Skip to content

runtime: map iteration not random in Go 1.3 #8410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
remyoudompheng opened this issue Jul 22, 2014 · 9 comments
Closed

runtime: map iteration not random in Go 1.3 #8410

remyoudompheng opened this issue Jul 22, 2014 · 9 comments
Milestone

Comments

@remyoudompheng
Copy link
Contributor

What steps will reproduce the problem?
1. Run the attached tests and benchmarks.

http://play.golang.org/p/KvcHbNlwia

What is the expected output? What do you see instead?

Expected: no changes from Go 1.2

PASS
BenchmarkFill10  1000000          2257 ns/op
BenchmarkFill100      100000         20675 ns/op
BenchmarkFill1000      10000        264515 ns/op
BenchmarkFill10000      1000       2432130 ns/op
BenchmarkFill100000      100      25084435 ns/op
BenchmarkPop10    500000          3868 ns/op
BenchmarkPop100    50000         38456 ns/op
BenchmarkPop1000        5000        586389 ns/op
BenchmarkPop10000        200       8301058 ns/op
BenchmarkPop100000        10     193763167 ns/op
ok      testmap 25.412s

Instead:

--- FAIL: TestMap (0.00s)
    a_test.go:34: [192 193 194 195 196 197 198 199 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47]
    a_test.go:35: constant iteration order after 80 tries

BenchmarkFill10   500000          2413 ns/op
BenchmarkFill100      100000         22547 ns/op
BenchmarkFill1000       5000        281903 ns/op
BenchmarkFill10000       500       2606072 ns/op
BenchmarkFill100000       50      26541367 ns/op
BenchmarkPop10    300000          4459 ns/op
BenchmarkPop100    20000         68517 ns/op
BenchmarkPop1000         300       5222174 ns/op
BenchmarkPop10000          3     396486260 ns/op
BenchmarkPop100000         1    35560982673 ns/op

Please use labels and text to provide additional information.

Attachments:

  1. a_test.go (1612 bytes)
@remyoudompheng
Copy link
Contributor Author

Comment 1:

The patch
changeset:   18847:6ea672cbfe43
user:        Josh Bleecher Snyder <josharian@gmail.com>
date:        Tue Jan 14 12:54:05 2014 -0800
summary:     runtime: change map iteration randomization to use intra-bucket offset
changed randomization to cyclically shuffle each bucket, but doesn't start iteration at
a random bucket. As a consequence When a map is very sparse (1 item per bucket),
randomization is ineffective.
Another issue is that a "pop" pattern: deleting an element of the map after first
iteration, will leave the map in a very unbalanced state. Popping all elements will take
a quadratic time instead of something like O(n log n) or maybe a bit more as it is in Go
1.2.

@bradfitz
Copy link
Contributor

Comment 2:

Status changed to Accepted.

@josharian
Copy link
Contributor

Comment 3:

Is there a real-world style program that this shows up in?
If we want to fix this by randomizing both the offset and the starting bucket (which has
a minor performance cost), there's an implementation of that in Patch Set 5 of CL
47370043.

@ianlancetaylor
Copy link
Member

Comment 4:

Labels changed: added release-go1.4.

@remyoudompheng
Copy link
Contributor Author

Comment 5:

The quadratic pattern happened as a visible performance regression in a real program. It
was possible to work it around by modifying the algorithm.

@randall77
Copy link
Contributor

Comment 6:

We're not shooting for perfect iteration randomness.  We only provide enough randomness
to make sure people aren't depending on iteration order.  We do it to ensure that we can
change the map implementation at some future time and not break people's incorrect code.
Yes, the iteration is not random if the first bucket has <=1 element.  That will only
happen with very sparse maps, which requires inserting and then deleting most elements. 
I'm not particularly worried about that corner case.
The O(n^2) pop problem is more worrying.  Starting at a random bucket as in 1.2 would
help.  Maybe better is to keep track of (a conservative approximation of) the minimum
nonempty bucket.

@randall77
Copy link
Contributor

Comment 7:

I've split out the O(n^2) performance problem into issue #8412.  Let's keep this bug
about the nonrandomness.

@gopherbot
Copy link
Contributor

Comment 8:

CL https://golang.org/cl/137560044 mentions this issue.

@josharian
Copy link
Contributor

Comment 9:

This issue was closed by revision d6cd230.

Status changed to Fixed.

@rsc rsc added this to the Go1.4 milestone Apr 14, 2015
@rsc rsc removed the release-go1.4 label Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 25, 2016
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jun 25, 2018
The behavior was fixed in CL 141270043. Add a test.

Fixes golang#8410.

LGTM=khr
R=khr, remyoudompheng
CC=golang-codereviews
https://golang.org/cl/137560044
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jul 9, 2018
The behavior was fixed in CL 141270043. Add a test.

Fixes golang#8410.

LGTM=khr
R=khr, remyoudompheng
CC=golang-codereviews
https://golang.org/cl/137560044
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jul 30, 2018
The behavior was fixed in CL 141270043. Add a test.

Fixes golang#8410.

LGTM=khr
R=khr, remyoudompheng
CC=golang-codereviews
https://golang.org/cl/137560044
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants