Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/imports: concurrent map iteration and map write #20109

Closed
karalabe opened this issue Apr 25, 2017 · 4 comments
Closed

x/tools/imports: concurrent map iteration and map write #20109

karalabe opened this issue Apr 25, 2017 · 4 comments
Labels
FrozenDueToAge
Milestone
Unreleased

Comments

@karalabe
Copy link
Contributor

What version of Go are you using (go version)?

go version go1.8.1 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/work:/opt/google/cloud-sdk/platform/google_appengine/goroot/src"
GORACE=""
GOROOT="/opt/google/go"
GOTOOLDIR="/opt/google/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build044665358=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

We use goimports internally to resolve the imports for some auto generated Go code. The code snippets are part of our tests and have worked correctly for more than a year now without being modified, however I just now managed to crash the tests in goimports with a concurrent map read/write. I can't really provide a repro code since it never happened until now so it probably won't happen again too fast either.

We're using a vendored version at commit be0fcc31ae2332374e800dfff29b721c585b35df, so it might have been an issue fixed since then.

2017/04/25 14:14:41 goimports: scanning directory /work/src: permission denied
fatal error: concurrent map iteration and map write

goroutine 11 [running]:
runtime.throw(0x9b006a, 0x26)
	/opt/google/go/src/runtime/panic.go:596 +0x95 fp=0xc420057c68 sp=0xc420057c48
runtime.mapiternext(0xc420057e80)
	/opt/google/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc420057d18 sp=0xc420057c68
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.findImportGoPath(0xc4201313f0, 0x8, 0xc4201d35f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fix.go:737 +0x349 fp=0xc420057ef0 sp=0xc420057d18
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fixImports.func2(0x0, 0x0, 0xc4200214a0, 0xc4201313f0, 0x8, 0xc4201d35f0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fix.go:227 +0x7b fp=0xc420057fb0 sp=0xc420057ef0
runtime.goexit()
	/opt/google/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420057fb8 sp=0xc420057fb0
created by github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fixImports
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fix.go:233 +0x614

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4200692b0, 0x97aa12, 0xc, 0x9c9808, 0xc42022bd20)
	/opt/google/go/src/testing/testing.go:698 +0x2f4
testing.runTests.func1(0xc4200692b0)
	/opt/google/go/src/testing/testing.go:882 +0x67
testing.tRunner(0xc4200692b0, 0xc42022bde0)
	/opt/google/go/src/testing/testing.go:657 +0x96
testing.runTests(0xc42021a180, 0xe4e920, 0x2, 0x2, 0xe800000000950fc0)
	/opt/google/go/src/testing/testing.go:888 +0x2c1
testing.(*M).Run(0xc420053f20, 0xc42022bf20)
	/opt/google/go/src/testing/testing.go:822 +0xfc
main.main()
	github.com/ethereum/go-ethereum/accounts/abi/bind/_test/_testmain.go:46 +0xf7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/opt/google/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 6 [chan receive]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc420020ae0, 0xc420020a20)
	/work/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:36 +0x5a
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree
	/work/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:29 +0x176

goroutine 7 [chan receive]:
github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc420020ae0, 0xc420020a80)
	/work/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:81 +0x94
created by github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify.newNonrecursiveTree
	/work/src/github.com/ethereum/go-ethereum/vendor/github.com/rjeczalik/notify/tree_nonrecursive.go:30 +0x1a2

goroutine 8 [chan receive]:
github.com/ethereum/go-ethereum/vendor/github.com/rcrowley/go-metrics.(*meterArbiter).tick(0xe8e7e0)
	/work/src/github.com/ethereum/go-ethereum/vendor/github.com/rcrowley/go-metrics/meter.go:221 +0x4c
created by github.com/ethereum/go-ethereum/vendor/github.com/rcrowley/go-metrics.NewMeter
	/work/src/github.com/ethereum/go-ethereum/vendor/github.com/rcrowley/go-metrics/meter.go:40 +0x127

goroutine 9 [chan receive]:
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fixImports(0xc420017d00, 0xc420077100, 0x0, 0x0, 0x57, 0x60, 0xc420055b10, 0xc420077100, 0x0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fix.go:237 +0x6ba
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.Process(0x0, 0x0, 0xc4200213e0, 0x57, 0x60, 0xc420055b10, 0x57, 0x0, 0x0, 0x0, ...)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/imports.go:58 +0x719
github.com/ethereum/go-ethereum/accounts/abi/bind.TestBindings(0xc420069380)
	/work/src/github.com/ethereum/go-ethereum/accounts/abi/bind/bind_test.go:462 +0x151
testing.tRunner(0xc420069380, 0x9c9808)
	/opt/google/go/src/testing/testing.go:657 +0x96
created by testing.(*T).Run
	/opt/google/go/src/testing/testing.go:697 +0x2ca

goroutine 15 [runnable]:
runtime.goexit1()
	/opt/google/go/src/runtime/proc.go:2319 +0x4e
runtime.goexit()
	/opt/google/go/src/runtime/asm_amd64.s:2198 +0x6
created by github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fastWalk
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:63 +0x205

goroutine 16 [runnable]:
syscall.Syscall(0xd9, 0x4, 0xc420283e48, 0x2000, 0x1e0, 0x2000, 0x0)
	/opt/google/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Getdents(0x4, 0xc420283e48, 0x2000, 0x2000, 0x0, 0x0, 0x0)
	/opt/google/go/src/syscall/zsyscall_linux_amd64.go:508 +0x55
syscall.ReadDirent(0x4, 0xc420283e48, 0x2000, 0x2000, 0xc400000000, 0x0, 0x0)
	/opt/google/go/src/syscall/syscall_linux.go:761 +0x49
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.readDir(0xc420266e10, 0x28, 0xc420285ec0, 0x0, 0x0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk_unix.go:37 +0x1b3
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.(*walker).walk(0xc4201d36e0, 0xc420266e10, 0x28, 0x1, 0x0, 0x0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:171 +0x6b
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.(*walker).doWork(0xc4201d36e0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:112 +0x124
created by github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fastWalk
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:63 +0x205

goroutine 18 [runnable]:
runtime.goexit1()
	/opt/google/go/src/runtime/proc.go:2319 +0x4e
runtime.goexit()
	/opt/google/go/src/runtime/asm_amd64.s:2198 +0x6
created by github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fastWalk
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:63 +0x205

goroutine 19 [runnable]:
runtime.goexit1()
	/opt/google/go/src/runtime/proc.go:2319 +0x4e
runtime.goexit()
	/opt/google/go/src/runtime/asm_amd64.s:2198 +0x6
created by github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fastWalk
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:63 +0x205

goroutine 20 [runnable]:
runtime.goexit1()
	/opt/google/go/src/runtime/proc.go:2319 +0x4e
runtime.goexit()
	/opt/google/go/src/runtime/asm_amd64.s:2198 +0x6
created by github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fastWalk
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:63 +0x205

goroutine 21 [runnable]:
path/filepath.Dir(0xc420322cc0, 0x5e, 0xc420295cc8, 0x4466b7)
	/opt/google/go/src/path/filepath/path.go:464 +0x98
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.scanGoDirs.func1(0xc420322cc0, 0x5e, 0x0, 0x9747a6, 0x1)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fix.go:522 +0x59
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.(*walker).onDirEnt(0xc4201d36e0, 0xc420261590, 0x45, 0xc4203cb2a0, 0x18, 0x0, 0x0, 0x0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:145 +0xbb
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.(*walker).(github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.onDirEnt)-fm(0xc420261590, 0x45, 0xc4203cb2a0, 0x18, 0xc400000000, 0x18, 0x0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:171 +0x5a
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.readDir(0xc420261590, 0x45, 0xc420297ec0, 0x0, 0x0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk_unix.go:64 +0x463
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.(*walker).walk(0xc4201d36e0, 0xc420261590, 0x45, 0x1, 0x0, 0x0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:171 +0x6b
github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.(*walker).doWork(0xc4201d36e0)
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:112 +0x124
created by github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports.fastWalk
	/work/src/github.com/ethereum/go-ethereum/vendor/golang.org/x/tools/imports/fastwalk.go:63 +0x205
@gopherbot gopherbot added this to the Unreleased milestone Apr 25, 2017
@mvdan
Copy link
Member

mvdan commented Apr 25, 2017

We're using a vendored version at commit be0fcc3, so it might have been an issue fixed since then.

Did you try the latest version from master?

@karalabe
Copy link
Contributor Author

The code snippets are part of our tests and have worked correctly for more than a year now without being modified [...] I can't really provide a repro code since it never happened until now so it probably won't happen again too fast either.

That being said, I think it has to do with low disk space. I actually hit the exact same issue rerunning my tests, but after freeing up some disk space all tests work correctly. I'm guessing some error path gets hit that doesn't have a proper synchronization. I'll try to repro with latets master if I can.

@mvdan
Copy link
Member

mvdan commented Apr 25, 2017

Could be that a disk error is found and the code keeps on going, causing a panic further down? Instead of just stopping, I mean.

@gopherbot
Copy link

CL https://golang.org/cl/41681 mentions this issue.

@golang golang locked and limited conversation to collaborators Apr 25, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@karalabe @mvdan @gopherbot