CC @aclements @RLH

I was unable to reproduce this on linux/amd64.

failed MSpanList_Insert 0x47bd10 0x189339c26f5e9 0x0 0x0

This is really weird. In order this is: span, span.next, span.prev, span.list. But the span pointer looks too low (maybe Darwin mmaps really low?), span.next is clearly nonsense, and span.next should really never be non-nil and span.list is nil.

@donrudo, could you paste a few more failures? I want to look for patterns. Assuming the tracebacks are the same, just the first line of the failure is all I need.

@aclements It does not reproduce it in the same my environment, but

failed MSpanList_Insert 0x47bd10 0x189339c26f5e9 0x0 0x0

I think the cause is different, but it is similar to first ▶︎ 1. ./make.bash: error from this issue thread.
#16570

No problem, here is a number of logs I created by running this:

gogetlog.tar.gz

rm -rf Code/go
mkdir Code/go
cd Code/go

 go get -u -v -x 9fans.net/go/acme/... > log/acme.log 2>&1
 go get -u -v -x 9fans.net/go/acme/... > log/acme-2.log 2>&1
 go get -u -v -x 9fans.net/go/acme/... > log/acme-3.log 2>&1
 go get -u -v -x 9fans.net/go/acme/... > log/acme-4.log 2>&1
 go get -u -v -x 9fans.net/go/acme/... > log/acme-5.log 2>&1
 go get -u -v -x github.com/nsf/gocode/... > log/gocode-2.log 2>&1
 go get -u -v -x github.com/nsf/gocode/... > log/gocode-3.log 2>&1
 go get -u -v -x golang.org/x/tools/... > log/xtools.log 2>&1
 mv xtools.log xtools-stuck-hadtoCtrlC.log
 go get -u -v -x golang.org/x/tools/... > log/xtools.log 2>&1
 mv log/xtools.log log/xtools-stuck-hadtoCtrlC-2.log
 go get -u -v -x golang.org/x/tools/... > log/xtools.log 2>&1
 mv log/xtools.log log/xtools-stuck-hadtoCtrlC-3.log
 go get -u -v -x golang.org/x/tools/... > log/xtools.log 2>&1
 mv log/xtools.log log/xtools-stuck-hadtoCtrlC-4.log
 go get -u -v -x golang.org/x/tools/... > log/xtools.log 2>&1
 mv log/xtools.log log/xtools-5.log
 go get -u -v -x golang.org/x/tools/... > log/xtools.log 2>&1
 mv log/xtools.log log/xtools-6.log
 go get -u -v -x golang.org/x/tools/... > log/xtools.log 2>&1
 mv log/xtools.log log/xtools-7.log
 go get -u -v -x github.com/derekparker/delve/cmd/dlv/...  > log/dlv.log 2>&1
 mv log/dlv.log log/dlv-1.log
 go get -u -v -x github.com/derekparker/delve/cmd/dlv/...  > log/dlv-2.log 2>&1

@zchee, thanks for the pointer. This does appear to be memory corruption, similar to #16570.

@donrudo, thanks for the logs. Digging into this more, it looks like Darwin does in fact allocate mspans from very low addresses, so the low span pointer is not a surprise. But if the span.next pointer were actually pointing to a span, it would also be a low value. That means this is almost certainly memory corruption. This is odd, because the mspans are in a very "stable" region of memory compared to the heap (once an mspan is allocated, it never gets freed, so the memory is type-stable) and are nowhere near the Go heap, so a bad heap pointer is unlikely to cause this. They are, however, close to the data segment (global variables) and intermingled with some runtime-internal data structures.

Also supporting the memory corruption theory is that one of your logs (xtools-5.log) shows a crash following a pointer in workbuf structure.

Also interesting is that the bad span.next value does look like a pointer (it's a "largish" 64-bit number, so it's probably not an int, and it's clearly not byte data). They're also all in roughly the same region: 0x[78]xxxxxxxxxx (even the workbuf crash has a pointer in this region if you undo the lfstack mangling). However, this isn't a Go pointer of any sort; it's nowhere near low enough to point to text, data, or runtime structures, and it doesn't have the right pattern to be a heap or heap bitmap pointer.

Also interesting is that the bad span.next value does look like a pointer

It's not a pointer. It's nanoseconds since roughly 2016-10-25T15:15 UTC. I have no idea what the significance of that epoch is, but the progression of the span.next values in the panics correlates precisely with mtimes of your log files.

/cc @quentinmit. Random timestamp values corrupting memory on Darwin. Sounds right up your alley. :)

Are you absolutely sure that the go tool you are running is one built from 1.7.3? Because this is what I would expect to see when running an earlier version of the go tool on Sierra. For example, what does go version print?

This is exactly consistent with using a Go version <1.7. "Nanoseconds since roughly 2016-10-25T15:15" sounds like the value of mach_absolute_time. Sierra started writing the value of mach_absolute_time into *rdx, which prior to 1.7 contained garbage. Now we put 0 there.

@donrudo You said that you are running 1.7.3 from brew, but I noticed that your logs are using GOROOT=/usr/local/go. That's where our .pkg installer places go; do you happen to have an older Go installed there, and it's shadowing the Go 1.7.3 from brew on your $PATH?

Maybe try running something like

which -a go
go version

to confirm that it's really 1.7.3.

@quentinmit is right, after running his commands I saw this:

Code/templates » which -a go
go version
/usr/local/bin/go
/usr/local/go/bin/go
go version go1.7.3 darwin/amd64

the second /usr/local/go/bin/go, the same as my $GOROOT pointed to an old 1.6.1 setup.

After fixing the $GOROOT to make it point to the correct brew install ( /usr/local/Cellar/go/1.7.3/libexec ) all the libraries were able to be correctly built.

sorry for making you lose your time with this.