I'm using a prebuilt toolchain stored in:

http://cdn.netbsd.org/pub/NetBSD/misc/kamil/llvm-clang-compilerrt-lldb-7.0.0beta_2018-02-28.tar.bz2

Generated on NetBSD/8.9.12. It contains the pristine sources from HEAD of Clang, LLVM, LLDB, compiler-rt patched with around 1500 lines of local diffs that is still not cleaned up and upstreamed. [to be clear: local patches are mainly with interceptors for MSan, nothing in particular pending for TSan]

We will need to redo the same for Memory Sanitizer in Go on NetBSD/amd64, as it's already stable enough and works for C and C++ programs.

@ceseo needs to push a change upstream for LLVM on ppc64le for Go. We have some internal users testing it.

FTR, here is an attempt to fix racebuild:
https://go-review.googlesource.com/c/build/+/101137

@laboger done. It's in revision 330122.

Is there need for changes in NetBSD Go files?

@krytarowski changes are needed in multiple places (e.g. adding support for netbsd to buildgo.sh in llvm repo, to go tool to accept -race flag on netbsd, to racebuild.go to build runtime on netbsd). They probably will be mostly "doing the same as freebsd", but still somebody needs to do them and then do testing.

@dvyukov I'm not sure of the order in which I submit my changes to add support for ppc64le. The code in x/build/cmd/racebuild/racebuild.go expects that the golang race support is there with the -race flag enabled, but it won't build if it doesn't have the syso file.

And I don't know how to run something with gomote to test the change to build the syso file.

@laboger I think it would be a good start to prepare all required changes in a single change (except for racebuild.go because you can't test it and because it's in a different repository). Then we can figure out how to split them. It will be easier when we can see all required changes.

Last time I was told that someone internal shall add the files, generate -race files etc. I can help to configure a NetBSD VM and instruct to build compiler-rt, is this enough?

We have NetBSD VMs. We just need instructions.

Here are the instructions in x/build/cmd/racebuild for all the other OSes:

https://github.com/golang/build/blob/master/cmd/racebuild/racebuild.go#L62

var platforms = []*Platform{
	&Platform{
		OS:   "netbsd",
		Arch: "amd64",
		Type: "netbsd-amd64-race",
		Script: `#!/usr/bin/env bash
set -e
git clone https://go.googlesource.com/go
git clone http://llvm.org/git/compiler-rt.git
(cd compiler-rt && git checkout $REV)
(cd compiler-rt/lib/tsan/go && CC=clang ./buildgo.sh)
cp compiler-rt/lib/tsan/go/race_netbsd_amd64.syso go/src/runtime/race
(cd go/src && ./race.bash)
			`,

You need to get git, bash and mozilla-rootcerts from pkgsrc/pkgin/pkg_add. For mozilla-rootcerts follow post-installation notes.

1. Manual build:

• feetch pkgsrc into /usr/pkgsrc
• cd /usr/pkgsrc/shells/bash && make install
• cd /usr/pkgsrc/devel/git-base && make install
• cd /usr/pkgsrc/security/mozilla-rootcerts && make install

2. pkgin

• Setup it accordingly to http://pkgin.net/
• pkgin install git-base mozilla-rootcerts bash

3. pkg_add

• setup PKG_PATH (environment variable)
• pkg_add git-base mozilla-rootcerts bash

In general you need clang to build compiler-rt, it's in pkgsrc under name 'clang'.

A general note, for (ASan/)TSan/MSan there is need to disable ASLR:

sysctl -w security.pax.aslr.enabled=0

@bcmills @FiloSottile, one of you want to work on this?

/cc @andybons

CC @bsiegert

Assigning to @bcmills. Bryan let us know if you have any issue with this :)

@bcmills, note that this bug is about several things:

• refresh existing race-enabled ports' syso files from upstream llvm
• add linux/ppc64le (https://go-review.googlesource.com/c/go/+/107935)
• add netbsd/amd64

Regarding NetBSD, we just tagged 8.0RC1.. please use this version.

http://www.nerv.org/netbsd/?q=id:20180419T155522Z.e759b7514ca9f1cf6bcf3bbc27ea04e91d2ceab9

ISO images are still not published, but images with a little bit older snapshot are already available on:

http://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-8/201804161630Z/images/

MSan has been ported to NetBSD too, but I've not been fiddling with a go variation so far.

@krytarowski, we don't have time to do much NetBSD-specific stuff here. At best we'll run the commands you gave us on our existing NetBSD 8 VM image and if it produces a usable & fully passing object file, we can ship it. But once we hit any problems, we'll need to punt until the next release.

I'll send a CL to update the builders once the binaries for 8.0_RC1 land.

@bsiegert could you install there the mentioned programs as well? bash, clang, git-base ...

It has most of these. See https://github.com/golang/build/blob/master/env/netbsd-amd64/mkvm.py:

• bash
• curl
• git
• mozilla-rootcerts
• go14
• emacs25-nox11
• vim
• screen

So you would like clang in addition?

Yes.

Change https://golang.org/cl/109035 mentions this issue: env: update netbsd builders to 8.0_RC1

Change https://golang.org/cl/107935 mentions this issue: runtime/race: implement race detector for ppc64le

I've got everything except for Windows building. Windows appears to be broken since https://reviews.llvm.org/D28596 (12 Jan. 2017).

I hope that NetBSD/amd64 is included in everything!

@krytarowski, well, we're not going to ship anything that we didn't build from source in an automated way. That means no slurping down random pre-built binaries. Is the NetBSD stuff ready to be built from source from a script?

I think it's ready.

Change https://golang.org/cl/112875 mentions this issue: cmd/racebuild: clarify LLVM revision in usage comment

Change https://golang.org/cl/112881 mentions this issue: cmd/racebuild: add linux/ppc64le

Change https://golang.org/cl/112876 mentions this issue: cmd/racebuild: simplify debugging

Change https://golang.org/cl/112878 mentions this issue: cmd/racebuild: destroy in-flight gomotes on interrupt

Change https://golang.org/cl/112877 mentions this issue: cmd/racebuild: stream stderr from gomote when building a single platform

Change https://golang.org/cl/112880 mentions this issue: cmd/racebuild: add netbsd/amd64

Change https://golang.org/cl/112879 mentions this issue: cmd/racebuild: update darwin to use 10.12

Change https://golang.org/cl/112895 mentions this issue: cmd/racebuild: install Git and GCC on the Windows builder

Change https://golang.org/cl/112896 mentions this issue: runtime/race: update most syso files to compiler-rt fe2c72

syso files are updated. The Windows LLVM is a bit old because that part of the LLVM build is broken on Windows.