@bsiegert, does this line of the Anita script restrict the machine to 1G of memory always, even if the GCE VM has much more than that?

a = anita.Anita(
    anita.URL(find_latest_release("netbsd-8", arch)),
    workdir="work-NetBSD-%s" % arch,
    disk_size="4G",
    memory_size = "1G",
    persist=True)
child = a.boot()
anita.login(child)

But that still doesn't explain how NetBSD built & tested all of the standard library successfully and only failed at this one test/ directory fork/exec.

No, this is just the amount of memory to use in the staging qemu environment. The VM comes up with as much memory as configured. However, there might be a ulimit in place. When I log in as root on another NetBSD instance, I get

time          (-t seconds    ) unlimited
file          (-f blocks     ) unlimited
data          (-d kbytes     ) 262144
stack         (-s kbytes     ) 4096
coredump      (-c blocks     ) unlimited
memory        (-m kbytes     ) 7303556
locked memory (-l kbytes     ) 2434518
thread        (-r threads    ) 1024
process       (-p processes  ) 1024
nofiles       (-n descriptors) 1024
vmemory       (-v kbytes     ) unlimited
sbsize        (-b bytes      ) unlimited

run.bash bumps several types of limits before running tests. Does the bot do the same?

Setting the ulimits as in run.bash allows the test to complete.

netbsd-current$ go run -- nilptr.go
fork/exec /tmp/go-build338049794/command-line-arguments/_obj/exe/nilptr: cannot allocate memory
netbsd-current$ [ "$(ulimit -H -n)" = "unlimited" ] || ulimit -S -n $(ulimit -H -n)
netbsd-current$ [ "$(ulimit -H -d)" = "unlimited" ] || ulimit -S -d $(ulimit -H -d)
netbsd-current$ go run run.go -- nilptr.go
netbsd-current$

I kinda doubt the 1024 file descriptor limit was the problem (unless the NetBSD port is leaking fds).

So it's probably the -d (RLIMIT_DATA, data segment) limit of 256 MB, eh? Still weird that everything else passed, but I'll try.

And no, the buildlet doesn't use run.bash, so I'll make it bump the limits with setrlimit in the buildlet.

Change https://golang.org/cl/79955 mentions this issue: buildlet: raise ulimit for NetBSD

Yup, RLIMIT_DATA did the trick. Now NetBSD builders are sometimes passing. (sometimes they're hanging in the runtime tests, but that's unrelated)