runtime: gdb tests hang on go1.8beta2 gentoo ebuild

[williamh]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.8beta2 Linux/amd64

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

I attempted to install go1.8beta2 using our package manager and an ebuild I'm about to add to Gentoo. Testss are turned on which causes the package manager to attempt to run the Go test suite.

What did you expect to see?

A successful installation of Go with all tests passing.

What did you see instead?

The tests failed which aborts installation.

The build log is attached. Any guidance you can provide for troubleshooting this will be greatly appreciated. :-)

build-log.txt

[davecheney]

Can you try building Go outside portage and confirming that the build passes or fails without portarge in the mix?

…

On Wed, Dec 28, 2016 at 2:33 PM, William ***@***.***> wrote: Please answer these questions before submitting your issue. Thanks! What version of Go are you using (go version)? go1.8beta2 Linux/amd64 What operating system and processor architecture are you using (go env)? linux/amd64 What did you do? I attempted to install go1.8beta2 using our package manager and an ebuild I'm about to add to Gentoo. Testss are turned on which causes the package manager to attempt to run the Go test suite. What did you expect to see? A successful installation of Go with all tests passing. What did you see instead? The tests failed which aborts installation. The build log is attached. Any guidance you can provide for troubleshooting this will be greatly appreciated. :-) build-log.txt <https://github.com/golang/go/files/674988/build-log.txt> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#18442>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA-6kKVonfG0rltdXuLq7ttgnAJHGks5rMdhxgaJpZM4LWplO> .

[bradfitz]

Looks like your host is just slower than the tests assume.

Try setting GO_TEST_TIMEOUT_SCALE=2 or GO_TEST_TIMEOUT_SCALE=3 in your environment.

But really, the Gentoo package manager should probably just run make.bash instead of all.bash. The tests are mostly interesting for us and others working on Go itself.

[davecheney]

I agree that portage shouldn't run the tests, but with that said I don't think this is a timeout % go test -short net/http os reflect runtime ok net/http 2.893s ok os 0.227s ok reflect 0.053s ok runtime 11.374s ^ my machine. Which is 3x faster than this sample. I'd expect the runtime tests to finish on the sample machine in < 40 seconds.

…

On Wed, Dec 28, 2016 at 2:40 PM, Brad Fitzpatrick ***@***.***> wrote: Looks like your host is just slower than the tests assume. Try setting GO_TEST_TIMEOUT_SCALE=2 or GO_TEST_TIMEOUT_SCALE=3 in your environment. But really, the Gentoo package manager should probably just run make.bash instead of all.bash. The tests are mostly interesting for us and others working on Go itself. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18442 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA5ZnK708TbaYC22dLCOPCsO73AFEks5rMdo2gaJpZM4LWplO> .

[williamh]

Portage does not run the tests unless a user requests that it do so.
We run make.bash to build Go, then optionally "run.bash --no-rebuild" to run the tests.

I can make portage block users from running the tests if they pass outside portage, but I thought we should give the option to run them since your build automatically runs them. :-)

I will run a quick build outside of portage and let you know how it goes.

[williamh]

Hi all,

the tests pass if the build is run without portage in the mix.

If you want, I am willing to attempt to help troubleshoot. Otherwise, I can just block the tests so that portage does not run them.

Which do you prefer?

[williamh]

Hi all,

I ran the build including tests inside portage again, setting GO_TEST_TIMEOUT_SCALE=3 in the environment as suggested.

It failed again, the log is attached.

build-log1.txt

The one thing I can think of is, do the tests write files somewhere on the filesystem? If they do, where? is that path programmable in some way?

[davecheney]

They write to /tmp. You can overwrite it with TMPDIR

…

On Wed, 28 Dec 2016, 16:29 William ***@***.***> wrote: Hi all, I ran the build including tests inside portage again, setting GO_TEST_TIMEOUT_SCALE=3 in the environment as suggested. It failed again, the log is attached. build-log1.txt <https://github.com/golang/go/files/675080/build-log1.txt> The one thing I can think of is, do the tests write files somewhere on the filesystem? If they do, where? is that path programmable in some way? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18442 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA8oWIvNqLTNb3hviowwNDLlsKRSzks5rMfOxgaJpZM4LWplO> .

[williamh]

I pointed TMPDIR to the appropriate location and still got a failure.
The log is attached.
It definitely looks like some kind of timeout, but I'm not sure what is causing it.

build-log2.txt

[davecheney]

I don't think increasing the timeout further will help, the test is blocked on something that portage has done to the environment. Does dmesg show any indication that the kernel has blocked the test from running a child process, I saw a mention of os/waitid blocking waiting for something.

…

On Wed, Dec 28, 2016 at 6:09 PM, William ***@***.***> wrote: I pointed TMPDIR to the appropriate location and still got a failure. The log is attached. It definitely looks like some kind of timeout, but I'm not sure what is causing it. build-log2.txt <https://github.com/golang/go/files/675174/build-log2.txt> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18442 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA3X5mDmLCiMFTDfZl-gnK1v-wbR6ks5rMgsMgaJpZM4LWplO> .

[williamh]

I cleared dmesg so I wouldn't see any old messages then re-ran the build with tests enabled. Nothing appeared in dmesg.

[rsc]

The runtime test is hung waiting for gdb in TestGdbBacktrace. It runs:

gdb -nx -batch \
	-ex 'set startup-with-shell off' \
	-ex 'break main.eee' \
	-ex 'run' \
	-ex 'backtrace' \
	-ex 'continue' \
	a.exe

where a.exe is the result of compiling:

package main

//go:noinline
func aaa() bool { return bbb() }

//go:noinline
func bbb() bool { return ccc() }

//go:noinline
func ccc() bool { return ddd() }

//go:noinline
func ddd() bool { return f() }

//go:noinline
func eee() bool { return true }

var f = eee

func main() {
	_ = aaa()
}

Gdb is not exiting. Can you try running that command by hand and see why?

It is also hung waiting for gdb in TestGdbAutotmpTypes, which does a similar thing.

[rsc]

Still waiting for a reply.

[williamh]

I will look at this in the next 48 hours. :-)

[williamh]

I am now working with go1.8rc3.
When I ran this by hand, a exited normally. However, when I ran the test inside our package manager environment by copying a.go into the build environment and modifying the test functionality to compile it and run this gdb command, it still hung.
The only relevant output appears to be the following showing that a breakpoint was set:

Breakpoint 1 at 0x44d680: file /usr/lib/go/src/a/a.go, line 16.

After this output, it hangs. The only option I have is to kill the build.

[rsc]

It sounds like portage is breaking things somehow? Can you attach to gdb and see if it is stuck on something interesting?

[williamh]

I am attaching output from strace, which will show all of the system calls.
strace.txt

[rsc]

The last few lines of the trace are

stat("/var/tmp/portage/dev-lang/go-1.8_rc3/work/go/a.exe", {st_mode=S_IFREG|0755, st_size=960330, ...}) = 0
close(7)                                = 0
stat("/var/tmp/portage/dev-lang/go-1.8_rc3/work/go/a.exe", {st_mode=S_IFREG|0755, st_size=960330, ...}) = 0
personality(0xffffffff)                 = 0 (PER_LINUX)
personality(PER_LINUX|ADDR_NO_RANDOMIZE) = 0 (PER_LINUX)
personality(0xffffffff)                 = 0x40000 (PER_LINUX|ADDR_NO_RANDOMIZE)
vfork(

It looks like vfork starts and just hangs? That's kind of strange. Or maybe you're running strace without -f and it got very confused?

The truncation in the strace log does match the behavior you see without strace: it tells about the breakpoint but then when it's time to start the program, bad things.

I don't know much about portage. Is there some reason it might block vfork?

[minux]

If I have to guess, perhaps Gentoo's sandbox uses ptrace based sandboxing to restrict writes to only be inside the work area? If that's the case, then it could explain that why gdb doesn't function correctly under the ptrace sandbox. I think you can verify this by running gdb on a regular C program inside portage and see if gdb can correct set a breakpoint and stop on breakpoint hit.

[rsc]

Is there some environment variable we can use to detect that we're running as part of portage? It would be easy enough to disable the test once we believe we understand the root cause and that it's not worth fixing.

[williamh]

You are correct that our sandbox is the culprit. I ran the same test a little while ago disabling the sandbox and it worked fine. I do not know the answers to either of your questions myself, but I will find out asap tomorrow morning ( it is 21:48 here now).

[minux]

If the sandbox is indeed using ptrace, then an easy solution is: detect nonzero TracerPid in /proc/self/status. (or try if ptrace(PTRACE_TRACEME) returns EPERM, but it's hard to implement in Go because the syscall package doesn't expose the raw ptrace syscall.) However, I expect that strace(1)-based sendfile tests in net/http package will also fail if we're being ptraced.

[rsc]

@minux, the net/http tests are passing (see original build log). Maybe strace is not available in the portage chroot but gdb is?

[minux]

Perhaps. I think it's probably better for the Gentoo dev-lang/go ebuild to disable (ptrace-based) sandbox for the test phase. (We can check for ptrace in select tests, but without a builder, this is really easy to regress. And there're at least two tests affected: net/http's sendfile and runtime's gdb, perhaps there will be more in the future.)

[bradfitz]

FWIW, net/http does:

        if _, err := exec.LookPath("strace"); err != nil {
                t.Skip("skipping; strace not found in path")
        }
...
        if err := child.Start(); err != nil {
                t.Skipf("skipping; failed to start straced child: %v", err)
        }

[tw4452852]

Just did a test. Gdb breakpoint doesn't work in sandbox with go program.

$ gdb -nx /tmp/go-build089897835/a.exe
GNU gdb (Gentoo 7.10.1 vanilla) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /tmp/go-build089897835/a.exe...done.
(gdb) b main.eee
Breakpoint 1 at 0x44dc60: file /tmp/go-build089897835/main.go, line 17.
(gdb) r
Starting program: /tmp/go-build089897835/a.exe 
During startup program exited normally.
(gdb) info br
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x000000000044dc60 in main.eee at /tmp/go-build089897835/main.go:17
(gdb) bt
No stack.

But it works on a c program.

// main.c

#include <stdio.h>

int
main(int argc, const char *argv[])
{
        printf("hello world\n");
        return 0;
}

$ gdb -nx ~/tmp/test
GNU gdb (Gentoo 7.10.1 vanilla) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/tw/tmp/test...done.
(gdb) b main
Breakpoint 1 at 0x400565: file main.c, line 6.
(gdb) r
Starting program: /home/tw/tmp/test 

Breakpoint 1, main (argc=1, argv=0x7fffffffd668) at main.c:6
6               printf("hello world\n");
(gdb) bt
#0  main (argc=1, argv=0x7fffffffd668) at main.c:6
(gdb) 

[minux]

I can reproduce the problem on Gentoo. I will take a deeper look. It seems unrelated to ptrace sandboxing.

[williamh]

I just looked at the sandbox sources, and we are using ptrace.

[williamh]

@minux There is not a way for an ebuild to disable the sandbox. The only option I would have is to disable the test phase for dev-lang/go.

[bradfitz]

@williamh, did you see @rsc's comment above? #18442 (comment)

Is there a way to detect when we're running in your specific sandbox? Is there some environment variable set like HEY_YOURE_BUILDING_IN_GENTOO_SANDBOX=1?

[williamh]

@bradfitz I don't know of a specific variable you can use for that, but I'm researching it. I'll give you a definite answer asap. :-)

[williamh]

@bradfitz @minux @rsc I was just advised that I may be wrong about disabling the sandbox for the tests.
SANDBOX_ON=1 in the environment means the sandbox is active, and I may be able to turn it off with SANDBOX_ON=0 only for the tests. I am testing this now and I will update you.

[williamh]

It looks like SANDBOX_ON=0 does not currently disable the sandbox. I verified this and also saw a comment in the source that says once the sandbox is active it can't be deactivated.
If you want to skip the tests in our sandbox, the best way to do that is to skip them if SANDBOX_ON=1 is in the environment in case we decide to honor SANDBOX_ON=0 in the future.

[ALTree]

Is this still a problem on 1.11?

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)