Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/vendor/github.com/google/pprof/internal/binutils: test failure inside a container: TestObjFile #23888

Closed
nehaljwani opened this issue Feb 17, 2018 · 14 comments
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@nehaljwani
Copy link
Contributor

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go 1.10

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GORACE=""
GOROOT="/opt/conda/conda-bld/go_1518860727469/work/go"
GOTMPDIR=""
GOTOOLDIR="/opt/conda/conda-bld/go_1518860727469/work/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build979971167=/tmp/go-build"

What did you do?

Ran this script: https://github.com/conda-forge/go-feedstock/blob/master/recipe/build.sh

What did you expect to see?

All tests should've passed

What did you see instead?

--- FAIL: TestObjFile (0.05s)
        binutils_test.go:231: SourceLine: unexpected error write |1: broken pipe
FAIL
FAIL    cmd/vendor/github.com/google/pprof/internal/binutils    0.224s
@nehaljwani nehaljwani changed the title Build failure inside a container Test failure inside a container: TestObjFile Feb 17, 2018
@davecheney
Copy link
Contributor

davecheney commented Feb 17, 2018 via email

@davecheney davecheney added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 17, 2018
@nehaljwani
Copy link
Contributor Author

Container is run using:

sudo docker run --rm --name conda-forge -it --net=host --entrypoint=/bin/bash docker.io/condaforge/linux-anvil:latest

Limits (nothing explicitly configured):

[root@centos7-nwani /]# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30399
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

@davecheney
Copy link
Contributor

At a guess your container ran out of ram and the kernel oom killed one of the tests child processes. My rule of thumb is ./all.bash needs ~768 mb of ram per 64bit core.

@davecheney
Copy link
Contributor

Dmesg on your host system might give you a clue

@nehaljwani
Copy link
Contributor Author

I have 8GB RAM on the host machine

No trace of OOM in dmesg.

@davecheney
Copy link
Contributor

Is docker applying any limits to the amount of memory a container can use?

@nehaljwani
Copy link
Contributor Author

No, no cgroup limits. I even ran the container with --privileged but I come across the same error.

Interestingly, the CI build on CircleCI passed.

@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Feb 17, 2018
@ianlancetaylor ianlancetaylor changed the title Test failure inside a container: TestObjFile cmd/vendor/github.com/google/pprof/internal/binutils: test failure inside a container: TestObjFile Feb 17, 2018
@ianlancetaylor
Copy link
Contributor

This test is going to start a separate program, either llvm-symbolizer or addr2line, and communicate with it over a pipe. Are either of those programs in your container on the default PATH? Are they standard versions of those programs?

@joenall
Copy link

joenall commented Feb 26, 2018

I see the same failure on RHEL 6.9. No OOM in dmesg. Very consistent. If someone will point out how to run the test in isolation without installing 1.10, I'll strace it.

@esproul
Copy link

esproul commented Mar 14, 2018

@joenall this appears to work for me, for running that one test. After a run of all.bash where the failure appears:

$ cd /path/to/go-1.10/src
$ GOPATH=/home/me/go ../bin/go test cmd/vendor/github.com/google/pprof/internal/binutils
--- FAIL: TestObjFile (0.00s)
	binutils_test.go:231: SourceLine: unexpected error write |1: broken pipe
FAIL
FAIL	cmd/vendor/github.com/google/pprof/internal/binutils	0.014s

I'm also on CentOS 6, in a QEMU VM with 1 CPU and 4G RAM. No evidence of OOM, and I have /usr/bin/addr2line, but not llvm-symbolizer. This build was done with the default gcc in CentOS 6.

If I use a newer gcc (e.g., from the devtoolset collection), the test succeeds, so maybe it's to do with old gcc/libgcc, since EL6 has GCC 4.4.x?

@grizzlybears
Copy link

I see the same failure on CentOS 6.9 too, yes, very consistent.

Details are:
$ cat /etc/redhat-release
CentOS release 6.9 (Final)

$ gcc --version
gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)

@nehaljwani
Copy link
Contributor Author

I finally figured out why this is happening.

The command which actually fails is:

120807 3650  execve("/usr/bin/addr2line", ["/usr/bin/addr2line", "-aif", "-e", "testdata/exe_linux_64"], [/* 15 vars */] <unfinished ...>

The source code for the generation of this command is available at:
https://github.com/google/pprof/blob/a74ae6fb3cd7047c79272e3ea0814b08154a2d3c/internal/binutils/addr2liner.go#L92

Now, the add2line file comes from the package: binutils

The command fails, because, on CentOS 6.9, the version of binutils is 2.20 and according to https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=binutils/NEWS;hb=refs/tags/binutils-2_27 , the -a argument to the utility addr2line was added in binutils version 2.21

So, to get around this, I built binutils from source and added the binaries built by it to PATH and then ran the tests, which passed.

@nehaljwani
Copy link
Contributor Author

Closing, as the only solution that seems to work is to get a newer binutils.

@ianlancetaylor
Copy link
Contributor

I think the test should be fixed to detect this case and skip the test when it happens.

However, this needs to be fixed in the pprof code, not the Go code. When the pprof code is fixed, it can be re-vendored into the Go code.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

7 participants