Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: docker build failure with go-1.6.2/musl libc on armhf #16081

Closed
ncopa opened this issue Jun 16, 2016 · 9 comments
Closed

runtime: docker build failure with go-1.6.2/musl libc on armhf #16081

ncopa opened this issue Jun 16, 2016 · 9 comments
Labels
FrozenDueToAge

Comments

@ncopa
Copy link
Contributor

ncopa commented Jun 16, 2016 •
edited
Loading

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?

go version go1.6.2 linux/arm

  1. What operating system and processor architecture are you using (go env)?

Alpine Linux edge.
Linux ncdev-edge-armhf 4.1.15-0-grsec #1-Alpine SMP Wed Dec 16 15:13:43 GMT 2015 armv7l Linux

On wandboard quad with 2G ram and 4G swap.

GOARCH="arm"
GOBIN=""
GOEXE=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_arm"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
  1. What did you do?
    If possible, provide a recipe for reproducing the error.
    A complete runnable program is good.
    A link on play.golang.org is best.

tried to build docker-1.11.2 as:

        export AUTO_GOPATH=1
        export DOCKER_GITCOMMIT=$_gitcommit
        export DOCKER_BUILDTAGS=$_buildtags
        unset CC # prevent possible ccache issues

        # docker
        cd "$builddir"
        ./hack/make.sh dynbinary || return 1

Full build script is found here: http://git.alpinelinux.org/cgit/aports/tree/community/docker/APKBUILD#n62

  1. What did you expect to see?

docker sucessfully built.

  1. What did you see instead?
---> Making bundle: dynbinary (in bundles/1.11.2/dynbinary)
Building: bundles/1.11.2/dynbinary/docker-1.11.2
# github.com/ugorji/go/codec
runtime: memory allocated by OS (0x7f092000) not in usable range [0x8f958000,0xffffffff)
fatal error: out of memory

runtime stack:
runtime.throw(0x40ba28, 0xd)
        /usr/lib/go/src/runtime/panic.go:547 +0x78
runtime.(*mcache).refill(0x7f8174b0, 0xe, 0x7f1c1e80)
        /usr/lib/go/src/runtime/mcache.go:121 +0xfc
runtime.mallocgc.func2()
        /usr/lib/go/src/runtime/malloc.go:642 +0x28
runtime.systemstack(0x82dd00)
        /usr/lib/go/src/runtime/asm_arm.s:247 +0x80
runtime.mstart()
        /usr/lib/go/src/runtime/proc.go:1051

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/lib/go/src/runtime/asm_arm.s:192 +0x4 fp=0xa3e7f6b4 sp=0xa3e7f6b0
runtime.mallocgc(0xd0, 0x3e42e8, 0x0, 0xaf8d7ee0)
        /usr/lib/go/src/runtime/malloc.go:643 +0x930 fp=0xa3e7f724 sp=0xa3e7f6b4
runtime.newobject(0x3e42e8, 0xaf8d7ee0)
        /usr/lib/go/src/runtime/malloc.go:781 +0x50 fp=0xa3e7f738 sp=0xa3e7f724
cmd/internal/obj/arm.preprocess(0x8f964300, 0xa8c07aa0)
        /usr/lib/go/src/cmd/internal/obj/arm/obj5.go:431 +0xc14 fp=0xa3e7fa7c sp=0xa3e7f738
cmd/internal/obj.Flushplist(0x8f964300)
        /usr/lib/go/src/cmd/internal/obj/objfile.go:299 +0x2b8 fp=0xa3e7fbd8 sp=0xa3e7fa7c
cmd/internal/obj.Writeobjdirect(0x8f964300, 0xa3cad560)
        /usr/lib/go/src/cmd/internal/obj/objfile.go:114 +0x1c fp=0xa3e7fbe4 sp=0xa3e7fbd8
cmd/compile/internal/gc.dumpobj()
        /usr/lib/go/src/cmd/compile/internal/gc/obj.go:88 +0xca8 fp=0xa3e7fd30 sp=0xa3e7fbe4
cmd/compile/internal/gc.Main()
        /usr/lib/go/src/cmd/compile/internal/gc/lex.go:495 +0x260c fp=0xa3e7fefc sp=0xa3e7fd30
cmd/compile/internal/arm.Main()
        /usr/lib/go/src/cmd/compile/internal/arm/galign.go:92 +0x538 fp=0xa3e7ff08 sp=0xa3e7fefc
main.main()
        /usr/lib/go/src/cmd/compile/main.go:34 +0x15c fp=0xa3e7ff74 sp=0xa3e7ff08
runtime.main()
        /usr/lib/go/src/runtime/proc.go:188 +0x2b4 fp=0xa3e7ff9c sp=0xa3e7ff74
runtime.goexit()
        /usr/lib/go/src/runtime/asm_arm.s:990 +0x4 fp=0xa3e7ff9c sp=0xa3e7ff9c
@ncopa
Copy link
Contributor Author

ncopa commented Jun 16, 2016

What is a bit strange is that it built earlier, with go1.6.1, but after upgrading to go1.6.2 it no longer builds. What is more strange is that i have downgraded various times and now it no longer build with go 1.6.1 either. I get different error messages with different go versions and different docker versions. All seems to be related memory management.

building docker 1.11.1 with go-1.6.2:

---> Making bundle: dynbinary (in bundles/1.11.1/dynbinary)
Building: bundles/1.11.1/dynbinary/docker-1.11.1
# github.com/ugorji/go/codec
fatal error: addspecial on invalid pointer

runtime stack:
runtime.throw(0x41d818, 0x1d)
        /usr/lib/go/src/runtime/panic.go:547 +0x78
runtime.addspecial(0x9f07d8c0, 0x6ee245e8, 0x74)
        /usr/lib/go/src/runtime/mheap.go:1004 +0xcc
runtime.setprofilebucket(0x9f07d8c0, 0x6e236960)
        /usr/lib/go/src/runtime/mheap.go:1158 +0xac
runtime.mProf_Malloc.func1()
        /usr/lib/go/src/runtime/mprof.go:249 +0x24
runtime.systemstack(0x7f0b8000)
        /usr/lib/go/src/runtime/asm_arm.s:247 +0x80
runtime.mstart()
        /usr/lib/go/src/runtime/proc.go:1051

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/lib/go/src/runtime/asm_arm.s:192 +0x4 fp=0x936d56a4 sp=0x936d56a0
runtime.mProf_Malloc(0x9f07d8c0, 0xc0)
        /usr/lib/go/src/runtime/mprof.go:250 +0x134 fp=0x936d5764 sp=0x936d56a4
runtime.profilealloc(0x7f0c6500, 0x9f07d8c0, 0xc0)
        /usr/lib/go/src/runtime/malloc.go:794 +0x38 fp=0x936d5770 sp=0x936d5764
runtime.mallocgc(0xc0, 0x3da658, 0x0, 0x18)
        /usr/lib/go/src/runtime/malloc.go:712 +0x640 fp=0x936d57e0 sp=0x936d5770
runtime.newarray(0x3da658, 0x8, 0x2c1738)
        /usr/lib/go/src/runtime/malloc.go:778 +0xe4 fp=0x936d5800 sp=0x936d57e0
runtime.growslice(0x3a7b80, 0x9f079920, 0x4, 0x4, 0x5, 0x0, 0x0, 0x0)
        /usr/lib/go/src/runtime/slice.go:100 +0x2a8 fp=0x936d5838 sp=0x936d5800
cmd/internal/obj/arm.asmout(0x7f0a6300, 0x5e231ad0, 0x538148, 0x936d599c, 0x9, 0x9)
        /usr/lib/go/src/cmd/internal/obj/arm/asm5.go:1642 +0xf3c fp=0x936d593c sp=0x936d5838
cmd/internal/obj/arm.span5(0x7f0a6300, 0x97ebbe00)
        /usr/lib/go/src/cmd/internal/obj/arm/asm5.go:756 +0x85c fp=0x936d5a7c sp=0x936d593c
cmd/internal/obj.Flushplist(0x7f0a6300)
        /usr/lib/go/src/cmd/internal/obj/objfile.go:300 +0x2d8 fp=0x936d5bd8 sp=0x936d5a7c
cmd/internal/obj.Writeobjdirect(0x7f0a6300, 0x935842f0)
        /usr/lib/go/src/cmd/internal/obj/objfile.go:114 +0x1c fp=0x936d5be4 sp=0x936d5bd8
cmd/compile/internal/gc.dumpobj()
        /usr/lib/go/src/cmd/compile/internal/gc/obj.go:88 +0xca8 fp=0x936d5d30 sp=0x936d5be4
cmd/compile/internal/gc.Main()
        /usr/lib/go/src/cmd/compile/internal/gc/lex.go:495 +0x260c fp=0x936d5efc sp=0x936d5d30
cmd/compile/internal/arm.Main()
        /usr/lib/go/src/cmd/compile/internal/arm/galign.go:92 +0x538 fp=0x936d5f08 sp=0x936d5efc
main.main()
        /usr/lib/go/src/cmd/compile/main.go:34 +0x15c fp=0x936d5f74 sp=0x936d5f08
runtime.main()
        /usr/lib/go/src/runtime/proc.go:188 +0x2b4 fp=0x936d5f9c sp=0x936d5f74
runtime.goexit()
        /usr/lib/go/src/runtime/asm_arm.s:990 +0x4 fp=0x936d5f9c sp=0x936d5f9c

buidling docker 1.11.1 with go 1.6.1:

---> Making bundle: dynbinary (in bundles/1.11.1/dynbinary)
Building: bundles/1.11.1/dynbinary/docker-1.11.1
# github.com/ugorji/go/codec
runtime: memory allocated by OS (0x83cbd000) not in usable range [0x94582000,0xffffffff)
fatal error: out of memory

runtime stack:
runtime.throw(0x40ba28, 0xd)
        /usr/lib/go/src/runtime/panic.go:530 +0x78
runtime.(*mcache).refill(0x84442258, 0x11, 0x83df1948)
        /usr/lib/go/src/runtime/mcache.go:121 +0xfc
runtime.mallocgc.func2()
        /usr/lib/go/src/runtime/malloc.go:642 +0x28
runtime.systemstack(0x945a0a00)
        /usr/lib/go/src/runtime/asm_arm.s:247 +0x80
runtime.mstart()
        /usr/lib/go/src/runtime/proc.go:1048

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/lib/go/src/runtime/asm_arm.s:192 +0x4 fp=0xa887781c sp=0xa8877818
runtime.mallocgc(0x100, 0x0, 0x3, 0x100)
        /usr/lib/go/src/runtime/malloc.go:643 +0x930 fp=0xa887788c sp=0xa887781c
runtime.rawmem(0x100, 0x40)
        /usr/lib/go/src/runtime/malloc.go:809 +0x30 fp=0xa88778a0 sp=0xa887788c
runtime.growslice(0x3a7828, 0xb4500000, 0x80, 0x80, 0x81, 0x0, 0x0, 0x0)
        /usr/lib/go/src/runtime/slice.go:95 +0x230 fp=0xa88778d8 sp=0xa88778a0
cmd/internal/obj.Symgrow(0x9458e300, 0xb273ca80, 0x88, 0x0)
        /usr/lib/go/src/cmd/internal/obj/data.go:50 +0x17c fp=0xa887793c sp=0xa88778d8
cmd/internal/obj/arm.span5(0x9458e300, 0xb273ca80)
        /usr/lib/go/src/cmd/internal/obj/arm/asm5.go:745 +0x758 fp=0xa8877a7c sp=0xa887793c
cmd/internal/obj.Flushplist(0x9458e300)
        /usr/lib/go/src/cmd/internal/obj/objfile.go:300 +0x2d8 fp=0xa8877bd8 sp=0xa8877a7c
cmd/internal/obj.Writeobjdirect(0x9458e300, 0xa86bcb60)
        /usr/lib/go/src/cmd/internal/obj/objfile.go:114 +0x1c fp=0xa8877be4 sp=0xa8877bd8
cmd/compile/internal/gc.dumpobj()
        /usr/lib/go/src/cmd/compile/internal/gc/obj.go:88 +0xca8 fp=0xa8877d30 sp=0xa8877be4
cmd/compile/internal/gc.Main()
        /usr/lib/go/src/cmd/compile/internal/gc/lex.go:495 +0x260c fp=0xa8877efc sp=0xa8877d30
cmd/compile/internal/arm.Main()
        /usr/lib/go/src/cmd/compile/internal/arm/galign.go:92 +0x538 fp=0xa8877f08 sp=0xa8877efc
main.main()
        /usr/lib/go/src/cmd/compile/main.go:34 +0x15c fp=0xa8877f74 sp=0xa8877f08
runtime.main()
        /usr/lib/go/src/runtime/proc.go:188 +0x2b4 fp=0xa8877f9c sp=0xa8877f74
runtime.goexit()
        /usr/lib/go/src/runtime/asm_arm.s:990 +0x4 fp=0xa8877f9c sp=0xa8877f9c

building docker-1.11.2 with go-1.6.2:

---> Making bundle: dynbinary (in bundles/1.11.2/dynbinary)
Building: bundles/1.11.2/dynbinary/docker-1.11.2
# github.com/ugorji/go/codec
fatal error: runtime.SetFinalizer: pointer not in allocated block

goroutine 1 [running]:
runtime.throw(0x431868, 0x34)
        /usr/lib/go/src/runtime/panic.go:547 +0x78 fp=0x9c0afa7c sp=0x9c0afa70
runtime.SetFinalizer(0x3cbe68, 0x9bf363c0, 0x0, 0x0)
        /usr/lib/go/src/runtime/mfinal.go:309 +0x204 fp=0x9c0afb7c sp=0x9c0afa7c
os.(*file).close(0x9bf363c0, 0x0, 0x0)
        /usr/lib/go/src/os/file_unix.go:146 +0x158 fp=0x9c0afbbc sp=0x9c0afb7c
os.(*File).Close(0x9bfd21a8, 0x0, 0x0)
        /usr/lib/go/src/os/file_unix.go:132 +0x54 fp=0x9c0afbcc sp=0x9c0afbbc
cmd/internal/obj.Bterm(0x9bf363d0, 0x0, 0x0)
        /usr/lib/go/src/cmd/internal/obj/util.go:181 +0x68 fp=0x9c0afbe4 sp=0x9c0afbcc
cmd/compile/internal/gc.dumpobj()
        /usr/lib/go/src/cmd/compile/internal/gc/obj.go:101 +0xe10 fp=0x9c0afd30 sp=0x9c0afbe4
cmd/compile/internal/gc.Main()
        /usr/lib/go/src/cmd/compile/internal/gc/lex.go:495 +0x260c fp=0x9c0afefc sp=0x9c0afd30
cmd/compile/internal/arm.Main()
        /usr/lib/go/src/cmd/compile/internal/arm/galign.go:92 +0x538 fp=0x9c0aff08 sp=0x9c0afefc
main.main()
        /usr/lib/go/src/cmd/compile/main.go:34 +0x15c fp=0x9c0aff74 sp=0x9c0aff08
runtime.main()
        /usr/lib/go/src/runtime/proc.go:188 +0x2b4 fp=0x9c0aff9c sp=0x9c0aff74
runtime.goexit()
        /usr/lib/go/src/runtime/asm_arm.s:990 +0x4 fp=0x9c0aff9c sp=0x9c0aff9c

@davecheney
Copy link
Contributor

You probably got lucky. Try reducing the amount of parallelism with the -P flag.

@ncopa
Copy link
Contributor Author

ncopa commented Jun 16, 2016

Good idea. I asssume you mean with -p 1. But it still fails:

---> Making bundle: dynbinary (in bundles/1.11.2/dynbinary)
Building: bundles/1.11.2/dynbinary/docker-1.11.2
# github.com/ugorji/go/codec
runtime: memory allocated by OS (0x7ec33000) not in usable range [0x8f538000,0xffffffff)
fatal error: out of memory

runtime stack:
runtime.throw(0x40ba28, 0xd)
        /usr/lib/go/src/runtime/panic.go:547 +0x78
runtime.(*mcache).refill(0x7f3f8000, 0xe, 0x7ed427a0)
        /usr/lib/go/src/runtime/mcache.go:121 +0xfc
runtime.mallocgc.func2()
        /usr/lib/go/src/runtime/malloc.go:642 +0x28
runtime.systemstack(0x8f557400)
        /usr/lib/go/src/runtime/asm_arm.s:247 +0x80
runtime.mstart()
        /usr/lib/go/src/runtime/proc.go:1051

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/lib/go/src/runtime/asm_arm.s:192 +0x4 fp=0xa3a156b4 sp=0xa3a156b0
runtime.mallocgc(0xd0, 0x3e42e8, 0x0, 0x0)
        /usr/lib/go/src/runtime/malloc.go:643 +0x930 fp=0xa3a15724 sp=0xa3a156b4
runtime.newobject(0x3e42e8, 0xaf4b7ad0)
        /usr/lib/go/src/runtime/malloc.go:781 +0x50 fp=0xa3a15738 sp=0xa3a15724
cmd/internal/obj/arm.preprocess(0x8f5ac180, 0xa8ad4ba0)
        /usr/lib/go/src/cmd/internal/obj/arm/obj5.go:382 +0x7b4 fp=0xa3a15a7c sp=0xa3a15738
cmd/internal/obj.Flushplist(0x8f5ac180)
        /usr/lib/go/src/cmd/internal/obj/objfile.go:299 +0x2b8 fp=0xa3a15bd8 sp=0xa3a15a7c
cmd/internal/obj.Writeobjdirect(0x8f5ac180, 0xa3854dc0)
        /usr/lib/go/src/cmd/internal/obj/objfile.go:114 +0x1c fp=0xa3a15be4 sp=0xa3a15bd8
cmd/compile/internal/gc.dumpobj()
        /usr/lib/go/src/cmd/compile/internal/gc/obj.go:88 +0xca8 fp=0xa3a15d30 sp=0xa3a15be4
cmd/compile/internal/gc.Main()
        /usr/lib/go/src/cmd/compile/internal/gc/lex.go:495 +0x260c fp=0xa3a15efc sp=0xa3a15d30
cmd/compile/internal/arm.Main()
        /usr/lib/go/src/cmd/compile/internal/arm/galign.go:92 +0x538 fp=0xa3a15f08 sp=0xa3a15efc
main.main()
        /usr/lib/go/src/cmd/compile/main.go:34 +0x15c fp=0xa3a15f74 sp=0xa3a15f08
runtime.main()
        /usr/lib/go/src/runtime/proc.go:188 +0x2b4 fp=0xa3a15f9c sp=0xa3a15f74
runtime.goexit()
        /usr/lib/go/src/runtime/asm_arm.s:990 +0x4 fp=0xa3a15f9c sp=0xa3a15f9c

@ncopa
Copy link
Contributor Author

ncopa commented Jun 16, 2016

one interesting thing is that it seems to always happen in github.com/ugorji/go/codec.

@davecheney
Copy link
Contributor

Try go 1.7beta1, usable address space should have been increased in that version.

@ncopa
Copy link
Contributor Author

ncopa commented Jun 16, 2016

i did try to backport commit e6ec820 to 1.6 but it did not help. (or my conflict resolving was done bad). I am building go1.7beta1 now.

In any case, i will need some kind of workaround since this made it to alpine linux 3.4-stable branch (due to success with go1.6.1)

@ianlancetaylor ianlancetaylor changed the title docker build failure with go-1.6.2/musl libc on armhf runtime: docker build failure with go-1.6.2/musl libc on armhf Jun 16, 2016
@ianlancetaylor
Copy link
Member 

runtime: memory allocated by OS (0x7f092000) not in usable range [0x8f958000,0xffffffff)

This error means that the runtime set up a bitmap describing all memory in the range [0x8f958000,0xffffffff), and it asked the OS for memory in that range using mmap, but the OS gave it back memory that was not in that range. Use strace -f to see the sequence of mmap calls. Just before the failure you should see an mmap requesting memory somewhere in that range, but the OS returning a different value. You should confirm that, to make sure that my analysis is correct.

These adjustments could be the result of mmap randomization, which I believe is more aggressive in newer kernels. What are the values of /proc/sys/vm/mmap_rnd_bits and /proc/sys/vm/mmap_rnd_compat_bits?

As was mentioned above, this should be fixed in the upcoming 1.7 release by https://golang.org/cl/20471. But we are not going to backport that to the 1.6 release series.

@ncopa
Copy link
Contributor Author

ncopa commented Jun 16, 2016

@ianlancetaylor sounds correct. This is a kernel patched with PaX which has a randmmap feature. Setting pax softmode makes it build.

I also tested go1.7beta1 and it also works like it should.

@ianlancetaylor
Copy link
Member

OK, thanks for trying it out. This doesn't sound like something we should touch in Go 1.6. Closing as fixed in 1.7.

ncopa added a commit to alpinelinux/aports that referenced this issue Jun 17, 2016
@ncopa
community/go: paxmark go on arm
e7dca0f 
this works around problem when building docker
golang/go#16081
ncopa added a commit to alpinelinux/aports that referenced this issue Jun 17, 2016
@ncopa
community/docker: enable on all archs
d0e56a6 
golang/go#16081
ncopa added a commit to alpinelinux/aports that referenced this issue Jun 27, 2016
@ncopa
community/go: paxmark go on arm
648f671 
this works around problem when building docker
golang/go#16081

(cherry picked from commit e7dca0f)
@golang golang locked and limited conversation to collaborators Jun 16, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@davecheney @ncopa @ianlancetaylor @gopherbot