Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: regular panic on Plan 9 #14650

Closed
0intro opened this issue Mar 4, 2016 · 5 comments
Closed

cmd/compile: regular panic on Plan 9 #14650

0intro opened this issue Mar 4, 2016 · 5 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge OS-Plan9
Milestone
Go1.7

Comments

@0intro
Copy link
Member

0intro commented Mar 4, 2016

Since early January we started to experience occasional cmd/compile failures reported by the Plan 9 (386) trybots on the dev.ssa branch. When the dev.ssa branch was merged to master on March 1, the failures started to become much more frequent, because the master branch is more active.

The failures happen when building go_bootstrap. They look like:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0x3 code=0x0 addr=0x0 pc=0x20b307]

goroutine 1 [running]:
bootstrap/internal/obj/x86.doasm(0x10adc000, 0x114481a0)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:3221 +0x11c27
bootstrap/internal/obj/x86.asmins(0x10adc000, 0x114481a0)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:4577 +0xf26
bootstrap/internal/obj/x86.span6(0x10adc000, 0x10fefd40)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:1778 +0x81f
bootstrap/internal/obj.Flushplist(0x10adc000)
    /tmp/workdir/go/src/cmd/internal/obj/objfile.go:300 +0x278
bootstrap/internal/obj.Writeobjdirect(0x10adc000, 0x119fff00)
    /tmp/workdir/go/src/cmd/internal/obj/objfile.go:114 +0x27
bootstrap/compile/internal/gc.dumpobj()
    /tmp/workdir/go/src/cmd/compile/internal/gc/obj.go:88 +0xafc
bootstrap/compile/internal/gc.Main()
    /tmp/workdir/go/src/cmd/compile/internal/gc/lex.go:497 +0x1c75
bootstrap/compile/internal/x86.Main()
    /tmp/workdir/go/src/cmd/compile/internal/x86/galign.go:108 +0x5ff
main.main()
    /tmp/workdir/go/src/cmd/compile/main.go:30 +0xe5

or

sys: trap: invalid opcode pc=0x0020c008
PC=0x20c008

goroutine 1 [running]:
bootstrap/internal/obj/x86.asmidx(0x10b16000, 0x0, 0x10b18ebc, 0x10b160de)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:2637 +0x78 fp=0x111d5774 sp=0x111d572c
bootstrap/internal/obj/x86.relput4(0x10b16000, 0x10b1903c, 0x10b19044)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:2700 +0x28b fp=0x111d5818 sp=0x111d5774
bootstrap/internal/obj/x86.doasm(0x10b16000, 0x10b1903c)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:3505 +0xebf9 fp=0x111d5b08 sp=0x111d5818
bootstrap/internal/obj/x86.asmins(0x10b16000, 0x10b1903c)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:4726 +0xcd5 fp=0x111d5bb0 sp=0x111d5b08
bootstrap/internal/obj/x86.span6(0x10b16000, 0x113655c0)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:1925 +0x836 fp=0x111d5c7c sp=0x111d5bb0
bootstrap/internal/obj.flushplist(0x10b16000, 0x10b1f601)
    /tmp/workdir/go/src/cmd/internal/obj/objfile.go:308 +0x281 fp=0x111d5cfc sp=0x111d5c7c
bootstrap/internal/obj.Flushplist(0x10b16000)
    /tmp/workdir/go/src/cmd/internal/obj/objfile.go:119 +0x34 fp=0x111d5d08 sp=0x111d5cfc
bootstrap/compile/internal/gc.funccompile(0x110009c0)
    /tmp/workdir/go/src/cmd/compile/internal/gc/dcl.go:1445 +0x1d3 fp=0x111d5d44 sp=0x111d5d08
bootstrap/compile/internal/gc.Main()
    /tmp/workdir/go/src/cmd/compile/internal/gc/lex.go:488 +0x1bd9 fp=0x111d5ef4 sp=0x111d5d44
bootstrap/compile/internal/x86.Main()
    /tmp/workdir/go/src/cmd/compile/internal/x86/galign.go:97 +0x5c6 fp=0x111d5f60 sp=0x111d5ef4
main.main()
    /tmp/workdir/go/src/cmd/compile/main.go:30 +0xe5 fp=0x111d5fcc sp=0x111d5f60
runtime.main()
    /tmp/buildlet-scatch798455816/src/runtime/proc.go:63 +0xcb fp=0x111d5ff0 sp=0x111d5fcc
runtime.goexit()
    /tmp/buildlet-scatch798455816/src/runtime/asm_386.s:2287 +0x1 fp=0x111d5ff4 sp=0x111d5ff0

or

unexpected fault address 0xf0000000
fatal error: fault
[signal 0x3 code=0x0 addr=0xf0000000 pc=0x20b307]

goroutine 1 [running]:
runtime.gothrow(0x40f108, 0x5)
    /tmp/buildlet-scatch798455816/src/runtime/panic.go:503 +0x67 fp=0x10ef3730 sp=0x10ef3724
runtime.sigpanic()
    /tmp/buildlet-scatch798455816/src/runtime/os_plan9.go:50 +0x151 fp=0x10ef3764 sp=0x10ef3730
bootstrap/internal/obj/x86.doasm(0x10adc000, 0x113a5930)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:3221 +0x11c27 fp=0x10ef3a60 sp=0x10ef3764
bootstrap/internal/obj/x86.asmins(0x10adc000, 0x113a5930)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:4577 +0xf26 fp=0x10ef3b40 sp=0x10ef3a60
bootstrap/internal/obj/x86.span6(0x10adc000, 0x10f58ba0)
    /tmp/workdir/go/src/cmd/internal/obj/x86/asm6.go:1778 +0x81f fp=0x10ef3c08 sp=0x10ef3b40
bootstrap/internal/obj.Flushplist(0x10adc000)
    /tmp/workdir/go/src/cmd/internal/obj/objfile.go:300 +0x278 fp=0x10ef3c88 sp=0x10ef3c08
bootstrap/internal/obj.Writeobjdirect(0x10adc000, 0x1112f240)
    /tmp/workdir/go/src/cmd/internal/obj/objfile.go:114 +0x27 fp=0x10ef3c94 sp=0x10ef3c88
bootstrap/compile/internal/gc.dumpobj()
    /tmp/workdir/go/src/cmd/compile/internal/gc/obj.go:88 +0xafc fp=0x10ef3d94 sp=0x10ef3c94
bootstrap/compile/internal/gc.Main()
    /tmp/workdir/go/src/cmd/compile/internal/gc/lex.go:495 +0x1c75 fp=0x10ef3ef4 sp=0x10ef3d94
bootstrap/compile/internal/x86.Main()
    /tmp/workdir/go/src/cmd/compile/internal/x86/galign.go:108 +0x5ff fp=0x10ef3f60 sp=0x10ef3ef4
main.main()
    /tmp/workdir/go/src/cmd/compile/main.go:30 +0xe5 fp=0x10ef3fcc sp=0x10ef3f60
runtime.main()
    /tmp/buildlet-scatch798455816/src/runtime/proc.go:63 +0xcb fp=0x10ef3ff0 sp=0x10ef3fcc
runtime.goexit()
    /tmp/buildlet-scatch798455816/src/runtime/asm_386.s:2287 +0x1 fp=0x10ef3ff4 sp=0x10ef3ff0

I looked at all the Plan 9 trybot failures for the past few months. Here are the failures that seem related to the issue:

2016-01-07 https://storage.googleapis.com/go-build-log/08d47646/plan9-386_a0c8a6de.log
2016-02-07 https://storage.googleapis.com/go-build-log/786cb6aa/plan9-386_3caaf30b.log
2016-02-08 https://storage.googleapis.com/go-build-log/6284fd7d/plan9-386_0eb811c9.log
2016-02-08 https://storage.googleapis.com/go-build-log/c3e8473c/plan9-386_a81fe2b3.log
2016-02-08 https://storage.googleapis.com/go-build-log/74693682/plan9-386_af2ede70.log
2016-02-10 https://storage.googleapis.com/go-build-log/fd458ba4/plan9-386_32443634.log
2016-02-10 https://storage.googleapis.com/go-build-log/c05edc82/plan9-386_d9e2152a.log
2016-03-02 https://storage.googleapis.com/go-build-log/bb017f22/plan9-386_675f6af8.log
2016-03-02 https://storage.googleapis.com/go-build-log/7a15fb26/plan9-386_8300a8d5.log
2016-03-03 https://storage.googleapis.com/go-build-log/45983827/plan9-386_964f732b.log
2016-03-03 https://storage.googleapis.com/go-build-log/8ce703f2/plan9-386_f1266bd9.log
2016-03-03 https://storage.googleapis.com/go-build-log/99f4402a/plan9-386_359912ab.log
2016-03-03 https://storage.googleapis.com/go-build-log/8ce703f2/plan9-386_f1266bd9.log
2016-03-03 https://storage.googleapis.com/go-build-log/b1f365e1/plan9-386_bb0c78ee.log
2016-03-03 https://storage.googleapis.com/go-build-log/628190d2/plan9-386_dd54d34f.log
2016-03-03 https://storage.googleapis.com/go-build-log/2a89b4fa/plan9-386_b11a651b.log
2016-03-03 https://storage.googleapis.com/go-build-log/e360bef5/plan9-386_dd273260.log
2016-03-04 https://storage.googleapis.com/go-build-log/39cfd5ed/plan9-386_ac6a3ddb.log
2016-03-04 https://storage.googleapis.com/go-build-log/d02312c3/plan9-386_416636ef.log
2016-03-04 https://storage.googleapis.com/go-build-log/4af86c05/plan9-386_1b187278.log
2016-03-04 https://storage.googleapis.com/go-build-log/1023603d/plan9-386_f379247a.log

The error appeared the first time on 2016-01-07. Then, it appeared on 2016-02-07, 2016-02-08 and 2016-02-10. All these failures are related to changes on the dev.ssa branch.
Then, the dev.ssa branch was merged to master on 2016-03-01 and the issue started to appear on master as well.

Here are the issues on the build dashboard for the dev.ssa branch:

2016-01-13 http://build.golang.org/log/16064baa23820f462d88b62091555e3f3206a2c2
2016-02-07 http://build.golang.org/log/5f7bd41b60d44c44553b8a6c5fa09a3c11e1919b
2016-02-07 http://build.golang.org/log/f18355544e9b52cac265ab46181915bced101a03
2016-02-08 http://build.golang.org/log/752ad760aa459d182d494fd78b2ee027ad9d37a4
2016-02-08 http://build.golang.org/log/1cc97297b15cc3537bfe0b355412af7c5d655e1d
2016-02-09 http://build.golang.org/log/ff9ad04c40f3618ea25c752879182cab5d06d9d9
2016-02-09 http://build.golang.org/log/4cfeffddb1ce4b2ef3beab43a7020a5bade1b27e

The very first failure (2016-01-07) was noticed by the Plan 9 trybot in CL 18390, which was submitted on 2016-01-13. Then, the first failure from the dashboard (2016-01-13) appeared when the [CL 18267 (https://golang.org/cl/18267) was submitted, just after CL 18390.

I tried to reproduce the issue by running make.rc dozens of times on various machines, including the same image running the Plan 9 builder on GCE. So far, I haven't been able to reproduce the issue.
@0intro 0intro added the OS-Plan9 label Mar 4, 2016
@0intro 0intro self-assigned this Mar 4, 2016
@0intro 0intro added this to the Go1.7 milestone Mar 4, 2016
@0intro
Copy link
Member Author

0intro commented Mar 4, 2016

CC @4ad

@0intro
Copy link
Member Author

0intro commented Mar 31, 2016

Here are the trybot failures since 2016-03-04:

2016-03-08 https://storage.googleapis.com/go-build-log/3296d017/plan9-386_4c71e40e.log
2016-03-08 https://storage.googleapis.com/go-build-log/2ec4941a/plan9-386_6d5070c6.log
2016-03-08 https://storage.googleapis.com/go-build-log/26c83298/plan9-386_e32656ca.log
2016-09-09 https://storage.googleapis.com/go-build-log/443a9cf3/plan9-386_7108a665.log
2016-09-09 https://storage.googleapis.com/go-build-log/0ef79836/plan9-386_3031270a.log
2016-03-10 https://storage.googleapis.com/go-build-log/9703d373/plan9-386_6f2b86fb.log
2016-03-10 https://storage.googleapis.com/go-build-log/e001a143/plan9-386_7724e041.log
2016-03-11 https://storage.googleapis.com/go-build-log/6a834cbd/plan9-386_d687aba4.log
2016-03-11 https://storage.googleapis.com/go-build-log/566fb02f/plan9-386_567af3ee.log
2016-03-11 https://storage.googleapis.com/go-build-log/1b673a21/plan9-386_ce752054.log
2016-03-13 https://storage.googleapis.com/go-build-log/ddba68a1/plan9-386_088a58ed.log
2016-03-14 https://storage.googleapis.com/go-build-log/203773ac/plan9-386_72928b25.log
2016-03-14 https://storage.googleapis.com/go-build-log/35d74c85/plan9-386_29d60965.log
2016-03-16 https://storage.googleapis.com/go-build-log/5ad81197/plan9-386_e3388eb1.log
2016-03-16 https://storage.googleapis.com/go-build-log/62a93256/plan9-386_820dd58b.log
2016-03-17 https://storage.googleapis.com/go-build-log/1d86fb81/plan9-386_fd5c5343.log
2016-03-17 https://storage.googleapis.com/go-build-log/fa8603d4/plan9-386_69acc8c2.log
2016-03-17 https://storage.googleapis.com/go-build-log/3043d472/plan9-386_88a7aa57.log
2016-03-17 https://storage.googleapis.com/go-build-log/718c5baf/plan9-386_29ca8105.log
2016-03-17 https://storage.googleapis.com/go-build-log/035ac48d/plan9-386_351690e8.log
2016-03-17 https://storage.googleapis.com/go-build-log/4a1ac65a/plan9-386_ead82991.log
2016-03-18 https://storage.googleapis.com/go-build-log/77beb915/plan9-386_db7b1fd2.log
2016-03-21 https://storage.googleapis.com/go-build-log/3c9e5e97/plan9-386_2eee8987.log
2016-03-21 https://storage.googleapis.com/go-build-log/942f7ff0/plan9-386_596b6ea1.log
2016-03-22 https://storage.googleapis.com/go-build-log/deebea0f/plan9-386_d3ece8a3.log
2016-03-22 https://storage.googleapis.com/go-build-log/4509a28d/plan9-386_6b70e99a.log
2016-03-23 https://storage.googleapis.com/go-build-log/c788568b/plan9-386_9f19889f.log
2016-03-24 https://storage.googleapis.com/go-build-log/bd54b513/plan9-386_f97c57e4.log
2016-03-24 https://storage.googleapis.com/go-build-log/f536cd9b/plan9-386_90fd2c86.log
2016-03-24 https://storage.googleapis.com/go-build-log/c6ee225b/plan9-386_9e47fd93.log
2016-03-25 https://storage.googleapis.com/go-build-log/2b6b844b/plan9-386_757d1c14.log
2016-03-25 https://storage.googleapis.com/go-build-log/6eac72f8/plan9-386_c7954962.log
2016-03-31 https://storage.googleapis.com/go-build-log/702ff8fb/plan9-386_be7d659b.log
2016-03-31 https://storage.googleapis.com/go-build-log/39334b89/plan9-386_23d16f85.log

@gopherbot
Copy link

CL https://golang.org/cl/21411 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Apr 1, 2016
@0intro @bradfitz
dashboard: use Go 1.6 to bootstrap Go on Plan 9
77ef5db 
On the Plan 9 builder, we are experiencing regular
failures in cmd/compile when building go_boostrap.

We suspect the failures to be related to a bug in
the Go 1.4 runtime. This is affecting the cmd/compile
program, since it has been built with Go 1.4 and
linked with the Go 1.4 runtime.

Updates golang/go#14650

Change-Id: I78729c8660314be381d7b668810c6f64214f2358
Reviewed-on: https://go-review.googlesource.com/21411
Reviewed-by: Richard Miller <millerresearch@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz bradfitz added the Builders x/build issues (builders, bots, dashboards) label Apr 1, 2016
@bradfitz
Copy link
Contributor

bradfitz commented Apr 1, 2016

Okay, I've updated the plan9-386 GOROOT_BOOTSTRAP tarball to Go tip (at 59fc42b), trimmed down for size (see #9797 (comment)), and rebuilt the plan9-386 buildlet at the same revision and deployed, restarted farmer.golang.org, and reset the failed plan9 builds on the dashboard, so they're trying again now.

@bradfitz
Copy link
Contributor

bradfitz commented Apr 1, 2016

This appears to be fixed. We have an entire column of "ok" now.

I'm going to optimistically close this bug.

@bradfitz bradfitz closed this as completed Apr 1, 2016
@golang golang locked and limited conversation to collaborators Apr 2, 2017
@rsc rsc unassigned 0intro Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge OS-Plan9
Projects
None yet
Milestone
Go1.7
Development

No branches or pull requests
3 participants
@bradfitz @0intro @gopherbot