Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: TestGdbPython flaky on linux/arm64 and linux/s390x builders #15599

Closed
eliasnaur opened this issue May 8, 2016 · 6 comments
Closed

runtime: TestGdbPython flaky on linux/arm64 and linux/s390x builders #15599

eliasnaur opened this issue May 8, 2016 · 6 comments
Labels
FrozenDueToAge Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Go1.7Maybe

Comments

@eliasnaur
Copy link
Contributor

The TestGdbPython test sometimes fails with:

GOMAXPROCS=2 runtime -cpu=1,2,4

--- FAIL: TestGdbPython (0.37s)
runtime-gdb_test.go:46: gdb version 7.9
runtime-gdb_test.go:171: goroutine 2 bt failed: Python Exception <class 'gdb.error'> Attempt to assign to an unmodifiable value.:
Error occurred in Python command: Attempt to assign to an unmodifiable value.
FAIL
FAIL runtime 25.005s

Recent failures on linux/arm64:
https://build.golang.org/log/5e16e3d5f5be40d93a12d7febf82d235d6c95ab6
https://build.golang.org/log/ca55586f93e0af899164dc759f5cb3c8f19fbdde

Recent failures on linux/s390x:
https://build.golang.org/log/0a806e53991d1d5c7b8f39cbc73f45ecfdff14b6
https://build.golang.org/log/dce649e576773871fdc529b29d35eb0fba5714fe
@bradfitz bradfitz added the Testing An issue that has been verified to require only test changes, not just a test failure. label May 10, 2016
@bradfitz
Copy link
Contributor

/cc @mundaym

@bradfitz bradfitz added this to the Go1.7Maybe milestone May 10, 2016
@bradfitz bradfitz changed the title TestGdbPython flaky on linux/arm64 and linux/s390x builders runtime: TestGdbPython flaky on linux/arm64 and linux/s390x builders May 10, 2016
@gopherbot
Copy link

CL https://golang.org/cl/23481 mentions this issue.

mk0x9 pushed a commit to mk0x9/go that referenced this issue May 27, 2016
@aclements
runtime: record Python stack on TestGdbPython failure
966baed 
For golang#15599.

Change-Id: Icc2e58a3f314b7a098d78fe164ba36f5b2897de6
Reviewed-on: https://go-review.googlesource.com/23481
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@aclements
Copy link
Member

This last happened with commit 6662897 on June 9th. A few commits later was cbc2686, which fixed a bug with GDB backtraces. There haven't been enough commits since 6662897 to be statistically sure that this is fixed, but I suspect it is.

$ greplogs -dashboard -E "Attempt to assign to an unmodifiable value" -l | findflakes -paths -dir ~/.cache/fetchlogs/rev
First observed e833749 26 Apr 23:39 2016 (554 commits ago)
Last observed  6662897 09 Jun 15:30 2016 (30 commits ago)
15% chance failure is still happening
6.1% failure probability (33 of 525 commits)
Likely culprits:
    6% e833749 runtime: free dead G stacks concurrently
    6% 1a2cf91 runtime: split gfree list into with-stacks and without-stacks
    5% 3b0efa6 cmd/compile: a rule's line number is at its ->
    5% 8d075be cmd/compile: lazily initialize litbuf
    5% 19912e1 cmd/dist: sort entries in zcgo.go generated file for deterministic build
    4% e607abb unicode: improve SimpleFold performance for ascii
    4% 6e4a861 gc: use AbsFileLine for deterministic binary export data
    4% 17db07f cmd/compile: don't discard inlineable but empty functions with binary export format
    4% 3a72d62 cmd/link: pass -no-pie (if supported) when creating a race-enabled executable.
    3% 55154cf cmd/link: fix gdb backtrace on architectures using a link register
No known past failures

CL 23481 did capture a stack at the failure:

--- FAIL: TestGdbPython (2.57s)
    runtime-gdb_test.go:50: gdb version 7.10
    runtime-gdb_test.go:177: goroutine 2 bt failed: Traceback (most recent call last):
          File "/tmp/workdir/go/src/runtime/runtime-gdb.py", line 454, in invoke
            gdb.parse_and_eval('$sp = {0}'.format(str(sp)))
        gdb.error: Attempt to assign to an unmodifiable value.
        Error occurred in Python command: Attempt to assign to an unmodifiable value.
FAIL
FAIL    runtime

This is also right where cbc2686 changed things.

I'm trying to reproduce this using gomote to confirm the fix, but not having any luck.

@aclements
Copy link
Member

aclements commented Jun 15, 2016 •
edited

Reproduced once out of ~100 runs at 6662897 on the linux-arm64-buildlet by running go test -short std.

git worktree add /tmp/issue15599
cd /tmp/issue15599/src
./make.bash
PATH=$PWD/../bin:$PATH
VM=$(gomote create linux-arm64-buildlet)
gomote push $VM && gomote run $VM /bin/sh -c 'PATH=$PWD/go/bin:$PATH; cd go/src && ./make.bash && while true; do go test -short std; done'

I'm going to run cbc2686 over night and see if it still reproduces.

@aclements
Copy link
Member

681 runs at cbc2686 on linux-arm64-buildlet with no TestGdbPython failures.

@aclements
Copy link
Member

Apparently I left the loop running. 2000 runs at cbc2686 on linux-arm64-buildlet with no TestGdbPython failures. I think we're good.

@golang golang locked and limited conversation to collaborators Jun 17, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Go1.7Maybe
Development

No branches or pull requests
4 participants
@bradfitz @eliasnaur @aclements @gopherbot