Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: TestGdbPythonCgo is flaky on linux-mips64le-rtrk builder #37794

Open
bcmills opened this issue Mar 11, 2020 · 6 comments
Open

runtime: TestGdbPythonCgo is flaky on linux-mips64le-rtrk builder #37794

bcmills opened this issue Mar 11, 2020 · 6 comments
Labels
arch-mips compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@bcmills
Copy link
Contributor

bcmills commented Mar 11, 2020

2020-03-10T16:25:09-9f74f0a/linux-mips64le-rtrk
2020-02-01T06:01:05-866920a/linux-mips64le-rtrk
2019-11-27T16:41:28-b11d02e/linux-mips64le-rtrk
2019-11-19T02:41:53-2d8c199/linux-mips64le-rtrk
2019-11-15T18:31:19-398f9e1/linux-mips64le-rtrk
2019-11-13T15:52:21-bf49905/linux-mips64le-rtrk
2019-11-12T22:30:48-995ade8/linux-mips64le-rtrk

CC @ianlancetaylor @bogojevic @milanknezevic

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
--- FAIL: TestGdbPythonCgo (6.02s)
    runtime-gdb_test.go:69: gdb version 8.2
    runtime-gdb_test.go:236: gdb output: Loading Go Runtime support.
        Loading Go Runtime support.
        Breakpoint 1 at 0x1200ad4b4: file /tmp/gobuilder-mips64le/tmp/go-build883242937/main.go, line 15.
        Couldn't get registers: No such process.
        BEGIN info goroutines
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 442, in invoke
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        BEGIN print mapvar
        No symbol "mapvar" in current context.
        END
        BEGIN print strvar
        No symbol "strvar" in current context.
        END
        BEGIN info locals
        No frame selected.
        END
        BEGIN goroutine 1 bt
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 536, in invoke
            self.invoke_per_goid(goid, cmd)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 539, in invoke_per_goid
            pc, sp = find_goroutine(goid)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 467, in find_goroutine
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        BEGIN goroutine 2 bt
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 536, in invoke
            self.invoke_per_goid(goid, cmd)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 539, in invoke_per_goid
            pc, sp = find_goroutine(goid)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 467, in find_goroutine
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        BEGIN goroutine all bt
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 530, in invoke
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        Breakpoint 2 at 0x1200ad55c: file /tmp/gobuilder-mips64le/tmp/go-build883242937/main.go, line 20.
        Cannot execute this command while the selected thread is running.
        BEGIN goroutine 1 bt at the end
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 536, in invoke
            self.invoke_per_goid(goid, cmd)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 539, in invoke_per_goid
            pc, sp = find_goroutine(goid)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 467, in find_goroutine
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        
    runtime-gdb_test.go:265: info goroutines failed: Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 442, in invoke
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
FAIL
FAIL	runtime	177.962s
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 11, 2020
@bcmills bcmills added this to the Backlog milestone Mar 11, 2020
@bcmills
Copy link
Contributor Author

bcmills commented Mar 11, 2020

See also #35743, #20821.

@bcmills bcmills mentioned this issue May 21, 2020
Open
@bcmills
Copy link
Contributor Author

bcmills commented Sep 16, 2020

2020-09-15T14:51:32-f1c400a/linux-mips64le-rtrk
2020-08-28T20:01:41-54e18f1/linux-mips64le-rtrk
2020-03-25T20:14:03-e8be350/linux-mips64le-rtrk

@bcmills
Copy link
Contributor Author

bcmills commented Jan 7, 2022

greplogs --dashboard -md -l -e '(?ms)FAIL: TestGdbPythonCgo .*Cannot access memory at address' --since=2020-09-17

2022-01-07T18:20:24-ade5488/linux-mips64le-rtrk
2021-12-20T15:13:56-7c94355/linux-mips64le-rtrk
2021-12-07T21:54:30-cf1ec17/linux-mips64le-rtrk
2021-11-13T00:23:16-39bc666/linux-mips64le-rtrk
2021-09-17T21:29:32-3fa7dbe/linux-mips64le-rtrk
2021-05-26T22:43:54-1d5298d/linux-mips64le-rtrk
2021-04-02T01:13:58-41e8a9f/linux-mips64le-rtrk
2021-03-23T05:11:39-e7aa0f9/linux-mips64le-rtrk

@bcmills
Copy link
Contributor Author

bcmills commented Jan 7, 2022

This test is still flaking regularly, creating noise in the dashboard and potentially masking other issues.

@bogojevic, @milanknezevic: is this a known bug for the gdb version installed on this builder? Is the gdb binary current, and if not can it be upgraded to see whether the underlying problem has been fixed?

@bcmills
Copy link
Contributor Author

bcmills commented Jan 7, 2022

There is already a skip on this builder for mipsle and mips64 due to #18784. 😠

@gopherbot
Copy link

Change https://golang.org/cl/376654 mentions this issue: runtime: expand TestGdbPythonCgo skip to include mips64le

@bcmills bcmills modified the milestones: Backlog, Unplanned Jan 7, 2022
gopherbot pushed a commit that referenced this issue Jan 10, 2022
runtime: expand TestGdbPythonCgo skip to include mips64le
9e7ea35 
The failure mode in #37794 does not match the failure mode described
in #18784. However, since the test is currently skipped on all other
MIPS variants, it may be that they suffer from the same underlying GDB
bug. Ideally one of the Go MIPS maintainers should file an upstream
bug and remove the skip once it is fixed; in the meantime, there is no
point in continuing to let the test fail on just one of the four MIPS
variants.

For #37794

Change-Id: I570f51cc04cbb7ef1ed7efd526e26886af53bfb6
Reviewed-on: https://go-review.googlesource.com/c/go/+/376654
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
jproberts pushed a commit to jproberts/go that referenced this issue Jun 21, 2022
@jproberts
runtime: expand TestGdbPythonCgo skip to include mips64le
ac9b72f 
The failure mode in golang#37794 does not match the failure mode described
in golang#18784. However, since the test is currently skipped on all other
MIPS variants, it may be that they suffer from the same underlying GDB
bug. Ideally one of the Go MIPS maintainers should file an upstream
bug and remove the skip once it is fixed; in the meantime, there is no
point in continuing to let the test fail on just one of the four MIPS
variants.

For golang#37794

Change-Id: I570f51cc04cbb7ef1ed7efd526e26886af53bfb6
Reviewed-on: https://go-review.googlesource.com/c/go/+/376654
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
@bcmills bcmills mentioned this issue Mar 8, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-mips compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
2 participants
@bcmills @gopherbot