Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: TestGdbBacktrace fails on Linux builders with gdb version >=11.1 due to AppArmor restrictions on /proc/.../task/.../mem #54352

Open
thanm opened this issue Aug 9, 2022 · 4 comments
Open

x/build: TestGdbBacktrace fails on Linux builders with gdb version >=11.1 due to AppArmor restrictions on /proc/.../task/.../mem #54352

thanm opened this issue Aug 9, 2022 · 4 comments
Assignees
@prattmic
Labels
Builders x/build issues (builders, bots, dashboards) compiler/runtime Issues related to the Go compiler and/or runtime. ExpertNeeded NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@thanm
Copy link
Contributor

thanm commented Aug 9, 2022 •
edited

What version of Go are you using (go version)?

This is with tip.

What operating system and processor architecture are you using (go env)?

linux/amd64 on the linux-amd64-alpine builder

What did you do?

go test -test.v -test.run=TestGdbBacktrace runtime

What did you expect to see?

pass

What did you see instead?

--- FAIL: TestGdbBacktrace (0.93s)
    runtime-gdb_test.go:76: gdb version 11.2
    testenv.go:473: [gdb -nx -batch -iex add-auto-load-safe-path /workdir/go/src/runtime -ex set startup-with-shell off -ex break main.eee -ex run -ex backtrace -ex continue /workdir/tmp/TestGdbBacktrace1177699378/001/a.exe] exit status: exit status 1
    runtime-gdb_test.go:428: gdb output:
        Loading Go Runtime support.
        Breakpoint 1 at 0x456f40: file /workdir/tmp/TestGdbBacktrace1177699378/001/main.go, line 17.
        Warning:
        Cannot insert breakpoint 1.
        Cannot access memory at address 0x456f40
        
        Command aborted.
        #0  _rt0_amd64_linux () at /workdir/go/src/runtime/rt0_linux_amd64.s:8
        Warning:
        Cannot insert breakpoint 1.
        Cannot access memory at address 0x456f40
        
        Command aborted.
    runtime-gdb_test.go:443: gdb exited with error: exit status 1

This needs to be looked into -- I am not sure what the issue might be. I did try:

echo 0 > /proc/sys/kernel/yama/ptrace_scope

on the builder, but I am still seeing the failure.
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Aug 9, 2022
@thanm thanm added this to the Backlog milestone Aug 9, 2022
@thanm thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 9, 2022
@gopherbot
Copy link

Change https://go.dev/cl/422294 mentions this issue: runtime: disable gdb testpoints on alpine pending builder fix

gopherbot pushed a commit that referenced this issue Aug 9, 2022
@thanm
runtime: disable gdb testpoints on alpine pending builder fix
662a729 
Disable the TestGdb* testpoints until we can figure out why they are
failing and reconfigure the machine properly.

Updates #54352.

Change-Id: Id9c76a0ba6e23b5deff24f521a1c8e0aafb6481a
Reviewed-on: https://go-review.googlesource.com/c/go/+/422294
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Than McIntosh <thanm@google.com>
jproberts pushed a commit to jproberts/go that referenced this issue Aug 10, 2022
@thanm @jproberts
runtime: disable gdb testpoints on alpine pending builder fix
b761e02 
Disable the TestGdb* testpoints until we can figure out why they are
failing and reconfigure the machine properly.

Updates golang#54352.

Change-Id: Id9c76a0ba6e23b5deff24f521a1c8e0aafb6481a
Reviewed-on: https://go-review.googlesource.com/c/go/+/422294
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Than McIntosh <thanm@google.com>
@rsc
Copy link
Contributor

rsc commented Aug 10, 2022 •
edited

Strace on gdb (without -f to avoid interfering entirely) shows

open("/proc/19662/task/19662/mem", O_RDWR|O_LARGEFILE|O_CLOEXEC) = 11
...
pread64(11, "\270", 1, 4554112)         = 1
pwrite64(11, "\314", 1, 4554112)        = -1 EACCES (Permission denied)

That EACCES is not in the /proc implementation itself. It could have been (and I suspect was) introduced by AppArmor ahead of the actual call into the /proc implementation.

This started with the update from Alpine 3.10 to Alpine 3.16 because Alpine 3.16 includes gdb 11.2. Starting in gdb 11.1 (Sept 2021), gdb uses the /proc/.../task/.../mem file to do short writes to the user process memory, such as when setting breakpoints. Prior to gdb 11.1, large writes used the mem file, but writes smaller than 3 words (the breakpoint is only 1 byte) used ptrace directly.

I suspect all our other builders have gdb prior to 11.1. I confirmed that the linux-amd64 builder has gdb 10.1.

Probably there is a way to change the container configuration to override the AppArmor configuration to allow the writes to proc. I found the 'docker-default' configuration here and it seems to allow writing to /proc/.../task/.../mem.

We don't seem to be doing anything explicitly with AppArmor for our containers in the coordinator. But perhaps the Container-Optimized OS image has a different default AppArmor config.

Leaving the test disabled for now seems fine. Someone who understands Container-Optimized OS and AppArmor will need to relax the permissions.

@mknyszek mknyszek modified the milestones: Backlog, Go1.20 Aug 17, 2022
@dmitshur dmitshur mentioned this issue Dec 30, 2022
Closed
@heschi
Copy link
Contributor

heschi commented Jan 3, 2023

It would be nice to get this prioritized -- based on Dmitri's experience in #57520, it seems like it's going to bite us every time we try to set up a modern Linux builder.

@gopherbot gopherbot modified the milestones: Go1.20, Go1.21 Feb 1, 2023
@mknyszek
Copy link
Contributor

mknyszek commented Jun 9, 2023

Since this has been punted a couple times already, I'm inclined to move this to Backlog. Though, based on Russ' comment, perhaps this should be in Unreleased? It sounds like the fix here is actually on the x/build side?

Updating the issue title optimistically. Please comment if you disagree. Thanks!

@mknyszek mknyszek changed the title runtime: TestGdbBacktrace failure on linux-amd64-alpine builder x/build: TestGdbBacktrace fails on Linux builders with gdb >=11.1 due to AppArmor restrictions on /proc/.../task/.../mem Jun 9, 2023
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jun 9, 2023
@mknyszek mknyszek changed the title x/build: TestGdbBacktrace fails on Linux builders with gdb >=11.1 due to AppArmor restrictions on /proc/.../task/.../mem x/build: TestGdbBacktrace fails on Linux builders with gdb version >=11.1 due to AppArmor restrictions on /proc/.../task/.../mem Jun 9, 2023
@mknyszek mknyszek modified the milestones: Go1.21, Unreleased Jun 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prattmic prattmic

Labels
Builders x/build issues (builders, bots, dashboards) compiler/runtime Issues related to the Go compiler and/or runtime. ExpertNeeded NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Compiler / Runtime
Status: Todo
Milestone
Unreleased
Development

No branches or pull requests
7 participants
@rsc @mknyszek @prattmic @dmitshur @gopherbot @thanm @heschi