Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: new linux-arm64-msan-clang15 builder isn't passing at tip yet #71614

Open
dmitshur opened this issue Feb 7, 2025 · 5 comments
Open

x/build: new linux-arm64-msan-clang15 builder isn't passing at tip yet #71614

dmitshur opened this issue Feb 7, 2025 · 5 comments
Assignees
@prattmic
Labels
arch-arm64 Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Linux
Milestone
Backlog

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Feb 7, 2025

CL 643918 added ASAN and MSAN builders for linux-arm64, related to issue #70054. The gotip-linux-arm64-asan-clang15 builder is passing, but the gotip-linux-arm64-msan-clang15 one is consistently failing with errors like:

FATAL: Code 0xb1ff902cc9f4 is out of application range. Non-PIE build?
FATAL: MemorySanitizer can not mmap the shadow memory.
FATAL: Make sure to compile with -fPIE and to link with -pie.
FATAL: Disabling ASLR is known to cause this error.
FATAL: If running under GDB, try 'set disable-randomization off'.
==13833==Process memory map follows:
	0xb1ff90210000-0xb1ff90665000	/home/swarming/.swarming/w/ir/x/t/go-build2622072025/b451/tar.test
	0xb1ff9067b000-0xb1ff90830000	/home/swarming/.swarming/w/ir/x/t/go-build2622072025/b451/tar.test
	0xb1ff90830000-0xb1ff90854000	/home/swarming/.swarming/w/ir/x/t/go-build2622072025/b451/tar.test
	0xb1ff90854000-0xb1ff92295000	
	0xfbc054200000-0xfbc054300000	
	0xfbc054400000-0xfbc054500000	
	0xfbc054600000-0xfbc054700000	
	0xfbc054800000-0xfbc054900000	
	0xfbc054a00000-0xfbc054b00000	
	0xfbc054bb2000-0xfbc054f40000	
	0xfbc054f40000-0xfbc0550c7000	/usr/lib/aarch64-linux-gnu/libc.so.6
	0xfbc0550c7000-0xfbc0550dc000	/usr/lib/aarch64-linux-gnu/libc.so.6
	0xfbc0550dc000-0xfbc0550e0000	/usr/lib/aarch64-linux-gnu/libc.so.6
	0xfbc0550e0000-0xfbc0550e2000	/usr/lib/aarch64-linux-gnu/libc.so.6
	0xfbc0550e2000-0xfbc0550ef000	
	0xfbc0550f0000-0xfbc055104000	/usr/lib/aarch64-linux-gnu/libgcc_s.so.1
	0xfbc055104000-0xfbc05511f000	/usr/lib/aarch64-linux-gnu/libgcc_s.so.1
	0xfbc05511f000-0xfbc055120000	/usr/lib/aarch64-linux-gnu/libgcc_s.so.1
	0xfbc055120000-0xfbc055121000	/usr/lib/aarch64-linux-gnu/libgcc_s.so.1
	0xfbc055130000-0xfbc05513d000	/usr/lib/aarch64-linux-gnu/libresolv.so.2
	0xfbc05513d000-0xfbc05514f000	/usr/lib/aarch64-linux-gnu/libresolv.so.2
	0xfbc05514f000-0xfbc055150000	/usr/lib/aarch64-linux-gnu/libresolv.so.2
	0xfbc055150000-0xfbc055151000	/usr/lib/aarch64-linux-gnu/libresolv.so.2
	0xfbc055151000-0xfbc055153000	
	0xfbc055160000-0xfbc0551e0000	/usr/lib/aarch64-linux-gnu/libm.so.6
	0xfbc0551e0000-0xfbc0551ef000	/usr/lib/aarch64-linux-gnu/libm.so.6
	0xfbc0551ef000-0xfbc0551f0000	/usr/lib/aarch64-linux-gnu/libm.so.6
	0xfbc0551f0000-0xfbc0551f1000	/usr/lib/aarch64-linux-gnu/libm.so.6
	0xfbc0551f1000-0xfbc055201000	
	0xfbc055201000-0xfbc055229000	/usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
	0xfbc05522c000-0xfbc055235000	
	0xfbc055235000-0xfbc05523c000	
	0xfbc05523c000-0xfbc05523e000	[vvar]
	0xfbc05523e000-0xfbc05523f000	[vdso]
	0xfbc05523f000-0xfbc055241000	/usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
	0xfbc055241000-0xfbc055243000	/usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
	0xfffffd87c000-0xfffffd89d000	[stack]
==13833==End of process memory map.
FAIL	archive/tar	0.071s

See https://ci.chromium.org/ui/p/golang/builders/luci.golang.ci/gotip-linux-arm64-msan-clang15.

CC @prattmic, @mknyszek.
@dmitshur dmitshur added arch-arm64 NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Linux labels Feb 7, 2025
@dmitshur dmitshur added this to the Backlog milestone Feb 7, 2025
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Feb 7, 2025
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/647777 mentions this issue: main.star: set known issue for linux-arm64-msan-clang15

gopherbot pushed a commit to golang/build that referenced this issue Feb 7, 2025
@dmitshur @gopherbot
main.star: set known issue for linux-arm64-msan-clang15
bdf0b84 
Annotate a known failure in a new builder. This helps various automation
(like relui releases and golang.org/x tagging) know not to report it as
an unknown problem.

For golang/go#71614.
For golang/go#70054.

Change-Id: Ia086c2de8da11975a2fcaf2809fcd695553882d0
Reviewed-on: https://go-review.googlesource.com/c/build/+/647777
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@prattmic
Copy link
Member

Apologies, I didn't watch this one closely. I need to double check that msan + arm64 works normally. :)

@prattmic prattmic self-assigned this Feb 10, 2025
@prattmic
Copy link
Member

MSAN for C/C++ is similarly broken on this builder. The example at https://clang.llvm.org/docs/MemorySanitizer.html#usage results in a similar error:

$ ./a.out
FATAL: Code 0xb1b06ec2bb74 is out of application range. Non-PIE build?
FATAL: MemorySanitizer can not mmap the shadow memory.
FATAL: Make sure to compile with -fPIE and to link with -pie.
FATAL: Disabling ASLR is known to cause this error.
FATAL: If running under GDB, try 'set disable-randomization off'.
==26552==Process memory map follows:
        0xb1b06ebe0000-0xb1b06ecae000   /home/swarming/.swarming/w/ir/a.out
        0xb1b06ecbd000-0xb1b06ecc0000   /home/swarming/.swarming/w/ir/a.out
        0xb1b06ecc0000-0xb1b06ecc3000   /home/swarming/.swarming/w/ir/a.out
        0xb1b06ecc3000-0xb1b070758000
        0xfe0a16f00000-0xfe0a17000000
        0xfe0a17100000-0xfe0a17200000
        0xfe0a17300000-0xfe0a17400000
        0xfe0a17500000-0xfe0a17600000
        0xfe0a176c2000-0xfe0a17a60000
        0xfe0a17a60000-0xfe0a17be7000   /usr/lib/aarch64-linux-gnu/libc.so.6
        0xfe0a17be7000-0xfe0a17bfc000   /usr/lib/aarch64-linux-gnu/libc.so.6
        0xfe0a17bfc000-0xfe0a17c00000   /usr/lib/aarch64-linux-gnu/libc.so.6
        0xfe0a17c00000-0xfe0a17c02000   /usr/lib/aarch64-linux-gnu/libc.so.6
        0xfe0a17c02000-0xfe0a17c0f000
        0xfe0a17c10000-0xfe0a17c24000   /usr/lib/aarch64-linux-gnu/libgcc_s.so.1
        0xfe0a17c24000-0xfe0a17c3f000   /usr/lib/aarch64-linux-gnu/libgcc_s.so.1
        0xfe0a17c3f000-0xfe0a17c40000   /usr/lib/aarch64-linux-gnu/libgcc_s.so.1
        0xfe0a17c40000-0xfe0a17c41000   /usr/lib/aarch64-linux-gnu/libgcc_s.so.1
        0xfe0a17c50000-0xfe0a17c5d000   /usr/lib/aarch64-linux-gnu/libresolv.so.2
        0xfe0a17c5d000-0xfe0a17c6f000   /usr/lib/aarch64-linux-gnu/libresolv.so.2
        0xfe0a17c6f000-0xfe0a17c70000   /usr/lib/aarch64-linux-gnu/libresolv.so.2
        0xfe0a17c70000-0xfe0a17c71000   /usr/lib/aarch64-linux-gnu/libresolv.so.2
        0xfe0a17c71000-0xfe0a17c73000
        0xfe0a17c80000-0xfe0a17d00000   /usr/lib/aarch64-linux-gnu/libm.so.6
        0xfe0a17d00000-0xfe0a17d0f000   /usr/lib/aarch64-linux-gnu/libm.so.6
        0xfe0a17d0f000-0xfe0a17d10000   /usr/lib/aarch64-linux-gnu/libm.so.6
        0xfe0a17d10000-0xfe0a17d11000   /usr/lib/aarch64-linux-gnu/libm.so.6
        0xfe0a17d20000-0xfe0a17f1b000   /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.30
        0xfe0a17f1b000-0xfe0a17f25000   /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.30
        0xfe0a17f25000-0xfe0a17f30000   /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.30
        0xfe0a17f30000-0xfe0a17f33000   /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.30
        0xfe0a17f33000-0xfe0a17f36000
        0xfe0a17f44000-0xfe0a17f6c000   /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
        0xfe0a17f6d000-0xfe0a17f78000
        0xfe0a17f78000-0xfe0a17f7f000
        0xfe0a17f7f000-0xfe0a17f81000   [vvar]
        0xfe0a17f81000-0xfe0a17f82000   [vdso]
        0xfe0a17f82000-0xfe0a17f84000   /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
        0xfe0a17f84000-0xfe0a17f86000   /usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1
        0xffffd2b4a000-0xffffd2b6b000   [stack]
==26552==End of process memory map.

So this isn't a Go-specific issue.

@prattmic
Copy link
Member

prattmic commented Feb 10, 2025 •
edited
Loading

google/sanitizers#1716 (comment)

IIUC, newer kernels (v6.6+) have increased the default number of bits of randomness in ASLR, resulting in binaries getting mapped outside of *SAN's expected range. LLVM 18.1+ include a fixed version of *SAN to handle this.

This builder has a 6.6.72 kernel and LLVM 15.0.6.

So the right thing to do is update to a newer version of LLVM.

@prattmic
Copy link
Member

Ah, @mknyszek already encountered this in #59418 (comment).

@dmitshur dmitshur moved this to In Progress in Go Release Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prattmic prattmic

Labels
arch-arm64 Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Linux
Projects
Go Release
Status: In Progress
Milestone
Backlog
Development

No branches or pull requests
3 participants
@prattmic @dmitshur @gopherbot