Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: linux/s390x ThreadSanitizer failed to allocate 0x7e0000 (8257536) bytes at address 900000180000 (errno: 12) in QEMU #67881

Open
tamird opened this issue Jun 7, 2024 · 9 comments
Open

runtime: linux/s390x ThreadSanitizer failed to allocate 0x7e0000 (8257536) bytes at address 900000180000 (errno: 12) in QEMU #67881

tamird opened this issue Jun 7, 2024 · 9 comments
Labels
arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@tamird
Copy link
Contributor

tamird commented Jun 7, 2024

Go version

go version go1.22.3 linux/amd64

Output of go env in your module/workspace:

GOARCH=s390x

What did you do?

Cross-compiled a race build test binary from linux/amd64 to linux/s390x:

      - name: 'BuildTestRace with ${{ matrix.go }} for ${{ matrix.arch }}'
        if: ${{ matrix.arch == 'aarch64' && !matrix.arm64_race_unsupported || matrix.arch == 's390x' && !matrix.s390x_race_unsupported }}
        env:
          CGO_ENABLED: 1
        shell: bash
        run: |
          set -euxo pipefail

          # Non-host *.syso files are missing from the Go toolchains provided
          # by setup-go. See https://github.com/actions/setup-go/issues/181.
          curl --location --output "$(go env GOROOT)"/src/runtime/race/race_linux_"$GOARCH".syso \
            https://github.com/golang/go/raw/release-branch.go${{ matrix.go }}/src/runtime/race/race_linux_"$GOARCH".syso

          sudo apt update
          sudo apt install -y gcc-${{ matrix.arch }}-linux-gnu

          CC=${{ matrix.arch }}-linux-gnu-gcc CC_FOR_TARGET=gcc-${{ matrix.arch }}-linux-gnu go test -c -race -o goid.race.test ./...

where GOARCH=s390x and ${{ matrix.arch }} == s390x, and then attempted to run that binary using docker-on-qemu:

      - name: 'Test and Bench with ${{ matrix.go }} on ${{ matrix.arch }}'
        if: ${{ matrix.arch != '386' && matrix.arch != 'x64' && !matrix.qemu_emulation_broken }}
        uses: uraimo/run-on-arch-action@v2
        with:
          arch: ${{ matrix.arch }}
          distro: bookworm
          dockerRunArgs: --mount type=bind,source="$(pwd)",target=/checkout,readonly
          shell: /bin/bash
          run: |
            set -euxo pipefail

            find /checkout -name '*.test' -type f -executable -print0 | \
              xargs -t -0 -I '{}' sh -c '{} -test.v && {} -test.bench=. -test.benchmem -test.v'

What did you see happen?

+ find /checkout -name '*.test' -type f -executable -print0
  + xargs -t -0 -I '{}' sh -c '{} -test.v && {} -test.bench=. -test.benchmem -test.v'
  sh -c '/checkout/goid.race.test -test.v && /checkout/goid.race.test -test.bench=. -test.benchmem -test.v'
  ==17==ERROR: ThreadSanitizer failed to allocate 0x7f0000 (8323072) bytes at address 9000001a0000 (errno: 12)
  sh -c '/checkout/goid.test -test.v && /checkout/goid.test -test.bench=. -test.benchmem -test.v'
  === RUN   TestGet
  --- PASS: TestGet (0.01s)
  PASS
  === RUN   TestGet
  --- PASS: TestGet (0.01s)
  goos: linux
  goarch: s390x
  pkg: github.com/petermattis/goid
  cpu: AMD EPYC 7763 64-Core Processor @ 3242.4[67](https://github.com/petermattis/goid/actions/runs/9417707064/job/25943521220?pr=44#step:16:68)MHz
  BenchmarkGet
  BenchmarkGet-4   	130759052	         9.021 ns/op	       0 B/op	       0 allocs/op
  PASS
  Error: The process '/home/runner/work/_actions/uraimo/run-on-arch-action/v2/src/run-on-arch.sh' failed with exit code 123

What did you expect to see?

I expected the test to pass.
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jun 7, 2024
@tamird
Copy link
Contributor Author

tamird commented Jun 7, 2024

This was observed in https://github.com/petermattis/goid/actions/runs/9417811829 in all Go versions between 1.19 (when race supported was added) and 1.22 (latest at the time of writing).

@tamird tamird changed the title runtime: ThreadSanitizer failed to allocate 0x7e0000 (8257536) bytes at address 900000180000 (errno: 12) runtime: linux/s390x ThreadSanitizer failed to allocate 0x7e0000 (8257536) bytes at address 900000180000 (errno: 12) in QEMU Jun 7, 2024
@tamird tamird changed the title runtime: linux/s390x ThreadSanitizer failed to allocate 0x7e0000 (8257536) bytes at address 900000180000 (errno: 12) in QEMU runtime: linux/s390x ThreadSanitizer failed to allocate 0x7e0000 (8257536) bytes at address 900000180000 (errno: 12) in QEMU Jun 7, 2024
@tamird tamird mentioned this issue Jun 7, 2024
Merged
@mauri870 mauri870 added the arch-s390x Issues solely affecting the s390x architecture. label Jun 7, 2024
@randall77
Copy link
Contributor

-race does use a lot more memory, maybe you are just running out. How much memory is it using when it fails? How much memory does the test use at peak without -race?

@mauri870
Copy link
Member

mauri870 commented Jun 7, 2024

Is this a qemu issue only or is that reproducible on a s390x machine? We don't have qemu as an official platform that Go supports, support for it has always being best-effort. I myself have experienced some failures in the past that were confirmed false positives when testing with one of our s390x builders.

@tamird
Copy link
Contributor Author

tamird commented Jun 7, 2024

@randall77 I would expect this to use next to no memory at all. We're running the unit tests for https://github.com/petermattis/goid which is implemented with 2 assembly instructions to read the goroutine ID from the g.

@mauri870 It may very well be a QEMU-only issue; I don't have a s390x machine myself. Support for s390x was contributed to https://github.com/petermattis/goid by @srinivas-pokala in petermattis/goid#44. @srinivas-pokala are you able to try it on a s390x machine?

@tamird
Copy link
Contributor Author

tamird commented Jun 7, 2024

cc @iii-i who contributed race support for s390x in 87cf92e.

@randall77
Copy link
Contributor

My suspicion is that the racefuncenter call is clobbering the g register somehow, causing the getg hack to return junk.

@mknyszek
Copy link
Contributor

mknyszek commented Jun 7, 2024

CC @golang/s390x

@mknyszek mknyszek added this to the Backlog milestone Jun 7, 2024
@mknyszek mknyszek added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. help wanted and removed help wanted labels Jun 7, 2024
@gabyhelp
Copy link

gabyhelp commented Jun 8, 2024

Similar Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@srinivas-pokala
Copy link
Contributor

@randall77 I would expect this to use next to no memory at all. We're running the unit tests for https://github.com/petermattis/goid which is implemented with 2 assembly instructions to read the goroutine ID from the g.

@mauri870 It may very well be a QEMU-only issue; I don't have a s390x machine myself. Support for s390x was contributed to https://github.com/petermattis/goid by @srinivas-pokala in petermattis/goid#44. @srinivas-pokala are you able to try it on a s390x machine?
@tamird , I could able to run on s390x machine with -race without any errors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Compiler / Runtime
Status: Todo
Milestone
Backlog
Development

No branches or pull requests
7 participants
@mknyszek @tamird @randall77 @gopherbot @mauri870 @srinivas-pokala @gabyhelp