Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: fatal error: malloc deadlock on s390x #58783

Closed
Vishwanatha-HD opened this issue Feb 28, 2023 · 3 comments
Closed

runtime: fatal error: malloc deadlock on s390x #58783

Vishwanatha-HD opened this issue Feb 28, 2023 · 3 comments
Assignees
@Vishwanatha-HD
Labels
arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Backlog

Comments

@Vishwanatha-HD
Copy link
Contributor

Vishwanatha-HD commented Feb 28, 2023 •
edited by mknyszek

What version of Go are you using (go version)?

go version - "devel go1.21-f36dc54e9c" on linux s390x architecture.

Does this issue reproduce with the latest release?

Yes. The issue is reproducible easily.

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE=""
GOARCH="s390x"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/vishwa/Workspaces/golang/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/vishwa/Workspaces/golang/go/pkg/tool/linux_s390x"
GOVCS=""
GOVERSION="devel go1.21-f36dc54e9c Thu Feb 23 06:44:23 2023 +0000"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/vishwa/Workspaces/golang/go/src/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3776423934=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Race detection feature is not working properly on linux s390x. In order to evaluate that, I ran the "race.bash" script from "go/src" directory and I see that the script fails, and this is one of the errors that I am seeing.

What did you expect to see?

I expected the "race.bash" to run completely without any errors or issues.

What did you see instead?

The stack trace of the error is as below:

fatal error: malloc deadlock
panic during panic

goroutine 55 [running]:
runtime.throw({0x1293c20, 0xf})
        /home/vishwa/Workspaces/golang/go/src/runtime/panic.go:1075 +0x58 fp=0xc000121688 sp=0xc000121660 pc=0x1054228
runtime.mallocgc(0x30, 0x1279920, 0x1)
        /home/vishwa/Workspaces/golang/go/src/runtime/malloc.go:937 +0xb06 fp=0xc000121700 sp=0xc000121688 pc=0x10200b6
runtime.newobject(0x1279920)
        /home/vishwa/Workspaces/golang/go/src/runtime/malloc.go:1254 +0x3a fp=0xc000121728 sp=0xc000121700 pc=0x102029a
os.newProcess(...)
        /home/vishwa/Workspaces/golang/go/src/os/exec.go:29
os.startProcess({0xc00022a090, 0x15}, {0xc00020c120, 0x2, 0x2}, 0xc000121b30)
        /home/vishwa/Workspaces/golang/go/src/os/exec_posix.go:63 +0x75c fp=0xc000121868 sp=0xc000121728 pc=0x10e350c
os.StartProcess({0xc00022a090, 0x15}, {0xc00020c120, 0x2, 0x2}, 0xc000121b30)
        /home/vishwa/Workspaces/golang/go/src/os/exec.go:109 +0x7e fp=0xc0001218d0 sp=0xc000121868 pc=0x10e2cae
os/exec.(*Cmd).Start(0xc000214160)
        /home/vishwa/Workspaces/golang/go/src/os/exec/exec.go:693 +0xad4 fp=0xc000121b78 sp=0xc0001218d0 pc=0x1171de4
os/exec.(*Cmd).Run(0xc000214160)
        /home/vishwa/Workspaces/golang/go/src/os/exec/exec.go:587 +0x42 fp=0xc000121bb0 sp=0xc000121b78 pc=0x1171262
syscall_test.testAmbientCaps(0xc00023e4e0, 0x0)
        /home/vishwa/Workspaces/golang/go/src/syscall/exec_linux_test.go:633 +0x1462 fp=0xc000121e80 sp=0xc000121bb0 pc=0x11fd752
syscall_test.TestAmbientCaps(0xc00023e4e0)
        /home/vishwa/Workspaces/golang/go/src/syscall/exec_linux_test.go:526 +0x46 fp=0xc000121e98 sp=0xc000121e80 pc=0x11fc256
testing.tRunner(0xc00023e4e0, 0x12a0ee0)
        /home/vishwa/Workspaces/golang/go/src/testing/testing.go:1579 +0x244 fp=0xc000121fb0 sp=0xc000121e98 pc=0x115bd84
testing.(*T).Run.func1()
        /home/vishwa/Workspaces/golang/go/src/testing/testing.go:1632 +0x8e fp=0xc000121fd8 sp=0xc000121fb0 pc=0x115db8e
runtime.goexit()
        /home/vishwa/Workspaces/golang/go/src/runtime/asm_s390x.s:749 +0x2 fp=0xc000121fd8 sp=0xc000121fd8 pc=0x1096202
created by testing.(*T).Run in goroutine 1
        /home/vishwa/Workspaces/golang/go/src/testing/testing.go:1632 +0x894
@Vishwanatha-HD Vishwanatha-HD added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. labels Feb 28, 2023
@Vishwanatha-HD Vishwanatha-HD changed the title runtime.mallocgc: fatal error: malloc deadlock runtime.mallocgc: fatal error: malloc deadlock on s390x Feb 28, 2023
@bcmills
Copy link
Contributor

bcmills commented Feb 28, 2023

(attn @golang/s390x)

@Vishwanatha-HD Vishwanatha-HD self-assigned this Mar 1, 2023
@mknyszek mknyszek changed the title runtime.mallocgc: fatal error: malloc deadlock on s390x runtime: fatal error: malloc deadlock on s390x Mar 1, 2023
@mknyszek
Copy link
Contributor

mknyszek commented Mar 1, 2023

That error message comes up when malloc is reentered from within malloc, but that's clearly not happening in the stack trace. That suggests to us (here in triage) that it's actually memory corruption of some kind, otherwise in theory we would be seeing this failure elsewhere. The mechanism for checking this condition is very simple, so I doubt it's actually a problem with the mechanism itself, either.

@mknyszek mknyszek added this to the Backlog milestone Mar 1, 2023
@Vishwanatha-HD Vishwanatha-HD added the NeedsFix The path to resolution is known, but the work has not been done. label Apr 1, 2023
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 1, 2023
@Vishwanatha-HD Vishwanatha-HD added Testing An issue that has been verified to require only test changes, not just a test failure. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 1, 2023
@gopherbot gopherbot removed the NeedsFix The path to resolution is known, but the work has not been done. label Apr 1, 2023
@Vishwanatha-HD
Copy link
Contributor Author

Vishwanatha-HD commented Apr 12, 2023 •
edited

Change https://go.dev/cl/481415 mentions this issue: syscall: add a Go directive "go:nocheckptr" to forkAndExecInChild1 and another change https://go.dev/cl/480575 mentions this issue: crypto/subtle: don't cast to *uintptr when word size is 0.

With above two CLs, this issue was taken care automatically. Hence closing this issue since the race detection is working fine on s390x arch.

@Vishwanatha-HD Vishwanatha-HD removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 12, 2023
@golang golang locked and limited conversation to collaborators Apr 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@Vishwanatha-HD Vishwanatha-HD

Labels
arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@mknyszek @bcmills @gopherbot @Vishwanatha-HD