Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: poor performance in simulated environment #56704

Closed
Hendrik-H opened this issue Nov 11, 2022 · 4 comments
Closed

runtime: poor performance in simulated environment #56704

Hendrik-H opened this issue Nov 11, 2022 · 4 comments
Labels
arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@Hendrik-H
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.19.2 linux/s390x

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="s390x"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/go/pkg/tool/linux_s390x"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"
AR="ar"
CC="s390x-linux-gnu-gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1969591831=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I wrote a little test tool that start 50 goroutines that waste some time and exits once all goroutines are done.
main.go.txt

What did you expect to see?

When running the code on real hardware it finishes in about 200ms. I expected it to execute in roughly the same time in a virtual environment. We are using cecsim, which is based on KVM.

What did you see instead?

In our virtual environment the execution takes at least 10 times longer and the system shows a very high steal time. When using GODEBUG=asyncpreemptoff=1 the problem does not seem to ocur.
Here is an strace of a run on real hardware and one in a simulation:
testtool-1.19.2-strace-real-machine.txt
testtool-1.19.2-strace-sim.txt

The trace shows many nanosleep calls. Or assumption is that this might be the cause of the problem.
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 11, 2022
@prattmic
Copy link
Member

One obvious difference is that the simulated version makes lots of clock_gettime system calls. Normally those are handled by the VDSO in userspace. Making full system calls will certainly slow things down, though 10x is much more extreme than I would expect.

cc @golang/s390x

@prattmic prattmic added the arch-s390x Issues solely affecting the s390x architecture. label Nov 11, 2022
@prattmic prattmic added this to the Backlog milestone Nov 11, 2022
@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 11, 2022
@mknyszek
Copy link
Contributor

CC @golang/runtime

@mknyszek
Copy link
Contributor

I'm not sure what we can do here. We can try to call nanotime less but that's a pretty massive change to the runtime at this point. To clarify, what do you mean by "high steal time"? Also, given that the issue mainly only arises with GODEBUG=asyncpreemptoff=0, it may be that the simulated environment doesn't handle the OS signals the runtime sends very well. I'm not sure what we can do here about this on our side.

@mknyszek mknyszek added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 16, 2022
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Dec 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@mknyszek @prattmic @Hendrik-H @gopherbot