Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: esbuild sometimes crashes with fatal: morestack on g0 on 64-bit Windows #51266

Closed
evanw opened this issue Feb 18, 2022 · 5 comments
Closed

runtime: esbuild sometimes crashes with fatal: morestack on g0 on 64-bit Windows #51266

evanw opened this issue Feb 18, 2022 · 5 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Unplanned

Comments

@evanw
Copy link

evanw commented Feb 18, 2022

What version of Go are you using (go version)?

$ go version
go version go1.17.7 darwin/arm64

Does this issue reproduce with the latest release?

Yes, 1.17.7 is the latest release.

What operating system and processor architecture are you using (go env)?

(this is the OS used to build the executable, but not to run it)

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/evan/Library/Caches/go-build"
GOENV="/Users/evan/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/evan/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/evan/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.17.7"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mj/fcy686697qn53dt7g4rvv2540000gn/T/go-build1751159435=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

This is a new bug report as suggested by #39457 (comment). See this issue for full context: evanw/esbuild#2031. Specifically I'm the author of esbuild, a JavaScript build tool written in Go, and I have a user reporting intermittent crashes. The crashes either say fatal: morestack on g0 or something like [signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x512894].

What did you expect to see?

I expect users to be able to run esbuild successfully without intermittent crashes.

What did you see instead?

The crashes look something like this (see evanw/esbuild#2031 for the full stack trace):

fatal: morestack on g0
  Exception 0x80000003 0x0 0x0 0x868781
  PC=0x868781
  
  runtime.abort()
  	runtime/asm_amd64.s:978 +0x1
  runtime.morestack()
  	runtime/asm_amd64.s:429 +0x2a

  goroutine 1 [syscall, locked to thread]:
  syscall.Syscall6(0x7ffe1f3a4ee0, 0x5, 0x234, 0xc000202000, 0x4000, 0xc000197c54, 0x0, 0x0)
  	runtime/syscall_windows.go:497 +0xfa
  syscall.ReadFile(0x26db6f7a758, {0xc000202000, 0x4000, 0x800000}, 0xc000197c54, 0x0)
  	syscall/zsyscall_windows.go:1024 +0xbe
  syscall.Read(0xc000198000, {0xc000202000, 0x0, 0xb})
  	syscall/syscall_windows.go:380 +0x2e
  internal/poll.(*FD).Read(0xc000198000, {0xc000202000, 0x4000, 0x4000})
  	internal/poll/fd_windows.go:427 +0x1b4
  os.(*File).read(...)
  	os/file_posix.go:32
  os.(*File).Read(0xc000006010, {0xc000202000, 0x0, 0x0})
  	os/file.go:119 +0x5e
  main.runService(0x1)
  	github.com/evanw/esbuild/cmd/esbuild/service.go:141 +0x2ed
  main.main()
  	github.com/evanw/esbuild/cmd/esbuild/main.go:207 +0x1f0

...

or like this:

  [signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x512894]
  
  goroutine 15 [running]:
  runtime.throw({0x98a902, 0x899d8ea3759f6296})
  	runtime/panic.go:1198 +0x76 fp=0xc000ca74f8 sp=0xc000ca74c8 pc=0x547dd6
  runtime.sigpanic()
  	runtime/signal_windows.go:260 +0x10c fp=0xc000ca7540 sp=0xc000ca74f8 pc=0x55c22c
  memeqbody()
  	internal/bytealg/equal_amd64.s:183 +0x114 fp=0xc000ca7548 sp=0xc000ca7540 pc=0x512894
  strings.HasSuffix(...)
  	strings/strings.go:450
  github.com/evanw/esbuild/internal/resolver.resolverQuery.esmPackageImportsExportsResolve({0xc00029e700, 0xc0002c4780, 0x0, 0x28}, {0xc00028ad60, 0xc0005623f0}, {{0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
  	github.com/evanw/esbuild/internal/resolver/package_json.go:802 +0x269 fp=0xc000ca79d8 sp=0xc000ca7548 pc=0x83b169
  github.com/evanw/esbuild/internal/resolver.resolverQuery.esmPackageExportsResolve({0xc00029e700, 0xc0002c4780, 0x0, 0x0}, {0x9899f2, 0x0}, {0xc00028ad60, 0x1b}, {{0x0, 0x0}, ...}, ...)
  	github.com/evanw/esbuild/internal/resolver/package_json.go:779 +0x771 fp=0xc000ca7d88 sp=0xc000ca79d8 pc=0x83ac91
  github.com/evanw/esbuild/internal/resolver.resolverQuery.loadNodeModules({0xc00029e700, 0xc0002c4780, 0x0, 0x0}, {0xc000342000, 0x0}, 0xc000491960, 0x0)
  	github.com/evanw/esbuild/internal/resolver/resolver.go:1684 +0x204f fp=0xc000ca88e0 sp=0xc000ca7d88 pc=0x85384f
  github.com/evanw/esbuild/internal/resolver.resolverQuery.resolveWithoutRemapping({0xc00029e700, 0xc0002c4780, 0x0, 0x0}, 0xc000491960, {0xc000342000, 0x28})
  	github.com/evanw/esbuild/internal/resolver/resolver.go:764 +0x1ed fp=0xc000ca8b10 sp=0xc000ca88e0 pc=0x84762d
  github.com/evanw/esbuild/internal/resolver.resolverQuery.resolveWithoutSymlinks({0xc00029e700, 0xc0002c4780, 0x0, 0x18}, {0xc00031a000, 0x5c}, {0xc000342000, 0x28})
  	github.com/evanw/esbuild/internal/resolver/resolver.go:751 +0x1445 fp=0xc000ca92e8 sp=0xc000ca8b10 pc=0x847005
  github.com/evanw/esbuild/internal/resolver.(*resolver).Resolve(0xc00029e700, {0xc00031a000, 0x5c}, {0xc000342000, 0x28}, 0x1)
  	github.com/evanw/esbuild/internal/resolver/resolver.go:380 +0x114d fp=0xc000ca9668 sp=0xc000ca92e8 pc=0x842c6d
  github.com/evanw/esbuild/internal/bundler.runOnResolvePlugins({0x0, 0x0, 0x0}, {0xa36fd8, 0xc00029e700}, {0x4, 0xc0002ce000, 0xc00028c0a8, 0xc00028c0c0, 0xc0002880c0}, ...)
  	github.com/evanw/esbuild/internal/bundler/bundler.go:778 +0x2ee fp=0xc000ca9bf8 sp=0xc000ca9668 pc=0x8687ee
  github.com/evanw/esbuild/internal/bundler.parseFile({{0xa3d6f0, 0xc0002d0040}, {0x4, 0xc0002ce000, 0xc00028c0a8, 0xc00028c0c0, 0xc0002880c0}, {0xa36fd8, 0xc00029e700}, 0xc0002cc060, ...})
  	github.com/evanw/esbuild/internal/bundler/bundler.go:396 +0x3e8d fp=0xc000cabbf8 sp=0xc000ca9bf8 pc=0x865b2d
  github.com/evanw/esbuild/internal/bundler.(*scanner).maybeParseFile·dwrap·3()
  	github.com/evanw/esbuild/internal/bundler/bundler.go:1163 +0x58 fp=0xc000cabfe0 sp=0xc000cabbf8 pc=0x86c9f8
  runtime.goexit()
  	runtime/asm_amd64.s:1581 +0x1 fp=0xc000cabfe8 sp=0xc000cabfe0 pc=0x578901
  created by github.com/evanw/esbuild/internal/bundler.(*scanner).maybeParseFile
  	github.com/evanw/esbuild/internal/bundler/bundler.go:1163 +0x865
  
  goroutine 1 [syscall, locked to thread]:
  syscall.Syscall6(0x7ffec61d4ee0, 0x5, 0x234, 0xc0001fc000, 0x4000, 0xc000197c04, 0x0, 0x0)
  	runtime/syscall_windows.go:497 +0xfa
  syscall.ReadFile(0x22561a4a7e0, {0xc0001fc000, 0x4000, 0x800000}, 0xc000197c04, 0x0)
  	syscall/zsyscall_windows.go:1024 +0xbe
  syscall.Read(0xc000198000, {0xc0001fc000, 0x0, 0xa})
  	syscall/syscall_windows.go:380 +0x2e
  internal/poll.(*FD).Read(0xc000198000, {0xc0001fc000, 0x4000, 0x4000})
  	internal/poll/fd_windows.go:427 +0x1b4
  os.(*File).read(...)
  	os/file_posix.go:32
  os.(*File).Read(0xc000006010, {0xc0001fc000, 0xc1f940, 0x20})
  	os/file.go:119 +0x5e
  main.runService(0x1)
  	github.com/evanw/esbuild/cmd/esbuild/service.go:101 +0x37b
  main.main()
  	github.com/evanw/esbuild/cmd/esbuild/main.go:203 +0x1eb
  
...

The problem appears to happen on 64-bit Windows. Note that esbuild does not make use of cgo. The machine is "a 3970x, i.e. 32 cores, 64 threads" according to the user. I cannot reproduce this issue myself (I don't even have a Windows machine) so I can't help debug this myself.
@evanw evanw changed the title affected/package: github.com/evanw/esbuild/pkg/cli esbuild sometimes crashes with fatal: morestack on g0 on 64-bit Windows Feb 18, 2022
@ALTree ALTree changed the title esbuild sometimes crashes with fatal: morestack on g0 on 64-bit Windows runtime: esbuild sometimes crashes with fatal: morestack on g0 on 64-bit Windows Feb 18, 2022
@ALTree ALTree added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows labels Feb 18, 2022
@dmitshur
Copy link
Contributor

CC @ianlancetaylor, @cherrymui.

@EamonNerbonne
Copy link

EamonNerbonne commented Feb 19, 2022 •
edited

I'm the user that reported this, but I don't use go directly. I'd be willing to add whatever instrumentation necessary if that's helpful however; I'm just somewhat out of my depth here. I don't know how to reproduce the issue consistently; it just ... rarely happens, but usually does not.

@randall77
Copy link
Contributor

Not sure what's going on with the first trace. It looks just like it is waiting for a file read, and the error happened in some other goroutine. We'd need more of the printed stack traces, I think.

That second trace has more information. It's trying to do a string == with presumably a bad string. I'm not sure if it is inside if !strings.HasSuffix(matchKey, "*") { (line 835 at tip) or if strings.HasSuffix(expansion.key, "*") { (line 847 at tip). (Neither line matches the stack trace line - what hash of your esbuild repository was that error capture at?)

The matchKey value in the trace looks maybe bad? It is {0xc00028ad60, 0xc0005623f0} and that second word should be the length of the string, which is way too big. That would cause strings.HasSuffix to make a string with a bad pointer to pass to memeq, which is where we see the crash. The matchKey value isn't necessarily correct in the traceback though, so it is hard to be sure. And the fault address is -1, which doesn't make a lot of sense, unless Windows doesn't report faulting addresses and that's just a "I don't know" fallback.

Have you run this under the race detector?

It would really help if you could catch this in the act under a debugger, so you can inspect the length of matchKey being passed to HasSuffix. Or give us instructions sufficient to reproduce ourselves.

unsafe uses in this module look ok, but I haven't checked the dependencies.

Some debugging things you could do:

  • Add if len(matchKey) > 1<<30 { panic... } in a few places. Same for the arg to the other HasSuffix call. See if you still get the same crashes.
  • Run under the race detector
  • Does this fail with 1.16? the current 1.18 release candidate? Did it ever work with a different Go release? The more you can bisect between a good and bad Go release, the better.

@evanw
Copy link
Author

evanw commented Mar 1, 2022

unsafe uses in this module look ok, but I haven't checked the dependencies.

FYI the only dependency is on the golang.org/x/sys library: https://github.com/evanw/esbuild/blob/a1ff9d144cdb8d50ea2fa79a1d11f43d5bd5e2d8/go.mod.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 20, 2022
@seankhliao seankhliao added this to the Unplanned milestone Aug 20, 2022
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
7 participants
@evanw @EamonNerbonne @dmitshur @ALTree @randall77 @gopherbot @seankhliao