cc @prattmic based on other compiler/runtime issues.

Could you collect a core file (with GOTRACEBACK=crash) and then open it in gdb and run thread apply all bt to see where all of the system threads are? (edit: or the equivalent in lldb)

Does this application have CPU profiling enabled? If so, please test using a toolchain built from master or release-branch.go1.20, which includes a fix for hung traceback (#58513).

cc @golang/runtime

Could you collect a core file (with GOTRACEBACK=crash) and then open it in gdb and run thread apply all bt to see where all of the system threads are? (edit: or the equivalent in lldb)

Is gdb available on macOS arm?

I couldn't find anything suggesting it was, only this thread from last year saying it wasn't available (yet): https://inbox.sourceware.org/gdb/3185c3b8-8a91-4beb-a5d5-9db6afb93713@Spark/.

Apologies if I'm missing something obvious. I actually develop on Linux - I'm just reproducing and tracking this down for others who develop on mac.

Does this application have CPU profiling enabled?

Please can you let me know how I determine this?

Could you collect a core file (with GOTRACEBACK=crash) and then open it in gdb and run thread apply all bt to see where all of the system threads are? (edit: or the equivalent in lldb)

Is gdb available on macOS arm?

I'm not sure, lldb should work fine too. bt all is the equivalent command in lldb.

Does this application have CPU profiling enabled?

Please can you let me know how I determine this?

If you aren't sure, but can readily reproduce this issue, it is probably simplest to just test with a Go toolchain built from release-branch.go1.20. To build one:

$ git clone https://go.googlesource.com/go
$ cd go/src
$ git checkout release-branch.go1.20
$ ./make.bash
$ # Now build hugo with `../bin/go`.

Oops, I forgot to add git checkout of the release branch to the instructions above. I've edited to add it.

Thanks.

I'm not sure, lldb should work fine too. bt all is the equivalent command in lldb.

A fresh stack trace and lldb output here:

https://gist.github.com/myitcv/32f362f6c12e829c348383b2617815c5

If you aren't sure, but can readily reproduce this issue, it is probably simplest to just test with a Go toolchain built from release-branch.go1.20

Doing so now.

Oops, I forgot to add git checkout of the release branch to the instructions above. I've edited to add it.

No problems. I actually have go version go1.20.1 darwin/arm64 available to perform a build.

Is that good enough?

Oops, I forgot to add git checkout of the release branch to the instructions above. I've edited to add it.

No problems. I actually have go version go1.20.1 darwin/arm64 available to perform a build.

Is that good enough?

No, the CL is merged to release-branch.go1.20 for 1.20.2, which hasn't been released yet.

* thread #8, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x16c107b50)
  * frame #0: 0x000000010411f8dc hugo`runtime.(*sigctxt).preparePanic + 76
    frame #1: 0x0000000104120a7c hugo`runtime.sighandler + 556
    frame #2: 0x000000010412051c hugo`runtime.sigtrampgo + 524
    frame #3: 0x000000010413f7bc hugo`runtime.sigtrampgo.abi0 + 28
    frame #4: 0x000000010413ec3c hugo`runtime.sigtramp.abi0 + 76
    frame #5: 0x00000001afbcc2a4 libsystem_platform.dylib`_sigtramp + 56
    frame #6: 0x00000001afbcc2a4 libsystem_platform.dylib`_sigtramp + 56
    frame #7: 0x000000010571d79c hugo`Sass::Eval::operator()(Sass::Binary_Expression*) + 4020
    frame #8: 0x000000010571d79c hugo`Sass::Eval::operator()(Sass::Binary_Expression*) + 4020
    frame #9: 0x0000000105728fa4 hugo`Sass::Eval::operator()(Sass::Argument*) + 72
    frame #10: 0x00000001057294e0 hugo`Sass::Eval::operator()(Sass::Arguments*) + 232
    frame #11: 0x0000000105722440 hugo`Sass::Eval::operator()(Sass::Function_Call*) + 1524
    frame #12: 0x0000000105728fa4 hugo`Sass::Eval::operator()(Sass::Argument*) + 72
    ... lots more frames (798 total) ...

This thread is panicking. I'm actually not 100% sure if this is just from the signal you sent to generate the core file, or it was actually stuck here before the core dump.

If the latter, it might be #58513, as that can get stuck in panic. OTOH, typically that bug should appear in traceback code in the panic, not in preparePanic.

That makes me think it is the former, in which case it looks like the C++ library was just stuck executing forever itself.

I guess another way to look at this would be to execute the program directly under lldb. Let it run until it seems stuck, then stop it and check bt all to see if it is in a panic, or just in the C++ library.

Thanks for this.

If the latter, it might be #58513, as that can get stuck in panic. OTOH, typically that bug should appear in traceback code in the panic, not in preparePanic.

In which case I will continue with my efforts to try and repro this building hugo with a later Go version, to conclusively rule that in/out.

No, the CL is merged to release-branch.go1.20 for 1.20.2, which hasn't been released yet.

Noted, thanks.

Ok, I was unable to reproduce this with the latest commit on the go1.20 branch. Which led me to bisect to find the fix.

Turns out the scenario I was running into was fixed by 7d3a5a5.

@prattmic - if you're happy 7d3a5a5 can explain what we've seen here, please feel free to close this issue.

Huh, well you certainly had a deep cgo stack, so that makes sense. I don't fully understand why the program was hanging rather than crashing with a SIGSEGV on stack overflow, but that is OK.