@cherrymui @suzmue reported this problem in go1.16 too (I am not sure if it's the same problem)

@cherrymui @suzmue reported this problem in go1.16 too (I am not sure if it's the same problem)

I looked at the DWARF produced by the 1.16 compiler. It is indeed broken, but it looks like in a different way (two different bugs).

Change https://golang.org/cl/332269 mentions this issue: cmd/compile: fix bug in dwarf-gen var location generation

With Go 1.17rc1, I am seeing that the location is correct within the systemstack call, but it is optimized out in the body of runtime.throw. Delve sets a breakpoint at runtime.fatalthrow for users to inspect the program state, so this is the point that it would be helpful to have access to the value of s.

suzmue@cloudshell:~/delve$ gdb fatalerror 
...
(gdb) b runtime.throw
Breakpoint 1 at 0x433820: file /usr/local/go/src/runtime/panic.go, line 1107.
(gdb) b runtime.fatalthrow
Breakpoint 2 at 0x433a20: file /usr/local/go/src/runtime/panic.go, line 1163.
(gdb) b /usr/local/go/src/runtime/panic.go:1111
Breakpoint 3 at 0x460eaf: file /usr/local/go/src/runtime/panic.go, line 1111.
(gdb) run
Starting program: /home/suzmue/delve/fatalerror 
...
Thread 1 "fatalerror" hit Breakpoint 1, runtime.throw (s=...) at /usr/local/go/src/runtime/panic.go:1107
1107    func throw(s string) {
(gdb) p s
$1 = <optimized out>
(gdb) c
Continuing.

Thread 1 "fatalerror" hit Breakpoint 3, runtime.throw.func1 () at /usr/local/go/src/runtime/panic.go:1111
1111                    print("fatal error: ", s, "\n")
(gdb) p s
$2 = 0x480a79 "go of nil func value"
(gdb) c
Continuing.
fatal error: go of nil func value

Thread 1 "fatalerror" hit Breakpoint 2, runtime.fatalthrow () at /usr/local/go/src/runtime/panic.go:1163
1163    func fatalthrow() {
(gdb) frame 1
#1  0x0000000000433892 in runtime.throw (s=...) at /usr/local/go/src/runtime/panic.go:1117
1117            fatalthrow()
(gdb) p s
$3 = <optimized out>

If this would be more appropriate as a new issue to open, since it seems that the incorrect location seems to be fixed I can do so.

Interesting. At the point of fatalthow call s is no longer live. Maybe we could keep it live in throw.

Why does it set a breakpoint in fatalthrow instead of throw?

I think its historical from when fatalpanic and fatalthrow split, but I'm not sure @derekparker @aarzilli would know better.

In both Go 1.16 and Go 1.17 the location info appears to only be available in the print function in runtime.throw:

systemstack(func() {
	print("fatal error: ", s, "\n")
})

It seems that s should also be live at the start of runtime.throw since it is not used print("fatal error: ", s, "\n"). Would it be possible to get the location information available then at least?

It seems that s should also be live at the start of runtime.throw

With @thanm 's fix I believe it should work for 1.17. I just tried at tip and it does seem to work

(gdb) b runtime.throw
Breakpoint 1 at 0x42b7e0: file /usr/local/google/home/cherryyz/src/go/src/runtime/panic.go, line 1188.
(gdb) run
Starting program: /tmp/d 
[New LWP 267089]
[New LWP 267090]
[New LWP 267091]
[New LWP 267092]
[Switching to LWP 267092]

Thread 5 "d" hit Breakpoint 1, runtime.throw (s="all goroutines are asleep - deadlock!") at /usr/local/google/home/cherryyz/src/go/src/runtime/panic.go:1188
1188	func throw(s string) {
(gdb) p s
$1 = "all goroutines are asleep - deadlock!"

I know it doesn't work with Go 1.16 due to some bug. @thanm would it be possible to fix 1.16 with something simple?

Thanks it does seem to be working at runtime.throw. Sorry about that those logs must have been using a binary built with 1.16.

@derekparker @aarzilli What do you think about moving the fatal throw breakpoint to runtime.throw instead of runtime.fatalthrow? This seems like it would be a more reliable place to get the information about the throw message.

Echoing what Cherry said, the crux of the problem is that "s" is no longer live at the call to fatalthrow; in general it isn't possible to generate a good location unless the lifetime is extended somehow.

The location at the start of runtime.throw should be accurate now in 1.17. There are still many other issues with debug locations, but that one at least is working.

accurate now in 1.16
Sorry, I meant 1.17 (I edited the comment)

@derekparker @aarzilli What do you think about moving the fatal throw breakpoint to runtime.throw instead of runtime.fatalthrow? This seems like it would be a more reliable place to get the information about the throw message.

I think that should be ok, but I haven't dug into the ramifications of the change. I'll take a closer look tomorrow, but it does seem like a more reliable spot.

On initial digging it seems nothing directly calls runtime.fatalthrow that I can see, so we wouldn't miss anything by setting the breakpoint on runtime.throw instead.

@suzmue ok if I close this issue out?

@thanm Yes definitely! Thanks for your help with this issue.

Delve has also switched to using runtime.throw in PR go-delve/delve#2616, so this information will be surfaced to users.

Cool, thanks.