runtime: unable to backtrace program using Go shared object #52608
Comments
|
Can you share a complete program that we can use to replicate the problem? Thanks. I note a call to |
|
CC @golang/runtime |
ianlancetaylor
added
the
WaitingForInfo
ianlancetaylor
changed the title
|
We've learned something new. The software is stable when Go runs as a separate process using identical code. We can "connect" the C++ application and Go application using a socket, replacing CGo, and the system is stable. If we run Go within the C++ application in a shared object, that initializes using module func init(), using a socket to replace CGo, the system is unstable - it appears to crash (SIGSEV) randomly with invalid stack traces, always in Go. In the shared object configuration, CGO_ENABLED must be set during the Go compile despite CGo not technically being used (there is no import "C" and no //export tagged Go functions). Essentially, CGo is eliminated by exchanging all data using a socket, instead of CGo, using a simple coupling like this. ` // Go listens on port 6001. "C" listens on port 6000 func ReadUDP() *GMessage { func WriteUDP(msg *GMessage) { in Go, with a similar implementation in C++. Both runtimes block on the sockets waiting for events from the other. We want to use CGo, because of its simplicity and speed, instead of a contraption like this. The larger application is too large to share - the Go part is smaller and dedicated to interfacing with Google gRPC services. The common denominator here is the appearence that the C++ and Go runtimes are not compatible in the same process. For Go to call "C" code, it may be that it may be that it can do so only as a Go process. Having a "C" process calling Go, while also having a Go runtime, might be the issue. But, if this is the case, then why support Go as a shared object for a "C" application? I saw a comment on a board that said something like, and I'm paraphrasing, "the C and Go runtimes cannot be used together" but I've never seen this in official documentation. One interesting official documentation comment is here: Regarding the large slices suspected in the traceback. We went as far as logging slice sizes along the C/Go boundary and did not encounter anything unusual. I think any invalid slice range seen is a product of some sort of trampling - which also explains the corrupted stacks - a problem that does not occur, running the same code, as separate C and Go processes. |
|
Calling a Go shared library from a C program is in general supported. I don't know what is happening here. Have you tried using When Go and C code are linked together, it's definitely possible for C to trample into Go, and it's possible for Go code that uses unsafe to trample into C. |
|
When linked using static libs, I notice a few things. First, the func init() is not reliable. Whether it breaks or not is a compile time issue. I've seen this before in general... sometimes an init() is not called. I work around that by calling what's in the init() for one go file from another and as long as it runs somewhere (like a factory registration) then it's ok. But there is something wrong with init() - that's only a nuisance. With the Go code linked static everything is now in the C++ binary (which makes it large, but about the sum of its parts) and the same issue occurs: Program received signal SIGSEGV, Segmentation fault. (gdb) frame 10 This is using a socket interface to exchange messages between C++ and Go so CGo is not used. The applications(s) run fine as separate processes (they ran the same test as such overnight in heavy traffic with no issues). |
|
This issue does not reproduce with a minimal application that drives the Go library in a similar manner than the real one. The real application is too complex to be able to share for diagnostic purposes. Here's some facts:
Something is coupling execution of the application and the Go code (as a library) causing corruption in thread stacks. |
seankhliao
added
WaitingForInfo
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
There appears to be an issue with the Golang GC when CGo is used with a "C++" application that uses a Go shared object.
go version go1.18 linux/amd64
It reproduces with previous releases, the very latest release has not been tried (the problem being investigated was observed in 1.17, which was updated to 1.18 along the way). It is possible this issue is related to https://github.com/golang/go/issues/19928 and a question was posted to that older report. This is a follow-up to that post after the issue was investigated further.
The build environment is:
GO111MODULE=""
GOARCH="amd64"
GOBIN="/home/defrain/legit3/ams/mediaserver/Golang/go/bin"
GOCACHE="/home/defrain/.cache/go-build"
GOENV="/home/defrain/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/defrain/legit3/ams/mediaserver/Golang/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/defrain/legit3/ams/mediaserver/Golang/go:/home/defrain/legit3/ams/mediaserver/Golang/src/libgo"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/defrain/legit3/ams/mediaserver/Golang/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/defrain/legit3/ams/mediaserver/Golang/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/defrain/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1959834666=/tmp/go-build -gno-record-gcc-switches"
The code uses CGo, two functions, to pass serialized messages buffers between "C++" <-> Go. The C++ application is the "main" which loads a Golang shared object, libso.so. The C++ application is compiled using the intel compiler, libgo uses "go build -v -buildmode=c-shared -o Linux/release64/libgo.so *.go"
Here is the CGo interface part of the code:
`// -----------------------------------------------------------
//
// C++ API - "C" functions used to access Go
//
// -----------------------------------------------------------
//
// typedef struct {
// void* buf;
// unsigned long length;
// } CSlice;
//
// typedef void (*send_to_go)(CSlice cs);
// extern void SendToGo(CSlice cs);
//
// typedef CSlice (*read_from_go)();
// extern CSlice ReadFromGo();
//
import "C"
import (
"os"
"fmt"
"net"
"time"
"sync"
"errors"
"unsafe"
"strconv"
)
var IncomingChannel chan *GMessage = make(chan *GMessage, 1000)
var OutgoingChannel chan *GMessage = make(chan *GMessage, 1000)
//export SendToGo
func SendToGo(cs C.CSlice) {
// Called from "C" to send a message to Go
var cptr unsafe.Pointer = unsafe.Pointer(cs.buf)
bp := unsafe.Slice((*byte)(cptr), cs.length)
msg := deserializeGMessage(bp)
IncomingChannel <- msg
}
//export ReadFromGo
func ReadFromGo() C.CSlice {
// Called by "C" to get a message from Go
msg := <- OutgoingChannel
var cs C.CSlice
cs.length = C.ulong(msg.getTransmitSize())
cs.buf = C.malloc(cs.length)
gbuf := unsafe.Slice((*byte)(cs.buf), cs.length)
msg.serializeGMessage(gbuf, int(cs.length))
return cs
}
`
where GMessage is used to marshal/unmarshal an object that stores label/value pairs. The "C++" code has a thread that sits on the OutgoingChannel and has other threads that post to the IncomingChannel.
I expect the code to run forever - and it does with GOGC=off until memory is exhausted. When GOGC is enabled, it will eventually panic here:
runtime: unexpected return pc for main.ReadUDP called from 0x7fff4c0112b0
stack: frame={sp:0xc00006ad38, fp:0xc00006adf8} stack=[0xc00006a000,0xc00006b000)
0x000000c00006ac38: 0x00007fffe44b7879 <fmt.Sprintf+0x0000000000000059> 0x000000c00011e270
0x000000c00006ac48: 0x00007fffe498e874 0x0000000000000036
0x000000c00006ac58: 0x000000c00006adb8 0x0000000000000003
0x000000c00006ac68: 0x0000000000000003 0x0000000000000000
0x000000c00006ac78: 0x0000000000000000 0x000000c00011e270
0x000000c00006ac88: 0x000000c00006ad28 0x00007fffe4965d45 <main.DebugLogV+0x00000000000000c5>
0x000000c00006ac98: 0x00007fffe498e874 0x0000000000000036
0x000000c00006aca8: 0x000000c00006adb8 0x0000000000000003
0x000000c00006acb8: 0x0000000000000003 0x00007ffff7f9b228
0x000000c00006acc8: 0x0000000000000008 0x0000000000000016
0x000000c00006acd8: 0x0000000000000010 0x000000c000154150
0x000000c00006ace8: 0x0000000000203000 0x000000c0005a3040
0x000000c00006acf8: 0x000000c00006ad28 0x000000c00006ad28
0x000000c00006ad08: 0x00007fffe43da105 <runtime.convT64+0x0000000000000045> 0x0000000000000008
0x000000c00006ad18: 0x00007fffe4cd6e20 0x0000000000000000
0x000000c00006ad28: 0x000000c00006ade8 0x00007fffe496a3ab <main.ReadUDP+0x000000000000026b>
0x000000c00006ad38: <0x0000000000003eca 0x00007fffe498e874
0x000000c00006ad48: 0x0000000000000036 0x000000c00006adb8
0x000000c00006ad58: 0x0000000000000003 0x0000000000000003
0x000000c00006ad68: 0x0000000000000005 0x0000000000003eca
0x000000c00006ad78: 0x00007fff4c015100 0x0000000000000003
0x000000c00006ad88: 0x000000000000000a 0x000000c00003a7b0
0x000000c00006ad98: 0x000000c0003bd800 0x000000c00006add0
0x000000c00006ada8: 0x00007ffff78df80d 0x0000000000056c9f
0x000000c00006adb8: 0x00007fffe4dada80 0x00007ffff78df80d
0x000000c00006adc8: 0x00007fffe4cd6e60 0x000000c0000321a0
0x000000c00006add8: 0x00007fffe4cd63e0 0x000000c00003a7b8
0x000000c00006ade8: 0x000000c000000000 !0x00007fff4c0112b0
0x000000c00006adf8: >0x000000000003ce14 0x000000000e3b6fef
0x000000c00006ae08: 0x00007fff40592630 0x000000c00006ef68
0x000000c00006ae18: 0x0000000000000005 0x0000000000000005
0x000000c00006ae28: 0x000000000000000a 0x00007fff4c0112b0
0x000000c00006ae38: 0x0000000000000000 0x000000c00006ae70
0x000000c00006ae48: 0x00007ffff78c38d6 0x0000000000000000
0x000000c00006ae58: 0x0000000e0000000a 0x0000000000000005
0x000000c00006ae68: 0x00007fff4c0112e8 0x000000c00006bbf0
0x000000c00006ae78: 0x00007fffe3b0a630 0x0000000000000001
0x000000c00006ae88: 0x0000000000000000 0x000000c000080000
0x000000c00006ae98: 0x0000000000000000 0x0000000000008000
0x000000c00006aea8: 0x000000c00006bc20 0x0000000000000000
0x000000c00006aeb8: 0x000000c000723000 0x0000000000000020
0x000000c00006aec8: 0x000000c000722000 0x0000000000000000
0x000000c00006aed8: 0x000000c0001029c0 0xffffffffffffffff
0x000000c00006aee8: 0x000000c000723520 0x000000c0007225a0
fatal error: unknown caller pc
runtime stack:
runtime.throw({0x7fffe4976978?, 0x7fffe52a4200?})
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/panic.go:992 +0x71
runtime.gentraceback(0x7fff3c03f8cb?, 0x200000004?, 0xc0000031e0?, 0x7fffb05f3d40?, 0x0, 0x0, 0x7fffffff, 0x7fffb05f3d28, 0xc00006ac78?, 0x0)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/traceback.go:254 +0x1a36
runtime.addOneOpenDeferFrame.func1()
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/panic.go:599 +0x6b
runtime.systemstack()
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/asm_amd64.s:469 +0x46
goroutine 29 [running]:
runtime.systemstack_switch()
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/asm_amd64.s:436 fp=0xc00006a588 sp=0xc00006a580 pc=0x7fffe4430b20
runtime.addOneOpenDeferFrame(0x4?, 0x1770?, 0xe?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/panic.go:598 +0x69 fp=0xc00006a5c8 sp=0xc00006a588 pc=0x7fffe44023e9
panic({0x7fffe4d10dc0, 0x7fffe4e1b1a0})
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/panic.go:798 +0x106 fp=0xc00006a688 sp=0xc00006a5c8 pc=0x7fffe4402c06
runtime.panicmakeslicelen(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/slice.go:29
runtime.makeslice(0xc0005a2820?, 0x7fffe496cbc0?, 0xc00006a720?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/slice.go:98 +0x78 fp=0xc00006a6b0 sp=0xc00006a688 pc=0x7fffe4419798
net.hexString(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/ip.go:384
net.IP.String({0xe8458b4801fc4583, 0x8f0ffc453b28408b, 0x0?})
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/ip.go:341 +0xc5 fp=0xc00006a7a0 sp=0xc00006a6b0 pc=0x7fffe4580ec5
net.ipEmptyString(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/ip.go:397
net.(*UDPAddr).String(0x7ffff78df80d)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock.go:51 +0x55 fp=0xc00006a858 sp=0xc00006a7a0 pc=0x7fffe4596175
fmt.(*pp).handleMethods(0xc00011e270, 0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/fmt/print.go:626 +0x30b fp=0xc00006aaa8 sp=0xc00006a858 pc=0x7fffe44b9fab
fmt.(*pp).printArg(0xc00011e270, {0x7fffe4dada80?, 0x7ffff78df80d}, 0x73)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/fmt/print.go:709 +0x693 fp=0xc00006ab48 sp=0xc00006aaa8 pc=0x7fffe44bab53
fmt.(*pp).doPrintf(0xc00011e270, {0x7fffe498e874, 0x36}, {0xc00006adb8?, 0x3, 0x3})
/home/defrain/legit3/ams/mediaserver/Golang/go/src/fmt/print.go:1026 +0x288 fp=0xc00006ac40 sp=0xc00006ab48 pc=0x7fffe44bd408
fmt.Sprintf({0x7fffe498e874, 0x36}, {0xc00006adb8, 0x3, 0x3})
/home/defrain/legit3/ams/mediaserver/Golang/go/src/fmt/print.go:219 +0x59 fp=0xc00006ac98 sp=0xc00006ac40 pc=0x7fffe44b7879
main.DebugLogV(0x3eca?, {0x7fffe498e874, 0x36}, {0xc00006adb8, 0x3, 0x3})
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:126 +0xc5 fp=0xc00006ad38 sp=0xc00006ac98 pc=0x7fffe4965d45
runtime: unexpected return pc for main.ReadUDP called from 0x7fff4c0112b0
stack: frame={sp:0xc00006ad38, fp:0xc00006adf8} stack=[0xc00006a000,0xc00006b000)
0x000000c00006ac38: 0x00007fffe44b7879 <fmt.Sprintf+0x0000000000000059> 0x000000c00011e270
0x000000c00006ac48: 0x00007fffe498e874 0x0000000000000036
0x000000c00006ac58: 0x000000c00006adb8 0x0000000000000003
0x000000c00006ac68: 0x0000000000000003 0x0000000000000000
0x000000c00006ac78: 0x0000000000000000 0x000000c00011e270
0x000000c00006ac88: 0x000000c00006ad28 0x00007fffe4965d45 <main.DebugLogV+0x00000000000000c5>
0x000000c00006ac98: 0x00007fffe498e874 0x0000000000000036
0x000000c00006aca8: 0x000000c00006adb8 0x0000000000000003
0x000000c00006acb8: 0x0000000000000003 0x00007ffff7f9b228
0x000000c00006acc8: 0x0000000000000008 0x0000000000000016
0x000000c00006acd8: 0x0000000000000010 0x000000c000154150
0x000000c00006ace8: 0x0000000000203000 0x000000c0005a3040
0x000000c00006acf8: 0x000000c00006ad28 0x000000c00006ad28
0x000000c00006ad08: 0x00007fffe43da105 <runtime.convT64+0x0000000000000045> 0x0000000000000008
0x000000c00006ad18: 0x00007fffe4cd6e20 0x0000000000000000
0x000000c00006ad28: 0x000000c00006ade8 0x00007fffe496a3ab <main.ReadUDP+0x000000000000026b>
0x000000c00006ad38: <0x0000000000003eca 0x00007fffe498e874
0x000000c00006ad48: 0x0000000000000036 0x000000c00006adb8
0x000000c00006ad58: 0x0000000000000003 0x0000000000000003
0x000000c00006ad68: 0x0000000000000005 0x0000000000003eca
0x000000c00006ad78: 0x00007fff4c015100 0x0000000000000003
0x000000c00006ad88: 0x000000000000000a 0x000000c00003a7b0
0x000000c00006ad98: 0x000000c0003bd800 0x000000c00006add0
0x000000c00006ada8: 0x00007ffff78df80d 0x0000000000056c9f
0x000000c00006adb8: 0x00007fffe4dada80 0x00007ffff78df80d
0x000000c00006adc8: 0x00007fffe4cd6e60 0x000000c0000321a0
0x000000c00006add8: 0x00007fffe4cd63e0 0x000000c00003a7b8
0x000000c00006ade8: 0x000000c000000000 !0x00007fff4c0112b0
0x000000c00006adf8: >0x000000000003ce14 0x000000000e3b6fef
0x000000c00006ae08: 0x00007fff40592630 0x000000c00006ef68
0x000000c00006ae18: 0x0000000000000005 0x0000000000000005
0x000000c00006ae28: 0x000000000000000a 0x00007fff4c0112b0
0x000000c00006ae38: 0x0000000000000000 0x000000c00006ae70
0x000000c00006ae48: 0x00007ffff78c38d6 0x0000000000000000
0x000000c00006ae58: 0x0000000e0000000a 0x0000000000000005
0x000000c00006ae68: 0x00007fff4c0112e8 0x000000c00006bbf0
0x000000c00006ae78: 0x00007fffe3b0a630 0x0000000000000001
0x000000c00006ae88: 0x0000000000000000 0x000000c000080000
0x000000c00006ae98: 0x0000000000000000 0x0000000000008000
0x000000c00006aea8: 0x000000c00006bc20 0x0000000000000000
0x000000c00006aeb8: 0x000000c000723000 0x0000000000000020
0x000000c00006aec8: 0x000000c000722000 0x0000000000000000
0x000000c00006aed8: 0x000000c0001029c0 0xffffffffffffffff
0x000000c00006aee8: 0x000000c000723520 0x000000c0007225a0
main.ReadUDP()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:704 +0x26b fp=0xc00006adf8 sp=0xc00006ad38 pc=0x7fffe496a3ab
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:613 +0x185
goroutine 19 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00020a080)
/home/defrain/legit3/ams/mediaserver/Golang/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
/home/defrain/legit3/ams/mediaserver/Golang/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x8d
goroutine 6 [sleep]:
time.Sleep(0x3b9aca00)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/time.go:194 +0x12e
main.libgoAudit()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/utility.go:111 +0x7d
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:640 +0x33e
goroutine 22 [semacquire]:
internal/poll.runtime_Semacquire(0xc00020fbb0?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00020aa80, 0xea?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).ReadFromInet4(0xc00020aa80, {0xc0000d0000, 0x4e20, 0x4e20}, 0xa000?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_unix.go:235 +0x69
net.(*netFD).readFromInet4(0xc00020aa80, {0xc0000d0000?, 0x0?, 0xc00020fc40?}, 0x4?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/fd_posix.go:66 +0x29
net.(*UDPConn).readFrom(0xc00005ec00?, {0xc0000d0000?, 0x0?, 0xa?}, 0xc0001f4a50)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock_posix.go:52 +0x1b8
net.(*UDPConn).readFromUDP(0xc00013a3d0, {0xc0000d0000?, 0xc00020fde8?, 0x7fffe4965e95?}, 0xc00006bdb8?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock.go:149 +0x31
net.(*UDPConn).ReadFromUDP(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock.go:141
main.ReadUDP()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:696 +0x8d
main.incomingMessagesWorker(0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:391 +0x7c
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:591 +0x9c
goroutine 23 [runnable]:
main.deserializeGMessage({0xc000218000, 0x4e20, 0xc000213de8?})
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/gmessage.go:285 +0x167
main.ReadUDP()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:698 +0xa9
main.incomingMessagesWorker(0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:391 +0x7c
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:613 +0x185
goroutine 24 [runnable]:
main.deserializeGMessage({0xc000b98000, 0x4e20, 0xc00020dde8?})
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/gmessage.go:285 +0x167
main.ReadUDP()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:698 +0xa9
main.incomingMessagesWorker(0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:391 +0x7c
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:613 +0x185
goroutine 25 [semacquire]:
internal/poll.runtime_Semacquire(0xc0003885a0?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/sema.go:61 +0x25
internal/poll.(*fdMutex).rwlock(0xc00020aa80, 0x34?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_mutex.go:154 +0xd2
internal/poll.(*FD).readLock(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).ReadFromInet4(0xc00020aa80, {0xc000605000, 0x4e20, 0x4e20}, 0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_unix.go:235 +0x69
net.(*netFD).readFromInet4(0xc00020aa80, {0xc000605000?, 0x0?, 0xc00020dc40?}, 0x4?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/fd_posix.go:66 +0x29
net.(*UDPConn).readFrom(0xc00005e800?, {0xc000605000?, 0xc0005a29c0?, 0xa?}, 0xc0001f4630)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock_posix.go:52 +0x1b8
net.(*UDPConn).readFromUDP(0xc00013a3d0, {0xc000605000?, 0xc00020dde8?, 0x7fffe4965e95?}, 0xc000068db8?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock.go:149 +0x31
net.(*UDPConn).ReadFromUDP(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock.go:141
main.ReadUDP()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:696 +0x8d
main.incomingMessagesWorker(0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:391 +0x7c
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:613 +0x185
goroutine 26 [runnable]:
main.deserializeGMessage({0xc0000d5000, 0x4e20, 0xc00065fde8?})
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/gmessage.go:285 +0x167
main.ReadUDP()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:698 +0xa9
main.incomingMessagesWorker(0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:391 +0x7c
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:613 +0x185
goroutine 27 [runnable]:
main.deserializeGMessage({0xc000b9d000, 0x4e20, 0xc00065dde8?})
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/gmessage.go:285 +0x185
main.ReadUDP()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:698 +0xa9
main.incomingMessagesWorker(0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:391 +0x7c
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:613 +0x185
goroutine 28 [IO wait]:
internal/poll.runtime_pollWait(0x7ffff7e21090, 0x72)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc00020aa80?, 0xc000677000?, 0x0)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).ReadFromInet4(0xc00020aa80, {0xc000677000, 0x4e20, 0x4e20}, 0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/internal/poll/fd_unix.go:250 +0x1e5
net.(*netFD).readFromInet4(0xc00020aa80, {0xc000677000?, 0x0?, 0x7fffe4e28f80?}, 0x1?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/fd_posix.go:66 +0x29
net.(*UDPConn).readFrom(0xc00005ec00?, {0xc000677000?, 0x0?, 0xa?}, 0xc00011d200)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock_posix.go:52 +0x1b8
net.(*UDPConn).readFromUDP(0xc00013a3d0, {0xc000677000?, 0xc000211de8?, 0x7fffe4965e95?}, 0xc000139db8?)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock.go:149 +0x31
net.(*UDPConn).ReadFromUDP(...)
/home/defrain/legit3/ams/mediaserver/Golang/go/src/net/udpsock.go:141
main.ReadUDP()
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:696 +0x8d
main.incomingMessagesWorker(0x0?)
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:391 +0x7c
created by main.GoInitialize
/home/defrain/legit3/ams/mediaserver/Golang/src/libgo/libgo.go:613 +0x185
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffb05f4700 (LWP 30894)]
runtime.raise () at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/sys_linux_amd64.s:168
168 /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/sys_linux_amd64.s: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-324.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libcom_err-1.42.9-13.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libstdc++-4.8.5-36.el7_6.2.x86_64 openssl-libs-1.0.2k-16.el7_6.1.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0 runtime.raise () at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/sys_linux_amd64.s:168
#1 0x00007fffe44188e5 in runtime.dieFromSignal (sig=6) at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/signal_unix.go:852
#2 0x00007fffe4418b18 in runtime.crash () at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/signal_unix.go:944
#3 0x00007fffe44035b1 in runtime.fatalthrow.func1 () at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/panic.go:1051
#4 0x00007fffe4403530 in runtime.fatalthrow () at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/panic.go:1044
#5 0x00007fffe44032f1 in runtime.throw (s=...) at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/panic.go:992
#6 0x00007fffe4426d36 in runtime.gentraceback (pc0=140737022891928, sp0=824634156680, lr0=, gp=, skip=0, pcbuf=0x0, max=2147483647, callback=
{void (runtime.stkframe *, void *, bool *)} 0x7fffb05f3d18, v=0x0, flags=0, ~r0=) at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/traceback.go:254
#7 0x00007fffe44024ab in runtime.addOneOpenDeferFrame.func1 () at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/panic.go:600
#8 0x00007fffe4430b86 in runtime.systemstack () at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/asm_amd64.s:469
#9 0x0000000000000000 in ?? ()
An alternative implementation that does not use CGo, but, instead uses UDP to exchange the buffer between C++ <-> Go, using the ReadUDP and WriteUDP, runs forever w/o issue.
The observed panic occurs in
runtime/traceback.go:254
where it finds an address, which appears to always be in "C++" memory space, it does not expect. Here is the frame (6) from GDB from a panic:
(gdb) frame 6
#6 0x00007fffe4426d36 in runtime.gentraceback (pc0=140737022891928, sp0=824634156680, lr0=, gp=, skip=0, pcbuf=0x0, max=2147483647, callback=
{void (runtime.stkframe *, void *, bool *)} 0x7fffb05f3d18, v=0x0, flags=0, ~r0=) at /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/traceback.go:254
254 /home/defrain/legit3/ams/mediaserver/Golang/go/src/runtime/traceback.go: No such file or directory.
(gdb) i args
pc0 = 140737022891928
sp0 = 824634156680
lr0 =
gp =
skip = 0
pcbuf = 0x0
max = 2147483647
callback = {void (runtime.stkframe *, void *, bool *)} 0x7fffb05f3d18
v = 0x0
flags = 0
~r0 =
(gdb) i lo
doPrint = false
lrPtr = 824634158576
flag =
flr = {_func = 0x7fffe52764e0, datap = }
level = 1
ctxt = 0x0
nprint = 0
waspanic = false
printing = false
lastFuncID = 0 '\000'
n = 12
f = {_func = 0x7fffe52764e0, datap = }
cgoCtxt = <error reading variable cgoCtxt (access outside bounds of object referenced via synthetic pointer)>
frame = {fn = {_func = 0x7fffe52764e0, datap = 0x7fffe52a4200 <runtime.firstmoduledata>}, pc = 140737028465579, continpc = 140737028447557, lr = 140734468526768, sp = 824634158392, fp = 824634158584,
varp = 824634158376, argp = 824634158392, arglen = 48, argmap = 0x0}
cache = {entries = {{{targetpc = 140737028447557, off = 1312617, val = 152}, {targetpc = 140737023552339, off = 292530, val = 152}, {targetpc = 0, off = 0, val = 0}, {targetpc = 0, off = 0, val = 0}, {
targetpc = 0, off = 0, val = 0}, {targetpc = 140737024364229, off = 477966, val = 232}, {targetpc = 140737024450933, off = 498209, val = 176}, {targetpc = 0, off = 0, val = 0}}, {{targetpc = 0, off = 0,
val = 0}, {targetpc = 0, off = 0, val = 0}, {targetpc = 140737023549355, off = 292208, val = 584}, {targetpc = 140737023539321, off = 289663, val = 80}, {targetpc = 0, off = 0, val = 0}, {targetpc = 0,
off = 0, val = 0}, {targetpc = 0, off = 0, val = 0}, {targetpc = 0, off = 49, val = 0}}}}
Many different approaches have been tried to avoid the panic. This includes alternative implementations where the Go channels are replaced with blocking queues on the "C++" side. Control of the Go GC has been tried using GOGC=off then by programmatically scheduling a GC (with "stoptheworld" either on and off and with any "C++" threads stopped too). Zeroing malloc memory has no impact. Never "free"ing the malloc memory has no impact. It seems that the GC eventually crashes if it finds anything "C++" related in a stack it is traversing. The the code (meaning the entire application) has no issue when CGo is removed and UDP is used to pass message buffers strongly suggests that there is a problem specific to the operation of the Go GC w.r.t. CGo. Valgrind used against the application code w/o libgo.so included, finds no issues.
The text was updated successfully, but these errors were encountered: