Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: unexpected return pc for runtime.selectgo called from 0xXXX #25602

Closed
meilihao opened this issue May 28, 2018 · 7 comments
Closed

runtime: unexpected return pc for runtime.selectgo called from 0xXXX #25602

meilihao opened this issue May 28, 2018 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@meilihao
Copy link

What version of Go are you using (go version)?

go version go1.10.2 linux/amd64

Does this issue reproduce with the latest release?

YES

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GORACE=""
GOROOT="/opt/go"
GOTMPDIR=""
GOTOOLDIR="/opt/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build250776484=/tmp/go-build -gno-record-gcc-switches"

What did you do?

run my binary

What did you expect to see?

no panic

What did you see instead?

log1:

[debug](cloud_adapt.c 1833) key: results Array Length: 0

[debug](ai.c         1281) ---remote_pid_list---
[error](ai.c         1200) ps -ef|grep pmanage|awk '{print $2,$9}' > /tmp/process error
[error](ai.c         1286) get_local_process_id error
[debug](ai.c         1370) child process revoked. pid[ 22937], exit code[0]

2018-05-28T11:00:24.137+0800	DEBUG	pmanage/util.go:81	url(GET): http://127.0.0.1:666/ais/getuser?aim_sn=107b44b1334d&sjc=1527476424&jmc=b256e841e377e28d6ba78063:606c7d8f1dd342f857cf47b908c0073bb1ccb304faea7a06dd7e419549d9120c
2018-05-28T11:00:24.145+0800	INFO	pmanage/manager.go:283	/ais/getuser ret: 2
2018-05-28T11:00:24.145+0800	DEBUG	pmanage/util.go:81	url(POST): http://127.0.0.1:666/ais/set_current_ais
2018-05-28T11:00:24.149+0800	INFO	pmanage/manager.go:168	ai processes num is 0
2018-05-28T11:00:24.149+0800	DEBUG	pmanage/util.go:81	url(GET): http://127.0.0.1:666/ais/get_templates?template_path=/data/template&sjc=1527476424&jmc=b256e841e377e28d6ba78063:606c7d8f1dd342f857cf47b908c0073bb1ccb304faea7a06dd7e419549d9120c
runtime: unexpected return pc for runtime.selectgo called from 0xc420042040
stack: frame={sp:0xc420041d20, fp:0xc420041ef8} stack=[0xc420041800,0xc420042000)
000000c420041c20:  000000c420216060  0000000000000000 
000000c420041c30:  000000c42007a000  0000000000000000 
000000c420041c40:  00007f12a9e8dd90  0000000000000000 
000000c420041c50:  0000000000000000  0000000000000000 
000000c420041c60:  0000000000000000  0000000000000000 
000000c420041c70:  000000c420041ca0  000000000041c048 <runtime.newobject+56> 
000000c420041c80:  0000000000000060  00000000008af860 
000000c420041c90:  0000000000000001  000000c420216060 
000000c420041ca0:  000000c420041d10  0000000000437032 <runtime.acquireSudog+642> 
000000c420041cb0:  00000000008af860  000000c420216060 
000000c420041cc0:  0000000000000000  0000000000000000 
000000c420041cd0:  0000000000000000  0000000000000000 
000000c420041ce0:  0000000000000000  000000c420041d10 
000000c420041cf0:  000000c42007a000  0000000000436c8a <runtime.gopark+282> 
000000c420041d00:  00000000008de2d0  000000c420028a00 
000000c420041d10:  000000c420041ee8  0000000000445dd0 <runtime.selectgo+3664> 
000000c420041d20: <00000000008de318  0000000000000000 
000000c420041d30:  00000000008c5439  0000000000000006 
000000c420041d40:  0000000000000018  0000000000000001 
000000c420041d50:  0000000000000000  0000000000000002 
000000c420041d60:  0000000000000002  0000000000000002 
000000c420041d70:  0000000000000000  0000000000000001 
000000c420041d80:  0000000000000000  0000000000000000 
000000c420041d90:  0000000000000000  0000000000000001 
000000c420041da0:  0000000000000002  0000000000000002 
000000c420041db0:  0000000000000000  0000000000000000 
000000c420041dc0:  000000c420041f50  000000c420041fb4 
000000c420041dd0:  000000c420216048  000000c420041fb0 
000000c420041de0:  000000c42007d380  0000000000000000 
000000c420041df0:  000000c420214000  000000c420214000 
000000c420041e00:  000000c420214000  0000000000000000 
000000c420041e10:  0000000000000000  000000c42007d408 
000000c420041e20:  000000c42007d4b8  00007f12a9cdd594 
000000c420041e30:  000000c420041f50  00007f12a982ef00 
000000c420041e40:  0000000000000002  0000000000000036 
000000c420041e50:  000000007dd7aaaa  00007f12a9cddccb 
000000c420041e60:  000000c420041fb0  000000c420041ea0 
000000c420041e70:  00007f12a982ee78  00007f12a9830b08 
000000c420041e80:  000000c420041fb0  000000c420041fa0 
000000c420041e90:  000000000000002a  0000000000000002 
000000c420041ea0:  0000000000000000  00007f12a9ed3060 
000000c420041eb0:  00007f12a9ee6000  00007f12a9a5112a 
000000c420041ec0:  00007f12a9830e38  00007f12a9a4c0a0 
000000c420041ed0:  0000000500000000  000000010000012b 
000000c420041ee0:  000000c4201e0c50  000000c420042068 
000000c420041ef0: !000000c420042040 >0000000000000001 
000000c420041f00:  00007f12a9ed3060  00007f12a9ef3a58 
000000c420041f10:  00007f12a9ef3700  00007f12a9cddfaf 
000000c420041f20:  0000000000000000  00007f12a9ed3060 
000000c420041f30:  0000000000000005  0000000000000000 
000000c420041f40:  000000c400000001  00007f12a9ef3700 
000000c420041f50:  0000000000000000  000000c420214000 
000000c420041f60:  00000000006e7ad1 <database/sql.(*DB).connectionResetter+241>  0000000000000001 
000000c420041f70:  0000000000000000 2018-05-28T11:00:24.152+0800	DEBUG	pmanage/manager.go:184	origin:map[b607221c251d11e88ad2107b44b1334e.tar.gz:{} 1c903288105311e8bed2107b44b1334e.tar.gz:{} 0b8bfe9608b911e8bbbd107b44b1334e.tar.gz:{} 34c2f8581dbf11e8a791107b44b1334e.tar.gz:{} 0bc05bb21c5211e8bdc1107b44b1334e.tar.gz:{} f19b8f6008b811e8992f107b44b1334e.tar.gz:{} 3bff4b76056e11e8b711107b44b1334e.tar.gz:{} b138f68059b511e892e6107b44b1334e.tar.gz:{} 5c69687a1c2411e8b039107b44b1334e.tar.gz:{} 51c32c800b3511e88c24107b44b1334e.tar.gz:{} 698500701c5411e893df107b44b1334e.tar.gz:{} 3d223c7a1c5211e8b764107b44b1334e.tar.gz:{} 6b40258c0b3511e894db107b44b1334e.tar.gz:{} 3d87944c1cef11e88d98107b44b1334e.tar.gz:{} 913ddbec1ac311e8b7db107b44b1334e.tar.gz:{} c04ad1d41ac211e88b21107b44b1334e.tar.gz:{} 290c2604581311e89c97107b44b1334e.tar.gz:{} 4a84fa2846b511e88013107b44b1334e.tar.gz:{} 79b633941c4d11e89ca4107b44b1334e.tar.gz:{} 1e3c7a602b6a11e8aa4f107b44b1334e.tar.gz:{} 0ff5f07c1dc211e8abbe107b44b1334e.tar.gz:{} b41f9796185b11e882b7107b44b1334e.tar.gz:{} a93f26de08a211e8985f107b44b1334e.tar.gz:{} 6e1550ac30e711e89358107b44b1334e.tar.gz:{} 24e146ee081911e8999a107b44b1334e.tar.gz:{} b26f90560b1b11e8bb05107b44b1334e.tar.gz:{} 1543f0aa3e2511e8918f107b44b1334e.tar.gz:{} 2698b91a080d11e8b03d107b44b1334e.tar.gz:{}]
 00007f12a9ef3a58 
000000c420041f80:  000000c420041fb0  000000c420041fa0 
000000c420041f90:  000000007dd7aaaa  00007f12a9a5112a 
000000c420041fa0:  00000000ffffffff  0000000000000000 
000000c420041fb0:  3033393232830b08  00007f12a9ee6000 
000000c420041fc0:  0000000000460ff1 <runtime.goexit+1>  000000c4200420ec 
000000c420041fd0:  000000c420042f40  00000000000000ff 
000000c420041fe0:  00000000000000ff  000000000000000f 
000000c420041ff0:  000000c4200426e8 
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x8c8f3f, 0x11)
	/opt/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42007d380, 0x0, 0x0, 0x7fffffff, 0x7f12997f9b88, 0x0, 0x0, ...)
	/opt/go/src/runtime/traceback.go:257 +0x1bdb
runtime.scanstack(0xc42007d380, 0xc420029c70)
	/opt/go/src/runtime/mgcmark.go:786 +0x14b
runtime.scang(0xc42007d380, 0xc420029c70)
	/opt/go/src/runtime/proc.go:893 +0x1f6
runtime.markroot.func1()
	/opt/go/src/runtime/mgcmark.go:264 +0x6d
runtime.markroot(0xc420029c70, 0xc40000000b)
	/opt/go/src/runtime/mgcmark.go:245 +0x2f6
runtime.gcDrain(0xc420029c70, 0xd)
	/opt/go/src/runtime/mgcmark.go:934 +0x111
runtime.gcBgMarkWorker.func2()
	/opt/go/src/runtime/mgc.go:1865 +0x187
runtime.systemstack(0x0)
	/opt/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/opt/go/src/runtime/proc.go:1175

goroutine 27 [GC worker (idle)]:
runtime.systemstack_switch()
	/opt/go/src/runtime/asm_amd64.s:363 fp=0xc420043f48 sp=0xc420043f40 pc=0x45e4a0
runtime.gcBgMarkWorker(0xc420028a00)
	/opt/go/src/runtime/mgc.go:1829 +0x1ee fp=0xc420043fd8 sp=0xc420043f48 pc=0x424b0e
runtime.goexit()
	/opt/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420043fe0 sp=0xc420043fd8 pc=0x460ff1
created by runtime.gcBgMarkStartWorkers
	/opt/go/src/runtime/mgc.go:1723 +0x79

goroutine 1 [runnable]:
syscall.ByteSliceFromString(0xc4206a48c0, 0x36, 0xc420000100, 0x38, 0xc8, 0xc4206b8dd0, 0x0)
	/opt/go/src/syscall/syscall.go:56 +0x96
syscall.BytePtrFromString(0xc4206a48c0, 0x36, 0x36, 0xc420669998, 0x44dea6)
	/opt/go/src/syscall/syscall.go:72 +0x35
syscall.Lstat(0xc4206a48c0, 0x36, 0xc4206b8e08, 0xc4206a48c0, 0xc4206a48cf)
	/opt/go/src/syscall/zsyscall_linux_amd64.go:1308 +0x39
os.lstatNolog(0xc4206a48c0, 0x36, 0x3, 0x3, 0xc4206a48c0, 0x36)
	/opt/go/src/os/stat_unix.go:42 +0x5c
os.Lstat(0xc4206a48c0, 0x36, 0xe, 0x8c469e, 0x1, 0xc42069c870)
	/opt/go/src/os/stat.go:22 +0x4d
os.(*File).readdir(0xc42066e018, 0xffffffffffffffff, 0x0, 0x0, 0xc42066e018, 0x0, 0x0)
	/opt/go/src/os/dir_unix.go:27 +0x19e
os.(*File).Readdir(0xc42066e018, 0xffffffffffffffff, 0xc42066e018, 0x0, 0x0, 0x0, 0x0)
	/opt/go/src/os/dir.go:26 +0x3e
io/ioutil.ReadDir(0xc4201b6e10, 0xe, 0x2, 0x2, 0xc42069a3e8, 0xc42069c510, 0x27)
	/opt/go/src/io/ioutil/ioutil.go:101 +0x7c
main.(*Manager).Run(0xc4201d4aa0, 0x1)
	/root/go/src/pmanage/manager.go:185 +0x196
main.main()
	/root/go/src/pmanage/main.go:22 +0x1a4

goroutine 20 [select]:
database/sql.(*DB).connectionOpener(0xc4201d4d20, 0x912ea0, 0xc4201e0c40)
	/opt/go/src/database/sql/sql.go:935 +0x119
created by database/sql.OpenDB
	/opt/go/src/database/sql/sql.go:634 +0x178

goroutine 21 [select (scan)]:
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x119 pc=0x4562d5]

runtime stack:
runtime.startpanic_m()
	/opt/go/src/runtime/panic.go:690 +0x179
runtime.startpanic()
	/opt/go/src/runtime/panic.go:589 +0x1e
runtime.throw(0x8d3f53, 0x2a)
	/opt/go/src/runtime/panic.go:615 +0x74
runtime.sigpanic()
	/opt/go/src/runtime/signal_unix.go:372 +0x28e
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42007d380, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, ...)
	/opt/go/src/runtime/traceback.go:245 +0x1515
runtime.traceback1(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42007d380, 0x0)
	/opt/go/src/runtime/traceback.go:684 +0xf9
runtime.traceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42007d380)
	/opt/go/src/runtime/traceback.go:645 +0x52
runtime.tracebackothers(0xc4202ce600)
	/opt/go/src/runtime/traceback.go:834 +0x186
runtime.dopanic_m(0xc4202ce600, 0x434fc1, 0x7f12997f96f0)
	/opt/go/src/runtime/panic.go:736 +0x2c5
runtime.dopanic.func1()
	/opt/go/src/runtime/panic.go:598 +0x3c
runtime.dopanic(0x0)
	/opt/go/src/runtime/panic.go:597 +0x4a
runtime.throw(0x8c8f3f, 0x11)
	/opt/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42007d380, 0x0, 0x0, 0x7fffffff, 0x7f12997f9b88, 0x0, 0x0, ...)
	/opt/go/src/runtime/traceback.go:257 +0x1bdb
runtime.scanstack(0xc42007d380, 0xc420029c70)
	/opt/go/src/runtime/mgcmark.go:786 +0x14b
runtime.scang(0xc42007d380, 0xc420029c70)
	/opt/go/src/runtime/proc.go:893 +0x1f6
runtime.markroot.func1()
	/opt/go/src/runtime/mgcmark.go:264 +0x6d
runtime.markroot(0xc420029c70, 0xc40000000b)
	/opt/go/src/runtime/mgcmark.go:245 +0x2f6
runtime.gcDrain(0xc420029c70, 0xd)
	/opt/go/src/runtime/mgcmark.go:934 +0x111
runtime.gcBgMarkWorker.func2()
	/opt/go/src/runtime/mgc.go:1865 +0x187
runtime.systemstack(0x0)
	/opt/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/opt/go/src/runtime/proc.go:1175

/root/go/src/pmanage/manager.go:
code1

log2:

[debug](cloud_adapt.c 1833) key: results Array Length: 0

[debug](ai.c         1281) ---remote_pid_list---
[error](ai.c         1200) ps -ef|grep pmanage|awk '{print $2,$9}' > /tmp/process error
[debug](ai.c         1370) child process revoked. pid[ 22814], exit code[0]

[error](ai.c         1286) get_local_process_id error
2018-05-28T11:00:04.275+0800	DEBUG	pmanage/util.go:81	url(GET): http://127.0.0.1:666/ais/getuser?aim_sn=107b44b1334d&sjc=1527476404&jmc=1d69e2f4478d2fce9db02d7f:ca5b5c116253f5bf2241154e57146df660565e1dee7d1427e4d40393e406ec17
2018-05-28T11:00:04.281+0800	INFO	pmanage/manager.go:283	/ais/getuser ret: 2
2018-05-28T11:00:04.282+0800	DEBUG	pmanage/util.go:81	url(POST): http://127.0.0.1:666/ais/set_current_ais
2018-05-28T11:00:04.285+0800	INFO	pmanage/manager.go:168	ai processes num is 0
2018-05-28T11:00:04.285+0800	DEBUG	pmanage/util.go:81	url(GET): http://127.0.0.1:666/ais/get_templates?template_path=/data/template&sjc=1527476404&jmc=1d69e2f4478d2fce9db02d7f:ca5b5c116253f5bf2241154e57146df660565e1dee7d1427e4d40393e406ec17
2018-05-28T11:00:04.288+0800	DEBUG	pmanage/manager.go:184	origin:map[51c32c800b3511e88c24107b44b1334e.tar.gz:{} 1543f0aa3e2511e8918f107b44b1334e.tar.gz:{} 0b8bfe9608b911e8bbbd107b44b1334e.tar.gz:{} 79b633941c4d11e89ca4107b44b1334e.tar.gz:{} c04ad1d41ac211e88b21107b44b1334e.tar.gz:{} 6b40258c0b3511e894db107b44b1334e.tar.gz:{} f19b8f6008b811e8992f107b44b1334e.tar.gz:{} 913ddbec1ac311e8b7db107b44b1334e.tar.gz:{} a93f26de08a211e8985f107b44b1334e.tar.gz:{} 34c2f8581dbf11e8a791107b44b1334e.tar.gz:{} 1e3c7a602b6a11e8aa4f107b44b1334e.tar.gz:{} 698500701c5411e893df107b44b1334e.tar.gz:{} 2698b91a080d11e8b03d107b44b1334e.tar.gz:{} 4a84fa2846b511e88013107b44b1334e.tar.gz:{} b41f9796185b11e882b7107b44b1334e.tar.gz:{} 0bc05bb21c5211e8bdc1107b44b1334e.tar.gz:{} 5c69687a1c2411e8b039107b44b1334e.tar.gz:{} 0ff5f07c1dc211e8abbe107b44b1334e.tar.gz:{} 24e146ee081911e8999a107b44b1334e.tar.gz:{} 290c2604581311e89c97107b44b1334e.tar.gz:{} 3bff4b76056e11e8b711107b44b1334e.tar.gz:{} b138f68059b511e892e6107b44b1334e.tar.gz:{} 1c903288105311e8bed2107b44b1334e.tar.gz:{} b607221c251d11e88ad2107b44b1334e.tar.gz:{} 6e1550ac30e711e89358107b44b1334e.tar.gz:{} 3d223c7a1c5211e8b764107b44b1334e.tar.gz:{} 3d87944c1cef11e88d98107b44b1334e.tar.gz:{} b26f90560b1b11e8bb05107b44b1334e.tar.gz:{}]
runtime: unexpected return pc for runtime.selectgo called from 0xc420046840
stack: frame={sp:0xc420046520, fp:0xc4200466f8} stack=[0xc420046000,0xc420046800)
000000c420046420:  000000c420206060  0000000000000000 
000000c420046430:  000000c420048c00  0000000000000000 
000000c420046440:  00007f346f271d90  0000000000000000 
000000c420046450:  0000000000000000  0000000000000000 
000000c420046460:  0000000000000000  0000000000000000 
000000c420046470:  000000c4200464a0  000000000041c048 <runtime.newobject+56> 
000000c420046480:  0000000000000060  00000000008af860 
000000c420046490:  0000000000000001  000000c420206060 
000000c4200464a0:  000000c420046510  0000000000437032 <runtime.acquireSudog+642> 
000000c4200464b0:  00000000008af860  000000c420206060 
000000c4200464c0:  0000000000000000  0000000000000000 
000000c4200464d0:  0000000000000000  0000000000000000 
000000c4200464e0:  0000000000000000  000000c420046510 
000000c4200464f0:  000000c420048c00  0000000000436c8a <runtime.gopark+282> 
000000c420046500:  00000000008de2d0  000000c420028a00 
000000c420046510:  000000c4200466e8  0000000000445dd0 <runtime.selectgo+3664> 
000000c420046520: <00000000008de318  0000000000000000 
000000c420046530:  00000000008c5439  0000000000000006 
000000c420046540:  0000000000000018  0000000000000001 
000000c420046550:  0000000000000000  0000000000000002 
000000c420046560:  0000000000000002  0000000000000002 
000000c420046570:  0000000000000000  0000000000000001 
000000c420046580:  0000000000000000  0000000000000000 
000000c420046590:  0000000000000000  0000000000000001 
000000c4200465a0:  0000000000000002  0000000000000002 
000000c4200465b0:  0000000000000000  0000000000000000 
000000c4200465c0:  000000c420046750  000000c4200467b4 
000000c4200465d0:  000000c420206048  000000c4200467b0 
000000c4200465e0:  000000c42016c300  00000000000000002018-05-28T11:00:04.288+0800	DEBUG	pmanage/manager.go:192	to download:map[]
 
000000c4200465f0:  000000c420195b60  000000c4201f4000 
000000c420046600:  000000c420195b60  0000000000000000 
000000c420046610:  0000000000000000  000000c42016c388 
000000c420046620:  000000c42016c438  00007f346f0c1594 
000000c420046630:  000000c420046750  00007f346ec12f00 
000000c420046640:  0000000000000002  0000000000000036 
000000c420046650:  000000007dd7aaaa  00007f346f0c1ccb 
000000c420046660:  000000c4200467b0  000000c4200466a0 
000000c420046670:  00007f346ec12e78  00007f346ec14b08 
000000c420046680:  000000c4200467b0  000000c4200467a0 
000000c420046690:  000000000000002a  0000000000000000 
000000c4200466a0:  0000000000000000  00007f346f2b7060 
000000c4200466b0:  00007f346f2ca000  00007f346ee3512a 
000000c4200466c0:  00007f346ec14e38  00007f346ee300a0 
000000c4200466d0:  0000000500000000  000000010000012b 
000000c4200466e0:  000000c4201c2cd0  000000c420046868 2018-05-28T11:00:04.288+0800	DEBUG	pmanage/util.go:81	url(POST): http://127.0.0.1:666/ais/setuserover

000000c4200466f0: !000000c420046840 >0000000000000001 
000000c420046700:  00007f346f2b7060  00007f346f2d7a58 
000000c420046710:  00007f346f2d7700  00007f346f0c1faf 
000000c420046720:  0000000000000000  00007f346f2b7060 
000000c420046730:  0000000000000005  0000000000000000 
000000c420046740:  000000c400000001  00007f346f2d7700 
000000c420046750:  0000000000000000  000000c4201f4000 
000000c420046760:  00000000006e7ad1 <database/sql.(*DB).connectionResetter+241>  0000000000000001 
000000c420046770:  0000000000000000  00007f346f2d7a58 
000000c420046780:  000000c4200467b0  000000c4200467a0 
000000c420046790:  000000007dd7aaaa  00007f346ee3512a 
000000c4200467a0:  00000000ffffffff  0000000000000000 
000000c4200467b0:  3031383232c14b08  00007f346f2ca000 
000000c4200467c0:  0000000000460ff1 <runtime.goexit+1>  000000c4200468ec 
000000c4200467d0:  000000c420047740  00000000000000ff 
000000c4200467e0:  00000000000000ff  000000000000000f 
000000c4200467f0:  000000c420046ee8 
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x8c8f3f, 0x11)
	/opt/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42016c300, 0x0, 0x0, 0x7fffffff, 0x7f34567fbb88, 0x0, 0x0, ...)
	/opt/go/src/runtime/traceback.go:257 +0x1bdb
runtime.scanstack(0xc42016c300, 0xc420029c70)
	/opt/go/src/runtime/mgcmark.go:786 +0x14b
runtime.scang(0xc42016c300, 0xc420029c70)
	/opt/go/src/runtime/proc.go:893 +0x1f6
runtime.markroot.func1()
	/opt/go/src/runtime/mgcmark.go:264 +0x6d
runtime.markroot(0xc420029c70, 0x7f340000000b)
	/opt/go/src/runtime/mgcmark.go:245 +0x2f6
runtime.gcDrain(0xc420029c70, 0xd)
	/opt/go/src/runtime/mgcmark.go:934 +0x111
runtime.gcBgMarkWorker.func2()
	/opt/go/src/runtime/mgc.go:1865 +0x187
runtime.systemstack(0x0)
	/opt/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/opt/go/src/runtime/proc.go:1175

goroutine 36 [GC worker (idle)]:
runtime.systemstack_switch()
	/opt/go/src/runtime/asm_amd64.s:363 fp=0xc4202d5748 sp=0xc4202d5740 pc=0x45e4a0
runtime.gcBgMarkWorker(0xc420028a00)
	/opt/go/src/runtime/mgc.go:1829 +0x1ee fp=0xc4202d57d8 sp=0xc4202d5748 pc=0x424b0e
runtime.goexit()
	/opt/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4202d57e0 sp=0xc4202d57d8 pc=0x460ff1
created by runtime.gcBgMarkStartWorkers
	/opt/go/src/runtime/mgc.go:1723 +0x79

goroutine 1 [select]:
net/http.(*persistConn).roundTrip(0xc4200b5b00, 0xc4202a01e0, 0x0, 0x0, 0x0)
	/opt/go/src/net/http/transport.go:2033 +0x5a7
net/http.(*Transport).RoundTrip(0xcd95c0, 0xc420602000, 0xcd95c0, 0x0, 0x0)
	/opt/go/src/net/http/transport.go:422 +0x8f2
net/http.send(0xc420602000, 0x90f7c0, 0xcd95c0, 0x0, 0x0, 0x0, 0xc420606018, 0x41ba0d, 0xc42028d830, 0x1)
	/opt/go/src/net/http/client.go:252 +0x185
net/http.(*Client).send(0xd0cee0, 0xc420602000, 0x0, 0x0, 0x0, 0xc420606018, 0x0, 0x1, 0x8)
	/opt/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).Do(0xd0cee0, 0xc420602000, 0xc42028dad8, 0x8cf1cf, 0x1f)
	/opt/go/src/net/http/client.go:615 +0x28d
main.HttpDoWithHeader(0x8c4b90, 0x4, 0xc4205fb140, 0x24, 0x90f300, 0xc4202a00c0, 0x0, 0x0, 0x0, 0x0, ...)
	/root/go/src/pmanage/util.go:97 +0x309
main.HttpDo(0x8c4b90, 0x4, 0xc4205fb140, 0x24, 0x90f300, 0xc4202a00c0, 0x24, 0x411, 0xe00, 0x0, ...)
	/root/go/src/pmanage/util.go:76 +0x76
main.(*Manager).Run(0xc4201b6aa0, 0x1)
	/root/go/src/pmanage/manager.go:213 +0x750
main.main()
	/root/go/src/pmanage/main.go:22 +0x1a4

goroutine 5 [select]:
database/sql.(*DB).connectionOpener(0xc4201b6d20, 0x912ea0, 0xc4201c2cc0)
	/opt/go/src/database/sql/sql.go:935 +0x119
created by database/sql.OpenDB
	/opt/go/src/database/sql/sql.go:634 +0x178

goroutine 6 [select (scan)]:
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x119 pc=0x4562d5]

runtime stack:
runtime.startpanic_m()
	/opt/go/src/runtime/panic.go:690 +0x179
runtime.startpanic()
	/opt/go/src/runtime/panic.go:589 +0x1e
runtime.throw(0x8d3f53, 0x2a)
	/opt/go/src/runtime/panic.go:615 +0x74
runtime.sigpanic()
	/opt/go/src/runtime/signal_unix.go:372 +0x28e
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42016c300, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, ...)
	/opt/go/src/runtime/traceback.go:245 +0x1515
runtime.traceback1(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42016c300, 0x0)
	/opt/go/src/runtime/traceback.go:684 +0xf9
runtime.traceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42016c300)
	/opt/go/src/runtime/traceback.go:645 +0x52
runtime.tracebackothers(0xc42016d680)
	/opt/go/src/runtime/traceback.go:834 +0x186
runtime.dopanic_m(0xc42016d680, 0x434fc1, 0x7f34567fb6f0)
	/opt/go/src/runtime/panic.go:736 +0x2c5
runtime.dopanic.func1()
	/opt/go/src/runtime/panic.go:598 +0x3c
runtime.dopanic(0x0)
	/opt/go/src/runtime/panic.go:597 +0x4a
runtime.throw(0x8c8f3f, 0x11)
	/opt/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42016c300, 0x0, 0x0, 0x7fffffff, 0x7f34567fbb88, 0x0, 0x0, ...)
	/opt/go/src/runtime/traceback.go:257 +0x1bdb
runtime.scanstack(0xc42016c300, 0xc420029c70)
	/opt/go/src/runtime/mgcmark.go:786 +0x14b
runtime.scang(0xc42016c300, 0xc420029c70)
	/opt/go/src/runtime/proc.go:893 +0x1f6
runtime.markroot.func1()
	/opt/go/src/runtime/mgcmark.go:264 +0x6d
runtime.markroot(0xc420029c70, 0x7f340000000b)
	/opt/go/src/runtime/mgcmark.go:245 +0x2f6
runtime.gcDrain(0xc420029c70, 0xd)
	/opt/go/src/runtime/mgcmark.go:934 +0x111
runtime.gcBgMarkWorker.func2()
	/opt/go/src/runtime/mgc.go:1865 +0x187
runtime.systemstack(0x0)
	/opt/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/opt/go/src/runtime/proc.go:1175

/root/go/src/pmanage/util.go:
code2

@davecheney
Copy link
Contributor

This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

@meilihao
Copy link
Author

I get no data race report by using go build -race.

log1

2018-05-28T12:08:40.464+0800	INFO	pmanage/manager.go:283	/ais/getuser ret: 2
2018-05-28T12:08:40.464+0800	DEBUG	pmanage/util.go:81	url(POST): http://127.0.0.1:666/ais/set_current_ais
2018-05-28T12:08:40.468+0800	INFO	pmanage/manager.go:168	ai processes num is 0
2018-05-28T12:08:40.468+0800	DEBUG	pmanage/util.go:81	url(GET): http://127.0.0.1:666/ais/get_templates?template_path=/data/template&sjc=1527480520&jmc=a3b916c7f2e94a240db2c4f5:413f6d028644b314d8db01a743a6960c15c0a4eef4e030a788fcf42e81351bc1
runtime: unexpected return pc for runtime.sigpanic called from 0x1
stack: frame={sp:0xc420058ef8, fp:0xc420058f48} stack=[0xc420058000,0xc420059000)
000000c420058df8:  0000000000ef60a0  000000c420058eb0 
000000c420058e08:  000000c420084120  000000c420214000 
000000c420058e18:  000000c4204b6180  000000c4204b4000 
000000c420058e28:  000000c420058ec8  0000000000447be5 <runtime.gopanic+933> 
000000c420058e38:  000000c420058ea0  000000c420040ea0 
000000c420058e48:  0000000000000036  000000007dd7aaaa 
000000c420058e58:  00007f066ac05ccb  000000000042c7fa <runtime.mallocgc+682> 
000000c420058e68:  000000c420040ea0  2018-05-28T12:08:40.472+0800	DEBUG	pmanage/manager.go:184	origin:map[3d87944c1cef11e88d98107b44b1334e.tar.gz:{} b26f90560b1b11e8bb05107b44b1334e.tar.gz:{} 51c32c800b3511e88c24107b44b1334e.tar.gz:{} f19b8f6008b811e8992f107b44b1334e.tar.gz:{} 6e1550ac30e711e89358107b44b1334e.tar.gz:{} 79b633941c4d11e89ca4107b44b1334e.tar.gz:{} 1e3c7a602b6a11e8aa4f107b44b1334e.tar.gz:{} 290c2604581311e89c97107b44b1334e.tar.gz:{} 6b40258c0b3511e894db107b44b1334e.tar.gz:{} c04ad1d41ac211e88b21107b44b1334e.tar.gz:{} 4a84fa2846b511e88013107b44b1334e.tar.gz:{} b41f9796185b11e882b7107b44b1334e.tar.gz:{} 0ff5f07c1dc211e8abbe107b44b1334e.tar.gz:{} 2698b91a080d11e8b03d107b44b1334e.tar.gz:{} 913ddbec1ac311e8b7db107b44b1334e.tar.gz:{} 1543f0aa3e2511e8918f107b44b1334e.tar.gz:{} 698500701c5411e893df107b44b1334e.tar.gz:{} 24e146ee081911e8999a107b44b1334e.tar.gz:{} 5c69687a1c2411e8b039107b44b1334e.tar.gz:{} 1c903288105311e8bed2107b44b1334e.tar.gz:{} 0b8bfe9608b911e8bbbd107b44b1334e.tar.gz:{} b607221c251d11e88ad2107b44b1334e.tar.gz:{} b138f68059b511e892e6107b44b1334e.tar.gz:{} 3d223c7a1c5211e8b764107b44b1334e.tar.gz:{} a93f26de08a211e8985f107b44b1334e.tar.gz:{} 34c2f8581dbf11e8a791107b44b1334e.tar.gz:{} 0bc05bb21c5211e8bdc1107b44b1334e.tar.gz:{} 3bff4b76056e11e8b711107b44b1334e.tar.gz:{}]
000000c420076d80 
000000c420058e78:  00007f066a758b08  000000c420040fb0 
000000c420058e88:  000000c420040fa0  010005c40000002a 
000000c420058e98:  000000c420076da0  0000000000000000 
000000c420058ea8:  0000000000a36360  0000000000ef60a0 
000000c420058eb8:  0000000000000000  0000000000000000 
000000c420058ec8:  000000c420058ee8  0000000000446ade <runtime.panicmem+94> 
000000c420058ed8:  0000000000a36360  0000000000ef60a0 
000000c420058ee8:  000000c420058f38  000000000045edda <runtime.sigpanic+378> 
000000c420058ef8: <000000c420076d80  00007f066adfb001 
000000c420058f08:  00007f066ae1ba58  00007f066ae1b700 
000000c420058f18:  00007f066ac05faf  000000c420076d80 
000000c420058f28:  00007f066adfb060  000000c400000005 
000000c420058f38:  000000c400000005 !0000000000000001 
000000c420058f48: >00007f066ae1b700  0000000000aa2fe8 
000000c420058f58:  0000000000000010  0000000000863f14 <database/sql.(*DB).connectionOpener+260> 
000000c420058f68:  0000000000000000  0000000000000000 
000000c420058f78:  00007f066ae1ba58  000000c420040fb0 
000000c420058f88:  000000c420040fa0  000000007dd7aaaa 
000000c420058f98:  00007f066a97912a  00000000ffffffff 
000000c420058fa8:  000000c4201b77d8  2018-05-28T12:08:40.472+0800	DEBUG	pmanage/manager.go:192	to download:map[]
303330316a758b08 
000000c420058fb8:  00007f066ae0e000  000000c420026500 
000000c420058fc8:  000000c4200410ec  000000c420067220 
000000c420058fd8:  0000000000000001  0000000000000000 
000000c420058fe8:  0000000000000000  000000c4200416e8 
000000c420058ff8:  00007f066a1e2ff7 
fatal error: unknown caller pc
2018-05-28T12:08:40.473+0800	DEBUG	pmanage/util.go:81	url(POST): http://127.0.0.1:666/ais/setuserover

runtime stack:
runtime.throw(0xaa3a6e, 0x11)
	/opt/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc420076d80, 0x0, 0x0, 0x7fffffff, 0xab8c48, 0x7f063fffea88, 0x0, ...)
	/opt/go/src/runtime/traceback.go:257 +0x1a62
runtime.copystack(0xc420076d80, 0x1000, 0x7f063fffeb01)
	/opt/go/src/runtime/stack.go:891 +0x270
runtime.newstack()
	/opt/go/src/runtime/stack.go:1063 +0x342
runtime.morestack()
	/opt/go/src/runtime/asm_amd64.s:480 +0x89

goroutine 35 [copystack]:
runtime.gcAssistAlloc(0xc420076d80)
	/opt/go/src/runtime/mgcmark.go:408 +0x24e fp=0xc420058b50 sp=0xc420058b48 pc=0x4392ce
runtime.mallocgc(0x40, 0x0, 0x0, 0x7f066ada56c8)
	/opt/go/src/runtime/malloc.go:614 +0x7bd fp=0xc420058bf8 sp=0xc420058b50 pc=0x42cd0d
runtime.rawstring(0x40, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/go/src/runtime/string.go:245 +0x53 fp=0xc420058c30 sp=0xc420058bf8 pc=0x4646f3
runtime.rawstringtmp(0x0, 0x40, 0xa8a080, 0xc420026500, 0xc420214058, 0x0, 0x0)
	/opt/go/src/runtime/string.go:115 +0x6f fp=0xc420058c70 sp=0xc420058c30 pc=0x463fdf
runtime.concatstrings(0x0, 0xc420058d50, 0x2, 0x2, 0xa36360, 0xafec40)
	/opt/go/src/runtime/string.go:46 +0xf2 fp=0xc420058d08 sp=0xc420058c70 pc=0x463a02
runtime.concatstring2(0x0, 0xaa2d84, 0xf, 0xab0920, 0x31, 0xc4204b4000, 0xc420040db0)
	/opt/go/src/runtime/string.go:55 +0x47 fp=0xc420058d48 sp=0xc420058d08 pc=0x463be7
runtime.errorString.Error(0xab0920, 0x31, 0x1, 0xafec40)
	/opt/go/src/runtime/error.go:52 +0x53 fp=0xc420058d90 sp=0xc420058d48 pc=0x41ecc3
runtime.(*errorString).Error(0xef60a0, 0xa36360, 0xef60a0)
	<autogenerated>:1 +0x45 fp=0xc420058dc0 sp=0xc420058d90 pc=0x47ac25
runtime.preprintpanics(0xc420058ea0)
	/opt/go/src/runtime/panic.go:401 +0xb1 fp=0xc420058e38 sp=0xc420058dc0 pc=0x447601
panic(0xa36360, 0xef60a0)
	/opt/go/src/runtime/panic.go:543 +0x3a5 fp=0xc420058ed8 sp=0xc420058e38 pc=0x447be5
runtime.panicmem()
	/opt/go/src/runtime/panic.go:63 +0x5e fp=0xc420058ef8 sp=0xc420058ed8 pc=0x446ade
runtime: unexpected return pc for runtime.sigpanic called from 0x1
stack: frame={sp:0xc420058ef8, fp:0xc420058f48} stack=[0xc420058000,0xc420059000)
000000c420058df8:  0000000000ef60a0  000000c420058eb0 
000000c420058e08:  000000c420084120  000000c420214000 
000000c420058e18:  000000c4204b6180  000000c4204b4000 
000000c420058e28:  000000c420058ec8  0000000000447be5 <runtime.gopanic+933> 
000000c420058e38:  000000c420058ea0  000000c420040ea0 
000000c420058e48:  0000000000000036  000000007dd7aaaa 
000000c420058e58:  00007f066ac05ccb  000000000042c7fa <runtime.mallocgc+682> 
000000c420058e68:  000000c420040ea0  000000c420076d80 
000000c420058e78:  00007f066a758b08  000000c420040fb0 
000000c420058e88:  000000c420040fa0  010005c40000002a 
000000c420058e98:  000000c420076da0  0000000000000000 
000000c420058ea8:  0000000000a36360  0000000000ef60a0 
000000c420058eb8:  0000000000000000  0000000000000000 
000000c420058ec8:  000000c420058ee8  0000000000446ade <runtime.panicmem+94> 
000000c420058ed8:  0000000000a36360  0000000000ef60a0 
000000c420058ee8:  000000c420058f38  000000000045edda <runtime.sigpanic+378> 
000000c420058ef8: <000000c420076d80  00007f066adfb001 
000000c420058f08:  00007f066ae1ba58  00007f066ae1b700 
000000c420058f18:  00007f066ac05faf  000000c420076d80 
000000c420058f28:  00007f066adfb060  000000c400000005 
000000c420058f38:  000000c400000005 !0000000000000001 
000000c420058f48: >00007f066ae1b700  0000000000aa2fe8 
000000c420058f58:  0000000000000010  0000000000863f14 <database/sql.(*DB).connectionOpener+260> 
000000c420058f68:  0000000000000000  0000000000000000 
000000c420058f78:  00007f066ae1ba58  000000c420040fb0 
000000c420058f88:  000000c420040fa0  000000007dd7aaaa 
000000c420058f98:  00007f066a97912a  00000000ffffffff 
000000c420058fa8:  000000c4201b77d8  303330316a758b08 
000000c420058fb8:  00007f066ae0e000  000000c420026500 
000000c420058fc8:  000000c4200410ec  000000c420067220 
000000c420058fd8:  0000000000000001  0000000000000000 
000000c420058fe8:  0000000000000000  000000c4200416e8 
000000c420058ff8:  00007f066a1e2ff7 
runtime.sigpanic()
	/opt/go/src/runtime/signal_unix.go:388 +0x17a fp=0xc420058f48 sp=0xc420058ef8 pc=0x45edda
created by runtime.gcBgMarkStartWorkers
	/opt/go/src/runtime/mgc.go:1723 +0x79

goroutine 1 [select]:
net/http.(*persistConn).roundTrip(0xc42021c120, 0xc4201ce1e0, 0x0, 0x0, 0x0)
	/opt/go/src/net/http/transport.go:2033 +0x7ca
net/http.(*Transport).RoundTrip(0xefc660, 0xc420584000, 0xefc660, 0x0, 0x0)
	/opt/go/src/net/http/transport.go:422 +0xcf4
net/http.send(0xc420584000, 0xafdd80, 0xefc660, 0x0, 0x0, 0x0, 0x0, 0xc420067698, 0x45ff40, 0x8)
	/opt/go/src/net/http/client.go:252 +0x353
net/http.(*Client).send(0xf32020, 0xc420584000, 0x0, 0x0, 0x0, 0xc420588018, 0x0, 0x1, 0xc420067790)
	/opt/go/src/net/http/client.go:176 +0x1b1
net/http.(*Client).Do(0xf32020, 0xc420584000, 0xc420067a40, 0xaa9cfe, 0x1f)
	/opt/go/src/net/http/client.go:615 +0x4de
main.HttpDoWithHeader(0xa9f6b0, 0x4, 0xc42030ec00, 0x24, 0xafd8c0, 0xc42018f410, 0x0, 0x0, 0x0, 0x0, ...)
	/root/go/src/pmanage/util.go:97 +0x439
main.HttpDo(0xa9f6b0, 0x4, 0xc42030ec00, 0x24, 0xafd8c0, 0xc42018f410, 0x24, 0x411, 0xe00, 0x0, ...)
	/root/go/src/pmanage/util.go:76 +0x9b
main.(*Manager).Run(0xc4201d4aa0, 0x1)
	/root/go/src/pmanage/manager.go:213 +0xa0b
main.main()
	/root/go/src/pmanage/main.go:22 +0x2b9

goroutine 19 [select]:
database/sql.(*DB).connectionOpener(0xc4201d4d20, 0xb01540, 0xc4201e0c80)
	/opt/go/src/database/sql/sql.go:935 +0x13f
created by database/sql.OpenDB
	/opt/go/src/database/sql/sql.go:634 +0x2bd

goroutine 20 [select]:
database/sql.(*DB).connectionResetter(0xc4201d4d20, 0xb01540, 0xc4201e0c80)
	/opt/go/src/database/sql/sql.go:948 +0x155
created by database/sql.OpenDB
	/opt/go/src/database/sql/sql.go:635 +0x2f3

goroutine 24 [IO wait]:
internal/poll.runtime_pollWait(0x7f066ac95f00, 0x72, 0xf04a78)
	/opt/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc4201e6798, 0x72, 0xafec00, 0xf04a78, 0xffffffffffffffff)
	/opt/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc4201e6798, 0xc420228000, 0x1000, 0x1000)
	/opt/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc4201e6780, 0xc420228000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc4201e6780, 0xc420228000, 0x1000, 0x1000, 0x0, 0x1, 0x1)
	/opt/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc4200a2cb8, 0xc420228000, 0x1000, 0x1000, 0x42d8ec, 0xc420053910, 0x42e042)
	/opt/go/src/net/net.go:176 +0x85
net/http.(*persistConn).Read(0xc42021c120, 0xc420228000, 0x1000, 0x1000, 0x0, 0xe6e, 0x1)
	/opt/go/src/net/http/transport.go:1453 +0x21b
bufio.(*Reader).fill(0xc420226000)
	/opt/go/src/bufio/bufio.go:100 +0x1a0
bufio.(*Reader).Peek(0xc420226000, 0x1, 0x0, 0x0, 0x1, 0xc4200845a0, 0x0)
	/opt/go/src/bufio/bufio.go:132 +0x51
net/http.(*persistConn).readLoop(0xc42021c120)
	/opt/go/src/net/http/transport.go:1601 +0x29b
created by net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1237 +0xb26

goroutine 25 [runnable]:
net/http.(*persistConn).writeLoop(0xc42021c120)
	/opt/go/src/net/http/transport.go:1822 +0x1a7
created by net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1238 +0xb4b

log2

[debug](cloud_adapt.c 1833) key: results Array Length: 0

[debug](ai.c         1281) ---remote_pid_list---
[debug](ai.c         1370) child process revoked. pid[  2527], exit code[0]

[error](ai.c         1200) ps -ef|grep pmanage|awk '{print $2,$9}' > /tmp/process error
[error](ai.c         1286) get_local_process_id error
2018-05-28T12:13:10.800+0800	DEBUG	pmanage/util.go:81	url(GET): http://127.0.0.1:666/ais/getuser?aim_sn=107b44b1334d&sjc=1527480790&jmc=4e6bb1b0ff4bfb7e26e26223:044f9c60e7c5faf2595a6f5de400c817aa895a8162afddcd8c97e00d5c01d121
2018-05-28T12:13:10.806+0800	INFO	pmanage/manager.go:283	/ais/getuser ret: 2
2018-05-28T12:13:10.806+0800	DEBUG	pmanage/util.go:81	url(POST): http://127.0.0.1:666/ais/set_current_ais
2018-05-28T12:13:10.810+0800	INFO	pmanage/manager.go:168	ai processes num is 0
2018-05-28T12:13:10.810+0800	DEBUG	pmanage/util.go:81	url(GET): http://127.0.0.1:666/ais/get_templates?template_path=/data/template&sjc=1527480790&jmc=4e6bb1b0ff4bfb7e26e26223:044f9c60e7c5faf2595a6f5de400c817aa895a8162afddcd8c97e00d5c01d121
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc420047710, fp:0xc420047740} stack=[0xc420047000,0xc420047800)
000000c420047610:  0000000000000002  0000000000000002 
000000c420047620:  0000000000457034 <runtime.racemalloc+68>  00007f34977fb594 
000000c420047630:  0000000000000000  00007f349734cf00 
000000c420047640:  0000000000000002  0000000000000036 
000000c420047650:  000000007dd7aaaa  00007f34977fbccb 
000000c420047660:  000000000042c7fa <runtime.mallocgc+682>  000000c4200476a0 
000000c420047670:  00007f349734ce78  00007f349734eb08 
000000c420047680:  000000c4200477b0  000000c4200477a0 
000000c420047690:  010005c40000002a  0000000000000000 
000000c4200476a0:  0000000000000000  00007f34979f1060 
000000c4200476b0:  00007f3497a04000  00007f349756f12a 
000000c4200476c0:  00007f349734ee38  00007f349756a0a0 
000000c4200476d0:  0000000500000000  000000010000012b 
000000c4200476e0:  0000000000000000  000000c420047868 
000000c4200476f0:  000000c420047840  0000000000000001 
000000c420047700:  00007f34979f1060  00007f3497a11a58 
000000c420047710: <00007f3497a11700  00007f34977fbfaf 
000000c420047720:  0000000000000000  00007f34979f1060 
000000c420047730:  000000c400000005 !0000000000000000 
000000c420047740: >0000000000000001  00007f3497a11700 
000000c420047750:  0000000000aa2fe8  0000000000000010 
000000c420047760:  0000000000000014  0000000000000000 
000000c420047770:  0000000000000000  00007f3497a11a58 
000000c420047780:  000000c4200477b0  000000c4200477a0 
000000c420047790:  000000007dd7aaaa  00007f349756f12a 
000000c4200477a0:  00000000ffffffff  000000c42033e058 
000000c4200477b0:  00007f349734eb08  00007f3497a04000 
000000c4200477c0:  000000c420024000  000000c4200478ec 
000000c4200477d0:  000000c420067220  0000000000000001 
000000c4200477e0:  0000000000000000  0000000000000000 
000000c4200477f0:  000000c420047ee8  00007f3496dd8ff7 
fatal error: unknown caller pc

runtime stack:
runtime.throw(0xaa3a6e, 0x11)
	/opt/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42021d080, 0x0, 0x0, 0x7fffffff, 0x7f348e51bb88, 0x0, 0x0, ...)
	/opt/go/src/runtime/traceback.go:257 +0x1a62
runtime.scanstack(0xc42021d080, 0xc42002c170)
	/opt/go/src/runtime/mgcmark.go:786 +0x174
runtime.scang(0xc42021d080, 0xc42002c170)
	/opt/go/src/runtime/proc.go:893 +0x1fc
runtime.markroot.func1()
	/opt/go/src/runtime/mgcmark.go:264 +0x6d
runtime.markroot(0xc42002c170, 0x7f3400000012)
	/opt/go/src/runtime/mgcmark.go:245 +0x1b6
runtime.gcDrain(0xc42002c170, 0xd)
	/opt/go/src/runtime/mgcmark.go:934 +0x111
runtime.gcBgMarkWorker.func2()
	/opt/go/src/runtime/mgc.go:1865 +0x135
runtime.systemstack(0x7f3400000000)
	/opt/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/opt/go/src/runtime/proc.go:1175

goroutine 19 [GC worker (idle)]:
runtime.systemstack_switch()
	/opt/go/src/runtime/asm_amd64.s:363 fp=0xc420482740 sp=0xc420482738 pc=0x474da0
runtime.gcBgMarkWorker(0xc42002af00)
	/opt/go/src/runtime/mgc.go:1829 +0x21e fp=0xc4204827d8 sp=0xc420482740 pc=0x436cce
runtime.goexit()
	/opt/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4204827e0 sp=0xc4204827d8 pc=0x4778f1
created by runtime.gcBgMarkStartWorkers
	/opt/go/src/runtime/mgc.go:1723 +0x79

goroutine 1 [runnable]:
strings.TrimRightFunc(0xc4205483a4, 0x21, 0xc4201ba960, 0xc4205483a4, 0x21)
	/opt/go/src/strings/strings.go:689 +0x10e
strings.TrimRight(0xc4205483a4, 0x21, 0xa9f1b6, 0x1, 0xc4205483a4, 0x21)
	/opt/go/src/strings/strings.go:808 +0x91
main.DecodePySet2Data(0xc420546000, 0x411, 0xe00, 0xb0, 0x0, 0x0)
	/root/go/src/pmanage/util.go:43 +0x2b0
main.(*Manager).Run(0xc4201c4aa0, 0x1)
	/root/go/src/pmanage/manager.go:178 +0x1860
main.main()
	/root/go/src/pmanage/main.go:22 +0x2b9

goroutine 5 [select]:
database/sql.(*DB).connectionOpener(0xc4201c4d20, 0xb01540, 0xc4201d0d00)
	/opt/go/src/database/sql/sql.go:935 +0x13f
created by database/sql.OpenDB
	/opt/go/src/database/sql/sql.go:634 +0x2bd

goroutine 6 [select]:
database/sql.(*DB).connectionResetter(0xc4201c4d20, 0xb01540, 0xc4201d0d00)
	/opt/go/src/database/sql/sql.go:948 +0x155
created by database/sql.OpenDB
	/opt/go/src/database/sql/sql.go:635 +0x2f3

goroutine 10 [IO wait]:
internal/poll.runtime_pollWait(0x7f349788bf00, 0x72, 0xf04a78)
	/opt/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc4201d6798, 0x72, 0xafec00, 0xf04a78, 0xffffffffffffffff)
	/opt/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc4201d6798, 0xc420218000, 0x1000, 0x1000)
	/opt/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc4201d6780, 0xc420218000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc4201d6780, 0xc420218000, 0x1000, 0x1000, 0x0, 0x1, 0x1)
	/opt/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc42000ecc0, 0xc420218000, 0x1000, 0x1000, 0x42d8ec, 0xc42020b910, 0x42e042)
	/opt/go/src/net/net.go:176 +0x85
net/http.(*persistConn).Read(0xc4200c3e60, 0xc420218000, 0x1000, 0x1000, 0x0, 0xd8b, 0x1)
	/opt/go/src/net/http/transport.go:1453 +0x21b
bufio.(*Reader).fill(0xc420216060)
	/opt/go/src/bufio/bufio.go:100 +0x1a0
bufio.(*Reader).Peek(0xc420216060, 0x1, 0x0, 0x0, 0x1, 0xc42022a720, 0x0)
	/opt/go/src/bufio/bufio.go:132 +0x51
net/http.(*persistConn).readLoop(0xc4200c3e60)
	/opt/go/src/net/http/transport.go:1601 +0x29b
created by net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1237 +0xb26

goroutine 11 [select]:
net/http.(*persistConn).writeLoop(0xc4200c3e60)
	/opt/go/src/net/http/transport.go:1822 +0x1a7
created by net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1238 +0xb4b

/root/go/src/pmanage/util.go:
code3

log3

[debug](ai.c         1274) domain: http://127.0.0.1:666  domain_ptr: 127.0.0.1
[debug](cloud_adapt.c 1786) GET: https://127.0.0.1:888/inapi/1.0/allaiport?aim_sn=107b44b1334d
[debug](cloud_adapt.c 512) 38 bytes retrieved
[debug](cloud_adapt.c 1798) msg :{"res":"ok","results":[],"state":200}


[debug](cloud_adapt.c 1833) key: results Array Length: 0

[debug](ai.c         1281) ---remote_pid_list---
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7fc3a318346c]

runtime stack:
runtime.throw(0xaaeaa3, 0x2a)
	/opt/go/src/runtime/panic.go:616 +0x81
runtime.sigpanic()
	/opt/go/src/runtime/signal_unix.go:372 +0x28e

goroutine 1 [syscall]:
runtime.cgocall(0x90b3f0, 0xc420561970, 0xc)
	/opt/go/src/runtime/cgocall.go:128 +0x74 fp=0xc420561940 sp=0xc420561908 pc=0x41a514
main._Cfunc_sync_process(0x7fc360001570, 0x7fc360000a60, 0x7fc360000a80, 0x0)
	_cgo_gotypes.go:97 +0x6b fp=0xc420561970 sp=0xc420561940 pc=0x901a3b
main.(*Manager).ProcessTasks(0xc4201c4aa0, 0x1, 0x24, 0x23, 0xafd8c0)
	/root/go/src/pmanage/manager.go:266 +0x5d0 fp=0xc420561b48 sp=0xc420561970 pc=0x905fa0
main.(*Manager).Run(0xc4201c4aa0, 0x1)
	/root/go/src/pmanage/manager.go:234 +0x1197 fp=0xc420561ef0 sp=0xc420561b48 pc=0x904fc7
main.main()
	/root/go/src/pmanage/main.go:22 +0x2b9 fp=0xc420561f88 sp=0xc420561ef0 pc=0x8ff849
runtime.main()
	/opt/go/src/runtime/proc.go:198 +0x1d0 fp=0xc420561fe0 sp=0xc420561f88 pc=0x449960
runtime.goexit()
	/opt/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420561fe8 sp=0xc420561fe0 pc=0x4778f1

goroutine 5 [select]:
database/sql.(*DB).connectionOpener(0xc4201c4d20, 0xb01540, 0xc4201d0d00)
	/opt/go/src/database/sql/sql.go:935 +0x13f
created by database/sql.OpenDB
	/opt/go/src/database/sql/sql.go:634 +0x2bd

goroutine 6 [select]:
database/sql.(*DB).connectionResetter(0xc4201c4d20, 0xb01540, 0xc4201d0d00)
	/opt/go/src/database/sql/sql.go:948 +0x155
created by database/sql.OpenDB
	/opt/go/src/database/sql/sql.go:635 +0x2f3

goroutine 107 [select]:
net/http.(*persistConn).writeLoop(0xc42044cfc0)
	/opt/go/src/net/http/transport.go:1822 +0x1a7
created by net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1238 +0xb4b

goroutine 106 [IO wait]:
internal/poll.runtime_pollWait(0x7fc3a3c1af00, 0x72, 0xf04a78)
	/opt/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc420310c18, 0x72, 0xafec00, 0xf04a78, 0xffffffffffffffff)
	/opt/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc420310c18, 0xc420539000, 0x1000, 0x1000)
	/opt/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc420310c00, 0xc420539000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc420310c00, 0xc420539000, 0x1000, 0x1000, 0x0, 0x1, 0x1)
	/opt/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc4202f8498, 0xc420539000, 0x1000, 0x1000, 0x42d8ec, 0xc420054910, 0x42e042)
	/opt/go/src/net/net.go:176 +0x85
net/http.(*persistConn).Read(0xc42044cfc0, 0xc420539000, 0x1000, 0x1000, 0x0, 0xf5e, 0x1)
	/opt/go/src/net/http/transport.go:1453 +0x21b
bufio.(*Reader).fill(0xc4201a3e60)
	/opt/go/src/bufio/bufio.go:100 +0x1a0
bufio.(*Reader).Peek(0xc4201a3e60, 0x1, 0x0, 0x0, 0x0, 0xc4205f68a0, 0x0)
	/opt/go/src/bufio/bufio.go:132 +0x51
net/http.(*persistConn).readLoop(0xc42044cfc0)
	/opt/go/src/net/http/transport.go:1601 +0x29b
created by net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1237 +0xb26

/root/go/src/pmanage/manager.go:
code5

@ianlancetaylor ianlancetaylor added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels May 28, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone May 28, 2018
@ianlancetaylor
Copy link
Contributor

Does this always fail in the same way?

Is there any way that we can get the code to recreate the problem ourselves?

@josharian
Copy link
Contributor

I see some CGo in the code snippet. If it fails reliably, please try adding some runtime.KeepAlive calls after the C calls to ensure that this is not a lifetime/liveness problem.

@meilihao
Copy link
Author

@ianlancetaylor The error message is same, but it's location is random. And these codes are for the company, i cannot provide them.

@meilihao
Copy link
Author

@ianlancetaylor I can provide remote desktop to review codes. Is it ok?

@bradfitz bradfitz modified the milestones: Go1.11, Unplanned Jul 16, 2018
@meilihao
Copy link
Author

pmanage is deprecated, so close the issue.

@golang golang locked and limited conversation to collaborators Sep 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

6 participants