Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: malloc deadlock #15436

Closed
hryamzik opened this issue Apr 25, 2016 · 3 comments
Closed

runtime: malloc deadlock #15436

hryamzik opened this issue Apr 25, 2016 · 3 comments
Labels
FrozenDueToAge
Milestone
Unplanned

Comments

@hryamzik
Copy link

hryamzik commented Apr 25, 2016 •
edited

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?

1.6.2

  1. What operating system and processor architecture are you using (go env)?

Build on MacOS, run on linux x64.

  1. What did you do?

Running prometheus/node_exporter build with 1.6.2 resulted in crashes multiple times. These crashes look like go runtime issues, here's the log:

fatal error: malloc deadlock

goroutine 340924 [running]:
runtime.throw(0xa7bc70, 0xf)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:547 +0x90 fp=0xc820292760 sp=0xc820292748
runtime.mallocgc(0x18, 0x9f77c0, 0x7f4100000000, 0xc820190180)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/malloc.go:544 +0x1cc fp=0xc820292838 sp=0xc820292760
runtime.newobject(0x9f77c0, 0x7f418af1a588)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/malloc.go:781 +0x42 fp=0xc820292860 sp=0xc820292838
runtime.convT2I(0x9f77c0, 0x9a9600, 0xd8fe48, 0xc8202928c8, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/iface.go:168 +0xf0 fp=0xc820292890 sp=0xc820292860
sort.Strings(0xd90008, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/sort/sort.go:305 +0x6c fp=0xc8202928e8 sp=0xc820292890
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.NewDesc(0xc8201900c0, 0x16, 0xc8207576e0, 0x2a, 0x0, 0x0, 0x0, 0x0, 0x2a)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/desc.go:139 +0xf3c fp=0xc820292ba0 sp=0xc8202928e8
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.NewUntyped(0xa68390, 0x4, 0xa6a9c8, 0x6, 0xc82022ffa0, 0xa, 0xc8207576e0, 0x2a, 0x0, 0x0, ...)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/untyped.go:51 +0xd1 fp=0xc820292c08 sp=0xc820292ba0
github.com/prometheus/node_exporter/collector.(*vmStatCollector).Update(0xd90008, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/vmstat_linux.go:59 +0x464 fp=0xc820292e00 sp=0xc820292c08
main.execute(0x7ffd10f57f55, 0x6, 0x7f418aede510, 0xd90008, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82 fp=0xc820292f50 sp=0xc820292e00
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f55, 0x6, 0x7f418aede510, 0xd90008)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49 fp=0xc820292f80 sp=0xc820292f50
runtime.goexit()
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820292f88 sp=0xc820292f80
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f418aedf710, 0x72, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201e1170, 0x72, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201e1170, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201e1110, 0x0, 0x7f418aedf808, 0xc82047a000)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc8201ac0e0, 0x456170, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc8201ac0e0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2427 +0x41
net/http.(*Server).Serve(0xc8201b0400, 0x7f418aedf7d0, 0xc8201ac0e0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2117 +0x129
net/http.(*Server).ListenAndServe(0xc8201b0400, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2098 +0x136
net/http.ListenAndServe(0xa61c38, 0x5, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2195 +0x98
main.main()
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:168 +0x8fc

goroutine 341046 [runnable]:
regexp.(*bitState).reset(0xc820058af0, 0x8, 0x4)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/backtrack.go:96 +0x189
regexp.(*machine).backtrack(0xc82032e6c0, 0x7f418af1a620, 0xc82032e798, 0x0, 0x8, 0x4, 0x5)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/backtrack.go:324 +0x106
regexp.(*Regexp).doExecute(0xc8209cafa0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc82049d100, 0x8, 0x0, 0x4, ...)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/exec.go:449 +0x423
regexp.(*Regexp).replaceAll(0xc8209cafa0, 0x0, 0x0, 0x0, 0xc82049d100, 0x8, 0x4, 0xc8209599d0, 0x0, 0x0, ...)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/regexp.go:501 +0x12d
regexp.(*Regexp).ReplaceAllString(0xc8209cafa0, 0xc82049d100, 0x8, 0xa64ca0, 0x5, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/regexp/regexp.go:460 +0x133
github.com/prometheus/node_exporter/collector.parseMemInfo(0x7f418af1a000, 0xc8204b2018, 0xc8204b2018, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/meminfo_linux.go:105 +0x3ac
github.com/prometheus/node_exporter/collector.getMemInfo(0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/meminfo_linux.go:79 +0x116
github.com/prometheus/node_exporter/collector.(*meminfoCollector).Update(0xc8201ac060, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/meminfo_linux.go:52 +0x43
main.execute(0x7ffd10f57f14, 0x7, 0x7f418aede3a8, 0xc8201ac060, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f14, 0x7, 0x7f418aede3a8, 0xc8201ac060)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 340981 [chan send]:
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc8201461c0, 0xc8201b46c0, 0x130)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:187 +0x1bb
created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:146 +0x18d

goroutine 340984 [runnable]:
syscall.Syscall(0x0, 0x7, 0xc82056b000, 0x1000, 0x540, 0x1000, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x7, 0xc82056b000, 0x1000, 0x1000, 0x379, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/zsyscall_linux_amd64.go:783 +0x5f
syscall.Read(0x7, 0xc82056b000, 0x1000, 0x1000, 0xc82056a000, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/syscall_unix.go:161 +0x4d
os.(*File).read(0xc8201ac1c8, 0xc82056b000, 0x1000, 0x1000, 0x6401a7, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file_unix.go:228 +0x53
os.(*File).Read(0xc8201ac1c8, 0xc82056b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file.go:95 +0x8a
bufio.(*Scanner).Scan(0xc820369900, 0x5)
    /usr/local/Cellar/go/1.6.2/libexec/src/bufio/scan.go:208 +0x97b
github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs.FS.NewStat(0xa619a0, 0x5, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs/stat.go:36 +0x1ee
github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs.ProcStat.StartTime(0xd5d8, 0xc82049f8d0, 0xd, 0xc82049f8e0, 0x1, 0x1904, 0x1904, 0x1904, 0x0, 0xffffffffffffffff, ...)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs/proc_stat.go:165 +0x49
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).processCollect(0xc820148480, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:127 +0x3fa
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).(github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.processCollect)-fm(0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:85 +0x2a
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).Collect(0xc820148480, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:103 +0x2a
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func2(0xc82049e990, 0xc8201b4720, 0x7f418af1a810, 0xc820148480)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:415 +0x58
created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:416 +0x34d

goroutine 340986 [semacquire]:
sync.runtime_Semacquire(0xc82055bcfc)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc82055bcf0)
    /usr/local/Cellar/go/1.6.2/libexec/src/sync/waitgroup.go:127 +0xb4
main.NodeCollector.Collect(0xc8201a2810, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:79 +0x163
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func2(0xc82049e990, 0xc8201b4720, 0x7f418aede628, 0xc8201a2810)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:415 +0x58
created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:416 +0x34d

goroutine 341047 [runnable]:
strings.Join(0xc8206efb38, 0x2, 0x2, 0xa61978, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strings/strings.go:369 +0x3c3
path.Join(0xc8206efb38, 0x2, 0x2, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/path/path.go:153 +0xfa
github.com/prometheus/node_exporter/collector.procFilePath(0xa68130, 0x7, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/paths.go:30 +0xd2
github.com/prometheus/node_exporter/collector.getNetDevStats(0xc8201b8dc0, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netdev_linux.go:34 +0x51
github.com/prometheus/node_exporter/collector.(*netDevCollector).Update(0xc8201ab7e0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netdev_common.go:55 +0x56
main.execute(0x7ffd10f57f1c, 0x6, 0x7f418aede3d0, 0xc8201ab7e0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f1c, 0x6, 0x7f418aede3d0, 0xc8201ab7e0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341049 [runnable]:
github.com/prometheus/node_exporter/collector.parseMegaCliAdapter(0x7f418af1a000, 0xc8201ac1d8, 0xc8201ac1d8, 0x7f418af1a000, 0xc8201ac1d8)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/megacli.go:127
github.com/prometheus/node_exporter/collector.(*megaCliCollector).updateAdapter(0xc8201a3d70, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/megacli.go:171 +0x1a4
github.com/prometheus/node_exporter/collector.(*megaCliCollector).Update(0xc8201a3d70, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/megacli.go:74 +0x33
main.execute(0x7ffd10f57f64, 0x7, 0x7f418aede560, 0xc8201a3d70, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f64, 0x7, 0x7f418aede560, 0xc8201a3d70)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 340980 [runnable]:
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB(0xc8200ac600, 0x7f418aedfb48, 0xc8203ba540, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:427 +0x407
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).ServeHTTP(0xc8200ac600, 0x7f418aedfa60, 0xc8201ac020, 0xc8201461c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:375 +0x173
net/http.(Handler).ServeHTTP-fm(0x7f418aedfa60, 0xc8201ac020, 0xc8201461c0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/h2_bundle.go:3847 +0x50
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x7f418aedf960, 0xc8209a80d0, 0xc8201461c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:158 +0x336
net/http.HandlerFunc.ServeHTTP(0xc8201a4640, 0x7f418aedf960, 0xc8209a80d0, 0xc8201461c0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc8200a25d0, 0x7f418aedf960, 0xc8209a80d0, 0xc8201461c0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc8201b0400, 0x7f418aedf960, 0xc8209a80d0, 0xc8201461c0)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc8206a0080)
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
    /usr/local/Cellar/go/1.6.2/libexec/src/net/http/server.go:2137 +0x44e

goroutine 340922 [runnable]:
syscall.Syscall(0x0, 0x6, 0xc8203a2c87, 0x379, 0x379, 0x379, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x6, 0xc8203a2c87, 0x379, 0x379, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/zsyscall_linux_amd64.go:783 +0x5f
syscall.Read(0x6, 0xc8203a2c87, 0x379, 0x379, 0xb779e9, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/syscall/syscall_unix.go:161 +0x4d
os.(*File).read(0xc820032028, 0xc8203a2c87, 0x379, 0x379, 0x6401a7, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file_unix.go:228 +0x53
os.(*File).Read(0xc820032028, 0xc8203a2c87, 0x379, 0x379, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file.go:95 +0x8a
bufio.(*Scanner).Scan(0xc820584ce8, 0xc8201b4720)
    /usr/local/Cellar/go/1.6.2/libexec/src/bufio/scan.go:208 +0x97b
github.com/prometheus/node_exporter/collector.(*statCollector).Update(0xc8201a8a40, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/stat_linux.go:96 +0x198
main.execute(0x7ffd10f57f34, 0x4, 0x7f418aede448, 0xc8201a8a40, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f34, 0x4, 0x7f418aede448, 0xc8201a8a40)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341051 [runnable]:
strconv.atof64exact(0x2b38, 0x0, 0x2b00, 0x0, 0x10000)
    /usr/local/Cellar/go/1.6.2/libexec/src/strconv/atof.go:369
strconv.atof64(0xc82066017c, 0x5, 0xc82061b4d0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strconv/atof.go:486 +0x106
strconv.ParseFloat(0xc82066017c, 0x5, 0x40, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strconv/atof.go:536 +0x82
github.com/prometheus/node_exporter/collector.(*diskstatsCollector).Update(0xc8201ab1c0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/diskstats_linux.go:198 +0x70b
main.execute(0x7ffd10f57ee2, 0x9, 0x7f418aede2b8, 0xc8201ab1c0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57ee2, 0x9, 0x7f418aede2b8, 0xc8201ab1c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 340982 [semacquire]:
sync.runtime_Semacquire(0xc82049e99c)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc82049e990)
    /usr/local/Cellar/go/1.6.2/libexec/src/sync/waitgroup.go:127 +0xb4
github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func1(0xc82049e990, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:409 +0x21
created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:411 +0x271

goroutine 341052 [runnable]:
bufio.(*Scanner).Scan(0xc8209b0a08, 0xc820112d80)
    /usr/local/Cellar/go/1.6.2/libexec/src/bufio/scan.go:128
github.com/prometheus/node_exporter/collector.mountPointDetails(0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/filesystem_linux.go:87 +0x1e8
github.com/prometheus/node_exporter/collector.(*filesystemCollector).GetStats(0xc8201a88c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/filesystem_linux.go:40 +0x72
github.com/prometheus/node_exporter/collector.(*filesystemCollector).Update(0xc8201a88c0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/filesystem_common.go:109 +0x47
main.execute(0x7ffd10f57efb, 0xa, 0x7f418aede330, 0xc8201a88c0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57efb, 0xa, 0x7f418aede330, 0xc8201a88c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341048 [runnable]:
strings.Join(0xc820295cb8, 0x2, 0x2, 0xa61978, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/strings/strings.go:369 +0x3c3
path.Join(0xc820295cb8, 0x2, 0x2, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/path/path.go:153 +0xfa
github.com/prometheus/node_exporter/collector.sysFilePath(0xa76ae0, 0x9, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/paths.go:34 +0xd2
github.com/prometheus/node_exporter/collector.(*bondingCollector).Update(0xc8201a71b0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/bonding_linux.go:61 +0x5c
main.execute(0x7ffd10f57f5c, 0x7, 0x7f418aede538, 0xc8201a71b0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f5c, 0x7, 0x7f418aede538, 0xc8201a71b0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341044 [runnable]:
runtime.SetFinalizer(0x983bc0, 0xc82009e000, 0x8f1c20, 0xb7a520)
    /usr/local/Cellar/go/1.6.2/libexec/src/runtime/mfinal.go:368 +0x461
os.NewFile(0x9, 0xa6e770, 0xc, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file_unix.go:57 +0xfc
os.OpenFile(0xa6e770, 0xc, 0x0, 0x0, 0xc81ffb158f, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file_unix.go:123 +0x11f
os.Open(0xa6e770, 0xc, 0xc8206cbad0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/os/file.go:244 +0x48
io/ioutil.ReadDir(0xa6e770, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/Cellar/go/1.6.2/libexec/src/io/ioutil/ioutil.go:101 +0x64
github.com/prometheus/node_exporter/vendor/github.com/soundcloud/go-runit/runit.GetServices(0xa6e770, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/vendor/github.com/soundcloud/go-runit/runit/runit.go:58 +0x90
github.com/prometheus/node_exporter/collector.(*runitCollector).Update(0xc8201dc2c0, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/runit.go:84 +0x67
main.execute(0x7ffd10f57f6c, 0x5, 0x7f418aede588, 0xc8201dc2c0, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f6c, 0x5, 0x7f418aede588, 0xc8201dc2c0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 340927 [runnable]:
github.com/prometheus/node_exporter/collector.parseNetStats(0x7f418af1a000, 0xc82009c4f8, 0xc8202e00a0, 0x11, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:117 +0x590
github.com/prometheus/node_exporter/collector.getNetStats(0xc8202e00a0, 0x11, 0x0, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:96 +0x113
github.com/prometheus/node_exporter/collector.(*netStatCollector).Update(0xc8201ac070, 0xc8201b4720, 0x0, 0x0)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:50 +0x8a
main.execute(0x7ffd10f57f23, 0x7, 0x7f418aede3f8, 0xc8201ac070, 0xc8201b4720)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:96 +0x82
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57f23, 0x7, 0x7f418aede3f8, 0xc8201ac070)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:75 +0x49
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137

goroutine 341050 [runnable]:
main.NodeCollector.Collect.func1(0xc8201b4720, 0xc82055bcf0, 0x7ffd10f57ed8, 0x9, 0x7f418aede230, 0xc8201a6680)
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:74
created by main.NodeCollector.Collect
    /Users/hryamzik/gopath/src/github.com/prometheus/node_exporter/node_exporter.go:77 +0x137
@hryamzik hryamzik mentioned this issue Apr 25, 2016
Closed
@bradfitz bradfitz added this to the Unplanned milestone May 4, 2016
@bradfitz bradfitz changed the title malloc deadlock runtime: malloc deadlock May 4, 2016
@aclements
Copy link
Member

@hryamzik, can you reproduce this with GOTRACEBACK=crash and send the backtrace?

@hryamzik
Copy link
Author

hryamzik commented May 5, 2016

@aclements I'll give it a try. I have currently moved this version from production but I think I could simulate this.

@Kubuxu Kubuxu mentioned this issue May 19, 2016
Closed
@hryamzik
Copy link
Author

I've updated to latest version of node_exporter and do not face this any more (with go 1.5). Unfortunately I don't have time for debugging this, so I'll close this for now.

@golang golang locked and limited conversation to collaborators Jun 22, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
4 participants
@bradfitz @hryamzik @aclements @gopherbot