Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: found bad pointer in Go heap #29362

Closed
methane opened this issue Dec 20, 2018 · 30 comments
Closed

runtime: fatal error: found bad pointer in Go heap #29362

methane opened this issue Dec 20, 2018 · 30 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone

Comments

@methane
Copy link
Contributor

methane commented Dec 20, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/force/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/force/.golang"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/n5/c1hcw05942s2pfbdf3pgjnh80000gq/T/go-build447375688=/tmp/go-build -gno-record-gcc-switches -fno-common"

We use GOOS=linux GOARCH=amd64 when build.
We ran the program on the other machine.

Running machine info
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux (hostname) 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux

What did you do?

We're building online game server with Go.
We faced a random crash like this.

crash report
runtime: pointer 0xc009a038ca to unused region of span span.base()=0xc0035c2000 span.limit=0xc0035c4000 span.state=1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
runtime stack:
runtime.throw(0xc046cf, 0x3e)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/panic.go:608 +0x72 fp=0xc0000abf00 sp=0xc0000abed0 pc=0x42bf02
runtime.findObject(0xc009a038ca, 0x0, 0x0, 0xc0024b5380, 0x7f67d219edc0, 0xd)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/mbitmap.go:399 +0x3b6 fp=0xc0000abf50 sp=0xc0000abf00 pc=0x413bf6
runtime.wbBufFlush1(0xc000086a00)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/mwbbuf.go:252 +0xd1 fp=0xc0000abfb8 sp=0xc0000abf50 pc=0x428121
runtime.wbBufFlush.func1()
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/mwbbuf.go:195 +0x3a fp=0xc0000abfd0 sp=0xc0000abfb8 pc=0x457e1a
runtime.systemstack(0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/asm_amd64.s:351 +0x66 fp=0xc0000abfd8 sp=0xc0000abfd0 pc=0x459af6
runtime.mstart()
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/proc.go:1229 fp=0xc0000abfe0 sp=0xc0000abfd8 pc=0x4307d0
goroutine 143 [running]:
runtime.systemstack_switch()
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/asm_amd64.s:311 fp=0xc0053e7d38 sp=0xc0053e7d30 pc=0x459a80
runtime.wbBufFlush(0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/mwbbuf.go:194 +0x4e fp=0xc0053e7d58 sp=0xc0053e7d38 pc=0x427fbe
runtime.typeBitsBulkBarrier(0xaf3640, 0xc006019b80, 0xc0053e7f28, 0x10)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/mbitmap.go:737 +0x111 fp=0xc0053e7dc0 sp=0xc0053e7d58 pc=0x4145c1
runtime.sendDirect(0xaf3640, 0xc00298c600, 0xc0053e7f28)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/chan.go:312 +0x50 fp=0xc0053e7df8 sp=0xc0053e7dc0 pc=0x4053d0
runtime.send(0xc0023b4f60, 0xc00298c600, 0xc0053e7f28, 0xc0053e7e88, 0x3)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/chan.go:283 +0xde fp=0xc0053e7e28 sp=0xc0053e7df8 pc=0x40533e
runtime.chansend(0xc0023b4f60, 0xc0053e7f28, 0x1, 0x755280, 0x1)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/chan.go:191 +0x4df fp=0xc0053e7ea8 sp=0xc0053e7e28 pc=0x40514f
runtime.chansend1(0xc0023b4f60, 0xc0053e7f28)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/chan.go:125 +0x35 fp=0xc0053e7ee0 sp=0xc0053e7ea8 pc=0x404c65
(snip)

Other info:

  • We don't use CGO.
  • Crash happens even with -race flag, without any race report.
  • We don't use many "unsafe" libraries. This is our glide.lock. Some packages including go-sqlite3 in the glide.lock are not used in the game server.
  • span.state is always 1.
  • Stacktrace is always runtime.wbBufFlush() ... runtime.findObject(). But caller of wbBufFlush() is various.
  • Crash happens more often when GOMAXPROCS=1. It happens less than 30 min.
  • Crash happens with Go 1.11.2, 1.11.3, 1.11.4, and 1.12beta1.
  • Crash doesn't happen when GOMAXPROCS=1 GODEBUG=invalidptr=0, more than 6 hours (false positive?)
  • Crash doesn't happen with Go 1.10.5 and GOMAXPROCS=1 more than 9 hours. (Go 1.11 regression?)
  • No crash with GODEBUG=gcstoptheworld=1

I don't think this issue is same to #26243 because stack trace and environment are different.

Can we do anything to investigate this issue?

@methane methane changed the title fatal error: found bad pointer in Go heap (Go 1.11, amd64, linux) runtime: fatal error: found bad pointer in Go heap (Go 1.11, amd64, linux) Dec 20, 2018
@agnivade
Copy link
Contributor

It would help if you could try with the 1.12 beta1 and see if the crash continues to happen.

Also, I guess it's not possible to reduce the code to a shortened reproducer which we can use ?

@agnivade agnivade changed the title runtime: fatal error: found bad pointer in Go heap (Go 1.11, amd64, linux) runtime: fatal error: found bad pointer in Go heap Dec 20, 2018
@agnivade agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 20, 2018
@agnivade agnivade added this to the Go1.13 milestone Dec 20, 2018
@ianlancetaylor
Copy link
Contributor

CC @aclements @mknyszek

@mknyszek
Copy link
Contributor

A short reproducer, or any more information would be great. It's really hard to tell what could be going wrong with just this information, though lowering GOMAXPROCS causing it to fail faster is an interesting point.

I think it's still too early to rule out a cgo and unsafe issue until we get more information. Although you say you don't use cgo, perusing at some of your dependencies in glide.lock indicates that some of them use cgo (e.g. parts of the systemd repository). Try building your application with CGO_ENABLED=0 and see what happens. Note that even if it fails to compile that doesn't necessarily mean you're making a cgo call anywhere in your code, but it'll at least show that your code may depend on cgo in some way.

The fact that it succeeds with invalidptr=0 suggests to me that there's a stale value (perhaps produced via cgo/unsafe) in a pointer-typed slot somewhere, but the fact that your program doesn't crash means it probably doesn't ever get dereferenced, it just gets placed in a pointer-typed slot. That pointer, 0xc009a038ca, looks like a heap pointer to me, but it's very far away from the span that ended up getting looked up, which makes it seem like a stale pointer value to me.

As for an example of where it could still be a cgo issue: consider if one of your dependencies was making some false assumption about cgo that's mostly true on a certain platform in an older toolchain version, but with a toolchain release that assumption became true less often. Similar case with unsafe. Not saying that this is exactly what's going on (in fact, it's probably not) but at this point it's still a possibility until we can figure out where that pointer came from.

Figuring out where the pointer came from is a bit tricky, because in this case the error is being caught during a write barrier buffer flush, which could be happening long after the write barrier for the pointer value was executed. One thing that might help fish this out is to open this up in gdb and try and see what values are at the offending address, and see if that looks like anything you might recognize?

As @agnivade suggests, could you run it with the 1.12 beta as well?

@methane
Copy link
Contributor Author

methane commented Dec 21, 2018

I think it's still too early to rule out a cgo and unsafe issue until we get more information. Although you say you don't use cgo, perusing at some of your dependencies in glide.lock indicates that some of them use cgo (e.g. parts of the systemd repository). Try building your application with CGO_ENABLED=0 and see what happens.

When I said "We don't use cgo", it meant we can build our application with CGO_ENABLED=0.
This is what go build -a -v shows, without our project code.
https://gist.github.com/methane/e1bb17851616d559a86ba71fa8d453c8#file-build_packages

On the other hand, it's difficult to build without unsafe. Even sync depends on it...

@mknyszek
Copy link
Contributor

mknyszek commented Dec 21, 2018

When I said "We don't use cgo", it meant we can build our application with CGO_ENABLED=0.
This is what go build -a -v shows, without our project code.
https://gist.github.com/methane/e1bb17851616d559a86ba71fa8d453c8#file-build_packages

Got it! Thanks for confirming and I'm sorry for the noise.

On the other hand, it's difficult to build without unsafe. Even sync depends on it...

For sure, and I'm certainly not suggesting you stop using unsafe. All I was trying to get at was that maybe a dependency isn't using the unsafe package right. I only mention it to leave it on the table as a possibility, because until we know where that pointer came from, or have other new information, it's really up in the air. It could very well be a bug in the runtime, but it could also be in user code someplace.

The fact that all of your backtraces have this failing in wbBufFlush and the lack of failure from invalidptr=0 leads me to believe that there's some harmless pointer value somewhere that's invalid but never dereferenced, yet a write barrier picks up on it because it gets deleted, or it gets placed into an already-marked object.

Another thing that might be helpful is finding which commit between 1.10 and 1.11 caused the issue to start happening via bisection. This would also give us a lot of information (though I suspect this might take a long time, if it takes ~30 minutes to reproduce).

@mark-rushakoff
Copy link
Contributor

I have a reproducer at https://github.com/mark-rushakoff/go-issue-24993 that produces a similar trace:

untime: pointer 0xc000336000 to unallocated span span.base()=0xc000334000 span.limit=0xc000338000 span.state=3
runtime: found in object at *(0xc0001cc420+0x10)
object=0xc0001cc420 s.base()=0xc0001cc000 s.limit=0xc0001cdfe0 s.spanclass=14 s.elemsize=96 s.state=mSpanInUse
 *(object+0) = 0x0
 *(object+8) = 0x10
 *(object+16) = 0xc000336000 <==
 *(object+24) = 0x100000010
 *(object+32) = 0x16d7800
 *(object+40) = 0x0
 *(object+48) = 0x0
 *(object+56) = 0x0
 *(object+64) = 0x0
 *(object+72) = 0x0
 *(object+80) = 0x0
 *(object+88) = 0x0
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x17a481e, 0x3e)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/panic.go:617 +0x72 fp=0x70000d71ed08 sp=0x70000d71ecd8 pc=0x102d892
runtime.findObject(0xc000336000, 0xc0001cc420, 0x10, 0xd150, 0x2a12800, 0xc000045270)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mbitmap.go:397 +0x3b4 fp=0x70000d71ed58 sp=0x70000d71ed08 pc=0x1015c44
runtime.scanobject(0xc0001cc420, 0xc000045270)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgcmark.go:1161 +0x216 fp=0x70000d71ede8 sp=0x70000d71ed58 pc=0x1021326
runtime.gcDrain(0xc000045270, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgcmark.go:919 +0x217 fp=0x70000d71ee40 sp=0x70000d71ede8 pc=0x1020b37
runtime.gcBgMarkWorker.func2()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1873 +0x80 fp=0x70000d71ee80 sp=0x70000d71ee40 pc=0x1054f20
runtime.systemstack(0x2800000)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:351 +0x66 fp=0x70000d71ee88 sp=0x70000d71ee80 pc=0x1056ee6
runtime.mstart()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:1153 fp=0x70000d71ee90 sp=0x70000d71ee88 pc=0x1031db0

...

Some commentary ongoing in #24993. Not 100% clear to me if the reproducer is related, but it does at least produce the found bad pointer in Go heap panic.

@methane
Copy link
Contributor Author

methane commented Dec 21, 2018

Updated: No crash with GODEBUG=gcstoptheworld=1

@methane
Copy link
Contributor Author

methane commented Dec 21, 2018

Updated: Crash with Go 1.12beta1

@mknyszek
Copy link
Contributor

Updated: No crash with GODEBUG=gcstoptheworld=1

Huh. I'm only loosely familiar with the precise semantics of gcstoptheworld (FWIU it's the same GC except the mark phases just don't let the program run concurrently). This points more toward a runtime bug I think?

I have a reproducer at https://github.com/mark-rushakoff/go-issue-24993 that produces a similar trace:

Awesome, I'll dig into this tomorrow and see if I can figure anything out.

@cherrymui
Copy link
Member

gcstoptheworld effectively disables the write barrier, as the write barrier is only needed for the program running concurrently with the GC. So if the bad pointer is always coming from the write barrier, this may explain. Maybe we inserted a write barrier where it should not have one?

I wonder if we could have a debug mode that the write barrier is always eager, i.e. not going through the buffer. So if the bad pointer is from the write barrier, we see a fault immediately. Effectively maybe we just set the buffer size to 0 so every write barrier triggers a flush?

@mark-rushakoff
Copy link
Contributor

My reproducer is able to get the found bad pointer in Go heap panic even with GODEBUG=gcstoptheworld=1. Using mark-rushakoff/go-issue-24993@52db6e7 and go version devel +90dca98d33 Thu Dec 20 22:11:45 2018 +0000 darwin/amd64.

Full stack trace:

$ GODEBUG=gcstoptheworld=1 gotip run main.go
runtime: pointer 0xc0002e5fd8 to unallocated span span.base()=0xc0002e4000 span.limit=0xc0002e6000 span.state=3
runtime: found in object at *(0xc000392798+0x8)
object=0xc000392798 s.base()=0xc00038c000 s.limit=0xc000394000 s.spanclass=0 s.elemsize=2048 s.state=mSpanManual
 *(object+0) = 0x17acc88
 *(object+8) = 0xc0002e5fd8 <==
 *(object+16) = 0x0
 *(object+24) = 0x1058f21
 *(object+32) = 0xc0000961e0
 *(object+40) = 0xc0000942a0
 *(object+48) = 0xc000300510
 *(object+56) = 0x18743a0
 *(object+64) = 0xc00013c1c0
 *(object+72) = 0xc00038d668
 *(object+80) = 0x4
 *(object+88) = 0x4
 *(object+96) = 0xc00038d680
 *(object+104) = 0xc0003d4000
 *(object+112) = 0x6
 *(object+120) = 0xc00038d688
 *(object+128) = 0x4
 *(object+136) = 0x4
 *(object+144) = 0xc00038d6a0
 *(object+152) = 0x6
 *(object+160) = 0x6
 *(object+168) = 0xc00038d6a8
 *(object+176) = 0x4
 *(object+184) = 0x4
 *(object+192) = 0xc00038d6c0
 *(object+200) = 0x6
 *(object+208) = 0x6
 *(object+216) = 0xc00038d6c8
 *(object+224) = 0x4
 *(object+232) = 0x4
 *(object+240) = 0xc00038d6e0
 *(object+248) = 0x6
 *(object+256) = 0x6
 *(object+264) = 0xc00038d6e8
 *(object+272) = 0x4
 *(object+280) = 0x4
 *(object+288) = 0xc00038d700
 *(object+296) = 0x6
 *(object+304) = 0x6
 *(object+312) = 0xc00038d708
 *(object+320) = 0x4
 *(object+328) = 0x4
 *(object+336) = 0xc00038d720
 *(object+344) = 0x6
 *(object+352) = 0x6
 *(object+360) = 0xc00038d728
 *(object+368) = 0x4
 *(object+376) = 0x4
 *(object+384) = 0xc00038d740
 *(object+392) = 0x6
 *(object+400) = 0x6
 *(object+408) = 0xc00038d748
 *(object+416) = 0x4
 *(object+424) = 0x4
 *(object+432) = 0xc00038d760
 *(object+440) = 0x6
 *(object+448) = 0x6
 *(object+456) = 0xc00038d768
 *(object+464) = 0x4
 *(object+472) = 0x4
 *(object+480) = 0xc00038d780
 *(object+488) = 0x6
 *(object+496) = 0x6
 *(object+504) = 0xc00038d788
 *(object+512) = 0x4
 *(object+520) = 0x4
 *(object+528) = 0xc00038d7a0
 *(object+536) = 0x6
 *(object+544) = 0x6
 *(object+552) = 0xc00038d7a8
 *(object+560) = 0x4
 *(object+568) = 0x4
 *(object+576) = 0xc00038d7c0
 *(object+584) = 0x6
 *(object+592) = 0x6
 *(object+600) = 0xc00038d7c8
 *(object+608) = 0x4
 *(object+616) = 0x4
 *(object+624) = 0xc00038d7e0
 *(object+632) = 0x6
 *(object+640) = 0x6
 *(object+648) = 0xc00038d7e8
 *(object+656) = 0x4
 *(object+664) = 0x4
 *(object+672) = 0xc00038d800
 *(object+680) = 0x6
 *(object+688) = 0x6
 *(object+696) = 0xc00038d808
 *(object+704) = 0x4
 *(object+712) = 0x4
 *(object+720) = 0xc00038d820
 *(object+728) = 0x6
 *(object+736) = 0x6
 *(object+744) = 0xc00038d828
 *(object+752) = 0x4
 *(object+760) = 0x4
 *(object+768) = 0xc00038d840
 *(object+776) = 0x6
 *(object+784) = 0x6
 *(object+792) = 0xc00038d848
 *(object+800) = 0x4
 *(object+808) = 0x4
 *(object+816) = 0xc00038d860
 *(object+824) = 0x6
 *(object+832) = 0x6
 *(object+840) = 0xc00038d868
 *(object+848) = 0x4
 *(object+856) = 0x4
 *(object+864) = 0xc00038d880
 *(object+872) = 0x6
 *(object+880) = 0x6
 *(object+888) = 0x1023922
 *(object+896) = 0x10187e0
 *(object+904) = 0x2000
 *(object+912) = 0x0
 *(object+920) = 0x6
 *(object+928) = 0x10157aa
 *(object+936) = 0x100d29d
 *(object+944) = 0x10157aa
 *(object+952) = 0x100d29d
 *(object+960) = 0x1d53ee0
 *(object+968) = 0x6
 *(object+976) = 0x1d53ee0
 *(object+984) = 0xc000392b90
 *(object+992) = 0x1d53ee0
 *(object+1000) = 0xc000392c20
 *(object+1008) = 0x100dbe0
 *(object+1016) = 0x3900000
 ...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x17a48be, 0x3e)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/panic.go:617 +0x72 fp=0x70000d599a38 sp=0x70000d599a08 pc=0x102d892
runtime.findObject(0xc0002e5fd8, 0xc000392798, 0x8, 0xc0005d6860, 0xc0005d6800, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mbitmap.go:397 +0x3b4 fp=0x70000d599a88 sp=0x70000d599a38 pc=0x1015c44
runtime.scanblock(0xc000392798, 0x10, 0x18420b9, 0xc000045770, 0x70000d599b88)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgcmark.go:1057 +0xa8 fp=0x70000d599ae8 sp=0x70000d599a88 pc=0x1021038
runtime.scanstack(0xc0002fa780, 0xc000045770)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgcmark.go:750 +0x24e fp=0x70000d599cd8 sp=0x70000d599ae8 pc=0x102036e
runtime.scang(0xc0002fa780, 0xc000045770)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:888 +0x1ea fp=0x70000d599d28 sp=0x70000d599cd8 pc=0x10313ba
runtime.markroot.func1()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgcmark.go:221 +0x6d fp=0x70000d599d68 sp=0x70000d599d28 pc=0x10550bd
runtime.markroot(0xc000045770, 0x700000000082)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgcmark.go:202 +0x2e6 fp=0x70000d599de8 sp=0x70000d599d68 pc=0x101f336
runtime.gcDrain(0xc000045770, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgcmark.go:886 +0x10e fp=0x70000d599e40 sp=0x70000d599de8 pc=0x1020a2e
runtime.gcBgMarkWorker.func2()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1873 +0x80 fp=0x70000d599e80 sp=0x70000d599e40 pc=0x1054f20
runtime.systemstack(0x3800000)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:351 +0x66 fp=0x70000d599e88 sp=0x70000d599e80 pc=0x1056ee6
runtime.mstart()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:1153 fp=0x70000d599e90 sp=0x70000d599e88 pc=0x1031db0

goroutine 68 [GC worker (idle)]:
runtime.systemstack_switch()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:311 fp=0xc0003d3f60 sp=0xc0003d3f58 pc=0x1056e70
runtime.gcBgMarkWorker(0xc000044500)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1860 +0x1b8 fp=0xc0003d3fd8 sp=0xc0003d3f60 pc=0x101d2c8
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003d3fe0 sp=0xc0003d3fd8 pc=0x1058f21
created by runtime.gcBgMarkStartWorkers
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1754 +0x77

goroutine 1 [chan receive]:
runtime.gopark(0x17acde0, 0xc0000942f8, 0xc00018170d, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0000e52f8 sp=0xc0000e52d8 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chanrecv(0xc0000942a0, 0xc0000e5430, 0xc000000101, 0x15e0101)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:524 +0x2bf fp=0xc0000e5388 sp=0xc0000e52f8 pc=0x1007c6f
runtime.chanrecv1(0xc0000942a0, 0xc0000e5430)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:406 +0x2b fp=0xc0000e53b8 sp=0xc0000e5388 pc=0x100795b
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply(0xc00016d110, 0xc000300510, 0x2701, 0xc0000aa0a0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:417 +0x152 fp=0xc0000e5468 sp=0xc0000e53b8 pc=0x15e50a2
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).deleteSeriesRange(0xc000256120, 0xc000426000, 0x1, 0x2710, 0x8000000000000000, 0x7fffffffffffffff, 0xc0003004e0, 0x1)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1501 +0x156 fp=0xc0000e56c0 sp=0xc0000e5468 pc=0x15d09e6
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRangeWithPredicate(0xc000256120, 0x185c000, 0xc0000aa080, 0xc0000e5810, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1470 +0x7ae fp=0xc0000e57e0 sp=0xc0000e56c0 pc=0x15d072e
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange(0xc000256120, 0x185c000, 0xc0000aa080, 0x8000000000000000, 0x7fffffffffffffff, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1362 +0x7e fp=0xc0000e5838 sp=0xc0000e57e0 pc=0x15cff4e
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteMeasurement(0xc000256120, 0xc000396000, 0x3, 0x3, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1770 +0x226 fp=0xc0000e5918 sp=0xc0000e5838 pc=0x15d1b66
main.main()
	/Users/mr/gomod/go-issue-24993/main.go:58 +0x858 fp=0xc0000e5f98 sp=0xc0000e5918 pc=0x1632708
runtime.main()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:200 +0x207 fp=0xc0000e5fe0 sp=0xc0000e5f98 pc=0x102f1c7
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000e5fe8 sp=0xc0000e5fe0 pc=0x1058f21

goroutine 2 [force gc (idle)]:
runtime.gopark(0x17acde0, 0x1d52d80, 0x1410, 0x1)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc000054fb0 sp=0xc000054f90 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.forcegchelper()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:250 +0xb6 fp=0xc000054fe0 sp=0xc000054fb0 pc=0x102f456
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000054fe8 sp=0xc000054fe0 pc=0x1058f21
created by runtime.init.5
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:239 +0x35

goroutine 18 [GC sweep wait]:
runtime.gopark(0x17acde0, 0x1d53600, 0x140c, 0x1)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0000507a8 sp=0xc000050788 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.bgsweep(0xc000092000)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgcsweep.go:89 +0x12d fp=0xc0000507d8 sp=0xc0000507a8 pc=0x10229bd
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000507e0 sp=0xc0000507d8 pc=0x1058f21
created by runtime.gcenable
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:208 +0x58

goroutine 3 [finalizer wait]:
runtime.gopark(0x17acde0, 0x1d70658, 0x140f, 0x1)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc000055758 sp=0xc000055738 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.runfinq()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mfinal.go:175 +0xa5 fp=0xc0000557e0 sp=0xc000055758 pc=0x1019685
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000557e8 sp=0xc0000557e0 pc=0x1058f21
created by runtime.createfing
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mfinal.go:156 +0x61

goroutine 19 [select]:
runtime.gopark(0x17ace90, 0x0, 0x1809, 0x1)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0002bec60 sp=0xc0002bec40 pc=0x102f5a6
runtime.selectgo(0xc0002bef48, 0xc0002bedd0, 0x2, 0xc0000545e0, 0x10466c2)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/select.go:313 +0xcaa fp=0xc0002bed88 sp=0xc0002bec60 pc=0x103df0a
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc000256120)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1920 +0x11c fp=0xc0002befa8 sp=0xc0002bed88 pc=0x15d329c
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc0000b2060, 0xc000256120)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:499 +0x51 fp=0xc0002befd0 sp=0xc0002befa8 pc=0x161edb1
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0002befd8 sp=0xc0002befd0 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:499 +0x128

goroutine 4 [syscall]:
runtime.notetsleepg(0x1d57d80, 0x3b9a8da4, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/lock_sema.go:280 +0x41 fp=0xc000055f60 sp=0xc000055f20 pc=0x100c681
runtime.timerproc(0x1d57d60)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/time.go:311 +0x2e8 fp=0xc000055fd8 sp=0xc000055f60 pc=0x104bbb8
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000055fe0 sp=0xc000055fd8 pc=0x1058f21
created by runtime.(*timersBucket).addtimerLocked
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/time.go:169 +0x10e

goroutine 34 [syscall]:
runtime.notetsleepg(0x1d57e80, 0x3b99727e, 0x102c5cf)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/lock_sema.go:280 +0x41 fp=0xc000054760 sp=0xc000054720 pc=0x100c681
runtime.timerproc(0x1d57e60)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/time.go:311 +0x2e8 fp=0xc0000547d8 sp=0xc000054760 pc=0x104bbb8
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000547e0 sp=0xc0000547d8 pc=0x1058f21
created by runtime.(*timersBucket).addtimerLocked
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/time.go:169 +0x10e

goroutine 1302 [runnable]:
syscall.syscallX(0x1076450, 0x26, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/sys_darwin_64.go:16 +0x2e fp=0xc0002bbbb0 sp=0xc0002bbb88 pc=0x104aede
syscall.Seek(0x26, 0x0, 0x0, 0xc00030e6d8, 0xc0002bbc90, 0x108fd4e)
	/Users/mr/gotip/src/github.com/golang/go/src/syscall/zsyscall_darwin_amd64.go:1272 +0x58 fp=0xc0002bbc00 sp=0xc0002bbbb0 pc=0x1074d88
internal/poll.(*FD).Seek(0xc00030e6c0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/internal/poll/fd_unix.go:404 +0xa6 fp=0xc0002bbc40 sp=0xc0002bbc00 pc=0x108cf66
os.(*File).seek(...)
	/Users/mr/gotip/src/github.com/golang/go/src/os/file_unix.go:281
os.(*File).Seek(0xc0000b8058, 0x0, 0x0, 0x4, 0x4, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/os/file.go:196 +0x5e fp=0xc0002bbca0 sp=0xc0002bbc40 pc=0x1090bee
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Tombstoner).Walk(0xc0000e8140, 0xc0002bbdc8, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/tombstone.go:248 +0x18d fp=0xc0002bbd08 sp=0xc0002bbca0 pc=0x160bc8d
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*TSMReader).applyTombstones(0xc0000de230, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/reader.go:262 +0x124 fp=0xc0002bbe00 sp=0xc0002bbd08 pc=0x15fee44
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*batchDelete).Commit(0xc0000be0e0, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/reader.go:622 +0xa7 fp=0xc0002bbe28 sp=0xc0002bbe00 pc=0x1600df7
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).deleteSeriesRange.func1(0x18743a0, 0xc0000de230, 0xc0000de230, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1537 +0x5ba fp=0xc0002bbf78 sp=0xc0002bbe28 pc=0x1620a4a
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc0000de230)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:411 +0xc2 fp=0xc0002bbfb8 sp=0xc0002bbf78 pc=0x1623162
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0002bbfc0 sp=0xc0002bbfb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1304 [runnable]:
runtime.gcStart(0x1, 0x0, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1199 +0x68c fp=0xc000385438 sp=0xc000385430 pc=0x101c0ec
runtime.mallocgc(0x18000, 0x168b4e0, 0x5c5bf01, 0xc000385528)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/malloc.go:1032 +0x3e8 fp=0xc0003854d8 sp=0xc000385438 pc=0x100d868
runtime.makeslice(0x168b4e0, 0x0, 0x1000, 0x102cab7)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/slice.go:49 +0x6c fp=0xc000385508 sp=0xc0003854d8 pc=0x10426bc
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*TSMReader).applyTombstones(0xc0000de460, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/reader.go:260 +0x86 fp=0xc000385600 sp=0xc000385508 pc=0x15feda6
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*batchDelete).Commit(0xc0000be0e8, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/reader.go:622 +0xa7 fp=0xc000385628 sp=0xc000385600 pc=0x1600df7
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).deleteSeriesRange.func1(0x18743a0, 0xc0000de460, 0xc0000de460, 0x1)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1537 +0x5ba fp=0xc000385778 sp=0xc000385628 pc=0x1620a4a
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc0000de460)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:411 +0xc2 fp=0xc0003857b8 sp=0xc000385778 pc=0x1623162
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003857c0 sp=0xc0003857b8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 69 [GC worker (idle)]:
runtime.gopark(0x17acc40, 0xc000038c80, 0x1417, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0003d2760 sp=0xc0003d2740 pc=0x102f5a6
runtime.gcBgMarkWorker(0xc000046a00)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1806 +0xfc fp=0xc0003d27d8 sp=0xc0003d2760 pc=0x101d20c
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003d27e0 sp=0xc0003d27d8 pc=0x1058f21
created by runtime.gcBgMarkStartWorkers
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1754 +0x77

goroutine 29 [GC worker (idle)]:
runtime.gopark(0x17acc40, 0xc0003d8030, 0x1417, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0003d5760 sp=0xc0003d5740 pc=0x102f5a6
runtime.gcBgMarkWorker(0xc000042000)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1806 +0xfc fp=0xc0003d57d8 sp=0xc0003d5760 pc=0x101d20c
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003d57e0 sp=0xc0003d57d8 pc=0x1058f21
created by runtime.gcBgMarkStartWorkers
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1754 +0x77

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x17acc40, 0xc0003a8590, 0x1417, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc000057760 sp=0xc000057740 pc=0x102f5a6
runtime.gcBgMarkWorker(0xc000048f00)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1806 +0xfc fp=0xc0000577d8 sp=0xc000057760 pc=0x101d20c
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000577e0 sp=0xc0000577d8 pc=0x1058f21
created by runtime.gcBgMarkStartWorkers
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1754 +0x77

goroutine 1303 [runnable]:
runtime.Gosched(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:266
runtime.gcStart(0x1, 0x0, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/mgc.go:1352 +0x352 fp=0xc000069c38 sp=0xc000069bc8 pc=0x101bdb2
runtime.mallocgc(0x18000, 0x168b4e0, 0x152a101, 0xc000351528)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/malloc.go:1032 +0x3e8 fp=0xc000069cd8 sp=0xc000069c38 pc=0x100d868
runtime.makeslice(0x168b4e0, 0x0, 0x1000, 0xc000046a01)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/slice.go:49 +0x6c fp=0xc000069d08 sp=0xc000069cd8 pc=0x10426bc
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*TSMReader).applyTombstones(0xc0000de380, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/reader.go:260 +0x86 fp=0xc000069e00 sp=0xc000069d08 pc=0x15feda6
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*batchDelete).Commit(0xc000012088, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/reader.go:622 +0xa7 fp=0xc000069e28 sp=0xc000069e00 pc=0x1600df7
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).deleteSeriesRange.func1(0x18743a0, 0xc0000de380, 0xc0000de380, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1537 +0x5ba fp=0xc000069f78 sp=0xc000069e28 pc=0x1620a4a
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc0000de380)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:411 +0xc2 fp=0xc000069fb8 sp=0xc000069f78 pc=0x1623162
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000069fc0 sp=0xc000069fb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1309 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0xc00039160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc000391ec0 sp=0xc000391ea0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc000391f98, 0x39a3001, 0x16230d9, 0xc0003183f0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc000391f40 sp=0xc000391ec0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc000391f98)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc000391f78 sp=0xc000391f40 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013c230)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc000391fb8 sp=0xc000391f78 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000391fc0 sp=0xc000391fb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 106 [syscall]:
runtime.notetsleepg(0x1d57e00, 0x3b9a9ee9, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/lock_sema.go:280 +0x41 fp=0xc0003cff60 sp=0xc0003cff20 pc=0x100c681
runtime.timerproc(0x1d57de0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/time.go:311 +0x2e8 fp=0xc0003cffd8 sp=0xc0003cff60 pc=0x104bbb8
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003cffe0 sp=0xc0003cffd8 pc=0x1058f21
created by runtime.(*timersBucket).addtimerLocked
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/time.go:169 +0x10e

goroutine 1305 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0x160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc000261ec0 sp=0xc000261ea0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc000261f98, 0x5cea001, 0x16230d9, 0xc0003183f0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc000261f40 sp=0xc000261ec0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc000261f98)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc000261f78 sp=0xc000261f40 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013c070)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc000261fb8 sp=0xc000261f78 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000261fc0 sp=0xc000261fb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1310 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0x102160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0002876c0 sp=0xc0002876a0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc000287798, 0x1, 0x16230d9, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc000287740 sp=0xc0002876c0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc000287798)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc000287778 sp=0xc000287740 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013c2a0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc0002877b8 sp=0xc000287778 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0002877c0 sp=0xc0002877b8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1314 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0x102160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc000353ec0 sp=0xc000353ea0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc000353f98, 0x1, 0x16230d9, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc000353f40 sp=0xc000353ec0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc000353f98)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc000353f78 sp=0xc000353f40 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013c770)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc000353fb8 sp=0xc000353f78 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000353fc0 sp=0xc000353fb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 798 [syscall]:
runtime.notetsleepg(0x1d57f00, 0x3b9aa1ce, 0x1623162)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/lock_sema.go:280 +0x41 fp=0xc00038ff60 sp=0xc00038ff20 pc=0x100c681
runtime.timerproc(0x1d57ee0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/time.go:311 +0x2e8 fp=0xc00038ffd8 sp=0xc00038ff60 pc=0x104bbb8
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00038ffe0 sp=0xc00038ffd8 pc=0x1058f21
created by runtime.(*timersBucket).addtimerLocked
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/time.go:169 +0x10e

goroutine 1307 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0x160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc00038c6c0 sp=0xc00038c6a0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc00038c798, 0x80a0001, 0x16230d9, 0xc0003183f0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc00038c740 sp=0xc00038c6c0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc00038c798)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc00038c778 sp=0xc00038c740 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00018c000)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc00038c7b8 sp=0xc00038c778 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00038c7c0 sp=0xc00038c7b8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1313 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0x102160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0003ceec0 sp=0xc0003ceea0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc0003cef98, 0x1, 0x16230d9, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc0003cef40 sp=0xc0003ceec0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc0003cef98)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc0003cef78 sp=0xc0003cef40 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013c700)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc0003cefb8 sp=0xc0003cef78 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003cefc0 sp=0xc0003cefb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1315 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0x102160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0003d0ec0 sp=0xc0003d0ea0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc0003d0f98, 0x1, 0x16230d9, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc0003d0f40 sp=0xc0003d0ec0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc0003d0f98)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc0003d0f78 sp=0xc0003d0f40 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013c850)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc0003d0fb8 sp=0xc0003d0f78 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003d0fc0 sp=0xc0003d0fb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1312 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0x102160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc00038d6c0 sp=0xc00038d6a0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc00038d798, 0x1, 0x16230d9, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc00038d740 sp=0xc00038d6c0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc00038d798)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc00038d778 sp=0xc00038d740 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013c540)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc00038d7b8 sp=0xc00038d778 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00038d7c0 sp=0xc00038d7b8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1300 [chan send]:
runtime.gopark(0x17acde0, 0xc000096238, 0xc00038160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc00038eec0 sp=0xc00038eea0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc00038ef98, 0x1062301, 0x16230d9, 0x17acf00)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc00038ef40 sp=0xc00038eec0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc00038ef98)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc00038ef78 sp=0xc00038ef40 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc0000de000)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc00038efb8 sp=0xc00038ef78 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00038efc0 sp=0xc00038efb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1308 [chan send (scan)]:
runtime.gopark(0x17acde0, 0xc000096238, 0xc00007160e, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:301 +0xe6 fp=0xc0003926c0 sp=0xc0003926a0 pc=0x102f5a6
runtime.goparkunlock(...)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/proc.go:307
runtime.chansend(0xc0000961e0, 0xc000392798, 0x1055e01, 0x16230d9, 0x3)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:236 +0x21f fp=0xc000392740 sp=0xc0003926c0 pc=0x100708f
runtime.chansend1(0xc0000961e0, 0xc000392798)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/chan.go:127 +0x35 fp=0xc000392778 sp=0xc000392740 pc=0x1006e65
github.com/influxdata/influxdb/pkg/limiter.Fixed.Take(...)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/pkg/limiter/fixed.go:40
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013c1c0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:406 +0x39 fp=0xc0003927b8 sp=0xc000392778 pc=0x16230d9
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003927c0 sp=0xc0003927b8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113

goroutine 1322 [runnable]:
syscall.syscall(0x1076310, 0x23, 0x33, 0x0, 0x0, 0x0, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/sys_darwin.go:63 +0x2e fp=0xc0002bdc98 sp=0xc0002bdc70 pc=0x104a2ee
syscall.fcntl(0x23, 0x33, 0x0, 0x8, 0xc0002bdd68, 0x113ae9b)
	/Users/mr/gotip/src/github.com/golang/go/src/syscall/zsyscall_darwin_amd64.go:318 +0x58 fp=0xc0002bdce8 sp=0xc0002bdc98 pc=0x1073708
internal/poll.(*FD).Fsync(0xc0000bd200, 0x0, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/internal/poll/fd_fsync_darwin.go:21 +0x97 fp=0xc0002bdd28 sp=0xc0002bdce8 pc=0x1089e57
os.(*File).Sync(0xc0001ee030, 0x0, 0x0)
	/Users/mr/gotip/src/github.com/golang/go/src/os/file_posix.go:133 +0x4a fp=0xc0002bdd78 sp=0xc0002bdd28 pc=0x1091c6a
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Tombstoner).commit(0xc0000e99a0, 0x17acfa8, 0xc0000e99a0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/tombstone.go:379 +0x88 fp=0xc0002bddc8 sp=0xc0002bdd78 pc=0x160cc28
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Tombstoner).Flush(0xc0000e99a0, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/tombstone.go:156 +0x6d fp=0xc0002bde00 sp=0xc0002bddc8 pc=0x160b27d
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*batchDelete).Commit(0xc000012008, 0x0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/reader.go:618 +0x6d fp=0xc0002bde28 sp=0xc0002bde00 pc=0x1600dbd
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).deleteSeriesRange.func1(0x18743a0, 0xc00013cee0, 0xc00013cee0, 0x0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/engine.go:1537 +0x5ba fp=0xc0002bdf78 sp=0xc0002bde28 pc=0x1620a4a
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply.func1(0xc0000961e0, 0xc0000942a0, 0xc000300510, 0x18743a0, 0xc00013cee0)
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:411 +0xc2 fp=0xc0002bdfb8 sp=0xc0002bdf78 pc=0x1623162
runtime.goexit()
	/Users/mr/gotip/src/github.com/golang/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0002bdfc0 sp=0xc0002bdfb8 pc=0x1058f21
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Apply
	/Users/mr/go/pkg/mod/github.com/influxdata/influxdb@v0.0.0-20181220221034-7db3db278bfa/tsdb/engine/tsm1/file_store.go:405 +0x113
exit status 2

@methane
Copy link
Contributor Author

methane commented Dec 26, 2018

I got more interesting information.

runtime: pointer 0xc00659e432 to unused region of span span.base()=0xc004000000 span.limit=0xc004001f80 span.state=1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0xc0f417, 0x3e)
    /home/ubuntu/local/go/src/runtime/panic.go:608 +0x72 fp=0x7ff10e99fd98 sp=0x7ff10e99fd68 pc=0x42dec2
runtime.findObject(0xc00659e432, 0x0, 0x0, 0xc004ab0180, 0x7ff10d5f3e70, 0x1)
    /home/ubuntu/local/go/src/runtime/mbitmap.go:399 +0x3b6 fp=0x7ff10e99fde8 sp=0x7ff10e99fd98 pc=0x415996
runtime.wbBufFlush1(0xc00003a000)
    /home/ubuntu/local/go/src/runtime/mwbbuf.go:264 +0xda fp=0x7ff10e99fe50 sp=0x7ff10e99fde8 pc=0x42a0ca
...

Note that the bad pointer is 0xc00659e432. Then, there is the pointer in full stack trace:

goroutine 29115 [runnable]:
XXX/game/connection.Keep.func1(0xc00659e432)
    XXX/game/connection/connection.go:22 fp=0xc0044a97d8 sp=0xc0044a97d0 pc=0x7f1ff0
runtime.goexit()
    /home/ubuntu/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0044a97e0 sp=0xc0044a97d8 pc=0x45dc51
created by XXX/game/connection.Keep
    XXX/game/connection/connection.go:22 +0x3d

The bad pointer is the argument of Keep.func1(). And it's code like this:

func Keep(playerID int32) {
    go func() {
        err := updateAliveTime(playerID)
        if err != nil {
            logger.Errorf("Failed to keep connection. err=[%v]", err)
        }
    }()
}

Note that playerID is int32, while GOARCH=amd64.
Is this bad pointer created by compiler?

@methane
Copy link
Contributor Author

methane commented Dec 26, 2018

I added quick check in runtime.(*wbBuf).putFast. Bad pointer is written by runtime.mstart().

runtime: bad pinter: 0xc007435e93
fatal error: XXXXXX

runtime stack:
runtime.throw(0xbeaeb3, 0x6)
    /home/ubuntu/local/go/src/runtime/panic.go:608 +0x72
runtime.(*wbBuf).putFast(0xc00003b290, 0xc007435e93, 0xc007435e93, 0x7f4b766b6d88)
    /home/ubuntu/local/go/src/runtime/mwbbuf.go:143 +0x1f8
runtime.bulkBarrierBitmap(0xc0038a6fd8, 0xc0038a6fd8, 0x8, 0x0, 0xc19258)
    /home/ubuntu/local/go/src/runtime/mbitmap.go:682 +0x12d
runtime.newproc1(0xc194f0, 0xc0002c8708, 0x8, 0xc00222e780, 0x7f1edd)
    /home/ubuntu/local/go/src/runtime/proc.go:3373 +0x441
runtime.newproc.func1()
    /home/ubuntu/local/go/src/runtime/proc.go:3309 +0x4f
runtime.systemstack(0x0)
    /home/ubuntu/local/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
    /home/ubuntu/local/go/src/runtime/proc.go:1229

goroutine 66219 [running]:
runtime.systemstack_switch()
    /home/ubuntu/local/go/src/runtime/asm_amd64.s:311 fp=0xc0002c86b0 sp=0xc0002c86a8 pc=0x45bb70
runtime.newproc(0xc000000008, 0xc194f0)
    /home/ubuntu/local/go/src/runtime/proc.go:3308 +0x6e fp=0xc0002c86f8 sp=0xc0002c86b0 pc=0x43750e
XXX/game/connection.Keep(0x7435e93)

@methane
Copy link
Contributor Author

methane commented Dec 26, 2018

@aclements Would you take a look? I'm not sure about Go's GC implementation.

In case of Keep.func1, stkmap.nbit is 0. But stkmap.bytedata[0] is 0xf.
So I think we should test stkmap.nbit>0 before calling bulkBarrierBitmap()

diff --git a/src/runtime/proc.go b/src/runtime/proc.go
index bdf73e0412..67467b2ee5 100644
--- a/src/runtime/proc.go
+++ b/src/runtime/proc.go
@@ -3303,9 +3303,11 @@ func newproc1(fn *funcval, argp *uint8, narg int32, callergp *g, callerpc uintpt
                if writeBarrier.needed && !_g_.m.curg.gcscandone {
                        f := findfunc(fn.fn)
                        stkmap := (*stackmap)(funcdata(f, _FUNCDATA_ArgsPointerMaps))
-                       // We're in the prologue, so it's always stack map index 0.
-                       bv := stackmapdata(stkmap, 0)
-                       bulkBarrierBitmap(spArg, spArg, uintptr(narg), 0, bv.bytedata)
+                       if stkmap.nbit > 0 {
+                               // We're in the prologue, so it's always stack map index 0.
+                               bv := stackmapdata(stkmap, 0)
+                               bulkBarrierBitmap(spArg, spArg, uintptr(narg), 0, bv.bytedata)
+                       }
                }
        }

@gopherbot
Copy link

Change https://golang.org/cl/155779 mentions this issue: runtime: Fix invalid ptr in write barrier buffer

@mknyszek
Copy link
Contributor

@methane yeah that's definitely problematic; non-existent (or "zero-sized") stack maps don't seem to be treated properly. The compiler seems to forgo emitting a stack map because the entrypoint of Keep()'s closure doesn't need one, but the new G code assumes that if nargs>0 then the stack map has at least one valid bit, which isn't true here. Furthermore this stack map points to another stack map which has a non-zero first bit, which doesn't match up with the fact that the first bit would be zero, if a stack map was necessary, so a bad value (some stack value, probably) gets placed into the write barrier buffer.

Nice catch, and thanks for putting up a fix!

This got me worried about this happening elsewhere, but I poked around and it seems OK. The bitvector's n field is also bypassed elsewhere, but there isn't as much of an issue because other checks make sure that valid data exists there (e.g. BSS/data section bitvectors).

Just to be clear, does this fix the issue you've been seeing? There was also a bunch of stuff figured out/solved in #24993 which could result in a similar error but I assume you saw that and the patch there didn't help.

@methane
Copy link
Contributor Author

methane commented Dec 27, 2018

I created small reproducer fot this. (amd64 only)
https://gist.github.com/methane/b61dcfb504d54de5bced1c6e3209a91d

@methane
Copy link
Contributor Author

methane commented Dec 27, 2018

Just to be clear, does this fix the issue you've been seeing? There was also a bunch of stuff figured out/solved in #24993 which could result in a similar error but I assume you saw that and the patch there didn't help.

The original issue was found in our company's load testing environment.
And I spotted this bug by adding assertion to putFast in runtime/mwbbuf.c.
So I'm sure this bug caused our issue.

We will run the load test today. I will share the result if we faced fatal error again.

@FiloSottile FiloSottile modified the milestones: Go1.13, Go1.12 Jan 2, 2019
@FiloSottile FiloSottile added the NeedsFix The path to resolution is known, but the work has not been done. label Jan 2, 2019
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 2, 2019
@FiloSottile
Copy link
Contributor

@aclements @randall77 Do you think golang.org/cl/155779 fixes #26243 as well?

@methane
Copy link
Contributor Author

methane commented Jan 3, 2019

I'm not sure, but I don't think so. CL-155779 fixes non-pointer is put into write barrier buffer.
Stacktrace in #26243 is not relating to write barrier buffer.

@randall77
Copy link
Contributor

@FiloSottile I don't think it's related to #26243, unfortunately.
That found a bad pointer in a heap object. This one is a bad pointer found on the stack.

@gopherbot
Copy link

Change https://golang.org/cl/156122 mentions this issue: runtime: add test for go function argument scanning

@gopherbot
Copy link

Change https://golang.org/cl/156123 mentions this issue: runtime: don't scan go'd function args past length of ptr bitmap

gopherbot pushed a commit that referenced this issue Jan 3, 2019
Derived	from Naoki's reproducer.

Update #29362

Change-Id: I1cbd33b38a2f74905dbc22c5ecbad4a87a24bdd1
Reviewed-on: https://go-review.googlesource.com/c/156122
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Jan 3, 2019
Use the length of the bitmap to decide how much to pass to the
write barrier, not the total length of the arguments.

The test needs enough arguments so that two distinct bitmaps
get interpreted as a single longer bitmap.

Update #29362

Change-Id: I78f3f7f9ec89c2ad4678f0c52d3d3def9cac8e72
Reviewed-on: https://go-review.googlesource.com/c/156123
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@methane
Copy link
Contributor Author

methane commented Jan 4, 2019

Would you backport this to Go 1.11?

@randall77
Copy link
Contributor

Yes, we should backport these.

@gopherbot please open an issue for backporting to 1.11.

@gopherbot
Copy link

Backport issue(s) opened: #29565 (for 1.11).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@randall77
Copy link
Contributor

@gopherbot please open an issue for backporting to 1.10.

@FiloSottile
Copy link
Contributor

@randall77 You can ask gopherbot to open backport issues for both releases at the same time, but in general if "compile with 1.11" is a valid workaround, we are not backporting to 1.10 anymore. (Second paragraph of https://github.com/golang/go/wiki/MinorReleases.)

@josharian
Copy link
Contributor

Apologies for my limited imagination, but under what scenarios would “compile with 1.11” not be a sufficient workaround?

@FiloSottile
Copy link
Contributor

I suppose if 1.11 dropped support for a relevant OS, but in practice I suspect the new policy is equivalent to "only security fixes are backported to two releases ago".

@golang golang locked and limited conversation to collaborators Jan 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
None yet
Development

No branches or pull requests

10 participants