In this weekends we came across a very unpleasant bug. The our server application is freezed after about 50min of works.
Using tool pprof/goroutine we did investigation and found that all 4 workers hang in the database/sql package:

ROUTINE ======================== database/sql.(*DB).conn in /usr/local/go/src/database/sql/sql.go
         0         20 (flat, cum) 0.0045% of Total
         .          .    965:           reqKey := db.nextRequestKeyLocked()
         .          .    966:           db.connRequests[reqKey] = req
         .          .    967:           db.mu.Unlock()
         .          .    968:
         .          .    969:           // Timeout the connection request with the context.
         .         20    970:           select {
         .          .    971:           case <-ctx.Done():
         .          .    972:                   // Remove the connection request and ensure no value has been sent
         .          .    973:                   // on it after removing.
         .          .    974:                   db.mu.Lock()
         .          .    975:                   delete(db.connRequests, reqKey)

What is most striking that until today's weekend the our server application for 2 past years worked stably. And hasn't meet such kind of bug.

Tool like netstat didn't see any network activity with mysql.

Mysql driver is here:
https://github.com/go-sql-driver/mysql

I know the sufficiency of information is poor and the bug is probably just not to repeat so easily.
But can someone tell what is going on in this part of that package. I think this will help me to find clue.

What version of Go are you using (go version)?

go version go1.9.2 linux/amd64

Does this issue reproduce with the latest release?

I don't know.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/dev-pon/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build053765279=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you expect to see?

Processing query or return error.

What did you see instead?

After 50 minutes it is impossible to make queries to the database and the whole goroutine freeze.