Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

database/sql: transaction leak #22976

Closed
kirk91 opened this issue Dec 3, 2017 · 7 comments
Closed

database/sql: transaction leak #22976

kirk91 opened this issue Dec 3, 2017 · 7 comments
Labels
CherryPickApproved Used during the release process for point releases FrozenDueToAge
Milestone

Comments

@kirk91
Copy link
Contributor

kirk91 commented Dec 3, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.9.2 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/kirk/golang"
GORACE=""
GOROOT="/usr/local/opt/go/libexec"
GOTOOLDIR="/usr/local/opt/go/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/j7/yr_n5r6j5jx4jb856mkk7kzr0000gp/T/go-build290212909=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

When context is canceled or timeout, the db transaction is leaked which cannot be commit or rollback in some extreme scenario. It may cause massive lock wait timeouts and high load on the underlying database.

The following describes the basic flow:

Goroutine 1

Call (*DB) BeginTx begins a transaction with a userContext. In (*DB)BeginTx, a new goroutine (*Tx)awaitDone which monitor context and rollback tx if needed will be created.

Goroutine 2(awaitDone)

block on tx.ctx.Done()

Goroutine 1

Execute some insert or update sqls on the database

Goroutine 1

Commit the transaction, (*Tx)Commit set the atomic variable tx.done to 1.

Goroutine 3(maybe global timer)

Cancel userContext which be passed to Tx

Goroutine 1

(*Tx)Commit checks tx.ctx.Done(). Due to context has been canceled, it will return context.Canceled or context.DeadlineExceeded error immediately and abort the real COMMIT of transaction.

Goroutine 2

Release with tx.ctx.Done() signal, execute (*Tx)rollback. However the atomic variable tx.done is 1 currently, it will return ErrTxDone error immediately and abort the real ROLLBACK of transaction.

What did you expect to see?

Rollback the transaction

What did you see instead?

The transaction hangs up and massive db lock wait timeouts

@gopherbot
Copy link

Change https://golang.org/cl/81736 mentions this issue: database/sql: fix transaction leak

@posener
Copy link

posener commented Jan 6, 2018

Hi, will this fix be in go 1.9.3?

@ianlancetaylor ianlancetaylor added this to the Go1.9.3 milestone Jan 7, 2018
@ianlancetaylor
Copy link
Contributor

Reopening to consider for 1.9.3.

@ianlancetaylor ianlancetaylor reopened this Jan 7, 2018
@ianlancetaylor
Copy link
Contributor

(But note that there may never be a 1.9.3 release, as 1.10 is expected in a few weeks.)

@andybons
Copy link
Member

CL 81736 OK for Go 1.9.3.

@andybons andybons added the CherryPickApproved Used during the release process for point releases label Jan 18, 2018
@gopherbot
Copy link

Change https://golang.org/cl/88323 mentions this issue: [release-branch.go1.9] database/sql: fix transaction leak

gopherbot pushed a commit that referenced this issue Jan 22, 2018
When the user context which passed in (*DB)BeginTx is canceled or
timeout, the current implementation could cause db transaction leak
in some extreme scenario.

Goroutine 1:
        Call (*DB) BeginTx begins a transaction with a userContext.
        In (*DB)BeginTx, a new goroutine (*Tx)awaitDone
        which monitor context and rollback tx if needed will be created

Goroutine 2(awaitDone):
        block on tx.ctx.Done()

Goroutine 1:
        Execute some insert or update sqls on the database

Goroutine 1:
        Commit the transaction, (*Tx)Commit set
        the atomic variable tx.done to 1

Goroutine 3(maybe global timer):
        Cancel userContext which be passed in Tx

Goroutine 1:
        (*Tx)Commit checks tx.ctx.Done().
        Due to the context has been canceled, it will return
        context.Canceled or context.DeadlineExceeded error immediately
        and abort the real COMMIT operation of transaction

Goroutine 2:
        Release with tx.ctx.Done() signal, execute (*Tx)rollback.
        However the atomic variable tx.done is 1 currently,
        it will return ErrTxDone error immediately and
        abort the real ROLLBACK operation of transaction

Fixes #22976

Change-Id: I3bc23adf25db823861d91e33d3cca6189fb1171d
Reviewed-on: https://go-review.googlesource.com/81736
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Daniel Theophanes <kardianos@gmail.com>
Reviewed-on: https://go-review.googlesource.com/88323
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@andybons
Copy link
Member

go1.9.3 has been packaged and includes:

  • CL 81736 database/sql: fix transaction leak

The release is posted at golang.org/dl.

— golang.org/x/build/cmd/releasebot, Jan 22 21:02:58 UTC

paco0x added a commit to paco0x/beego that referenced this issue Jul 23, 2018
upgrade to avoid bug: golang/go#22976

Signed-off-by: Penghui Liao <liaoishere@gmail.com>
@golang golang locked and limited conversation to collaborators Jan 22, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CherryPickApproved Used during the release process for point releases FrozenDueToAge
Projects
None yet
Development

No branches or pull requests

6 participants