Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/sync/singleflight: hangs after the first panic #41133

Closed
kirk91 opened this issue Aug 30, 2020 · 4 comments
Closed

x/sync/singleflight: hangs after the first panic #41133

kirk91 opened this issue Aug 30, 2020 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Unreleased

Comments

@kirk91
Copy link
Contributor

kirk91 commented Aug 30, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/kirk/Library/Caches/go-build"
GOENV="/Users/kirk/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="*.byted.org,*.everphoto.cn,git.smartisan.com"
GONOSUMDB="*.byted.org,*.everphoto.cn,git.smartisan.com"
GOOS="darwin"
GOPATH="/Users/kirk/go"
GOPRIVATE="*.byted.org,*.everphoto.cn,git.smartisan.com"
GOPROXY="https://go-mod-proxy.byted.org,https://goproxy.cn,https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.2_1/libexec"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.2_1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3x/blq4wl4x6q512nmyr651qt_w0000gp/T/go-build088646219=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I use singleflight to do something like creating a singleton client, all subsequent calls hang after the first panic.

Here is the code:

package main

import (
	"flag"
	"fmt"
	"log"
	"net"
	"net/http"
	_ "net/http/pprof"

	"golang.org/x/sync/singleflight"
)

var (
	port = flag.Int("port", 8000, "listening port")
)

func main() {
	flag.Parse()
	addr := fmt.Sprintf("127.0.0.1:%d", *port)
	lis, err := net.Listen("tcp", addr)
	if err != nil {
		panic(err)
	}

	http.HandleFunc("/ping", handlePing)
	log.Println("Server is listening on:", addr)
	http.Serve(lis, nil)
}

var doSthGroup singleflight.Group

func handlePing(w http.ResponseWriter, r *http.Request) {
	doSthGroup.Do("", func() (interface{}, error) {
		// simulating panic
		panic("invalid memory address or nil pointer dereference")
	})

	w.WriteHeader(http.StatusOK)
	fmt.Fprintf(w, "pong")
}

Using curl do tests after starting the above program, all requests are timeout expect the first.

$ # timeout is one second
$ for (( i=0; i<20; i++)); do curl -m 1 http://localhost:8000/ping; done
curl: (52) Empty reply from server
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1002 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1003 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1003 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1002 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1002 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1003 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1000 milliseconds with 0 bytes received
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received

And cloud sees there are 19 goroutines get stuck in singleflight.(*Group).Do through goroutine stack traces

$  curl "http://localhost:8000/debug/pprof/goroutine?debug=1"
goroutine profile: total 23
19 @ 0x10380a0 0x1048a40 0x1048a2b 0x1048692 0x1081474 0x1352aea 0x13536c5 0x12c3b04 0x12c5945 0x12c6e93 0x12c290c 0x1067431
#       0x1048691       sync.runtime_Semacquire+0x41                    /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/sema.go:56
#       0x1081473       sync.(*WaitGroup).Wait+0x63                     /usr/local/Cellar/go/1.14.2_1/libexec/src/sync/waitgroup.go:130
#       0x1352ae9       golang.org/x/sync/singleflight.(*Group).Do+0xa9 /Users/kirk/go/src/golang.org/x/sync/singleflight/singleflight.go:59
#       0x13536c4       main.handlePing+0x44                            /Users/kirk/sg_hang.go:34
#       0x12c3b03       net/http.HandlerFunc.ServeHTTP+0x43             /usr/local/Cellar/go/1.14.2_1/libexec/src/net/http/server.go:2012
#       0x12c5944       net/http.(*ServeMux).ServeHTTP+0x1a4            /usr/local/Cellar/go/1.14.2_1/libexec/src/net/http/server.go:2387
#       0x12c6e92       net/http.serverHandler.ServeHTTP+0xa2           /usr/local/Cellar/go/1.14.2_1/libexec/src/net/http/server.go:2807
#       0x12c290b       net/http.(*conn).serve+0x86b                    /usr/local/Cellar/go/1.14.2_1/libexec/src/net/http/server.go:1895

What did you expect to see?

All requests fail immediately

What did you see instead?

All requests are timeout expect the first
@gopherbot gopherbot added this to the Unreleased milestone Aug 30, 2020
@gopherbot
Copy link

Change https://golang.org/cl/251677 mentions this issue: singleflight: fix hangs after first Do panic

@dmitshur dmitshur changed the title x/sync: singleflight hangs after the first panic x/sync/singleflight: hangs after the first panic Sep 1, 2020
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 1, 2020
@dmitshur
Copy link
Contributor

dmitshur commented Sep 1, 2020

Thanks for the report. It looks like it might be a duplicate of #33519. Is that your understanding too?

@dmitshur dmitshur added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 1, 2020
@kirk91
Copy link
Contributor Author

kirk91 commented Sep 1, 2020

@dmitshur It should be, I am sorry missing it when raising the issue

@dmitshur
Copy link
Contributor

dmitshur commented Sep 4, 2020

Thanks for confirming—that is quite okay. I'll close this in favor of that issue.

@dmitshur dmitshur closed this as completed Sep 4, 2020
@dmitshur dmitshur mentioned this issue Sep 4, 2020
Closed
gopherbot pushed a commit to golang/sync that referenced this issue Sep 30, 2020
@kirk91
singleflight: fix hangs after first Do panic
3042136 
When first Do panic, the related wait group will never be done,
and all the subsequent calls would block on the same wait group forever.

Fixes golang/go#41133

Change-Id: I0ad9bfb387b6133b10766a34fc0040f200eae27e
Reviewed-on: https://go-review.googlesource.com/c/sync/+/251677
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Trust: Ian Lance Taylor <iant@golang.org>
Trust: Bryan C. Mills <bcmills@google.com>
sherifabdlnaby pushed a commit to sherifabdlnaby/sync that referenced this issue Feb 24, 2021
@kirk91 @sherifabdlnaby
singleflight: fix hangs after first Do panic
656f3e7 
When first Do panic, the related wait group will never be done,
and all the subsequent calls would block on the same wait group forever.

Fixes golang/go#41133

Change-Id: I0ad9bfb387b6133b10766a34fc0040f200eae27e
Reviewed-on: https://go-review.googlesource.com/c/sync/+/251677
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Trust: Ian Lance Taylor <iant@golang.org>
Trust: Bryan C. Mills <bcmills@google.com>
@golang golang locked and limited conversation to collaborators Sep 4, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@dmitshur @kirk91 @gopherbot