Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime/cgo: deadlock #24608

Closed
YijinLiu opened this issue Mar 30, 2018 · 5 comments
Closed

runtime/cgo: deadlock #24608

YijinLiu opened this issue Mar 30, 2018 · 5 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@YijinLiu
Copy link

YijinLiu commented Mar 30, 2018 •
edited by ALTree

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.9.2 and 1.10

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

Calling Go from native code may cause deadlock if it holds some mutex that is needed by another CGO call. The following code calls CGO to increase a native counter, which is protected by a mutex.
if you pass "1" to C.inc_counter, it'll call Go to print the counter.

////////////////////////////// main.go //////////////////////////

package main
import (
    "flag"
    "fmt"
    "time"
)
// #include <inttypes.h>
// extern void inc_counter(int print);
// #cgo CXXFLAGS: -g -O2 -pthread -std=gnu++11
import "C"
var logCounterIntervalFlag = flag.Duration("log-counter-interval", time.Second, "")
func logCounter() {
    go C.inc_counter(1)
    time.AfterFunc(*logCounterIntervalFlag, logCounter)
}
//export printCounter
func printCounter(counter C.uint64_t) {
    fmt.Printf("Counter=%d", counter)
}
func main() {
    flag.Parse()
    go logCounter()
    for {
        go C.inc_counter(0)
    }
}

//////////////////////////////////// inc_counter ////////////////////////////////////

#include <inttypes.h>
#include <mutex>
extern "C" {
extern void printCounter(uint64_t counter);
uint64_t counter = 0;
std::mutex mu;
void inc_counter(int print) {
    std::unique_lock<std::mutex> lock(mu);
    ++counter;
    if (print) printCounter(counter);
}
}  // extern "C"

What did you expect to see?

The program should run till I interrupt it.

What did you see instead?

It's killed by the system because it uses too many threads.
The printCounter Go call is stuck here (go1.9.2):

#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:439
#1  0x0000000000a2fc72 in runtime.futexsleep (addr=0xc4251da810, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2  0x0000000000a1644b in runtime.notesleep (n=0xc4251da810) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000a3857c in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:1827
#4  0x0000000000a39caf in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2213
#5  0x0000000000a39e26 in runtime.park_m (gp=0xc426d87080) at /usr/local/go/src/runtime/proc.go:2318
#6  0x0000000000a60dbb in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:286
#7  0x00007f31ba40ec88 in ?? ()
#8  0x00000008110aa0a8 in ?? ()
#9  0x00007f31ba40e868 in ?? ()
#10 0x0000000000a626be in runtime.cgocallback () at /usr/local/go/src/runtime/asm_amd64.s:673
@AlexRouSg
Copy link
Contributor

cc @ianlancetaylor

@cznic
Copy link
Contributor

cznic commented Mar 30, 2018

Where's the mutex unlock?

@AlexRouSg
Copy link
Contributor

@cznic
He is using a http://www.cplusplus.com/reference/mutex/unique_lock/
Which unlocks on function return

@AlexRouSg
Copy link
Contributor

After root causing this, I've determined that the for loop is simply creating too many os threads too quickly due to cgo calls. So I believe this is user error? @cznic

Simplified repo below:

package main

import (
	"runtime"
	"time"
)

func foo() {
	runtime.LockOSThread()
	time.Sleep(time.Millisecond * 100)
}

func main() {
	for {
		go foo()
	}
}

@FiloSottile FiloSottile changed the title Cgo deadlock runtime/cgo: deadlock Mar 30, 2018
@FiloSottile FiloSottile added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 30, 2018
@bcmills
Copy link
Contributor

bcmills commented Apr 2, 2018

@YijinLiu, it does look like your example program is creating goroutines faster than it can retire them.

Do you have a more realistic example that properly bounds the number of goroutines in flight? Otherwise, I don't think there's anything for us to fix here.

@bcmills bcmills closed this as completed Apr 2, 2018
@golang golang locked and limited conversation to collaborators Apr 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@cznic @FiloSottile @bcmills @AlexRouSg @gopherbot @YijinLiu