runtime/cgo: pthread_create failed: Resource temporarily unavailable #37006

nadermx · 2020-02-03T21:56:00Z

What version of Go are you using (`go version`)?

$ go version 1.13

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

$ go env linux amd64

What did you do?

I currently have this function, that pipes multiple youtubedl commands thru ffmpeg, and then pipes the output of ffmpeg to an HTTP client.

func pipeThruFfmpegToMp4(vi *VideoInfo, rw web.ResponseWriter) error {
	var ffmpeg *exec.Cmd

	ffmpeg = exec.Command(
		"ffmpeg",
		"-i", "-",
		"-i", "pipe:3",
		"-c:v", "copy", "-c:a", "copy",
		"-preset", "veryfast",
		"-metadata", fmt.Sprintf(`title=%s`, vi.GetTitle()),
		"-movflags", "frag_keyframe+empty_moov",
		"-f", "mp4",
		"-")


	youtubevideo := exec.Command(YoutubeDLPath,  "-c", "-f", fmt.Sprintf("%s/bestvideo[ext=mp4]/bestvideo/best", vi.GetFormat()), "--no-cache-dir", "--restrict-filenames", "--hls-prefer-native", "-o", "-", fmt.Sprintf("%s", vi.GetVideoUrl()))

	youtube := exec.Command(YoutubeDLPath, "-c", "-f", "bestaudio[ext=m4a]/bestaudio/best", "--no-cache-dir", "--restrict-filenames", "--hls-prefer-native", "-o", "-", fmt.Sprintf("%s", vi.GetVideoUrl()))

	video, err := youtubevideo.StdoutPipe()
	if err != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", err)
		return err
	}

	pipe3, err := youtube.StdoutPipe()
	if err != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", err)
		return err
	}
	youtubestderr, youtubestderrErr := youtube.StderrPipe()
	if youtubestderrErr != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", youtubestderrErr)
		return youtubestderrErr
	}

	youtubevideostderr, youtubevideostderrErr := youtubevideo.StderrPipe()
	if youtubevideostderrErr != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", youtubevideostderrErr)
		return youtubevideostderrErr
	}

	if err := youtubevideo.Start(); err != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", err)
		return err
	}
	defer func() {
		if err != nil {
			youtubevideo.Process.Kill()
			youtubevideo.Wait()
		}
	}()

	if err := youtube.Start(); err != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", err)
		return err
	}
	defer func() {
		if err != nil {
			youtube.Process.Kill()
			youtube.Wait()
		}
	}()

	ffmpeg.Stdin = video
	ffmpeg.ExtraFiles = []*os.File{pipe3.(*os.File)}
	defer pipe3.Close()
	fmt.Println(ffmpeg)
	stdout, stdoutErr := ffmpeg.StdoutPipe()
	if stdoutErr != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", stdoutErr)
		return stdoutErr
	}

	stderr, stderrErr := ffmpeg.StderrPipe()
	if stderrErr != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", stderrErr)
		return stderrErr
	}

	if err := ffmpeg.Start(); err != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", err)
		return err
	}
    defer func() {
		if err != nil {
			ffmpeg.Process.Kill()
			ffmpeg.Wait()
		}
	}()

	// Headers sent, no turning back now
	rw.Header().Set("Content-Type", "video/mp4")
	rw.Header().Set("Content-Disposition", fmt.Sprintf("attachment;filename=\"%s.mp4\"", vi.GetSlug()))
	rw.Flush()

	var wg sync.WaitGroup
	wg.Add(2)

	go func() {
		if _, err := io.Copy(rw, stdout); err != nil {
			if !WritePipeBroken.MatchString(err.Error()) &&
				!ConnectionResetByPeer.MatchString(err.Error()) {
				log.Printf("pipeThruFfmpegToMp4: %v\n", err)
			}
			log.Printf("pipeThruFfmpegToMp4: %v\n", err)
			if err := ffmpeg.Process.Kill(); err != nil {
				log.Printf("pipeThruFfmpegToMp4: %v\n", err)
			}
		}
		rw.Flush()
		wg.Done()
	}()

	var errContents []byte
	go func() {
		var err error
		errContents, err = ioutil.ReadAll(stderr)
		if err != nil {
			log.Printf("pipeThruFfmpegToMp4: %v\n", err)
			if err := ffmpeg.Process.Kill(); err != nil {
				log.Printf("pipeThruFfmpegToMp4: %v\n", err)
			}
		}
		wg.Done()
	}()

	wg.Wait()
	if err := ffmpeg.Wait(); err != nil {
		log.Printf("pipeThruFfmpegToMp4: %v\n", err)
		log.Printf("pipeThruFfmpegToMp4: %v\n", string(errContents))
	}
	if err := youtubevideo.Wait(); err != nil {
		data, _ := ioutil.ReadAll(youtubevideostderr)
		log.Printf("pipeThruFfmpegToMp4: %v\n", err)
		log.Printf("pipeThruFfmpegToMp4: %v\n", string(data))
	}
	if err := youtube.Wait(); err != nil {
		data, _ := ioutil.ReadAll(youtubestderr)
		log.Printf("pipeThruFfmpegToMp4: %v\n", err)
		log.Printf("pipeThruFfmpegToMp4: %v\n", string(data))
	}
	return nil

}

The problem is that everything runs fine, but after a while the ram starts filling up on the server, till it gets to the point that it errors out with runtime/cgo: pthread_create failed: Resource temporarily unavailable

I'm unsure if it's a memory leak, or if either instance of youtube-dl is not closing right or if ffmpeg isn't closing right and just consuming more and more ram as the program runs more, until the program crashes with this error

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f083501fe97 m=128 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f083501fe97
stack: frame={sp:0x7f05ff7fd7d0, fp:0x0} stack=[0x7f05feffe288,0x7f05ff7fde88)
00007f05ff7fd6d0:  00007f05ff7fd700  0000ffff00001fa0
00007f05ff7fd6e0:  00007f05ff7fdbe0  00007f05f8000da0
00007f05ff7fd6f0:  0000000000000000  000000000093032c
00007f05ff7fd700:  0000000000000000  00007f0835600ec3
00007f05ff7fd710:  0000000000000005  0000000000000000
00007f05ff7fd720:  000000c0000ce120  00007f0834ff1ce0
00007f05ff7fd730:  00007f05ff7fdaf0  00007f083560870a
00007f05ff7fd740:  0000000000000000  0000000000000000
00007f05ff7fd750:  0000000000000000  00007f05ff7fdbe0
00007f05ff7fd760:  2525252525252525  2525252525252525
00007f05ff7fd770:  000000ffffffffff  0000000000000000
00007f05ff7fd780:  000000ffffffffff  0000000000000000
00007f05ff7fd790:  000000c00010d1a0  000000c000953740
00007f05ff7fd7a0:  000000c0000ce120  000000c000cf2300
00007f05ff7fd7b0:  000000c00010d260  000000c001f4e180
00007f05ff7fd7c0:  000000c001f4e000  000000c00169f680
00007f05ff7fd7d0: <0000000000000000  000000c001f34180
00007f05ff7fd7e0:  6e75720000000000  6f67632f656d6974
00007f05ff7fd7f0:  0000000000000000  0000000000000000
00007f05ff7fd800:  000000c000cf2300  000000c00010d260
00007f05ff7fd810:  000000c001f4e180  000000c001f4e000
00007f05ff7fd820:  000000c00169f680  000000c00169f500
00007f05ff7fd830:  000000c001f34180  000000c001f34000
00007f05ff7fd840:  000000c000c92780  000000c001ec2600
00007f05ff7fd850:  fffffffe7fffffff  ffffffffffffffff
00007f05ff7fd860:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd870:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd880:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd890:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd8a0:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd8b0:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd8c0:  ffffffffffffffff  ffffffffffffffff
runtime: unknown pc 0x7f083501fe97
stack: frame={sp:0x7f05ff7fd7d0, fp:0x0} stack=[0x7f05feffe288,0x7f05ff7fde88)
00007f05ff7fd6d0:  00007f05ff7fd700  0000ffff00001fa0
00007f05ff7fd6e0:  00007f05ff7fdbe0  00007f05f8000da0
00007f05ff7fd6f0:  0000000000000000  000000000093032c
00007f05ff7fd700:  0000000000000000  00007f0835600ec3
00007f05ff7fd710:  0000000000000005  0000000000000000

I've also tried building the binary with CGO_ENABLED=0 even though I don't even have a import "c" but that also ends up erroring out as well with

runtime: failed to create new OS thread (have 21 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

my limit's are already very generouse, which maybe could also be a problem? Or do I maybe have to increase the pipe size?

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 192907
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 65536
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63883
virtual memory          (kbytes, -v) unlimited

Either way, I apprecate any and all the help I can get on this.

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2020-02-04T01:49:29Z

You will most likely get better and faster answers on a forum, rather than on this issue tracker. See https://golang.org/wiki/Questions.

The error about pthread_create failed: Resource temporarily unavailable normally means that you are trying to create too many threads. It can occasionally mean that you are allocating too much, memory, but that is much less likely. Note that number of permitted threads is across all your programs, not just one.

nadermx · 2020-02-04T06:51:23Z

I went ahead and posted it on StackOverflow with a 200 point bounty. But this happens regardless of the server size, 20 core box or 40 core box with 50+ gig's of ram, with even a low number of requests and time. So I think it's more of a leak somewhere or something hanging.

networkimprov · 2020-02-04T07:11:23Z

It would help if you could post a complete program in your Q.

I haven't found the quality of Go responses on SO to be that good, tho I haven't looked lately. You might at least post a link to that Q on golang-nuts.

nadermx · 2020-02-04T07:56:32Z

Thank you, just posted it there, and at the end of the SO question I posted a gist to the entire program.

nadermx closed this as completed Feb 4, 2020

golang locked and limited conversation to collaborators Feb 3, 2021

gopherbot added the FrozenDueToAge label Feb 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime/cgo: pthread_create failed: Resource temporarily unavailable #37006

runtime/cgo: pthread_create failed: Resource temporarily unavailable #37006

nadermx commented Feb 3, 2020

ianlancetaylor commented Feb 4, 2020

nadermx commented Feb 4, 2020

networkimprov commented Feb 4, 2020

nadermx commented Feb 4, 2020

runtime/cgo: pthread_create failed: Resource temporarily unavailable #37006

runtime/cgo: pthread_create failed: Resource temporarily unavailable #37006

Comments

nadermx commented Feb 3, 2020

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

ianlancetaylor commented Feb 4, 2020

nadermx commented Feb 4, 2020

networkimprov commented Feb 4, 2020

nadermx commented Feb 4, 2020

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?