Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

os/exec: "fork/exec ... cannot allocate memory" on linux,!amd64 #31936

Closed
4a6f656c opened this issue May 9, 2019 · 11 comments
Closed

os/exec: "fork/exec ... cannot allocate memory" on linux,!amd64 #31936

4a6f656c opened this issue May 9, 2019 · 11 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Backlog

Comments

@4a6f656c
Copy link
Contributor

4a6f656c commented May 9, 2019

What version of Go are you using (go version)?

$ go version
go version devel +5286b2a Thu May 9 14:58:03 2019 +0000 linux/s390x

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

linux/s390x

What did you do?

On a linux/s390x LPAR with 12GB of RAM:

package main

import (
        "log"
        "os/exec"
)

func main() {
        c := 8 * 1024 * 1024 * 1024
        a := make([]byte, c, c)
        for i := range a {
                a[i] = byte(i % 255)
        }

        log.Print(exec.Command("sleep", "1").Run())
}
$ go build -o fork fork.go

What did you expect to see?

$ ./fork
2019/05/07 04:21:46 <nil>

What did you see instead?

$ ./fork
2019/05/07 03:59:54 fork/exec /bin/sleep: cannot allocate memory

This causes a production impact since various Go programs cannot fork/exec (including the Go compiler at times).

This is reproducable under Linux on most architectures aside from linux/amd64, where the problem has been addressed via issue #5838. The memory consumption will need to be altered depending on the available host memory.
@bradfitz bradfitz added the NeedsFix The path to resolution is known, but the work has not been done. label May 9, 2019
@bradfitz bradfitz added this to the Go1.14 milestone May 9, 2019
@gopherbot
Copy link

Change https://golang.org/cl/175697 mentions this issue: syscall: implement rawVforkSyscall for remaining linux platforms

gopherbot pushed a commit that referenced this issue May 15, 2019
@4a6f656c
syscall: implement rawVforkSyscall for linux/ppc64x and linux/s390x
ab242dc 
This allows the use of CLONE_VFORK and CLONE_VM for fork/exec, preventing
"fork/exec ...: cannot allocate memory" failures from occuring when attempting
to execute commands from a Go process that has a large memory footprint.
Additionally, this should reduce the latency of fork/exec on these platforms.

The same problem was addressed on linux/amd64 via issue #5838.

Updates #31936

Change-Id: I7ae0fbbeaa29cab944a49a11272a380d497eb2d0
Reviewed-on: https://go-review.googlesource.com/c/go/+/175697
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/189418 mentions this issue: syscall: implement rawVforkSyscall for linux/arm64

gopherbot pushed a commit that referenced this issue Sep 11, 2019
@zhlhahaha @tklauser
syscall: implement rawVforkSyscall for linux/arm64
b25ec50 
This allows the use of CLONE_VFORK and CLONE_VM for fork/exec, preventing
"fork/exec ...: cannot allocate memory" failures from occuring when attempting
to execute commands from a Go process that has a large memory footprint.
Additionally, this should reduce the latency of fork/exec on linux/arm64.

With CLONE_VM the child process shares the same memory with the parent
process. On its own this would lead to conflicting use of the same
memory, so CLONE_VFORK is used to suspend the parent process until the
child releases the memory when switching to the new program binary
via the exec syscall. When the parent process continues to run, one
has to consider the changes to memory that the child process did,
namely the return address of the syscall function needs to be restored
from a register.

exec.Command() callers can start in a faster manner, as child process who
do exec commands job can be cloned faster via vfork than via fork on arm64.

The same problem was addressed on linux/amd64 via issue #5838.

Updates #31936
Contributed by Howard Zhang <howard.zhang@arm.com> and Bin Lu <bin.lu@arm.com>

Change-Id: Ia99d81d877f564ec60d19f17e596276836576eaf
Reviewed-on: https://go-review.googlesource.com/c/go/+/189418
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@acls
Copy link

acls commented Jan 15, 2020

It would be nice to also have this fixed in linux/arm. There was a fix in https://go-review.googlesource.com/c/go/+/175697/5 but was removed.

@albertoangelici
Copy link

Still have the problem with linux/mipsle (compiled with go 1.13.4 windows/amd64)

@gopherbot
Copy link

Change https://golang.org/cl/234960 mentions this issue: syscall: support rawVforkSyscall on linux/riscv64

gopherbot pushed a commit that referenced this issue Aug 16, 2020
@4a6f656c
syscall: support rawVforkSyscall on linux/riscv64
d303630 
Updates #31936

Change-Id: I7dcb8987d4c306ccc97704b9c1b12313ba8bf242
Reviewed-on: https://go-review.googlesource.com/c/go/+/234960
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@csystem-it
Copy link

this issue is impacting us pretty hard even on go 1.15 on linux/mipsle.
Is there any workaround? Adding swap space (with zram) helps a bit, but it still happens.

@gopherbot
Copy link

Change https://golang.org/cl/295849 mentions this issue: syscall: implement rawVforkSyscall for remaining linux platforms

@4a6f656c
Copy link
Contributor Author

@csystem-it - there is no real work around for this issue, short of either fixing the the syscall package, or rearchitecting your application. Feel free to try https://go-review.googlesource.com/c/go/+/295849 (the mipsle portion of that should be easy to backport to Go 1.15).

@csystem-it
Copy link

I tried to backport and it seems to be working.
Any idea when we can see this in a stable release?
I see that it is merged, but I'm not familiar with the Go development cycle, so I don't know if it will make it for a 1.16.x release, 1.17 or if it has not been decied.

@ianlancetaylor
Copy link
Contributor

@csystem-it The change in https://golang.org/cl/295849 will be in the upcoming Go 1.17 release. It was not in the Go 1.16 release.

1 similar comment
@ianlancetaylor
Copy link
Contributor

@csystem-it The change in https://golang.org/cl/295849 will be in the upcoming Go 1.17 release. It was not in the Go 1.16 release.

@golang golang locked and limited conversation to collaborators May 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
8 participants
@bradfitz @rsc @acls @ianlancetaylor @4a6f656c @gopherbot @albertoangelici @csystem-it