runtime: macOS syscall.Exec can get SIGILL due to preemption signal #41702

ianlancetaylor · 2020-09-30T04:49:15Z

On macOS 10.14 and 10.15 (at least) a Go program calling syscall.Exec can fail due to SIGILL in the newly execed image because of preemption signals.

This is a bug in macOS. Here is a C program that demonstrates the problem. This program will occasionally fail with SIGILL.

For Go, we should have a workaround, which means in syscall.Exec we should disable preemption via signal. We already have hooks that we can use for this (syscall.runtime_BeforeExec and syscall.runtime_AfterExec).

#include <sys/errno.h>
#include <assert.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
 
extern char **environ;
 
static pthread_t sleep_thread;
 
void sighandler(int signal) {}
 
void* signal_thread_main(void* unused_args) {
  while (1) {
    int ret = pthread_kill(sleep_thread, SIGURG);
    assert(ret == 0);
  }
  return NULL;
}
 
void* exec_thread_main(void* unused_args) {
  char* exec_args[] = {"/usr/bin/python", "--version", NULL};
  int ret;
 
  ret = execve(exec_args[0], exec_args, environ);
  assert(ret == 0);
  return NULL;
}
 
void* sleep_thread_main(void* unused_args) {
  unsigned int to_sleep = 60;
  while (to_sleep > 0) {
    to_sleep = sleep(to_sleep);
  }
  return NULL;
}
 
int main(int argc, char* argv[]) {
  pthread_t signal_thread, exec_thread;
  int ret;
 
  signal(SIGURG, &sighandler);
 
  ret = pthread_create(&sleep_thread, NULL, &sleep_thread_main, NULL);
  assert(ret == 0);
 
  ret = pthread_create(&signal_thread, NULL, &signal_thread_main, NULL);
  assert(ret == 0);
 
  sleep(1);  // Give signalling time to start                             
  ret = pthread_create(&exec_thread, NULL, &exec_thread_main, NULL);
  assert(ret == 0);
 
  sleep(30);
}

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2020-09-30T04:50:32Z

@gopherbot Please open backport issues

This issue can cause any Go program that uses syscall.Exec on macOS to crash during the exec. We should backport the fix (when we have it) so that this will work for older releases.

gopherbot · 2020-09-30T04:50:44Z

Backport issue(s) opened: #41703 (for 1.14), #41704 (for 1.15).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

ianlancetaylor · 2020-10-08T18:47:49Z

FYI, a bug was reportedly filed with Apple at https://feedbackassistant.apple.com/feedback/8759414 (but I can't see that report because I don't have an Apple ID).

randall77 · 2020-10-08T19:08:14Z

I have an Apple ID and I can't see it.
Feedback assistant seems to make things private by default. There are only 15 bugs I have access to, and that can't be all the bugs in existence.

bcmills · 2020-10-08T19:42:14Z

I have access to the Google Apple account from which the report was filed (obtained for #33041 (comment)) and can confirm that that feedback report describes this issue.

gopherbot · 2020-10-14T23:08:22Z

Change https://golang.org/cl/262438 mentions this issue: runtime: stop preemption during syscall.Exec on Darwin

gopherbot · 2020-10-15T18:34:25Z

Change https://golang.org/cl/262717 mentions this issue: [release-branch.go1.15] runtime: stop preemption during syscall.Exec on Darwin

gopherbot · 2020-10-15T18:35:30Z

Change https://golang.org/cl/262737 mentions this issue: [release-branch.go1.14] runtime: stop preemption during syscall.Exec on Darwin

gopherbot · 2020-10-15T21:20:30Z

Change https://golang.org/cl/262738 mentions this issue: syscall: use MustHaveExec in TestExec

gopherbot · 2020-10-15T21:40:29Z

Change https://golang.org/cl/262817 mentions this issue: runtime: wait for preemption signals before syscall.Exec

rsc · 2020-10-16T17:58:33Z

Reopening until CL 262817 is submitted. (Test is failing right now.)

For #41702 Change-Id: Ib2b15e52aa1fef2f5e644b316c726150252fa9f8 Reviewed-on: https://go-review.googlesource.com/c/go/+/262738 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>

…on Darwin On current macOS versions a program that receives a signal during an execve can fail with a SIGILL signal. This appears to be a macOS kernel bug. It has been reported to Apple. This CL partially works around the problem by using execLock to not send preemption signals during execve. Of course some other stray signal could occur, but at least we can avoid exacerbating the problem. We can't simply disable signals, as that would mean that the exec'ed process would start with all signals blocked, which it likely does not expect. For #41702 Fixes #41703 Change-Id: I91b0add967b315671ddcf73269c4d30136e579b4 Reviewed-on: https://go-review.googlesource.com/c/go/+/262438 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 64fb6ae) Reviewed-on: https://go-review.googlesource.com/c/go/+/262737

…on Darwin On current macOS versions a program that receives a signal during an execve can fail with a SIGILL signal. This appears to be a macOS kernel bug. It has been reported to Apple. This CL partially works around the problem by using execLock to not send preemption signals during execve. Of course some other stray signal could occur, but at least we can avoid exacerbating the problem. We can't simply disable signals, as that would mean that the exec'ed process would start with all signals blocked, which it likely does not expect. For #41702 Fixes #41704 Change-Id: I91b0add967b315671ddcf73269c4d30136e579b4 Reviewed-on: https://go-review.googlesource.com/c/go/+/262438 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 64fb6ae) Reviewed-on: https://go-review.googlesource.com/c/go/+/262717

gopherbot · 2020-10-20T21:39:16Z

Change https://golang.org/cl/264020 mentions this issue: [possible] syscall: use MustHaveExec in TestExec

gopherbot · 2020-10-20T21:40:17Z

Change https://golang.org/cl/264021 mentions this issue: [release-branch.go1.14] syscall: use MustHaveExec in TestExec

gopherbot · 2020-10-20T21:41:02Z

Change https://golang.org/cl/264022 mentions this issue: [release-branch.go1.15] runtime: wait for preemption signals before syscall.Exec

gopherbot · 2020-10-20T21:41:45Z

Change https://golang.org/cl/264023 mentions this issue: [release-branch.go1.14] runtime: wait for preemption signals before syscall.Exec

For #41702 For #41703 Change-Id: Ib2b15e52aa1fef2f5e644b316c726150252fa9f8 Reviewed-on: https://go-review.googlesource.com/c/go/+/262738 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 11cfb48) Reviewed-on: https://go-review.googlesource.com/c/go/+/264021

For #41702 For #41704 Change-Id: Ib2b15e52aa1fef2f5e644b316c726150252fa9f8 Reviewed-on: https://go-review.googlesource.com/c/go/+/262738 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 11cfb48) Reviewed-on: https://go-review.googlesource.com/c/go/+/264020

…yscall.Exec For #41702 For #41704 For #42023 Change-Id: If07f40b1d73b8f276ee28ffb8b7214175e56c24d Reviewed-on: https://go-review.googlesource.com/c/go/+/262817 Trust: Ian Lance Taylor <iant@golang.org> Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 05739d6) Reviewed-on: https://go-review.googlesource.com/c/go/+/264022

…yscall.Exec For #41702 For #41703 For #42023 Change-Id: If07f40b1d73b8f276ee28ffb8b7214175e56c24d Reviewed-on: https://go-review.googlesource.com/c/go/+/262817 Trust: Ian Lance Taylor <iant@golang.org> Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 05739d6) Reviewed-on: https://go-review.googlesource.com/c/go/+/264023

…on Darwin On current macOS versions a program that receives a signal during an execve can fail with a SIGILL signal. This appears to be a macOS kernel bug. It has been reported to Apple. This CL partially works around the problem by using execLock to not send preemption signals during execve. Of course some other stray signal could occur, but at least we can avoid exacerbating the problem. We can't simply disable signals, as that would mean that the exec'ed process would start with all signals blocked, which it likely does not expect. For golang#41702 Fixes golang#41704 Change-Id: I91b0add967b315671ddcf73269c4d30136e579b4 Reviewed-on: https://go-review.googlesource.com/c/go/+/262438 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 64fb6ae) Reviewed-on: https://go-review.googlesource.com/c/go/+/262717

For golang#41702 For golang#41704 Change-Id: Ib2b15e52aa1fef2f5e644b316c726150252fa9f8 Reviewed-on: https://go-review.googlesource.com/c/go/+/262738 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 11cfb48) Reviewed-on: https://go-review.googlesource.com/c/go/+/264020

…yscall.Exec For golang#41702 For golang#41704 For golang#42023 Change-Id: If07f40b1d73b8f276ee28ffb8b7214175e56c24d Reviewed-on: https://go-review.googlesource.com/c/go/+/262817 Trust: Ian Lance Taylor <iant@golang.org> Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> (cherry picked from commit 05739d6) Reviewed-on: https://go-review.googlesource.com/c/go/+/264022

gopherbot · 2020-12-03T21:56:53Z

Change https://golang.org/cl/275293 mentions this issue: runtime: avoid receiving preemotion signal while exec'ing

The iOS kernel has the same problem as the macOS kernel. Extend the workaround of #41702 (CL 262438 and CL 262817) to iOS. Updates #35851. Change-Id: I7ccec00dc96643c08c5be8b385394856d0fa0f64 Reviewed-on: https://go-review.googlesource.com/c/go/+/275293 Trust: Cherry Zhang <cherryyz@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>

ianlancetaylor added OS-Darwin NeedsFix The path to resolution is known, but the work has not been done. release-blocker labels Sep 30, 2020

ianlancetaylor added this to the Go1.16 milestone Sep 30, 2020

This was referenced Sep 30, 2020

runtime: macOS syscall.Exec can get SIGILL due to preemption signal [1.14 backport] #41703

Closed

runtime: macOS syscall.Exec can get SIGILL due to preemption signal [1.15 backport] #41704

Closed

ianlancetaylor self-assigned this Oct 14, 2020

gopherbot closed this as completed in 64fb6ae Oct 15, 2020

rsc mentioned this issue Oct 16, 2020

syscall: flaky TestExec failure on macOS: signal: Illegal instruction #42023

Closed

rsc reopened this Oct 16, 2020

gopherbot closed this as completed in 05739d6 Oct 16, 2020

cherrymui mentioned this issue Nov 22, 2020

runtime: sporadic but frequent "signal: illegal instruction" during all.bash on darwin/arm64 at CL 272258 PS 1 #42774

Closed

eliasnaur mentioned this issue Dec 2, 2020

runtime: signal: illegal instruction on ios/arm64 #35851

Closed

golang locked and limited conversation to collaborators Dec 3, 2021

gopherbot added the FrozenDueToAge label Dec 3, 2021

rsc unassigned ianlancetaylor Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: macOS syscall.Exec can get SIGILL due to preemption signal #41702

runtime: macOS syscall.Exec can get SIGILL due to preemption signal #41702

ianlancetaylor commented Sep 30, 2020

ianlancetaylor commented Sep 30, 2020

gopherbot commented Sep 30, 2020

ianlancetaylor commented Oct 8, 2020

randall77 commented Oct 8, 2020

bcmills commented Oct 8, 2020 •

edited

gopherbot commented Oct 14, 2020

gopherbot commented Oct 15, 2020

gopherbot commented Oct 15, 2020

gopherbot commented Oct 15, 2020

gopherbot commented Oct 15, 2020

rsc commented Oct 16, 2020

gopherbot commented Oct 20, 2020

gopherbot commented Oct 20, 2020

gopherbot commented Oct 20, 2020

gopherbot commented Oct 20, 2020

gopherbot commented Dec 3, 2020

runtime: macOS syscall.Exec can get SIGILL due to preemption signal #41702

runtime: macOS syscall.Exec can get SIGILL due to preemption signal #41702

Comments

ianlancetaylor commented Sep 30, 2020

ianlancetaylor commented Sep 30, 2020

gopherbot commented Sep 30, 2020

ianlancetaylor commented Oct 8, 2020

randall77 commented Oct 8, 2020

bcmills commented Oct 8, 2020 • edited

gopherbot commented Oct 14, 2020

gopherbot commented Oct 15, 2020

gopherbot commented Oct 15, 2020

gopherbot commented Oct 15, 2020

gopherbot commented Oct 15, 2020

rsc commented Oct 16, 2020

gopherbot commented Oct 20, 2020

gopherbot commented Oct 20, 2020

gopherbot commented Oct 20, 2020

gopherbot commented Oct 20, 2020

gopherbot commented Dec 3, 2020

bcmills commented Oct 8, 2020 •

edited