runtime: runtime test timeout in TestCrashDumpsAllThreads #19196
Comments
|
Possibly related: #18442 Also: why did you delete the following sections:
from the issue template? |
What version of Go are you using (
|
ALTree
added
the
NeedsInvestigation
|
The hang is occurring in |
ianlancetaylor
changed the title
|
I wonder if this has something to do with having a controlling terminal, and or running under a pseudo-terminal? When I ssh into my build box and build go by hand, I have: That environment varible is not set in a Jenkins slave's environment. We're sending
If the process doesn't have a controlling terminal it might be ignoring SIGQUIT. |
|
This change to 26 func TestCrashDumpsAllThreads(t *testing.T) {
27 _, err := os.Open("/dev/tty")
28 if err != nil {
29 t.Skipf("Skipping, unable to open /dev/tty: %v")
30 }Newbie mystery, I thought that the |
|
The presence or absence of a controlling terminal should be irrelevant here. We are sending The |
|
Sorry for the false start and thanks for the explanation! |
|
For now , Is it possible to skip runtime test ? If yes , then what we have to do ? Thanks! |
|
@ianlancetaylor -- Is there a clue to be had in the fact that this test (tests) tend to fail when run in a Jenkins job and not from a command line? Do you have any ideas of where to start poking at it? |
|
Well, I'm sure there is a clue that the failure is in a Jenkins job, but I don't know what that clue is. If you can recreate this at will, I would suggest running the child process under |
|
@ianlancetaylor -- that looks doable-by-me. Thanks! |
|
Sending the output to std{out,err} resulted in Jenkins spinning for 15 minutes before I got bored and killed the job. I sent the output to a pair of files instead. Here's the diff (no snickering...): $ diff crash_unix_test.go.orig crash_unix_test.go
10c10
< "bytes"
---
> // "bytes"
59c59
< cmd = exec.Command(filepath.Join(dir, "a.exe"))
---
> cmd = exec.Command("strace", "-f", filepath.Join(dir, "a.exe"))
74,76c74,89
< var outbuf bytes.Buffer
< cmd.Stdout = &outbuf
< cmd.Stderr = &outbuf
---
> // var outbuf bytes.Buffer
> // cmd.Stdout = &outbuf
> // cmd.Stderr = &outbuf
>
> // cmd.Stdout = os.Stdout
> // cmd.Stderr = os.Stderr
> so, err := os.Create("/tmp/poodle-stdout")
> if err != nil {
> t.Fatalf("Unable to create /tmp/poodle-stdout: %v", err)
> }
> cmd.Stdout = so
> se, err := os.Create("/tmp/poodle-stderr")
> if err != nil {
> t.Fatalf("Unable to create /tmp/poodle-stderr: %v", err)
> }
> cmd.Stderr = se
106,111c119,125
< out = outbuf.Bytes()
< n := bytes.Count(out, []byte("main.loop("))
< if n != 4 {
< t.Errorf("found %d instances of main.loop; expected 4", n)
< t.Logf("%s", out)
< }
---
> // out = outbuf.Bytes()
> // n := bytes.Count(out, []byte("main.loop("))
> // if n != 4 {
> // t.Errorf("found %d instances of main.loop; expected 4", n)
> // t.Logf("%s", out)
> // }
> t.Logf("Falling out the bottom")
What can I do next (and thanks for helping!)? |
|
ps. That test "passed" (noticed that I lopped the testy parts out) and/but took a bit over 20 seconds. |
|
I connected the dots with instructions that @minux gave me in a related thread. I ran a build with an unmodified It's not what I expected, but here's the result: Here is the |
|
@hartzell, sorry for dropping this on the floor. Could you re-do what you did with gdb in your last comment, but first add |
|
Hi @aclements -- I'm on vacation (miles and worlds away from work). I'll be back in about 10 days and will see if I can do anything with this. I no longer have access to that particularly large machine but will see I can get it again. |
bradfitz
added
the
Testing
|
Ping @hartzell |
aclements
added
the
WaitingForInfo
|
It looks like that when you start a program from Java, not just Jenkins, the SIGQUIT signal is blocked. I do not know if this blocking is a JVM bug or feature: The Java runtime prints a stack trace and keeps on running on SIGQUIT, while most programs would of course terminate. It might be that Java tries to protect other programs from When I patch the Go runtime to unblock SIGQUIT, diff --git a/src/runtime/sigtab_linux_generic.go b/src/runtime/sigtab_linux_generic.go
index 874148e1d2..8f4508dd90 100644
--- a/src/runtime/sigtab_linux_generic.go
+++ b/src/runtime/sigtab_linux_generic.go
@@ -19,7 +19,7 @@ var sigtable = [...]sigTabT{
/* 0 */ {0, "SIGNONE: no trap"},
/* 1 */ {_SigNotify + _SigKill, "SIGHUP: terminal line hangup"},
/* 2 */ {_SigNotify + _SigKill, "SIGINT: interrupt"},
- /* 3 */ {_SigNotify + _SigThrow, "SIGQUIT: quit"},
+ /* 3 */ {_SigNotify + _SigThrow + _SigUnblock, "SIGQUIT: quit"},
/* 4 */ {_SigThrow + _SigUnblock, "SIGILL: illegal instruction"},
/* 5 */ {_SigThrow + _SigUnblock, "SIGTRAP: trace trap"},
/* 6 */ {_SigNotify + _SigThrow, "SIGABRT: abort"},
all.bash passes. Are there any good reasons for leaving SIGQUIT blocked, except for the terminal Here |
|
@chlunde, thanks for figuring that out! I suspect you're right about the reasoning behind Java blocking SIQQUIT.
This seems like the right approach to me. Could someone try the following patch? (This isn't the right way to do it, but if this works I'll put together a real CL) --- a/src/runtime/crash_test.go
+++ b/src/runtime/crash_test.go
@@ -12,18 +12,24 @@ import (
"io/ioutil"
"os"
"os/exec"
+ "os/signal"
"path/filepath"
"regexp"
"runtime"
"strconv"
"strings"
"sync"
+ "syscall"
"testing"
"time"
)
var toRemove []string
+func init() {
+ signal.Notify(make(chan os.Signal), syscall.SIGQUIT)
+}
+
func TestMain(m *testing.M) {
status := m.Run()
for _, file := range toRemove { |
|
@aclements It kind of works, a little bit. For the parent process, I can send a signal. The test still hangs. Reproducer: public class Wrap {
public static void main(String... args) throws Exception {
Process proc = new ProcessBuilder(args).inheritIO().start();
System.exit(proc.waitFor());
}
}java -cp ~/tmp Wrap ./runtime.test -test.run TestCrashDumpsAllThreads -test.v |
|
Thanks. I added some prints in the runtime and confirmed that that change to the test does cause it to unblock SIGQUIT with |
|
Unblocking the signal in the runtime.test program does not unblock the signal in child processes started by the runtime.test program. Using |
|
It's likely that this patch in addition to yours would fix the problem. |
|
I'm not sure if I understand the goal here, is it
or something else? If it is 1), maybe it would be less intrusive to skip this test if an environment variable set by Jenkins is defined, and drop that if Java/Jenkins changes the behaviour. |
|
I agree that for the 1.9 release, if there is an environment variable that we can use to reliably detect whether we are running under Jenkins, then we should skip that test if the environment variable is set. Do you know if there is such an environment variable? For 1.10 I think a patch along the lines of the one I posted above would be reasonable, so that Go programs have a way to control the signal mask of child processes. Or, of course, we could also simply add a field to |
Or we could export something to the runtime test that checks if SIGQUIT is blocked. |
|
RE: Jenkins environment variables (scroll down to "Jenkins set environment variables): https://wiki.jenkins.io/display/JENKINS/Building+a+software+project
|
|
@ianlancetaylor that patch works, tested with Wrap.java and in Jenkins. @hartzell I don't have JENKINS_URL here on a freshly started jenkins server. I also took a stab at trying to detect if SIGQUIT is blocked, but I'm not sure if this is the right approach and portable, given that sigset is different and there are buildtags involved. diff --git a/src/runtime/crash_unix_test.go b/src/runtime/crash_unix_test.go
index fdb3267006..73da55d267 100644
--- a/src/runtime/crash_unix_test.go
+++ b/src/runtime/crash_unix_test.go
@@ -31,6 +31,10 @@ func TestCrashDumpsAllThreads(t *testing.T) {
t.Skipf("skipping; not supported on %v", runtime.GOOS)
}
+ if runtime.Sigisblocked(int(sigquit)) {
+ t.Skip("skipping; SIGQUIT is blocked, see golang.org/issue/19196")
+ }
+
// We don't use executeTest because we need to kill the
// program while it is running.
diff --git a/src/runtime/export_linux_test.go b/src/runtime/export_linux_test.go
index ef0c111677..c8c94702b6 100644
--- a/src/runtime/export_linux_test.go
+++ b/src/runtime/export_linux_test.go
@@ -8,3 +8,15 @@ package runtime
var NewOSProc0 = newosproc0
var Mincore = mincore
+
+func sigismember(mask *sigset, i int) bool {
+ clear := *mask
+ sigdelset(&clear, i)
+ return clear != *mask
+}
+
+func Sigisblocked(i int) bool {
+ var sigmask sigset
+ sigprocmask(_SIG_SETMASK, nil, &sigmask)
+ return sigismember(&sigmask, i)
+} |
I just created a new job on my Jenkins server (v2.32.2) that had a single "Execute shell" build step that contained #!/bin/sh
printenvand its output included Wonder what's different? I guess if you really don't have it then it's not universal and so the "why?" doesn't matter. Is there another variable on the list that is in your jobs' environments? |
|
I just ran java -jar jenkins-war-2.70-SNAPSHOT.war and accepted the default plugins, without configuring jenkins: |
|
There are some things that don't have default values, instead it forces you to the configuration page the first time you connect and provides default values that get used if/when you save the config page w/out providing anything specific. This is one of them. I have a setup that stands up a Jenkins server w/out requiring any user intervention, I had to add a bit of groovy to ensure that it got set, similar to what's described here on the jenkins wiki.
|
|
@chlunde, your patch looks pretty reasonable to me, other than some minor build tag issues. May I turn it into a CL? (You're welcome to, of course, but I'm happy to.) |
|
@aclements Sure, go ahead |
|
CL https://golang.org/cl/50330 mentions this issue. |
Hi,
I could able to build and test successfully on RHEL7.3 VM, however on jenkins server I am getting runtime test timeout issue. Is there any way to solve this issue ?
Please find attached logfile.
golang_runtime_test_failure.txt
Thanks!
The text was updated successfully, but these errors were encountered: