Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/internal/testdir: Test failures #64050

Closed
gopherbot opened this issue Nov 9, 2023 · 31 comments
Closed

cmd/internal/testdir: Test failures #64050

gopherbot opened this issue Nov 9, 2023 · 31 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@gopherbot
Copy link 

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"

Issue created automatically to collect these failures.

Example (log):

panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m57s)

syscall.Syscall6(0x110, 0x1, 0x362e1, 0xc0000d2700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000d26a8 sp=0xc0000d2638 pc=0x941b8
os.(*Process).blockUntilWaitable(0xc0005b6120)
	/workdir/go/src/os/wait_waitid.go:32 +0x94 fp=0xc0000d2790 sp=0xc0000d26a8 pc=0xbb294
os.(*Process).wait(0xc0005b6120)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc0000d2810 sp=0xc0000d2790 pc=0xb5e6c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0000a82c0)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000d28a0 sp=0xc0000d2810 pc=0x10b414
os/exec.(*Cmd).Run(0xc0000a82c0)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000d28c8 sp=0xc0000d28a0 pc=0x10a230
cmd/internal/testdir_test.test.run.func1({0xc0000d3508?, 0x1c1f4f?, 0xc000354000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc0000d2a38 sp=0xc0000d28c8 pc=0x16ac2c
cmd/internal/testdir_test.test.run({{{0xc00000e960, 0x10}, 0xc00005e150}, 0xc0003fc4e0, {0x1c1596, 0x9}, {0xc0002b8ec0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x22ac fp=0xc0000d3ea8 sp=0xc0000d2a38 pc=0x165d2c
cmd/internal/testdir_test.Test.func1(0xc0003fc4e0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc0000d3f30 sp=0xc0000d3ea8 pc=0x161444
testing.tRunner(0xc0003fc4e0, 0xc000399680)

— watchflakes
@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 9, 2023
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-09 22:35 linux-ppc64le-buildlet go@ff7cf2d4 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m57s)

syscall.Syscall6(0x110, 0x1, 0x362e1, 0xc0000d2700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000d26a8 sp=0xc0000d2638 pc=0x941b8
os.(*Process).blockUntilWaitable(0xc0005b6120)
	/workdir/go/src/os/wait_waitid.go:32 +0x94 fp=0xc0000d2790 sp=0xc0000d26a8 pc=0xbb294
os.(*Process).wait(0xc0005b6120)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc0000d2810 sp=0xc0000d2790 pc=0xb5e6c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0000a82c0)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000d28a0 sp=0xc0000d2810 pc=0x10b414
os/exec.(*Cmd).Run(0xc0000a82c0)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000d28c8 sp=0xc0000d28a0 pc=0x10a230
cmd/internal/testdir_test.test.run.func1({0xc0000d3508?, 0x1c1f4f?, 0xc000354000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc0000d2a38 sp=0xc0000d28c8 pc=0x16ac2c
cmd/internal/testdir_test.test.run({{{0xc00000e960, 0x10}, 0xc00005e150}, 0xc0003fc4e0, {0x1c1596, 0x9}, {0xc0002b8ec0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x22ac fp=0xc0000d3ea8 sp=0xc0000d2a38 pc=0x165d2c
cmd/internal/testdir_test.Test.func1(0xc0003fc4e0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc0000d3f30 sp=0xc0000d3ea8 pc=0x161444
testing.tRunner(0xc0003fc4e0, 0xc000399680)
2023-11-09 22:35 linux-ppc64le-power9osu go@ff7cf2d4 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/rangegen.go (19m47s)

syscall.Syscall6(0x110, 0x1, 0x360ef, 0xc000164700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0001646a8 sp=0xc000164638 pc=0x93c68
os.(*Process).blockUntilWaitable(0xc000016150)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc000164790 sp=0xc0001646a8 pc=0xbac8c
os.(*Process).wait(0xc000016150)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000164810 sp=0xc000164790 pc=0xb587c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0001282c0)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0001648a0 sp=0xc000164810 pc=0x10abf4
os/exec.(*Cmd).Run(0xc0001282c0)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0001648c8 sp=0xc0001648a0 pc=0x109a20
cmd/internal/testdir_test.test.run.func1({0xc000000300?, 0x1c2814?, 0xc000e34000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000164a38 sp=0xc0001648c8 pc=0x16a10c
cmd/internal/testdir_test.test.run({{{0xc00009e1b0, 0x10}, 0xc0000ca000}, 0xc00028c820, {0x1c0358, 0x1}, {0xc00009e5e0, 0xb}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1076 +0x33b4 fp=0xc000165ea8 sp=0xc000164a38 pc=0x166404
cmd/internal/testdir_test.Test.func1(0xc00028c820)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000165f30 sp=0xc000165ea8 pc=0x160a14
testing.tRunner(0xc00028c820, 0xc0000a4830)

— watchflakes

@bcmills bcmills added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 10, 2023
@mknyszek
Copy link
Contributor

That's convenient; I was just about to file an issue.

When I enabled the allocation headers by default, two builders started timing out (linux-ppc64le-buildlet and linux-ppc64-sid-power10). However, I'm a little skeptical this is related to that change just because there was another issue briefly masking it so we're not 100% sure that change is implicated. On top of that, some of the other ppc64 builders don't show this slowdown and some appear to have the tests running faster than before. This makes me wonder if it's potentially a builder issue? It seems to me like the builder might be already hovering somewhat close to its overall timeout?

CC @golang/ppc64

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-09 22:38 linux-ppc64le-buildlet go@130baf3d cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m54s)

syscall.Syscall6(0x110, 0x1, 0x36c70, 0xc0000c2700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000c26a8 sp=0xc0000c2638 pc=0x942d8
os.(*Process).blockUntilWaitable(0xc0001842d0)
	/workdir/go/src/os/wait_waitid.go:32 +0x94 fp=0xc0000c2790 sp=0xc0000c26a8 pc=0xbb3b4
os.(*Process).wait(0xc0001842d0)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc0000c2810 sp=0xc0000c2790 pc=0xb5f8c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc00018d080)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000c28a0 sp=0xc0000c2810 pc=0x10b534
os/exec.(*Cmd).Run(0xc00018d080)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000c28c8 sp=0xc0000c28a0 pc=0x10a350
cmd/internal/testdir_test.test.run.func1({0xc0000c3508?, 0x1c1f4f?, 0xc0001a0000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc0000c2a38 sp=0xc0000c28c8 pc=0x16ad4c
cmd/internal/testdir_test.test.run({{{0xc0001981a0, 0x10}, 0xc0001b4000}, 0xc0003cfba0, {0x1c1596, 0x9}, {0xc00029a6f0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x22ac fp=0xc0000c3ea8 sp=0xc0000c2a38 pc=0x165e4c
cmd/internal/testdir_test.Test.func1(0xc0003cfba0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc0000c3f30 sp=0xc0000c3ea8 pc=0x161564
testing.tRunner(0xc0003cfba0, 0xc000397080)
2023-11-10 04:46 linux-ppc64le-buildlet go@f7c5cbb8 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/rangegen.go (19m51s)

syscall.Syscall6(0x110, 0x1, 0x35e2c, 0xc000090700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000906a8 sp=0xc000090638 pc=0x942d8
os.(*Process).blockUntilWaitable(0xc000212000)
	/workdir/go/src/os/wait_waitid.go:32 +0x94 fp=0xc000090790 sp=0xc0000906a8 pc=0xbb3b4
os.(*Process).wait(0xc000212000)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000090810 sp=0xc000090790 pc=0xb5f8c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0005d6160)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000908a0 sp=0xc000090810 pc=0x10b534
os/exec.(*Cmd).Run(0xc0005d6160)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000908c8 sp=0xc0000908a0 pc=0x10a350
cmd/internal/testdir_test.test.run.func1({0xc000000240?, 0x1c2814?, 0xc000de8000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000090a38 sp=0xc0000908c8 pc=0x16ad4c
cmd/internal/testdir_test.test.run({{{0xc0000c41a0, 0x10}, 0xc0000de000}, 0xc0002a8680, {0x1c0358, 0x1}, {0xc0000c45d0, 0xb}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1076 +0x3464 fp=0xc000091ea8 sp=0xc000090a38 pc=0x167004
cmd/internal/testdir_test.Test.func1(0xc0002a8680)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000091f30 sp=0xc000091ea8 pc=0x161564
testing.tRunner(0xc0002a8680, 0xc0000ba7f0)

— watchflakes

@pmur
Copy link
Contributor

pmur commented Nov 10, 2023

I don't think it's a builder issue. I ran into a deadlock running all.bash against tip on a development machine on my first try. Likewise, one ppc64le builder is stuck bootstrapping the compiler (which seems to bypass the buildlet timeouts).

Attaching gdb to the stuck ppc64le bootstrapping buildlet, the sysmon task is running, but everything else is stuck on a futex, the backtrace isn't helpful.

@mknyszek
Copy link
Contributor

@pmur Thanks for checking. I'll take a look.

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-10 15:49 linux-ppc64le-power10osu go@43ffe2a8 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m52s)

syscall.Syscall6(0x110, 0x1, 0x38c0d, 0xc000090700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000906a8 sp=0xc000090638 pc=0x95668
os.(*Process).blockUntilWaitable(0xc0004603f0)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc000090790 sp=0xc0000906a8 pc=0xbcc4c
os.(*Process).wait(0xc0004603f0)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000090810 sp=0xc000090790 pc=0xb767c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0001ec580)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000908a0 sp=0xc000090810 pc=0x10d154
os/exec.(*Cmd).Run(0xc0001ec580)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000908c8 sp=0xc0000908a0 pc=0x10bf80
cmd/internal/testdir_test.test.run.func1({0xc000091508?, 0x1c1fcf?, 0xc0001c2000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7b8 fp=0xc000090a38 sp=0xc0000908c8 pc=0x16c438
cmd/internal/testdir_test.test.run({{{0xc00000e9a0, 0x10}, 0xc00005e150}, 0xc0003e81a0, {0x1c1616, 0x9}, {0xc0002a2ef0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x21c4 fp=0xc000091ea8 sp=0xc000090a38 pc=0x167684
cmd/internal/testdir_test.Test.func1(0xc0003e81a0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x98 fp=0xc000091f30 sp=0xc000091ea8 pc=0x162e98
testing.tRunner(0xc0003e81a0, 0xc0003836a0)
2023-11-10 15:49 linux-ppc64le-power9osu go@43ffe2a8 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m56s)

syscall.Syscall6(0x110, 0x1, 0x376a9, 0xc000506700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0005066a8 sp=0xc000506638 pc=0x93f28
os.(*Process).blockUntilWaitable(0xc0000bc510)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc000506790 sp=0xc0005066a8 pc=0xbaf4c
os.(*Process).wait(0xc0000bc510)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000506810 sp=0xc000506790 pc=0xb5b3c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0000e0580)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0005068a0 sp=0xc000506810 pc=0x10aeb4
os/exec.(*Cmd).Run(0xc0000e0580)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0005068c8 sp=0xc0005068a0 pc=0x109ce0
cmd/internal/testdir_test.test.run.func1({0xc000507508?, 0x1c1fcf?, 0xc000418000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000506a38 sp=0xc0005068c8 pc=0x16a3bc
cmd/internal/testdir_test.test.run({{{0xc00009a850, 0x10}, 0xc00010a0e0}, 0xc000322d00, {0x1c1616, 0x9}, {0xc000166da0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x2240 fp=0xc000507ea8 sp=0xc000506a38 pc=0x165540
cmd/internal/testdir_test.Test.func1(0xc000322d00)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000507f30 sp=0xc000507ea8 pc=0x160cc4
testing.tRunner(0xc000322d00, 0xc0002cb650)

— watchflakes

@mknyszek
Copy link
Contributor

I'm struggling a bit to get a gomote. I'll just disable the experiment on ppc64 for now.

@gopherbot
Copy link
Author

Change https://go.dev/cl/541555 mentions this issue: internal/buildcfg: disable the allocheaders GOEXPERIMENT on ppc64

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-10 15:51 linux-ppc64le-power10osu go@3b303fa9 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (20m0s)

syscall.Syscall6(0x110, 0x1, 0x379b7, 0xc0002aa700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0002aa6a8 sp=0xc0002aa638 pc=0x95668
os.(*Process).blockUntilWaitable(0xc0003fc4b0)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc0002aa790 sp=0xc0002aa6a8 pc=0xbcc4c
os.(*Process).wait(0xc0003fc4b0)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc0002aa810 sp=0xc0002aa790 pc=0xb767c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc000128420)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0002aa8a0 sp=0xc0002aa810 pc=0x10d154
os/exec.(*Cmd).Run(0xc000128420)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0002aa8c8 sp=0xc0002aa8a0 pc=0x10bf80
cmd/internal/testdir_test.test.run.func1({0xc0002ab508?, 0x1c1fcf?, 0xc0002b0000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7b8 fp=0xc0002aaa38 sp=0xc0002aa8c8 pc=0x16c438
cmd/internal/testdir_test.test.run({{{0xc00000e9a0, 0x10}, 0xc00005e150}, 0xc0003ded00, {0x1c1616, 0x9}, {0xc000220ef0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x21c4 fp=0xc0002abea8 sp=0xc0002aaa38 pc=0x167684
cmd/internal/testdir_test.Test.func1(0xc0003ded00)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x98 fp=0xc0002abf30 sp=0xc0002abea8 pc=0x162e98
testing.tRunner(0xc0003ded00, 0xc00039d080)

— watchflakes

@mknyszek
Copy link
Contributor

mknyszek commented Nov 10, 2023 •
edited

I'm trying to submit https://go.dev/cl/541555 but it seems to be stuck even with the GOEXPERIMENT disabled.

log 
  builder: linux-ppc64le-buildlet
      rev: a078a9ce8db85d07e20cb20b5a11c4e1216e20f0
 buildlet: http://power_04 reverse peer power_04/140.211.169.160:56314 for host type host-linux-ppc64le-osu
  started: 2023-11-10 17:22:02.401510352 +0000 UTC m=+1530450.839428897
   status: still running

Events:
  2023-11-10T17:22:02Z get_buildlet 
  2023-11-10T17:28:05Z finish_get_buildlet after 6m2.8s
  2023-11-10T17:28:05Z using_buildlet power_04
  2023-11-10T17:28:05Z write_go_bootstrap_tar 
  2023-11-10T17:28:05Z write_version_tar 
  2023-11-10T17:28:05Z get_source go@a078a9ce8db85d07e20cb20b5a11c4e1216e20f0
  2023-11-10T17:28:05Z finish_get_source after 0s; go@a078a9ce8db85d07e20cb20b5a11c4e1216e20f0
  2023-11-10T17:28:05Z write_go_src_tar 
  2023-11-10T17:28:09Z finish_write_go_bootstrap_tar after 3.88s
  2023-11-10T17:28:09Z finish_write_go_src_tar after 3.89s
  2023-11-10T17:28:09Z make_and_test 
  2023-11-10T17:28:09Z make src/make.bash
 +2085.6s (now)

Build log:
linux-ppc64le-buildlet at a078a9ce8db85d07e20cb20b5a11c4e1216e20f0

:: Running /workdir/go/src/make.bash with args ["/workdir/go/src/make.bash"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=power_04" "GO_BUILDER_ENV=host-linux-ppc64le-osu" "GO_BUILD_KEY_PATH=/buildkey/gobuildkey" "GO_BUILD_KEY_DELETE_AFTER_READ=true" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=100ms" "GO_STAGE0_DL_DELAY=400ms" "WORKDIR=/workdir" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GO_BUILDER_NAME=linux-ppc64le-buildlet" "GO_BUILDER_FLAKY_NET=1" "GO_TEST_TIMEOUT_SCALE=2" "GOBIN=" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "GOPLSCACHE=/workdir/goplscache" "PWD=/workdir/go/src"] in dir /workdir/go/src

Building Go cmd/dist using /workdir/go1.4. (go1.20.6 linux/ppc64le)
Building Go toolchain1 using /workdir/go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/ppc64le.

@mknyszek
Copy link
Contributor

Oh! Actually, the first timeout is on https://go.dev/cl/533455. I was looking at the wrong thing. That... certainly seems more likely to cause a deadlock.

@mknyszek
Copy link
Contributor

Yes, I believe I've narrowed it to down to https://go.dev/cl/533455. But why?

@mknyszek
Copy link
Contributor

@pmur Since I assume you're able to run on a ppc64 machine directly fairly easily, do you mind posting a stack dump of one of these deadlocks? I can't seem to ssh into the gomote.

@pmur
Copy link
Contributor

pmur commented Nov 10, 2023

The bootstrap binaries seem to lack debug information. I am poking at the stuck compile binary on ppc64le/power_03. Are the following stack dumps helpful?

dumps of the 6 thread stacks 
0x7fffe8cf4ab0:	0x1fc40 <runtime.lock2+160>	0x3f
0x7fffe8cf4ac0:	0xc00003c500	0x4c06c <runtime.wbBufFlush+76>
0x7fffe8cf4ad0:	0x108a480 <runtime.mheap_>	0x200000080
0x7fffe8cf4ae0:	0x0	0x0
0x7fffe8cf4af0:	0xc000000000	0xc000002680
0x7fffe8cf4b00:	0xc0000026b8	0x200000001
0x7fffe8cf4b10:	0x1fb90 <runtime.lock+48>	0x105b4e0 <runtime.m0>
0x7fffe8cf4b20:	0x5c440 <runtime.park_m+352>	0x32cc4 <runtime.gcBgMarkWorker.func2+372>
0x7fffe8cf4b30:	0xc00000001e	0x2000
0x7fffe8cf4b40:	0xc000002680	0x200002000
0x7fffe8cf4b50:	0x4	0x0
0x7fffe8cf4b60:	0x1	0x3e370 <runtime.(*sweepLocked).sweep.(*mheap).freeSpan.func4+112>
0x7fffe8cf4b70:	0xc0039b2000	0x0
0x7fffe8cf4b80:	0xc00003c500	0x108a480 <runtime.mheap_>
0x7fffe8cf4b90:	0x88078 <runtime.systemstack.abi0+104>	0x0
0x7fffe8cf4ba0:	0x1ff	0xc0039b2000
0x7fffe8cf4bb0:	0x200000004	0x105b4e0 <runtime.m0>
0x7fffe8cf4bc0:	0x0	0x0
0x7fffe8cf4bd0:	0x7116f18f74e8	0x108a480 <runtime.mheap_>
0x7fffe8cf4be0:	0xc000040f00	0x87f5c <runtime.mstart.abi0+12>
0x7fffe8cf4bf0:	0x87f00 <runtime.rt0_go.abi0+208>	0x0
0x7fffe8cf4c00:	0x0	0x0
0x7fffe8cf4c10:	0x400000000	0x0
0x7fffe8cf4c20:	0x87f08 <runtime.rt0_go.abi0+216>	0x0
0x7fffe8cf4c30:	0x0	0xae8880 <runtime.mainPC>
0x7fffe8cf4c40:	0x0	0x0
0x7fffe8cf4c50:	0x0	0x0
0x7fffe8cf4c60:	0x66	0x7fffe8cf4c98
0x7fffe8cf4c70:	0x0	0x0
0x7fffe8cf4c80:	0x0	0x0
0x7fffe8cf4c90:	0x66	0x7fffe8cfe63d

0xc000033ec8:	0x60eb4 <runtime.sysmon+228>	0x0
0xc000033ed8:	0x989680	0x5c9470127ba1
0xc000033ee8:	0xc000040f00	0x0
0xc000033ef8:	0x56e5c <runtime.mstart1+156>	0xc000000002
0xc000033f08:	0xc000033f20	0x0
0xc000033f18:	0xa03000002710	0x0
0xc000033f28:	0x4e484 <runtime.minit+36>	0x1000000 <cmd/internal/obj/x86.avxOptab+44416>
0xc000033f38:	0x2710	0x5c787dcbeeaf
0xc000033f48:	0x0	0x0
0xc000033f58:	0x0	0x67b721 <cmd/compile/internal/ssagen.InitConfig+6433>
0xc000033f68:	0xa030f20c8e33	0x56da4 <runtime.mstart0+116>
0xc000033f78:	0x0	0x0
0xc000033f88:	0x0	0x0
0xc000033f98:	0x0	0xc0000024e0
0xc000033fa8:	0x87f5c <runtime.mstart.abi0+12>	0x0
0xc000033fb8:	0x0	0x0
0xc000033fc8:	0x0	0x0
0xc000033fd8:	0x0	0x8c36c <runtime.clone.abi0+156>
0xc000033fe8:	0x87f50 <runtime.mstart.abi0>	0xc0000024e0

0xc00006de58:	0x20020 <runtime.notesleep+192>	0x30012 <runtime.SetFinalizer+1554>
0xc00006de68:	0x41198 <runtime.(*mheap).allocSpan+1608>	0x40c1c <runtime.(*mheap).allocSpan+204>
0xc00006de78:	0xc0000526c8	0x80
0xc00006de88:	0x0	0x0
0xc00006de98:	0x0	0x40744 <runtime.(*mheap).alloc.func1+132>
0xc00006dea8:	0x586c4 <runtime.stopm+100>	0xc0007c3340
0xc00006deb8:	0x58700 <runtime.stopm+160>	0x108a480 <runtime.mheap_>
0xc00006dec8:	0x2000	0x586e4 <runtime.stopm+132>
0xc00006ded8:	0xc0049f4000	0x586d0 <runtime.stopm+112>
0xc00006dee8:	0x5dcd8 <runtime.exitsyscall0+120>	0xffffffffffffffff
0xc00006def8:	0xc0000029c0	0x5de24 <runtime.exitsyscall0+452>
0xc00006df08:	0x2000	0x5dde8 <runtime.exitsyscall0+392>
0xc00006df18:	0x81ba0 <runtime.wbBufFlush.func1+48>	0xc0000526c8
0xc00006df28:	0xffffffffffffe000	0xc0000029c0
0xc00006df38:	0xc0000029c0	0x87fd4 <runtime.mcall+84>
0xc00006df48:	0xc00003c500	0x88078 <runtime.systemstack.abi0+104>
0xc00006df58:	0x0	0xc000173520
0xc00006df68:	0x100000003	0xc00003e768
0xc00006df78:	0x0	0x0
0xc00006df88:	0x0	0x0
0xc00006df98:	0x0	0xc000173520
0xc00006dfa8:	0x87f5c <runtime.mstart.abi0+12>	0x0
0xc00006dfb8:	0x0	0x0
0xc00006dfc8:	0x0	0x0
0xc00006dfd8:	0x0	0x8c36c <runtime.clone.abi0+156>
0xc00006dfe8:	0x87f50 <runtime.mstart.abi0>	0xc0000029c0
0xc00006dff8:	0xc000052580	0x0
0xc00006e008:	0x2	0xc00006e060
0xc00006e018:	0x8	0x8e2c80
0xc00006e028:	0x0	0x0
0xc00006e038:	0x0	0x0
0xc00006e048:	0x0	0x0
0xc00006e058:	0x0	0x0
0xc00006e068:	0x0	0x44656c69666f7250
0xc00006e078:	0x6464413a3a617461	0x656c69666f72507c
0xc00006e088:	0x72703a3a61746144	0x6c646e61685f666f
0xc00006e098:	0x72507570437c7265	0x3a3a72656c69666f
0xc00006e0a8:	0x6e61685f666f7270	0x705f5f7c72656c64
0xc00006e0b8:	0x735f646165726874	0x656c646e61686769
0xc00006e0c8:	0x747365725f5f7c72	0x65726f <cmd/compile/internal/ssa.tighten+991>

0xc000069a08:	0x1fc40 <runtime.lock2+160>	0x223fc <runtime.persistentalloc1+524>
0xc000069a18:	0x22324 <runtime.persistentalloc1+308>	0x0
0xc000069a28:	0x108a480 <runtime.mheap_>	0x200000080
0xc000069a38:	0x0	0x0
0xc000069a48:	0x0	0x3
0xc000069a58:	0xc000893fff	0x2
0xc000069a68:	0x40cc0 <runtime.(*mheap).allocSpan+368>	0xffffffffffffefe8
0xc000069a78:	0x80	0xffffffffffffff80
0xc000069a88:	0xc00000001e	0x40c1c <runtime.(*mheap).allocSpan+204>
0xc000069a98:	0x44e90 <runtime.(*pageAlloc).free+576>	0x1f1600000
0xc000069aa8:	0x4	0x0
0xc000069ab8:	0x1	0x407ac <runtime.(*mheap).allocManual+76>
0xc000069ac8:	0x1018	0x7116f1625d00
0xc000069ad8:	0x10aaad8 <runtime.memstats+3576>	0x108a480 <runtime.mheap_>
0xc000069ae8:	0x22180 <runtime.persistentalloc+128>	0xda58c <time.Time.Format+268>
0xc000069af8:	0x221d8 <runtime.persistentalloc.func1+56>	0x3
0xc000069b08:	0xc001715fff	0x3
0xc000069b18:	0x0	0xffffffffffffefe8
0xc000069b28:	0x80	0xffffffffffffff80
0xc000069b38:	0xc000052b00	0x0
0xc000069b48:	0x7116f163a380	0x0
0xc000069b58:	0x880b8 <runtime.systemstack.abi0+168>	0x80
0xc000069b68:	0xc00003ea00	0x6cda0 <runtime.stackpoolalloc+112>
0xc000069b78:	0x1018	0x7116f1639b80
0xc000069b88:	0x10aaad8 <runtime.memstats+3576>	0x108a480 <runtime.mheap_>
0xc000069b98:	0x4	0x60001 <runtime.procresize+929>
0xc000069ba8:	0x6d0d4 <runtime.stackcacherefill+116>	0x800
0xc000069bb8:	0x4abc4 <runtime.(*spanSetBlockAlloc).alloc+212>	0x10aaad8 <runtime.memstats+3576>
0xc000069bc8:	0x438e4 <runtime.(*pageAlloc).update+868>	0x22180 <runtime.persistentalloc+128>
0xc000069bd8:	0xc000069be8	0x7116f163a380
0xc000069be8:	0x221a0 <runtime.persistentalloc.func1>	0x10a4648 <runtime.stackpool+136>
0xc000069bf8:	0x6d624 <runtime.stackalloc+660>	0x80
0xc000069c08:	0x10aaad8 <runtime.memstats+3576>	0x4a570 <runtime.(*spanSet).push+432>
0xc000069c18:	0x711713de0001	0x221a0 <runtime.persistentalloc.func1>
0xc000069c28:	0x0	0x0
0xc000069c38:	0x10a4640 <runtime.stackpool+128>	0x1
0xc000069c48:	0x6e194 <runtime.copystack+244>	0x6000


0xc00008fe70:	0x1fc40 <runtime.lock2+160>	0x59114 <runtime.wakep+196>
0xc00008fe80:	0xc00003a000	0x4c06c <runtime.wbBufFlush+76>
0xc00008fe90:	0x108a480 <runtime.mheap_>	0x200000080
0xc00008fea0:	0x0	0x0
0xc00008feb0:	0x0	0x56110 <runtime.casgstatus+624>
0xc00008fec0:	0xc000002b98	0x200000001
0xc00008fed0:	0x1fb90 <runtime.lock+48>	0xc000053080
0xc00008fee0:	0x5c440 <runtime.park_m+352>	0x32cc4 <runtime.gcBgMarkWorker.func2+372>
0xc00008fef0:	0xc00000001e	0x5c3b0 <runtime.park_m+208>
0xc00008ff00:	0xd5d44 <syscall.Uname+148>	0x2000221d8
0xc00008ff10:	0x4	0x0
0xc00008ff20:	0x1	0x3e370 <runtime.(*sweepLocked).sweep.(*mheap).freeSpan.func4+112>
0xc00008ff30:	0xffffffffffffefe8	0x80
0xc00008ff40:	0xffffffffffffff80	0x108a480 <runtime.mheap_>
0xc00008ff50:	0x88078 <runtime.systemstack.abi0+104>	0x7116f170a680
0xc00008ff60:	0xc000172680	0x88078 <runtime.systemstack.abi0+104>
0xc00008ff70:	0x200000004	0xc000053080
0xc00008ff80:	0x0	0x1018
0xc00008ff90:	0x7116f1adab00	0x108a480 <runtime.mheap_>
0xc00008ffa0:	0xc00003c500	0x87f5c <runtime.mstart.abi0+12>
0xc00008ffb0:	0x0	0x0
0xc00008ffc0:	0x0	0x0
0xc00008ffd0:	0x0	0x0
0xc00008ffe0:	0x8c36c <runtime.clone.abi0+156>	0x87f50 <runtime.mstart.abi0>
0xc00008fff0:	0xc000003380	0xc000053080


0xc001d41e08:	0x1fc40 <runtime.lock2+160>	0x1800d0 <cmd/compile/internal/base.readCoverageCfg+112>
0xc001d41e18:	0x3d404 <runtime.(*mspan).ensureSwept+276>	0xc002806808
0xc001d41e28:	0x108a480 <runtime.mheap_>	0x200000080
0xc001d41e38:	0x0	0x0
0xc001d41e48:	0x0	0x42058 <runtime.addspecial+168>
0xc001d41e58:	0xffffffffffffff80	0x3001a <runtime.SetFinalizer+1562>
0xc001d41e68:	0x400c0 <runtime.(*mheap).reclaim+480>	0x40c1c <runtime.(*mheap).allocSpan+204>
0xc001d41e78:	0x42170 <runtime.addspecial+448>	0xc0069fe000
0xc001d41e88:	0x1e	0x7116f16bb2fb
0xc001d41e98:	0x100000002	0x200000000
0xc001d41ea8:	0x4	0x0
0xc001d41eb8:	0x1	0x40730 <runtime.(*mheap).alloc.func1+112>
0xc001d41ec8:	0x7116f17f94d8	0x223fc <runtime.persistentalloc1+524>
0xc001d41ed8:	0x1000000000000	0x108a480 <runtime.mheap_>
0xc001d41ee8:	0x19e0	0xc0013d8000
0xc001d41ef8:	0x7116f17f9550	0xd2d14 <syscall.anyToSockaddr+260>
0xc001d41f08:	0x221d8 <runtime.persistentalloc.func1+56>	0xa
0xc001d41f18:	0x2	0x2000
0xc001d41f28:	0x0	0xc0013d8000
0xc001d41f38:	0x7116f1c30000	0x109a610 <runtime.mheap_+65936>
0xc001d41f48:	0x109a608 <runtime.mheap_+65928>	0x88078 <runtime.systemstack.abi0+104>
0xc001d41f58:	0x7116f1585200	0x88078 <runtime.systemstack.abi0+104>
0xc001d41f68:	0x88078 <runtime.systemstack.abi0+104>	0x108a480 <runtime.mheap_>
0xc001d41f78:	0xc0000021a0	0xc005be99e0
0xc001d41f88:	0x8200000000001018	0xa
0xc001d41f98:	0x108a480 <runtime.mheap_>	0xc002805d50
0xc001d41fa8:	0x87f5c <runtime.mstart.abi0+12>	0x0
0xc001d41fb8:	0x0	0x0
0xc001d41fc8:	0x0	0x0
0xc001d41fd8:	0x0	0x8c36c <runtime.clone.abi0+156>
0xc001d41fe8:	0x87f50 <runtime.mstart.abi0>	0xc0013dc1a0

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-10 16:18 linux-ppc64le-buildlet go@abf84221 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/rangegen.go (19m45s)

syscall.Syscall6(0x110, 0x1, 0x3639a, 0xc000156700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0001566a8 sp=0xc000156638 pc=0x94478
os.(*Process).blockUntilWaitable(0xc0004520c0)
	/workdir/go/src/os/wait_waitid.go:32 +0x94 fp=0xc000156790 sp=0xc0001566a8 pc=0xbb554
os.(*Process).wait(0xc0004520c0)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000156810 sp=0xc000156790 pc=0xb612c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0000f0160)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0001568a0 sp=0xc000156810 pc=0x10b6d4
os/exec.(*Cmd).Run(0xc0000f0160)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0001568c8 sp=0xc0001568a0 pc=0x10a4f0
cmd/internal/testdir_test.test.run.func1({0xc00017e1e0?, 0x1c2894?, 0xc000de8000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000156a38 sp=0xc0001568c8 pc=0x16aedc
cmd/internal/testdir_test.test.run({{{0xc00000e2c0, 0x10}, 0xc00005e070}, 0xc00022f1e0, {0x1c03d8, 0x1}, {0xc00000e6f0, 0xb}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1076 +0x3464 fp=0xc000157ea8 sp=0xc000156a38 pc=0x167194
cmd/internal/testdir_test.Test.func1(0xc00022f1e0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000157f30 sp=0xc000157ea8 pc=0x1616f4
testing.tRunner(0xc00022f1e0, 0xc00001e800)

— watchflakes

@mknyszek
Copy link
Contributor

Oof. It's a bit hard for me to make sense of these tracebacks. I think whatever is inferring symbols is probably being a little overzealous.

@pmur
Copy link
Contributor

pmur commented Nov 10, 2023

Here is a proper log of backtraces from running go build -a cmd/compile/internal/ssa in a loop until deadlocking on ppc64.
deadlock_ppc64.txt

@mknyszek
Copy link
Contributor

Oh that's great, thanks! I think I have a couple leads now.

@mknyszek
Copy link
Contributor

Yeah, that seems to suggest everything is stuck on mheap_.lock, which is very weird. That doesn't really line up with what I thought was the culprit... It's also unclear what's holding the lock. That suggests to me that it might be a self-deadlock somewhere. mheap_.lock must be held on the system stack because any stack growth could end up trying to acquire it again.

Notably, there is an attempt to allocate a new stack in the tracebacks, which supports this theory. That suggests there's a rogue stack growth at a point where mheap_.lock is held.

@mknyszek
Copy link
Contributor

Oh, this might suggest it's the change below the one I apparently identified as a culprit. That certainly introduced new calls in potentially sensitive contexts like this.

@mknyszek mknyszek mentioned this issue Nov 10, 2023
Closed
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-10 16:18 linux-ppc64le-power9osu go@abf84221 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m56s)

syscall.Syscall6(0x110, 0x1, 0x37339, 0xc0003b6700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0003b66a8 sp=0xc0003b6638 pc=0x93f28
os.(*Process).blockUntilWaitable(0xc00027ec60)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc0003b6790 sp=0xc0003b66a8 pc=0xbaf4c
os.(*Process).wait(0xc00027ec60)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc0003b6810 sp=0xc0003b6790 pc=0xb5b3c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc000448c60)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0003b68a0 sp=0xc0003b6810 pc=0x10aeb4
os/exec.(*Cmd).Run(0xc000448c60)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0003b68c8 sp=0xc0003b68a0 pc=0x109ce0
cmd/internal/testdir_test.test.run.func1({0xc0003b7508?, 0x1c1fcf?, 0xc0002a4000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc0003b6a38 sp=0xc0003b68c8 pc=0x16a3bc
cmd/internal/testdir_test.test.run({{{0xc00000e990, 0x10}, 0xc00005e150}, 0xc0002f0000, {0x1c1616, 0x9}, {0xc0001a0ee0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x2240 fp=0xc0003b7ea8 sp=0xc0003b6a38 pc=0x165540
cmd/internal/testdir_test.Test.func1(0xc0002f0000)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc0003b7f30 sp=0xc0003b7ea8 pc=0x160cc4
testing.tRunner(0xc0002f0000, 0xc000283690)

— watchflakes

@gopherbot
Copy link
Author

Change https://go.dev/cl/541635 mentions this issue: runtime: call enableMetadataHugePages and its callees on the systemstack

@mknyszek
Copy link
Contributor

With the CL I just put up, I can no longer reproduce the timeouts I was seeing before in a gomote on tip. It's definitely fixing a problem so we should land it anyway, but it looks like it might be the problem.

If this really is the root cause, then it turns out it had very little to do with any of the changes landed today.

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-10 20:43 linux-ppc64-sid-buildlet go@4346ba34 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m50s)

syscall.Syscall6(0x110, 0x1, 0x27de4, 0xc0000b6708, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000b66b0 sp=0xc0000b6640 pc=0x931a8
os.(*Process).blockUntilWaitable(0xc000016240)
	/workdir/go/src/os/wait_waitid.go:32 +0x94 fp=0xc0000b6798 sp=0xc0000b66b0 pc=0xba544
os.(*Process).wait(0xc000016240)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc0000b6818 sp=0xc0000b6798 pc=0xb511c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc00054e000)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000b68a8 sp=0xc0000b6818 pc=0x10a864
os/exec.(*Cmd).Run(0xc00054e000)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000b68d0 sp=0xc0000b68a8 pc=0x109680
cmd/internal/testdir_test.test.run.func1({0xc0000b7508?, 0x1c1f3c?, 0xc0001c0000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc0000b6a40 sp=0xc0000b68d0 pc=0x169d8c
cmd/internal/testdir_test.test.run({{{0xc00000e970, 0x10}, 0xc00005e150}, 0xc0003fc340, {0x1c1536, 0x9}, {0xc0002b6ed0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x23dc fp=0xc0000b7ea8 sp=0xc0000b6a40 pc=0x164fac
cmd/internal/testdir_test.Test.func1(0xc0003fc340)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc0000b7f30 sp=0xc0000b7ea8 pc=0x1604f4
testing.tRunner(0xc0003fc340, 0xc000399650)

— watchflakes

@mknyszek mknyszek mentioned this issue Nov 10, 2023
Closed
@mknyszek
Copy link
Contributor

Closing as a duplicate of #64067.

@mknyszek mknyszek closed this as not planned Won't fix, can't repro, duplicate, stale Nov 10, 2023
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-10 18:45 linux-ppc64le-power10osu go@505dff4f cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (20m0s)

syscall.Syscall6(0x110, 0x1, 0x37927, 0xc0000b6700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000b66a8 sp=0xc0000b6638 pc=0x95668
os.(*Process).blockUntilWaitable(0xc000016150)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc0000b6790 sp=0xc0000b66a8 pc=0xbcc4c
os.(*Process).wait(0xc000016150)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc0000b6810 sp=0xc0000b6790 pc=0xb767c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0001c6000)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000b68a0 sp=0xc0000b6810 pc=0x10d154
os/exec.(*Cmd).Run(0xc0001c6000)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000b68c8 sp=0xc0000b68a0 pc=0x10bf80
cmd/internal/testdir_test.test.run.func1({0xc0000b7508?, 0x1c1fcf?, 0xc00023c000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7b8 fp=0xc0000b6a38 sp=0xc0000b68c8 pc=0x16c438
cmd/internal/testdir_test.test.run({{{0xc00000e9a0, 0x10}, 0xc00005e150}, 0xc0003e61a0, {0x1c1616, 0x9}, {0xc0002a0ef0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x21c4 fp=0xc0000b7ea8 sp=0xc0000b6a38 pc=0x167684
cmd/internal/testdir_test.Test.func1(0xc0003e61a0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x98 fp=0xc0000b7f30 sp=0xc0000b7ea8 pc=0x162e98
testing.tRunner(0xc0003e61a0, 0xc0003816a0)
2023-11-10 21:09 linux-ppc64le-power10osu go@ea14b633 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m59s)

syscall.Syscall6(0x110, 0x1, 0x37f2e, 0xc00041a700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc00041a6a8 sp=0xc00041a638 pc=0x95668
os.(*Process).blockUntilWaitable(0xc000016150)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc00041a790 sp=0xc00041a6a8 pc=0xbcc4c
os.(*Process).wait(0xc000016150)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc00041a810 sp=0xc00041a790 pc=0xb767c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0000f0160)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc00041a8a0 sp=0xc00041a810 pc=0x10d154
os/exec.(*Cmd).Run(0xc0000f0160)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc00041a8c8 sp=0xc00041a8a0 pc=0x10bf80
cmd/internal/testdir_test.test.run.func1({0xc00041b508?, 0x1c1fcf?, 0xc000474000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7b8 fp=0xc00041aa38 sp=0xc00041a8c8 pc=0x16c438
cmd/internal/testdir_test.test.run({{{0xc0000b0860, 0x10}, 0xc00011a0e0}, 0xc0003b51e0, {0x1c1616, 0x9}, {0xc00017adb0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x21c4 fp=0xc00041bea8 sp=0xc00041aa38 pc=0x167684
cmd/internal/testdir_test.Test.func1(0xc0003b51e0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x98 fp=0xc00041bf30 sp=0xc00041bea8 pc=0x162e98
testing.tRunner(0xc0003b51e0, 0xc00035b660)

— watchflakes

gopherbot pushed a commit that referenced this issue Nov 13, 2023
@mknyszek @gopherbot
runtime: call enableMetadataHugePages and its callees on the systemstack
5f08b44 
These functions acquire the heap lock. If they're not called on the
systemstack, a stack growth could cause a self-deadlock since stack
growth may allocate memory from the page heap.

This has been a problem for a while. If this is what's plaguing the
ppc64 port right now, it's very surprising (and probably just
coincidental) that it's showing up now.

For #64050.
For #64062.
Fixes #64067.

Change-Id: I2b95dc134d17be63b9fe8f7a3370fe5b5438682f
Reviewed-on: https://go-review.googlesource.com/c/go/+/541635
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Paul Murphy <murp@ibm.com>
@gopherbot gopherbot reopened this Nov 13, 2023
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-10 20:43 linux-ppc64-sid-power10 go@4346ba34 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m59s)

syscall.Syscall6(0x110, 0x1, 0x271db, 0xc000220700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0002206a8 sp=0xc000220638 pc=0x942a8
os.(*Process).blockUntilWaitable(0xc000016480)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc000220790 sp=0xc0002206a8 pc=0xbbacc
os.(*Process).wait(0xc000016480)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000220810 sp=0xc000220790 pc=0xb64fc
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0005a5080)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0002208a0 sp=0xc000220810 pc=0x10c154
os/exec.(*Cmd).Run(0xc0005a5080)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0002208c8 sp=0xc0002208a0 pc=0x10afa0
cmd/internal/testdir_test.test.run.func1({0xc000221508?, 0x1c1f3c?, 0xc000524000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7b8 fp=0xc000220a38 sp=0xc0002208c8 pc=0x16b138
cmd/internal/testdir_test.test.run({{{0xc0000b0850, 0x10}, 0xc00011a0e0}, 0xc000185ba0, {0x1c1536, 0x9}, {0xc000178db0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x22f0 fp=0xc000221ea8 sp=0xc000220a38 pc=0x166470
cmd/internal/testdir_test.Test.func1(0xc000185ba0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x98 fp=0xc000221f30 sp=0xc000221ea8 pc=0x161af8
testing.tRunner(0xc000185ba0, 0xc0001a2300)
2023-11-11 02:02 linux-ppc64-sid-power10 go@8da6405e cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m59s)

syscall.Syscall6(0x110, 0x1, 0x26f2d, 0xc000494700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0004946a8 sp=0xc000494638 pc=0x942a8
os.(*Process).blockUntilWaitable(0xc00049c960)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc000494790 sp=0xc0004946a8 pc=0xbbacc
os.(*Process).wait(0xc00049c960)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000494810 sp=0xc000494790 pc=0xb64fc
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc000536dc0)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0004948a0 sp=0xc000494810 pc=0x10c154
os/exec.(*Cmd).Run(0xc000536dc0)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0004948c8 sp=0xc0004948a0 pc=0x10afa0
cmd/internal/testdir_test.test.run.func1({0xc000495508?, 0x1c1f3c?, 0xc00047c000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7b8 fp=0xc000494a38 sp=0xc0004948c8 pc=0x16b138
cmd/internal/testdir_test.test.run({{{0xc0000b0850, 0x10}, 0xc00011a0e0}, 0xc0003b31e0, {0x1c1536, 0x9}, {0xc000178db0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x22f0 fp=0xc000495ea8 sp=0xc000494a38 pc=0x166470
cmd/internal/testdir_test.Test.func1(0xc0003b31e0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x98 fp=0xc000495f30 sp=0xc000495ea8 pc=0x161af8
testing.tRunner(0xc0003b31e0, 0xc000359660)

— watchflakes

@mknyszek
Copy link
Contributor

These are failures from before the fix.

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test"
2023-11-10 21:09 linux-ppc64le-power9osu go@ea14b633 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/rangegen.go (19m47s)

syscall.Syscall6(0x110, 0x1, 0x369b4, 0xc000490700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0004906a8 sp=0xc000490638 pc=0x93f28
os.(*Process).blockUntilWaitable(0xc0003b0000)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc000490790 sp=0xc0004906a8 pc=0xbaf4c
os.(*Process).wait(0xc0003b0000)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000490810 sp=0xc000490790 pc=0xb5b3c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0001fc000)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0004908a0 sp=0xc000490810 pc=0x10aeb4
os/exec.(*Cmd).Run(0xc0001fc000)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0004908c8 sp=0xc0004908a0 pc=0x109ce0
cmd/internal/testdir_test.test.run.func1({0xc00033e420?, 0x1c2894?, 0xc000e6c000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000490a38 sp=0xc0004908c8 pc=0x16a3bc
cmd/internal/testdir_test.test.run({{{0xc00000e990, 0x10}, 0xc00005e150}, 0xc0001a8680, {0x1c03d8, 0x1}, {0xc00000edc0, 0xb}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1076 +0x33b4 fp=0xc000491ea8 sp=0xc000490a38 pc=0x1666b4
cmd/internal/testdir_test.Test.func1(0xc0001a8680)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000491f30 sp=0xc000491ea8 pc=0x160cc4
testing.tRunner(0xc0001a8680, 0xc00001ee00)
2023-11-10 21:25 linux-ppc64-sid-buildlet go@31887586 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/rangegen.go (19m48s)

syscall.Syscall6(0x110, 0x1, 0x26a8d, 0xc000234708, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0002346b0 sp=0xc000234640 pc=0x931a8
os.(*Process).blockUntilWaitable(0xc00046c000)
	/workdir/go/src/os/wait_waitid.go:32 +0x94 fp=0xc000234798 sp=0xc0002346b0 pc=0xba544
os.(*Process).wait(0xc00046c000)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000234818 sp=0xc000234798 pc=0xb511c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc000072000)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0002348a8 sp=0xc000234818 pc=0x10a864
os/exec.(*Cmd).Run(0xc000072000)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0002348d0 sp=0xc0002348a8 pc=0x109680
cmd/internal/testdir_test.test.run.func1({0xc000550060?, 0x1c2824?, 0xc000e20000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000234a40 sp=0xc0002348d0 pc=0x169d8c
cmd/internal/testdir_test.test.run({{{0xc0001b4190, 0x10}, 0xc0001d0000}, 0xc00029a340, {0x1c0299, 0x1}, {0xc0001b45d0, 0xb}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1076 +0x358c fp=0xc000235ea8 sp=0xc000234a40 pc=0x16615c
cmd/internal/testdir_test.Test.func1(0xc00029a340)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000235f30 sp=0xc000235ea8 pc=0x1604f4
testing.tRunner(0xc00029a340, 0xc0001aa7f0)
2023-11-10 21:25 linux-ppc64le-power9osu go@31887586 cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/fixedbugs/issue9110.go (19m55s)

syscall.Syscall6(0x110, 0x1, 0x37679, 0xc000470700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0004706a8 sp=0xc000470638 pc=0x93f28
os.(*Process).blockUntilWaitable(0xc000017860)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc000470790 sp=0xc0004706a8 pc=0xbaf4c
os.(*Process).wait(0xc000017860)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000470810 sp=0xc000470790 pc=0xb5b3c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0001e1340)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0004708a0 sp=0xc000470810 pc=0x10aeb4
os/exec.(*Cmd).Run(0xc0001e1340)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0004708c8 sp=0xc0004708a0 pc=0x109ce0
cmd/internal/testdir_test.test.run.func1({0xc000471508?, 0x1c1fcf?, 0xc0000e8000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000470a38 sp=0xc0004708c8 pc=0x16a3bc
cmd/internal/testdir_test.test.run({{{0xc00000e990, 0x10}, 0xc00005e150}, 0xc000214d00, {0x1c1616, 0x9}, {0xc0002a0ee0, 0xc}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1035 +0x2240 fp=0xc000471ea8 sp=0xc000470a38 pc=0x165540
cmd/internal/testdir_test.Test.func1(0xc000214d00)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000471f30 sp=0xc000471ea8 pc=0x160cc4
testing.tRunner(0xc000214d00, 0xc0002061e0)
2023-11-11 02:02 linux-ppc64-sid-buildlet go@8da6405e cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/rangegen.go (19m41s)

syscall.Syscall6(0x110, 0x1, 0x268c8, 0xc000074708, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000746b0 sp=0xc000074640 pc=0x931a8
os.(*Process).blockUntilWaitable(0xc000154000)
	/workdir/go/src/os/wait_waitid.go:32 +0x94 fp=0xc000074798 sp=0xc0000746b0 pc=0xba544
os.(*Process).wait(0xc000154000)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000074818 sp=0xc000074798 pc=0xb511c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0000e02c0)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000748a8 sp=0xc000074818 pc=0x10a864
os/exec.(*Cmd).Run(0xc0000e02c0)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000748d0 sp=0xc0000748a8 pc=0x109680
cmd/internal/testdir_test.test.run.func1({0xc00039b260?, 0x1c2824?, 0xc000e08000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000074a40 sp=0xc0000748d0 pc=0x169d8c
cmd/internal/testdir_test.test.run({{{0xc000222190, 0x10}, 0xc00023e000}, 0xc0002901a0, {0x1c0299, 0x1}, {0xc0002225d0, 0xb}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1076 +0x358c fp=0xc000075ea8 sp=0xc000074a40 pc=0x16615c
cmd/internal/testdir_test.Test.func1(0xc0002901a0)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000075f30 sp=0xc000075ea8 pc=0x1604f4
testing.tRunner(0xc0002901a0, 0xc00021a7f0)
2023-11-11 02:02 linux-ppc64le-power9osu go@8da6405e cmd/internal/testdir.Test (log) 
panic: test timed out after 20m0s
running tests:
	Test (20m0s)
	Test/rangegen.go (19m47s)

syscall.Syscall6(0x110, 0x1, 0x36933, 0xc000090700, 0x1000004, 0x0, 0x0)
	/workdir/go/src/syscall/syscall_linux.go:91 +0x28 fp=0xc0000906a8 sp=0xc000090638 pc=0x93f28
os.(*Process).blockUntilWaitable(0xc0004ec000)
	/workdir/go/src/os/wait_waitid.go:32 +0x8c fp=0xc000090790 sp=0xc0000906a8 pc=0xbaf4c
os.(*Process).wait(0xc0004ec000)
	/workdir/go/src/os/exec_unix.go:22 +0x3c fp=0xc000090810 sp=0xc000090790 pc=0xb5b3c
os.(*Process).Wait(...)
	/workdir/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc000128160)
	/workdir/go/src/os/exec/exec.go:897 +0x54 fp=0xc0000908a0 sp=0xc000090810 pc=0x10aeb4
os/exec.(*Cmd).Run(0xc000128160)
	/workdir/go/src/os/exec/exec.go:607 +0x50 fp=0xc0000908c8 sp=0xc0000908a0 pc=0x109ce0
cmd/internal/testdir_test.test.run.func1({0xc000000300?, 0x1c2894?, 0xc001010000?})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:659 +0x7cc fp=0xc000090a38 sp=0xc0000908c8 pc=0x16a3bc
cmd/internal/testdir_test.test.run({{{0xc0001a21a0, 0x10}, 0xc0001be000}, 0xc000202000, {0x1c03d8, 0x1}, {0xc0001a25d0, 0xb}})
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:1076 +0x33b4 fp=0xc000091ea8 sp=0xc000090a38 pc=0x1666b4
cmd/internal/testdir_test.Test.func1(0xc000202000)
	/workdir/go/src/cmd/internal/testdir/testdir_test.go:136 +0x94 fp=0xc000091f30 sp=0xc000091ea8 pc=0x160cc4
testing.tRunner(0xc000202000, 0xc0001987f0)

— watchflakes

@pmur
Copy link
Contributor

pmur commented Nov 13, 2023 •
edited

These are unrelated timeout failures due to #64067.

@gopherbot
Copy link
Author

Change https://go.dev/cl/541955 mentions this issue: [release-branch.go1.21] runtime: call enableMetadataHugePages and its callees on the systemstack

gopherbot pushed a commit that referenced this issue Nov 28, 2023
@mknyszek @gopherbot
[release-branch.go1.21] runtime: call enableMetadataHugePages and its…
3684d19 
… callees on the systemstack

These functions acquire the heap lock. If they're not called on the
systemstack, a stack growth could cause a self-deadlock since stack
growth may allocate memory from the page heap.

This has been a problem for a while. If this is what's plaguing the
ppc64 port right now, it's very surprising (and probably just
coincidental) that it's showing up now.

For #64050.
For #64062.
For #64067.
Fixes #64073.

Change-Id: I2b95dc134d17be63b9fe8f7a3370fe5b5438682f
Reviewed-on: https://go-review.googlesource.com/c/go/+/541635
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Paul Murphy <murp@ibm.com>
(cherry picked from commit 5f08b44)
Reviewed-on: https://go-review.googlesource.com/c/go/+/541955
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Test Flakes
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mknyszek @bcmills @gopherbot @pmur