Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

windows/arm: fatal error: minpc or maxpc invalid #42786

Closed
zx2c4 opened this issue Nov 23, 2020 · 18 comments
Closed

windows/arm: fatal error: minpc or maxpc invalid #42786

zx2c4 opened this issue Nov 23, 2020 · 18 comments
Labels
arch-arm Issues solely affecting the 32-bit arm architecture. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows

Comments

@zx2c4
Copy link
Contributor

zx2c4 commented Nov 23, 2020

Now that we've got the windows/arm builder running, we can start looking at the first failures. So far there are no successful tests. That could indicate an issue with the test rig, or it might indicate a latent problem with windows/arm, while it's bitrotted without a builder.

Various issues seen on the farmer dashboard:

Building Go cmd/dist using C:\Program Files (Arm)\Go
Building Go toolchain1 using C:\Program Files (Arm)\Go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: minpc or maxpc invalid
runtime: panic before malloc heap initialized

runtime stack:
fatal error: findfunc: bad findfunctab entry idx
runtime: panic before malloc heap initialized
panic during panic

runtime stack:
fatal error: findfunc: bad findfunctab entry idx
runtime: panic before malloc heap initialized
stack trace unavailable
go tool dist: FAILED: C:\workdir\go\pkg\tool\windows_arm\go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 4
Building Go cmd/dist using C:\Program Files (Arm)\Go
Building Go toolchain1 using C:\Program Files (Arm)\Go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: minpc or maxpc invalid
runtime: panic before malloc heap initialized

runtime stack:
fatal: morestack on g0
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
runtime: signal received on thread not created by Go.
Building Go cmd/dist using C:\Program Files (Arm)\Go
Building Go toolchain1 using C:\Program Files (Arm)\Go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: minpc or maxpc invalid
runtime: panic before malloc heap initialized

runtime stack:
cmd/go/internal/work.init()
	C:/workdir/go/src/cmd/go/internal/work/security.go:198 +0x2400 fp=0x12ffc60 sp=0x12ffc4c pc=0x6f6c6c
cmd/go/internal/modcmd.runVendor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	C:/workdir/go/src/cmd/go/internal/modcmd/vendor.go:124 +0x904 fp=0x12fffe0 sp=0x12ffc60 pc=0x710e4c
runtime: unexpected return pc for cmd/go/internal/clean.clean called from 0x0
stack: frame={sp:0x12fffe0, fp:0x1300198} stack=[0x12efd60,0x12ffcf8)

cmd/go/internal/clean.clean(0x0)
	C:/workdir/go/src/cmd/go/internal/clean/clean.go:237 +0x14 fp=0x1300198 sp=0x12fffe0 pc=0x6fa12c
go tool dist: FAILED: C:\workdir\go\pkg\tool\windows_arm\go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2

CC @bcmills @alexbrainman @bradfitz @cagedmantis

@ALTree ALTree added OS-Windows arch-arm Issues solely affecting the 32-bit arm architecture. labels Nov 23, 2020
@ALTree
Copy link
Member

ALTree commented Nov 23, 2020

Note that you reported essentially the same crash at #39465. I suggest keeping only one open (the old one has no additional info anyway).

@ALTree ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 23, 2020
@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 23, 2020

Note that you reported essentially the same crash at #39465. I suggest keeping only one open (the old one has no additional info anyway).

Oh thanks! I had completely forgotten about that! I was in the process of beefing up the hardware, thinking that maybe this was just a out of memory issue, but now clearly that's not it. Time for me to jump into the actual bug I guess.

@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 24, 2020

Looking at the farmer dashboard, things seem to be okay on the 1.14 release branch but are broken by 1.15.

@alexbrainman
Copy link
Member

Looking at the farmer dashboard, things seem to be okay on the 1.14 release branch but are broken by 1.15.

Use git bisect to find commit that breaks. No?

Alex

@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 24, 2020

Looking at the farmer dashboard, things seem to be okay on the 1.14 release branch but are broken by 1.15.

Use git bisect to find commit that breaks. No?

Seems reasonable. I'm in the process of installing a beefier drive so I can do that performantly.

@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 25, 2020

Bisection identified c364079 as the first bad commit.

commit c364079a533c172bd071af38b6f6ffd7dc68186d
Author: Cherry Zhang <cherryyz@google.com>
Date:   Wed Apr 15 22:23:41 2020 -0400

    [dev.link] cmd/link: use function address directly in pclntab generation

    If we are internal linking a static executable, in pclntab
    generation, the function addresses are known, so we can just use
    them directly instead of emitting relocations.

    For external linking or other build modes,  we are generating a
    relocatable binary so we still need to emit relocations.

    Reduce some allocations: for linking cmd/compile,

    name           old alloc/op   new alloc/op   delta
    Pclntab_GC       38.8MB ± 0%    36.4MB ± 0%   -6.19%  (p=0.008 n=5+5)

    TODO: can we also do this in DWARF generation?

    Change-Id: I43920d930ab1da97c205871027e01844a07a5e60
    Reviewed-on: https://go-review.googlesource.com/c/go/+/228478
    Run-TryBot: Cherry Zhang <cherryyz@google.com>
    TryBot-Result: Gobot Gobot <gobot@golang.org>
    Reviewed-by: Than McIntosh <thanm@google.com>

 src/cmd/link/internal/ld/pcln.go   | 28 ++++++++++++++++++++--------
 src/cmd/link/internal/ld/target.go |  4 ++++
 2 files changed, 24 insertions(+), 8 deletions(-)
bisect run success

cc @cherrymui @thanm

@zx2c4 zx2c4 changed the title windows/arm: build failures with new builder windows/arm: fatal error: minpc or maxpc invalid Nov 25, 2020
@jeremyfaller
Copy link
Contributor

The flagged CL is almost certainly causing it. I will take a look and try to fix.

Is there gomotes available?

@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 26, 2020

Is there gomotes available?

I'm not super familiar with how the gomote infrastructure works. I assume I run some sort of trojan on my arm machine, and then you're able to easily run commands on it through the testing infra? If so, let me know what you want me to run, and I'll do that. Otherwise, I'm happy to just test patches out manually. Or, can't you tell gobot to use the new builder for trybot for a CL?

@jeremyfaller
Copy link
Contributor

I am not familiar with the infra beyond, "it just works, most of the time." This will likely be difficult to debug without access to a machine. Perhaps @dmitshur can fill us in on what needs to happen to get access to this machine? I checked gomote create and it was listing windows-arm-zx2c4 as a valid gomote, but gomote ssh failed.

@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 26, 2020

Can you just use 'gomote put' and 'gomote run' instead? In theory that should be enough, right? There's also 'gomote rdp', though I'm not sure how authentication works there.

@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 26, 2020

When it crashes, I'm seeing:

datap.minpc, datap.maxpc = 0xb11000, 0xe50630
datap.ftab[0].entry, datap.ftab[nftab].entry = 0x401000, 0x740630

So it looks like it's being relocated by a fixed offset.

@cherrymui
Copy link
Member

It is always PIE on Windows/ARM, right? But it doesn't set buildmode to PIE. The CL above excludes PIE (by testing ctxt.IsExe()). So, if setting buildmode to PIE works, we probably want to do that. Or we could hardcode an alwaysPIE predicate.

@dmitshur
Copy link
Contributor

Perhaps @dmitshur can fill us in on what needs to happen to get access to this machine? I checked gomote create and it was listing windows-arm-zx2c4 as a valid gomote, but gomote ssh failed.

The "windows-arm-zx2c4" builder was configured with SSH support turned off, because the HostConfig it uses has a zero value for SSHUsername field. If that configuration is changed, it should work, or it may fail for another reason. Also, gomote ssh uses a different authentication scheme, documented at https://golang.org/wiki/Gomote#gomote-ssh.

The gomote put and gomote run commands that @zx2c4 suggested are in general less finicky and easier to rely on.

@gopherbot
Copy link

Change https://golang.org/cl/273566 mentions this issue: link: windows/arm is all pie, so mark it as such

@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 26, 2020

I think I've got it fixed!

@zx2c4
Copy link
Contributor Author

zx2c4 commented Nov 26, 2020

@gopherbot please backport this to 1.15

@gopherbot
Copy link

Backport issue(s) opened: #42849 (for 1.15).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@gopherbot
Copy link

Change https://golang.org/cl/273567 mentions this issue: [release-branch.go1.15] cmd/link: windows/arm is all pie, so mark it as such

@golang golang locked and limited conversation to collaborators Nov 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-arm Issues solely affecting the 32-bit arm architecture. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Projects
None yet
Development

No branches or pull requests

7 participants