Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: crash in memmove on loongson 3a4000 (mips64le) #50728

Open
mei-rune opened this issue Jan 21, 2022 · 12 comments
Open

runtime: crash in memmove on loongson 3a4000 (mips64le) #50728

mei-rune opened this issue Jan 21, 2022 · 12 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@mei-rune
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.13.15 linux/mips64le

Does this issue reproduce with the latest release?

fail

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="mips64le"
GOBIN=""
GOCACHE="/home/lshf/.cache/go-build"
GOENV="/home/lshf/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="mips64le"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/lshf/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/lshf/go1.17"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/lshf/go1.17/pkg/tool/linux_mips64le"
GCCGO="gccgo"
GOMIPS64="hardfloat"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -mabi=64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build062192263=/tmp/go-build -gno-record-gcc-switches"
[lshf@localhost ~]$

What did you do?

  1. I build go1.13.15 with mips64le on the centos7 amd64, and golang version is 1.4
    GOOS=linux GOARCH=mips64le ./bootstrap.bash

  2. copy go-linux-mips64le-bootstrap.tbz to target host
    target host:
    Linux localhost.localdomain 3.10.0-693.lns7.18.mips64el cmd/cgo: fails with gcc 4.4.1 #1 SMP PREEMPT Fri Jul 10 15:28:05 CST 2020 mips64 mips64 mips64 GNU/Linux
    target os:
    Loongnix-Server release 1.7.2007 (Core)
    Loongnix-Server release 1.7.2007 (Core)
    NAME="Loongnix"
    VERSION="7 (Core)"
    ID="loongnix"
    ID_LIKE="rhel fedora"
    VERSION_ID="7"
    PRETTY_NAME="Loongnix 7 (Core)"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:loongnix:loongnix:7"
    HOME_URL="https://www.loongnix.org/"
    BUG_REPORT_URL="https://bugs.loongnix.org/"

                  Loongnix-Server release 1.7.2007 (Core)
              Loongnix-Server release 1.7.2007 (Core)

  3. compile go1.13 with go-linux-mips64le-bootstrap.tbz , and successful.

  4. compile go1.17 with go1.13 , and failure

What did you expect to see?

build go1.17 successful

What did you see instead?

error.txt
@ianlancetaylor
Copy link
Contributor

Go 1.17 doesn't support loongarch.

@mei-rune
Copy link
Author

@ianlancetaylor loong 3a4000 is not loongarch, it is mips64le.
loong 3a5000 is loongarch, loong 3a4000 and 3a5000 is not same.

@ianlancetaylor
Copy link
Contributor

OK, reopening. But the Go team does not have access to any loong 3a4000 systems, and our mips64le systems are not showing any problems. This will most likely have to be investigated by somebody with access to a machine that shows the failure.

@ianlancetaylor ianlancetaylor changed the title golang/src: I build go1.17.6 fail in loongarch 3a4000 runtime: crash in memmove on loongarch 3a4000 (mips64le) Jan 21, 2022
@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 21, 2022
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Jan 21, 2022
@mei-rune
Copy link
Author

mei-rune commented Jan 22, 2022 •
edited

@ianlancetaylor I send user and password to your mail (ian@airs.com).

No privacy or secret on this host!

@mengzhuo
Copy link
Contributor

OK, reopening. But the Go team does not have access to any loong 3a4000 systems, and our mips64le systems are not showing any problems. This will most likely have to be investigated by somebody with access to a machine that shows the failure.

@ianlancetaylor there is a loong 3a4000 (host-linux-mipsle-mengzhuo) in the build farmer, you can access it with gomote

p.s. I've tried on mips64le builder (Loongson 3A4K, go 1.13.15 to bootstrap 1.17 ) and it works fine.

error log shows an access memory to 0 at memmove_mips64x.s:97

unexpected fault address 0x0
fatal error: fault
[signal SIGBUS: bus error code=0x80 addr=0x0 pc=0x919a0]

go/src/runtime/memmove_mips64x.s

Line 97 in ec51703

MOVV R4, (R6)

@mei-rune
Copy link
Author

mei-rune commented Jan 22, 2022 •
edited

@mengzhuo Can you provide your email if you are interested.

@randall77
Copy link
Contributor

Why 1.13 and multiple builds? Can't you just bootstrap build 1.17 on your amd64 system and use that result?

I can't get your error.txt file - it appears empty.

Is this reproducible? Can you figure out what test is failing, set a breakpoint on the faulting instruction, and see where the presumably bad R6 is coming from? Is it a bad argument to memmove? From where? Or is it a bug in memmove itself?

@mei-rune
Copy link
Author

mei-rune commented Jan 23, 2022 •
edited

@randall77 I send mail to your mail (keithr@alum.mit.edu)

** go1.17 amd64 - > go.1.17 mips64le bootstrap - > go.1.17 mips64le**

  1. download go1.17.5 on my amd64 system
  2. download go1.17.6 source on my amd64 system
  3. bootstrap build 1.17 on my amd64 system
  4. copy bootstrap build result to my loongson 3a4000
  5. download go1.17.6 source on my loongson 3a4000
  6. build it fail

output.log

** go1.4 amd64 - > go.1.17 mips64le bootstrap - > go.1.17 mips64le**

  1. download go1.4 on my amd64 system
  2. download go1.17.6 source on my amd64 system
  3. bootstrap build 1.17 on my amd64 system
  4. copy bootstrap build result to my loongson 3a4000
  5. download go1.17.6 source on my loongson 3a4000
  6. build it fail

output-1.17.log

Is this reproducible? yes! but I`m a junior programmer, don't know how to debug

@mei-rune mei-rune changed the title runtime: crash in memmove on loongarch 3a4000 (mips64le) runtime: crash in memmove on loongson 3a4000 (mips64le) Jan 23, 2022
@randall77
Copy link
Contributor

output.log is empty.
output-1.17.log is more informative. What command was run to generate that log? It shows two errors, the one you showed above and another one:

warning: GOPATH set to GOROOT (/home/lshf/go1.17) has no effect
Building Go cmd/dist using /home/lshf/golang_bootstrap_1.17_by_1.4/go-linux-mips64le-bootstrap. (go1.17.6 linux/mips64le)
Building Go toolchain1 using /home/lshf/golang_bootstrap_1.17_by_1.4/go-linux-mips64le-bootstrap.
# bootstrap/sort
panic: runtime error: index out of range [48087] with length 282

goroutine 113 [running]:
cmd/internal/obj/mips.(*ctxt0).oplook(0xc000b5bd58, 0xc0001fba20)
        /root/go-linux-mips64le-bootstrap/src/cmd/internal/obj/mips/asm0.go:756 +0x224
cmd/internal/obj/mips.span0(0xc000398200, 0xc000481380, 0xc000424060)
        /root/go-linux-mips64le-bootstrap/src/cmd/internal/obj/mips/asm0.go:431 +0x260
cmd/internal/obj.Flushplist(0xc000398200, 0xc000b5bf40, 0xc000424060, {0xffffb871cd, 0xe})
        /root/go-linux-mips64le-bootstrap/src/cmd/internal/obj/plist.go:134 +0x5f4
cmd/compile/internal/objw.(*Progs).Flush(...)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/objw/prog.go:124
cmd/compile/internal/ssagen.Compile(0xc0001271e0, 0x1)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/ssagen/pgen.go:190 +0x67c
cmd/compile/internal/gc.compileFunctions.func4.1(0x1)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/gc/compile.go:153 +0x58
cmd/compile/internal/gc.compileFunctions.func3.1(0xc000034080, 0xc000788220)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/gc/compile.go:140 +0x64
created by cmd/compile/internal/gc.compileFunctions.func3
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/gc/compile.go:138 +0x5c
# regexp
unexpected fault address 0x0
fatal error: fault
[signal SIGBUS: bus error code=0x80 addr=0x0 pc=0x12e14]

goroutine 60 [running]:
runtime.throw({0xb093d9, 0x5})
        /root/go-linux-mips64le-bootstrap/src/runtime/panic.go:1198 +0x6c fp=0xc00115bb10 sp=0xc00115bae8 pc=0x55214
runtime.sigpanic()
        /root/go-linux-mips64le-bootstrap/src/runtime/signal_unix.go:732 +0x1e4 fp=0xc00115bb40 sp=0xc00115bb10 pc=0x73eec
runtime.interhash(0xc00115bd10, 0xa46dd2b27f70fc2)
        /root/go-linux-mips64le-bootstrap/src/runtime/alg.go:102 +0x3c fp=0xc00115bb88 sp=0xc00115bb48 pc=0x12e14
type..hash.cmd/compile/internal/ssa.vkey(0xc00115bd00, 0xa524b086)
        <autogenerated>:1 +0x8c fp=0xc00115bbb0 sp=0xc00115bb88 pc=0x7db274
runtime.mapaccess1(0xa98400, 0xc00115bd28, 0xc00115bd00)
        /root/go-linux-mips64le-bootstrap/src/runtime/map.go:413 +0x7c fp=0xc00115bbe8 sp=0xc00115bbb0 pc=0x2038c
cmd/compile/internal/ssa.zcse(0xc001068000)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/ssa/zcse.go:22 +0x5a8 fp=0xc00115bee8 sp=0xc00115bbe8 pc=0x7d5020
cmd/compile/internal/ssa.Compile(0xc001068000)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/ssa/compile.go:96 +0xc6c fp=0xc00115fb60 sp=0xc00115bee8 pc=0x3e206c
cmd/compile/internal/ssagen.buildssa(0xc00008a2c0, 0x0)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/ssagen/ssa.go:642 +0x1a6c fp=0xc00115feb0 sp=0xc00115fb60 pc=0x7f5074
cmd/compile/internal/ssagen.Compile(0xc00008a2c0, 0x0)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/ssagen/pgen.go:165 +0x54 fp=0xc00115ff68 sp=0xc00115feb0 pc=0x7e91c4
cmd/compile/internal/gc.compileFunctions.func4.1(0x0)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/gc/compile.go:153 +0x58 fp=0xc00115ffa0 sp=0xc00115ff68 pc=0xa10f60
cmd/compile/internal/gc.compileFunctions.func3.1(0xc000034280, 0xc00037a220)
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/gc/compile.go:140 +0x64 fp=0xc00115ffc8 sp=0xc00115ffa0 pc=0xa11094
runtime.goexit()
        /root/go-linux-mips64le-bootstrap/src/runtime/asm_mips64x.s:632 +0x4 fp=0xc00115ffc8 sp=0xc00115ffc8 pc=0x95354
created by cmd/compile/internal/gc.compileFunctions.func3
        /root/go-linux-mips64le-bootstrap/src/cmd/compile/internal/gc/compile.go:138 +0x5c

(+ some other goroutines)

I suspect the first error is the salient one.

I think a mips64le maintainer should take a look.
@mengzhuo (and @krytarowski ?)

Can the bootstrap-built distribution build and run a hello world program? Or is it just when trying to build the whole distribution again that it fails?

@mei-rune
Copy link
Author

@mengzhuo and @krytarowski

I`m a junior programmer, don't know how to debug. but I can privode the user accout of loongson hosts!

@mengzhuo
Copy link
Contributor

mengzhuo commented Jan 24, 2022 •
edited

@randall77 I've tried to reproduce by the OP's bootstrap approach (3 times)
The build is fine with everything.
These unexpected memory errors occurs to me that the 3A4000 machine run in build farmer had the same issue before.
After I changed the RAM (physically) to a more stable one and all the panic gone.

@runner-mei Maybe you can run a memtest on your machine and AFAIK the original distribution from Loongson (Loognix) runs on a buggy fork of Linux (3.19)
Or could you upload the bootstrap.tbz?

Here is my build log and script

git name-rev --tags --name-only $(git rev-parse HEAD) && PATH=$HOME/go-linux-mips64le-bootstrap/bin:$PATH ./make.bash

go1.17.6
Building Go cmd/dist using /root/go-linux-mips64le-bootstrap. (go1.17 linux/mips64le)
Building Go toolchain1 using /root/go-linux-mips64le-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/mips64le.
---
Installed Go for linux/mips64le in /root/godev
Installed commands in /root/godev/bin

kernel 5.4.110-3.fc28.lemote.mips64el
gcc 8.4.0

@guoh1988
Copy link

guoh1988 commented Feb 26, 2022 •
edited

@mengzhuo
build golang 1.16.14
build in loongnix 3A4000 + kernel 3.10 error

# uname -a
Linux master3 3.10.0 #1 SMP PREEMPT Tue Dec 1 16:45:16 CST 2020  mips64 mips64 mips64 GNU/Linux
GOOS=linux GOARCH=mips64le ./bootstrap.bash 
#### Copying to ../../go-linux-mips64le-bootstrap

#### Cleaning ../../go-linux-mips64le-bootstrap

#### Building ../../go-linux-mips64le-bootstrap

Building Go cmd/dist using /usr/local/go. (go1.14.4 linux/mips64le)
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
unexpected fault address 0x0
fatal error: fault
[signal SIGBUS: bus error code=0x80 addr=0x0 pc=0x26dfc]

goroutine 1086 [running]:
runtime.throw(0x4a8bcc, 0x5)
	/opt/go-linux-mips64le-bootstrap/src/runtime/panic.go:1117 +0x6c fp=0xc00008ec70 sp=0xc00008ec48 pc=0x4f9b4
runtime.sigpanic()
	/opt/go-linux-mips64le-bootstrap/src/runtime/signal_unix.go:731 +0x3dc fp=0xc00008eca0 sp=0xc00008ec70 pc=0x6c814
runtime.evacuate_faststr(0x441040, 0xc0002f2570, 0x26)
	/opt/go-linux-mips64le-bootstrap/src/runtime/map_faststr.go:425 +0x124 fp=0xc00008ed50 sp=0xc00008eca8 pc=0x26dfc
runtime.growWork_faststr(0x441040, 0xc0002f2570, 0xad)
	/opt/go-linux-mips64le-bootstrap/src/runtime/map_faststr.go:394 +0xc8 fp=0xc00008ed70 sp=0xc00008ed50 pc=0x26cc0
runtime.mapassign_faststr(0x441040, 0xc0002f2570, 0xc00050d8b0, 0x47, 0xc00034b000)
	/opt/go-linux-mips64le-bootstrap/src/runtime/map_faststr.go:226 +0x434 fp=0xc00008edd0 sp=0xc00008ed70 pc=0x2662c
cmd/go/internal/cache.SetFileHash(0xc00050d8b0, 0x47, 0x55b463fadb1e3de6, 0x838b8ad8ecc0b705, 0xf769a96203a971a3, 0x184a198f91682ce5)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/cache/hash.go:172 +0xbc fp=0xc00008ee00 sp=0xc00008edd0 pc=0x219894
cmd/go/internal/cache.FileHash(0xc00050d8b0, 0x47, 0x0, 0x0, 0x0, 0x0, 0x47, 0x0)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/cache/hash.go:162 +0x518 fp=0xc00008eef8 sp=0xc00008ee00 pc=0x2195e8
cmd/go/internal/work.(*Builder).fileHash(0xc0004af860, 0xc00050d8b0, 0x47, 0xc00050d8b0, 0x47)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/buildid.go:379 +0x60 fp=0xc00008ef80 sp=0xc00008eef8 pc=0x361628
cmd/go/internal/work.(*Builder).buildActionID(0xc0004af860, 0xc0002b4000, 0x0, 0x0, 0x0, 0x0)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:345 +0xabc fp=0xc00008f3a8 sp=0xc00008ef80 pc=0x3653ec
cmd/go/internal/work.(*Builder).build(0xc0004af860, 0x54a810, 0xc000022158, 0xc0002b4000, 0x0, 0x0)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:426 +0x5010 fp=0xc00008fd98 sp=0xc00008f3a8 pc=0x36c548
cmd/go/internal/work.(*Builder).Do.func2(0x54a810, 0xc000022158, 0xc0002b4000)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:136 +0x56c fp=0xc00008ff08 sp=0xc00008fd98 pc=0x39f92c
cmd/go/internal/work.(*Builder).Do.func3(0xc0003d6640, 0xc000352ea0, 0xc0004af860, 0xc00039f578)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:198 +0xb4 fp=0xc00008ffb8 sp=0xc00008ff08 pc=0x39fb34
runtime.goexit()
	/opt/go-linux-mips64le-bootstrap/src/runtime/asm_mips64x.s:631 +0x4 fp=0xc00008ffb8 sp=0xc00008ffb8 pc=0x8e6c4
created by cmd/go/internal/work.(*Builder).Do
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:184 +0x420

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc000352ea8)
	/opt/go-linux-mips64le-bootstrap/src/runtime/sema.go:56 +0x4c
sync.(*WaitGroup).Wait(0xc000352ea0)
	/opt/go-linux-mips64le-bootstrap/src/sync/waitgroup.go:130 +0xac
cmd/go/internal/work.(*Builder).Do(0xc0004af860, 0x54a810, 0xc000022158, 0xc0002bb400)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:207 +0x448
cmd/go/internal/work.InstallPackages(0x54a810, 0xc000022158, 0xc00001e0e0, 0x4, 0x4, 0xc000220920, 0x4, 0x4)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/build.go:724 +0x4c4
cmd/go/internal/work.runInstall(0x54a810, 0xc000022158, 0x722860, 0xc00001e0e0, 0x4, 0x4)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/build.go:632 +0x29c
main.main()
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/main.go:195 +0x99c

goroutine 1083 [select]:
cmd/go/internal/work.(*Builder).Do.func3(0xc0003d6640, 0xc000352ea0, 0xc0004af860, 0xc00039f578)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:188 +0x110
created by cmd/go/internal/work.(*Builder).Do
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:184 +0x420

goroutine 1084 [select]:
cmd/go/internal/work.(*Builder).Do.func3(0xc0003d6640, 0xc000352ea0, 0xc0004af860, 0xc00039f578)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:188 +0x110
created by cmd/go/internal/work.(*Builder).Do
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:184 +0x420

goroutine 1085 [select]:
cmd/go/internal/work.(*Builder).Do.func3(0xc0003d6640, 0xc000352ea0, 0xc0004af860, 0xc00039f578)
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:188 +0x110
created by cmd/go/internal/work.(*Builder).Do
	/opt/go-linux-mips64le-bootstrap/src/cmd/go/internal/work/exec.go:184 +0x420
go tool dist: FAILED: /opt/go-linux-mips64le-bootstrap/pkg/tool/linux_mips64le/go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2

build in loongnix 3A4000 + kernel 4.19 is ok

Linux 192.168.1.11 4.19.90-2.lns7.1.mips64el #1 SMP PREEMPT Thu Jul 23 15:15:38 CST 2020 mips64 mips64 mips64 GNU/Linux

# GOOS=linux GOARCH=mips64le ./bootstrap.bash
#### Copying to ../../go-linux-mips64le-bootstrap

#### Cleaning ../../go-linux-mips64le-bootstrap

#### Building ../../go-linux-mips64le-bootstrap

Building Go cmd/dist using /usr/local/go. (go1.14.4 linux/mips64le)
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/mips64le.
----
Bootstrap toolchain for linux/mips64le installed in /mnt/go-linux-mips64le-bootstrap.
Building tbz.
-rw-r--r-- 1 root root 119017462 2月  26 10:19 /mnt/go-linux-mips64le-bootstrap.tbz

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
6 participants
@mengzhuo @mei-rune @ianlancetaylor @randall77 @gopherbot @guoh1988