We find this problem is that go crash on Cavium Octeon machine:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960674

Since currently, golang for mips32le requires mips32r2, we can switch to FP64 directly,

I think we require only mips32r1, at least according to https://github.com/golang/go/wiki/MinimumRequirements

What's the difference between "MIPS II", "MIPS32r1", and "MIPS32r2" (at least, relevant parts for floating-point support)?

It's easy enough to ban odd-numbered FP registers. I'm not sure what is involved with changing lwc* uses.

We find this problem is that go crash on Cavium Octeon machine:

What's special about that machine? Is it something hardware-related? OS? Is it running FP64-only?

Since currently, golang for mips32le requires mips32r2, we can switch to FP64 directly,

I think we require only mips32r1, at least according to https://github.com/golang/go/wiki/MinimumRequirements

What's the difference between "MIPS II", "MIPS32r1", and "MIPS32r2" (at least, relevant parts for floating-point support)?

It's easy enough to ban odd-numbered FP registers. I'm not sure what is involved with changing lwc* uses.

For FP64/FPXX mode, we can only use ldc1 to load float64 into a single FPR.
Since it may be really a single 64bit FPR.

We find this problem is that go crash on Cavium Octeon machine:

What's special about that machine? Is it something hardware-related? OS? Is it running FP64-only?
It is a Octeon 7130 CPU
https://db.debian.org/machines.cgi?host=eller

Yes, it is in FP64 mode. We find this problem due to that we enable CONFIG_MIPS_O32_FP64_SUPPORT

For FP64/FPXX mode, we can only use ldc1 to load float64 into a single FPR.
Since it may be really a single 64bit FPR.

Is that instruction available in mips32r1?

For FP64/FPXX mode, we can only use ldc1 to load float64 into a single FPR.
Since it may be really a single 64bit FPR.

Is that instruction available in mips32r1?

Yes. ldc1 is from MIPS II, which is quite old.
and that's why FPXX can work with MIPS II but not MIPS I.

Another tip: MTC1, MFC1 should not be used to access the high part of a float64.

for mips32r1, it is also possible to support FP64, if we:
don't use MTHC1, MFHC1 to access the high part of a float64,
since MTHC1, MFHC1 are introduced in mips32r2
Instead, we have to use memory as a bridge.

@wzssyqa If I remeber corectly, GO has an issue of not being able to guarantee the 8-byte alignment for float64 on mips, at least on stack. So using ldc1 and sdc1 could trap.

This issue sounds something more like a https://go-review.googlesource.com/c/go/+/97735. Maybe the culprit is in https://github.com/golang/go/blob/master/src/runtime/preempt_mipsx.s#L78. It seems to load odd FP regs first if that means something. Maybe there are other places that do the same.

@randall77 Is there a way to run a bootstrap process w/o async preemption to check this?

GODEBUG=asyncpreemptoff=1 should turn off async preemption.

@wzssyqa If I remeber corectly, GO has an issue of not being able to guarantee the 8-byte alignment for float64 on mips, at least on stack. So using ldc1 and sdc1 could trap.

If so, we may have 2 choice:
raise ISA level to r2, so that we can use MTHC1/MFHC1
use memory to help alignment.

This issue sounds something more like a https://go-review.googlesource.com/c/go/+/97735. Maybe the culprit is in https://github.com/golang/go/blob/master/src/runtime/preempt_mipsx.s#L78. It seems to load odd FP regs first if that means something. Maybe there are other places that do the same.

@randall77 Is there a way to run a bootstrap process w/o async preemption to check this?

I think there is some alignment issue. Currently float64 is only 4-byte aligned on 32-bit architectures. If we use 64-bit FP load/store and the address is not 8-byte aligned, it faults. This is why we use two MOVFs instead of one MOVD.

Preemption is not a problem if we don't use odd-numbered FP registers. We can change preempt_mipsx.s easily.

currently, go is FP32, while it claims that it is FPXX.
So, on FP64 enabled machine, the kernel will set FPU mode to FR=1 for it,
make it cannot work.

I cannot find where it is set.

How are we claiming FPXX? Is that in the elf header somewhere? If we explicitly mark as FP32, will that fix the issue?

How are we claiming FPXX? Is that in the elf header somewhere? If we explicitly mark as FP32, will that fix the issue?

In section of .MIPS.abiflags.
objdump -s -j .MIPS.abiflags
told me that it does have this section, while I cannot find it in our source code.

the content of this section is:
typedef struct
{
/* Version of flags structure. /
uint16_t version;
/ The level of the ISA: 1-5, 32, 64. /
uint8_t isa_level;
/ The revision of ISA: 0 for MIPS V and below, 1-n otherwise. /
uint8_t isa_rev;
/ The size of general purpose registers. /
uint8_t gpr_size;
/ The size of co-processor 1 registers. /
uint8_t cpr1_size;
/ The size of co-processor 2 registers. /
uint8_t cpr2_size;
/ The floating-point ABI. /
uint8_t fp_abi;
/ Processor-specific extension. /
uint32_t isa_ext;
/ Mask of ASEs used. /
uint32_t ases;
/ Mask of general flags. */
uint32_t flags1;
uint32_t flags2;
} Elf_Internal_ABIFlags_v0;

ohh, my fault, it doesn't have this section at all.

Strange thing, this section is added by some tool of Debian...

anyway, it is time for us to consider to migrate to FPXX.

Now we get the real problem: when link with C libraires, go-link will copy the section from libc.

While in fact we should avoid to just copy .MIPS.abiflags, since
FP32 + FPXX -> FP32
FP32 + FP64 -> rejected

@wzssyqa It seems that binutils exhibits this "wrong" behavior, at least on 2.31.

$ echo "int main(void) { return 0; }" > test.c

$ gcc -EL  -mabi=32 -mfp32 test.c -c -o test.o
$ gcc -EL  -mabi=32 test.o  -o test.exe

$ readelf -a test.exe | grep FP
  Tag_GNU_MIPS_ABI_FP: Hard float (double precision)
FP ABI: Hard float (double precision)

$ objcopy --remove-section .gnu.attributes test.o
$ objcopy --remove-section .MIPS.abiflags test.o
$ gcc -EL  -mabi=32 test.o  -o test.exe

$ readelf -a test.exe | grep FP

  Tag_GNU_MIPS_ABI_FP: Hard float (32-bit CPU, Any FPU)
FP ABI: Hard float (32-bit CPU, Any FPU)

Edit: If I preserve the .gnu.attributes the abiflags gets correctly inferred. I guess we have to start emitting gnu.attributes here with Tag_GNU_MIPS_ABI_FP set to Val_GNU_MIPS_ABI_FP_DOUBLE.

Yes. I think that we should do so, aka add these attributions.
.gnu.attributes and .MIPS.abiflags etc

Ok, then on the affected machine, can someone try adding a FP32 marker to a Go binary which would otherwise fail, to see if it then passes?

@wzssyqa @randall77 I beileve just placing .gnu_attribute 4, 1 under #ifndef __mips_soft_float in src/runtime/cgo/gcc_mipsx.S should do the trick, but I cannot check it for some time. I assume that this file gets linked on mips every time we do externel linking.

That will work for cgo builds but that file won't be included (and the external linker won't be called) for pure Go builds.
We'll need internal linking fix also. Any maybe a cgo-disabled external link solution also? (Not sure why one would do that, but I think it is possible.)

I think we can just emit the marker in the linker, as long as we know what the right marker should be. Softfloat is an interesting case. What happens if we emit the marker in softfloat mode (where FP registers are not used at all)? Does it hurt? (Currently the linker doesn't know/care softfloat or not. We could plumbing the information through, if necessary.)

@cherrymui @randall77 I did a quick check linking object with only hard-float .gnu.attibutes into soft-float exe and I get a warring from bfd-ld that I'm mixing soft-float and hard-float objects.
Ideally this should emited in the linker (I'm just proposing a quick fix, that can be easily backpacked), but this only matters when we do external linking. Internal-linked binaries w/o .MIPS.abiflags are treated same as FP32 by the kernel.
Does the ARM have the same problem?

@draganmladjenovic just to be clear, do you mean that for internally linked executables it already works fine, and we only need the marker for external linking? Thanks.

@cherrymui Yes, sorry, we only need marker for external linking, because mips doesn't support internal linking w/ cgo (edit: Actually that is irrelevant, even if mips target did support that use case, marker is needed only for external linking).

@draganmladjenovic thanks for clarifying.

For ARM, #15862 seems related. But it doesn't seem as bad there.

I tried to add a .gnu.attributes section for non-cgo, or static linked exe.
It does have a .gnu.attributes section, while the kernel cannot recognize it.

I think that there must be something I am wrong.

diff --git a/src/cmd/link/internal/ld/elf.go b/src/cmd/link/internal/ld/elf.go
index 78298be..e2414c3 100644
--- a/src/cmd/link/internal/ld/elf.go
+++ b/src/cmd/link/internal/ld/elf.go
@@ -207,6 +207,7 @@ const (
 	SHT_SYMTAB_SHNDX     = 18
 	SHT_LOOS             = 0x60000000
 	SHT_HIOS             = 0x6fffffff
+	SHT_GNU_ATTRIBUTES   = 0x6ffffff5
 	SHT_GNU_VERDEF       = 0x6ffffffd
 	SHT_GNU_VERNEED      = 0x6ffffffe
 	SHT_GNU_VERSYM       = 0x6fffffff
@@ -467,6 +468,8 @@ var (
 	shdr [NSECT]*ElfShdr
 
 	interp string
+
+	BigEndian bool
 )
 
 type Elfstring struct {
@@ -487,6 +490,9 @@ var buildinfo []byte
 func Elfinit(ctxt *Link) {
 	ctxt.IsELF = true
 
+	if ctxt.IsBigEndian () {
+		BigEndian = true
+	}
 	if ctxt.Arch.InFamily(sys.AMD64, sys.ARM64, sys.MIPS64, sys.PPC64, sys.RISCV64, sys.S390X) {
 		elfRelType = ".rela"
 	} else {
@@ -826,6 +832,37 @@ func elfwriteinterp(out *OutBuf) int {
 	return int(sh.size)
 }
 
+func elfgnuattributes(sh *ElfShdr, startva uint64, resoff uint64) int {
+	n := 16
+	sh.addr = startva + resoff - uint64(n)
+	sh.off = resoff - uint64(n)
+	println("sh.off in elfgnuattributes: ", sh.off);
+	sh.size = uint64(n)
+
+	return n
+}
+
+func elfgnuattributescontent() []byte {
+	if BigEndian {
+		return []byte{0x41, 0x00, 0x00, 0x00,
+				0x0f, 0x67, 0x6e, 0x75,
+				0x00, 0x01, 0x00, 0x00,
+				0x00, 0x07, 0x04, 0x05}
+	}
+	return []byte{0x41, 0x0f, 0x00, 0x00,
+			0x00, 0x67, 0x6e, 0x75,
+			0x00, 0x01, 0x07, 0x00,
+			0x00, 0x00, 0x04, 0x05}
+}
+func elfwritegnuattributes(out *OutBuf) int {
+	sh := elfshname(".gnu.attributes")
+	sh.type_ = SHT_GNU_ATTRIBUTES
+	out.SeekSet(int64(sh.off))
+	println("sh.off in elfwritegnuattributes: ", sh.off);
+	out.Write(elfgnuattributescontent())
+	return int(sh.size)
+}
+
 func elfnote(sh *ElfShdr, startva uint64, resoff uint64, sz int) int {
 	n := 3*4 + uint64(sz) + resoff%4
 
@@ -1490,6 +1527,20 @@ func addgonote(ctxt *Link, sectionName string, tag uint32, desc []byte) {
 	s.SetAlign(4)
 }
 
+func addgnuattributes(ctxt *Link) {
+	ldr := ctxt.loader
+	s := ldr.CreateSymForUpdate(".gnu.attributes", 0)
+	s.SetReachable(true)
+	s.SetType(sym.SELFROSECT)
+	s.AddUint32(ctxt.Arch, uint32(16))
+	s.AddBytes(elfgnuattributescontent())
+	for len(s.Data())%4 != 0 {
+		s.AddUint8(0)
+	}
+	s.SetSize(int64(len(s.Data())))
+	s.SetAlign(4)
+}
+
 func (ctxt *Link) doelf() {
 	ldr := ctxt.loader
 
@@ -1507,6 +1558,9 @@ func (ctxt *Link) doelf() {
 	shstrtab.Addstring(".noptrbss")
 	shstrtab.Addstring("__libfuzzer_extra_counters")
 	shstrtab.Addstring(".go.buildinfo")
+	if ctxt.IsMIPS() {
+		shstrtab.Addstring(".gnu.attributes")
+	}
 
 	// generate .tbss section for dynamic internal linker or external
 	// linking, so that various binutils could correctly calculate
@@ -1557,6 +1611,9 @@ func (ctxt *Link) doelf() {
 			shstrtab.Addstring(elfRelType + ".data.rel.ro")
 		}
 		shstrtab.Addstring(elfRelType + ".go.buildinfo")
+		if ctxt.IsMIPS() {
+			shstrtab.Addstring(elfRelType + ".gnu.attributes")
+		}
 
 		// add a .note.GNU-stack section to mark the stack as non-executable
 		shstrtab.Addstring(".note.GNU-stack")
@@ -1760,6 +1817,9 @@ func (ctxt *Link) doelf() {
 	if ctxt.LinkMode == LinkExternal && *flagBuildid != "" {
 		addgonote(ctxt, ".note.go.buildid", ELF_NOTE_GOBUILDID_TAG, []byte(*flagBuildid))
 	}
+	if ctxt.IsMIPS() {
+		addgnuattributes(ctxt)
+	}
 }
 
 // Do not write DT_NULL.  elfdynhash will finish it.
@@ -1886,6 +1946,11 @@ func Asmbelf(ctxt *Link, symo int64) {
 			sh.flags = SHF_ALLOC
 		}
 
+		if ctxt.IsMIPS() {
+			sh := elfshname(".gnu.attributes")
+			sh.type_ = SHT_NOTE
+		}
+
 		goto elfobj
 	}
 
@@ -2003,6 +2068,16 @@ func Asmbelf(ctxt *Link, symo int64) {
 		phsh(pnote, sh)
 	}
 
+	if ctxt.IsMIPS() {
+		sh := elfshname(".gnu.attributes")
+		resoff -= int64(elfgnuattributes(sh, uint64(startva), uint64(resoff)))
+
+		pnote := newElfPhdr()
+		pnote.type_ = PT_NOTE
+		pnote.flags = PF_R
+		phsh(pnote, sh)
+	}
+
 	// Additions to the reserved area must be above this line.
 
 	elfphload(&Segtext)
@@ -2344,6 +2419,10 @@ elfobj:
 		a += int64(elfwritenetbsdpax(ctxt.Out))
 	}
 
+	if ctxt.IsMIPS() {
+		a += int64(elfwritegnuattributes(ctxt.Out))
+	}
+
 	if a > elfreserve {
 		Errorf(nil, "ELFRESERVE too small: %d > %d with %d text sections", a, elfreserve, numtext)
 	}

This is blocked by #39677

This patch has been merged in 2020. So let's close this bug report.