Apparently that Realtek RTD1296 CPU is:

• ARMv8-A (64-bit)
• 4x ARM Cortex-A53 @ 1.4 GHz

(attn @golang/arm)

The instruction is mrs x0, ID_AA64ISAR0_EL1. Maybe that system register isn't accessible in user space on your machine? I assume the OS is Linux?

I think EL1 here is meant to refer to kernel mode. My copy of the ARM manual, section "D13.2.61 ID_AA64ISAR0_EL1, AArch64 Instruction Set Attribute Register 0" even says that access from EL0 (user mode) either traps or is undefined behavior. But clearly it must work on most systems.

Looks like it's:

Linux fanli-218play 4.4.180+ #42661 SMP Mon Jun 27 15:09:04 CST 2022 aarch64 GNU/Linux synology_rtd1296_ds218play

On a https://www.synology.com/en-us/products/DS218play

FWIW, on Synology, this program runs as non-root but with some additional capabilities (as in capabilities(7)).

Looks like this is the same CPU across:

https://www.synology.com/en-us/products/DS220j#specs
https://www.synology.com/en-us/products/DS218#specs
https://www.synology.com/en-us/products/DS418#specs
https://www.synology.com/en-us/products/DS218play#specs
https://www.synology.com/en-us/products/DS118#specs

... which are the 5 of the top 6 arm64 Synology models per our stats. (the DS120j has a Marvell Armada 3700 88F3720)

It looks like these instructions do trap. Linux emulates the instructions and returns to userspace: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=77c97b4ee21290f5f083173d957843b615abbff2

It looks like emulation support was added in Linux 4.11, so it makes sense that 4.4 is crashing. From the patch, it seems that we should check for HWCAP_CPUID before executing these instructions.

Out of curiosity, I checked: no Synology model (for any architecture) runs a kernel newer than 4.4.x. Top kernel versions on Synology we see used:

But who knows how much was cherry-picked into those +. Apparently not that commit you referenced :)

So in this case I guess the /proc/self/auxv read is failing, causing the readARM64Registers call, which then gets the SIGILL:

func doinit() {
        if err := readHWCAP(); err != nil {
                // failed to read /proc/self/auxv, try reading registers directly                                                                               
                readARM64Registers()
                return
        }
...

But these same hwcap flags are available in /proc/cpuinfo too, no?

pi@pi400arm64:~ $ uname -r
5.10.103-v8+
pi@pi400arm64:~ $ cat /proc/cpuinfo  | grep ^Features
Features	: fp asimd evtstrm crc32 cpuid
Features	: fp asimd evtstrm crc32 cpuid
Features	: fp asimd evtstrm crc32 cpuid
Features	: fp asimd evtstrm crc32 cpuid

Should we try parsing that first as another fallback before the mrs x0, ID_AA64ISAR0_EL1?

Or would y'all prefer looking at the kernel version and hope that's indicative of reality?

Related issue with x/sys/cpu on arm64 but on OpenBSD is #42517

@lovitus notes in tailscale/tailscale#5793 (comment) that they're able to cat /proc/cpuinfo:

fanli@fanli-218play:~$ cat /proc/cpuinfo
processor       : 0
model name      : ARMv8 Processor rev 4 (v8l)
BogoMIPS        : 54.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4
...

I am fairly surprised that /proc/self/auxv is not readable, especially if /proc/cpuinfo is. From tailscale/tailscale#5793 (comment), it sounds like /proc/self/auxv is generally readable.

Is tailscale running inside of some kind of sandbox? Running under `strace -f`` could at least give a sense of what kind of error we get trying to open/read auxv.

(As an aside, I'm sad we can't just get auxv from runtime main. It shouldn't be necessary to open a file to read auxv...)

As far as changing x/sys/cpu to avoid SIGILL when /proc/self/auxv can't be read, I see a few options, none stellar:

1. Attempt to read CPU capabilities from /proc/cpuinfo. I don't have high confidence in cpuinfo being available if auxv is not.
2. Use uname to do a best-effort version check before readARM64Registers.
3. Install a SIGILL handler and manually recover if the instructions fault. This is going to get really messy.
4. Don't use readARM64Registers at all and assume minimal CPU features.
5. Accept the status quo that this might crash on old kernels.

@tklauser For more context, I'm curious what your motivation for https://go.dev/cl/209478 was. What systems did not have auxv available that you wanted to support?

Is tailscale running inside of some kind of sandbox? Running under `strace -f`` could at least give a sense of what kind of error we get trying to open/read auxv.

Synology has two major OS versions in the wild. In the newer one (DSM7), packages run non-root but can request certain capabilities(7) in their manifest. I'd never noticed a filesystem chroot or anything but our existing four Synology test devices don't yet include arm64. One is arriving Monday, though, and then I'll be able to see.

Checking uname kernel version seems easiest. We do that already elsewhere so should work.

x/sys/cpu already reaches into the standard library for a couple of cases. We could save the auxv information in sysauxv and provide a hidden runtime package function for x/sys/cpu to call to retrieve it.

Change https://go.dev/cl/458256 mentions this issue: runtime: expose auxv for use by x/sys/cpu

x/sys/cpu already reaches into the standard library for a couple of cases. We could save the auxv information in sysauxv and provide a hidden runtime package function for x/sys/cpu to call to retrieve it.

Something like this? https://go-review.googlesource.com/c/go/+/458256

I then made a change to x/sys/cpu using that. Seems to work.

Amazon arrived and I have yet another test device so I can reproduce now.

/proc/self/auxv isn't readable because the binary runs as a non-root user but its binary is additionally given /bin/setcap cap_net_admin,cap_net_raw+eip which changes the permissions in procfs it seems (per https://lore.kernel.org/lkml/6ddd00bd-87d9-484e-8f2a-06f15a75a4df@app.fastmail.com/T/#md2736e8bf1b5a87e3913f2f558887bc012ae75c8).

If I remove those capabilities, then x/sys/cpu reads /proc/self/auxv and is fine. (but runs in a degraded mode where certain operations aren't permitted due to the Synology sandbox)

Minimal repro against Linux 4.4.180+ (arm64):

bradfitz@tsdev:~/src/golang.org/x/sys$ git di
diff --git a/cpu/cpu_linux_arm64.go b/cpu/cpu_linux_arm64.go
index 79a38a0..fee6f12 100644
--- a/cpu/cpu_linux_arm64.go
+++ b/cpu/cpu_linux_arm64.go
@@ -4,6 +4,11 @@
 
 package cpu
 
+import (
+       "io/ioutil"
+       "log"
+)
+
 // HWCAP/HWCAP2 bits. These are exposed by Linux.
 const (
        hwcap_FP       = 1 << 0
@@ -34,6 +39,9 @@ const (
 
 func doinit() {
        if err := readHWCAP(); err != nil {
+               log.Printf("XXX readHWCap error: %v", err)
+               cpuinfo, err := ioutil.ReadFile("/proc/cpuinfo")
+               log.Printf("XXX /proc/cpuinfo = %q, %v", cpuinfo, err)
                // failed to read /proc/self/auxv, try reading registers directly
                readARM64Registers()
                return

Then, after a go.mod replace:

$ cat cputest.go
package main

import (
        "fmt"

        "golang.org/x/sys/cpu"
)

func main() {
        fmt.Printf("%+v\n", cpu.ARM64)
}

$ GOARCH=arm64 go build  && scp cputest ts@10.2.201.2: && ssh root@10.2.201.2  /bin/setcap cap_net_admin,cap_net_raw+eip /var/services/homes/ts/cputest && ssh ts@10.2.201.2 ./cputest 2>&1 | head -20
cputest                                                                                                                                 100% 2029KB  59.2MB/s   00:00    
2022/12/18 18:50:46 XXX readHWCap error: open /proc/self/auxv: permission denied
2022/12/18 18:50:46 XXX /proc/cpuinfo = "processor\t: 0\nmodel name\t: ARMv8 Processor rev 4 (v8l)\nBogoMIPS\t: 54.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 1\nmodel name\t: ARMv8 Processor rev 4 (v8l)\nBogoMIPS\t: 54.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 2\nmodel name\t: ARMv8 Processor rev 4 (v8l)\nBogoMIPS\t: 54.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 3\nmodel name\t: ARMv8 Processor rev 4 (v8l)\nBogoMIPS\t: 54.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\n", <nil>
SIGILL: illegal instruction
PC=0x9feb0 m=0 sigcode=1
instruction bytes: 0x0 0x6 0x38 0xd5 0xe0 0x7 0x0 0xf9 0xc0 0x3 0x5f 0xd6 0x0 0x0 0x0 0x0

goroutine 1 [running, locked to thread]:
golang.org/x/sys/cpu.getisar0()
        /home/bradfitz/src/golang.org/x/sys/cpu/cpu_arm64.s:14 fp=0x4000041c20 sp=0x4000041c20 pc=0x9feb0
golang.org/x/sys/cpu.readARM64Registers()
        /home/bradfitz/src/golang.org/x/sys/cpu/cpu_arm64.go:65 +0x2c fp=0x4000041c60 sp=0x4000041c20 pc=0x9f69c
golang.org/x/sys/cpu.doinit()
        /home/bradfitz/src/golang.org/x/sys/cpu/cpu_linux_arm64.go:46 +0xc8 fp=0x4000041ce0 sp=0x4000041c60 pc=0x9fa28
golang.org/x/sys/cpu.archInit(...)
        /home/bradfitz/src/golang.org/x/sys/cpu/cpu_arm64.go:48
golang.org/x/sys/cpu.init.0()
        /home/bradfitz/src/golang.org/x/sys/cpu/cpu.go:199 +0x20 fp=0x4000041cf0 sp=0x4000041ce0 pc=0x9ed20
runtime.doInit(0x1522e0)
        /home/bradfitz/sdk/go1.19/src/runtime/proc.go:6321 +0x128 fp=0x4000041e30 sp=0x4000041cf0 pc=0x51828
runtime.doInit(0x1516e0)

@prattmic wrote:

I don't have high confidence in cpuinfo being available if auxv is not.

Neither did I, but it seems to work.

I'll send a change to fall back to parsing that for now for Linux kernels prior to 4.11.

Change https://go.dev/cl/458315 mentions this issue: cpu: parse /proc/cpuinfo on linux/arm64 on old kernels when needed

Change https://go.dev/cl/465295 mentions this issue: cpu: get hwcap/auxv from the Go 1.21+ runtime