Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proposal: x/sys/unix, x/sys/cpu: use the RISC-V Hardware Probing Interface on Linux #61416

Open
markdryan opened this issue Jul 18, 2023 · 8 comments
Open

proposal: x/sys/unix, x/sys/cpu: use the RISC-V Hardware Probing Interface on Linux #61416

markdryan opened this issue Jul 18, 2023 · 8 comments
Labels
arch-riscv Issues solely affecting the riscv64 architecture. OS-Linux Proposal
Milestone
Proposal

Comments

@markdryan
Copy link
Contributor

markdryan commented Jul 18, 2023 •
edited

Overview

The RISC-V port of Go is currently restricted to the RV64G instruction set. It does not take advantage of any other RISC-V extensions that could be used to boost performance or reduce code size, for example the Vector or Bit manipulation extensions. This is in stark contrast to other architectures supported by Go where SIMD and bit manipulation instructions are used when they are available. For example, amd64 builds of Go can use AVX2 optimised versions of certain critical functions, e.g., compare and memmove, to improve performance. Until recently, one issue preventing the use of RISC-V extensions beyond RV64G in Go was the lack of a mechanism to detect these extensions. This issue was resolved in Linux 6.4.

Version 6.4 of the Linux kernel introduces a new syscall, sys_riscv_hwprobe that can be used to determine interesting pieces of information about the underlying RISC-V CPUs on which a user space program runs. In particular, it exposes the RISC-V extensions supported by the CPUs in addition to other performance information such as whether and how efficiently the CPUs support unaligned memory accesses.

What about HWCAP?

Where available sys_riscv_hwprobe is preferable to using HWCAP as:

  1. It's extensible (there's enough bits for all future extensions).
  2. It can be used to query for extensions with multi-letter names, e.g., Zba, Zbb, which are likely to be useful in Go going forward.
  3. The hwprobe extension bits are versioned and their meaning is concretely defined.
  4. It can be used to determine additional information about the CPU. One potentially interesting example is its ability to return information about how well the CPU handles unaligned accesses. The current optimised RISC-V primitives for Go, e.g., memequal go out of their way to avoid unaligned accesses, i.e., they do 8 individual byte loads and stores instead of two unaligned 8 byte memory accesses. The current implementations might not be optimal on CPUs that support fast unaligned accesses. Hwprobe provides a way to detect fast unaligned accesses and an opportunity to switch to implementations that use them.

sys_riscv_hwprobe should be sufficient by itself for detecting the presence of V, Zba, Zbb, Zbs in vanilla kernels. However, there may be cases where extensions have been back ported to an earlier kernel but hw_probe has not. Such situations could be handled by first trying sys_hw_probe and then falling back to HWCAP if hw_probe is not available. This is the strategy used in OpenJDK.

Update 26/09/2023 However, it is probably not safe to use hwcap to detect the Vector extension as some RISC-V boards, e.g., the SiPeed LicheePi, indicate support for Vector 0.7 ( which is not compatible with Vector 1.0) through the hwcap 'V' bit.

Note that there's already a patch pending review that detects some extensions using HWCAP. Here we propose calling sys_riscv_hwprobe first and if that fails falling back to the code in this HWCAP patch. We would not fall back for Vector.

What extensions can sys_riscv_hwprobe detect?

It can detect FD and C in Linux 6.4

Support for detecting V, Zba, Zbb and Zbs are merged and are available in Linux 6.5.

What are we proposing?

The proposal is then to

(Updated on 26th of September 2023.)

  1. Add support for the sys_riscv_hwprobe to golang.org/x/sys/unix (510795). This would add a new syscall, a new type and some new constants to riscv64 builds.
  2. Update golang.org/x/sys/unix when Linux 6.5 is released to add support for the new extensions it supports. (530895)
  3. Update golang.org/x/sys/cpu to invoke the new syscall and record which extensions are supported. We'd fall back to HWCAP where sys_riscv_hwprobe is not available.
  4. Add support for the sys_riscv_hwprobe to internal/syscall/unix runtime (522995).
  5. Update internal/cpu to invoke the new syscall in internal/syscall/unix and record which extensions are supported. We'd fall back to HWCAP (not for Vector) where sys_riscv_hwprobe is not available (522995).

Proposal #62238 introduces the golang/x/sys/cpu.RISCV64 structure with one member, HasV. Here we propose adding the following new members to this structure.

        IsMisalignedFast bool // Fast misaligned accesses 
        HasC bool              // The C extension
        HasZba bool            // The Zba address generation extension, version 1.0 or greater
        HasZbb bool            // The Zbb extension, version 1.0 or greater
        HasZbs bool            // The Zbs extension, version 1.0 or greater
@gopherbot gopherbot added this to the Proposal milestone Jul 18, 2023
@ianlancetaylor
Copy link
Contributor

The proposal committee is concerned primarily with user-visible API changes. Switching to use the Hardware Probing Interface doesn't sound like a user-visible change, so that is fine. And adding new system calls and types to x/sys/unix is fine. So the only question is: what new API would be added to x/sys/cpu? If you can describe that, that would be helpful. Thanks.

@seankhliao seankhliao changed the title proposal: affected/package: golang.org/x/sys/unix and golang.org/x/sys/cpu: use the RISC-V Hardware Probing Interface on Linux proposal: x/sys/unix, x/sys/cpu: use the RISC-V Hardware Probing Interface on Linux Jul 18, 2023
@seankhliao seankhliao added OS-Linux arch-riscv Issues solely affecting the riscv64 architecture. labels Jul 18, 2023
@gopherbot
Copy link

Change https://go.dev/cl/510795 mentions this issue: unix/linux: Add sys_riscv_hwprobe for riscv64

@markdryan
Copy link
Contributor Author

So the only question is: what new API would be added to x/sys/cpu? If you can describe that, that would be helpful. Thanks.

Patch 508676 already proposes some changes to golang/x/sys/cpu. It adds a new struct, RISCV64. The intention would be to expand this structure with some additional fields.

        HasMisalignedFast bool // Fast misaligned accesses 
        HasC bool              // The C extension
        HasZBA bool            // The Zba address generation extension
        HasZBB bool            // The Zbb extension
        HasZBS bool            // The Zbs extension

The sys_riscv_hwprobe syscall can detect different types of CPU support for misaligned accesses (unknown, emulated, slow, fast, unsupported). My feeling here is that we only care if fast misaligned access are supported and, if they're not, misaligned accesses should be avoided. One boolean field should then suffice.

HasZBA, HasZBB and HasZBS, would require golang.org/x/sys/unix/linux/Dockerfile to be updated to use LInux 6.5 before they could be set to any value other than false. More fields for new extensions would be added over time as they become supported by hwprobe and the kernel itself.

One issue I did not consider in the initial proposal above is that there appears to be some parallel cpu feature detection code in internal/cpu. IIUC we'd need to duplicate the RISCV64 structure in internal/cpu if we wanted to inspect any of these fields in Go itself, from internal/bytealg for example. To populate the fields in internal/cpu we'd need to make a syscall. How would this work? Would we need to manually add the new syscall to the syscall package in the main Go repo?

@markdryan markdryan mentioned this issue Jul 20, 2023
Closed
@ianlancetaylor
Copy link
Contributor

You might have to add the new syscall to internal/syscall/unix in the standard library.

gopherbot pushed a commit to golang/sys that referenced this issue Jul 26, 2023
@markdryan @gopherbot
unix: add riscv_hwprobe for riscv64
104d401 
The riscv_hwprobe system call was introduced in Linux 6.4 and allows
the caller to determine a number of interesting pieces of information
about the underlying RISC-V CPUs, e.g., which extensions they support
and whether they allow fast unaligned memory accesses.  For more information
please see:

https://docs.kernel.org/riscv/hwprobe.html

We also update linux/mksysnum.go to ensure that the generated syscall constants
written to the zsysnum_linux_*.go files are always sorted by their syscall numbers
in ascending order.

Updates golang/go#61416

Change-Id: Iedb0a86adb65faac9061b9a5969ffa09eb5b303a
Reviewed-on: https://go-review.googlesource.com/c/sys/+/510795
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
@mengzhuo mengzhuo mentioned this issue Aug 23, 2023
Open
@gopherbot
Copy link

Change https://go.dev/cl/522995 mentions this issue: cpu/internal: provide runtime detection of RISC-V extensions on Linux

@4a6f656c
Copy link
Contributor

So the only question is: what new API would be added to x/sys/cpu? If you can describe that, that would be helpful. Thanks.

Patch 508676 already proposes some changes to golang/x/sys/cpu. It adds a new struct, RISCV64. The intention would be to expand this structure with some additional fields.

        HasMisalignedFast bool // Fast misaligned accesses 
        HasC bool              // The C extension
        HasZBA bool            // The Zba address generation extension
        HasZBB bool            // The Zbb extension
        HasZBS bool            // The Zbs extension

HasMisalignedFast seems awkward - HasFastMisaligned at least matches the comment, or we potentially drop the Has and use FastMisaligned or MisalignedIsFast...

Re HasZB*, given that they're usually referered to as mixed case (Zba and not ZBA), it would probably be better to stick with that here and use HasZb[abs].

The sys_riscv_hwprobe syscall can detect different types of CPU support for misaligned accesses (unknown, emulated, slow, fast, unsupported). My feeling here is that we only care if fast misaligned access are supported and, if they're not, misaligned accesses should be avoided. One boolean field should then suffice.

HasZBA, HasZBB and HasZBS, would require golang.org/x/sys/unix/linux/Dockerfile to be updated to use LInux 6.5 before they could be set to any value other than false. More fields for new extensions would be added over time as they become supported by hwprobe and the kernel itself.

Not necessarily - I suspect there is prior art for adding hard coded constants, which can be replaced with autogenerated ones once available.

One issue I did not consider in the initial proposal above is that there appears to be some parallel cpu feature detection code in internal/cpu. IIUC we'd need to duplicate the RISCV64 structure in internal/cpu if we wanted to inspect any of these fields in Go itself, from internal/bytealg for example. To populate the fields in internal/cpu we'd need to make a syscall. How would this work? Would we need to manually add the new syscall to the syscall package in the main Go repo?

The necessary code should be copied from x/sys/cpu to internal/cpu - re making a system call, there is prior art for this in the form of sysctl - see cpu_arm64_darwin.go or cpu_arm64_openbsd.go for example.

@markdryan
Copy link
Contributor Author

HasMisalignedFast seems awkward - HasFastMisaligned at least matches the comment, or we potentially drop the Has and use FastMisaligned or MisalignedIsFast...

How about IsMisalignedFast? The fields in the existing structures in src/internal/cpu.go all start with either 'Is' or 'Has' so this would be consistent with what's already there for the other architectures and hopefully, a bit more readable.

Re HasZB*, given that they're usually referered to as mixed case (Zba and not ZBA), it would probably be better to stick with that here and use HasZb[abs].

Agreed. This is better. I'll change this.

The necessary code should be copied from x/sys/cpu to internal/cpu - re making a system call, there is prior art for this in the form of sysctl - see cpu_arm64_darwin.go or cpu_arm64_openbsd.go for example.

I used cpu_arm64_openbsd.go as a model when creating https://go.dev/cl/522995

@gopherbot
Copy link

Change https://go.dev/cl/530895 mentions this issue: unix: update riscv_hwprobe constants

gopherbot pushed a commit to golang/sys that referenced this issue Sep 26, 2023
@markdryan @gopherbot
unix: update riscv_hwprobe constants
8858c72 
Linux 6.5 enhanced the riscv_hwprobe syscall to detect four new
RISC-V extensions, V, Zba, Zbb and Zbs.  Update the hwprobe
constants in unix so these extensions can be detected by Go
programs.

Updates golang/go#61416

Change-Id: Id6b4566c5c96fe3429fad54e93d3459cb5317642
Reviewed-on: https://go-review.googlesource.com/c/sys/+/530895
Reviewed-by: Ian Lance Taylor <iant@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Than McIntosh <thanm@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-riscv Issues solely affecting the riscv64 architecture. OS-Linux Proposal
Projects
Proposals
Status: Incoming
Milestone
Proposal
Development

No branches or pull requests
5 participants
@markdryan @ianlancetaylor @4a6f656c @gopherbot @seankhliao