cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled #48112

igalic · 2021-09-01T10:46:40Z

What version of Go are you using (`go version`)?

$ go version
go version go1.17 freebsd/amd64

Does this issue reproduce with the latest release?

This is the latest version.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/meena/.cache/go-build"
GOENV="/home/meena/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOINSECURE=""
GOMODCACHE="/home/meena/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="freebsd"
GOPATH="/home/meena/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build337677643=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I compiled a simple hello world program:

package main

import (
        "fmt"
)

func main() {
        fmt.Println("Hello, World!")
}

, and tried to run it in a FreeBSD 13 environment with W^X enabled:

meena@bsdix ~/s/wxorx> go build hello.go
meena@bsdix ~/s/wxorx> sudo -H sysctl kern.elf64.allow_wx=0
kern.elf64.allow_wx: 1 -> 0
meena@bsdix ~/s/wxorx>

What did you expect to see?

meena@bsdix ~/s/wxorx> ./hello
Hello, World!
meena@bsdix ~/s/wxorx>

What did you see instead?

meena@bsdix ~/s/wxorx> ./hello
exec_new_vmspace: mapping stack size 0x20000000 prot 0x7 failed mach error 2 errno 13
fish: Job 1, './hello' terminated by signal SIGABRT (Abort)
meena@bsdix ~/s/wxorx [SIGABRT]>

This also happens if you run any go program:

meena@bsdix ~/s/wxorx [SIGABRT]> go version
exec_new_vmspace: mapping stack size 0x20000000 prot 0x7 failed mach error 2 errno 13
fish: Job 1, 'go version' terminated by signal SIGABRT (Abort)
meena@bsdix ~/s/wxorx [SIGABRT]>

Data Execution Prevention Windows is supported since go 1.14:
https://golang.org/doc/go1.14#windows "Go binaries on Windows now have DEP (Data Execution Prevention) enabled."

I also don't hear any complaints from OpenBSD people, who have W^X enabled for about 20 years now.

Note that workarounds of marking a binary as wxneeded with elfctl(1) also don't work:

meena@bsdix ~/s/wxorx> elfctl -e +wxneeded hello
elfctl: NT_FREEBSD_FEATURE_CTL note not found
elfctl: NT_FREEBSD_FEATURE_CTL note not found
meena@bsdix ~/s/wxorx [1]>

Somebody managed to extract that "Note" and link it against their rclone: https://forums.freebsd.org/threads/rclone-not-working-with-w-x.80279/ which allowed them to work around this.
I have not yet been able to reproduce that work-around.

The text was updated successfully, but these errors were encountered:

igalic · 2021-09-01T11:15:15Z

Addendum: With go build -buildmode=pie the resulting binary works!

meena@bsdix ~/s/wxorx> go build -buildmode=pie hello.go
meena@bsdix ~/s/wxorx> ./hello
Hello, World!
meena@bsdix ~/s/wxorx> sudo -H sysctl kern.elf64.allow_wx=0
Password:
kern.elf64.allow_wx: 1 -> 0
meena@bsdix ~/s/wxorx> ./hello
Hello, World!
meena@bsdix ~/s/wxorx> go version
exec_new_vmspace: mapping stack size 0x20000000 prot 0x7 failed mach error 2 errno 13
fish: Job 1, 'go version' terminated by signal SIGABRT (Abort)
meena@bsdix ~/s/wxorx [SIGABRT]>

not only that, elfctl is happy with the binary:

meena@bsdix ~/s/wxorx> elfctl hello
File 'hello' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user VA to 48bit' is unset.
noaslrstkgap    'Disable ASLR stack gap' is unset.
meena@bsdix ~/s/wxorx>

emaste · 2021-09-01T13:03:05Z

kib@ explained at least part of the issue: -buildmode=exe results in a binary with no PT_GNU_STACK, so defaults to rwx stack and thus fails with allow_wx=0

It seems -buildmode=pie causes go to use (parts of) the standard host tool chain, and so gets both a PT_GNU_STACK for rw stack, and the NT_FREEBSD_FEATURE_CTL note that elfctl manipulates.

igalic · 2021-09-01T14:41:01Z

We can modify our ports system to set -buildmode=pie for ports depending on go.
That means, we can produce packages, built with go, that will be compiled as PIEs.

However, we cannot seem to be able to produce a go compiler that is PIE.

Put differently: we can build packages with go on a system with WX disabled, that will work on a system with WX enabled, but we cannot produce a go compiler that will work on such a system.

randall77 · 2021-09-01T14:55:28Z

We already set PT_GNU_STACK for Linux, we just need to do so for freebsd also (and all the other bsds?). That would solve the problem for internal linking, and I think that's the situation when it is happening?

emaste · 2021-09-01T15:25:51Z

Yes, we need PT_GNU_STACK on FreeBSD. I believe both OpenBSD and NetBSD provide a no-X stack in any case and would be fine without PT_GNU_STACK (although it might not cause any trouble to have it).

That would solve the problem for internal linking

I believe so, yes. I'm not completely familiar with go's linking/tool chain use, but I assume that -buildmode=pie ends up relying on the standard host linker as a side effect? (And, this may be an implementation detail that could change in the future?)

gopherbot · 2021-09-01T15:35:07Z

Change https://golang.org/cl/346872 mentions this issue: cmd/link: mark stacks as non-executable on freebsd

randall77 · 2021-09-01T15:36:00Z

@igalic @emaste Try the above linked CL and tell me if that works for you.

emaste · 2021-09-01T17:52:32Z

CC @dmgk, FreeBSD's lang/go port maintainer

igalic · 2021-09-02T07:57:00Z

with that patch it now works with w^x enabled:

meena@bsdix /u/p/l/go (main)> file /usr/local/go/bin/go*
/usr/local/go/bin/go:    ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, Go BuildID=XLIT5y7zsZF5lR-GsGHn/5sUPdtoWTSvCjrig1rac/oUGhGkYcwZr5aypGE13h/PyCGVZmPPahQmZyis-c2, not stripped
/usr/local/go/bin/gofmt: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, Go BuildID=23dNAVgDQa9cM8y7STWR/ZMhOBGxd00N-4hKWF1zI/-MZO7QHiu4q61cHkSr4w/107R6PUcs881Y9-en7pT, not stripped
meena@bsdix /u/p/l/go (main)> elfctl /usr/local/go/bin/go*
elfctl: NT_FREEBSD_FEATURE_CTL note not found
elfctl: NT_FREEBSD_FEATURE_CTL note not found
meena@bsdix /u/p/l/go (main) [1]> sudo -H sysctl kern.elf64.allow_wx=0
kern.elf64.allow_wx: 1 -> 0
meena@bsdix /u/p/l/go (main)> go version
go version go1.17 freebsd/amd64
meena@bsdix /u/p/l/go (main)>

(this means that whenever the bootstrap compiler switches to Go1.18, we could start building lang/go on w^x enabled systems)

cherrymui · 2021-09-02T15:25:44Z

Do we want to backport this, so Go 1.16 and 1.17 will work on newer FreeBSD? Thanks.

randall77 · 2021-09-02T15:31:22Z

I guess that depends on how often people enable W^X mode. My intuition is not very often, or we'd have seen this before.
Is there a plan to make W^X the default?

emaste · 2021-09-02T15:45:58Z

My intuition is not very often, or we'd have seen this before. Is there a plan to make W^X the default?

You're right that it's not enabled often at present. Right now it's really "adventurous" users trying it out; I expect we will make a more comprehensive effort within the FreeBSD developer community to find showstoppers (e.g. major runtimes that fail with it enabled, as here), before a broader call for testing and any discussion of defaults.

That said, independent of W^X with no PT_GNU_STACK we get a RWX stack, and IMO it's worth backporting this just to avoid that.

igalic · 2021-09-02T20:56:30Z

@ cherrymui Our "quarterly" branch currently has 1.16 and our "latest" branch has 1.17, so i think it would make sense to backport it to both of those.

I'll also open an issue to have NT_FREEBSD_FEATURE_CTL note added, as @emaste mentioned in the review for https://golang.org/cl/346872
I see that NetBSD and OpenBSD already have a .note section, so there's precedent.

randall77 · 2021-09-02T21:10:02Z

I don't think this qualifies for backporting. https://github.com/golang/go/wiki/MinorReleases
It's always been this way, so it isn't a regression. A workaround is to just turn W^X mode off.
It's not really a security issue either - at least not something exploitable (and not new). In fact, I think this would only affect the single stack the program starts with - for freebsd all the other stacks (both G and M stacks) are allocated by the Go runtime and are already not marked executable.

igalic · 2021-09-03T10:19:40Z

Aye.
I've submitted this as patch to our lang/go port then: https://bugs.freebsd.org/258241 /cc @dmgk

yzgyyang · 2021-11-09T02:18:06Z

We can modify our ports system to set -buildmode=pie for ports depending on go. That means, we can produce packages, built with go, that will be compiled as PIEs.

However, we cannot seem to be able to produce a go compiler that is PIE.

Any specific blockers on why we cannot have a go compiler that is PIE? Currently, go itself builds fail if set kern.elf64.allow_wx=0.

ianlancetaylor · 2021-11-09T02:33:24Z

I don't know why the Go tools would fail with allow_wx=0. Can you provide more details?

You should be able to build the Go tools with pie by running something like go install -buildmode=pie cmd.

ianlancetaylor · 2021-11-09T02:34:17Z

Oh, I see, the problem is the missing PT_GNU_STACK. But that problem should be fixed for 1.18.

If you still have problems on tip, please open a new bug. Thanks.

Switch to the newer go1.17.9 bootstrap with backported CL346872 [1,2], fixes ports 259709. [1] https://go-review.googlesource.com/c/go/+/346872/ [2] golang/go#48112 PR: 263726 PR: 259709 Approved by: ygy@ (maintainer)

This was referenced Sep 1, 2021

Trying to run syncthing on a FreeBSD system with W^X enabled syncthing/syncthing#7915

Closed

FreeBSD 13 memory mapping policy allow_wx=0 causes crash DNSCrypt/dnscrypt-proxy#1730

Closed

cherrymui changed the title ~~Unable to execute go binaries on a FreeBSD system with W^X enabled~~ cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled Sep 1, 2021

cherrymui added the NeedsFix The path to resolution is known, but the work has not been done. label Sep 1, 2021

cherrymui added this to the Go1.18 milestone Sep 1, 2021

gopherbot closed this as completed in 782aa42 Sep 2, 2021

igalic mentioned this issue Sep 2, 2021

cmd/link: define ELF .note section on FreeBSD #48164

Closed

fundon mentioned this issue Sep 21, 2021

Error: esbuild: Failed to install correctly evanw/esbuild#1603

Closed

golang locked and limited conversation to collaborators Nov 9, 2022

gopherbot added the FrozenDueToAge label Nov 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled #48112

cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled #48112

igalic commented Sep 1, 2021

igalic commented Sep 1, 2021

emaste commented Sep 1, 2021

igalic commented Sep 1, 2021

randall77 commented Sep 1, 2021

emaste commented Sep 1, 2021

gopherbot commented Sep 1, 2021

randall77 commented Sep 1, 2021

emaste commented Sep 1, 2021

igalic commented Sep 2, 2021

cherrymui commented Sep 2, 2021

randall77 commented Sep 2, 2021

emaste commented Sep 2, 2021

igalic commented Sep 2, 2021

randall77 commented Sep 2, 2021

igalic commented Sep 3, 2021

yzgyyang commented Nov 9, 2021

ianlancetaylor commented Nov 9, 2021

ianlancetaylor commented Nov 9, 2021

cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled #48112

cmd/link: unable to execute go binaries on a FreeBSD system with W^X enabled #48112

Comments

igalic commented Sep 1, 2021

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

igalic commented Sep 1, 2021

emaste commented Sep 1, 2021

igalic commented Sep 1, 2021

randall77 commented Sep 1, 2021

emaste commented Sep 1, 2021

gopherbot commented Sep 1, 2021

randall77 commented Sep 1, 2021

emaste commented Sep 1, 2021

igalic commented Sep 2, 2021

cherrymui commented Sep 2, 2021

randall77 commented Sep 2, 2021

emaste commented Sep 2, 2021

igalic commented Sep 2, 2021

randall77 commented Sep 2, 2021

igalic commented Sep 3, 2021

yzgyyang commented Nov 9, 2021

ianlancetaylor commented Nov 9, 2021

ianlancetaylor commented Nov 9, 2021

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?