Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: add a flag to the linker to do not write function names to runtime.pclntab #36555

Open
xaionaro opened this issue Jan 14, 2020 · 16 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@xaionaro
Copy link
Contributor

xaionaro commented Jan 14, 2020

What version of Go are you using (go version)?

go version
go version devel +71154e061f Tue Jan 14 17:13:34 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/experiment0/.cache/go-build"
GOENV="/home/experiment0/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/experiment0/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/experiment0/.gimme/versions/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/experiment0/.gimme/versions/go/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build891839909=/tmp/go-build -gno-record-gcc-switches"

What did you do?

go get github.com/u-root/u-root
u-root -o /dev/null -build bb core boot
go get github.com/u-root/u-root/bb
go tool nm -size -sort size "$(go env GOPATH)"/bin/bb | head -1

What did you expect to see?

  e02ba0    3836272 r runtime.pclntab

What did you see instead?

  e39500    4808314 r runtime.pclntab

Proposal

I found here https://groups.google.com/d/msg/golang-nuts/hEdGYnqokZc/zQojaoWlAgAJ that a binary size could be reduced by avoiding of wasting space on function names within runtime.pclntab. So I propose to add a linker flag (for example -stripfnnames) to do that. We need such feature for applying Golang into embedded environments (with very limited total space).

I would like to prepare a PR if it will be approved.

@networkimprov
Copy link

See #36313

@xaionaro
Copy link
Contributor Author

xaionaro commented Jan 14, 2020

@networkimprov : thank you. However I saw that ticket. It contains discussion how to reduce runtime.pclntab for a generic case. While my proposal is about a special case (embedded systems), so I created a new ticket :)


UPD: Wrote a comment to #36313 with a proposal to add an option to select a simple GC which does not require so complex runtime.pclntab. If the proposal will be approved then this ticket is indeed not actual.

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Jan 14, 2020

Removing function names from runtime.pclntab would break programs that use runtime.Callers, which is a lot of programs. For example, the testing and log packages would partially break.

@xaionaro
Copy link
Contributor Author

xaionaro commented Jan 14, 2020

Removing function names from runtime.pclntab would break programs that use runtime.Callers, which is a lot of programs. For example, the testing and log packages would partially break.

@ianlancetaylor :

  1. It's only for embedded (where it's extremely important to reduce the size, and people are ready to sacrifice with log/testing for that). Moreover we can add a build-tag for -stripfnnames and use it to remove functions which uses runtime.Callers or to change their behavior. In my opinion it would good enough to just change the behavior to print <unknown> instead of function names.
  2. If I understand correctly testing and log will actually continue to work even without modifications (but I did not look into to this part of the code, yet). They will just print emptiness instead of function names.

@ianlancetaylor
Copy link
Contributor

I believe the testing.(*T).Helper function, for example, will break.

This will also likely break profiling information.

I understand that you would find it useful but I'm reluctant to endorse a change that will break code in ways that people will not know how to expect. I would rather find other ways to reduce binary size.

@xaionaro
Copy link
Contributor Author

that will break code in ways that people will not know how to expect.

We can forbid using broken functions via the build-tag (added automatically by -stripfnnames).

@networkimprov
Copy link

Ian may be concerned that you'll eventually realize that Go isn't the best choice for an embedded app, and switch to Rust or http://ziglang.org ;-)

@beoran
Copy link

beoran commented Jan 15, 2020

For embedded Go, where size is really important, for the time being, https://tinygo.org/ is the better solution.

Rather than having a specific build flag for stripping runtime.pclntab, I think that in the long run, what we need is new GOOS, such as "none" for running on raw iron, and "linux-embedded" for running an on embedded linux OS should be added. The "none" variants then allow building an OS kernel, and the -embedded variants then do a few things differently to save on space.

@xaionaro
Copy link
Contributor Author

xaionaro commented Jan 15, 2020

For embedded Go, where size is really important, for the time being, https://tinygo.org/ is the better solution.

I tried tinygo few times for u-root. And tinygo is a nice and interesting project, but it appears it has invalid CGo parser (which has problems with #define and other stuff), (if I remember correctly) does not support Golang's assembly language, it does not implement too much stuff from standard packages (which it overrides) and it even does not support something related to func() variables so it panics while compilation of u-root/bb. I've spent some time and realized that it's would require too much time to port the project to tinygo. Also tinygo's README.md says:

Non-goals: ...; Be able to compile every Go program out there.


Ian may be concerned that you'll eventually realize that Go isn't the best choice for an embedded app, and switch to Rust or http://ziglang.org ;-)

I believe the community of Golang is wise and will be able to find a solution for cases like mine :)
And I will be glad to somehow help to implement an approved solution when there will be one.

Anyway it would take much more time to port project to Rust than even to port it to tinygo :)

@cagedmantis cagedmantis added this to the Backlog milestone Jan 15, 2020
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 15, 2020
@embeddedgo
Copy link

embeddedgo commented Dec 6, 2020

@xaionaro I think we can try to introduce such -stripfnnames flag to the embeddedgo fork. Can you help?

I'd love to test such flag because I've just reached the limit of 1 MiB Flash in STM32L476RG:

$ size shell.elf
text data bss dec hex filename
1043876 3736 14004 1061616 1032f0 shell.elf

where pclntab is 345 KiB (34%):

$ nm --size-sort -S shell.elf|tail -5
08050104 00000fd4 T time.Time.AppendFormat
08059f98 00001444 T fmt.(*pp).printValue
200024e8 00001a74 B runtime.mheap_
08040ce0 00003728 T unicode.init
080a89c0 000565ac r runtime.pclntab

The embeddedgo fork adds support for noos/thumb, noos/riscv64 and linux/thumb.

I use linux/thumb only to run tests but it is usable and you can give it a try (no cgo support, sorry). It generates compressed Thumb2 instructions instead of ARM instructions so it produces smaller binaries that can be run on almost any ARMv7-A machine.

The noos ones are for bare metal programming.

@xaionaro
Copy link
Contributor Author

xaionaro commented Dec 7, 2020

I think we can try to introduce such -stripfnnames flag to the embeddedgo fork.

Sounds very interesting. It could become very useful for https://github.com/u-root/u-root.

But I do not understand: why do you do a fork instead of working with the upstream?

Can you help?

I would love to try. How can I help? :)

@embeddedgo
Copy link

embeddedgo commented Dec 7, 2020

why do you do a fork instead of working with the upstream?

You can much easily introduce new things in a fork. Even Go teem itself creates forks branches for for a bigger changes.

There is clearly no place for rarely used architectures or operating systems in the upstream but linux/thumb can be probably upstreamed as the 32-bit ARM seems to be now mainly used in embedded niche.

I would love to try. How can I help? :)

Simply send a pull request that implements -stripfnnames flag to the embeddedgo fork.

The embeddedgo compiler is released as patches to the original Go source so you can relatively easy see what was added/changed.

@embeddedgo
Copy link

embeddedgo commented Dec 12, 2020

@xaionaro I've just implemented stripfnn flag:

-stripfn int
    strip function names in pclntab, 1: remove package path, 2: blank names

Test results:

stripfn       text    data     bss
       0   1044754    3744   14004
       1   1026745    3744   14004
       2    973563    3744   14004

Sample panic outputs:

stripfn=1

panic: stripfn test

goroutine 1 [running]:
ramfs.(*dir).Close(0x2000b440, 0x20008f33, 0x0)
        dir.go:76 +0xaa
os.(*File).Close(0x2000a7c8, 0xffffffff, 0x20008070)
        file_noos.go:171 +0x1e
main.ls(0x20008050, 0x2, 0x2)
        ls.go:28 +0x70
main.runCmd(0x20008050, 0x2, 0x2)
        main.go:62 +0x80
main.main()
        main.go:25 +0x9e

stripfn=2

panic: stripfn test

goroutine 1 [running]:
(0x806d008, 0x8096f58)
        :1064 +0x3c6
(0x2000b600, 0x20008f33, 0x0)
        :76 +0xaa
(0x2000a7d8, 0xffffffff, 0x200080a0)
        :171 +0x1e
(0x20008080, 0x2, 0x2)
        :28 +0x70
(0x20008080, 0x2, 0x2)
        :62 +0x80
()
        :25 +0x9e
()
        :206 +0x11c
()
        :525 +0x2

In case of stripfn=2 panics are almost unusable. But in case of stripfn=1 there is enough information to find the problem.

Clone https://github.com/embeddedgo/go and try it.

@dedo1911
Copy link

Would be nice to have stripfn flag in official Go too 🤔

@eripi
Copy link

eripi commented Aug 24, 2021

@embeddedgo I clone https://github.com/embeddedgo/go and run all.bash to build go tools, when I do compile with stripfn=2 go build -ldflags '-s -w -stripfn 2' -o test main.go), when I run my test program, I get a stack of

panic: http new request fail

goroutine 1 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001f80c0, 0x0, 0x0, 0x0)
	:232 +0x532
go.uber.org/zap.(*Logger).Panic(0xc000196420, 0x5c5cba, 0x15, 0x0, 0x0, 0x0)
	:230 +0x85
main.main()
	:17 +0xc5

there are function names , my system is
Ubuntu 18.04.5 LTS \n \l

my test code is

package main

import "go.uber.org/zap"

var Olog *zap.Logger

func main() {
	opts := []zap.Option{zap.AddStacktrace(zap.PanicLevel)}

  logger, err := zap.NewProduction(opts...)
	if err != nil {
		panic("new logger error")
	}

	Olog = logger

   Olog.Panic("http new request fail")
}

I don't know why is there any configuration that needs to do?

@embeddedgo
Copy link

I didn't tested the stripfn flag much with go1.16.x but it seems with the last pcln changes it only strips the file names:

func A(s string) {
 B(s)
}

func B(s string) {
 panic(s)
}

func main() {
 A("hello!")
}

stripfn=0

panic: hello!

goroutine 1 [running]:
main.B(...)
 /home/michal/P/go/src/testy/hello_world/main.go:9
main.A(...)
 /home/michal/P/go/src/testy/hello_world/main.go:4
main.main()
 /home/michal/P/go/src/testy/hello_world/main.go:13 +0x51

stripfn=1

panic: hello!

goroutine 1 [running]:
main.B(...)
 main.go:9
main.A(...)
 main.go:4
main.main()
 main.go:13 +0x51

stripfn=2

panic: hello!

goroutine 1 [running]:
main.B(...)
 :9
main.A(...)
 :4
main.main()
 :13 +0x51

It seems the stripfn=1 shrinks the binary enough to meet my current requirements therefore i didn't investigated it further.

Bellow are the numbers for the above code (linux/amd64):

stripfn=0, text=814635 bytes
stripfn=1, text=810635 bytes
stripfn=2, text=809483 bytes

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Status: Triage Backlog
Development

No branches or pull requests

9 participants