Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: binary on darwin takes up more space on disk #39044

Closed
kokes opened this issue May 13, 2020 · 7 comments
Closed

cmd/link: binary on darwin takes up more space on disk #39044

kokes opened this issue May 13, 2020 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone
Go1.16

Comments

@kokes
Copy link

kokes commented May 13, 2020 •
edited

What version of Go are you using (go version)?

$ go version
go version devel +8ab37b1baf Mon Apr 20 18:32:58 2020 +0000 darwin/amd64

Does this issue reproduce with the latest release?

No (as in this is a regression in the current master, the latest stable is fine)

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/okokes/Library/Caches/go-build"
GOENV="/Users/okokes/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/okokes/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.2_1/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.2_1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/81/4jydp7kn51n6p68z88sqnkzc0000gn/T/go-build271357823=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I noticed my binary went from 8MB (1.14.2) to 13MB (master), but only when running du -sh or inspecting the binary in Finder (under "how much it takes on disk").

When stating both binaries, they are very similar in size, so this issue only revolves around disk usage, not size.

I replicated the issue by creating a hello world app

package main

import "fmt"


func main() {

fmt.Println("ahoy")

}

And then I bisected it, starting at 1811533 (good) and ending at cb11c98 (bad).

package main

import (

"log"

"os/exec"

)


func main() {

build := exec.Command("/Users/okokes/git/go/src/make.bash")

build.Dir = "/Users/okokes/git/go/src"

err := build.Run()

if err != nil {

log.Fatal("toolchain build failed", err)

}


cmd := exec.Command("/Users/okokes/git/go/bin/go", "build", "src/hello.go")
err = cmd.Run()
if err != nil {
	log.Fatal("program build failed", err)
}

sz := exec.Command("du", "-sh", "hello")
out, err := sz.Output()
if err != nil {
	log.Fatal("du failed", err)
}
num := out[0]
if num != '2' {
	log.Fatal(string(out))
}



}

Bisect identified 8ab37b1 as the first offending commit. I verified it manually - the commit before that leads to a 2.1MB binary on disk, this commit leads to 4.1MB on disk.

I could not replicate this on a Ubuntu 18.04 box, so I presume it's a Darwin thing.

What did you expect to see?

$ stat -f '%z %N' hello_*
2216280 hello_8ab37b1
2174008 hello_go1.14
$ du -sh hello_*
2.1M	hello_8ab37b1
2.1M	hello_go1.14

What did you see instead?

$ stat -f '%z %N' hello_*
2216280 hello_8ab37b1
2174008 hello_go1.14
$ du -sh hello_*
4.1M	hello_8ab37b1     <-- note the binary size here
2.1M	hello_go1.14
@bradfitz bradfitz added this to the Go1.16 milestone May 13, 2020
@josharian
Copy link
Contributor

cc @jeremyfaller

@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 18, 2020
@cagedmantis
Copy link
Contributor

/cc @cherrymui @rsc @mdempsky @ianlancetaylor

@randall77
Copy link
Contributor

Note that copying the result to a new file fixes the problem.

@cherrymui
Copy link
Member

Hmmm. https://tip.golang.org/src/cmd/link/internal/ld/outbuf_darwin.go

func (out *OutBuf) fallocate(size uint64) error {
	store := &syscall.Fstore_t{
		Flags:   syscall.F_ALLOCATEALL,
		Posmode: syscall.F_PEOFPOSMODE,
		Offset:  0,
		Length:  int64(size),
	}

	_, _, err := syscall.Syscall(syscall.SYS_FCNTL, uintptr(out.f.Fd()), syscall.F_PREALLOCATE, uintptr(unsafe.Pointer(store)))
	if err != 0 {
		return err
	}

	return nil
}

I wonder if F_PEOFPOSMODE is right here. It seems

F_PEOFPOSMODE Allocate from the physical end of file.

So if we preallocate more than once, the second and later calls will allocate the whole size on top of the existing size...

@cherrymui
Copy link
Member

Or we should subtract the existing size.

@gopherbot
Copy link

Change https://golang.org/cl/234481 mentions this issue: cmd/link: fix size calculation for file space preallocation on darwin

@networkimprov
Copy link

Shouldn't this be milestoned for 1.15?

@golang golang locked and limited conversation to collaborators May 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
None yet
Milestone
Go1.16
Development

No branches or pull requests
8 participants
@bradfitz @josharian @cagedmantis @networkimprov @randall77 @kokes @gopherbot @cherrymui