cmd/link: `link` reports "symbol listed multiple times" if go:linkname into both vendor package and non-vendor package #42401

chenzhuoyu · 2020-11-05T16:22:17Z

What version of Go are you using (`go version`)?

$ go version
go version go1.15.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/myusername/Library/Caches/go-build"
GOENV="/Users/myusername/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/myusername/GolangProjects/pkg/mod"
GONOPROXY="*.mycompanydomain.org"
GONOSUMDB="*.mycompanydomain.org"
GOOS="darwin"
GOPATH="/Users/myusername/GolangProjects"
GOPRIVATE="*.mycompanydomain.org"
GOPROXY="https://go-mod-proxy.mycompanydomain.org,https://goproxy.cn,https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15.3/libexec"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/myusername/Sources/tests/test_go/main/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/h3/c5bgpd9921d0tbkpf5n9vh2w0000gn/T/go-build518026853=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Create a directory structure like this:

.
├── foo
│   ├── foo.go
│   └── go.mod
└── main
    ├── go.mod
    └── main.go

// $ cat foo/go.mod 
module github.com/chenzhuoyu/env

go 1.15

// $ cat foo/foo.go 
package foo

var bar string

func GetBar() string {
    return bar
}

func init() {
    bar = ""
}

// $ cat main/go.mod 
module github.com/chenzhuoyu/linker_issue

go 1.15

replace github.com/chenzhuoyu/foo v0.0.0 => ../foo

require github.com/chenzhuoyu/foo v0.0.0

// $ cat main/main.go 
package main

import (
    _ `unsafe`

    `github.com/chenzhuoyu/foo`
)

//go:linkname v1 github.com/chenzhuoyu/foo.bar
var v1 string

//go:linkname v2 github.com/chenzhuoyu/linkname/vendor/github.com/chenzhuoyu/foo.bar
var v2 string

func main() {
    v1 = v2
    foo.GetBar()
}

Build the main package.

$ cd main
$ go build .

What did you expect to see?

The main package builds just fine.

What did you see instead?

$ go build .
# github.com/chenzhuoyu/linker_issue
2020/11/05 23:56:41 symbol go.info.github.com/chenzhuoyu/foo.bar listed multiple times

I have also tried:

go 1.14.10 works fine.

$ cd main
$ ls
go.mod  main.go
$ /usr/local/opt/go@1.14/bin/go version
go version go1.14.10 darwin/amd64
$ /usr/local/opt/go@1.14/bin/go build .
$ ls
go.mod  linker_issue*  main.go

Passing -ldflags='-w' to the compiler also works fine.

$ go build -ldflags='-w' .
$ ls
go.mod  linker_issue*  main.go

After searching for several hours, I found these related issues, but they are not very helpful to me.

#11699
#15926
#28159

The text was updated successfully, but these errors were encountered:

thanm · 2020-11-05T20:56:42Z

I'll take a look.

thanm · 2020-11-05T22:10:47Z

A note before getting into the details of the bug: please be aware that the behavior + implementation of "//go:linkname" is explicitly not covered by the Go 1 compatibility guarantee; the fact that importing "unsafe" is required is intended to reinforce this.

As the toolchain changes over time, the implementation and behavior of "//go:linkname" can evolve. There are uses of "//go:linkname" that may work with version X of the Go toolchain and then not work with version X+1. For example, using "//go:linkname" to reach from package A into a dependent package B to access one of B's non-exported symbols is something that works today, but may very well not work with future versions of Go.

Regarding the Go program in the bug:

This program contains behavior that happened to work with the Go 1.14 toolchain implementation of "//go:linkname" but does not work with Go 1.15. In Go 1.14, linking and symbol resolution is done using an entirely name-driven basis; all (or virtuall all) symbols are named, and at link time two symbols with the same name wind up mapping to same linker symbol, regardless of which package or compilation unit the symbol was defined in.

What this means is that for Go 1.14 when you write

  //go:linkname v1 github.com/chenzhuoyu/foo.bar
  var v1 string

this has the effect of making "v1" (defined in main.go) and "bar" (defined in foo.go) virtually the same symbol at link time. If you look in the final binary produced by Go 1.14 for this example, there are only two "foo.bar" variables.

In Go 1.15 we moved to a new object file format and a new way of doing symbol resolution. For most regular Go symbol definitions (variables, functions) the compiler assigns the symbol an index within the object file; when the linker reads the object file, it sees the symbol (roughly) as a tuple <P,S> where P is a numeric package ID and S is a numeric symbol ID.

What this means for Go 1.15 is that there are actually three "foo.bar" strings: one defined in package foo and two in package main; each has a unique identity using the new index-based scheme. Two of the three symbols happen to have the same name, which results in a collision during DWARF generation.

cherrymui · 2020-11-05T22:38:17Z

I'm not sure I understand the issue.

github.com/chenzhuoyu/linkname/vendor/github.com/chenzhuoyu/foo.bar

Where do you vendor this? I didn't see any vendor directory or any vendor command.

Also, do you run into this issue in real code, or just this synthetic example?

chenzhuoyu · 2020-11-06T03:47:24Z

Where do you vendor this? I didn't see any vendor directory or any vendor command.

I don't, I just write it there.

Also, do you run into this issue in real code, or just this synthetic example?

I ran into this issue in real code in our library, it is only used for unittests, so there are a lot of "hacks" in it. The code above is just a simplification.

Some of our legacy projects use dep to manage dependencies, so the code needs to work with both vendored projects and non-vendored projects, but upgrading to Go 1.15.3 breaks a lot of our tests. :(

Maybe we should consider refactoring the library and get rid of go:linkname, but that will be a lot of work. :(

chenzhuoyu · 2020-11-06T04:05:47Z

A note before getting into the details of the bug: please be aware that the behavior + implementation of "//go:linkname" is explicitly not covered by the Go 1 compatibility guarantee; the fact that importing "unsafe" is required is intended to reinforce this.

We only use "//go:linkname" for unittests, not in our production code.

There are some not-so-well-designed libraries that come with a lot of side-effects (performing I/O within init, reading environment variables, etc), which can cause our tests to be very unstable.

The "//go:linkname" is a way we found to "hack" these libraries and set them into a certain state for our tests.

As the toolchain changes over time, the implementation and behavior of "//go:linkname" can evolve. There are uses of "//go:linkname" that may work with version X of the Go toolchain and then not work with version X+1. For example, using "//go:linkname" to reach from package A into a dependent package B to access one of B's non-exported symbols is something that works today, but may very well not work with future versions of Go.

Should I consider "//go:linkname" as some sort of "undefined behavior" ?

ianlancetaylor · 2020-11-06T05:55:17Z

I wouldn't say that //go:linkname is undefined behavior. But it is unsafe, and it is not covered by the Go 1 compatibility guarantee. If you choose to use it, you must be prepared to adjust your use with each new Go release.

In general I would advise against using it.

gopherbot · 2020-11-07T16:37:13Z

Change https://golang.org/cl/268178 mentions this issue: cmd/compile: make sure linkname'd symbol is non-package

thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 5, 2020

thanm added this to the Go1.16 milestone Nov 5, 2020

thanm removed this from the Go1.16 milestone Nov 6, 2020

gopherbot closed this as completed in a444458 Nov 9, 2020

golang locked and limited conversation to collaborators Nov 9, 2021

gopherbot added the FrozenDueToAge label Nov 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/link: `link` reports "symbol listed multiple times" if go:linkname into both vendor package and non-vendor package #42401

cmd/link: `link` reports "symbol listed multiple times" if go:linkname into both vendor package and non-vendor package #42401

chenzhuoyu commented Nov 5, 2020

thanm commented Nov 5, 2020

thanm commented Nov 5, 2020

cherrymui commented Nov 5, 2020

chenzhuoyu commented Nov 6, 2020

chenzhuoyu commented Nov 6, 2020 •

edited

ianlancetaylor commented Nov 6, 2020

gopherbot commented Nov 7, 2020

cmd/link: link reports "symbol listed multiple times" if go:linkname into both vendor package and non-vendor package #42401

cmd/link: link reports "symbol listed multiple times" if go:linkname into both vendor package and non-vendor package #42401

Comments

chenzhuoyu commented Nov 5, 2020

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

I have also tried:

thanm commented Nov 5, 2020

thanm commented Nov 5, 2020

cherrymui commented Nov 5, 2020

chenzhuoyu commented Nov 6, 2020

chenzhuoyu commented Nov 6, 2020 • edited

ianlancetaylor commented Nov 6, 2020

gopherbot commented Nov 7, 2020

cmd/link: `link` reports "symbol listed multiple times" if go:linkname into both vendor package and non-vendor package #42401

cmd/link: `link` reports "symbol listed multiple times" if go:linkname into both vendor package and non-vendor package #42401

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

chenzhuoyu commented Nov 6, 2020 •

edited