Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plugin: cross-compiling a binary that uses a plugin does not work #54075

Closed
merschformann opened this issue Jul 26, 2022 · 3 comments
Closed
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@merschformann
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.18.3 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env of linux/amd64 machine
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/marius/.cache/go-build"
GOENV="/home/marius/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/marius/.asdf/installs/golang/1.18.3/packages/pkg/mod"
GONOPROXY="github.com/nextmv-io/*"
GONOSUMDB="github.com/nextmv-io/*"
GOOS="linux"
GOPATH="/home/marius/.asdf/installs/golang/1.18.3/packages"
GOPRIVATE="github.com/nextmv-io/*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/marius/.asdf/installs/golang/1.18.3/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/marius/.asdf/installs/golang/1.18.3/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1293390508=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.18.3 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.18.3
uname -sr: Linux 5.15.55-1-MANJARO
LSB Version: n/a
Distributor ID: ManjaroLinux
Description: Manjaro Linux
Release: 21.3.5
Codename: Ruah
/usr/lib/libc.so.6: GNU C Library (GNU libc) stable release version 2.35.
go env of linux/arm64 machine
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/ec2-user/.cache/go-build"
GOENV="/home/ec2-user/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ec2-user/.asdf/installs/golang/1.18.3/packages/pkg/mod"
GONOPROXY="github.com/nextmv-io/*"
GONOSUMDB="github.com/nextmv-io/*"
GOOS="linux"
GOPATH="/home/ec2-user/.asdf/installs/golang/1.18.3/packages"
GOPRIVATE="github.com/nextmv-io/*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ec2-user/.asdf/installs/golang/1.18.3/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ec2-user/.asdf/installs/golang/1.18.3/go/pkg/tool/linux_arm64"
GOVCS=""
GOVERSION="go1.18.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2003570466=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Goal: cross-compile a binary (which consumes a plugin) targeting linux/arm64. The plugin was built on the target platform.

  1. Compile plugin on linux/arm64

    Plugin code
    package main
    
    // MustExist defines some symbol we expect to find when consuming the plugin.
    var MustExist = "bunny"
    Build call
    go build -buildmode plugin -trimpath -o plugin-$(go env GOOS)-$(go env GOARCH).so main.go
  2. Cross-compile binary on linux/amd64 machine (or any other non-linux/arm64) targeting linux/arm64

    Binary code
    package main
    
    import (
        "fmt"
        "io/ioutil"
        "log"
        "os"
        "path/filepath"
        "plugin"
        "runtime"
    )
    
    func main() {
        // Determine plugin to load (expect it in the binary's directory).
        ex, err := os.Executable()
        if err != nil {
            panic(err)
        }
        exPath := filepath.Dir(ex)
        pluginPath := "plugin-linux-arm64.so"
        if runtime.GOARCH == "amd64" {
            pluginPath = "plugin-linux-amd64.so"
        }
        pluginPath = filepath.Join(exPath, pluginPath)
    
        // Load the plugin
        p, err := plugin.Open(pluginPath)
        if err != nil {
            fmt.Fprintf(os.Stderr, "error loading plugin %q\n", pluginPath)
            panic(err)
        }
    
        // Verify that we can find the expected symbol
        expectedSymbol := "MustExist"
        sym, err := p.Lookup(expectedSymbol)
        if err != nil {
            fmt.Fprintf(os.Stderr, "error connecting symbol %q\n", expectedSymbol)
            panic(err)
        }
        _ = sym
    
        // Verify value of the symbol is "bunny"
        expectedValue := "bunny"
        value := sym.(*string)
        if *value != expectedValue {
            fmt.Fprintf(os.Stderr, "error: expected symbol %q to have value %q, got %q\n", expectedSymbol, expectedValue, *value)
            panic(err)
        }
    
        // Echo input
        in, err := ioutil.ReadAll(os.Stdin)
        if err != nil {
            log.Fatal(err)
        }
        if string(in) == "fail" {
            fmt.Fprintf(os.Stderr, "%s", in)
            os.Exit(1)
        }
        fmt.Fprintf(os.Stdout, "%s", string(in))
    }
    Build call
    GOOS=linux GOARCH=arm64 go build -trimpath -o main main.go
  3. Execute binary on linux/arm64 machine

Also, find the repository with the code here.

What did you expect to see?

$ echo hello | ./main
hello

What did you see instead?

Fails as follows:

$ echo hello | ./main
error loading plugin "/home/ec2-user/plugintest/plugin-linux-arm64.so"
panic: plugin: not implemented

goroutine 1 [running]:
main.main()
        ./main.go:30 +0x110
Additional information about the files
$ file plugin-linux-arm64.so
plugin-linux-arm64.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fb52a1851195748e408d2d17b2ecf26472d4f955, not stripped
$ ldd plugin-linux-arm64.so
        linux-vdso.so.1 (0x0000ffff8b437000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000ffff8b2bd000)
        libc.so.6 => /lib64/libc.so.6 (0x0000ffff8b137000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff8b3f9000)
$ file main
main: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped
$ ldd main
        not a dynamic executable

Note: If plugin and binary are compiled on the same platform, everything works as expected.

Thanks for the great work on this project! 🤗

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 26, 2022
@cherrymui
Copy link
Member

Plugins needs cgo. You need to enable cgo and use a C cross compiler. Try something like
CGO_ENABLED=1 CC=/path/to/C-cross-compiler go build ....

@cherrymui cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jul 26, 2022
@merschformann
Copy link
Author

Thank you for the clarification! That makes sense.

Out of curiosity, we face the challenge of compiling on various platforms (darwin/xxx, linux/xxx - diverse distros) targeting one specific platform (linux/arm64 - docker image under our control). Are there best practices for building so that links don't break? (both plugin and binary need to be dynamically linked as far as I understand) I.e., is it possible to bundle the cross-compiler and make sure it's linking the dependencies so that it works on the target platform?

Anyway, any further guidance appreciated, but this is off-topic. The cross-compiler works when building plugin & binary the same way (currently using aarch64-linux-gnu-gcc). Closing this.

Thanks @cherrymui !

@cherrymui
Copy link
Member

Thanks for confirming!

C cross compilation is by itself a tricky topic. I'm sorry I don't have suggestion for best practices. For cross compiling Go programs using cgo, CGO_ENABLED=1 CC=/path/to/C-cross-compiler is generally a good starting point.

@golang golang locked and limited conversation to collaborators Jul 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

3 participants