Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: import paths of plugin dependencies redefined using modules #28983

Closed
oszika opened this issue Nov 28, 2018 · 6 comments
Closed

cmd/go: import paths of plugin dependencies redefined using modules #28983

oszika opened this issue Nov 28, 2018 · 6 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@oszika
Copy link

oszika commented Nov 28, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/oszika/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/oszika/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build685214377=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Using plugins with modules has become very restrictive. All common dependencies between plugins and binary that loads them, must be exactly at the same version.
With GOPATH, to avoid that, we can use vendoring because import paths are rewritten. (Ex: gopkg.yaml.v2 is rewritten to package/sub/vendor/gopkg.in/yaml.v2)
But with modules and vendoring, import paths are preserved.

Is this the desired behavior? If so, then the plugins are difficult to use by other users.
Maybe it would be possible to accept that module import paths are rewritten for better modularity, even if it increases the number of dependencies?
In addition, if the vendoring is removed, this feature will be removed too.

Example:

  • main/main.go : to load plugins
  • p1/p1.go : plugin
  • p2/p2.go : like p1.go (only print differs)
  • script.sh : builds plugins using GOPATH, or modules, and prints import path of gopkg.in/yaml.v2 dependency.

p1/p1.go

package main

import (
    "fmt"

    yaml "gopkg.in/yaml.v2"
)

var e = yaml.Encoder{}

func init() { fmt.Printf("p1: Type: %T\n", e) }

func main() {}

main/main.go

package main

import "plugin"

func main() {
    if _, err := plugin.Open("../p1/p1.so"); err != nil {
        panic(err)
    }
    if _, err := plugin.Open("../p2/p2.so"); err != nil {
        panic(err)
    }
}

script.sh

#!/bin/bash
set -e

GOPATH=~/go
p1=$PWD/p1
p2=$PWD/p2
main=$PWD/main

rm -f $p1/p1.so $p2/p2.so
GO111MODULE=on go clean -modcache
go clean -cache

# init vendoring
cd $p1 && GO111MODULE=on go mod vendor
cd $p2 && GO111MODULE=on go mod vendor

echo -e "package yaml\n\nimport \"fmt\"\nfunc init() { fmt.Println(\"Init yaml (p1)\") }" > $p1/vendor/gopkg.in/yaml.v2/init.go
echo -e "package yaml\n\nimport \"fmt\"\nfunc init() { fmt.Println(\"Init yaml (p2)\") }" > $p2/vendor/gopkg.in/yaml.v2/init.go

# GOPATH + vendoring
echo -n "GOPATH+vendoring:"
cd $p1 && GO111MODULE=off go build -buildmode=plugin -n 2>&1 | grep '#' | grep gopkg.in/yaml

cd $p1 && GO111MODULE=off go build -buildmode=plugin 2>/dev/null >/dev/null
cd $p2 && GO111MODULE=off go build -buildmode=plugin 2>/dev/null >/dev/null
cd $main && GO111MODULE=off go run main.go

# GOMOD + vendoring
echo -n "GOMOD+vendoring:"
cd $p1 && GO111MODULE=on go build -buildmode=plugin -mod=vendor -n 2>&1 | grep '#' | grep gopkg.in/yaml

cd $p1 && GO111MODULE=on go build -buildmode=plugin -mod=vendor 2>/dev/null >/dev/null
cd $p2 && GO111MODULE=on go build -buildmode=plugin -mod=vendor 2>/dev/null >/dev/null
cd $main && GO111MODULE=off go run main.go

(Post on golang-nuts: https://groups.google.com/forum/#!topic/golang-nuts/qVg90eU7zfw)

What did you expect to see?

Using module, i would like to redefine import path plugin dependencies, and be able to load several plugins with same dependency at different version (using vendoring?).

Init yaml (p1)
p1: Type: yaml.Encoder
Init yaml (p2)
p2: Type: yaml.Encoder

What did you see instead?

  • output
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: downloading gopkg.in/yaml.v2 v2.2.1
go: finding gopkg.in/yaml.v2 v2.2.0
go: downloading gopkg.in/yaml.v2 v2.2.0
GOPATH+vendoring:# plugin/p1/vendor/gopkg.in/yaml.v2
Init yaml (p1)
p1: Type: yaml.Encoder
Init yaml (p2)
p2: Type: yaml.Encoder
GOMOD+vendoring:# gopkg.in/yaml.v2
Init yaml (p1)
p1: Type: yaml.Encoder
panic: plugin.Open("../p2/p2"): plugin was built with a different version of package gopkg.in/yaml.v2

goroutine 1 [running]:
main.main()
        /home/oszika/go/src/plugin/main/main.go:10 +0x97
exit status 2
@andybons andybons changed the title Import paths of plugin dependencies redefined using modules cmd/go: import paths of plugin dependencies redefined using modules Nov 28, 2018
@andybons andybons added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. modules labels Nov 28, 2018
@andybons andybons added this to the Unplanned milestone Nov 28, 2018
@andybons
Copy link
Member

@jayconrod @bcmills

@bcmills
Copy link
Contributor

bcmills commented Nov 28, 2018

But with modules and vendoring, import paths are preserved.
Is this the desired behavior?

Yes. (See also #20481.)

If so, then the plugins are difficult to use by other users.

If I understand correctly, in general the user must use a build of the plugin that exactly matches the sources and build configuration of the thing it plugs in to.

As far as I can tell, the use of modules does not change that fact: it just makes violations easier to detect.

@bcmills
Copy link
Contributor

bcmills commented Nov 28, 2018

In fact, if you build all of the plugins from within the main module, you'll find that both plugins use the version of gopkg.in/yaml.v2 selected by the main module itself, resolving the vendor version skew entirely.

That seems strictly preferable to using inconsistent versions of the vendored package and pretending that they're actually identical.


src/main$ find .. -type f
../main/p1.so
../main/main.go
../main/p2.so
../main/go.mod
../main/go.sum
../p2/main.go
../p2/vendor/gopkg.in/yaml.v2/yamlh.go
../p2/vendor/gopkg.in/yaml.v2/README.md
../p2/vendor/gopkg.in/yaml.v2/yaml.go
../p2/vendor/gopkg.in/yaml.v2/LICENSE
../p2/vendor/gopkg.in/yaml.v2/sorter.go
../p2/vendor/gopkg.in/yaml.v2/resolve.go
../p2/vendor/gopkg.in/yaml.v2/yamlprivateh.go
../p2/vendor/gopkg.in/yaml.v2/emitterc.go
../p2/vendor/gopkg.in/yaml.v2/scannerc.go
../p2/vendor/gopkg.in/yaml.v2/writerc.go
../p2/vendor/gopkg.in/yaml.v2/encode.go
../p2/vendor/gopkg.in/yaml.v2/go.mod
../p2/vendor/gopkg.in/yaml.v2/decode.go
../p2/vendor/gopkg.in/yaml.v2/init.go
../p2/vendor/gopkg.in/yaml.v2/readerc.go
../p2/vendor/gopkg.in/yaml.v2/parserc.go
../p2/vendor/gopkg.in/yaml.v2/.travis.yml
../p2/vendor/gopkg.in/yaml.v2/NOTICE
../p2/vendor/gopkg.in/yaml.v2/apic.go
../p2/vendor/gopkg.in/yaml.v2/LICENSE.libyaml
../p2/vendor/modules.txt
../p2/go.mod
../p2/go.sum
../p1/main.go
../p1/vendor/gopkg.in/yaml.v2/yamlh.go
../p1/vendor/gopkg.in/yaml.v2/README.md
../p1/vendor/gopkg.in/yaml.v2/yaml.go
../p1/vendor/gopkg.in/yaml.v2/LICENSE
../p1/vendor/gopkg.in/yaml.v2/sorter.go
../p1/vendor/gopkg.in/yaml.v2/resolve.go
../p1/vendor/gopkg.in/yaml.v2/yamlprivateh.go
../p1/vendor/gopkg.in/yaml.v2/emitterc.go
../p1/vendor/gopkg.in/yaml.v2/scannerc.go
../p1/vendor/gopkg.in/yaml.v2/writerc.go
../p1/vendor/gopkg.in/yaml.v2/encode.go
../p1/vendor/gopkg.in/yaml.v2/go.mod
../p1/vendor/gopkg.in/yaml.v2/decode.go
../p1/vendor/gopkg.in/yaml.v2/init.go
../p1/vendor/gopkg.in/yaml.v2/readerc.go
../p1/vendor/gopkg.in/yaml.v2/parserc.go
../p1/vendor/gopkg.in/yaml.v2/.travis.yml
../p1/vendor/gopkg.in/yaml.v2/NOTICE
../p1/vendor/gopkg.in/yaml.v2/apic.go
../p1/vendor/gopkg.in/yaml.v2/LICENSE.libyaml
../p1/vendor/modules.txt
../p1/go.mod
../p1/go.sum

src/main$ cat go.mod
module main

go 1.12

require (
        p1 v0.0.0
        p2 v0.0.0
)

replace (
        p1 => ../p1
        p2 => ../p2
)

src/main$ go build -buildmode=plugin p1 && go build -buildmode=plugin p2 && go run .
p1: Type: yaml.Encoder
p2: Type: yaml.Encoder

@bcmills bcmills closed this as completed Nov 28, 2018
@oszika
Copy link
Author

oszika commented Nov 28, 2018

Great! Thanks for the tip.
However, this still involves distributing the main application and the plugins together. This shifts the problem to the end user who would be forced to follow the same versions of plugins dependencies (in case we distribute plugins without the source code).

@bcmills
Copy link
Contributor

bcmills commented Nov 28, 2018

Yes, fundamentally you have to build the plugin with the same dependencies as the binary.

If you build in module mode without local-file replacements, you should be able to run go list -m all, filter the results to remove non-public modules, and ship the result for the other authors to use in their go.mod requirements.

@oszika
Copy link
Author

oszika commented Nov 29, 2018

Ok, thank you.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants