I also discovered this oddity: in addition to a plug-in needing to be built against the same sources, the value of GOPATH must also be the same:

# extract the GOPATH to two temp dirs
$ tar xzf ../rexray/gopath-0.9.0+15-1276481.tar.gz -C /tmp/tmp.9jaJCm8Aqd
$ tar xzf ../rexray/gopath-0.9.0+15-1276481.tar.gz -C /tmp/tmp.Na8C0iypRG

# no differences
$ diff -qr /tmp/tmp.9jaJCm8Aqd /tmp/tmp.Na8C0iypRG

# build plugin with tmp.9jaJCm8Aqd
$ GOPATH="/tmp/tmp.9jaJCm8Aqd:$GOPATH" go build \
  -tags "pflag gofig" -o /home/akutz/.libstorage/var/lib/mod/mock.so \
  -buildmode plugin ./mod

# build lsx-linux with tmp.Na8C0iypRG
$ GOPATH="/tmp/tmp.Na8C0iypRG:$GOPATH" ./gob ./cli/lsx/lsx-linux

# plug-in fails to load
$ ./lsx-linux
ERRO[0000] error opening module                          error=plugin.Open: plugin was built with a different version of package github.com/spf13/pflag path=/home/akutz/.libstorage/var/lib/mod/mock.so time=1495663533919

# build lsx-linux with tmp.9jaJCm8Aqd, same GOPATH used to build plug-in
$ GOPATH="/tmp/tmp.9jaJCm8Aqd:$GOPATH" ./gob ./cli/lsx/lsx-linux

# plug-in loads successfully
$ ./lsx-linux
usage: ./lsx-linux <executor> supported
                              instanceID
                              nextDevice
                              localDevices <scanType>
                              wait <scanType> <attachToken> <timeout>
                              mounts
                              mount [-l label] [-o options] device path
                              umount path

       executor:    mock
                    vfs

       scanType:    0,quick | 1,deep

       attachToken: <token>

       timeout:     30s | 1h | 5m

Even when using a common GOPATH to build the program and the plug-in, the contents of the GOPATH aren't enough, but they have to share the same root path as well. In essence, even if GOPATH=/tmp/1 and GOPATH=/tmp/2 are completely identical in what sources they have, down to the checksum of every file, if you build the program against one and the plug-in against the other, it fails. When the code is built, the value of GOPATH must be the same in both instances.

# remove the GOPATH used to build lsx-linux & mock.so
$ rm -fr /tmp/tmp.9jaJCm8Aqd

# recreate the GOPATH with identical contents
$ cp -fr /tmp/tmp.Na8C0iypRG /tmp/tmp.9jaJCm8Aqd

# rebuild lsx-linux
$ GOPATH="/tmp/tmp.9jaJCm8Aqd:$GOPATH" ./gob ./cli/lsx/lsx-linux

# the plug-in is loaded successfully
$ ./lsx-linux
usage: ./lsx-linux <executor> supported
                              instanceID
                              nextDevice
                              localDevices <scanType>
                              wait <scanType> <attachToken> <timeout>
                              mounts
                              mount [-l label] [-o options] device path
                              umount path

       executor:    mock
                    vfs

       scanType:    0,quick | 1,deep

       attachToken: <token>

       timeout:     30s | 1h | 5m

tl;dr: For plug-ins to work, not only must the sources in the GOPATH be the same for a plug-in and the program that loads it, the value of GOPATH must be the same when building the program and plug-in.

Hi,

It just occurred to me that the behavior outlined in the above comment results in the inability to build Go programs targeting a UNIX GOOS from a Windows host OS as the host can not have an identical GOPATH value as a UNIX host. Not unless using Cygwin or some other tool to mimic a UNIX path structure.

Allow plug-ins to link directly against the Go program binary that will load the programs.

This sort of linking is directly supported by the Mach-O binary format used by Darwin (macOS/iOS/etc). The linker flag is -bundle_loader, and the ld(1) manpage documents it as:

      -bundle_loader executable
                 This specifies the executable that will be loading the bundle output file being linked.
                 Undefined symbols from the bundle are checked against the specified executable like it was
                 one of the dynamic libraries the bundle was linked with.

This is used when building a special "dynamically bound bundle" file type, which is a separate file type from any of a shared library, static library, or executable.

I guess that bodes well for when plugin support gets added for Darwin, but I don't know that ELF has baked in support for plugins using their host binary's names like that.

Hi @jeremy-w,

Interesting. That is very useful...for Darwin. Which doesn't yet have Go plug-in support :) Which sucks, because Darwin is my primary development platform...

For now, this is what I'm doing:

• Declaring that all participants in my software ecosystem (projects/products) must define their GOPATH as /tmp/go if they wish to either load or be plug-ins thanks to the issue I discovered in this comment above.
• On Travis-CI I'm using the following steps to copy my vendored sources to the appropriate location:

  - glide install
  - export PROJECT_NAME="rexray"
  - export GOPATH_OLD="$GOPATH"
  - export GOPATH="/tmp/go"
  - mkdir -p "$GOPATH"/{bin,pkg,src}
  - mv "$GOPATH_OLD"/bin/* "$GOPATH"/bin/
  - export PATH="${GOPATH}/bin:${PATH}"
  - mkdir -p "$GOPATH"/src/github.com/codedellemc
  - rsync -ax vendor/ "$GOPATH"/src/ && rm -fr vendor
  - cd .. && mv "$PROJECT_NAME" "$GOPATH"/src/github.com/codedellemc/
  - cd "$GOPATH"/src/github.com/codedellemc/"$PROJECT_NAME"

• Before I remove the vendor directory I use it to create a tarball to make it easy for others to build plug-ins. The below commands use tar's transform functionality to change the leading path from vendor/ to src/, making it easy to treat the contents of the tarball, once extracted, as a GOPATH:

Darwin

tar \
  --exclude '.git' \
  --exclude 'vendor' \
  -czf "gopath.tar.gz" \
  -s ',^./,src/,' \
  -C vendor .

Linux

tar \
  --exclude '.git' \
  --exclude 'vendor' \
  -czf "gopath.tar.gz" \
  --xform 's,^./,src/,S' \
  -C vendor .

@akutz What does a dependency update strategy look like for an application and its plugins?

Hi @mattfarina,

I'm moving ahead with a different approach that renders your question moot. You'll see why in a minute.

Hi @spf13 / @mattfarina / @bradfitz / @jeremy-w,

As I just indicated to @mattfarina, I am moving forward with a solution that essentially circumvents the original problem. The complete details can be found at akutz/gpds. For convenience I am copying the document below. For the members of the Go team, I am really curious about the section Curious Exception. It's something I never considered, and if that's not weird enough, check out Exception to the Exception.

Go Plug-ins & Vendored Dependencies: A Solution

This document outlines a solution for the problem described in golang/go#20481. Please review the original problem description before continuing.

The problem was fairly straight-forward and ultimately so is the solution: a Go plug-in should not depend on a host process's symbols. That means:

• Go plug-ins should use a unidirectional model for type registration
• Go plug-ins should use interface{} for all non-stdlib types involved
  in ingress and egress host-plug-in communications

Unidirectional Model

Go supports the dependency inversion principle (DIP) through the use of interface abstractions, but there still must exist a mechanism to provide implementations of the abstractions on which a program depends. One such solution can be found in the list of suggested implementations of inversion of control (IoC): the service locator pattern.

The service locator pattern is very easy to implement in Go as a simple type registry. Consumers that require an implementation of some interface are able to query the type registry and receive an object instance that fulfills the abstraction. There are two models that can be used to prime the registry with types: bidirectional and unidirectional.

The above diagram is an example of the bidirectional model, but it fails when used in concert with Go plug-ins due to the issues with dependencies outlined in golang/go#20481. The solution is a unidirectional model:

Illustrated in the diagram above, the unidirectional model provides the same type registry that the bidirectional model does but relocates type registration from the plug-ins' init functions to the host process. This change means the plug-ins no longer depend on the type registry in the host process, and that's very important because a plug-in cannot depend on a host process's symbols.

Interface In / Interface Out

Go interfaces are really powerful, but they are also quick to cause issues when used with plug-ins for two reasons:

1. Interface equality is not as simple as it seems
2. The fully-qualified path to an interface matters

Interface Equality

The following examples demonstrate the power and peril of using Go interfaces interchangeably having assumed equality. The first example defines two, identical interfaces, dog and fox, and two structs, best friends that implement the interfaces, copper and todd (run example):

package main

import (
	"fmt"
)

type dog interface {
	bark() string
}

type fox interface {
	bark() string
}

type copper struct{}

func (c *copper) bark() string { return "woof!" }

type todd struct{}

func (t *todd) bark() string { return "woof!" }

func barkWithDog(d dog) { fmt.Println(d.bark()) }
func barkWithFox(f fox) { fmt.Println(f.bark()) }

func main() {
	var d dog = &copper{}
	var f fox = &todd{}
	barkWithDog(d)
	barkWithFox(f)
}

The above code, when executed, will print woof! on two lines. The first line is the result of the dog Copper barking, and the second line is his friend Todd the fox taking a turn. However, what makes Copper a dog or Todd a fox? According to the code it's because copper implements the function bark() string from the dog interface and todd implements the same function from the fox interface.

Does that mean that copper and todd are interchangeable? In fact, the two friends decided to pretend to be one another in order to play a trick on the kind old lady and hunter (run example):

func main() {
	var d dog = &todd{}
	var f fox = &copper{}
	barkWithDog(f)
	barkWithFox(d)
}

How can Todd be a fox and Copper a dog? According to Go's interface rules, a variable of type fox can be assigned any type that implements the bark() string function. A function that has an
argument of type dog or fox can also accept any type that implements the bark() string function, even if that type is another interface.

It would appear then that multiple Go interfaces, if they define the same abstraction, are identical. However, thanks to Go's strong type system, interfaces are not as interchangeable as they first appear (run example):

package main

import (
	"fmt"
)

type dog interface {
	bark() string
	same(d dog) bool
}

type fox interface {
	bark() string
	same(f fox) bool
}

type copper struct{}

func (c *copper) bark() string    { return "woof!" }
func (c *copper) same(d dog) bool { return c == d }

type todd struct{}

func (t *todd) bark() string    { return "woof!" }
func (t *todd) same(f fox) bool { return t == f }

func barkWithDog(d dog) { fmt.Println(d.bark()) }
func barkWithFox(f fox) { fmt.Println(f.bark()) }

func main() {
	var d dog = &todd{}
	var f fox = &copper{}
	barkWithDog(f)
	barkWithFox(d)
}

The above example will no longer emit the sound of two friends barking, but rather the following errors:

tmp/sandbox006620983/main.go:31: cannot use todd literal (type *todd) as type dog in assignment:
	*todd does not implement dog (wrong type for same method)
		have same(fox) bool
		want same(dog) bool
tmp/sandbox006620983/main.go:32: cannot use copper literal (type *copper) as type fox in assignment:
	*copper does not implement fox (wrong type for same method)
		have same(dog) bool
		want same(fox) bool
tmp/sandbox006620983/main.go:33: cannot use f (type fox) as type dog in argument to barkWithDog:
	fox does not implement dog (wrong type for same method)
		have same(fox) bool
		want same(dog) bool
tmp/sandbox006620983/main.go:34: cannot use d (type dog) as type fox in argument to barkWithFox:
	dog does not implement fox (wrong type for same method)
		have same(dog) bool
		want same(fox) bool

The relevant piece of information from the above error text is the following:

have same(fox) bool
want same(dog) bool

In other words, even though Go interfaces A and B are identical, A{A} and B{B} are not. If A==B and C==D, A{C} != B{D}.

Because of this rule, without a shared types library, even with Go interfaces, it's not possible for Go plug-ins to expect to share or use symbols provided by the host process.

Fully-Qualified Type Path

However, even redefining interfaces inside plug-ins to match types found in the host process will fail if those interfaces are used by exported symbols. This section uses this project's dog package. The following command will get the package and build its plug-ins:

$ go get github.com/akutz/gpds && \
  cd $GOPATH/src/github.com/akutz/gpds/dog && \
  for d in $(find . -maxdepth 1 -type d | grep -v '^.$'); do \
    go build -buildmode plugin -o $d.so $d; \
  done

Run the program using the sit.so plug-in:

$ go run main.go dog.go sit.so
error: invalid Command func: func(main.dog)
exit status 1

An error occurs because the sit.so plug-in defines an interface dog to match the host program's interface Dog. Both interfaces include a single function: Name() string. However, these types are different because their fully-qualified type paths (FQTP) are not the same. An FQTP includes the package path to a type and the type's name, where the name is case sensitive (since case sensitivity is used by Go to indicate public and private members).

Therefore invoking the Command(Dog) function fails, because while the interface definitions are identical with regards to the equality ruleset outlined above, the two interfaces do not have the same FQTP.

Curious Exception

There is one curious exception to this rule: when an interface is defined in the main package of the host program as well as the main package of the plug-in. Run the program using the speak.so plug-in:

$ go run main.go dog.go speak.so
Lucy

The program should have printed the name "Lucy". However, if the code is examined, the Dog interface is defined in both the host program and in the plug-in package. Yet it works. Why? The answer is almost so embarrassingly obvious that it makes this author hesitant to admit it took him an hour of looking at the problem to figure it out.

Both interfaces have a fully-qualified package path of main.Dog.

When interfaces are defined in the main package of the hosting program and in the main package of a plug-in, their symbols are identical. However, like most things, there's an exception to even this.

Exception to the Exception

What happens if the Dog interface references itself? The answer is an error this author has never seen before in his history of working with the Go programming language. To reproduce this error, run the program using the stay.so plug-in:

$ go run main.go self.go stay.so
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

runtime stack:
runtime.throw(0x534aad, 0xe)
	/home/akutz/.go/1.8.1/src/runtime/panic.go:596 +0x95
runtime.newstack(0x0)
	/home/akutz/.go/1.8.1/src/runtime/stack.go:1089 +0x3f2
runtime.morestack()
	/home/akutz/.go/1.8.1/src/runtime/asm_amd64.s:398 +0x86

goroutine 1 [running]:
runtime.(*_type).string(0x7f2488279520, 0x0, 0x0)
	/home/akutz/.go/1.8.1/src/runtime/type.go:45 +0xad fp=0xc44009c358 sp=0xc44009c350
runtime.typesEqual(0x7f2488279520, 0x51d0c0, 0x50a310)
	/home/akutz/.go/1.8.1/src/runtime/type.go:543 +0x73 fp=0xc44009c480 sp=0xc44009c358
runtime.typesEqual(0x7f2488270740, 0x5137c0, 0x5137c0)
	/home/akutz/.go/1.8.1/src/runtime/type.go:586 +0x368 fp=0xc44009c5a8 sp=0xc44009c480
runtime.typesEqual(0x7f2488279520, 0x51d0c0, 0x50a310)
	/home/akutz/.go/1.8.1/src/runtime/type.go:615 +0x740 fp=0xc44009c6d0 sp=0xc44009c5a8
...additional frames elided...

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/home/akutz/.go/1.8.1/src/runtime/asm_amd64.s:2197 +0x1
exit status 2

The above program fails due to a Go runtime panic where Go is recursively trying to determine if the main.Dog interface from the host program is the same type as the main.Dog interface defined in the plug-in. The interfaces were considered the same when they did not reference themselves with their respective Self() Dog functions.

The Solution

The proposed solution adheres to the crucial restriction outlined at the beginning of this document -- Go plug-ins should not depend on a host program's symbols. This project is used to demonstrate a program and plug-ins that:

• Use a unidirectional model for type registration
• Use interface{} for all non-stdlib types involved in ingress and egress
  host-plug-in communications

To get started please clone this repository:

$ go get github.com/akutz/gpds && cd $GOPATH/src/github.com/akutz/gpds

Running the program will emit a little ditty by Mr. Chapin:

$ go run main.go
Yes, we have no bananas,
We have no bananas today.

Next, build the plug-in mod.so:

$ go build -buildmode plugin -o mod.so ./mod

Running the program with the plug-in will cause the output to be somewhat altered:

$ go run main.go mod.so
*main.v2Config
Yes there were thirty, thousand, pounds...
Of...bananas.

*main.v2Config
Bottom-line, sh*t kicking country choir
You'll see your part come by

The above steps do not appear to illustrate anything too fancy, but under the covers is a model that enables Go plug-ins to work alongside vendoered dependencies with ease. Pulling back the covers ever so slightly reveals how it all works.

Lib

For starters, this is the Module interface defined in ./lib/lib.go:

// Module is the interface implemented by types that
// register themselves as modular plug-ins.
type Module interface {

	// Init initializes the module.
	//
	// The config argument can be asserted as an implementation of the
	// of the github.com/akutz/gpds/lib/v2.Config interface or older.
	Init(ctx context.Context, config interface{}) error
}

A Module interface includes a single function, Init, which accepts a Go context and second argument of type interface{}. The second argument is expected to be a sort of configuration provider, provided to plug-ins to inform their initialization routine. Please note the Godoc for the argument:

The config argument can be asserted as an implementation of the
of the github.com/akutz/gpd/lib/v2.Config interface or older.

The documentation indicates which type the argument can be asserted as, and more importantly explains that the object provided can be asserted as a specific version of that type or older. All types should be versioned and plug-ins that assert v1.Type should continue to work even if v1+.Type is provided.

Please note that this model could be further enhanced so that plug-ins provide a symbol that contains the expected API version so that host programs can eventually deprecate older types by restricting which plug-ins get loaded based on their supported API type.

Mod

The file ./mod/mod.go is the core of the plug-in. At the top of the
file is the Types symbol:

// Types is the symbol the host process uses to
// retrieve the plug-in's type map
var Types = map[string]func() interface{}{
	"mod_go": func() interface{} { return &module{} },
}

The Types symbol is very important, and the model proposed in this project expects all plug-ins to define this symbol. The symbol is a type map that is used by the host program to register the plug-in's type names and functions to construct them.

The module mod_go referenced in the plug-in's type map looks like this:

type module struct{}

func (m *module) Init(ctx context.Context, configObj interface{}) error {
	config, configOk := configObj.(Config)
	if !configOk {
		return errInvalidConfig
	}
	fmt.Fprintf(os.Stdout, "%T\n", config)
	fmt.Fprintln(os.Stdout, config.Get(ctx, "bananas"))
	return nil
}

Since this is an example the plug-in's module only defines a barebones implementation of the Module interface. The module's Init function first asserts that the provided configObj argument can be asserted as the Config interface and then uses the typed object to retrieve and print bananas.

But wait, how is the plug-in able to assert the interface Config if the plug-in is not sharing any symbols with the host process? Simple, the plug-in simply treats the sources in the versioned lib package as C headers and copies the v1 or v2 headers into the plug-in's own package.

Conclusion

Hopefully this document has not only shown how to solve the issue of Go plug-ins and vendored dependencies, but also clearly articulated the reasoning behind the decisions that led to the proposed solution.

I believe this is a duplicate of #18827, no?

I just made a comment over on #18827. Ideally we would have one issue for discussing this. If the problem here is the same as that one (packages under the vendor directory are not the same at run time, just like in non-plugin programs), let's discuss it over there and please close this.

That said, @akutz I cannot follow how your solution "a Go plug-in should not depend on a host process's symbols" would work. If we don't use the host symbols, then there is no overlap between packages at all. A plugin cannot implement an http handler.

Hi @crawshaw,

Thank you very much for the response. Please allow me to address a few things:

1. My remarks regarding host symbols do not include types belonging to stdlib. However, a plug-in built with one version of Go cannot be loaded into a program built with a different version of Go. Even the host symbols are beholden to the same constraints as any other package.
2. There are two issues with regards to shared symbols:
   1. Loading a plug-in into a host program - A host program and plug-in that both import github.com/hello/world are compatible only if one of the following is true:
      1. The host program and plug-in both import the exact same sources from the same location for github.com/hello/world.
      2. Either one of the host program or plug-in vendor the sources, same or different, for github.com/hello/world.
   2. Sharing data between a plug-in and a host program - A plug-in and host program can use types from github.com/hello/world to share data only iff:
      1. The host program and plug-in both import the exact same sources from the same location for github.com/hello/world.

The point I was clumsily trying to make is that it's so overwhelmingly cumbersome to build plug-ins that can be loaded into host programs and share data via shared types that the utility of what plug-ins generally provide is diminished.

Having thought about this matter quite a bit, these are the occasions when, in my mind, Go plug-ins make sense:

1. Go plug-ins are built at the same time as the host program in order to provide a base program and additional functionality that can be enabled at runtime by loading plug-ins. Building the plug-ins at the same time as the host program, using the exact same sources, can ensure the exact same dependency graph and ensure that the plug-ins will load into the host program and use shared types to share data.
2. Shared types are not used to share data unless:
   1. The types belong to stdlib.
   2. The types are interface{} references that can be asserted as Go interfaces defined in both the host program and in the plug-in.
3. Shared types are not used to share data, but instead data is shared via marshalling/unmarshalling via JSON/gRPC/etc. into types defined both in the host program and in the plug-in.

Hopefully this helps clear up some of my original intent. Thank you again for your response!

--
-a

Hi,

I'm also getting same error,
**linux1@sfhyperledger:~/go/src/syndicatedLoans> go build

github.com/hyperledger/fabric/vendor/github.com/miekg/pkcs11

exec: "s390x-linux-gnu-gcc": executable file not found in $PATH

github.com/hyperledger/fabric/vendor/plugin

../github.com/hyperledger/fabric/vendor/plugin/plugin.go:32: undefined: open
../github.com/hyperledger/fabric/vendor/plugin/plugin.go:40: undefined: lookup**

Could u tell me solution?

@SreekanthSimplyfi That is a different problem. Please see https://golang.org/wiki/Questions .

@ianlancetaylor problem is solved from this link
https://github.com/linux-on-ibm-z/docs/wiki/Building-Go

Would it be feasible to add a command line option to go build that would direct the compiler to modify (prefix or otherwise) names in the symbol table?
It occurs to me that in my .go sources I can refer to a common dependency by a different name and satisfy my build with a replace directive in go.mod. That allows me to have a main application project and a plugin project owned by separate teams and cycles using the same dependency on different versions for internal stuff.
They key word in my opinion here is "internal". If my plugin architecture only exports symbols that deal with primitives, the main application and the plugin should not be forced to share verbatim machine code from common packages even from the standard library, and this could be achieved by manipulating the names in the symbol table on the plugin build.

Note that the replace approach for changing the symbol table will probably stop working — or, at least, need to be applied in the opposite direction — when we address #26904.

Duplicate of #18827