I'm a little confused. When you build the plugin, you use -trimpath for go build. But when you build the main binary, you only use -trimpath for -gcflags. How does other pkghash
equal, such as runtime? As I know, the hash of every package is related to the path of it. The path is different when "-trimpath" is enabled.

I'm a little confused. When you build the plugin, you use -trimpath for go build. But when you build the main binary, you only use -trimpath for -gcflags. How does other pkghash equal, such as runtime? As I know, the hash of every package is related to the path of it. The path is different when "-trimpath" is enabled.

The general -trimpath is used for the main binary in addition to the asmflags and gcflags trimpath, but it is set via GOFLAGS env var

Our main binary is compiled with the following go build and GOFLAGS contains -trimpath

Am pretty confident this is working, as we don't see any issue with Go 1.17.8 or older - this problem only occurs with Go 1.18... and the error occurs only for a particular package that uses cgo. Other plugins that don't use our github.com/sylabs/singularity/internal/pkg/util/user package are fine with Go 1.18.

I can pull out the strings section for Go 1.17.8 (which works) tomorrow, with exactly the same build flags / GOFLAGS to clarify a bit further.

I moved -trimpath from GOFLAGS to an explicit flag on go build, so that's less confusing. Have also added below a direct comparison of the package strings between Go 1.17.8 and Go 1.18.

The main binary now builds with:

go build 
  -mod=readonly
  -buildmode=pie
  -trimpath
  -tags "containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper sylog singularity_engine fakeroot_engine apparmor selinux seccomp"
  -ldflags="-B 0x`head -c20 /dev/urandom|od -An -tx1|tr -d ' \n'`"
  -gcflags=github.com/sylabs/singularity/...="-trimpath /home/dtrudg/Git_sylabs/singularity=>github.com/sylabs/singularity@v0.0.0"
  -asmflags=github.com/sylabs/singularity/...="-trimpath /home/dtrudg/Git_sylabs/singularity=>github.com/sylabs/singularity@v0.0.0" 
  -o cmd/starter/c/starter
  /home/dtrudg/Git_sylabs/singularity/cmd/starter/main_linux.go

The plugin now builds with:

go build 
  -a 
  -o /tmp/plugin-1227855955/src/plugin.so 
  -mod=readonly 
  -ldflags -X main.PluginRootDir=/usr/local/libexec/singularity/plugin 
  -trimpath
  -buildmode=plugin 
  -tags containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper sylog singularity_engine fakeroot_engine apparmor selinux seccomp 
  .

As before, everything works in Go 1.17.8 (or prior), but not with Go 1.18...

With Go 1.17.8 plugin.Open succeeds

The main binary and plugin build correctly, plugin.Open succeeds, and the plugin can be used.

Strings for internal/pkg/util/user during main binary build with 1.17.8:

Lgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/identity_unix.go�Name�UID�GID�Gecos�Dir�Shell�uid�User�~r1�~r2
.autotmp_3
.autotmp_4
lookupUnixUidNgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/cgo_lookup_unix.go�name
lookupUser�username�gid�Group
lookupUnixGid�lookupGroup	groupname�~r0
.autotmp_2�currentIgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/membership.go�list�esc:�user	.inittask�Current�CurrentOriginal�GetGrGID�GetGrNam�GetPwNam�GetPwUID
UIDInAnyGroup	UIDInList

Strings for internal/pkg/util/user/ during plugin build with 1.17.8:

Lgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/identity_unix.go�Name�UID�GID�Gecos�Dir�Shell�uid�User�~r1�~r2
.autotmp_3
.autotmp_4
lookupUnixUidNgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/cgo_lookup_unix.go�name
lookupUser�username�gid�Group
lookupUnixGid�lookupGroup	groupname�~r0
.autotmp_2�currentIgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/membership.go�list�esc:�user	.inittask�Current�CurrentOriginal�GetGrGID�GetGrNam�GetPwNam�GetPwUID
UIDInAnyGroup	UIDInList

These are identical between main binary and plugin, so there is no pkghash mismatch / failure with plugin.Open

Note that there is no path information in either. There are references to autotmp_X which aren't present in the Go 1.18 strings below.

There is also no reference to _cgo_gotypes.go which is the string which has an extra prefixed path in the main binary build for Go 1.18, causing the mismatch with the Go 1.18 plugin.

With Go 1.18 plugin.Open fails

plugin.Open("/tmp/plugin-1227855955/src/plugin"): plugin was built with a different version of package github.com/sylabs/singularity/internal/pkg/util/user

Strings for internal/pkg/util/user during main binary build with 1.18:

Lgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/identity_unix.go�Name�UID�GID�Gecos�Dir�Shell�uid�User�~r0�~r1�$autotmp
lookupUnixUidNgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/cgo_lookup_unix.go�name
lookupUser�username�gid�Group
lookupUnixGid�lookupGroup	groupname�currentIgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/membership.go�list�esc:+/tmp/go-build552663777/b050/_cgo_gotypes.go�user	.inittask�Current�CurrentOriginal�GetGrGID�GetGrNam�GetPwNam�GetPwUID
UIDInAnyGroup	UIDInList

Strings for internal/pkg/util/user during plugin build with 1.18.

Lgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/identity_unix.go�Name�UID�GID�Gecos�Dir�Shell�uid�User�~r0�~r1�$autotmp
lookupUnixUidNgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/cgo_lookup_unix.go�name
lookupUser�username�gid�Group
lookupUnixGid�lookupGroup	groupname�currentIgithub.com/sylabs/singularity@v0.0.0/internal/pkg/util/user/membership.go�list�esc:�_cgo_gotypes.go�user	.inittask�Current�CurrentOriginal�GetGrGID�GetGrNam�GetPwNam�GetPwUID
UIDInAnyGroup	UIDInList

The strings for internal/pkg/util/user from compilation of the main binary differ by having a: /tmp/go-build552663777/b050/ path prior to _cgo_gotypes.go while the strings from compilation of the plugin does not have this path. Therefore the pkghash differs so the plugin fails to load.

I can build other plugins that don't use internal/pkg/util/user without issue. There is no mismatch of pkghash / strings for the other packages between the main binary and plugin build.

By looking into src/cmd/compile/internal/noder/noder.go trimFilename() which calls src/cmd/internal/objabi/line.go Absfile I think I've found this is related to building with a -trimpath inside -gcflags on Go 1.18.

When building with a -trimpath in -gcflags...

/home/dave/Git_sylabs/go/bin/go build 
-mod=readonly 
-buildmode=pie 
-trimpath 
-tags "containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper sylog singularity_engine fakeroot_engine apparmor selinux seccomp" 
-ldflags="-B 0x`head -c20 /dev/urandom|od -An -tx1|tr -d ' \n'`" 
-gcflags=github.com/sylabs/singularity/...="-trimpath /home/dtrudg/Git_sylabs/singularity=>github.com/sylabs/singularity@v0.0.0" 
-asmflags=github.com/sylabs/singularity/...="-trimpath /home/dtrudg/Git_sylabs/singularity=>github.com/sylabs/singularity@v0.0.0" 
-o cmd/starter/c/starter-suid 
/home/dtrudg/Git_sylabs/singularity/cmd/starter/main_linux.go

... it's apparent that AbsFile doesn't receive the rewrite that will remove the /tmp/gobuildxxxxx path for the _cgo_gotypes.go. It only recieves the rewrite that was set via the -gcflags:

# github.com/sylabs/singularity/internal/pkg/util/user
AbsFile dir:/home/dtrudg/Git_sylabs/singularity file:/tmp/go-build2238438623/b050/_cgo_gotypes.go rewrites:/home/dtrudg/Git_sylabs/singularity=>github.com/sylabs/singularity@v0.0.0
trimFilename: b.Filename()=/tmp/go-build2238438623/b050/_cgo_gotypes.go b.Trimmed()=false b.IsFileBase()=true filename=/tmp/go-build2238438623/b050/_cgo_gotypes.go

If I drop the -gcflags from go build then AbsFile is getting a rewrite that removes the /tmp/go-buildxxxx from the _cgo_gotypes.go` path:

/home/dave/Git_sylabs/go/bin/go build
-mod=readonly
-buildmode=pie
-trimpath
-tags "containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper sylog singularity_engine fakeroot_engine apparmor selinux seccomp"
-ldflags="-B 0x`head -c20 /dev/urandom|od -An -tx1|tr -d ' \n'`"
-asmflags=github.com/sylabs/singularity/...="-trimpath /home/dtrudg/Git_sylabs/singularity=>github.com/sylabs/singularity@v0.0.0"
-o cmd/starter/c/starter-suid
/home/dtrudg/Git_sylabs/singularity/cmd/starter/main_linux.go 

# github.com/sylabs/singularity/internal/pkg/util/user
AbsFile dir:/home/dtrudg/Git_sylabs/singularity file:/tmp/go-build1956690545/b050/_cgo_gotypes.go rewrites:/home/dtrudg/Git_sylabs/singularity/internal/pkg/util/user=>github.com/sylabs/singularity/internal/pkg/util/user;/tmp/go-build1956690545/b050=>
trimFilename: b.Filename()=/tmp/go-build1956690545/b050/_cgo_gotypes.go b.Trimmed()=false b.IsFileBase()=true filename=_cgo_gotypes.go

So, similar to #47256 , use the high level -trimpath and not the individual tool flags.
In which case I don't think there's anything to do here?

So, similar to #47256 , use the high level -trimpath and not the individual tool flags.
In which case I don't think there's anything to do here?

Our main binary compilation uses the -trimpath in the tool flags as we have a somewhat wacky process of using runtime.Caller(0) and some substitution to identify our own source path so that our CLI can provide a command to compile plugins code that is not within our source tree.

I'll look into whether we can do this more sanely, but at the time it was the implementation that a developer came up with to allow people to build plugins via our own CLI.

I guess I'm interested in pursuing this a bit more, as it used to work up to and including 1.17 - and presumably the gcflag trimpath could just be appended to the rewrites that AbsFile sees, rather than replace them... which would maintain the 1.17 behavior?

Edit - understood that there may not be any appetite for pursuing this further, though, so I'll put most of my efforts on us coming up with something a bit more straightforward to avoid the issue :-)

Edit 2 - there may more to it, w.r.t. the reason for asmflags and gcflags... we're using CGO in our main binary and plugins, and I noticed mention of #27751, #36072 in the linked #47256 - I'll stop speculating and look into it properly.

Having come across a definitive message, while browsing through issues, that...

rsc commented on Apr 22, 2019
Direct use of -gcflags=-trimpath is unsupported.

... we'll drop back to a more restrictive approach for building plugins in our app, that doesn't require that.

Apologies for the noise!