Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: embed directory fails when at / #49570

Closed
thomaspeugeot opened this issue Nov 13, 2021 · 5 comments
Closed

cmd/go: embed directory fails when at / #49570

thomaspeugeot opened this issue Nov 13, 2021 · 5 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@thomaspeugeot
Copy link

thomaspeugeot commented Nov 13, 2021

What version of Go are you using (go version)?

/ # go version
go version go1.16.10 linux/amd64

Does this issue reproduce with the latest release?

I do not know

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
 # go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.10"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2312574173=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Training for docker with go, I forked github.com/olliefr/docker-gs-ping and added embed directive

//go:embed foo
var foo embed.FS

it worked fined in my working directory.
But when I configured Dockerfile for having everything at root, it did not work.

# Set destination for COPY
WORKDIR /

To reproduce the bug

git clone https://github.com/fullstack-lang/docker-gs-ping.git
cd docker-gs-ping
docker build --tag docker-gs-ping .

What did you expect to see?

a successfull go build

What did you see instead?

 > [7/8] RUN go build -o /docker-gs-ping:
#11 2.002 # github.com/olliefr/docker-gs-ping
#11 2.002 /main.go:14:5: embed oo/foo.txt: open /oo/foo.txt: no such file or directory

A workaround is to have the WORKDIR different from "/"

@seankhliao seankhliao changed the title go embed fails at root level (first character is mangled) cmd/go: embed directory fails when module root is at / Nov 13, 2021
@seankhliao seankhliao changed the title cmd/go: embed directory fails when module root is at / cmd/go: embed directory fails when at / Nov 13, 2021
@seankhliao seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 13, 2021
@seankhliao
Copy link
Member

cc @bcmills @matloob

@ronaldpetty
Copy link

ronaldpetty commented Feb 26, 2022

This fails for 1.18rc1 as well (using root directory). Trying to learn a little here, seems to fail almost immediately in the build process. Shows truncating the leading / if it is in root.

Here we are in "/ (root)":

root@82481f861c36:/# go build -x -work -o a2 app.go 
WORK=/tmp/go-build2643592769
mkdir -p $WORK/b001/
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile embed=/usr/local/go/pkg/linux_amd64/embed.a
packagefile fmt=/usr/local/go/pkg/linux_amd64/fmt.a
packagefile log=/usr/local/go/pkg/linux_amd64/log.a
packagefile net/http=/usr/local/go/pkg/linux_amd64/net/http.a
packagefile time=/usr/local/go/pkg/linux_amd64/time.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
EOF
cat >$WORK/b001/embedcfg << 'EOF' # internal
{
	"Patterns": {
		"images/*": [
			"mages/me.jpg"
		],
		"index.html": [
			"index.html"
		]
	},
	"Files": {
		"index.html": "/index.html",
		"mages/me.jpg": "/mages/me.jpg"
	}
}EOF
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p main -complete -buildid iQqxv8YiKWDrDnzfPYmP/iQqxv8YiKWDrDnzfPYmP -goversion go1.18rc1 -c=2 -nolocalimports -importcfg $WORK/b001/importcfg -embedcfg $WORK/b001/embedcfg -pack /app.go
# command-line-arguments
/app.go:13:5: embed mages/me.jpg: open /mages/me.jpg: no such file or directory
root@82481f861c36:/# export WORK=/tmp/go-build2643592769
root@82481f861c36:/# /usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p main -complete -buildid iQqxv8YiKWDrDnzfPYmP/iQqxv8YiKWDrDnzfPYmP -goversion go1.18rc1 -c=2 -nolocalimports -importcfg $WORK/b001/importcfg -embedcfg $WORK/b001/embedcfg -pack /app.go
/app.go:13:5: embed mages/me.jpg: open /mages/me.jpg: no such file or directory
root@82481f861c36:/# ls images/
me.jpg
root@82481f861c36:/# ls mages
ls: cannot access 'mages': No such file or directory

Symlink fixes it.

root@82481f861c36:/# ln -s /images /mages
root@82481f861c36:/# /usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p main -complete -buildid iQqxv8YiKWDrDnzfPYmP/iQqxv8YiKWDrDnzfPYmP -goversion go1.18rc1 -c=2 -nolocalimports -importcfg $WORK/b001/importcfg -embedcfg $WORK/b001/embedcfg -pack /app.go
root@82481f861c36:/#

I have been digging around, this code seems to fail when the pkgdir is "/".

> cmd/go/internal/load.resolveEmbed() .usr/local/go/src/cmd/go/internal/load/pkg.go:2077 (hits goroutine(1):1 total:1) (PC: 0x831cec)
Warning: debugging optimized function
  2072:					}
  2073:				}
  2074:	
  2075:				switch {
  2076:				default:
=>2077:					return nil, nil, fmt.Errorf("cannot embed irregular file %s", rel)
  2078:	
  2079:				case info.Mode().IsRegular():
  2080:					if have[rel] != pid {
  2081:						have[rel] = pid
  2082:						list = append(list, rel)
(dlv) stack
0  0x0000000000831cec in cmd/go/internal/load.resolveEmbed
   at .usr/local/go/src/cmd/go/internal/load/pkg.go:2077
1  0x000000000082e454 in cmd/go/internal/load.(*Package).load
   at .usr/local/go/src/cmd/go/internal/load/pkg.go:1871
2  0x0000000000827b3d in cmd/go/internal/load.loadImport
   at .usr/local/go/src/cmd/go/internal/load/pkg.go:722
3  0x0000000000836cf2 in cmd/go/internal/load.PackagesAndErrors
   at .usr/local/go/src/cmd/go/internal/load/pkg.go:2706
4  0x00000000008c8433 in cmd/go/internal/list.runList
   at .usr/local/go/src/cmd/go/internal/list/list.go:507
5  0x00000000009035ae in main.invoke
   at .usr/local/go/src/cmd/go/main.go:218
6  0x0000000000902f8e in main.main
   at .usr/local/go/src/cmd/go/main.go:175
7  0x0000000000438212 in runtime.main
   at .usr/local/go/src/runtime/proc.go:250
8  0x0000000000465661 in runtime.goexit
   at .usr/local/go/src/runtime/asm_amd64.s:1571
(dlv) args
pkgdir = "/"
patterns = []string len: 2, cap: 0, [...]
files = []string len: 0, cap: 0, nil
pmap = map[string][]string []
(err) = error nil
(dlv) locals
pattern = "images/*"
have = (unreadable empty OP stack)
dirOK = (unreadable empty OP stack)
pid = 1
glob = "images/*"
all = false
~R0 = (unreadable empty OP stack)
match = (unreadable read out of bounds)
(err) = (unreadable could not find loclist entry at 0x366a1b for address 0x831cec)
list = []string len: 0, cap: 0, nil
file = "/images/mages"
rel = ""
what = "file"
info = io/fs.FileInfo(*os.fileStat) 0xbeef000000000608
err = (unreadable could not find loclist entry at 0x366cf0 for address 0x831cec)
(dlv)

I had thought I caused "go list all" to also fail, but after cleaning things up its seems to no longer fail in the same way.

@tenkoh
Copy link

tenkoh commented Mar 25, 2022

Hi, I have investigated about this issue.

My guess:

This issue would be caused by resolveEmbed function in src/cmd/go/internal/load/pkg.go.

Detail:

Reproduce and check the problem:

First of all, I had confirmed go list output changing working directory. I focused on EmbedPattern and EmbedFiles fields.

./
 |- Dockerfile
 |- main.go
 |- go.mod
 |- foo/
     |- foo.txt

main.go

package main

import (
	"embed"
	"fmt"
)

//go:embed foo
var foo embed.FS

func main() {
    // do something
}

Dockerfile

FROM golang:1.18-alpine
# git is required to run `go list`
RUN apk add git --no-cache

WORKDIR /
COPY ./ ./

CMD ["go", "list", "-f", "'pattern: {{.EmbedPatterns}}, files: {{.EmbedFiles}}'"]

Result:

EmbedPatterns are same in both cases. But EmbedFiles is different. So, some codes generating EmbedFiles would have a problem.

working directory EmbedPattern EmbedFiles
/ foo oo/foo.txt
/app foo foo/foo.txt

Read codes:

I found a suspicious part in resolveEmbed function in src/cmd/go/internal/load/pkg.go.

https://cs.opensource.google/go/go/+/refs/tags/go1.18:src/cmd/go/internal/load/pkg.go;l=2089;drc=c016133c50512e9a83e7442bd7ac614fe7ca62de

err := fsys.Walk(file, func(path string, info os.FileInfo, err error) error {
    if err != nil {
      return err
    }
    rel := filepath.ToSlash(path[len(pkgdir)+1:])

This function recursively walks around in the working directory, then picks up files. The files' paths are stored as rel shown in the code above.

Here, pkgdir varies by working directory. It is set in GoFilesPackage in src/cmd/go/internal/load/pkg.go

https://cs.opensource.google/go/go/+/refs/tags/go1.18:src/cmd/go/internal/load/pkg.go;l=2929;drc=c016133c50512e9a83e7442bd7ac614fe7ca62de

if dir == "" {
    dir = base.Cwd()
}
dir, err = filepath.Abs(dir)
if err != nil {
    base.Fatalf("%s", err)
}
bp, err := ctxt.ImportDir(dir, 0)

The dir is equal to pkgdir and it differs from the working directory. It is trivial.

working directory dir
/ /
/app /app

Considering all above, the rel (i.e. EmbedFile) becomes as below. Wow, we got oo/foo.txt !

pkgdir path (just an example) rel = filepath.ToSlash(path[len(pkgdir)+1:])
/ /foo/foo.txt oo/foo.txt
/app /app/foo/foo.txt foo/foo.txt

https://go.dev/play/p/ySwcRSEYVd9

Solution (guess):

So, solution could be adding an exception when pkgdir='/'.

If I have a time, I will challenge a bugfix. But it's the first time to try, so it will take a little long...

@tenkoh
Copy link

tenkoh commented Mar 30, 2022

I have fixed resolveEmbed and confirm the result.

https://github.com/tenkoh/fix-resolveEmbed-on-root

@gopherbot
Copy link

Change https://go.dev/cl/396694 mentions this issue: load: fix resolveEmbed to avoid a build error on a specific situation.

tyzbit added a commit to tyzbit/bitcoin-balance-notifier that referenced this issue May 1, 2022
@golang golang locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants