Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go.work: go build requires versioned replace directive when replacing a used module with a local path #54264

Open
mlaventure opened this issue Aug 4, 2022 · 13 comments
Labels
GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Thinking
Milestone

Comments

@mlaventure
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.18.5 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mlaventure/.cache/go-build"
GOENV="/home/mlaventure/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mlaventure/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mlaventure/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mlaventure/tmp/go/workspace/realdeal/go.mod"
GOWORK="/home/mlaventure/tmp/go/workspace/go.work"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2178506260=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Given the following workspace structure:

~/tmp/go/workspace $ tree
.
├── fubar
│   ├── fubar.go
│   └── go.mod
├── go.work
└── realdeal
    ├── go.mod
    ├── realdeal
    └── realdeal.go

2 directories, 6 files

With the following go.work:

~/tmp/go/workspace $ cat go.work 
go 1.18

use ./fubar
use ./realdeal

replace github.com/foo/fubar   => ./fubar

Build realdeal which depends on fubar:

~/tmp/go/workspace/realdeal $ go build .

What did you expect to see?

Nothing, the build should succeed

What did you see instead?

go: workspace module github.com/foo/fubar is replaced at all versions in the go.work file. To fix, remove the replacement from the go.work file or specify the version at which to replace the module.

Found workaround

The only way I've found to fix this is to update the go.work to be as follow:

~/tmp/go/workspace $ cat go.work 
go 1.18

use ./fubar
use ./realdeal

replace github.com/foo/fubar v0.0.0-00010101000000-000000000000 => ./fubar

This is not needed if a workspace is not used. It also seems to contradict the example from the documentation (https://go.dev/ref/mod#workspaces):

replace (
    golang.org/x/net v1.2.3 => example.com/fork/net v1.4.5
    golang.org/x/net => example.com/fork/net v1.4.5
    golang.org/x/net v1.2.3 => ./fork/net
    golang.org/x/net => ./fork/net
)
@dmitshur dmitshur added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. GoCommand cmd/go labels Aug 4, 2022
@dmitshur dmitshur added this to the Backlog milestone Aug 4, 2022
@dmitshur
Copy link
Contributor

dmitshur commented Aug 4, 2022

CC @bcmills, @matloob.

@BourgeoisBear
Copy link

BourgeoisBear commented Nov 16, 2022

workspace replace lines also cause problems with go mod tidy

go: downloading mydomain.com/module v0.0.0-00010101000000-000000000000
github.com/acct/testcmd imports
        mydomain.com/module: unrecognized import path "mydomain.com/module": https fetch: Get "https://mydomain.com/module?go-get=1": ...

@wangmir
Copy link

wangmir commented May 3, 2023

@dmitshur @bcmills Is there any progress in this? I also have similar problem, when I trying to migrate existing project to use go workspace. It seems replace in go.work is not available to me too. I cannot run go mod tidy

@intel352
Copy link

intel352 commented May 9, 2023

Same problem for me.
I presume this could be getting caused by the fact that I'm specifying the same local module in the use directive as in the replace directive, which I suspect is not intended behavior, but considering Go keeps trying to remotely fetch workspace modules during build rather than respecting the go.work, I was hoping the replace would solve the situation.

Frankly I'm surprised, most of the go tooling is generally high quality, but go.work seems quite error prone...

@bcmills
Copy link
Contributor

bcmills commented Jun 21, 2023

In general we would expect modules that are included in a workspace to not also require a replace directive in the go.work file. Can you explain more about the circumstance that led you to that arrangement?

@sudo-bmitch
Copy link

The reason I was looking to do this was to handle an environment where I cannot directly connect to the upstream git repo. I'm seeing go execute git ls-remote -q origin ... (which fails in my environment) for the modules referenced with use but not with replace.

I can remove the use and only have a replace, which works, but only for libraries where I'm not running a go build.

@theaog
Copy link

theaog commented Jul 11, 2023

don't know where you came up w/ that long versioning string, fyi: v0.0.0 also works to satisfy go work complaints.

use ./cmd/fubar
replace github.com/foo/fubar v0.0.0 => ./cmd/fubar

to note that you shouldn't need the replace directive when in a workspace managed by go.work. it should automatically resolve your remote module github.com/foo/fubar to the local version available at ./cmd/fubar.

@sudo-bmitch
Copy link

it should automatically resolve your remote module github.com/foo/fubar to the local version available at ./cmd/fubar.

It appears to do that by running git ls-remote -q origin ... which requires access to the remote git server.

@wangmir
Copy link

wangmir commented Aug 7, 2023

@bcmills Please check this issue. I think it's related.

#50750

go.work should support replace, when we think about monorepo situation. Placing replace on all of the modules of microservices in a monorepo is not ideal case i think.

@independentid
Copy link

I am having the same problems in 1.19. Once I put in go.work everything goes haywire.

  • Replace works in go.mod but can cause problems elsewhere in the workspace (Goland Issues suggestions to create a workspace)
  • go.work - using use block causes go work sync to issue errors about missing versions for modules in the use clause. Using the long version number, a new number, or using v.0.0.0 will still generate the same errors
  • Replace in go.work sort of solves some problems but the versioning issue remains if any module is an indirect requirement in another module.

Thinking about trying the latest go to see if the behavior changes.

@folays
Copy link

folays commented Dec 27, 2023

The reason I was looking to do this was to handle an environment where I cannot directly connect to the upstream git repo. I'm seeing go execute git ls-remote -q origin ... (which fails in my environment) for the modules referenced with use but not with replace.

As said above, since the beginning of go workspaces, and at least up to current Go 1.21.5 ;

Tree structure :

./module-a/go.mod
./module-a/module-a.go

./main-project/go.mod
./main-project/main.go
./main-project/go.work

CONTENT of module-A (no dependencies) :

$ cat module-a/go.mod
module gitlab.unpublished.private/modules/module-a.git

go 1.21.5
----8<----8<----8<----8<----8<----8<----8<----8<----8<
$ cat module-a/module-a.go
package module_a

import "fmt"

func ModuleA() {
	fmt.Println("ModuleA()")
}

CONTENT of mains-project (depends on moduleA) :

$ cat main-project/go.mod
module main-project

go 1.21.5

require (
	gitlab.unpublished.private/modules/module-a.git v0.9.9
)
----8<----8<----8<----8<----8<----8<----8<----8<----8<
$ cat main-project/main.go
package main

import "fmt"
import "gitlab.unpublished.private/modules/module-a.git"

func main() {
	fmt.Println("main project")
	module_a.ModuleA()
}
$ cat main-project/go.work
----8<----8<----8<----8<----8<----8<----8<----8<----8<
go 1.21.5

use (
	.
	../module-a
)

So, they are manually-crafted files, and the repos/gitlab doesn't event exists, go.mod.sum are missing, the v0.9.9 doesn't really exists, etc etc... but it reflects the SAME PROBLEM than really existing private git-cloned repos.

Inside ./main-project/, upon try to go build -v or go run -v, I encounter :

$ go build -v .
# cd /Users/folays/go/pkg/mod; git ls-remote https://gitlab.unpublished.private/modules/module-a
fatal: unable to access 'https://gitlab.unpublished.private/modules/module-a/': Could not resolve host: gitlab.unpublished.private
# cd /Users/folays/go/pkg/mod; git ls-remote git+ssh://gitlab.unpublished.private/modules/module-a
ssh: Could not resolve hostname gitlab.unpublished.private: nodename nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
# cd /Users/folays/go/pkg/mod; git ls-remote ssh://gitlab.unpublished.private/modules/module-a
ssh: Could not resolve hostname gitlab.unpublished.private: nodename nor servname provided, or not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

So the full text above is not really interesting. The MAIN POINT is that go it trying to git ls-remote unjoignable repos, which in real scenarios, I would myself git clone with my own means, and which I would only be able to git pull / push with my own means.

I would also like to specify, in aim to alleviate unneedy discussions, that for a real scenarios, my $HOME/.gitconfig would alias insteadOf = https://gitlab.unpublished.private/ to [url "ssh://git@gitlab.unpublished.private/"]

So, for the described scenario above, using only go.work's use ../PATH-to-immediate-dependency, Go WILL NOT build the main module, because it wants to git ls-remote the dep.

This WONT happen for even-dummy-er testbed scenarios where you would omit this from main-project/go.mod :

require (
	gitlab.unpublished.private/modules/module-a.git v0.9.9
)

Indeed, if you only use local-only never-git-pushed tagless/unversionned dependencies, the problem would not occur.

If for some reasons (which seems valid to me), your main-project has go.mod containing requiring versioned dependencies (because, from time to time, you want to publish a new version of main-project to your org), then only using use ../PATH directives from go.work without replace directives won't work at all.

----8<----8<----8<----8<----8<----8<----8<----8<

I severely miss a simple method to having this workflow working seamlessly :

  • fetch all org-private Go modules for which I may want to modify some of their code
    • some of them having dependencies on some others
  • fetch the org-private main-project which is use those dependencies
  • having anybody able to just go get https://org-private/url/to/the/main-project.git
  • having a local, private git-uncommited go.work which would just use . and use ../EACH/DEPENDENCIES and use ../EACH/SUBDEPENDENCIES

I mean, all the above works seamlessly, except for me, the dev, which needs to constantly jungle with some go.work, use, and replace directives.

This just doesn't work seamlessly, go.work use ../each-dependency does not suffice, I need to add replace everywhere, and when I finally want to commit everything I may have modified, I have to JUNGLE BACK to remove those replace directive for making the tools able to find and fill the go.mod files with coherent requires ... vVERSION before git push'ing everything.

----8<----8<----8<----8<----8<----8<----8<----8<

To me, it feels like the go.work stuff has been made with a majority of its usage being in spirit of using a "monorepo", but there is lot of juggling needing to be done to use it in a lot of cases.

Googling around on "go.work", "git ls-remote", "use", "ignored", "without replace", etc... yields a lot of results of confused people which seems to have a bad time figuring out a workflow which would suits to them.

I would just like a simple way to :

  • locally fetch (OUTSIDE of Go handling) every go repos I may be willing to modify locally (on which I have authority to git push),
  • have Go finds them the local directories to where I fetched them, with go.work
  • being able to LOCALLY git-commit new versioned versions of dependencies which I modify
  • being able to LOCALLY git-commit a new version of the main project, using the modified dependencies, WITHOUT FORCING ME beforehand to push the deps, WITHOUT TRYING to git ls-remote them at any time, since they are ALL POINTED BY go work's use directives
  • when I'm satisfied on all the work done locally, I myself git push everything to the remote (sub dependencies, dependencies, main project).

That's just hard in Go, having to constantly jungle with replaces directives.

I cannot seem to have a fixed-once-figured-out-do-not-constantly-modify-replaces set of go.mod/go.work files where I would just git clone everything I need, modify my .go files locally, then update the dependencies in go.mod, and git push everything.

No, Go is willing to git ls-remote constantly, OR requires me to have some replace directives in go.* files, which then would prevent go.mod to contains coherent requires DEPENDENCY vVERSION versions, which would make those go.mod (and the whole project/dependencies) unpublishable without first juggling with the go.* directives, to make the Go tool produce coherent go.mod vVERSION requires.

@mweibel
Copy link

mweibel commented Feb 14, 2024

I'd like to chime in too with the same issue and an additional potential bug (not sure if I should open a separate issue for that though). I felt it's most appropriate in this issue, but it's certainly also related to #50750 .

FTR we're using Go 1.21.

In our monorepo we gitignored go.work until recently. We made the switch because updating a dependent module was very cumbersome.
Imagine the following repo structure:

.
├── go.work                     # gitignored so just local
├── lib
│   ├── go.mod
│   └── lib.go
├── moduletwo               # depends on lib
│   ├── go.mod
│   └── moduletwo.go
├── README.md
└── test
    └── moduleone           # depends on lib
        ├── go.mod
        └── moduleone.go

A local Go workspace enabled us to efficiently make updates to multiple modules in one change and test them without having to update versions. Push to CI/CD however, requires an update to the versions (e.g. if module lib has an update the other modules depending on it need an updated version). This requires a separate commit just to update the revision in go.mod and making sure we don't need to rebase after that, since otherwise commit hashes change.

To avoid this, we decided to add go.work (not go.work.sum) to the repo. This seems to work only in some cases. In one module we encountered issues like these:

# linux CI runner
cmd/cmd.go:9:2: path/to/module/wrappers@v0.0.0-20240202141310-9ed56a5b95df: invalid version: unknown revision 9ed56a5b95df

# windows CI runner
path\cli\root.go:26:2: path/to/lib@v0.0.0-20231207112259-11b3c6dc08c9: invalid version: git ls-remote -q origin in C:\Windows\system32\config\systemprofile\go\pkg\mod\cache\vcs\927cbc5a0bf57c000feeddfdd18d1586382b2a626237337481dc6f6b33425a3a: exit status 128:
	Unhandled Exception: System.ComponentModel.Win32Exception: The directory name is invalid

Removing go.work for this change in CI fixed the issue.
To get around this I tried to use replace directives just as outlined in this issue. A commited go.work plus each go.mod containing replace directives to replace the modules in the same folder structure with one above.

Repository with demo: https://github.com/mweibel/gomodreplace/tree/fail (please note, this is only a partial reproduction because we have an additional issue in our monorepo due to the fact that we are on an on-premise GitLab instance without a go module proxy in front).

The issue with this approach is visible when running go vet ./...:

$ go vet ./...
conflicting replacements found for github.com/mweibel/gomodreplace/lib@v0.0.0 in workspace modules defined by /tmp/gomodreplace/moduletwo/go.mod and /tmp/gomodreplace/test/moduleone/go.mod

Because moduletwo replaces lib as ../lib and moduleone replaces lib as ../../lib, go vet errors.
"Fixed" monorepo structure in main branch: https://github.com/mweibel/gomodreplace
This works because both moduleone and moduletwo replace lib in the same way.

I certainly might be doing something wrong and would love to get some help/documentation if that's the case.

Thank you!

@bcmills
Copy link
Contributor

bcmills commented Feb 28, 2024

There are a couple of solutions we could consider here:

  • We could allow the replace directive as long as it names the same directory as what is in the use directive, so that the workspace module is “replaced with itself” but all prior versions are also replaced with what is in the workspace (so replace github.com/foo/fubar => ./fubar is allowed but must be a no-op).
  • We could define that wildcard replace directives don't apply to the version of the module that is injected via the workspace (so replace github.com/foo/fubar => ./fubar is allowed but does not apply to the workspace module).

(CC @matloob @samthanawalla)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Thinking
Projects
None yet
Development

No branches or pull requests