Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: confusing go mod experience #49416

Closed
go101 opened this issue Nov 6, 2021 · 16 comments
Closed

cmd/go: confusing go mod experience #49416

go101 opened this issue Nov 6, 2021 · 16 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Unfortunate WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@go101
Copy link

go101 commented Nov 6, 2021 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.17.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

myname@hp:/tmp/gomod$ pwd
/tmp/gomod
myname@hp:/tmp/gomod$ ls
go.mod  main.go
myname@hp:/tmp/gomod$ cat main.go
package main

import (
	"os"

	"github.com/shurcooL/github_flavored_markdown"
)

const md = `hello`

func main() {
	text := []byte(md)

	os.WriteFile(
		"github_flavored_markdown.html",
		github_flavored_markdown.Markdown(text),
		0600)
}
myname@hp:/tmp/gomod$ cat go.mod
module a.b/app

go 1.17

myname@hp:/tmp/gomod$ go mod tidy
go: finding module for package github.com/shurcooL/github_flavored_markdown
go: found github.com/shurcooL/github_flavored_markdown in github.com/shurcooL/github_flavored_markdown v0.0.0-20210228213109-c3a9aa474629
go: finding module for package github.com/microcosm-cc/bluemonday
go: finding module for package github.com/shurcooL/octicon
go: finding module for package github.com/shurcooL/highlight_go
go: finding module for package github.com/shurcooL/highlight_diff
go: finding module for package github.com/shurcooL/sanitized_anchor_name
go: finding module for package github.com/sourcegraph/annotate
go: finding module for package github.com/sourcegraph/syntaxhighlight
go: finding module for package golang.org/x/net/html
go: finding module for package golang.org/x/net/html/atom
go: found github.com/microcosm-cc/bluemonday in github.com/microcosm-cc/bluemonday v1.0.16
go: found github.com/shurcooL/highlight_diff in github.com/shurcooL/highlight_diff v0.0.0-20181222201841-111da2e7d480
go: found github.com/shurcooL/highlight_go in github.com/shurcooL/highlight_go v0.0.0-20191220051317-782971ddf21b
go: found github.com/shurcooL/octicon in github.com/shurcooL/octicon v0.0.0-20191102190552-cbb32d6a785c
go: found github.com/shurcooL/sanitized_anchor_name in github.com/shurcooL/sanitized_anchor_name v1.0.0
go: found github.com/sourcegraph/annotate in github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
go: found github.com/sourcegraph/syntaxhighlight in github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: found golang.org/x/net/html in golang.org/x/net v0.0.0-20211105192438-b53810dc28af
go: found golang.org/x/net/html/atom in golang.org/x/net v0.0.0-20211105192438-b53810dc28af
go: finding module for package github.com/kr/pretty
go: finding module for package github.com/sergi/go-diff/diffmatchpatch
go: finding module for package github.com/shurcooL/go-goon
go: finding module for package github.com/shurcooL/go/reflectsource
go: found github.com/sergi/go-diff/diffmatchpatch in github.com/sergi/go-diff v1.2.0
go: found github.com/shurcooL/go-goon in github.com/shurcooL/go-goon v0.0.0-20210110234559-7585751d9a17
go: found github.com/shurcooL/go/reflectsource in github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636
go: found github.com/kr/pretty in github.com/kr/pretty v0.3.0
myname@hp:/tmp/gomod$ go build
myname@hp:/tmp/gomod$ vi main.go
myname@hp:/tmp/gomod$ cat main.go
package main

import (
	"os"

	commonmark_markdown "gitlab.com/golang-commonmark/markdown"
)

const md = `hello`

func main() {
	text := []byte(md)

	os.WriteFile(
		"commonmark_markdown.html",
		[]byte(commonmark_markdown.New(commonmark_markdown.XHTMLOutput(true)).RenderToString(text)),
		0600)
}
myname@hp:/tmp/gomod$ go mod tidy
go: finding module for package gitlab.com/golang-commonmark/markdown
go: found gitlab.com/golang-commonmark/markdown in gitlab.com/golang-commonmark/markdown v0.0.0-20191127184510-91b5b3c99c19
myname@hp:/tmp/gomod$ go build
myname@hp:/tmp/gomod$ vi main.go
myname@hp:/tmp/gomod$ cat main.go
package main

import (
	"os"

	"github.com/shurcooL/github_flavored_markdown"
)

const md = `hello`

func main() {
	text := []byte(md)

	os.WriteFile(
		"github_flavored_markdown.html",
		github_flavored_markdown.Markdown(text),
		0600)
}
myname@hp:/tmp/gomod$ go mod tidy
go: finding module for package github.com/shurcooL/github_flavored_markdown
go: found github.com/shurcooL/github_flavored_markdown in github.com/shurcooL/github_flavored_markdown v0.0.0-20210228213109-c3a9aa474629
go: finding module for package github.com/shurcooL/octicon
go: finding module for package github.com/microcosm-cc/bluemonday
go: finding module for package github.com/shurcooL/highlight_go
go: finding module for package github.com/shurcooL/highlight_diff
go: finding module for package github.com/sourcegraph/annotate
go: finding module for package github.com/sourcegraph/syntaxhighlight
go: finding module for package golang.org/x/net/html
go: finding module for package golang.org/x/net/html/atom
go: found github.com/microcosm-cc/bluemonday in github.com/microcosm-cc/bluemonday v1.0.16
go: found github.com/shurcooL/highlight_diff in github.com/shurcooL/highlight_diff v0.0.0-20181222201841-111da2e7d480
go: found github.com/shurcooL/highlight_go in github.com/shurcooL/highlight_go v0.0.0-20191220051317-782971ddf21b
go: found github.com/shurcooL/octicon in github.com/shurcooL/octicon v0.0.0-20191102190552-cbb32d6a785c
go: found github.com/sourcegraph/annotate in github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
go: found github.com/sourcegraph/syntaxhighlight in github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: found golang.org/x/net/html in golang.org/x/net v0.0.0-20211105192438-b53810dc28af
go: found golang.org/x/net/html/atom in golang.org/x/net v0.0.0-20211105192438-b53810dc28af
go: finding module for package github.com/kr/pretty
go: finding module for package github.com/shurcooL/go-goon
go: finding module for package github.com/sergi/go-diff/diffmatchpatch
go: finding module for package github.com/shurcooL/go/reflectsource
go: found github.com/sergi/go-diff/diffmatchpatch in github.com/sergi/go-diff v1.2.0
go: found github.com/shurcooL/go-goon in github.com/shurcooL/go-goon v0.0.0-20210110234559-7585751d9a17
go: found github.com/shurcooL/go/reflectsource in github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636
go: found github.com/kr/pretty in github.com/kr/pretty v0.3.0
myname@hp:/tmp/gomod$ go build
# github.com/shurcooL/github_flavored_markdown
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:36:30: undefined: blackfriday.HtmlRenderer
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:37:37: too many arguments to conversion to blackfriday.Markdown: blackfriday.Markdown(text, renderer, extensions)
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:71:20: undefined: blackfriday.EXTENSION_NO_INTRA_EMPHASIS
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:72:2: undefined: blackfriday.EXTENSION_TABLES
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:73:2: undefined: blackfriday.EXTENSION_FENCED_CODE
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:74:2: undefined: blackfriday.EXTENSION_AUTOLINK
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:75:2: undefined: blackfriday.EXTENSION_STRIKETHROUGH
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:76:2: undefined: blackfriday.EXTENSION_SPACE_HEADERS
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:77:2: undefined: blackfriday.EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:93:2: undefined: blackfriday.Html
/home/myname/go/pkg/mod/github.com/shurcoo!l/github_flavored_markdown@v0.0.0-20210228213109-c3a9aa474629/main.go:37:37: too many errors
myname@hp:/tmp/gomod$ 
myname@hp:/tmp/gomod$ ls
app  go.mod  go.sum  main.go
myname@hp:/tmp/gomod$ vi go.mod
myname@hp:/tmp/gomod$ cat go.mod
module a.b/app

go 1.17

myname@hp:/tmp/gomod$ go mod tidy
go: finding module for package github.com/shurcooL/github_flavored_markdown
go: found github.com/shurcooL/github_flavored_markdown in github.com/shurcooL/github_flavored_markdown v0.0.0-20210228213109-c3a9aa474629
go: finding module for package github.com/microcosm-cc/bluemonday
go: finding module for package github.com/shurcooL/octicon
go: finding module for package github.com/shurcooL/highlight_diff
go: finding module for package github.com/shurcooL/highlight_go
go: finding module for package github.com/shurcooL/sanitized_anchor_name
go: finding module for package github.com/sourcegraph/annotate
go: finding module for package github.com/sourcegraph/syntaxhighlight
go: finding module for package golang.org/x/net/html
go: finding module for package golang.org/x/net/html/atom
go: found github.com/microcosm-cc/bluemonday in github.com/microcosm-cc/bluemonday v1.0.16
go: found github.com/shurcooL/highlight_diff in github.com/shurcooL/highlight_diff v0.0.0-20181222201841-111da2e7d480
go: found github.com/shurcooL/highlight_go in github.com/shurcooL/highlight_go v0.0.0-20191220051317-782971ddf21b
go: found github.com/shurcooL/octicon in github.com/shurcooL/octicon v0.0.0-20191102190552-cbb32d6a785c
go: found github.com/shurcooL/sanitized_anchor_name in github.com/shurcooL/sanitized_anchor_name v1.0.0
go: found github.com/sourcegraph/annotate in github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
go: found github.com/sourcegraph/syntaxhighlight in github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: found golang.org/x/net/html in golang.org/x/net v0.0.0-20211105192438-b53810dc28af
go: found golang.org/x/net/html/atom in golang.org/x/net v0.0.0-20211105192438-b53810dc28af
go: finding module for package github.com/kr/pretty
go: finding module for package github.com/shurcooL/go/reflectsource
go: finding module for package github.com/sergi/go-diff/diffmatchpatch
go: finding module for package github.com/shurcooL/go-goon
go: found github.com/sergi/go-diff/diffmatchpatch in github.com/sergi/go-diff v1.2.0
go: found github.com/shurcooL/go-goon in github.com/shurcooL/go-goon v0.0.0-20210110234559-7585751d9a17
go: found github.com/shurcooL/go/reflectsource in github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636
go: found github.com/kr/pretty in github.com/kr/pretty v0.3.0
myname@hp:/tmp/gomod$ go build
myname@hp:/tmp/gomod$

What did you expect to see?

There are 4 go build runs, the 3rd one is expected to succeed.

What did you see instead?

It fails with many errors.
@go101 go101 changed the title cmd/go: confused go mod experience cmd/go: confusing go mod experience Nov 6, 2021
@go101
Copy link
Author

go101 commented Nov 6, 2021

This might be caused by the fact that the two markdown libraries use
two different version of github.com/russross/blackfriday.
One uses v1.5.2, the other uses v2.0.0+incompatible.

But any way, all the builds should succeed.

@seankhliao
Copy link
Member

output of go env?

@go101
Copy link
Author

go101 commented Nov 6, 2021

Generally, I don't post the go env output, for the output is often not very helpful, and it contains some privacy info.

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myname/.cache/go-build"
GOENV="/home/myname/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/myname/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myname/go"
GOPRIVATE=""
GOPROXY="proxy.golang.org"
GOROOT="/home/myname/software/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myname/software/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.3"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/gomod/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build6996316699=/tmp/go-build -gno-record-gcc-switches"

@go101
Copy link
Author

go101 commented Nov 7, 2021 •
edited

In short. the go mod tidy command does nothing for a project with the following two files.

  • go.mod
module a.b/app

go 1.17

require github.com/shurcooL/github_flavored_markdown v0.0.0-20210228213109-c3a9aa474629

require (
	github.com/aymerick/douceur v0.2.0 // indirect
	github.com/gorilla/css v1.0.0 // indirect
	github.com/kr/pretty v0.3.0 // indirect
	github.com/microcosm-cc/bluemonday v1.0.16 // indirect
	github.com/russross/blackfriday v2.0.0+incompatible // indirect
	github.com/sergi/go-diff v1.2.0 // indirect
	github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636 // indirect
	github.com/shurcooL/go-goon v0.0.0-20210110234559-7585751d9a17 // indirect
	github.com/shurcooL/highlight_diff v0.0.0-20181222201841-111da2e7d480 // indirect
	github.com/shurcooL/highlight_go v0.0.0-20191220051317-782971ddf21b // indirect
	github.com/shurcooL/octicon v0.0.0-20191102190552-cbb32d6a785c // indirect
	github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
	github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d // indirect
	github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e // indirect
	golang.org/x/net v0.0.0-20211105192438-b53810dc28af // indirect
    )
  • main.go
package main

import _ "github.com/shurcooL/github_flavored_markdown"

func main() {}

@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 8, 2021
@cagedmantis cagedmantis added this to the Backlog milestone Nov 8, 2021
@cagedmantis
Copy link
Contributor

cc @bcmills @matloob

@bcmills
Copy link
Contributor

bcmills commented Nov 9, 2021

At the first go mod tidy, you resolve github.com/shurcooL/github_flavored_markdown to its latest version, and upgrade (but do not downgrade) other dependencies to the versions it requires. That adds the specific version of github.com/russross/blackfriday required by that dependency (v1.5.2).

$ go list -m github.com/shurcooL/github_flavored_markdown github.com/russross/blackfriday
github.com/shurcooL/github_flavored_markdown v0.0.0-20210228213109-c3a9aa474629
github.com/russross/blackfriday v1.5.2

At the second go mod tidy, you resolve gitlab.com/golang-commonmark/markdown to its latest version, and upgrade (but do not downgrade) other dependencies to the versions it requires. That upgrades github.com/russross/blackfriday to v2.0.0+incompatible, and golang.org/x/text to

$ go list -m gitlab.com/golang-commonmark/markdown github.com/russross/blackfriday
gitlab.com/golang-commonmark/markdown v0.0.0-20191127184510-91b5b3c99c19
github.com/russross/blackfriday v2.0.0+incompatible

At the third go mod tidy, you resolve github.com/shurcooL/github_flavored_markdown to its latest version, and upgrade (but do not downgrade) other dependencies to the versions it requires.

$ go list -m github.com/shurcooL/github_flavored_markdown github.com/russross/blackfriday
github.com/shurcooL/github_flavored_markdown v0.0.0-20210228213109-c3a9aa474629
github.com/russross/blackfriday v2.0.0+incompatible

The operation of go mod tidy is the same in every case: it adds dependencies needed to resolve imports, then upgrades other dependencies so that they satisfy the requirements of those dependencies.

@bcmills
Copy link
Contributor

bcmills commented Nov 9, 2021

FWIW, I agree that the behavior here is a bit confusing but I don't see a better alternative.

(It would be arguably even more confusing if go mod tidy — which is intended to add missing dependencies and remove irrelevant ones — also downgraded dependencies that are still relevant!)

@bcmills
Copy link
Contributor

bcmills commented Nov 9, 2021

@go101, can you give a bit more detail about why you expected the third go build to succeed? I'm wondering if there is something we can do to at least clarify the documentation for go mod tidy.

(I've also sent https://gitlab.com/golang-commonmark/markdown/-/merge_requests/3 upstream to avoid the incompatible upgrade in this case to begin with.)

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 9, 2021
@go101
Copy link
Author

go101 commented Nov 9, 2021

go mod tidy is the only command I know of to resolve dependencies.
Should I use another command here?
Or editing go.mod manually is the only way?

@bcmills
Copy link
Contributor

bcmills commented Nov 9, 2021

go get is the command to use to downgrade existing dependencies.

In this case, go get -d github.com/russross/blackfriday@v1 would downgrade to the most recent v1.*.* release for that module, which allows go build to succeed.

@go101
Copy link
Author

go101 commented Nov 9, 2021 •
edited

But I don't know what is the exact version of the indirect dependency module.
I even don't know it is a v1 version.

@bcmills
Copy link
Contributor

bcmills commented Nov 9, 2021

See https://golang.org/doc/modules/version-numbers.

You don't have to request the exact version; go get supports prefixes. (In the above example, @v1 means “at the latest minor release for major version 1”.)

@go101
Copy link
Author

go101 commented Nov 9, 2021

Assume I know it is a v1 version and I know the problem is caused by the github.com/russross/blackfriday indirect module (in this case, this might be clear, but for some cases, this might not). The following output is got:

$ go get -d github.com/russross/blackfriday@v1
go get: downgraded github.com/russross/blackfriday v2.0.0+incompatible => v1.6.0

The go get command downgrade the version of the indirect dependency to v1.6.0, but the version required by the direct dependency (github.com/shurcooL/github_flavored_markdown) is 1.5.2. This might cause some subtle behavior differences.

@bcmills
Copy link
Contributor

bcmills commented Nov 9, 2021

Yes, that's true. We don't in general have a way to downgrade a dependency to the minimum of the versions already required by other dependencies, and I haven't seen a strong use-case for it. (Typically if you had a working build with an earlier set of dependencies you can use git checkout HEAD go.mod to go back to exactly those dependencies.)

It would also be fairly straightforward to build a third-party tool to perform that downgrade using go mod graph and go get.

@go101
Copy link
Author

go101 commented Nov 9, 2021

I find vacuuming the require lines in the go.mod file is the simplest way. ;D

@bcmills
Copy link
Contributor

bcmills commented Nov 9, 2021

I don't think there's anything specific we can do here, but I'll bear this report in mind when we're looking for patterns of workflows and/or documentation to improve.

@bcmills bcmills closed this as completed Nov 9, 2021
@golang golang locked and limited conversation to collaborators Nov 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Unfortunate WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
5 participants
@cagedmantis @bcmills @gopherbot @seankhliao @go101