cmd/go: legacy dependency conversion bypasses GOPROXY #33767

aofei · 2019-08-21T21:29:37Z

Well, one of my friends (@eddycjy) tried to migrate a project from their company from golang/dep to Go modules. When he executes go mod init example.com/foo/bar, go parses the Gopkg.lock file, which causes the initialization to fail because the endpoints of some of the modules in Gopkg.lock are blocked by their company's firewall. We thought the firewall problem could be solved by setting GOPROXY until I found modfetch.ImportRepoRev:

go/src/cmd/go/internal/modfetch/repo.go

Lines 282 to 327 in 2b8b34a

    
           // ImportRepoRev returns the module and version to use to access 
        
           // the given import path loaded from the source code repository that 
        
           // the original "go get" would have used, at the specific repository revision 
        
           // (typically a commit hash, but possibly also a source control tag). 
        
           func ImportRepoRev(path, rev string) (Repo, *RevInfo, error) { 
        
           	if cfg.BuildMod == "vendor" || cfg.BuildMod == "readonly" { 
        
           		return nil, nil, fmt.Errorf("repo version lookup disabled by -mod=%s", cfg.BuildMod) 
        
           	} 
        
           	// Note: Because we are converting a code reference from a legacy 
        
           	// version control system, we ignore meta tags about modules 
        
           	// and use only direct source control entries (get.IgnoreMod). 
        
           	security := web.SecureOnly 
        
           	if get.Insecure { 
        
           		security = web.Insecure 
        
           	} 
        
           	rr, err := get.RepoRootForImportPath(path, get.IgnoreMod, security) 
        
           	if err != nil { 
        
           		return nil, nil, err 
        
           	} 
        
           	code, err := lookupCodeRepo(rr) 
        
           	if err != nil { 
        
           		return nil, nil, err 
        
           	} 
        
           	revInfo, err := code.Stat(rev) 
        
           	if err != nil { 
        
           		return nil, nil, err 
        
           	} 
        
           	// TODO: Look in repo to find path, check for go.mod files. 
        
           	// For now we're just assuming rr.Root is the module path, 
        
           	// which is true in the absence of go.mod files. 
        
           	repo, err := newCodeRepo(code, rr.Root, rr.Root) 
        
           	if err != nil { 
        
           		return nil, nil, err 
        
           	} 
        
           	info, err := repo.(*codeRepo).convert(revInfo, rev) 
        
           	if err != nil { 
        
           		return nil, nil, err 
        
           	} 
        
           	return repo, info, nil 
        
           }

I also found that modfetch.ImportRepoRev first appeared in the golang/vgo@97ff4ad submitted by @rsc. And the reason is:

Add ImportRepoRev, because neither Lookup nor Import
is appropriate for translating legacy versioning system configs.
There's more to do in ImportRepoRev, but it's already more
accurate than before.

I investigated it as much as I can. Then I think the "inappropriate" that @rsc said before may have gone now, since modload.Query is now very powerful. So I added support for the module proxy for modfetch.ImportRepoRev by simply doing something like modload.Query.

I just submitted a patch. If the patch can be merged, then I suggest applying those changes to Go 1.13 since it has not yet been released. 😊

The text was updated successfully, but these errors were encountered:

gopherbot · 2019-08-21T21:38:25Z

Change https://golang.org/cl/191218 mentions this issue: cmd/go: add module proxy support for modfetch.ImportRepoRev``

bcmills · 2019-08-22T03:26:35Z

I suggest applying those changes to Go 1.13 since it has not yet been released. 😊

Go 1.13 is well into the release freeze at this point, so we're pretty much only fixing critical regressions from here on out. The good news is, the tree will be opening for 1.14 very soon.

aofei · 2019-08-22T03:29:55Z

And this problem isn't just for Gopkg.lock. It exists when translating legacy versioning system configs. So the title should be corrected.

aofei · 2019-09-09T12:23:06Z

Hi @bcmills, someone has reported this bug to me (again) today. This bug isn't only triggered when go mod init is called, but in every operation (such as go test, go run, go build etc) that might generate a go.mod file from a legacy dependency (when GO111MODULE=on). The people who have encountered this bug are blaming their Go module proxies (at least goproxy.cn was blamed by someone). They think that Go module proxy should be able to proxy this kind of initialization operation.

So, please merge https://golang.org/cl/191218 and consider applying those changes to Go 1.13.1 (or 1.13.2). We really need it. 🙏

Thanks again. ❤️

bcmills · 2019-09-11T19:05:25Z

CL 191218 is in, although I don't think it meets the backport criteria (it wasn't a regression and the workaround is to run go mod tidy after go mod init).

Regarding other operations that might generate a go.mod file: they no longer do as of 1.13. (Only go mod init creates the go.mod file now.) That was #29433.

aofei · 2019-09-11T19:10:40Z

Alright, I got it. Thank you for your reply. :-)

bcmills · 2019-09-16T13:40:35Z

This was reverted in CL 194797 due to a test failure, possibly related to #34092. Investigating now.

bcmills · 2019-09-16T14:39:11Z

The reason for the test failure seems to be that some of the converted formats specify package paths rather than module paths, and modfetch.Stat does not know how to resolve a package path to a module.

The usual function that converts a package path to a module path is modload.QueryPackage, but we can't call that within modconv due to a circular dependency.

The circular dependency exists because modload is currently performing (at least) two different roles:

Managing the contents of the main module's build list, and
managing the mapping of module paths and versions to source file locations.

Role (2) will need significant refactoring for #26904 anyway, so I'm planning to revisit this once that has been addressed.

CC @jayconrod

bcmills · 2019-10-24T17:48:24Z

Role (2) will need significant refactoring for #26904 anyway, so I'm planning to revisit this once that has been addressed.

I don't think I'll have time to get to that in 1.14 after all. 😞

scbizu · 2021-04-02T03:30:56Z

This will still make sense for people under firewall(AKA MOSTLY Chinese Gopher) , can we put it in Go1.17 or other recent releases , thanks.

Create the empty go.mod file and run go mod tidy seems hack and inconvenient for developers 🙏

aofei · 2021-04-02T06:43:03Z

@scbizu Sorry, I almost forgot this one, thanks for the ping.

I just checked the code, and modload still imports modconv, so there is no way to directly call anything in modload in modconv (import cycle not allowed). So before #26904 is resolved, we only have one way to fix this bug, which is to pass modload.QueryPackages as a parameter to modconv.ConvertLegacyConfig.

I'll submit a CL later.

gopherbot · 2021-04-02T07:11:16Z

Change https://golang.org/cl/306809 mentions this issue: cmd/go/internal/modconv: involve GOPROXY in ConvertLegacyConfig

scbizu · 2021-04-03T07:15:38Z

I find out that maybe the way create go.mod first and run go mod tidy do not work in some cases that go mod tidy do not recognize the old vendored version and use the latest version . In these cases , we must use go mod init to generate the correct version go.mod .

bcmills changed the title ~~cmd/go: add module proxy support for modfetch.ImportRepoRev~~ cmd/go: Gopkg.lock conversion bypasses GOPROXY Aug 22, 2019

bcmills added modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 22, 2019

bcmills added this to the Go1.14 milestone Aug 22, 2019

aofei changed the title ~~cmd/go: Gopkg.lock conversion bypasses GOPROXY~~ cmd/go: legacy dependency conversion bypasses GOPROXY Aug 22, 2019

gopherbot closed this as completed in 04867cd Sep 11, 2019

bcmills reopened this Sep 16, 2019

bcmills mentioned this issue Sep 18, 2019

cmd/go: in 'go mod init', prefer the 'version' field from Gopkg.lock instead of a pseudo-version #31251

Closed

aofei mentioned this issue Sep 28, 2019

我设置了GOPROXY还是无法下载 goproxy/goproxy.cn#29

Closed

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

bcmills modified the milestones: Backlog, Go1.15 Oct 24, 2019

bcmills mentioned this issue Dec 4, 2019

cmd/go: go mod init doesn't recognize GOPROXY #35931

Closed

bcmills modified the milestones: Go1.15, Backlog Mar 2, 2020

lujiaxing mentioned this issue Mar 16, 2020

aarch64 release? gogs/gogs#5990

Closed

gopherbot closed this as completed in 699a7c0 Apr 15, 2021

golang locked and limited conversation to collaborators Apr 15, 2022

gopherbot added the FrozenDueToAge label Apr 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: legacy dependency conversion bypasses GOPROXY #33767

cmd/go: legacy dependency conversion bypasses GOPROXY #33767

aofei commented Aug 21, 2019

gopherbot commented Aug 21, 2019

bcmills commented Aug 22, 2019

aofei commented Aug 22, 2019

aofei commented Sep 9, 2019

bcmills commented Sep 11, 2019 •

edited

aofei commented Sep 11, 2019

bcmills commented Sep 16, 2019

bcmills commented Sep 16, 2019 •

edited

bcmills commented Oct 24, 2019

scbizu commented Apr 2, 2021

aofei commented Apr 2, 2021

gopherbot commented Apr 2, 2021

scbizu commented Apr 3, 2021

Navigation Menu

cmd/go: legacy dependency conversion bypasses GOPROXY #33767

cmd/go: legacy dependency conversion bypasses GOPROXY #33767

Comments

aofei commented Aug 21, 2019

gopherbot commented Aug 21, 2019

bcmills commented Aug 22, 2019

aofei commented Aug 22, 2019

aofei commented Sep 9, 2019

bcmills commented Sep 11, 2019 • edited

aofei commented Sep 11, 2019

bcmills commented Sep 16, 2019

bcmills commented Sep 16, 2019 • edited

bcmills commented Oct 24, 2019

scbizu commented Apr 2, 2021

aofei commented Apr 2, 2021

gopherbot commented Apr 2, 2021

scbizu commented Apr 3, 2021

bcmills commented Sep 11, 2019 •

edited

bcmills commented Sep 16, 2019 •

edited