vendor: no docs on how to update a module in std #31806

bradfitz · 2019-05-02T17:23:14Z

I've been flailing around all morning fighting the new system of updating modules in std.

I want to update x/net in std because the golang.org/x/net/http/httpproxy we have in std is old and doesn't support ALL_PROXY.

Note that all_proxy is present in golang.org/x/net:

bradfitz@go:~/src/golang.org/x/net$ git show-ref HEAD
9ce7a6920f093fc0b908c4a5f66ae049110f417e refs/remotes/origin/HEAD
bradfitz@go:~/src/golang.org/x/net$ git grep -i all_proxy
proxy/proxy.go:         names: []string{"ALL_PROXY", "all_proxy"},
proxy/proxy_test.go:            fmt.Fprintf(&buf, "all_proxy=%q", t.allProxyEnv)
proxy/proxy_test.go:            os.Setenv("ALL_PROXY", tt.allProxyEnv)

But it's not present in std:

bradfitz@go:~$ cd $GOROOT/src/
bradfitz@go:~/go/src$ git grep -i all_proxy

So, let's update it...

bradfitz@go:~/go/src$ go get -u golang.org/x/net/...@latest
go: finding golang.org/x/net latest
go: finding golang.org/x/sys latest
go: finding golang.org/x/crypto latest
bradfitz@go:~/go/src$ git diff go.mod                                                                                                                                                                       diff --git a/src/go.mod b/src/go.mod
index a527f9a244..047e43f864 100644
--- a/src/go.mod
+++ b/src/go.mod
@@ -3,9 +3,9 @@ module std
 go 1.12
 
 require (
-       golang.org/x/crypto v0.0.0-20190424203555-c05e17bb3b2d
-       golang.org/x/net v0.0.0-20190424112056-4829fb13d2c6
-       golang.org/x/sys v0.0.0-20190425145619-16072639606e // indirect
+       golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734
+       golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09
+       golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd // indirect
        golang.org/x/text v0.3.2 // indirect
-       golang.org/x/tools v0.0.0-20190425214124-2d660fb8a000 // indirect
+       golang.org/x/tools v0.0.0-20190501045030-23463209683d // indirect
 )

And go mod vendor perhaps?

bradfitz@go:~/go/src$ go mod vendor
bradfitz@go:~/go/src$ git grep -i all_proxy

Nope. Still not present.

What changed?

bradfitz@go:~/go/src$ git diff --stat
 src/go.mod                                                |   8 ++---
 src/go.sum                                                |   7 ++++
 src/vendor/golang.org/x/sys/unix/mkall.sh                 |  14 +++++---
 src/vendor/golang.org/x/sys/unix/mksysctl_openbsd.pl      | 265 ----------------------------------------------------------------------------------------------------------------------------------------------------
 src/vendor/golang.org/x/sys/unix/sockcmsg_unix.go         |   8 ++---
 src/vendor/golang.org/x/sys/unix/syscall.go               |   1 -
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_386.go   |   2 ++
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_amd64.go |   2 +-
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_arm.go   |   4 ++-
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_arm64.go |   2 +-
 src/vendor/modules.txt                                    |  14 ++++----
 11 files changed, 39 insertions(+), 288 deletions(-)

... other stuff changed, but not golang.org/x/net/...

I tried to get get a specific version rather than "latest"

bradfitz@go:~/go/src$ go get -u golang.org/x/net/http/httpproxy@v0.0.0-20190501004415-9ce7a6920f09
go: finding golang.org/x/net/http/httpproxy v0.0.0-20190501004415-9ce7a6920f09
go: finding golang.org/x/net/http v0.0.0-20190501004415-9ce7a6920f09
bradfitz@go:~/go/src$ go mod vendor
bradfitz@go:~/go/src$ git grep -i all_proxy
bradfitz@go:~/go/src$ git di --stat
 src/go.mod                                                |   8 ++---
 src/go.sum                                                |   7 ++++
 src/vendor/golang.org/x/sys/unix/mkall.sh                 |  14 +++++---
 src/vendor/golang.org/x/sys/unix/mksysctl_openbsd.pl      | 265 ----------------------------------------------------------------------------------------------------------------------------------------------------
 src/vendor/golang.org/x/sys/unix/sockcmsg_unix.go         |   8 ++---
 src/vendor/golang.org/x/sys/unix/syscall.go               |   1 -
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_386.go   |   2 ++
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_amd64.go |   2 +-
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_arm.go   |   4 ++-
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_arm64.go |   2 +-
 src/vendor/modules.txt                                    |  14 ++++----
 11 files changed, 39 insertions(+), 288 deletions(-)

Still no good.

A README.txt would really help.

Also, what is vendor/modules.txt? What role does it play and is it hand maintained or tool maintained and where is it documented? Does it support comments? Could it have some comments saying what it is?

Please help. (and document)

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2019-05-02T17:52:34Z

This is the thread that discussed this: https://groups.google.com/d/msg/golang-dev/xcVJDj5GJ84/6XF1X8yhBwAJ .

(Austin asked for docs there, but I don't know if any were written.)

CC @bcmills

bradfitz · 2019-05-02T18:15:43Z

I also tried go get -m. Same results.

bradfitz · 2019-05-02T18:49:32Z

This is the thread that discussed this:

I'm still not able to make it work.

thepudds · 2019-05-02T19:09:45Z

Maybe this is "expected" for some reason, or maybe there is something specific to how GOROOT is set, or some other stdlib-specific step, but if this was a normal module (rather than stdlib), then this would seem to be a bug.

In this case, it looks like running go mod vendor does update src/vendor/modules.txt with the proper pseudoversion for golang.org/x/net (including expected commit of 9ce7a6920f09):

# golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09
golang.org/x/net/dns/dnsmessage
golang.org/x/net/lif
golang.org/x/net/route
golang.org/x/net/http/httpguts
golang.org/x/net/http/httpproxy
golang.org/x/net/http2/hpack
golang.org/x/net/idna
golang.org/x/net/nettest

...but the corresponding .go source files in vendor are not.

Also, what is vendor/modules.txt? What role does it play and is it hand maintained or tool maintained and where is it documented?

vendor/modules.txt is maintained by the go tool, and is not normally hand edited. I believe the format is consider an implementation detail and not documented externally... but it often (always?) matches the output of go mod vendor -v, which is lightly documented:

usage: go mod vendor [-v]

Vendor resets the main module's vendor directory to include all packages
needed to build and test all the main module's packages.
It does not include test code for vendored packages.

The -v flag causes vendor to print the names of vendored
modules and packages to standard error.

In this case, go mod vendor -v does match the contents of ./vendor/modules.txt.

bradfitz · 2019-05-02T19:32:09Z

Okay, @jayconrod points out that I'm confused. I thought the httpproxy package (which we use) was the same or depended on the golang.org/x/net/proxy package (which contains the all_proxy stuff), but it doesn't. They're separate. I think that transition never completed.

But docs would still be nice.

thepudds · 2019-05-02T19:33:03Z

Two other quick comments.

It seems go mod vendor -v includes the package golang.org/x/net/http/httpproxy in its output, but not the package golang.org/x/net/proxy

The module cache in GOPATH/pkg/mod does seem to end up with golang.org/x/net/proxy with all_proxy:

$ cd $(go list -f '{{.Dir}}' -m golang.org/x/net)
$ pwd
.../go/pkg/mod/golang.org/x/net@v0.0.0-20190502183928-7f726cade0ab
$ grep -ri all_proxy
proxy/dial_test.go:             if err = os.Setenv("ALL_PROXY", fmt.Sprintf("socks5://%s", s.Addr().String())); 
...

I have never tried to vendor std before, so it could be I am making a simple mistake.

edit: ok, seems like mystery is now solved.

jayconrod · 2019-05-02T19:36:35Z

So it seems vendoring is working as intended; I was initially confused because I misinterpreted bisect results during a meeting.

Documentation definitely needed. I thought there was a README at some point, but I'm unable to find it in the history.

rsc · 2019-05-02T21:19:36Z

We'll add the docs. I also have a pending CL that makes essentially all go commands after 'go get whatever@latest' die with a message saying 'now you run go mod vendor'.

gopherbot · 2019-05-03T22:09:57Z

Change https://golang.org/cl/174999 mentions this issue: all: document vendoring in the standard library

bradfitz added Documentation NeedsFix The path to resolution is known, but the work has not been done. release-blocker modules labels May 2, 2019

bradfitz added this to the Go1.13 milestone May 2, 2019

bradfitz assigned jayconrod May 2, 2019

gopherbot closed this as completed in a44c3ed May 9, 2019

golang locked and limited conversation to collaborators May 8, 2020

gopherbot added the FrozenDueToAge label May 8, 2020

rsc unassigned jayconrod Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vendor: no docs on how to update a module in std #31806

vendor: no docs on how to update a module in std #31806

bradfitz commented May 2, 2019

ianlancetaylor commented May 2, 2019

bradfitz commented May 2, 2019

bradfitz commented May 2, 2019

thepudds commented May 2, 2019

bradfitz commented May 2, 2019

thepudds commented May 2, 2019 •

edited

jayconrod commented May 2, 2019

rsc commented May 2, 2019

gopherbot commented May 3, 2019

vendor: no docs on how to update a module in std #31806

vendor: no docs on how to update a module in std #31806

Comments

bradfitz commented May 2, 2019

ianlancetaylor commented May 2, 2019

bradfitz commented May 2, 2019

bradfitz commented May 2, 2019

thepudds commented May 2, 2019

bradfitz commented May 2, 2019

thepudds commented May 2, 2019 • edited

jayconrod commented May 2, 2019

rsc commented May 2, 2019

gopherbot commented May 3, 2019

thepudds commented May 2, 2019 •

edited