Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: special case for chiselapp.com interferes with '.fossil' extension #31014

Open
urandom2 opened this issue Mar 23, 2019 · 12 comments
Open
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@urandom2
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
go version go1.12.1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build279953534=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I tried to clone a fossil repo from chiselapp using the .fossil to test self-hosted fossil repos. It looks like the chiselapp.com/user/kyle/repository/fossilgg format is handled separately in cmd/go/internal/get/vcs.go.

$ go get chiselapp.com/user/kyle/repository/fossilgg.fossil

What did you expect to see?

The go get succeeds and documentation can be listed via go doc for confirmation:

$ go doc -u -all -src fossilgg
package fossilgg.fossil // import "chiselapp.com/user/kyle/repository/fossilgg.fossil"


CONSTANTS

const greeting = "Hello from fossil!"

FUNCTIONS

func Hello() string {
        return greeting
}

What did you see instead?

The fossil clone fails:

# cd .; fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil /home/user/go/src/chiselapp.com/user/kyle/repository/fossilgg.fossil/.fossil
redirect limit exceeded
server returned an error - clone aborted
package chiselapp.com/user/kyle/repository/fossilgg.fossil: exit status 1

This also leaves behind an empty fossilgg.fossil directory in $GOPATH that has the side effect of preventing future calls to go get:

$ go get chiselapp.com/user/kyle/repository/fossilgg.fossil
can't load package: package chiselapp.com/user/kyle/repository/fossilgg.fossil: no Go files in /home/user/go/src/chiselapp.com/user/kyle/repository/fossilgg.fossil
@ALTree ALTree added this to the Go1.13 milestone Mar 24, 2019
@ALTree ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 24, 2019
@bcmills
Copy link
Contributor

bcmills commented Mar 28, 2019

Are you sure that the repository can be cloned in general? If so, what command did you use to verify that, and in what way (if any) does it differ from the fossil clone command printed by go get?

How does the behavior change, if at all, in module mode (with GO111MODULE=on and the working directory within a module)?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 28, 2019
@ksshannon
Copy link
Contributor

@arnottcr you need to supply the path to the repository with out the .fossil extension:

go get chiselapp.com/user/kyle/repository/fossilgg

@urandom2
Copy link
Contributor Author

urandom2 commented Mar 29, 2019

@kamanson: I am aware that the clone works without the .fossil suffix, however per the spec I should be able to clone with either, right?

@bcmills: while the output is not identical, modules appear to preserve the failing behaviour:

$ GO111MODULE=on go get chiselapp.com/user/kyle/repository/fossilgg.fossil
go: finding chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db
go get chiselapp.com/user/kyle/repository/fossilgg.fossil: fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil .fossil in /tmp/go/pkg/mod/cache/vcs/351aec3555812167b2e4d08a2027fe8df20a3e623d2d5ffa60cfba15a9da2437: exit status 1:
        redirect limit exceeded
        server returned an error - clone aborted
$ GO111MODULE=off go get chiselapp.com/user/kyle/repository/fossilgg.fossil
# cd .; fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil /tmp/go/src/chiselapp.com/user/kyle/repository/fossilgg.fossil/.fossil
redirect limit exceeded
server returned an error - clone aborted
package chiselapp.com/user/kyle/repository/fossilgg.fossil: exit status 1

It looks like an underlying fossil clone failure may be the root cause:

$ fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil fossilgg.fossil
redirect with status 302 to http://chiselapp.com/notfound
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect limit exceeded
Clone done, sent: 6998  received: 13314  ip: 2607:f1c0:84b:4b02:68e8:7a3f:2812:3fc0
server returned an error - clone aborted
$ fossil clone https://chiselapp.com/user/kyle/repository/fossilgg bare
Round-trips: 2   Artifacts sent: 0  received: 16
Clone done, sent: 572  received: 14731  ip: 10.0.1.187
Rebuilding repository meta-data...
  100.0% complete...
Extra delta compression... 
Vacuuming the database... 
project-id: e85b3a05650ffbb291dd1f398ab5b602d1859efb
server-id:  199d504c2d1c6d90f045d5f4e906fb93ff0b526d
admin-user: user (password is "password")

For completeness, I have included output for the successful go get calls without vcs suffix:

$ GO111MODULE=off go get chiselapp.com/user/kyle/repository/fossilgg
$ GO111MODULE=on go get chiselapp.com/user/kyle/repository/fossilgg
go: finding chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db
go: finding chiselapp.com/user/kyle/repository/fossilgg latest
go: downloading chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db
go: extracting chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db

Further investigation shows that this vcs suffix does work from some endpoints:

however this one uses a meta tag, so I am unsure if it counts:
<meta name="go-import" content="fossil.boisestate.edu/fossilgg fossil https://fossil.boisestate.edu/fossilgg">

$ GO111MODULE=on go get fossil.boisestate.edu/fossilgg.fossil
go: finding fossil.boisestate.edu/fossilgg.fossil latest
go: downloading fossil.boisestate.edu/fossilgg.fossil v0.0.0-20160617224315-0373066a09db
go: extracting fossil.boisestate.edu/fossilgg.fossil v0.0.0-20160617224315-0373066a09db
$ GO111MODULE=off go get fossil.boisestate.edu/fossilgg.fossil
$ GO111MODULE=on go get fossil.boisestate.edu/fossilgg
go: finding fossil.boisestate.edu/fossilgg.fossil v0.0.0-20160617224315-0373066a09db
go: finding fossil.boisestate.edu/fossilgg v0.0.0-20160617224315-0373066a09db
go: finding chiselapp.com/user/kyle/repository/fossilgg v0.0.0-20160617224315-0373066a09db
go: finding fossil.boisestate.edu/fossilgg latest
go: downloading fossil.boisestate.edu/fossilgg v0.0.0-20160617224315-0373066a09db
go: extracting fossil.boisestate.edu/fossilgg v0.0.0-20160617224315-0373066a09db
$ GO111MODULE=off go get fossil.boisestate.edu/fossilgg

@ksshannon
Copy link
Contributor

@arnottcr can you point to

@kamanson: I am aware that the clone works without the .fossil suffix, however per the spec I should be able to clone with either, right?

Chisel can handle redirects however it wants, same as any other code host. It doesn't work with git either:

$ go get github.com/golang/text.git
go get github.com/golang/text.git: invalid version control suffix in github.com/ path

Can you point me to the spec you are referring to?

@bcmills
Copy link
Contributor

bcmills commented Mar 29, 2019

Can you point me to the spec you are referring to?

I'm guessing the section in https://tip.golang.org/cmd/go/#hdr-Remote_import_paths starting at “For code hosted on other servers […]”.

But note that that is an either/or, not both: in general we expect each repository to have one canonical path, not multiple variations. You should import via only that canonical path.

@urandom2
Copy link
Contributor Author

Yeah, that was the paragraph I was citing as spec, but my reading was that the package/module owner had the choice of either option, not the entire hosting provider. So if a vcs suffix was provided, it would trump the http/meta driven resolution.

I guess I do not see the benefit of preventing a package/module owner from using github.com/user/repo.git/pkg if they want to? But if this is working as expected, I can understand that, it just feels like a hidden rule set[0], that only serves to confuse users.

0] This rule only applies for GitHub and JazzHub, and leverages get.noVCSSuffix.

@bcmills
Copy link
Contributor

bcmills commented Apr 1, 2019

my reading was that the package/module owner had the choice of either option, not the entire hosting provider

chiselapp.com is literally a special case:

// chiselapp.com for fossil
{
prefix: "chiselapp.com/",
regexp: lazyregexp.New(`^(?P<root>chiselapp\.com/user/[A-Za-z0-9]+/repository/[A-Za-z0-9_.\-]+)$`),
vcs: "fossil",
repo: "https://{root}",
},

We should probably try to migrate all of those special cases over to go-import meta tags, but it doesn't seem critical enough to do right now.

@bcmills bcmills changed the title cmd/go: go get fossil failures cmd/go: special case for chiselapp.com interferes with '.fossil' extension Apr 1, 2019
@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 1, 2019
@bcmills bcmills modified the milestones: Go1.13, Unplanned Apr 1, 2019
@urandom2
Copy link
Contributor Author

urandom2 commented Apr 2, 2019

sgtm; would you mind if I worked on a patch, or is there a larger refactor in the works that would nullify such efforts? (e.g. modules)

@ksshannon
Copy link
Contributor

If you add the .fossil extension, fossil itself fails with the redirect error:

fossil clone https://chiselapp.com/user/kyle/repository/fossilgg.fossil tmp.fossil
Click... redirect with status 302 to http://chiselapp.com/notfound redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/ redirect with status 301 to http://chiselapp.com/notfound/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/xfer/
redirect limit exceeded
Clone done, sent: 7018  received: 13314  ip: 74.208.146.128
server returned an error - clone aborted

I don't understand how this can be fixed. Would we drop the .fossil extension? This seems like even more of a special case to me. Fossil itself is hosted at https://fossil-scm.org/, which can't have an extension.

@bcmills
Copy link
Contributor

bcmills commented Apr 2, 2019

@arnottcr, I think that deprecating any (or all) of the cmd/go special-cases (presumably replacing them with go-import tags) would need to go through the proposal committee.

In particular, we would need buy-in from folks who currently maintain or use those hosting services, since either they would need to add the go-import tags for their servers or folks would need to migrate off of those services.

@bcmills
Copy link
Contributor

bcmills commented Apr 2, 2019

I'd like to take a step back, though: what's the concrete problem you're trying to address using the .fossil extension?

@urandom2
Copy link
Contributor Author

urandom2 commented Apr 4, 2019

I was looking into .fossil support for gddo, but was unable to find non-chiselapp fossil hosting[0]. Thus, I was hoping that chiselapp.com/user/<user>/repository/<repo>.fossil would be a decent substitute, however it failed to go get so I opened a ticket.

I do not have a workflow that is broken by this issue, and realistically it would probably be more useful to implement gddo support as a provider/special case like github or bitbucket, since chisel has somewhat standard APIs.

I was not aware when I opened this ticket that github.com/user/repo.git also failed, so I am happy to leave this to languish until such a time that it makes sense to clean up all, or at least most, the special cases.

0] I think this is because fossil is intended as a github in a binary, so most people run their own, however I was trying to find something official.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants