cmd/go: document restrictions on module zip extraction #33778
Comments
|
This is unfortunate but it seems necessary to ensure reproducible builds, which is one of the major design goals of modules. It's a stricter system than GOPATH, and some adaptation is needed. There are a number of other constraints, too, for example import paths are restricted to ASCII alphanumeric characters with a handful of allowed punctuation characters.
I'm not sure what you mean here. Could you elaborate? There's no way to avoid an issue on systems with case-insensitive file systems. We'd have to pick one of the files but not the other. If this were a .go file, code would be missing. Even on a case-sensitive file system, if you were cross-compiling to some target platform, you'd get a different result than if you had built from a case-insensitive file system. |
jayconrod
added
the
WaitingForInfo
I expected to be able to run Is not supporting files that share the same name that differ only by case a known and accepted limitation of Go modules? I couldn't find it reported or noted anywhere in the Go Modules wiki. |
It can't succeed on a case-insensitive file system, and it can't produce the same result on a case-sensitive file system as on another system though.
It's intended behavior, but like many aspects of modules, it's not adequately documented. I think probably the best thing to do here is to document exactly how the Go command produces a module zip file from a repository. There's a lot of undocumented, nuanced behavior around here, for example around symbolic links, directories without source files, and file size limits. |
jayconrod
added
Documentation
modules
NeedsInvestigation
|
In the framework proposed in #33637, this would be a page in the specification section. |
|
For context, I'm attempting to use the dependency on a case sensitive filesystem, where it is been used with the
Could you elaborate on this statement? |
Suppose you check out a repo on a |
jayconrod
changed the title
|
If this isn't an issue that will be fixed, rather than document restrictions regarding zip extraction, I think it would be clearer to document that Go Modules do not support containing two files with names that differ only by case. Anyone can get into this situation. In the |
|
I think this needs more attention than just documentation. I'm trying to avoid the version of I think the tool might be traversing available versions, and getting stuck along the way to a good version, because the version I need is older than the version with the filename collision. This could become a problem for any repo that accidentally commits two files that only differ in name by their case. Any version prior to that version becomes difficult to get to. Repos shouldn't become inaccessible just because a version was accidentally released with The only way I've managed to get around this is to manually hack my go.mod file to be specific versions to avoid the Go tool from traversing known versions. You can simulate this by importing these two packages side-by-side in a new module. Below is a set of commands to see what I see. I even try excluding the version. I see that when @rsc first added the limitation on case folding (5c622a5) he commented that the goal was to restrict what was supported up front then maybe loosen requirements. I agree portability is important, but probably not at the sacrifice of the Go tool being resilient to simple mistakes in discarded versions of our modules. Side note, this problem is great exasperated because the dependencies of the tests in my dependencies are included in the dependency tree, even though I have no interest in running my dependencies tests. |
|
I built a LAN private code repository proxy to speed up the construction of docker images, because I usually need to pull a lot of packages when I build. I use Goproxy for storage. However, when using Go1.13, it was found that there would be conflicts between the zip packages retrieved, not only case-insensitive, but also between the zip packages containing |
|
macOS is a dominate platform, and the default filesystem isn't case sensitive. This makes it distressingly easy to create a condition like:
These files don't conflict, and on macOS, they will appear in the same directory. A developer may not even know there's a case discrepancy until they try to import Though I respect the ideological stance of case-sensitivity, I think not providing a workarounds pushes a lot of frustrating make-work out to people just trying to use modules. |
|
Change https://golang.org/cl/202042 mentions this issue: |
|
This also happened in alessiosavi/GoGPUtils#2. /cc @alessiosavi |
zip provides three new functions: * Create - build a zip from an abstract list of files, filtering out files in submodules and vendor directories. This is useful for filtering a zip produced by a VCS tool (as the go command does). * CreateFromDir - build a zip from a directory. This is a convenience wrapper for Create. * Unzip - extract a zip file, checking various restrictions. A list of restrictions on module paths, versions, files within zips, and size limits is included in the package documentation. Both Create and Unzip enforce these restrictions. Also: copied cmd/go/internal/txtar to internal/txtar for testing. Updates golang/go#31302 Updates golang/go#33312 Updates golang/go#33778 Change-Id: I6fedb8b839a0cd991c9b210e73bafedc4b286ec5 Reviewed-on: https://go-review.googlesource.com/c/mod/+/202042 Run-TryBot: Jay Conrod <jayconrod@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
Attempted to
go geta package at a version.What did you expect to see?
The packages to be
go getd without any issue.What did you see instead?
go geterror'd saying that the package in the sum db is410 Gone.If I visit the
sum.golang.orgURL manually in a browser, I see this error:The
packrproject did indeed have two files namedSHOULDERS.mdandshoulders.mdinv1.21.4.Setting
GONOSUMDBand turning offGOPROXYmoves past the limitation in sum.golang.org, but then a similar error appears locally:It appears that versions of projects that contain case insensitive collisions are currently not supported in Go Modules. Checking out the package into a GOPATH manually as
go getused to still works, anddepsuccessfully works with it too.I think for portability there's some value in not allowing files that collide on case insensitive file systems, but existing packages that were published are now inaccessible.
This issue was encountered while attempting to upgrade a repo from dep to modules stellar/go#1634.
The text was updated successfully, but these errors were encountered: