Skip to content

os/exec: Combined number of symbols in file paths are causing error if it exceeds ~2^15 symbols combined. #21145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ghost opened this issue Jul 24, 2017 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Milestone

Comments

@ghost
Copy link

ghost commented Jul 24, 2017

Hello,

What version of Go are you using (go version)?

go1.8.3

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\trumpe\PROJECTS\Go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

To begin, I pulled 507 files on my work repository. All files were in one folder and package. When I tried to compile I got an error:

go build good: C:\Go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\Go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.
exit status 1
Process exiting with code: 1

After some investigation I found out that no filename or extension were too long, also the number of files in the package has nothing to do with. The problem occurs when a sum of all file paths in a package exceeds 2^15(My guess only, because it depends on other paths as well. When I was testing it was always between 31740 and 31830 symbols, so my best guess some internal files are counted as well and 2^15 makes most sense), or around 32k symbols.
To clarify you can have around 640 files with filepaths "C:\Users\trumpe\PROJECTS\Go\src\testing1\file01.go" --> (50 symbols)
or for example 800 files with filepaths
"C:\Users\PROJECTS\Go\src\testfile0001.go" --> (40 symbols)
As long as it adds up to approx. 32000 symbols from all the files in the package.

I tried to recreate the problem by generating files and compiling them and everytime I came up to the same conclusion.

The files that I pulled were written on Linux Centos6.8 so I believe it is a Windows problem only.

Unfortunately it is against my company policy to reveal any source code I was using, that's why I created random files to recreate the problem. Powershell scripts to recreate the problem are provided below.

https://play.golang.org/p/6bjmwD9gS4

What did you expect to see?

I expected to compile the package and run the program.

What did you see instead?

go build good: C:\Go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\Go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.
exit status 1
Process exiting with code: 1

@josharian josharian changed the title Combined number of symbols in file paths are causing error if it exceeds ~2^15 symbols combined. os/exec: Combined number of symbols in file paths are causing error if it exceeds ~2^15 symbols combined. Jul 24, 2017
@josharian
Copy link
Contributor

Tentatively calling this an os/exec bug. Ideas, @alexbrainman ?

@josharian josharian added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows labels Jul 24, 2017
@josharian josharian added this to the Go1.10 milestone Jul 24, 2017
@ALTree
Copy link
Member

ALTree commented Jul 24, 2017

Looks like #18468.

@ianlancetaylor
Copy link
Member

I concur. Closing as dup of #18468.

Would be nice if somebody could sort this out.

@kardianos
Copy link
Contributor

kardianos commented Jul 24, 2017

@ianlancetaylor and @ALTree I don't think these are the same issue.

This issue (#21145) appears to be hitting a windows limitation of passing the max size of arguments to the compiler. In other projects what was often done was to create a temp file on windows if > 32k chars and redirect input to use that file. For instance: https://blogs.msdn.microsoft.com/oldnewthing/20031210-00/?p=41553 and https://bitbucket.org/scons/scons/wiki/LongCmdLinesOnWin32 and ninja-build/ninja#53 . Note, Linux also has a similar limitation, it is just much higher.

Issue #18468 appears to be dealing with the executable being called having a path length longer then 250 chars. So if the "go.exe" command happens to be under a long path itself. The prior issue none of the files individually were under a long path.

I'm going to re-open for now. @SokIsKedu does the above analysis sound correct to you?

@kardianos kardianos reopened this Jul 24, 2017
@ALTree
Copy link
Member

ALTree commented Jul 24, 2017

Issue #18468 appears to be dealing with the executable being called having a path length longer then 250 chars. So if the "go.exe" command happens to be under a long path itself.

But the error message in #18468 says:

fork/exec C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.

Where's this 250 characters path? The go folder's path looks quite short to me... what am I missing?

Also the OP on that old issue wrote:

I [..] have issues with large go build calls:

and

Tried to compile several files, many with long names, in a folder

so it looks to me that the OP in #18468 is having the same issue described in this issue (many files, some with long names, adds up to > 32k, go build on the folder fails). I don't know about the other 250 chars issue you are describing, but to me it looks like the #18468 OP is not reporting about that.

But I could be wrong.

@kardianos
Copy link
Contributor

@ALTree I think I see what you mean.

I think Brad was trying to fix the wrong thing in #18468. I'll post there.

@ghost
Copy link
Author

ghost commented Jul 24, 2017

@kardianos Sorry for late response. Yes you are correct, that is exactly what I described and it seems to me that #18468 issue might be similar, but it might be that he is just getting the same error for slightly different problem with the 250 chars in a path.

@golang golang locked and limited conversation to collaborators Jul 24, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Projects
None yet
Development

No branches or pull requests

5 participants