Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: FileServer on WSL fails with wget and curl for files over ~4MB #27128

Closed
upsampled opened this issue Aug 21, 2018 · 5 comments
Closed

net/http: FileServer on WSL fails with wget and curl for files over ~4MB #27128

upsampled opened this issue Aug 21, 2018 · 5 comments
Labels
OS-Windows
Milestone
Unplanned

Comments

@upsampled
Copy link

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build493773858=/tmp/go-build -gno-record-gcc-switches"

using WSL

Ran the same test on a CentOS Machine and I did not see this problem

What did you do?

Attempted to server files via HTTP that an application using curl can download.

package main

import (
        "log"
        "net/http"
)

func main() {
        fs := http.FileServer(http.Dir("./"))
        http.Handle("/repo/", http.StripPrefix("/repo/", fs))

        log.Println("Listening...")
        http.ListenAndServe(":80", nil)
}

What did you expect to see?

Curl to download the files.

What did you see instead?

The file server worked for file that were roughly less than 5M, but failed for larger files with curl error 18. The curl command line tool also reproduces this error:

curl http://<IP>/repo/<FILE> --output <FILE>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                    Dload  Upload   Total   Spent     Left  Speed
 56 7257k   56 4096k    0     0      4970k      0       0:00:01 --:--:--  0:00:01 4970k
curl: (18) transfer closed with 3237034 bytes remaining to read

Wget (busybox version) also has the same problem:

wget http://<IP>/repo/<FILE> -O <FILE>
Connecting to <IP> (<IP>:80)
pa-config-1-r0.corte   6% |**********                                                      |   460k  0:00:14 ETA
<STOPS and the file is only ~4M>

However using pythons web server, wget and curl both work
sudo python3 -m http.server 80
@bradfitz
Copy link
Contributor

The WSL team want you to file bugs to them about any differences from actual Linux.

We generally don't investigate these things, or only do some when it appears we might be violating Linux or POSIX APIs in undocumented ways, or do so out of curiosity.

But please file a WSL bug and link it here.

We probably won't be investigating this. It's WSL's job to act like Linux and they don't want projects treating them like its own port.

@bradfitz bradfitz added this to the Unplanned milestone Aug 22, 2018
@upsampled upsampled mentioned this issue Aug 22, 2018
Closed
@hullarb
Copy link

hullarb commented Jun 18, 2019

i run into the same issue and while debugging it i saw that CopyN in the fileserver returns EOF early:
https://golang.org/src/net/http/fs.go#L296
(changing that line to readall and write works just fine if you need a quick workaround)

@bradfitz i wondered if the ignored error could be checked and logged to make easier to debug issues happening there?

@agnivade agnivade mentioned this issue Jul 20, 2019
Closed
hullarb added a commit to hullarb/go that referenced this issue Aug 27, 2019
@hullarb
net/http: log error if fileserver fails to copy the file content
f66c1bb 
Logging the error returned by CopyN could help finding the root cause
of issues like golang#27128.
@hullarb hullarb mentioned this issue Aug 27, 2019
Closed
@gopherbot
Copy link

Change https://golang.org/cl/191971 mentions this issue: net/http: log error if fileserver fails to copy the file content

hullarb added a commit to hullarb/go that referenced this issue Sep 12, 2019
@hullarb
net/http: improve FileServer error logging
9bf1d0f 
The errors returned during copying the file content in ServeContent,
ServeFile and FileServer were ignored to avoid excessive, meaningless
logging. This commit introduces error filtering and logging to ensure
that the errors occurring during reading the content are logged.

Updates golang#27128
hullarb added a commit to hullarb/go that referenced this issue Sep 12, 2019
@hullarb
net/http: improve FileServer error logging
9b69487 
The errors returned during copying the file content in ServeContent,
ServeFile and FileServer were ignored to avoid excessive, meaningless
logging. This commit introduces error filtering and logging to ensure
that the errors occurring during reading the content are logged.

Updates golang#27128
hullarb added a commit to hullarb/go that referenced this issue Sep 12, 2019
@hullarb
net/http: improve FileServer error logging
7976f48 
The errors returned during copying the file content in ServeContent,
ServeFile and FileServer were ignored to avoid excessive, meaningless
logging. This commit introduces error filtering and logging to ensure
that the errors occurring during reading the content are logged.

Updates golang#27128
hullarb added a commit to hullarb/go that referenced this issue Oct 24, 2019
@hullarb
net/http: improve FileServer error logging
2ae816e 
The errors returned during copying the file content in ServeContent,
ServeFile and FileServer were ignored to avoid excessive, meaningless
logging. This commit introduces error filtering and logging to ensure
that the errors occurring during reading the content are logged.

Updates golang#27128
hullarb added a commit to hullarb/go that referenced this issue Oct 24, 2019
@hullarb
net/http: improve FileServer error logging
3e18e1a 
Errors returned during copying the file content in ServeContent,
ServeFile and FileServer were ignored to avoid excessive, meaningless
logging. This commit introduces error filtering and logging to ensure
that the errors occurring during reading the content are logged.

Updates golang#27128
hullarb added a commit to hullarb/go that referenced this issue Oct 30, 2019
@hullarb
net/http: improve FileServer error logging
df595d2 
Errors returned during copying the file content in ServeContent,
ServeFile and FileServer were ignored to avoid excessive, meaningless
logging. This commit introduces error filtering and logging to ensure
that the errors occurring during reading the content are logged.

Updates golang#27128
@janpfeifer
Copy link

janpfeifer commented Nov 22, 2020 •
edited

Curiously I'm seeing the same effect. But instead of trimming the file, net/http (or something in between) changes the file somehow, starting at position 0x00400000 (exactly 4Mb) serves something different, it's very odd.

As HTTP client, I tried downloading the file in Chrome running in windows (while the server was local in WSL). And then I tried wget runnint from within WSL. Both Chrome and wget in WSL download the same altered file (the size remains the same).

go 1.15.5 here. But I've noticed that werving in WSL didn't work for a long time now.

If I serve the same file with:

python3 -m http.server -b localhost 8888

it works.

@janpfeifer
Copy link

janpfeifer commented Nov 22, 2020 •
edited

For others bumping into this, if I change the following in my server code:

	http.Handle("/", http.FileServer(http.Dir(*flagStaticFilesPath)))

To:

	fs := dotFileHidingFileSystem{http.Dir(*flagStaticFilesPath)}
	http.Handle("/", http.FileServer(fs))

Using the dotFileHidingFileSystem implemented in the documentation's example, it works as intended.

@golang golang locked and limited conversation to collaborators Mar 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
OS-Windows
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
6 participants
@bradfitz @hullarb @janpfeifer @gopherbot @upsampled and others