net/http: ServeFile fails to serve files over 4GB on Windows #33193

NickTikhonov · 2019-07-19T17:10:55Z

What version of Go are you using (`go version`)?

$ go version
go version go1.12.7 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\nicktikhonov\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=...
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\NICKTI~1\AppData\Local\Temp\go-build372822890=/tmp/go-build -gno-record-gcc-switches

What did you do?

Ran the following script. Before that, you can run

fsutil file createnew bigfile 8589934610

to generate a large file.

package main

import (
    "context"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "time"
)

// Create a random file in the same directory
// fsutil file createnew bigfile 8589934610

type indexHandler struct {
}

func (h *indexHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "bigfile")
}

func main() {
    fmt.Println("Starting server....")
    httpServer := &http.Server{Addr: fmt.Sprintf("%v:%v", "localhost", "8080"), Handler: &indexHandler{}}
    go func() {
        err := httpServer.ListenAndServe()
        if err != nil {
            fmt.Println("Err from server")
            fmt.Println(err)
        }
    }()

    time.Sleep(5 * time.Second)

    fmt.Println("Starting client....")
    client := &http.Client{}
    req, err := http.NewRequest("GET", "http://localhost:8080/", nil)
    req.Header.Add("Connection", "Keep-Alive")
    req.Header.Add("Keep-Alive", "timeout=1000")
    res, err := client.Do(req)
    if err != nil {
        panic(err)
    }

    fmt.Println(req)

    totalLen := uint64(res.ContentLength)
    bodyReader := res.Body
    fmt.Printf("totalLen: %v\n", totalLen)
    written, err := io.Copy(ioutil.Discard, bodyReader)
    fmt.Printf("downloaded: %v\n", written)
    fmt.Printf("err: %v\n", err)
    httpServer.Shutdown(context.Background())
}

What did you expect to see?

I expected the download to have finished,

What did you see instead?

Instead, only 18 bytes are downloaded, and the client receives an unexpected EOF

totalLen: 8589934610
downloaded: 18
err: unexpected EOF

The text was updated successfully, but these errors were encountered:

agnivade · 2019-07-20T03:42:01Z

What do you get when you try to fetch the file using curl ?

What is the cutoff size beyond which you are seeing failures ? Are 1GB files working fine ?

Are you using any virtualization layer like WSL (#27128) or VM (#30082) by any chance ?

alexbrainman · 2019-07-21T07:20:58Z

@NickTikhonov I could reproduce your problem on current tip. Thank you very much.

I did not have enough time to understand where the problem is.

Alex

NickTikhonov · 2019-07-22T09:23:42Z

@alexbrainman I believe the problem could be some sort of uint overflow. The number of bytes downloaded seems close to the total size modulo 2 ^ 32 - 1

gopherbot · 2019-07-22T09:28:10Z

Change https://golang.org/cl/187037 mentions this issue: net: do not call Windows TransmitFile for large files

alexbrainman · 2019-07-22T09:30:15Z

@NickTikhonov I think I found the problem. Can you, please, try https://go-review.googlesource.com/c/go/+/187037 and see if it fixes the problem for you. Thank you.

Alex

mattn · 2019-07-24T14:07:40Z

@alexbrainman Thank your fixing this. (I implemented sendfile on Windows)

alexbrainman · 2019-07-28T02:39:30Z

@NickTikhonov

You did not say, if https://go-review.googlesource.com/c/go/+/187037 fixes your problem. Please confirm.

Thank you.

Alex

odeke-em · 2019-08-27T17:01:25Z

I'll reopen this issue as @bradfitz raised a comment on the CL https://go-review.googlesource.com/c/go/+/187037 about perhaps trying to send the file in 2GB chunks instead of failing the first time.

gopherbot · 2019-08-28T20:19:43Z

Change https://golang.org/cl/192143 mentions this issue: Revert "net: do not call Windows TransmitFile for large files"

gopherbot · 2019-08-30T06:13:57Z

Change https://golang.org/cl/192518 mentions this issue: net: handle >=2GiB files with sendfile on Windows

TransmitFile does not allow for number of bytes that can be transmitted to be larger than 2147483646. See https://docs.microsoft.com/en-us/windows/win32/api/mswsock/nf-mswsock-transmitfile for details. So adjust sendFile accordingly. No test added, because this would require creating large file (more than 2GB file). Fixes golang#33193. Change-Id: I82e0cb104d112264e4ea363bb20b6d02ac30b38e Reviewed-on: https://go-review.googlesource.com/c/go/+/187037 Run-TryBot: Alex Brainman <alex.brainman@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>

gopherbot · 2019-09-08T19:43:48Z

Change https://golang.org/cl/194218 mentions this issue: net: permit sendFile on more than *io.LimitedReader on Windows

CL 192518 was a minimal simplification to get sendfile on Windows to work with chunked files, but as I had mentioned, I would add even more improvements. This CL improves it by: * If the reader is not an *io.LimitedReader, since the underlying reader is anyways an *os.File, we fallback and stat that file to determine the file size and then also invoke the chunked sendFile on the underlying reader. This issue existed even before the prior CL. * Extracting the chunked TransmitFile logic and moving it directly into internal/poll.SendFile. Thus if the callers of net.sendFile don't use *io.LimitedReader, but have a huge file (>2GiB), we can still invoke the chunked internal/poll.SendFile on it directly. The test case is not included in this patch as it requires creating a 3GiB file, but that if anyone wants to view it, they can find it at https://go-review.googlesource.com/c/go/+/194218/13/src/net/sendfile_windows_test.go Updates #33193. Change-Id: I97a67c712d558c84ced716d8df98b040cd7ed7f7 Reviewed-on: https://go-review.googlesource.com/c/go/+/194218 Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>

dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 19, 2019

dmitshur added this to the Go1.14 milestone Jul 19, 2019

dmitshur changed the title ~~http.ServeFile fails to serve files over 4GB on Windows~~ net/http: ServeFile fails to serve files over 4GB on Windows Jul 19, 2019

dmitshur added the OS-Windows label Jul 19, 2019

gopherbot closed this as completed in b963149 Aug 27, 2019

odeke-em reopened this Aug 27, 2019

gopherbot closed this as completed in 3a067f7 Sep 7, 2019

golang locked and limited conversation to collaborators Sep 7, 2020

gopherbot added the FrozenDueToAge label Sep 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: ServeFile fails to serve files over 4GB on Windows #33193

net/http: ServeFile fails to serve files over 4GB on Windows #33193

NickTikhonov commented Jul 19, 2019

agnivade commented Jul 20, 2019

alexbrainman commented Jul 21, 2019

NickTikhonov commented Jul 22, 2019

gopherbot commented Jul 22, 2019

alexbrainman commented Jul 22, 2019

mattn commented Jul 24, 2019

alexbrainman commented Jul 28, 2019

odeke-em commented Aug 27, 2019

gopherbot commented Aug 28, 2019

gopherbot commented Aug 30, 2019

gopherbot commented Sep 8, 2019

net/http: ServeFile fails to serve files over 4GB on Windows #33193

net/http: ServeFile fails to serve files over 4GB on Windows #33193

Comments

NickTikhonov commented Jul 19, 2019

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

agnivade commented Jul 20, 2019

alexbrainman commented Jul 21, 2019

NickTikhonov commented Jul 22, 2019

gopherbot commented Jul 22, 2019

alexbrainman commented Jul 22, 2019

mattn commented Jul 24, 2019

alexbrainman commented Jul 28, 2019

odeke-em commented Aug 27, 2019

gopherbot commented Aug 28, 2019

gopherbot commented Aug 30, 2019

gopherbot commented Sep 8, 2019

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?