New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
net/http: slow http.ServeFile performance on Windows VM #30082
Comments
I decided to compare some other servers to see what kind of speeds they get in the same setup. In the Windows VM, I ran nodejs Python serves up the file slightly slower than golang, but nodejs is much faster than both. macOS <- Windows (golang port 9000)
macOS <- Windows (nodejs port 5000)
macOS <- Windows (python port 4000)
|
I also tried nginx (on port 80), which is the fastest so far but still similar to the 8s nodejs result:
I think this proves there is some sort of bottleneck in the golang runtime which is limiting throughput at around ~150MB/s, and causing the same file to take 37% longer to upload (8s vs 11s). |
I am not an expert in performance tuning. But, I think, your setup / source code has too many moving parts - you use http server, you use VM. The slow down can be anywhere. I expect your code uses syscall.TransmitFile optimisation https://github.com/golang/go/blob/master/src/internal/poll/sendfile_windows.go It might help, or might hurt. See, if you do use syscall.TransmitFile or not. Adjust your code, so you could try both "with syscall.TransmitFile" and "without syscall.TransmitFile" versions. If that does not help, maybe try and replace net/http package with simple custom TCP implementation. See if that makes any difference. I don't see how I could reproduce your problem here. And, even if I do, I would need C source of your alternative program to be able to try and reproduce your top performance with Go. Alex |
No difference with the following: package main
import (
"fmt"
"io"
"net"
"os"
)
func main() {
server, err := net.Listen("tcp", ":27001")
if err != nil {
fmt.Println("Error listetning: ", err)
os.Exit(1)
}
defer server.Close()
fmt.Println("Server started! Waiting for connections...")
for {
connection, err := server.Accept()
if err != nil {
fmt.Println("Error: ", err)
os.Exit(1)
}
go sendFileToClient(connection)
}
}
func sendFileToClient(connection net.Conn) {
fmt.Println("A client has connected!")
defer connection.Close()
file, err := os.Open("stream.mpg")
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
io.Copy(connection, file)
fmt.Println("File has been sent, closing connection!")
} I believe sendBuffer := make([]byte, 1024*64)
for {
_, err = file.Read(sendBuffer)
if err == io.EOF {
break
}
connection.Write(sendBuffer)
} In both cases, using
This should be trivial to reproduce. Run the tcp or http script above on a Windows host to serve up any large file on disk, then download it from another host and measure the speed/time. You will notice that it is impossible to get download speeds above 150MB/s regardless of available network/disk bandwidth. |
I am away from my computer for few weeks. It will have to wait until then. Alex |
No problem, enjoy your time off! I will continue to run some tests to see if I can narrow things down further. |
Do you mean that from inside your VirtualBox Windows VM you're serving a file that's using one of those funky shared filesystems that shares files from your host? If so, this bug is not actionable or very important. Those funky filesystems are for ease of development but not for production. It's very likely they're just not very performant. |
No, I have a separate copy I downloaded into the VM, and am not using any shared filesystem. nginx and nodejs can serve up the file at 200MB/s, but golang never goes over 150MB/s I'll find some time this week to boot up a physical Windows host and re-run the tests, just to rule out the VM altogether. |
I downloaded curl for windows (from https://curl.haxx.se/windows/ ), and tried running your program and curl.exe on my Windows PC.
It looks like What do I do now? Alex |
This was apparently a bug in 7.64.0 and is fixed in the new 7.64.1 release available at https://curl.haxx.se/windows/
(Note also that modern versions of Windows 10 ship with curl @ |
I started fresh on a physical windows machine with go1.12.2, and I figured out one source of my confusion:
Next, I added an endpoint that just serves up zeros to rule out anything disk/file related:
I hit this endpoint from another machine and confirmed I was able to achieve ~110MB/s against both a macOS server and Windows server. Then, I tried again with the stream.mpg endpoint and confirmed again that I'm getting ~110MB/s regardless of what OS the golang server is running on. It's clear this bug as originally described is not present, or if it was it is no longer in go1.12. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
I tried 1.9.7 and 1.10.7 with the same results. I assume 1.11.5 is the same as there are no new relevant commits.
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
I have a 4.2ghz i7 iMac, which is running macOS 10.14.2
On the iMac, I'm also using VirtualBox to run a VM with Windows 10 Pro 10.0.17134 Build 17134
I have a 1.65GB video file named
stream.mpg
that exists both on my macOS host and inside the VM.I'm using the following go program to serve up that file via http:
I'm running this server both on macOS and Windows. The macOS host is
10.0.1.10
and the Windows VM is10.0.1.20
I observe the following results when using curl to download the file to measure download speed/time. Note the "Average DLoad" and "Time Total" columns in the middle. Each result below is the best of three runs. (The first result not shown is usually much slower, as the file is not memory mapped into the fs cache yet).
Windows <- Windows
Windows <- macOS
macOS <- macOS
macOS <- Windows
What did you expect to see?
Similar speeds when serving the file from Windows and macOS.
What did you see instead?
When running the server on macOS, the local host can download 1.65GB in 1s, and the Windows VM can download the file in 3s.
When running the server on Windows, the local VM can download 1.65GB in 3s, but the host takes over 11s to fetch the file and maxes out at 145Mbps.
I've observed the same behavior with physical hosts, and the VMs in this test were only setup to remove any network hardware from the test.
The text was updated successfully, but these errors were encountered: