Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: ParseMultipartForm using 8x maxMemory #53109

Closed
git001 opened this issue May 27, 2022 · 3 comments
Closed

net/http: ParseMultipartForm using 8x maxMemory #53109

git001 opened this issue May 27, 2022 · 3 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@git001
Copy link

git001 commented May 27, 2022

What version of Go are you using (go version)?

$ go version
go version go1.18.1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/alex/.cache/go-build"
GOENV="/home/alex/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/alex/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/alex/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2728191052=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I have created a small upload tool for the caddy v2 server
https://github.com/git001/caddyv2-upload

I set the MaxBytesReader ( https://github.com/git001/caddyv2-upload/blob/main/upload.go#L125 ) and ParseMultipartForm ( https://github.com/git001/caddyv2-upload/blob/main/upload.go#L126 ) to limit the memory usage.

What we have observed is that even when we limit the ParseMultipartForm() maxMemory parameter is the initial memory consumption 7-8 times higher then the given maxMemory.
git001/caddyv2-upload#2

What did you expect to see?

We expect that the memory usage is not 7-8 times higher at the initial upload phase then the configured one for ParseMultipartForm

What did you see instead?

We see that the memory usage 7-8 times higher then configured maxMemory in ParseMultipartForm
@seankhliao
Copy link
Member

Please try profiling to identify where the memory might be used.
Note maxMemory refers to the max file size that will be stored in memory, not the maximum memory that will be used during the operation.

@seankhliao seankhliao changed the title affected/package: net/http => ParseMultipartForm net/http: ParseMultipartForm using 8x maxMemory May 27, 2022
@seankhliao seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 27, 2022
@git001
Copy link
Author

git001 commented May 27, 2022

Note maxMemory refers to the max file size that will be stored in memory, not the maximum memory that will be used during the operation.

thank you for the info. I will profile the app.

@git001
Copy link
Author

git001 commented May 28, 2022

I have now created a simple module which shows my observations.
https://github.com/git001/golang-multiparttest

What I have seen is that the parameter maxMemory is a very important "flag" between memory usage and filesystem usage.
My conclusion is that the Multipart upload requires 2*times the memory of maxMemory but I was not able to use the tool pprof in that way to find out why this happen.

Finally I understand now the sentence in the doc which explains exactly the same which could be the reason for the doubled memory usage.

The whole request body is parsed and up to a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files.

@git001 git001 closed this as completed May 28, 2022
@golang golang locked and limited conversation to collaborators May 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@git001 @gopherbot @seankhliao