Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: cannot map pages in arena address space on Windows #8491

Open
gopherbot opened this issue Aug 7, 2014 · 22 comments
Open
Milestone

Comments

@gopherbot
Copy link

by kaixinlaoshu2008:

What does 'go version' print?
go 1.3 windows/386
What steps reproduce the problem?

file, handler, err := r.FormFile("file")
...
defer file.Close()

Use FormFile to receive documents.
When i upload a flie(100M) 4 or 5 times will get errors.

What happened?

fatal error: runtime: cannot map pages in arena address space

What should have happened instead?

go free memory after the FormFile receive a document.
@ianlancetaylor
Copy link
Contributor

Comment 1:

I think we need a lot more information.  What is FormFile?  Can you show us a complete
program?  Have you tried looking at a memory profile (see
http://blog.golang.org/profiling-go-programs)?

Labels changed: added repo-main, release-none, os-windows.

@bradfitz
Copy link
Contributor

bradfitz commented Aug 7, 2014

Comment 2:

OP means net/http.Request.FormFile but I'm also curious to see actual code.

Status changed to WaitingForReply.

@alexbrainman
Copy link
Member

Comment 3:

The platform is windows/386. So heap is small.
Alex

@gopherbot
Copy link
Author

Comment 4 by kaixinlaoshu2008:

my code:
//golang
package main
import (
    "html/template"
    "io"
    "log"
    "net/http"
    "os"
)
var uploadTemplate = template.Must(template.ParseFiles("page.html"))
func indexHandle(w http.ResponseWriter, r *http.Request) {
    if err := uploadTemplate.Execute(w, nil); err != nil {
        log.Fatal("Execute: ", err.Error())
        return
    }
}
func uploadHandle(w http.ResponseWriter, r *http.Request) {
    log.Println("method:", r.Method)
    if r.Method == "POST" {
        //128MB
        r.ParseMultipartForm(128 << 20)
        file, _, err := r.FormFile("file")
        if err != nil {
            log.Println(err)
            return
        }
        defer file.Close()
        path := "./"
        log.Println("create file")
        t, err := os.Create(path + "file")
        if err != nil {
            log.Println(err)
            return
        }
        defer t.Close()
        io.Copy(t, file)
        http.Redirect(w, r, "/", 302)
        return
    }
}
func main() {
    http.HandleFunc("/", indexHandle)
    http.HandleFunc("/upload", uploadHandle)
    http.ListenAndServe(":8080", nil)
}
//html
<html>
    <head>
        <title>Golang upload</title>
    </head>
    <body>
        <form id="uploadForm" method="POST" enctype="multipart/form-data" action="/upload">
            <p>Golang upload</p>
            <input type="file" id="file" name="file" />
            <input type="submit" value="upload">
        </form>
    </body>
</html>

@gopherbot
Copy link
Author

Comment 5 by kaixinlaoshu2008:

who can help me?

@alexbrainman
Copy link
Member

Comment 6:

kaixinlaoshu2008, I can run your program from #4 with no problems. How do I make it
crash? Thank you.
Alex

@gopherbot
Copy link
Author

Comment 7 by kaixinlaoshu2008:

today i try go 1.3 windows/386 on win7(x64),the same error.
Keep uploading a file(120MB) for 4 times in a short time on my computer what will make
it crash.

@gopherbot
Copy link
Author

Comment 8 by kaixinlaoshu2008:

today i try go 1.3.1 windows/386 on win7(x64),the same error.

@gopherbot
Copy link
Author

Comment 9 by kaixinlaoshu2008:

today i try go 1.3.1 windows/386 on win7(x64),not "go 1.3 windows/386".

@alexbrainman
Copy link
Member

Comment 10:

The problem is that your address space is quite small - 32-bit windows program have only
2GB address space. So Go cannot allocate a lot of memory. If you try to run
windows/amd64 (64-bit) version of Go, the limit is much high. So it should not crash as
quickly.
@Brad, in your opinion, how much memory should that server use under described
conditions? I don't have time to investigate it properly now. Thank you.
Alex

@gopherbot
Copy link
Author

Comment 11 by kaixinlaoshu2008:

Immediately free memory when a file upload is complete,the problem will be resolved.
In fact,the program don't immediately free memory when a file upload is complete,and
memory goes up with the size of the uploaded files before crash.
Monitoring the changes of program's memory,i found the program don't free memory when a
file upload is complete and time has passed for 5 minutes or more.

@alexbrainman
Copy link
Member

Comment 12:

kaixinlaoshu2008,
You use r.ParseMultipartForm in your program. It uses a lot of memory, because it parses
multi-part form in memory, and, if large file is part of that form, then
ParseMultipartForm will use as much memory as you've allowed. You have allowed to go up
to (128 << 20) bytes. So ParseMultipartForm tries to get as much memory as
possible on your computer.
You use windows/386 version of Go, so your total runtime heap is about 500M, so you
don't have much when you are playing with 300M files. You cannot increase heap, because
your OS allows for max of 2G virtual address space, and you need to fit many other
things apart from Go runtime heap.
I think your program behaves as expected. But I will let Brad to make that decision. He
wrote this code, so he might have better suggestions.
Sorry.
Alex

Status changed to New.

@gopherbot
Copy link
Author

Comment 13 by kaixinlaoshu2008:

If the program immediately free memory,the runtime heap(500M) is enough.Because i am 
uploading a single file(size<120MB) in every time.

@gopherbot
Copy link
Author

Comment 14 by kaixinlaoshu2008:

where is Brad?

@bradfitz bradfitz removed the new label Dec 18, 2014
@meilihao
Copy link

Today I tried again on go1.4.2 linux/amd64,but got the same problem.go does't free the uploaded file's memory.
ps:程序还是不会立即释放文件上传所使用过的内存,导致内存随文件上传而不断减少直至crash.

@minux
Copy link
Member

minux commented Mar 27, 2015 via email

@meilihao
Copy link

It happens on ubuntu 14.04 x86_84/go1.4.2 linux/amd64 too.The program eats memory each time until crash when files are uploaded.

My core questions is "how to free memory immediately".

And my whole code is in this issue ,a go file and a html file.you can test it.

@randall77
Copy link
Contributor

I looked into this a little bit (with 1.3.3). I don't see quite the behavior you are.

I tried uploading the same 90MB file 10+ times. The heap grows to about 500MB over the first three or four uploads and then stops. After I stop uploading, GC eventually gets back to <1MB. It takes two background cycles to get there because of the concurrent sweep. You can speed that up by calling runtime.GC(), although in your example program there really isn't a good place to do that (you'd need it inside ListenAndServe after the http.Request is dropped).

We should ideally be able to handle this workload in ~256MB. I'm not entirely sure why it gets to 500MB.

@meilihao
Copy link

I am so sorry,my tests are not carefully enough on amd64.The program on amd64 is ok,not crash.

"r.ParseMultipartForm(128 << 20)" makes the heap grows to about 511MB and then stops.If not use it, the heap upper limit is 256MB. And the memory eventually gets back to <17MB after 9 Minutes.The long time is ok?

@rsc rsc removed the os-windows label Apr 10, 2015
@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@mikioh
Copy link
Contributor

mikioh commented May 7, 2015

Now we see this issue at tip: http://build.golang.org/log/d9c532d3a1b8266fae407518d131bc872f41a617.

@mikioh mikioh changed the title runtime: cannot map pages in arena address space on Windows runtime: fatal error: cannot map pages in arena address space on Windows May 7, 2015
@alexbrainman
Copy link
Member

@mikioh please create new issue instead. Thank you. I suspect some tests use too much memory for windows/386.

Alex

@mikioh
Copy link
Contributor

mikioh commented May 7, 2015

Filed #10737.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants