runtime: fatal error: cannot map pages in arena address space on Windows #8491

gopherbot · 2014-08-07T08:13:04Z

by kaixinlaoshu2008:

What does 'go version' print?
go 1.3 windows/386
What steps reproduce the problem?

file, handler, err := r.FormFile("file")
...
defer file.Close()

Use FormFile to receive documents.
When i upload a flie(100M) 4 or 5 times will get errors.

What happened?

fatal error: runtime: cannot map pages in arena address space

What should have happened instead?

go free memory after the FormFile receive a document.

ianlancetaylor · 2014-08-07T15:16:35Z

Comment 1:

I think we need a lot more information.  What is FormFile?  Can you show us a complete
program?  Have you tried looking at a memory profile (see
http://blog.golang.org/profiling-go-programs)?

Labels changed: added repo-main, release-none, os-windows.

bradfitz · 2014-08-07T16:05:15Z

Comment 2:

OP means net/http.Request.FormFile but I'm also curious to see actual code.

Status changed to WaitingForReply.

alexbrainman · 2014-08-08T00:40:01Z

Comment 3:

The platform is windows/386. So heap is small.
Alex

gopherbot · 2014-08-12T03:21:45Z

Comment 4 by kaixinlaoshu2008:

my code:
//golang
package main
import (
    "html/template"
    "io"
    "log"
    "net/http"
    "os"
)
var uploadTemplate = template.Must(template.ParseFiles("page.html"))
func indexHandle(w http.ResponseWriter, r *http.Request) {
    if err := uploadTemplate.Execute(w, nil); err != nil {
        log.Fatal("Execute: ", err.Error())
        return
    }
}
func uploadHandle(w http.ResponseWriter, r *http.Request) {
    log.Println("method:", r.Method)
    if r.Method == "POST" {
        //128MB
        r.ParseMultipartForm(128 << 20)
        file, _, err := r.FormFile("file")
        if err != nil {
            log.Println(err)
            return
        }
        defer file.Close()
        path := "./"
        log.Println("create file")
        t, err := os.Create(path + "file")
        if err != nil {
            log.Println(err)
            return
        }
        defer t.Close()
        io.Copy(t, file)
        http.Redirect(w, r, "/", 302)
        return
    }
}
func main() {
    http.HandleFunc("/", indexHandle)
    http.HandleFunc("/upload", uploadHandle)
    http.ListenAndServe(":8080", nil)
}
//html
<html>
    <head>
        <title>Golang upload</title>
    </head>
    <body>
        <form id="uploadForm" method="POST" enctype="multipart/form-data" action="/upload">
            <p>Golang upload</p>
            <input type="file" id="file" name="file" />
            <input type="submit" value="upload">
        </form>
    </body>
</html>

gopherbot · 2014-08-14T04:11:40Z

Comment 5 by kaixinlaoshu2008:

who can help me?

alexbrainman · 2014-08-14T04:48:15Z

Comment 6:

kaixinlaoshu2008, I can run your program from #4 with no problems. How do I make it
crash? Thank you.
Alex

gopherbot · 2014-08-14T06:50:22Z

Comment 7 by kaixinlaoshu2008:

today i try go 1.3 windows/386 on win7(x64),the same error.
Keep uploading a file(120MB) for 4 times in a short time on my computer what will make
it crash.

gopherbot · 2014-08-14T07:03:45Z

Comment 8 by kaixinlaoshu2008:

today i try go 1.3.1 windows/386 on win7(x64),the same error.

gopherbot · 2014-08-14T07:06:18Z

Comment 9 by kaixinlaoshu2008:

today i try go 1.3.1 windows/386 on win7(x64),not "go 1.3 windows/386".

alexbrainman · 2014-08-14T07:18:28Z

Comment 10:

The problem is that your address space is quite small - 32-bit windows program have only
2GB address space. So Go cannot allocate a lot of memory. If you try to run
windows/amd64 (64-bit) version of Go, the limit is much high. So it should not crash as
quickly.
@Brad, in your opinion, how much memory should that server use under described
conditions? I don't have time to investigate it properly now. Thank you.
Alex

gopherbot · 2014-08-15T02:03:57Z

Comment 11 by kaixinlaoshu2008:

Immediately free memory when a file upload is complete,the problem will be resolved.
In fact,the program don't immediately free memory when a file upload is complete,and
memory goes up with the size of the uploaded files before crash.
Monitoring the changes of program's memory,i found the program don't free memory when a
file upload is complete and time has passed for 5 minutes or more.

alexbrainman · 2014-08-15T04:52:43Z

Comment 12:

kaixinlaoshu2008,
You use r.ParseMultipartForm in your program. It uses a lot of memory, because it parses
multi-part form in memory, and, if large file is part of that form, then
ParseMultipartForm will use as much memory as you've allowed. You have allowed to go up
to (128 << 20) bytes. So ParseMultipartForm tries to get as much memory as
possible on your computer.
You use windows/386 version of Go, so your total runtime heap is about 500M, so you
don't have much when you are playing with 300M files. You cannot increase heap, because
your OS allows for max of 2G virtual address space, and you need to fit many other
things apart from Go runtime heap.
I think your program behaves as expected. But I will let Brad to make that decision. He
wrote this code, so he might have better suggestions.
Sorry.
Alex

Status changed to New.

gopherbot · 2014-08-18T07:50:09Z

Comment 13 by kaixinlaoshu2008:

If the program immediately free memory,the runtime heap(500M) is enough.Because i am 
uploading a single file(size<120MB) in every time.

gopherbot · 2014-08-25T01:50:30Z

Comment 14 by kaixinlaoshu2008:

where is Brad?

meilihao · 2015-03-27T03:53:54Z

Today I tried again on go1.4.2 linux/amd64,but got the same problem.go does't free the uploaded file's memory.
ps:程序还是不会立即释放文件上传所使用过的内存,导致内存随文件上传而不断减少直至crash.

minux · 2015-03-27T03:59:48Z

The program is asking to parse up to 128MB of the uploaded data in memory. On 32-bit platforms, there is only about 768MB of usable heap, it's easy to run out of memory. I didn't read the whole program to see if there is any other memory problems.

meilihao · 2015-03-27T08:45:17Z

It happens on ubuntu 14.04 x86_84/go1.4.2 linux/amd64 too.The program eats memory each time until crash when files are uploaded.

My core questions is "how to free memory immediately".

And my whole code is in this issue ,a go file and a html file.you can test it.

randall77 · 2015-03-27T17:55:17Z

I looked into this a little bit (with 1.3.3). I don't see quite the behavior you are.

I tried uploading the same 90MB file 10+ times. The heap grows to about 500MB over the first three or four uploads and then stops. After I stop uploading, GC eventually gets back to <1MB. It takes two background cycles to get there because of the concurrent sweep. You can speed that up by calling runtime.GC(), although in your example program there really isn't a good place to do that (you'd need it inside ListenAndServe after the http.Request is dropped).

We should ideally be able to handle this workload in ~256MB. I'm not entirely sure why it gets to 500MB.

meilihao · 2015-03-28T07:09:52Z

I am so sorry,my tests are not carefully enough on amd64.The program on amd64 is ok,not crash.

"r.ParseMultipartForm(128 << 20)" makes the heap grows to about 511MB and then stops.If not use it, the heap upper limit is 256MB. And the memory eventually gets back to <17MB after 9 Minutes.The long time is ok?

mikioh · 2015-05-07T01:59:24Z

Now we see this issue at tip: http://build.golang.org/log/d9c532d3a1b8266fae407518d131bc872f41a617.

alexbrainman · 2015-05-07T02:35:49Z

@mikioh please create new issue instead. Thank you. I suspect some tests use too much memory for windows/386.

Alex

mikioh · 2015-05-07T02:41:40Z

Filed #10737.

gopherbot added new labels Aug 25, 2014

bradfitz removed the new label Dec 18, 2014

rsc removed the os-windows label Apr 10, 2015

rsc added this to the Unplanned milestone Apr 10, 2015

rsc added OS-Windows and removed release-none labels Apr 10, 2015

mikioh changed the title ~~runtime: cannot map pages in arena address space on Windows~~ runtime: fatal error: cannot map pages in arena address space on Windows May 7, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: fatal error: cannot map pages in arena address space on Windows #8491

runtime: fatal error: cannot map pages in arena address space on Windows #8491

gopherbot commented Aug 7, 2014

ianlancetaylor commented Aug 7, 2014

bradfitz commented Aug 7, 2014

alexbrainman commented Aug 8, 2014

gopherbot commented Aug 12, 2014

gopherbot commented Aug 14, 2014

alexbrainman commented Aug 14, 2014

gopherbot commented Aug 14, 2014

gopherbot commented Aug 14, 2014

gopherbot commented Aug 14, 2014

alexbrainman commented Aug 14, 2014

gopherbot commented Aug 15, 2014

alexbrainman commented Aug 15, 2014

gopherbot commented Aug 18, 2014

gopherbot commented Aug 25, 2014

meilihao commented Mar 27, 2015

minux commented Mar 27, 2015 via email

meilihao commented Mar 27, 2015

randall77 commented Mar 27, 2015

meilihao commented Mar 28, 2015

mikioh commented May 7, 2015

alexbrainman commented May 7, 2015

mikioh commented May 7, 2015

runtime: fatal error: cannot map pages in arena address space on Windows #8491

runtime: fatal error: cannot map pages in arena address space on Windows #8491

Comments

gopherbot commented Aug 7, 2014

ianlancetaylor commented Aug 7, 2014

bradfitz commented Aug 7, 2014

alexbrainman commented Aug 8, 2014

gopherbot commented Aug 12, 2014

gopherbot commented Aug 14, 2014

alexbrainman commented Aug 14, 2014

gopherbot commented Aug 14, 2014

gopherbot commented Aug 14, 2014

gopherbot commented Aug 14, 2014

alexbrainman commented Aug 14, 2014

gopherbot commented Aug 15, 2014

alexbrainman commented Aug 15, 2014

gopherbot commented Aug 18, 2014

gopherbot commented Aug 25, 2014

meilihao commented Mar 27, 2015

minux commented Mar 27, 2015 via email

meilihao commented Mar 27, 2015

randall77 commented Mar 27, 2015

meilihao commented Mar 28, 2015

mikioh commented May 7, 2015

alexbrainman commented May 7, 2015

mikioh commented May 7, 2015