Related: #28081 (large core files)

CC @aclements

Presumably related to https://golang.org/cl/85888 and #10460.

Interesting. I had thought that mlocking memory would only lock pages from being paged out, but wouldn't eagerly fault the pages in. But looking more closely at the manpage, it looks like that's only true with mlock2 with the MLOCK_ONFAULT flag or mlockall with the MCL_ONFAULT flag. I assume you're using mlockall? MCL_ONFAULT has only been available since Linux 4.4, but would that be a reasonable workaround, at least for now?

The solution I proposed in #28081 should also fix this.

More discussion: https://news.ycombinator.com/item?id=18184541

Thanks for the hint @aclements.

Indeed, couple of my system programs are using mlockall, often with both syscall.MCL_CURRENT | syscall.MCL_FUTURE. For this issue report, MCL_CURRENT alone is sufficient:

package main

import "syscall"

func main() {
        if err := syscall.Mlockall(syscall.MCL_CURRENT); err != nil {
                panic(err)
        }
        for {
        }
}

As observed by host OS, it appears to have indeed copied all of the allocated virtual memory into main memory:

root 29281 156 2.5 101836 101736 pts/0 RLl+ 05:27 0:01 ./usagetest

Hehe also I was wondering why this issue attracted so many emojis, it turns out someone put it on hackernews..

Yeah, I had just upgraded to 1.11 while not noticing anything significant in the release notes (at that time). And just after that, I opened hacker news, and this was the top result. Imagine the panic! (pun intended)

@HouzuoGuo, yes, that's what I would expect. Thanks for confirming.

I don't really understand when it makes sense to use mlockall instead of mlock. There's a ton of stuff in memory that certainly doesn't have to be locked, and using mlock for just the things that need to be locked wouldn't have the problem with virtual memory. What's the reason for using mlockall instead of mlock? And if you do need mlockall, is there a reason not to use MCL_ONFAULT?

Even if we do modify arena mapping to be incremental (which I believe should be relatively easy to do), I don't want to change how the arena index is mapped because that would impact both performance and simplicity. So mlockall would still fault in about 32MB, even with the change to arena mapping.

Thanks @aclements , TIL about MCL_ONFAULT, apparently available since Linux 4.4. I shall use it from now and onward.

Beyond the scope of this issue report, would you please offer some hints on the proper invocation of mlock in protecting a sensitive instance of structure, especially to determine its memory address range? Take this structure for example

type Daemon struct {
    ListenAddress string
    ListenPort int
    InternalData map[string]interface{}
}

Folding this issue into #28081, which I've re-titled, since the root cause is the same.