FWIW, adding

defer func() { println(x) }()

at the top of func f in the test program makes the test pass (because then SSA knows x is needed at various points in the function).

Another variant on the test program, making sure that writes are flushed even when they're about to be overwritten, if code between the write and the overwrite might panic.

package main

import "runtime"

func f(x []byte, start int64) {
    defer func() {
        recover()
        if delta := inuse() - start; delta > 1<<20 {
            println("after alloc: expected delta below 1MB, got: ", delta)
        }
    }()
    if delta := inuse() - start; delta < 9<<20 {
        println("after alloc: expected delta at least 9MB, got: ", delta)
    }
    n1 := runtime.GOMAXPROCS(-1)
    n2 := runtime.GOMAXPROCS(-1)
    x = nil
    y := 1 / (n1 - n2)
    x = make([]byte, 10<<20)
    println(y)
}

func main() {
    x := inuse()
    f(make([]byte, 10<<20), x)
}

func inuse() int64 {
    runtime.GC()
    var st runtime.MemStats
    runtime.ReadMemStats(&st)
    return int64(st.Alloc)
}

https://go-review.googlesource.com/22365 is a start at this.
This CL illuminates two problems with the "keep value live until return" strategy.

The first is this:

func f(p *int) {
    p = nil
    foo()
}

So the nil value is the one we want to keep live until the end of the function. But nil is rematerializeable, so we will never generate a spill for it. Since there is no spill, we never get a chance to decide to spill back to p.

So spill nil anyway if it is known to be assigned to p, you say? Then what about

func g(p, q *int) {
    p = nil
    q = nil
    foo()
}

The nils get CSEd, so there's only one value to spill. It must get spilled to two places.

Long story short, this is tricky. Continuing to think about it.

Note that the defer trick Russ added does work, but only because it (implicitly) takes the address of the argument. I don't think we want to mark all input args as addrtaken to fix this problem.

By the way, we don't spill other variables to PARAM slots (any more). The only variables that can share a stack slot are two AUTO variables.

Here's another tricky case:

func f(a int, p *int) int {
    t := a
    a = *p
    g()
    return t + *p + a
}

The t := a assignment is a no-op for SSA, it just records that t takes on the value of a at the start of the function. The a=*p assignment turns into a load. The value loaded must be spilled across the g() call. SSA knows the original value of a is still live at this point, so it spills the loaded value to a temporary location. It is not until after the call that it issues the load from the argument section (to evaluate t) and the load from the temporary (to evaluate 'a').

Forcing the compiler to write back the value of a to the argument section at the call means we can't wait until after the call to load the original a value.

Not sure what to do about this either. Yet more trickiness...

Another fun example

type big [1 << 20]byte
type T struct {
    x, y *big
}
var p *big
func f(t T) *big {
    t = T{p, nil}
    p = nil
    runtime.GC()
    return t.x
}

t is decomposed by SSA into its individual parts. t.y is never used, so it isn't spilled. That leaves the old value of t.y on the stack when GC is called. (The legacy compiler overwrote the old value of t.y with nil.)

This is similar to the p = nil example I mentioned a few comments ago, but adds the decomposition problem to the mix.

Do we want a special gc.NilForEffect "value".

type big [1 << 20]byte
type T struct {
    x, y *big
}
var p *big
func f(t T) *big {
    t = T{p, gc.NilForEffect}
    p = nil
    runtime.GC()
    return t.x
}

NFE would be a minor pain to implement because of its interactions with CSE, register allocation, and deadcode, but at least we would be communicating our intent clearly. There's also need to be some sort of a "don't get cute" rule (a go vet check?) for use of gc.NilForEffect passed as a parameter or flowing into a conditional assignment.

For Go 1.7, what if you just treat nil specially? Always flush stores of nil to argument values before every function call. Seems like that would address 99% of the cases for 1.7, and for 1.8 we can perhaps not keep arguments live through the function.

This is not a maybe. This is a critical bug that needs to be fixed for Go 1.7.

I don't think we should special-case nils. There are other writes that could happen too.

What is the downside to marking all the arguments and results addrtaken for Go 1.7?

I just mailed a potential fix for this. It will need some discussion.

Basically, it implements runtime.KeepAlive internally and calls it for pointer arguments after each call and at each return.

Pointer arguments only, because those are the ones that matter for finalizers, and compound objects (slices, structs, ...) are a much more invasive change.

The only extra code this CL will end up generating is spills for otherwise dead values. It does not explicitly write back nil to argument slots, instead it marks the input argument slot as not live at the appropriate safe points. This requires turning off the "inputs are always live everywhere" code you will see commented out in the CL. We need to decide whether that code matters. (The comment says it is there to help with unnamed input parameters, not sure how much we care about those.)

CL https://golang.org/cl/22365 mentions this issue.

I am still seeing memory leaks in the production code from which my first test case was distilled. Here is a slightly different distillation that still demonstrates the leaks.

// run

// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package main

import "runtime"

type big [10 << 20]byte

func f(x interface{}, start int64) {
    x1, x := x, nil
    if delta := inuse() - start; delta < 9<<20 {
        println("after alloc: expected delta at least 9 MB, got", delta>>20, "MB")
    }
    g(x1)
    if delta := inuse() - start; delta > 1<<20 {
        println("after drop: expected delta below 1 MB, got", delta>>20, "MB")
    }
}

func main() {
    x := inuse()
    f(new(big), x)
}

func inuse() int64 {
    runtime.GC()
    var st runtime.MemStats
    runtime.ReadMemStats(&st)
    return int64(st.Alloc)
}

//go:noinline
func g(interface{}) {}

This program runs fine with go run -gcflags=-ssa=0 but with go run it complains about 10 MB in use when it expects 1 MB.

The code is trying to allocate a local variable x1 initialized from the parameter x and then nil out x, under the theory that, in contrast to x, the local variable will not be kept alive for the lifetime of the function. In particular, after the call to g, x1 is no longer needed and should stop being live, allowing the underlying allocation to be reclaimed. The theory works in the old back end.

In the SSA back end, there's a leak. Looking at the liveness analysis that runs on the SSA-generated assembly:

$ go tool compile -live /tmp/x.go
/tmp/x.go:13: live at entry to f: x
/tmp/x.go:15: live at call to inuse: x
/tmp/x.go:16: live at call to printlock: x
/tmp/x.go:16: live at call to printstring: x
/tmp/x.go:16: live at call to printsp: x
/tmp/x.go:16: live at call to printint: x
/tmp/x.go:16: live at call to printsp: x
/tmp/x.go:16: live at call to printstring: x
/tmp/x.go:16: live at call to printnl: x
/tmp/x.go:16: live at call to printunlock: x
/tmp/x.go:18: live at call to g: x
/tmp/x.go:19: live at call to inuse: x
/tmp/x.go:20: live at call to printlock: x
/tmp/x.go:20: live at call to printstring: x
/tmp/x.go:20: live at call to printsp: x
/tmp/x.go:20: live at call to printint: x
/tmp/x.go:20: live at call to printsp: x
/tmp/x.go:20: live at call to printstring: x
/tmp/x.go:20: live at call to printnl: x
/tmp/x.go:20: live at call to printunlock: x

All those live x are supposed to be OK because I set x to nil. But it appears that even though SSA needs x1 across the initial calls to inuse and printxxx, it does not put x1 itself on the stack. Instead it chooses not to nil x out and simply reload x1 from x later. I infer this from the fact that x1 must be saved somewhere yet does not appear in the liveness bitmaps, and therefore x1 is being reloaded from x rather than being saved. But that has the effect of never nilling out x, and then since x is live for the entire function, so is the allocation I'm trying to make available for garbage collection.

I don't believe we can reasonably drop the whole-function-lifetime liveness of function parameters at this point in the Go 1.7 cycle, but it's clear we probably need to do that at the start of the Go 1.8 cycle. This is too confusing. (On the other hand, I expect subtle keepalive problems in Go 1.8.)

I can fix my program by inserting _ = &x at the top of the function. That records x as having its address taken, at which point SSA is no longer comfortable assuming that x1 can be reloaded from x later in the function nor that the store of nil can be elided, and then all the "right" things start happening. I will do that, taking this off any kind of critical path and giving us a pattern to suggest to others who run into this.