Sure enough. The goroutines don't even have to be doing anything. The cost is something like 25ns/goroutine on my laptop. Amusingly, I can cut that roughly in half just by putting gcAssistBytes on the same cache line as gcscandone and gcscanvalid. :)

I think we can just do this concurrently in gcStart before stopping the world. The downside of that is that it will still block allocation on Gs that go outside of their span cache, but that's better than blocking everything.

We could also make it just plain faster by pulling those fields out into their own contiguous arrays we can bulk zero.

We're paying the same per-goroutine cost in markroot during STW mark termination when we look for stacks to scan. That's trickier to avoid. We might be able to fix that by lifting the gcscanvalid check up in to markroot if we also move gcscanvalid in to a contiguous array.

@RLH, any thoughts?

Here's a simplified reproducer (I love the term "ballast", BTW):

package main

import (
    "os"
    "time"
)

const (
    ballastSize   = 100 << 20
    garbageSize   = 10 << 20
    garbagePeriod = 100 * time.Millisecond

    // This uses about 2GB of memory for stacks.
    numGs = 4e5
)

var (
    ballast []byte
    garbage []byte
)

func churn() {
    ballast = make([]byte, ballastSize)

    for {
        time.Sleep(garbagePeriod)
        garbage = make([]byte, garbageSize)
    }
}

func main() {
    for i := 0; i < numGs; i++ {
        go func() { select {} }()
    }

    go churn()

    time.AfterFunc(10*time.Second, func() { os.Exit(0) })
    select {}
}

Looks like I caused this performance degradation in b0d5e5c.

CL https://golang.org/cl/20147 mentions this issue.

CL https://golang.org/cl/22043 mentions this issue.