Text file src/runtime/HACKING.md

Documentation: runtime

     1This is a living document and at times it will be out of date. It is
     2intended to articulate how programming in the Go runtime differs from
     3writing normal Go. It focuses on pervasive concepts rather than
     4details of particular interfaces.
     6Scheduler structures
     9The scheduler manages three types of resources that pervade the
    10runtime: Gs, Ms, and Ps. It's important to understand these even if
    11you're not working on the scheduler.
    13Gs, Ms, Ps
    16A "G" is simply a goroutine. It's represented by type `g`. When a
    17goroutine exits, its `g` object is returned to a pool of free `g`s and
    18can later be reused for some other goroutine.
    20An "M" is an OS thread that can be executing user Go code, runtime
    21code, a system call, or be idle. It's represented by type `m`. There
    22can be any number of Ms at a time since any number of threads may be
    23blocked in system calls.
    25Finally, a "P" represents the resources required to execute user Go
    26code, such as scheduler and memory allocator state. It's represented
    27by type `p`. There are exactly `GOMAXPROCS` Ps. A P can be thought of
    28like a CPU in the OS scheduler and the contents of the `p` type like
    29per-CPU state. This is a good place to put state that needs to be
    30sharded for efficiency, but doesn't need to be per-thread or
    33The scheduler's job is to match up a G (the code to execute), an M
    34(where to execute it), and a P (the rights and resources to execute
    35it). When an M stops executing user Go code, for example by entering a
    36system call, it returns its P to the idle P pool. In order to resume
    37executing user Go code, for example on return from a system call, it
    38must acquire a P from the idle pool.
    40All `g`, `m`, and `p` objects are heap allocated, but are never freed,
    41so their memory remains type stable. As a result, the runtime can
    42avoid write barriers in the depths of the scheduler.
    44User stacks and system stacks
    47Every non-dead G has a *user stack* associated with it, which is what
    48user Go code executes on. User stacks start small (e.g., 2K) and grow
    49or shrink dynamically.
    51Every M has a *system stack* associated with it (also known as the M's
    52"g0" stack because it's implemented as a stub G) and, on Unix
    53platforms, a *signal stack* (also known as the M's "gsignal" stack).
    54System and signal stacks cannot grow, but are large enough to execute
    55runtime and cgo code (8K in a pure Go binary; system-allocated in a
    56cgo binary).
    58Runtime code often temporarily switches to the system stack using
    59`systemstack`, `mcall`, or `asmcgocall` to perform tasks that must not
    60be preempted, that must not grow the user stack, or that switch user
    61goroutines. Code running on the system stack is implicitly
    62non-preemptible and the garbage collector does not scan system stacks.
    63While running on the system stack, the current user stack is not used
    64for execution.
    66`getg()` and `getg().m.curg`
    69To get the current user `g`, use `getg().m.curg`.
    71`getg()` alone returns the current `g`, but when executing on the
    72system or signal stacks, this will return the current M's "g0" or
    73"gsignal", respectively. This is usually not what you want.
    75To determine if you're running on the user stack or the system stack,
    76use `getg() == getg().m.curg`.
    78Error handling and reporting
    81Errors that can reasonably be recovered from in user code should use
    82`panic` like usual. However, there are some situations where `panic`
    83will cause an immediate fatal error, such as when called on the system
    84stack or when called during `mallocgc`.
    86Most errors in the runtime are not recoverable. For these, use
    87`throw`, which dumps the traceback and immediately terminates the
    88process. In general, `throw` should be passed a string constant to
    89avoid allocating in perilous situations. By convention, additional
    90details are printed before `throw` using `print` or `println` and the
    91messages are prefixed with "runtime:".
    93For runtime error debugging, it's useful to run with
    94`GOTRACEBACK=system` or `GOTRACEBACK=crash`.
    99The runtime has multiple synchronization mechanisms. They differ in
   100semantics and, in particular, in whether they interact with the
   101goroutine scheduler or the OS scheduler.
   103The simplest is `mutex`, which is manipulated using `lock` and
   104`unlock`. This should be used to protect shared structures for short
   105periods. Blocking on a `mutex` directly blocks the M, without
   106interacting with the Go scheduler. This means it is safe to use from
   107the lowest levels of the runtime, but also prevents any associated G
   108and P from being rescheduled. `rwmutex` is similar.
   110For one-shot notifications, use `note`, which provides `notesleep` and
   111`notewakeup`. Unlike traditional UNIX `sleep`/`wakeup`, `note`s are
   112race-free, so `notesleep` returns immediately if the `notewakeup` has
   113already happened. A `note` can be reset after use with `noteclear`,
   114which must not race with a sleep or wakeup. Like `mutex`, blocking on
   115a `note` blocks the M. However, there are different ways to sleep on a
   116`note`:`notesleep` also prevents rescheduling of any associated G and
   117P, while `notetsleepg` acts like a blocking system call that allows
   118the P to be reused to run another G. This is still less efficient than
   119blocking the G directly since it consumes an M.
   121To interact directly with the goroutine scheduler, use `gopark` and
   122`goready`. `gopark` parks the current goroutine—putting it in the
   123"waiting" state and removing it from the scheduler's run queue—and
   124schedules another goroutine on the current M/P. `goready` puts a
   125parked goroutine back in the "runnable" state and adds it to the run
   128In summary,
   131<tr><th></th><th colspan="3">Blocks</th></tr>
   141The runtime uses its own atomics package at `runtime/internal/atomic`.
   142This corresponds to `sync/atomic`, but functions have different names
   143for historical reasons and there are a few additional functions needed
   144by the runtime.
   146In general, we think hard about the uses of atomics in the runtime and
   147try to avoid unnecessary atomic operations. If access to a variable is
   148sometimes protected by another synchronization mechanism, the
   149already-protected accesses generally don't need to be atomic. There
   150are several reasons for this:
   1521. Using non-atomic or atomic access where appropriate makes the code
   153   more self-documenting. Atomic access to a variable implies there's
   154   somewhere else that may concurrently access the variable.
   1562. Non-atomic access allows for automatic race detection. The runtime
   157   doesn't currently have a race detector, but it may in the future.
   158   Atomic access defeats the race detector, while non-atomic access
   159   allows the race detector to check your assumptions.
   1613. Non-atomic access may improve performance.
   163Of course, any non-atomic access to a shared variable should be
   164documented to explain how that access is protected.
   166Some common patterns that mix atomic and non-atomic access are:
   168* Read-mostly variables where updates are protected by a lock. Within
   169  the locked region, reads do not need to be atomic, but the write
   170  does. Outside the locked region, reads need to be atomic.
   172* Reads that only happen during STW, where no writes can happen during
   173  STW, do not need to be atomic.
   175That said, the advice from the Go memory model stands: "Don't be
   176[too] clever." The performance of the runtime matters, but its
   177robustness matters more.
   179Unmanaged memory
   182In general, the runtime tries to use regular heap allocation. However,
   183in some cases the runtime must allocate objects outside of the garbage
   184collected heap, in *unmanaged memory*. This is necessary if the
   185objects are part of the memory manager itself or if they must be
   186allocated in situations where the caller may not have a P.
   188There are three mechanisms for allocating unmanaged memory:
   190* sysAlloc obtains memory directly from the OS. This comes in whole
   191  multiples of the system page size, but it can be freed with sysFree.
   193* persistentalloc combines multiple smaller allocations into a single
   194  sysAlloc to avoid fragmentation. However, there is no way to free
   195  persistentalloced objects (hence the name).
   197* fixalloc is a SLAB-style allocator that allocates objects of a fixed
   198  size. fixalloced objects can be freed, but this memory can only be
   199  reused by the same fixalloc pool, so it can only be reused for
   200  objects of the same type.
   202In general, types that are allocated using any of these should be
   203marked `//go:notinheap` (see below).
   205Objects that are allocated in unmanaged memory **must not** contain
   206heap pointers unless the following rules are also obeyed:
   2081. Any pointers from unmanaged memory to the heap must be garbage
   209   collection roots. More specifically, any pointer must either be
   210   accessible through a global variable or be added as an explicit
   211   garbage collection root in `runtime.markroot`.
   2132. If the memory is reused, the heap pointers must be zero-initialized
   214   before they become visible as GC roots. Otherwise, the GC may
   215   observe stale heap pointers. See "Zero-initialization versus
   216   zeroing".
   218Zero-initialization versus zeroing
   221There are two types of zeroing in the runtime, depending on whether
   222the memory is already initialized to a type-safe state.
   224If memory is not in a type-safe state, meaning it potentially contains
   225"garbage" because it was just allocated and it is being initialized
   226for first use, then it must be *zero-initialized* using
   227`memclrNoHeapPointers` or non-pointer writes. This does not perform
   228write barriers.
   230If memory is already in a type-safe state and is simply being set to
   231the zero value, this must be done using regular writes, `typedmemclr`,
   232or `memclrHasPointers`. This performs write barriers.
   234Runtime-only compiler directives
   237In addition to the "//go:" directives documented in "go doc compile",
   238the compiler supports additional directives only in the runtime.
   243`go:systemstack` indicates that a function must run on the system
   244stack. This is checked dynamically by a special function prologue.
   249`go:nowritebarrier` directs the compiler to emit an error if the
   250following function contains any write barriers. (It *does not*
   251suppress the generation of write barriers; it is simply an assertion.)
   253Usually you want `go:nowritebarrierrec`. `go:nowritebarrier` is
   254primarily useful in situations where it's "nice" not to have write
   255barriers, but not required for correctness.
   257go:nowritebarrierrec and go:yeswritebarrierrec
   260`go:nowritebarrierrec` directs the compiler to emit an error if the
   261following function or any function it calls recursively, up to a
   262`go:yeswritebarrierrec`, contains a write barrier.
   264Logically, the compiler floods the call graph starting from each
   265`go:nowritebarrierrec` function and produces an error if it encounters
   266a function containing a write barrier. This flood stops at
   267`go:yeswritebarrierrec` functions.
   269`go:nowritebarrierrec` is used in the implementation of the write
   270barrier to prevent infinite loops.
   272Both directives are used in the scheduler. The write barrier requires
   273an active P (`getg().m.p != nil`) and scheduler code often runs
   274without an active P. In this case, `go:nowritebarrierrec` is used on
   275functions that release the P or may run without a P and
   276`go:yeswritebarrierrec` is used when code re-acquires an active P.
   277Since these are function-level annotations, code that releases or
   278acquires a P may need to be split across two functions.
   283`go:notinheap` applies to type declarations. It indicates that a type
   284must never be allocated from the GC'd heap. Specifically, pointers to
   285this type must always fail the `runtime.inheap` check. The type may be
   286used for global variables, for stack variables, or for objects in
   287unmanaged memory (e.g., allocated with `sysAlloc`, `persistentalloc`,
   288`fixalloc`, or from a manually-managed span). Specifically:
   2901. `new(T)`, `make([]T)`, `append([]T, ...)` and implicit heap
   291   allocation of T are disallowed. (Though implicit allocations are
   292   disallowed in the runtime anyway.)
   2942. A pointer to a regular type (other than `unsafe.Pointer`) cannot be
   295   converted to a pointer to a `go:notinheap` type, even if they have
   296   the same underlying type.
   2983. Any type that contains a `go:notinheap` type is itself
   299   `go:notinheap`. Structs and arrays are `go:notinheap` if their
   300   elements are. Maps and channels of `go:notinheap` types are
   301   disallowed. To keep things explicit, any type declaration where the
   302   type is implicitly `go:notinheap` must be explicitly marked
   303   `go:notinheap` as well.
   3054. Write barriers on pointers to `go:notinheap` types can be omitted.
   307The last point is the real benefit of `go:notinheap`. The runtime uses
   308it for low-level internal structures to avoid memory barriers in the
   309scheduler and the memory allocator where they are illegal or simply
   310inefficient. This mechanism is reasonably safe and does not compromise
   311the readability of the runtime.

View as plain text