Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: use frame pointer unwind to get backtrace in goroutine profile #66915

Open
zdyj3170101136 opened this issue Apr 19, 2024 · 7 comments
Open
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@zdyj3170101136
Copy link

Proposal Details

now, the goroutine profile use gopcinfotab to get backtrace.

if we use frame pointer, it would reduce lots of cost.

@gopherbot gopherbot added this to the Proposal milestone Apr 19, 2024
@randall77
Copy link
Contributor

Closing as dup of #16638

@zdyj3170101136
Copy link
Author

以#16638的重复结束

not a same question.

for now, the tracer use fp trace back.
but the goroutine profile do not use it.

@randall77
Copy link
Contributor

Ok, then I don't think I understand exactly what you are proposing. Could you be more explicit?

@randall77 randall77 reopened this Apr 19, 2024
@cherrymui cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 19, 2024
@cherrymui cherrymui changed the title proposal: runtime: use frame pointer unwind to get backtrace. proposal: runtime: use frame pointer unwind to get backtrace in goroutine profile Apr 19, 2024
@ianlancetaylor
Copy link
Contributor

I don't see any API change here, so I don't think this needs to be a proposal. Taking it out of the proposal process.

@ianlancetaylor ianlancetaylor changed the title proposal: runtime: use frame pointer unwind to get backtrace in goroutine profile runtime: use frame pointer unwind to get backtrace in goroutine profile Apr 19, 2024
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 19, 2024
@ianlancetaylor ianlancetaylor modified the milestones: Proposal, Backlog Apr 19, 2024
@zdyj3170101136
Copy link
Author

好吧,那么我想我不太明白你的建议。你能说得更明确一点吗?

for each goroutine, add a field fp:

type gobuf struct {
	// The offsets of sp, pc, and g are known to (hard-coded in) libmach.
	//
	// ctxt is unusual with respect to GC: it may be a
	// heap-allocated funcval, so GC needs to track it, but it
	// needs to be set and cleared from assembly, where it's
	// difficult to have write barriers. However, ctxt is really a
	// saved, live register, and we only ever exchange it between
	// the real register and the gobuf. Hence, we treat it as a
	// root during stack scanning, which means assembly that saves
	// and restores it doesn't need write barriers. It's still
	// typed as a pointer so that any other writes from Go get
	// write barriers.
	sp   uintptr
	pc   uintptr
        fp uintptr // new filed fp
	g    guintptr
	ctxt unsafe.Pointer
	ret  uintptr
	lr   uintptr
	bp   uintptr // for framepointer-enabled architectures
}

modify func runtime.save to save fp:

// save updates getg().sched to refer to pc and sp so that a following
// gogo will restore pc and sp.
//
// save must not have write barriers because invoking a write barrier
// can clobber getg().sched.
//
//go:nosplit
//go:nowritebarrierrec
func save(pc, sp, fp uintptr) {
	gp := getg()

	if gp == gp.m.g0 || gp == gp.m.gsignal {
		// m.g0.sched is special and must describe the context
		// for exiting the thread. mstart1 writes to it directly.
		// m.gsignal.sched should not be used at all.
		// This check makes sure save calls do not accidentally
		// run in contexts where they'd write to system g's.
		throw("save on system g not allowed")
	}

	gp.sched.pc = pc
	gp.sched.sp = sp
        gp.sched.fp = fp
	gp.sched.lr = 0
	gp.sched.ret = 0
	// We need to ensure ctxt is zero, but can't have a write
	// barrier here. However, it should always already be zero.
	// Assert that.
	if gp.sched.ctxt != nil {
		badctxt()
	}
}

after these change, user can use below code to get all goroutine's traceback:

package main

import (
	"fmt"
	"runtime"
	"sync/atomic"
	"time"
	"unsafe"
)

type stack struct {
	lo uintptr
	hi uintptr
}

type gobuf struct {
	sp   uintptr
	pc   uintptr
	fp   uintptr
	g    uintptr
	ctxt unsafe.Pointer
	ret  uintptr
	lr   uintptr
	bp   uintptr
}
type g struct {
	stack        stack
	stackguard0  uintptr
	stackguard1  uintptr
	_panic       uintptr
	_defer       uintptr
	m            uintptr
	sched        gobuf
	syscallsp    uintptr
	syscallpc    uintptr
	stktopsp     uintptr
	param        unsafe.Pointer
	atomicstatus atomic.Uint32
}

//go:linkname forEachGRace runtime.forEachG
func forEachGRace(fn func(gp *g)) bool

//go:linkname systemstack runtime.systemstack
func systemstack(fn func())

//go:noinline
func deepcall(n int) {
	if n > 0 {
		deepcall(n - 1)
	}
	time.Sleep(time.Second * 10)
}

func main() {
	max := 1000
	for i := 0; i < max; i++ {
		go deepcall(10)
	}
	fmt.Printf("got num of goroutine: %v\n", runtime.NumGoroutine())
	time.Sleep(time.Second)

	i := 0
	forEachGRace(func(gp *g) {
		if i == 0 {
			// skip current goroutine
			i++
			return
		}
		systemstack(func() {
			myfpTracebackPCs(unsafe.Pointer(gp.sched.fp), make([]uintptr, 32))
		})
	})
}

func myfpTracebackPCs(fp unsafe.Pointer, pcBuf []uintptr) (i int) {
	for i = 0; i < len(pcBuf) && fp != nil; i++ {
		// return addr sits one word above the frame pointer
		pcBuf[i] = *(*uintptr)(unsafe.Pointer(uintptr(fp) + PtrSize))
		prevfp := fp
		// follow the frame pointer to the next one
		fp = unsafe.Pointer(*(*uintptr)(fp))
		if uintptr(fp) <= uintptr(prevfp) {
			break
		}
	}
	return i

}

const PtrSize = 4 << (^uintptr(0) >> 63)

the benefit is the goroutine profile overhead would much enhancement.
those we could open goroutine profile continously in prod.

@nsrip-dd
Copy link
Contributor

I have a stack of CLs to bring frame pointer unwinding to the other runtime profilers, starting here: https://go-review.googlesource.com/c/go/+/543715. I intend to send a CL to use frame pointer unwinding for the goroutine profiler as well.

@zdyj3170101136
Copy link
Author

I have a stack of CLs to bring frame pointer unwinding to the other runtime profilers, starting here: https://go-review.googlesource.com/c/go/+/543715. I intend to send a CL to use frame pointer unwinding for the goroutine profiler as well.

amazing work!

may be the runtime could provide two type of api.

first is compatiable with current profile, which provide funcname, filename and line number.
the second is only provide address, and the symbolization could be run async in another program to reduce program overhead.
as far as i know, the software parca could extrace dwarf from executable and symbolization it in server.
see https://github.com/parca-dev/parca.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Status: No status
Development

No branches or pull requests

6 participants