I can reproduce

git clone https://github.com/leoleoasd/EduOJBackend
cd EduOJBackend
git checkout 0764cbfd0f8eaea90414c1e833651e8c961b64e5
GODEBUG=cgocheck=2 GOGC=2 go test -gcflags=-d=checkptr ./... -race

The data in your zombie object looks like gibberish to me: $2a$10$qHxkg64bA82YtgNI/l.3H.xEoZWMxOOotzRcZoFGmlMXVby1n3jAS. But in my crash, it contains test_get_run_comparer_output_fail_submit_user_0. So it is almost certainly the backing store of a string.
That string look familiar?

@randall77 $2a$10$qHxkg64bA82YtgNI/l.3H.xEoZWMxOOotzRcZoFGmlMXVby1n3jAS is output of a password hashing algorithm (bcrypt), and should be at user struct's password field

User with username test_get_run_comparer_output_fail_submit_user_0 is constructed at app/controller/submission_test.go

Checked every password used in tests, test_get_run_compiler_output_fail_user_0_pwd has the hash mentioned above.
User with that password is constructed app/controller/submission_test.go#L714

Some more info

Compile with

~/go1.16/bin/go test -c ./app/controller

Run with

GOGC=2 ./controller.test -test.run=GetRunCompilerOutput -test.count=10

Will fairly regularly crash with a bad pointer found in the stack frame of app/controller_test.TestGetRunCompilerOutput at offset 0x2f8.

It looks like the slot at 0x2f8 is initialized from the results of a Echo.Reverse call. But those calls only happen later in the function than where the failure occurs (Echo.Reverse calls are >= line 730, whereas the failure occurs when at line 722). Which makes me think the slot in question is not yet initialized but is somehow marked as live.

Still investigating.

This is a nasty compiler liveness bug.

The bug in the OP's code is in app/controller/submission_test.go:createSubmissionForTest. The user argument is never walked during GC, but is returned inside the result struct. Subsequent uses of user by the caller reference already-freed values.

I think a simple workaround would be to use unnamed return values. Use submission := ... and return submission. I think the bug can only happen for named return values.

Here's a reproducer.

package main

import (
	"runtime"
	"strings"
)

type T struct {
	str string
}

func main() {
	// Allocate something on the heap.
	str := strings.Repeat("x", 1024)

	// Allocate a stack object that points to str.
	t := &T{str: str}

	// Pass a pointer to the stack object to g.
	s := g(t)

	// At this point, neither str or t are statically live.
	// They should only be held onto by the reference in s.

	if s.t.str[0] != 'x' {
		panic("bad")
	}
}

type S struct {
	t *T
}

//go:noinline
func g(t *T) (s S) {
	s = S{t: t}

	// s escapes, so it is moved to the heap.
	escape(&s)

	// t is stored in the heap copy of s, but it is a stack pointer.
	// We need that t pointer to keep str alive, and we have to find it during
	// stack scanning. Having it live in the heap means the stack scan of this
	// goroutine will not find the stack object at *t, and hence not keep
	// str alive.
	runtime.GC() // This collects str!

	// At this return the heap copy of s is copied back to the stack, so main can
	// read it (and find t, even though we never scanned *t).
	return
}

//go:noinline
func escape(x interface{}) {
	sink = x
	sink = nil
}

var sink interface{}

$ GODEBUG=clobberfree=1 go run ~/gowork/issue44614.go
panic: bad

goroutine 1 [running]:
main.main()
	/Users/khr/gowork/issue44614.go:26 +0xb7
exit status 2

The problem here is that when we move a return value to the heap, it can still contain pointers into the stack. That's a big no-no, as we won't find any stack objects that those stack pointers point to.
I think this is just an escape analysis bug. Making t escape should fix it.
@mdempsky

This bug appears from 1.14 onwards. 1.13 and earlier are unaffected.

@gopherbot please open backport issues.

Backport issue(s) opened: #44658 (for 1.15), #44659 (for 1.16).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

Thanks a lot! @randall77

@randall77 Thanks for minimizing the reproducer. I agree this looks like an escape analysis issue.

I think the problem is in location.leakTo, that if a parameter flows to a result parameter that itself needs to be heap allocated, we need to record that as a heap leak rather than a result leak. We probably just need to include !sink.escapes in that check.

Thank you so much @randall77

Further minimized repro:

$ cat test.go
package p

func f(p *int) (r *int) {
	sink = &r
	return p
}

var sink interface{}

$ go tool compile -m -l test.go
test.go:3:8: leaking param: p to result r level=0
test.go:3:17: moved to heap: r

It should be simply leaking param: p without to result r.

I suspect the culprit CL that caused the regression in Go 1.14 is 2620467.

Change https://golang.org/cl/297289 mentions this issue: cmd/compile: fix escape analysis of heap-allocated results