Skip to content

x/vuln: govulncheck hangs when collecting stacks of all runtime usages #60708

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
karelbilek opened this issue Jun 9, 2023 · 7 comments
Closed
Assignees
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. vulncheck or vulndb Issues for the x/vuln or x/vulndb repo

Comments

@karelbilek
Copy link

karelbilek commented Jun 9, 2023

What version of Go are you using (go version)?

$ go version
go version go1.19.9 darwin/arm64

Does this issue reproduce at the latest version of golang.org/x/vuln?

Yes

What operating system and processor architecture are you using (go env)?

Darwin

What did you do?

I have a rather big project.

I was using go 1.19.9. Then, this vulnerability was published

https://pkg.go.dev/vuln/GO-2023-1840

When I run govulncheck ./..., it eats 8 cores for 5 minutes.

That is not normal. Go vulncheck should not take that much longer than code build. When before this vuln, it took seconds.

I know that I am using an old go version, which I did not notice. But then govulncheck should warn me, not hang.

Below is the actual issue in the govulncheck code.

When doing pprof, I figured out it's collecting all the traces where any function from runtime was used, and spending all the time on sorting slices.

The issue is at func CallStacks(res *Result) map[*Vuln][]CallStack {

It seems that the for _, vuln := range res.Vulns { is run 2052 times, and for each one, there is about 50000 of them for each.

When I made a small adjustment to the code, it fixed it instantly. That is, in witness.go - https://go.googlesource.com/vuln/+/refs/heads/master/internal/vulncheck/witness.go , instead of

		for _, cs := range callsites(f.CallSites, seen) {
			nStack := &callChain{f: cs.Parent, call: cs, child: c}
			if entries[cs.Parent] {
				stacks = append(stacks, nStack.CallStack())
			}
			queue.PushBack(nStack)
		}

I added (very dumb)

		for _, cs := range callsites(f.CallSites, seen) {
			nStack := &callChain{f: cs.Parent, call: cs, child: c}
			if entries[cs.Parent] {
				stacks = append(stacks, nStack.CallStack())
                                 if len(stacks)>10 {return stacks}
			}
			queue.PushBack(nStack)
		}

it worked well.

I think govulncheck is trying to get all the usages of all runtime functions, which then tries to somehow sort thousands of call stacks over and over. (2000 times getting and sorting 50.000 call stacks, I guess)

The simple fix is to just limit how many stacks will this collect. As the user still sees just 27 stacks....

What did you expect to see?

See above

What did you see instead?

See above

@karelbilek karelbilek added the vulncheck or vulndb Issues for the x/vuln or x/vulndb repo label Jun 9, 2023
@gopherbot gopherbot modified the milestones: Unreleased, vuln/unplanned Jun 9, 2023
@karelbilek karelbilek changed the title x/vuln: govulnchecks hangs when collecting runtime traces x/vuln: govulnchecks hangs when collecting stacks of all runtime usages Jun 9, 2023
@karelbilek karelbilek changed the title x/vuln: govulnchecks hangs when collecting stacks of all runtime usages x/vuln: govulncheck hangs when collecting stacks of all runtime usages Jun 9, 2023
@timothy-king
Copy link
Contributor

cc @zpavlinovic
There was also some relevant discussion here https://gophers.slack.com/archives/C0VPK4Z5E/p1686300492622229

@karelbilek
Copy link
Author

I don't think I'm the only one with this problem, I think it will also be present with old go 1.20 (but I cannot test now)

@dr2chase dr2chase added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 9, 2023
@dr2chase
Copy link
Contributor

dr2chase commented Jun 9, 2023

@golang/vulndb

@zpavlinovic
Copy link
Contributor

zpavlinovic commented Jun 9, 2023

Thanks for reporting this.

Indeed, govulncheck tries to find the "best" call stack for each reachable vulnerable symbol for better user experience. It does so by choosing from all "representative" call stacks of a symbol. If there are a lot of reachable vulnerable symbols, computing all representative call stacks for each such symbol might take time.

We should probably introduce some heuristic, in the lines of what you described, to counter examples such as GO-2023-1840 where all symbols of a package are deemed vulnerable, which should not happen too often.

@ianthehat @julieqiu

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504155 mentions this issue: internal/vulncheck: compute only one call stack for a symbol

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/504715 mentions this issue: internal/vulncheck: remove stdlib confidence in call stacks

gopherbot pushed a commit to golang/vuln that referenced this issue Jun 20, 2023
Call stacks going through standard library were designated as more
likely to be true positive, i.e., we have more confidence in them. This
does not seem to have a firm footing.

If the call stack consists of static calls, then the vulnerable symbol
is a std symbol, in which case comparison to other call stacks boils
down to length comparison (which we do) since standard library packages
import only standard library packages.

Otherwise if the call stacks have dynamic call sites, there is no reason
to believe that a callee of a dynamic site is more likely to be
correctly inferred compared to a dynamic site outside of the standard
library.

Updates golang/go#60708

Change-Id: Ib8cdd778ccf8902f5e42473fe145e452d28f3960
Reviewed-on: https://go-review.googlesource.com/c/vuln/+/504715
Run-TryBot: Zvonimir Pavlinovic <zpavlinovic@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Cottrell <iancottrell@google.com>
@zpavlinovic
Copy link
Contributor

Feel free to re-open the issue if the problem persists. We couldn't verify if it solves the problem on your code, but it did for another large project we considered.

@zpavlinovic zpavlinovic self-assigned this Jun 30, 2023
@golang golang locked and limited conversation to collaborators Jun 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. vulncheck or vulndb Issues for the x/vuln or x/vulndb repo
Projects
None yet
Development

No branches or pull requests

5 participants