Skip to content

go/build: Quadratic complexity in Context.Import #63966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
catenacyber opened this issue Nov 6, 2023 · 5 comments
Closed

go/build: Quadratic complexity in Context.Import #63966

catenacyber opened this issue Nov 6, 2023 · 5 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed Unfortunate
Milestone

Comments

@catenacyber
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
go version go1.21 linux/amd64

Does this issue reproduce with the latest release?

It happens only on gotip, not on go 1.21

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/root/.go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/root/.go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.21"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/src/ngolo-fuzzing/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2481516251=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run https://go.dev/play/p/R_pJ6FL70Ok

What did you expect to see?

The program finishing and printing Hello

What did you see instead?

timeout running program

Found by https://github.com/catenacyber/ngolo-fuzzing with oss-fuzz :
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63873

This looks due to a quadratic complexity between the number of directories in GOPATH for the build.Context (which do not seem to get deduplicated) and the number of colon-separated entries in srcDir argument for https://pkg.go.dev/go/build@go1.21.3#Context.Import
The documentation does not seem to mention that srcDir gets interpreted as a colon-separated list of entries...
Is this a problem ?

@bcmills
Copy link
Contributor

bcmills commented Nov 6, 2023

Found by https://github.com/catenacyber/ngolo-fuzzing with oss-fuzz :
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63873

I appear not to have the permissions needed to view that issue. Can you summarize it here?

@bcmills
Copy link
Contributor

bcmills commented Nov 6, 2023

This looks due to a quadratic complexity between the number of directories in GOPATH for the build.Context (which do not seem to get deduplicated)

How would the GOPATH field of the build.Context end up set to a long list of invalid or duplicated paths in the first place? (What is the threat model that results in treating quadratic complexity here as a problem?)

and the number of colon-separated entries in srcDir argument for [Context.Import]
The documentation does not seem to mention that srcDir gets interpreted as a colon-separated list of entries...

I don't see how it would end up being treated as such a list — what leads you to believe that it does?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 6, 2023
@catenacyber
Copy link
Contributor Author

I appear not to have the permissions needed to view that issue. Can you summarize it here?

There is nothing more there beyond the reproducer https://go.dev/play/p/R_pJ6FL70Ok

How would the GOPATH field of the build.Context end up set to a long list of invalid or duplicated paths in the first place? (What is the threat model that results in treating quadratic complexity here as a problem?)

I am not sure there is an issue indeed... But it looks a bit fishy (I do not see any warning in the documentation, and this is the first time ngolo-fuzzing finds something for go/build)

I don't see how it would end up being treated as such a list — what leads you to believe that it does?

If the value gets shorter, we do not get the timeout any longer...
If GOPATH gets shorter, we do not get the timeout any longer...
So, it looks to me that there is a multiplying effect between both... but I did not check the code...

@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 6, 2023
@bcmills
Copy link
Contributor

bcmills commented Nov 6, 2023

So, it looks to me that there is a multiplying effect between both... but I did not check the code...

I would guess that the quadratic behavior comes from the calls to filepath.EvalSymlinks in hasSubdir, although of course I would want to see a profile or trace to confirm that guess.
(In my experience, calls to EvalSymlinks are usually not the best solution to problems, but in this case they've been there for a long time — I don't think we can just rip them out without a proposal and some analysis.)

Since we don't expect go/build to be used with an attacker-controlled GOPATH, I don't plan to make any changes here. We could probably accept a fix if someone wants to gather a profile or set up a benchmark and can identify a fix that isn't too invasive, but this probably isn't worth adding much (if any!) code complexity to address.

@bcmills bcmills added Unfortunate ToolSpeed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 6, 2023
@bcmills bcmills added this to the Unplanned milestone Nov 6, 2023
@catenacyber
Copy link
Contributor Author

I ran a profile.
Most time is spent in
go/build.(*Context).Import doing a first loop over gopath

for _, root := range gopath { 
    if searchVendor(root, false) { 

Then searchVendor looping over the srcDir being split over `/

for { 
    vendor := ctxt.joinPath(root, sub, "vendor") 
    if ctxt.isDir(vendor) { 
...
    i := strings.LastIndex(sub, "/")
    if i < 0 {
        break
    }					
    sub = sub[:i]

I am ok for closing this

@bcmills bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Nov 6, 2023
@golang golang locked and limited conversation to collaborators Nov 5, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed Unfortunate
Projects
None yet
Development

No branches or pull requests

3 participants