Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: restrict outbound network access from builds by default #30612

Closed
bradfitz opened this issue Mar 6, 2019 · 10 comments
Closed

x/build: restrict outbound network access from builds by default #30612

bradfitz opened this issue Mar 6, 2019 · 10 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Mar 6, 2019

We should make sure our tests never do outbound network requests by default, or that we only hit endpoints that we control.

For instance, the longtest builder is often failing now on:

--- FAIL: TestScript (0.00s)
    --- FAIL: TestScript/mod_gopkg_unstable (54.35s)
        script_test.go:188: 
            > env GO111MODULE=on
            > cp go.mod.empty go.mod
            > go get -d gopkg.in/dummy.v2-unstable
....
            go: golang.org/x/net@v0.0.0-20180906233101-161cd47e91fd: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in $WORK/gopath/pkg/mod/cache/vcs/4a22365141bc4eea5d5ac4a1395e653f2669485db75ef119e7bbec8e19b12a21: exit status 128:
            	fatal: remote error: Access denied to IP 35.193.215.129

... because we're hitting some Gerrit quota issue it seems.

I'm going to firewall outbound port 443 and port 22 connections for now and set the GOPROXY to our existing private Athens service on GKE to fix the longtest issue above.

/cc @dmitshur @bcmills

@bradfitz bradfitz self-assigned this Mar 6, 2019
@gopherbot gopherbot added this to the Unreleased milestone Mar 6, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Mar 6, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Mar 6, 2019

We should make sure our tests never do outbound network requests by default, or that we only hit endpoints that we control.

I agree.

Package cmd/go tests doing go get of import paths we don't control came up previously at least in #27692. In #27692 (comment), I said:

It might also be worth rethinking whether the test should depend on your repo tags not changing.

And the commit message of CL 153460 that resolved it by @bcmills said:

I think we ought to make these tests hermetic

Edit: Another related open issue is #28856.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/165637 mentions this issue: env, buildlet, dashboard: support outbound firewalls for Linux

gopherbot pushed a commit to golang/build that referenced this issue Mar 6, 2019
Updates golang/go#30612

Change-Id: Ib13a286d0944a7f4a13b9e93a01533693052858b
Reviewed-on: https://go-review.googlesource.com/c/build/+/165637
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/165618 mentions this issue: net/http: failing test for new outbound firewall

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/165638 mentions this issue: dashboard: enable outbound firewall on a few more Linux-based builders

gopherbot pushed a commit to golang/build that referenced this issue Mar 6, 2019
Updates golang/go#30612

Change-Id: Id472e753043976b8671c15491b82bb57e3804d95
Reviewed-on: https://go-review.googlesource.com/c/build/+/165638
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/165777 mentions this issue: kubernetes/gke: skip API tests in short mode

gopherbot pushed a commit to golang/build that referenced this issue Mar 6, 2019
Updates golang/go#30612

Change-Id: I752c57f8631025f1784951e701340d2c21ee277c
Reviewed-on: https://go-review.googlesource.com/c/build/+/165777
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/167039 mentions this issue: dashboard: set GOPROXY=off for the "go" repo to catch more network usage

gopherbot pushed a commit to golang/build that referenced this issue Mar 13, 2019
Updates golang/go#30612
Updates golang/go#30760

Change-Id: I296f4f7a163c1bdd59e839229c0961b21f26da89
Reviewed-on: https://go-review.googlesource.com/c/build/+/167039
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 2, 2019

This is sufficiently done. We do it on enough builders now that we catch network usage.

@bradfitz bradfitz closed this as completed Apr 2, 2019
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/192317 mentions this issue: dashboard: disable outbound network restriction

gopherbot pushed a commit to golang/build that referenced this issue Aug 29, 2019
Issue golang.org/issue/30612 implemented a restriction on outbound
network access to help catch unintended uses of internet in tests.

We're currently relying on GOPROXY=https://proxy.golang.org for running
tests in module mode in Go 1.13+, but proxy.golang.org can serve some
module zip files by 302 redirecting to other URLs. Those URLs may not
be accessible due to this builder network restriction.

Disable the outbound network restriction as the first step to return
builders to a working state. The next step will be implement a long
term solution, then we can re-enable the network restriction again.

Updates golang/go#33928
Updates golang/go#30612

Change-Id: I1a87b8d107df8806753117fc60a3e795e1e1b182
Reviewed-on: https://go-review.googlesource.com/c/build/+/192317
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur
Copy link
Contributor

This is temporarily undone by CL 192317, re-opening so we don't lose track of it.

@dmitshur dmitshur reopened this Aug 29, 2019
@julieqiu julieqiu added the NeedsFix The path to resolution is known, but the work has not been done. label Aug 29, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Sep 4, 2019

Resolved once again by CL 192341. /cc @bradfitz

@dmitshur dmitshur closed this as completed Sep 4, 2019
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
Issue golang.org/issue/30612 implemented a restriction on outbound
network access to help catch unintended uses of internet in tests.

We're currently relying on GOPROXY=https://proxy.golang.org for running
tests in module mode in Go 1.13+, but proxy.golang.org can serve some
module zip files by 302 redirecting to other URLs. Those URLs may not
be accessible due to this builder network restriction.

Disable the outbound network restriction as the first step to return
builders to a working state. The next step will be implement a long
term solution, then we can re-enable the network restriction again.

Updates golang/go#33928
Updates golang/go#30612

Change-Id: I1a87b8d107df8806753117fc60a3e795e1e1b182
Reviewed-on: https://go-review.googlesource.com/c/build/+/192317
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@golang golang locked and limited conversation to collaborators Sep 3, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

4 participants