Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: solaris builders are flaky #19369

Closed
bradfitz opened this issue Mar 2, 2017 · 2 comments
Closed

x/build: solaris builders are flaky #19369

bradfitz opened this issue Mar 2, 2017 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted OS-Solaris

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Mar 2, 2017

The solaris (smartos) builders are flaky.

The failures are often in TestServerAllowsBlockingRemoteAddr, as reproduced here:

$ gomote run user-bradfitz-solaris-amd64-smartosbuildlet-0 go/bin/go test -short net/http
--- FAIL: TestServerAllowsBlockingRemoteAddr (1.59s)
        serve_test.go:959: Request 1: Get http://127.0.0.1:38230: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
        serve_test.go:1008: response 1 addr = ""; want "RA:21.21.21.21:21"
2017/03/02 19:33:21 http2: server: error reading preface from client 127.0.0.1:50770: read tcp 127.0.0.1:57544->127.0.0.1:50770: use of closed file or network connection
FAIL
FAIL    net/http        7.934s

But running just that test (even 2000 times) is fine:

$ gomote run user-bradfitz-solaris-amd64-smartosbuildlet-0 go/bin/go test -short -run=TestServerAllowsBlockingRemoteAddr -count=2000 net/http
ok      net/http        7.161s

Note that the net/http tests only go passively parallel in -short mode, which results in a higher CPU rate. Are we getting throttled?

Notably, the tests pass when run without short:

$ gomote run user-bradfitz-solaris-amd64-smartosbuildlet-0 go/bin/go test net/http
ok      net/http        47.750s
$ gomote run user-bradfitz-solaris-amd64-smartosbuildlet-0 go/bin/go test net/http
ok      net/http        48.087s

Investigate.

/cc @4ad
@bradfitz bradfitz added Builders x/build issues (builders, bots, dashboards) help wanted OS-Solaris labels Mar 2, 2017
@bradfitz
Copy link
Contributor Author

bradfitz commented Mar 2, 2017

Oh, our 5 Joyent instances are "t4-standard-1G" which is 1G RAM, 1/2 CPU, 25G disk. Doesn't seem like enough CPU.

@bradfitz
Copy link
Contributor Author

bradfitz commented Mar 3, 2017

Quadrupled its CPU, wiped the red column, and things look happy now.

@bradfitz bradfitz closed this as completed Mar 3, 2017
@golang golang locked and limited conversation to collaborators Mar 3, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted OS-Solaris
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bradfitz @gopherbot