Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: BenchmarkTCP{4,6}ConcurrentReadWrite failures on freebsd-amd64-race with "read: connection reset by peer" #45214

Closed
bcmills opened this issue Mar 24, 2021 · 15 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-FreeBSD release-blocker
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Mar 24, 2021

2021-03-23T05:12:12-cd26192/freebsd-amd64-race
2021-03-23T04:23:52-0265b64/freebsd-amd64-race

(See also #20960.)

CC @bradfitz @ianlancetaylor

These failures are both very recent, so I wonder if something has recently changed in the net package or the freebsd-amd64-race builders. (CC @golang/release)

@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-FreeBSD labels Mar 24, 2021
@bcmills bcmills added this to the Backlog milestone Mar 24, 2021
@bcmills
Copy link
Contributor Author

bcmills commented Mar 24, 2021

#19229 also features connection reset by fail errors on FreeBSD. I wonder if it's related...

@ayang64
Copy link
Member

ayang64 commented Mar 25, 2021

which version of freebsd? i can't reproduce on FreeBSD kiwi 14.0-CURRENT FreeBSD 14.0-CURRENT #5 main-n245559-d22883d71544: Sun Mar 21 16:51:44 EDT 2021 root@kiwi:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64

@ayang64
Copy link
Member

ayang64 commented Mar 25, 2021

also, are you running freebsd in a virtual machine for tests?

@bcmills
Copy link
Contributor Author

bcmills commented Mar 25, 2021

Looks like it's FreeBSD 12.0 of some flavor, on a GCE n1-highcpu-16 VM:
https://github.com/golang/build/blob/d7600ec889c89d486672e9ccd1b30bab50c15047/dashboard/builders.go#L317-L324

@golang/release could probably get you more details.

@bcmills
Copy link
Contributor Author

bcmills commented Mar 25, 2021

The failure rate is not particularly high, so you may also need a lot of runs to reproduce the failure.

@ayang64
Copy link
Member

ayang64 commented Mar 25, 2021

okay -- i just ran it on my local machine with -count 100 without any failures. i'll continue trying to reproduce and maybe try to isolate the commit that caused it. i'll also try under 12.0.

@bcmills
Copy link
Contributor Author

bcmills commented Mar 25, 2021

There's one more in the logs today:
2021-03-25T00:15:42-80157b5/freebsd-amd64-race

That's three in two days, and none that I can find before that. So the regression is probably very recent, whatever it is.

@bcmills bcmills changed the title net: BenchmarkTCP4ConcurrentReadWrite failures on freebsd-amd64-race with "read: connection reset by peer" net: BenchmarkTCP{4,6}ConcurrentReadWrite failures on freebsd-amd64-race with "read: connection reset by peer" Apr 15, 2021
@bcmills bcmills modified the milestones: Backlog, Go1.17 Apr 15, 2021
@ayang64
Copy link
Member

ayang64 commented Apr 15, 2021

i still can't reproduce using 12.2-RELEASE-p6.

it looks like a likely culprit is commit 6694a6888 though.

@ayang64
Copy link
Member

ayang64 commented Apr 15, 2021

maybe grasping at straws but what if you update the test image to the latest release in the freebsd 12 line?

it seems like that'd be a good start. it looks like you're running 12.0 but the latest production release is 12.2.

we have the following errata that might cause this issue for which the fix is to update the kernel version:

https://www.freebsd.org/security/advisories/FreeBSD-EN-19:02.tcp.asc

i iust ran 100 iterations of the benchmark on a 16 core GCP instance running FreeBSD 12.2 without errors.

it seems reasonable to try the latest stable release in the 12 line instead of 12.0.

@toothrot
Copy link
Contributor

We're working on updating the builders soon in #44431.
/cc @cagedmantis

@ayang64
Copy link
Member

ayang64 commented Apr 26, 2021

looks like the default builder has been updated to 12.2. is there a way to see the results?

@dmitshur
Copy link
Contributor

The new builder will be used in future testing (existing commit with test results are not automatically re-tested at this time), so its results will show up when new commits are pushed, or by explicitly sending a (possibly empty) change and requesting it via TRY=freebsd-amd64-race SlowBot syntax.

@ayang64
Copy link
Member

ayang64 commented May 6, 2021

has this error returned? it looks like the freebsd builders have been relatively happy:

https://build.golang.org/

@toothrot
Copy link
Contributor

toothrot commented May 6, 2021

There doesn't seem to be any recent occurrences. Closing this issue.

@toothrot toothrot closed this as completed May 6, 2021
@golang golang locked and limited conversation to collaborators May 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-FreeBSD release-blocker
Projects
None yet
Development

No branches or pull requests

5 participants