Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/net/ipv6: TestPacketConnReadWriteMulticast{UDP,ICMP} failing with "i/o timeout" on OpenBSD 6.8 and 7.0 #42064

Closed
bcmills opened this issue Oct 19, 2020 · 29 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 OS-OpenBSD release-blocker
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Oct 19, 2020

TestPacketConnReadWriteMulticastUDP is failing fairly consistently on the openbsd-arm-jsing builders. It should either be fixed or skipped.

--- FAIL: TestPacketConnReadWriteMulticastUDP (0.21s)
    multicast_test.go:114: read udp [::]:6423: raw-read udp6 [::]:6423: i/o timeout
FAIL
FAIL	golang.org/x/net/ipv6	0.294s

Note that this failure mode is different from the one on Darwin (#37319) and the one previously reported for OpenBSD (#33843).

2020-10-16T16:51:38-7b1cca2/openbsd-arm-jsing
2020-10-16T16:51:38-7b1cca2/openbsd-arm64-jsing
2020-10-10T22:47:23-4f7140c/openbsd-arm-jsing
2020-10-09T03:24:41-dbdefad/openbsd-arm-jsing
2020-10-08T22:28:04-59f7323/openbsd-arm-jsing
2020-10-06T15:34:59-a7d1128/openbsd-arm-jsing
2020-10-02T20:24:02-0a1ea39/openbsd-arm-jsing
2020-09-30T14:50:03-4acb6c0/openbsd-arm-jsing
2020-09-27T03:25:02-5d4f700/openbsd-arm-jsing
2020-09-25T08:00:53-05aa5d4/openbsd-arm-jsing
2020-09-23T18:22:12-328152d/openbsd-arm-jsing
2020-09-04T19:48:48-62affa3/openbsd-arm-jsing
2020-08-22T12:43:28-c890458/openbsd-arm-jsing
2020-08-13T13:45:08-3edf25e/openbsd-arm-jsing
2020-07-07T03:43:11-ab34263/openbsd-arm-jsing
2020-06-25T00:16:55-4c52546/openbsd-arm-jsing
2020-06-02T11:40:24-627f964/openbsd-arm-jsing
2020-05-28T22:51:25-3c3fba1/openbsd-arm-jsing
2020-05-20T18:23:14-0ba52f6/openbsd-arm-jsing

CC @4a6f656c @golang/osp-team

@bcmills bcmills added OS-OpenBSD NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 19, 2020
@gopherbot gopherbot added this to the Unreleased milestone Oct 19, 2020
@dmitshur dmitshur added the arch-arm Issues solely affecting the 32-bit arm architecture. label Oct 19, 2020
@tklauser
Copy link
Member

tklauser commented Dec 2, 2020

This test consistently fails on the openbsd-arm64-jsing and the 386/amd64 builders running OpenBSD 6.8 as well:

openbsd-arm64-jsing: https://build.golang.org/log/be54c3c8bfe2313b55b63b76bc5af92d6c39a524
openbsd-386-68: https://build.golang.org/log/eea81ada598c39d224d3a9a40b4739bbbe739b09
openbsd-amd64-68: https://build.golang.org/log/9039640721536372ca5170299ee6e923aa3775dd

The test also failed on https://golang.org/cl/274772 on the openbsd-mips64-jsing builder.

It doesn't seem to fail on builder running OpenBSD 6.2 or 6.4.

@dmitshur
Copy link
Contributor

dmitshur commented Dec 14, 2020

It doesn't seem to fail on builder running OpenBSD 6.2 or 6.4.

It'd be helpful to find out if the test runs or gets skipped on OpenBSD 6.4. (It's possible it's passing in 6.4 because IPv6 isn't supported there and it gets skipped.)

@bcmills bcmills changed the title x/net/ipv6: TestPacketConnReadWriteMulticastUDP failing with "i/o timeout" on openbsd-arm-jsing builders x/net/ipv6: TestPacketConnReadWriteMulticastUDP failing with "i/o timeout" on OpenBSD builders Dec 21, 2020
@bcmills bcmills removed the arch-arm Issues solely affecting the 32-bit arm architecture. label Dec 21, 2020
@bcmills
Copy link
Contributor Author

bcmills commented Dec 21, 2020

Marking as release-blocker via #11811, especially since this doesn't seem builder- or arch-specific.

But note that OpenBSD is not a first-class port, so it may be viable to unblock the release by skipping the test on the affected builders — especially given that this appears to be either a regression in the kernel or an incompatible ABI change.

@bcmills bcmills modified the milestones: Unreleased, Go1.16 Dec 21, 2020
@gopherbot
Copy link

Change https://golang.org/cl/279512 mentions this issue: dashboard: promote OpenBSD 6.8 builders to be default

gopherbot pushed a commit to golang/build that referenced this issue Dec 23, 2020
The known issue with OpenBSD 6.8 builders appears to be resolved
via CL 278732 and CL 279134. Promote them to the primary OpenBSD
post-submit builders and TryBots.

Having test coverage from OpenBSD 6.8 and 6.4 builders gives us
us more confidence that Go works on supported OpenBSD versions
(which are 6.8 and 6.7 at this time, per past policy decision
in https://golang.org/issue/15227#issuecomment-293319876).

Reduce numTryTestHelpers from 5 to 4 based on some data from
golang.org/issue/37439 showing that going from 3 to 5 helpers
doesn't make a significant difference. We can adjust further
if we learn that OpenBSD TryBots become the bottleneck.

Fixes golang/go#35712.
For golang/go#42542.
For golang/go#42064.
Updates golang/go#42426.

Change-Id: Id2fa4be7c3161f89752c1428146846fe06ca63db
Reviewed-on: https://go-review.googlesource.com/c/build/+/279512
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/280052 mentions this issue: ipv6: skip known-failing tests on OpenBSD 6.8 builders

gopherbot pushed a commit to golang/net that referenced this issue Dec 24, 2020
These tests are known to be failing on OpenBSD 6.8 and need
investigation. The OpenBSD 6.8 builders are default TryBots
now, so skip them to prevent people working on other packages
from constantly getting a failure.

Issue 42064 remains open to do the work of determining if the
ipv6 package needs to be fixed, or if OpenBSD 6.8 has a problem
and the test can be permanently skipped as unsupported.

For golang/go#11811.
Updates golang/go#42064.
Updates golang/go#35712.

Change-Id: I59883384e4dbce679cb1a4e8e2c3f0fa7983010b
Reviewed-on: https://go-review.googlesource.com/c/net/+/280052
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
@dmitshur dmitshur changed the title x/net/ipv6: TestPacketConnReadWriteMulticastUDP failing with "i/o timeout" on OpenBSD builders x/net/ipv6: TestPacketConnReadWriteMulticast{UDP,ICMP} failing with "i/o timeout" on OpenBSD 6.8 Dec 24, 2020
@dmitshur
Copy link
Contributor

dmitshur commented Dec 24, 2020

@4a6f656c Do openbsd-{arm,arm64,mips64}-jsing builders all run OpenBSD 6.8? If not, the issue title may need to be adjusted further.

@toothrot
Copy link
Contributor

toothrot commented Jan 7, 2021

@4a6f656c Just a friendly ping on this release-blocker in the new year. Do you have any updates?

@cagedmantis
Copy link
Contributor

@4a6f656c Have there been any updates with this issue. We are approaching the release and this is a release-blocker.

@tklauser
Copy link
Member

I quickly checked the arm64 and mips64 builders using gomote (the arm builder seems to be offline). They seem to be running OpenBSD 6.8 (uname -a output):

openbsd-arm64-jsing:
OpenBSD gopher-arm64.sing.id.au 6.8 GENERIC.MP#2 arm64

openbsd-mips64-jsing:
OS: OpenBSD er4.sing.id.au 6.8 GENERIC.MP#0 octeon

Assuming that the offline openbsd-arm-jsing builder also runs at least 6.7 (see #25498 (comment)) and that OpenBSD version might be affected the same, I've sent https://golang.org/cl/283752 to include openbsd-{arm,arm64,mips64}-jsing in the list of builders to skip the test on.

@gopherbot
Copy link

Change https://golang.org/cl/283752 mentions this issue: ipv6: skip known-failing tests on openbsd-{arm,arm64,mips64}-jsing builders

gopherbot pushed a commit to golang/net that referenced this issue Jan 19, 2021
…ilders

The arm64 and mips64 builder run OpenBSD 6.8:

openbsd-arm64-jsing:
OpenBSD gopher-arm64.sing.id.au 6.8 GENERIC.MP#2 arm64

openbsd-mips64-jsing:
OS: OpenBSD er4.sing.id.au 6.8 GENERIC.MP#0 octeon

The openbsd-arm-jsing builder was offline but according to
golang/go#25498 (comment)
it's running at least OpenBSD 6.7. Assuming that this version is
affected as well (or the builder was upgraded to 6.8 in the meantime),
skip TestPacketConnReadWriteMulticastUDP on this builder as well.

For golang/go#11811.
Updates golang/go#42064.

Change-Id: Ia9919057bfc2a69b8884228508ae919ce9d624e9
Reviewed-on: https://go-review.googlesource.com/c/net/+/283752
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Joel Sing <joel@sing.id.au>
@dmitshur
Copy link
Contributor

This is still a valid issue: the golang.org/x/net/ipv6 package has a problem on OpenBSD 6.8 and possibly 6.7, which may be a bug in Go code or a limitation/bug of OpenBSD itself. Work needs to be done to determine which, and the next steps.

The known-failing test is now skipped (for #11811), and this doesn't need to block the Go 1.16 release.

@dmitshur dmitshur modified the milestones: Go1.16, Unreleased Jan 21, 2021
@gopherbot
Copy link

Change https://golang.org/cl/314598 mentions this issue: [internal-branch.go1.16-vendor] ipv6: skip known-failing tests on openbsd-{arm,arm64,mips64}-jsing builders

@gopherbot
Copy link

Change https://golang.org/cl/314599 mentions this issue: [release-branch.go1.15] ipv6: skip known-failing tests on openbsd-{arm,arm64,mips64}-jsing builders

@gopherbot
Copy link

Change https://golang.org/cl/314749 mentions this issue: [internal-branch.go1.16-vendor] ipv6: skip known-failing tests on OpenBSD 6.8 builders

@bcmills bcmills modified the milestones: Unreleased, Go1.18 Nov 17, 2021
@bcmills bcmills added the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Nov 17, 2021
@bcmills bcmills changed the title x/net/ipv6: TestPacketConnReadWriteMulticast{UDP,ICMP} failing with "i/o timeout" on OpenBSD 6.8 x/net/ipv6: TestPacketConnReadWriteMulticast{UDP,ICMP} failing with "i/o timeout" on OpenBSD 6.8 and 7.0 Nov 17, 2021
@gopherbot
Copy link

Change https://golang.org/cl/366181 mentions this issue: ipv6: remove arbitrary deadlines from tests

@ianlancetaylor
Copy link
Contributor

Some notes from looking at this in a gomote.

When I run TestPacketConnReadWriteMulticastUDP by itself it always fails.

> go test -test.v -test.run=TestPacketConnReadWriteMulticastUDP
=== RUN   TestPacketConnReadWriteMulticastUDP
    multicast_test.go:120: read udp [::]:20756: raw-read udp6 [::]:20756: i/o timeout
--- FAIL: TestPacketConnReadWriteMulticastUDP (0.21s)
FAIL

When I simply run go test it pretty much always passes. The essential difference is that when I run plain go test, the test is skipped.

> go test -v
...
=== RUN   TestPacketConnReadWriteMulticastUDP
    multicast_test.go:48: not available on openbsd
...

The test is apparently being skipped because nettest.RoutedInterface is unable to find an interface that supports multicast loopback. But I don't yet understand why that changes based on whether I run other tests or not.

By the way, the test loops over addresses; the failure is always on the first address, [ff02::].

@ianlancetaylor
Copy link
Contributor

For #36435 we've converted the standard library syscall package to use OpenBSD library functions. However, that conversion has not been in golang.org/x/net/route. When I make that change, the test consistently fails when run with plain go test.

@ianlancetaylor
Copy link
Contributor

As far as I can tell these tests just don't work on OpenBSD. I haven't looked deeper into it but I'm guessing that they fail for the same reason that they fail on FreeBSD. I think the fact that they consistently fail on OpenBSD has been hidden by the fact that they are normally skipped on OpenBSD; when I correct the code that was incorrectly skipping the test, they consistently fail.

I will send a CL.

@gopherbot
Copy link

Change https://golang.org/cl/366354 mentions this issue: route, internal/socket: always go through syscall package

@gopherbot
Copy link

Change https://golang.org/cl/366355 mentions this issue: ipv6: OpenBSD does not appear to support multicast loopback

@gopherbot
Copy link

Change https://golang.org/cl/366195 mentions this issue: internal/socket: always go through syscall package

gopherbot pushed a commit to golang/net that referenced this issue Nov 23, 2021
Instead of relying on syscall.Syscall, always reach into the syscall
package to call the appropriate functions on Unix systems. We were
already doing this on Darwin. We also have to do this on
OpenBSD. Rather than sometimes reach into syscall and sometimes not,
just always reach in.

For golang/go#42064

Change-Id: Ie292a56766080d0c5ae6b6723d42b5475128290c
Reviewed-on: https://go-review.googlesource.com/c/net/+/366354
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
gopherbot pushed a commit to golang/net that referenced this issue Nov 23, 2021
Instead of relying on syscall.Syscall, always reach into the syscall
package to call the appropriate functions on Unix systems. We were
already doing this on Darwin and AIX. We also have to do this on
OpenBSD, and it's simpler to do it on Linux 386 and s390x. Rather than
sometimes reach into syscall and sometimes not, just always reach in.

For golang/go#42064

Change-Id: I0adb1c7cc623f2c1247465b3852fefd8d09975d2
Reviewed-on: https://go-review.googlesource.com/c/net/+/366195
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/366756 mentions this issue: vendor: update golang.org/x/net to tip

gopherbot pushed a commit that referenced this issue Nov 29, 2021
This brings in a fix for OpenBSD that lets it correctly gather network
interface information.

For #42064

Change-Id: Ib88fd2f494bb2ee86fd2725d8375b2df1404c4ca
Reviewed-on: https://go-review.googlesource.com/c/go/+/366756
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@neild
Copy link
Contributor

neild commented Dec 1, 2021

@gopherbot Please open backport issues for 1.16 and 1.17. This is causing trybot failures.

@gopherbot
Copy link

Backport issue(s) opened: #49908 (for 1.16), #49909 (for 1.17).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@gopherbot
Copy link

Change https://golang.org/cl/368534 mentions this issue: [internal-branch.go1.17-vendor] ipv6: OpenBSD does not appear to support multicast loopback

@gopherbot
Copy link

Change https://golang.org/cl/368514 mentions this issue: [internal-branch.go1.16-vendor] ipv6: OpenBSD does not appear to support multicast loopback

gopherbot pushed a commit to golang/net that referenced this issue Dec 1, 2021
…ort multicast loopback

Skip the tests for it, as we already skip them on FreeBSD.

For golang/go#42064
For golang/go#49908

Change-Id: I058f25309020367e686625033b4f25cb614c7f99
Reviewed-on: https://go-review.googlesource.com/c/net/+/366355
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-on: https://go-review.googlesource.com/c/net/+/368514
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
gopherbot pushed a commit to golang/net that referenced this issue Dec 1, 2021
…ort multicast loopback

Skip the tests for it, as we already skip them on FreeBSD.

For golang/go#42064
For golang/go#49909

Change-Id: I058f25309020367e686625033b4f25cb614c7f99
Reviewed-on: https://go-review.googlesource.com/c/net/+/366355
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-on: https://go-review.googlesource.com/c/net/+/368534
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
gopherbot pushed a commit to golang/net that referenced this issue Dec 6, 2021
These deadlines were added back in CL 21080043, apparently in an
attempt to increase code coverage numbers. However, nothing in the
tests actually exercises the deadline logic: the tests would pass even
if these methods were no-ops. Their only apparent effect is to make
the tests flaky on slower builders, and to destroy goroutine traces
if the test should ever happen to deadlock.

Updates golang/go#42064
For golang/go#37319

Change-Id: I530a8f3cb80d6d93d1625bc88f0ec7958d4ec35e
Reviewed-on: https://go-review.googlesource.com/c/net/+/366181
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@golang golang locked and limited conversation to collaborators Dec 1, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 OS-OpenBSD release-blocker
Projects
None yet
Development

No branches or pull requests

8 participants