Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: upgrade FreeBSD builders to avoid flakes #25289

Closed
ianlancetaylor opened this issue May 8, 2018 · 17 comments
Closed

x/build: upgrade FreeBSD builders to avoid flakes #25289

ianlancetaylor opened this issue May 8, 2018 · 17 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-FreeBSD
Milestone
Unplanned

Comments

@ianlancetaylor
Copy link
Contributor

2018-05-01T14:50:33-7f6105f/freebsd-386-11_1
2018-05-01T18:22:06-a5b80d4/freebsd-amd64-11_1
2018-05-02T17:52:14-e8d417d/freebsd-amd64-11_1
2018-05-04T14:47:49-166c37a/freebsd-386-11_1
2018-05-05T00:53:45-2a7e19e/freebsd-amd64-11_1
2018-05-05T00:54:52-f7c767e/freebsd-amd64-11_1
2018-05-06T04:28:23-a8a60ac/freebsd-386-10_3
2018-05-08T13:29:22-b00f72e/freebsd-386-11_1

findflakes reports

First observed 7f6105f 01 May 14:50 2018 (92 commits ago)
Last observed  b00f72e 08 May 13:29 2018 (0 commits ago)
100% chance failure is still happening
7.6% failure probability (8 of 93 commits)
Likely culprits:
    8% 7f6105f net, syscall: make ECONNRESET/ECONNABORTED only temporary for Accept
    7% 1129ec2 database/sql: trivial documentation fixes
    6% 60c76f7 cmd/compile: optimize bvec routines
    6% d91e970 runtime: avoid unnecessary scanblock calls
    6% dda4591 runtime: add BenchmarkScanStack
    5% 9ecf899 cmd: remove some unnecessary gotos
    5% 030ac2c cmd/doc: skip directories like other go tools
    4% 8c67ca1 cmd/compile: apply TODO now that gcCompat is gone
    4% 16d1a8e runtime: move open/close/read/write from syscall to libc on Darwin
    4% ce5c387 runtime: move more syscalls to libc on Darwin
No known past failures
@ianlancetaylor ianlancetaylor added OS-FreeBSD NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 8, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone May 8, 2018
@tklauser tklauser mentioned this issue May 22, 2018
Closed
@ayang64
Copy link
Member

ayang64 commented Jun 24, 2018

I just ran this test on 12-CURRENT and 11.1-RELEASE-p4 and it passed.

@ianlancetaylor
Copy link
Contributor Author

@ayang64 Thanks. To be clear, the test normally passes. As the message above suggests, it is only failing about 7.6% of the time. How many times did you run the test?

@ayang64
Copy link
Member

ayang64 commented Jun 27, 2018

I just tried 10000 times under 12-CURRNET (02bd6c4) and 12.0-CURRENT (58e53e526dc) and 500 under 11.1-RELEASE-p4 with no failures.

@ianlancetaylor
Copy link
Contributor Author

Great, thanks!

Changing this issue to upgrade our builders.

@ianlancetaylor ianlancetaylor changed the title net: flakes in TestNotTemporaryRead on FreeBSD builders: upgrade FreeBSD builders to avoid flakes Jun 27, 2018
@ianlancetaylor ianlancetaylor added NeedsFix The path to resolution is known, but the work has not been done. and removed release-blocker labels Jun 27, 2018
@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Unplanned Jun 27, 2018
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 27, 2018
@ianlancetaylor
Copy link
Contributor Author

CC @bradfitz

@bcmills bcmills mentioned this issue Aug 3, 2018
Closed
@bradfitz bradfitz changed the title builders: upgrade FreeBSD builders to avoid flakes x/build: upgrade FreeBSD builders to avoid flakes Aug 3, 2018
@gopherbot
Copy link

Change https://golang.org/cl/127837 mentions this issue: net: skip flaky TestNotTemporaryRead on FreeBSD

gopherbot pushed a commit that referenced this issue Aug 3, 2018
@bradfitz
net: skip flaky TestNotTemporaryRead on FreeBSD
1b87007 
Updates #25289

Change-Id: I662760b921be625aca988cd0b43c648ac5dfd814
Reviewed-on: https://go-review.googlesource.com/127837
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/135475 mentions this issue: env/freebsd-amd64: add FreeBSD 10.4, 11.2 images, update build script

@paulzhol paulzhol added the Builders x/build issues (builders, bots, dashboards) label Sep 22, 2018
@bradfitz
Copy link
Contributor

FreeBSD people, is it reasonable to take down our 10.3 and 11.1 builders and replace them with 10.4 or 11.2? I'd rather not double the number of FreeBSD builder's we're running, but we could if necessary. I don't know what versions are important/commonly used in practice.

/cc @dmitshur

gopherbot pushed a commit to golang/build that referenced this issue Sep 27, 2018
@paulzhol @bradfitz
env/freebsd-amd64: add FreeBSD 10.4, 11.2 images, update build script
2e001a4 
Use freebsd-update when preparing the downloaded -RELEASE image to get errata and security
updates made since the release.

Switch kern.timecounter.hardware from ACPI-safe to TSC on GCE
https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/000080.html

Drop vtnet hw.vtnet.csum_disable=1 workaround, should be fixed in all supported versions.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207446

Update the expect script to switch the console to serial, this makes it possible to match
prompts via a regexp (11.2 has 'root@freebsd:~ #', while the older images are 'root@:~ #').
Otherwise qemu with -display curses emits ansi escape codes which make a mess of everything.
This requires the sgabios smbios extension installed on the host preparing the images.
https://code.google.com/archive/p/sgabios/
https://packages.ubuntu.com/xenial/sgabios
SeaBIOS 1.11.0 and up should have a compatible rom builtin,
but it's not even available in Ubuntu 18.04.
qemu/seabios@d6728f3

Additionaly drop the -smp 2 directive for qemu, until MTTCG is enabled on amd64 targets.

Updates golang/go#25289

Change-Id: If33c2b35b4221f27d719938bcfc013efe3549305
Reviewed-on: https://go-review.googlesource.com/135475
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/138258 mentions this issue: dashboard: add FreeBSD 11.2 and 10.4 builders

@paulzhol
Copy link
Member

@bradfitz our builders are horribly out of date, for eg. our 11.1 builder is 11.1-RELEASE and is missing around a year worth of security patches and updates (~15 of them).
10.3-RELEASE was released in April 4, 2016 and was marked EoL since April 30, 2018 (that's over two years of missed updates).

Unfortunately 11.1 is expected to receive EoL on the 30th of September, and 10.4 a month after that.
https://www.freebsd.org/security/security.html#sup

I think we have no choice but to switch the 10.3-RELEASE to 10.4-RELEASE-p13. Maybe it's worth while having a 11.1 builder as well, but I'd rather we update it to 11.1-RELEASE-p15 if we'd need to test future Go 1.10/1.11 releases?

@ayang64
Copy link
Member

ayang64 commented Sep 28, 2018

Since 10.4 and 11.2 are currently supported releases (even though they're close to EoL), I think it is reasonable to bump up the revs used by the builders.

It may also be helpful to start eyeing a 12.0 builder since it is due in Novemberish.

It looks like there are a few system call/ABI changes in 12 that might need addressing. I wonder how incompatibilities across GOOS versions are handled. It might be useful to be able to target FreeBSD 11.x and FreeBSD 12.x as separate GOOS.

gopherbot pushed a commit to golang/build that referenced this issue Oct 2, 2018
@bradfitz
dashboard: add FreeBSD 11.2 and 10.4 builders
05d9e9b 
Not yet the defaults or trybots until we see they're working regularly
for a few days and compare timings, etc.

Updates golang/go#25289

Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0
Reviewed-on: https://go-review.googlesource.com/138258
Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@bradfitz
Copy link
Contributor

bradfitz commented Oct 2, 2018

It might be useful to be able to target FreeBSD 11.x and FreeBSD 12.x as separate GOOS.

That introduces a bunch of complexity which we don't want for ourselves or our users. Making it Just Work is much more preferred.

@paulzhol
Copy link
Member

@bradfitz can we drop the 10.3 (and maybe 11.1) builders so that we can remove the workaround for regular files being added to kqueue (#19093)?

@bradfitz
Copy link
Contributor

@paulzhol, not until Go 1.13 (#27619).

@paulzhol
Copy link
Member

paulzhol commented Nov 20, 2018 •
edited

@bradfitz I'm not sure if the kernel fix was added to 10.4 (I'll try to reproduce in a VM). If it's fixed, can we drop the 10.3 builder (and maybe the 11.1 one) for Go 1.12?

@gopherbot
Copy link

Change https://golang.org/cl/155977 mentions this issue: env/freebsd-amd64: update FreeBSD 12.0 to a -RELEASE image

@paulzhol paulzhol mentioned this issue Dec 29, 2018
Closed
gopherbot pushed a commit to golang/build that referenced this issue Jan 8, 2019
@paulzhol @bradfitz
env/freebsd-amd64: update FreeBSD 12.0 to a -RELEASE image
9730d8b 
Install GDB from the ports collection, the in-tree was removed from the base system
in FreeBSD 12.0, and the older releases have an unsupported (6.1.1 or older version).

Updates golang/go#25289

Change-Id: I8899a2a70c0b9ee9171621206431ca24a7e97704
Reviewed-on: https://go-review.googlesource.com/c/155977
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor

bradfitz commented Jan 8, 2019

I rebuilt the FreeBSD amd64 image with https://golang.org/cl/155977 & updated the coordinator from git rev 7b78c2042368d5c56ee9dbd92ab5fa988c763944 to 9730d8b657054ead7ffc67bfb2dff24b89a368a5

@bradfitz bradfitz closed this as completed Jan 8, 2019
@bcmills bcmills mentioned this issue Jan 9, 2019
Closed
@golang golang locked and limited conversation to collaborators Jan 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-FreeBSD
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
5 participants
@bradfitz @ayang64 @paulzhol @ianlancetaylor @gopherbot