Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/env/windows: disable Windows Defender in windows-amd64-2016 #26055

Closed
bradfitz opened this issue Jun 25, 2018 · 5 comments
Closed

x/build/env/windows: disable Windows Defender in windows-amd64-2016 #26055

bradfitz opened this issue Jun 25, 2018 · 5 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted NeedsFix The path to resolution is known, but the work has not been done. OS-Windows Performance
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

The Windows Defender (new in Windows Server 2016) is making filesystem access super slow on our windows-amd64-2016 builder.

Instead of a handful of seconds to extract the source tarball, it takes about a minute on windows-amd64-2016.

We need to disable PowerShell somehow programmatically from the buildlet, or change our image.

/cc @johnsonj
@bradfitz bradfitz added Performance help wanted OS-Windows NeedsFix The path to resolution is known, but the work has not been done. labels Jun 25, 2018
@gopherbot gopherbot added this to the Unreleased milestone Jun 25, 2018
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jun 25, 2018
@johnsonj johnsonj self-assigned this Jun 25, 2018
@bradfitz
Copy link
Contributor Author

FTR,

bradfitz@gdev:~/src/golang.org/x/build/cmd$ gomote create windows-amd64-2016
user-bradfitz-windows-amd64-2016-0
bradfitz@gdev:~/src/golang.org/x/build/cmd$ gomote run -system user-bradfitz-windows-amd64-2016-0 sc query WinDefend

SERVICE_NAME: WinDefend 
        TYPE               : 10  WIN32_OWN_PROCESS  
        STATE              : 4  RUNNING 
                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0

bradfitz@gdev:~/src/golang.org/x/build/cmd$ time gomote push user-bradfitz-windows-amd64-2016-0
2018/06/25 21:05:21 installing go1.4
2018/06/25 21:05:42 Remote doesn't have "src/math/unsafe.go"
2018/06/25 21:05:42 Remote doesn't have "test/fixedbugs/issue6703o.go"
2018/06/25 21:05:42 Remote doesn't have "misc/nacl/go_nacl_386_exec"
2018/06/25 21:05:42 Remote doesn't have "src/debug/gosym/pclinetest.h"
2018/06/25 21:05:42 Remote doesn't have "src/runtime/defs_netbsd.go"
2018/06/25 21:05:42 Remote doesn't have 7982 files (only showed 5).
2018/06/25 21:05:42 Remote lacks a VERSION file; sending a fake one
2018/06/25 21:05:53 Uploading 7983 new/changed files; 145856045 byte .tar.gz

real    1m18.972s
user    0m11.612s
sys     0m0.580s
bradfitz@gdev:~/src/golang.org/x/build/cmd$ time gomote rm user-bradfitz-windows-amd64-2016-0 .

real    0m2.559s
user    0m0.104s
sys     0m0.012s

That 1m18.982s is pretty slow.

Unfortunately the buildlet is running as user gopher (not admin) and the admin has no password set so we can't use runas to sc stop it.

So updating the image is our best option. Then we can re-benchmark it with the above.

@bradfitz
Copy link
Contributor Author

Also FTR,

SELECT Builder, AVG(Seconds) as Sec, COUNT(*) FROM builds.Spans WHERE Event='write_go_src_tar' AND Error='' AND StartTime > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 500 HOUR) GROUP BY 1 ORDER BY Sec DESC;

# Builder Seconds Count
1 dragonfly-amd64 185.60650053108293 229
2 linux-mips 152.23976293918628 220
3 plan9-arm 80.30161213704082 245
4 freebsd-arm-paulzhol 63.06639745541143 175
5 windows-amd64-2016 55.55313342421522 618
6 linux-arm 49.075175881510845 231
7 linux-arm-nativemake 48.70805966784979 233
8 linux-arm64-packet 19.515660791081444 221
9 android-386-emulator 18.119463975340423 235
10 darwin-arm64-a1549ios 16.899542428412843 218
11 darwin-arm-a1428ios 16.739108357539738 239
12 android-amd64-emulator 16.54923103103448 232
13 android-arm-wiko-fever 15.080117587781658 229
14 android-arm64-wiko-fever 14.462792876952378 231
15 plan9-amd64-9front 14.405663703632353 68
16 solaris-amd64-oraclerel 12.100369053424782 226
17 linux-mipsle 10.584515092049774 221
18 plan9-386 9.966798731117123 222
19 darwin-amd64-10_12 8.794949175008892 225
20 darwin-amd64-race 8.605581733506607 227
21 darwin-amd64-10_10 8.154243777534242 219
22 darwin-386-10_11 7.946001849290908 220
23 darwin-amd64-10_11 7.703967139495968 248
24 windows-amd64-2008 7.034785014407078 226
25 openbsd-386-62 7.010430919705883 221
26 windows-amd64-race 6.797259301486726 226
27 windows-386-2008 6.646885542157986 576
28 solaris-amd64-smartosbuildlet 6.024883029986425 221
29 windows-amd64-2012 5.337799377800866 231
30 linux-s390x-ibm 4.947826236703196 219
31 linux-ppc64le-buildlet 4.35824075608889 225
32 linux-ppc64-buildlet 4.307531873781248 224
33 openbsd-amd64-62 3.0256817755499097 551
34 netbsd-amd64-8_0 2.956358088454129 218
35 freebsd-386-10_3 2.3237422244598203 224
36 freebsd-amd64-10_3 2.294154283222727 220
37 freebsd-amd64-race 2.269509212908676 219
38 freebsd-386-11_1 2.2659248075675666 222
39 freebsd-amd64-11_1 2.102135369425992 554
40 linux-arm-arm5spacemonkey 1.1791402133607305 219
41 misc-compile-nacl 1.108447655222222 90
42 linux-amd64-nocgo 1.0897813647031966 219
43 misc-compile-ppc 1.080169373228659 328
44 linux-amd64-ssacheck 1.0798409478082194 219
45 linux-amd64-sid 1.0794736670136982 219
46 nacl-386 1.0792323511771218 542
47 js-wasm 1.0791127955347828 230
48 linux-amd64-race 1.078254141806331 537
49 nacl-amd64p32 1.0779594923314915 543
50 misc-compile-freebsd 1.077641247359756 328

@johnsonj
Copy link
Contributor

johnsonj commented Jul 2, 2018 •
edited

Uninstalling Windows Defender looks like a big speedup. CL incoming.

Fresh instance, n1-standard-4, gomote push:

.. clipped by buffer
2018/07/02 09:58:00 Uploading 7171 new/changed files; 20499864 byte .tar.gz
     
real    0m33.565s
user    0m3.976s
sys     0m0.508s

gomote put14

real	0m3.477s
user	0m0.008s
sys	0m0.008s

@gopherbot
Copy link

Change https://golang.org/cl/121937 mentions this issue: env/windows: disable windows defender on builders

@bradfitz
Copy link
Contributor Author

bradfitz commented Jul 2, 2018

Yup, it went from about 56 seconds to 6.8:

2018/07/02 19:38:04 SERIAL: 2018/07/02 19:38:03 extracted tarball into C:\workdir\go: 7908 files, 980 dirs (6.8310531s)

@golang golang locked and limited conversation to collaborators Jul 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted NeedsFix The path to resolution is known, but the work has not been done. OS-Windows Performance
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @johnsonj @gopherbot