Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: test failures on linux-arm, freebsd-386-12_0, and linux-s390x-ibm at CL 171758 #32452

Closed
bcmills opened this issue Jun 5, 2019 · 4 comments
Closed

runtime: test failures on linux-arm, freebsd-386-12_0, and linux-s390x-ibm at CL 171758 #32452

bcmills opened this issue Jun 5, 2019 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker Soon This needs to be done soon. (regressions, serious bugs, outages)
Milestone
Go1.13

Comments

@bcmills
Copy link
Contributor

bcmills commented Jun 5, 2019 •
edited

As of CL 171758, the linux-arm builder is consistently failing the ../test phase in fixedbugs/issue27518b.go, with the following error:

##### ../test
# go run run.go -- fixedbugs/issue27518b.go
exit status 1
runtime: nelems=128 nalloc=2 previous allocCount=1 nfreed=65535
fatal error: sweep increased allocation count

goroutine 1 [running]:
runtime.throw(0x88eac, 0x20)
	/workdir/go/src/runtime/panic.go:758 +0x5c fp=0x426690 sp=0x42667c pc=0x37044
runtime.(*mspan).sweep(0xb6f75878, 0x0, 0x1f)
	/workdir/go/src/runtime/mgcsweep.go:328 +0x888 fp=0x42671c sp=0x426690 pc=0x2df14
runtime.sweepone(0xd1e3c)
	/workdir/go/src/runtime/mgcsweep.go:136 +0x2d4 fp=0x426764 sp=0x42671c pc=0x2d3b8
runtime.GC()
	/workdir/go/src/runtime/mgc.go:1076 +0x1c0 fp=0x426784 sp=0x426764 pc=0x24014
main.gc(0x426700)
	/workdir/go/test/fixedbugs/issue27518b.go:36 +0x14 fp=0x426788 sp=0x426784 pc=0x61af0
main.main()
	/workdir/go/test/fixedbugs/issue27518b.go:53 +0x84 fp=0x4267a4 sp=0x426788 pc=0x61bf4
runtime.main()
	/workdir/go/src/runtime/proc.go:203 +0x208 fp=0x4267e4 sp=0x4267a4 pc=0x389f0
runtime.goexit()
	/workdir/go/src/runtime/asm_arm.s:868 +0x4 fp=0x4267e4 sp=0x4267e4 pc=0x5d8f0
exit status 2

FAIL	fixedbugs/issue27518b.go	1.908s

(https://build.golang.org/log/fc5cdd37ce154f561cb55d74243500c9432365f0)

It appears that there may be a bad interaction between stack-allocated defer records and the garbage collector.

CC @randall77 @aclements @mdempsky @josharian @mknyszek
@bcmills bcmills added NeedsFix The path to resolution is known, but the work has not been done. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Jun 5, 2019
@bcmills bcmills added this to the Go1.13 milestone Jun 5, 2019
@gopherbot gopherbot removed the NeedsFix The path to resolution is known, but the work has not been done. label Jun 5, 2019
@bcmills
Copy link
Contributor Author

bcmills commented Jun 5, 2019 •
edited

The freebsd-386-12_0 builder started failing consistently at the same CL, but in the syscall test and with a totally different failure mode.

--- FAIL: TestGetdirentries (0.04s)
    --- FAIL: TestGetdirentries/n=10 (0.00s)
        getdirentries_test.go:83: names don't match
             names: ["." ".." "file000" "file001" "file002" "file003" "file004" "file005" "file006" "file007" "file008" "file009"]
            names2: []
    --- FAIL: TestGetdirentries/n=1000 (0.04s)
        getdirentries_test.go:83: names don't match
             names: ["." ".." "file000" "file001" "file002" "file003" "file004" "file005" "file006" "file007" "file008" "file009" "file010" "file011" "file012" "file013" "file014" "file015" "file016" "file017" "file018" "file019" "file020" "file021" "file022" "file023" "file024" "file025" "file026" "file027" "file028" "file029" "file030" "file031" "file032" "file033" "file034" "file035" "file036" "file037" "file038" "file039" "file040" "file041" "file042" "file043" "file044" "file045" "file046" "file047" "file048" "file049" "file050" "file051" "file052" "file053" "file054" "file055" "file056" "file057" "file058" "file059" "file060" "file061" "file062" "file063" "file064" "file065" "file066" "file067" "file068" "file069" "file070" "file071" "file072" "file073" "file074" "file075" "file076" "file077" "file078" "file079" "file080" "file081" "file082" "file083" "file084" "file085" "file086" "file087" "file088" "file089" "file090" "file091" "file092" "file093" "file094" "file095" "file096" "file097" "file098" "file099" "file100" "file101" "file102" "file103" "file104" "file105" "file106" "file107" "file108" "file109" "file110" "file111" "file112" "file113" "file114" "file115" "file116" "file117" "file118" "file119" "file120" "file121" "file122" "file123" "file124" "file125" "file126" "file127" "file128" "file129" "file130" "file131" "file132" "file133" "file134" "file135" "file136" "file137" "file138" "file139" "file140" "file141" "file142" "file143" "file144" "file145" "file146" "file147" "file148" "file149" "file150" "file151" "file152" "file153" "file154" "file155" "file156" "file157" "file158" "file159" "file160" "file161" "file162" "file163" "file164" "file165" "file166" "file167" "file168" "file169" "file170" "file171" "file172" "file173" "file174" "file175" "file176" "file177" "file178" "file179" "file180" "file181" "file182" "file183" "file184" "file185" "file186" "file187" "file188" "file189" "file190" "file191" "file192" "file193" "file194" "file195" "file196" "file197" "file198" "file199" "file200" "file201" "file202" "file203" "file204" "file205" "file206" "file207" "file208" "file209" "file210" "file211" "file212" "file213" "file214" "file215" "file216" "file217" "file218" "file219" "file220" "file221" "file222" "file223" "file224" "file225" "file226" "file227" "file228" "file229" "file230" "file231" "file232" "file233" "file234" "file235" "file236" "file237" "file238" "file239" "file240" "file241" "file242" "file243" "file244" "file245" "file246" "file247" "file248" "file249" "file250" "file251" "file252" "file253" "file254" "file255" "file256" "file257" "file258" "file259" "file260" "file261" "file262" "file263" "file264" "file265" "file266" "file267" "file268" "file269" "file270" "file271" "file272" "file273" "file274" "file275" "file276" "file277" "file278" "file279" "file280" "file281" "file282" "file283" "file284" "file285" "file286" "file287" "file288" "file289" "file290" "file291" "file292" "file293" "file294" "file295" "file296" "file297" "file298" "file299" "file300" "file301" "file302" "file303" "file304" "file305" "file306" "file307" "file308" "file309" "file310" "file311" "file312" "file313" "file314" "file315" "file316" "file317" "file318" "file319" "file320" "file321" "file322" "file323" "file324" "file325" "file326" "file327" "file328" "file329" "file330" "file331" "file332" "file333" "file334" "file335" "file336" "file337" "file338" "file339" "file340" "file341" "file342" "file343" "file344" "file345" "file346" "file347" "file348" "file349" "file350" "file351" "file352" "file353" "file354" "file355" "file356" "file357" "file358" "file359" "file360" "file361" "file362" "file363" "file364" "file365" "file366" "file367" "file368" "file369" "file370" "file371" "file372" "file373" "file374" "file375" "file376" "file377" "file378" "file379" "file380" "file381" "file382" "file383" "file384" "file385" "file386" "file387" "file388" "file389" "file390" "file391" "file392" "file393" "file394" "file395" "file396" "file397" "file398" "file399" "file400" "file401" "file402" "file403" "file404" "file405" "file406" "file407" "file408" "file409" "file410" "file411" "file412" "file413" "file414" "file415" "file416" "file417" "file418" "file419" "file420" "file421" "file422" "file423" "file424" "file425" "file426" "file427" "file428" "file429" "file430" "file431" "file432" "file433" "file434" "file435" "file436" "file437" "file438" "file439" "file440" "file441" "file442" "file443" "file444" "file445" "file446" "file447" "file448" "file449" "file450" "file451" "file452" "file453" "file454" "file455" "file456" "file457" "file458" "file459" "file460" "file461" "file462" "file463" "file464" "file465" "file466" "file467" "file468" "file469" "file470" "file471" "file472" "file473" "file474" "file475" "file476" "file477" "file478" "file479" "file480" "file481" "file482" "file483" "file484" "file485" "file486" "file487" "file488" "file489" "file490" "file491" "file492" "file493" "file494" "file495" "file496" "file497" "file498" "file499" "file500" "file501" "file502" "file503" "file504" "file505" "file506" "file507" "file508" "file509" "file510" "file511" "file512" "file513" "file514" "file515" "file516" "file517" "file518" "file519" "file520" "file521" "file522" "file523" "file524" "file525" "file526" "file527" "file528" "file529" "file530" "file531" "file532" "file533" "file534" "file535" "file536" "file537" "file538" "file539" "file540" "file541" "file542" "file543" "file544" "file545" "file546" "file547" "file548" "file549" "file550" "file551" "file552" "file553" "file554" "file555" "file556" "file557" "file558" "file559" "file560" "file561" "file562" "file563" "file564" "file565" "file566" "file567" "file568" "file569" "file570" "file571" "file572" "file573" "file574" "file575" "file576" "file577" "file578" "file579" "file580" "file581" "file582" "file583" "file584" "file585" "file586" "file587" "file588" "file589" "file590" "file591" "file592" "file593" "file594" "file595" "file596" "file597" "file598" "file599" "file600" "file601" "file602" "file603" "file604" "file605" "file606" "file607" "file608" "file609" "file610" "file611" "file612" "file613" "file614" "file615" "file616" "file617" "file618" "file619" "file620" "file621" "file622" "file623" "file624" "file625" "file626" "file627" "file628" "file629" "file630" "file631" "file632" "file633" "file634" "file635" "file636" "file637" "file638" "file639" "file640" "file641" "file642" "file643" "file644" "file645" "file646" "file647" "file648" "file649" "file650" "file651" "file652" "file653" "file654" "file655" "file656" "file657" "file658" "file659" "file660" "file661" "file662" "file663" "file664" "file665" "file666" "file667" "file668" "file669" "file670" "file671" "file672" "file673" "file674" "file675" "file676" "file677" "file678" "file679" "file680" "file681" "file682" "file683" "file684" "file685" "file686" "file687" "file688" "file689" "file690" "file691" "file692" "file693" "file694" "file695" "file696" "file697" "file698" "file699" "file700" "file701" "file702" "file703" "file704" "file705" "file706" "file707" "file708" "file709" "file710" "file711" "file712" "file713" "file714" "file715" "file716" "file717" "file718" "file719" "file720" "file721" "file722" "file723" "file724" "file725" "file726" "file727" "file728" "file729" "file730" "file731" "file732" "file733" "file734" "file735" "file736" "file737" "file738" "file739" "file740" "file741" "file742" "file743" "file744" "file745" "file746" "file747" "file748" "file749" "file750" "file751" "file752" "file753" "file754" "file755" "file756" "file757" "file758" "file759" "file760" "file761" "file762" "file763" "file764" "file765" "file766" "file767" "file768" "file769" "file770" "file771" "file772" "file773" "file774" "file775" "file776" "file777" "file778" "file779" "file780" "file781" "file782" "file783" "file784" "file785" "file786" "file787" "file788" "file789" "file790" "file791" "file792" "file793" "file794" "file795" "file796" "file797" "file798" "file799" "file800" "file801" "file802" "file803" "file804" "file805" "file806" "file807" "file808" "file809" "file810" "file811" "file812" "file813" "file814" "file815" "file816" "file817" "file818" "file819" "file820" "file821" "file822" "file823" "file824" "file825" "file826" "file827" "file828" "file829" "file830" "file831" "file832" "file833" "file834" "file835" "file836" "file837" "file838" "file839" "file840" "file841" "file842" "file843" "file844" "file845" "file846" "file847" "file848" "file849" "file850" "file851" "file852" "file853" "file854" "file855" "file856" "file857" "file858" "file859" "file860" "file861" "file862" "file863" "file864" "file865" "file866" "file867" "file868" "file869" "file870" "file871" "file872" "file873" "file874" "file875" "file876" "file877" "file878" "file879" "file880" "file881" "file882" "file883" "file884" "file885" "file886" "file887" "file888" "file889" "file890" "file891" "file892" "file893" "file894" "file895" "file896" "file897" "file898" "file899" "file900" "file901" "file902" "file903" "file904" "file905" "file906" "file907" "file908" "file909" "file910" "file911" "file912" "file913" "file914" "file915" "file916" "file917" "file918" "file919" "file920" "file921" "file922" "file923" "file924" "file925" "file926" "file927" "file928" "file929" "file930" "file931" "file932" "file933" "file934" "file935" "file936" "file937" "file938" "file939" "file940" "file941" "file942" "file943" "file944" "file945" "file946" "file947" "file948" "file949" "file950" "file951" "file952" "file953" "file954" "file955" "file956" "file957" "file958" "file959" "file960" "file961" "file962" "file963" "file964" "file965" "file966" "file967" "file968" "file969" "file970" "file971" "file972" "file973" "file974" "file975" "file976" "file977" "file978" "file979" "file980" "file981" "file982" "file983" "file984" "file985" "file986" "file987" "file988" "file989" "file990" "file991" "file992" "file993" "file994" "file995" "file996" "file997" "file998" "file999"]
            names2: []
FAIL
FAIL	syscall	0.061s

(https://build.golang.org/log/150949ad287d3008b2d92ba74e61bd5b8795f515)

@bcmills bcmills changed the title runtime: "sweep increased allocation count" in fixedbugs/issue27518b.go after CL 171758 runtime: test failures on linux-arm, freebsd-386-12_0, and linux-s390x-ibm at CL 171758 Jun 5, 2019
@bcmills
Copy link
Contributor Author

bcmills commented Jun 5, 2019

linux-s390x-ibm also started failing at that CL, and this failure mode is a doozy: it doesn't even make it through bootstrapping.

linux-s390x-ibm at fff4f599fe1c21e411a99de5c9b3777d06ce0ce6

:: Running /data/golang/workdir/go/src/make.bash with args ["/data/golang/workdir/go/src/make.bash"] and env ["LC_CTYPE=en_US.UTF-8" "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOME=/root" "LOGNAME=root" "USER=root" "SHELL=/bin/sh" "META_BUILDLET_BINARY_URL=https://storage.googleapis.com/go-builder-data/buildlet.linux-s390x" "WORKDIR=/data/golang/workdir" "GOROOT_BOOTSTRAP=/data/golang/workdir/go1.4" "GO_BUILDER_NAME=linux-s390x-ibm" "GOROOT_BOOTSTRAP=/var/buildlet/go-linux-s390x-bootstrap" "GOBIN=" "TMPDIR=/data/golang/workdir/tmp" "GOCACHE=/data/golang/workdir/gocache"] in dir /data/golang/workdir/go/src

Building Go cmd/dist using /var/buildlet/go-linux-s390x-bootstrap.
Building Go toolchain1 using /var/buildlet/go-linux-s390x-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel fff4f599fe1c21e411a99de5c9b3777d06ce0ce6"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos linux goarch s390x\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "modinfo \"\"\n"
HASH[build runtime/internal/sys]: "compile Ju8I4bJGBKqsO5TJa6uN [] []\n"
HASH[build runtime/internal/sys]: "=\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/arch_s390x.go: 1860cf11a7c72ce0db5a7a73b14c52004d97c6af6d513540ece8d890e3f3ec16
HASH[build runtime/internal/sys]: "file arch_s390x.go GGDPEafHLODbWnpzsUxS\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/stubs.go: 23b3e5c631b086fe7a2dec4bf044600e034bf6a8eeb25e0a19efc4ce6311423d
HASH[build runtime/internal/sys]: "file stubs.go I7PlxjGwhv56LexL8ERg\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/zgoarch_s390x.go: a0333e25c1ec5662104b2b1e8128b274a0218ace80c3a905210fe49223d663a6
HASH[build runtime/internal/sys]: "file zgoarch_s390x.go oDM-JcHsVmIQSysegSiy\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/zgoos_linux.go: 806c088d7491b4560a28a5af86a52b459ebbf155ea455af873baa0bf697355e4
HASH[build runtime/internal/sys]: "file zgoos_linux.go gGwIjXSRtFYKKKWvhqUr\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/zversion.go: 447c7f41ab18d0b6265c2a94f93269075a518bad8df751d07fc7ee9eab0e2275
HASH[build runtime/internal/sys]: "file zversion.go RHx_QasY0LYmXCqU-TJp\n"
HASH[build runtime/internal/sys]: e8ab1b435529fa39c8c480e61b2271a65ebd4b10f852a1ef18f3516817609de5
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /data/golang/workdir/go/pkg/tool/linux_s390x/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch

@bcmills bcmills added the Soon This needs to be done soon. (regressions, serious bugs, outages) label Jun 5, 2019
@bcmills
Copy link
Contributor Author

bcmills commented Jun 5, 2019

Given the number and diversity of failing builders, I think we should roll back CL 171758 until it can be fixed.

@gopherbot
Copy link

Change https://golang.org/cl/180761 mentions this issue: Revert "cmd/compile,runtime: allocate defer records on the stack"

@golang golang locked and limited conversation to collaborators Jun 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker Soon This needs to be done soon. (regressions, serious bugs, outages)
Projects
None yet
Milestone
Go1.13
Development

No branches or pull requests
3 participants
@bcmills @randall77 @gopherbot