Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/pprof: TestStackBarrierProfiling fails occasionally on linux/mips64le #15313

Closed
cherrymui opened this issue Apr 15, 2016 · 15 comments
Closed
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@cherrymui
Copy link
Member

The failure is rare, e.g. with commit 44f80f6, it failed 29 times out of 1500 runs. When it fails, it looks like

=== RUN TestStackBarrierProfiling
--- FAIL: TestStackBarrierProfiling (0.65s)
pprof_test.go:393: subprocess failed with signal: segmentation fault:
found next stack barrier at 0xc820137670; expected [*0xc820135c00=0x348d0 *0xc820135c38=0x238f8 *0xc820135cb8=0x24a1c *0xc820135d98=0x6a318 *0xc820135dd8=0xbe274 *0xc820135e28=0xbe2d4 *0xc820135e60=0xbe2d4 *0xc820135e98=0xbe2d4 *0xc820135ed0=0xbe2d4 *0xc820135f08=0xbe2d4 *0xc820135f40=0xbe2d4 *0xc820135f78=0xbe2d4 *0xc820135fb0=0xbe2d4 *0xc820135fe8=0xbe2d4 *0xc820136020=0xbe2d4 *0xc820136058=0xbe2d4 *0xc820136090=0xbe2d4 *0xc8201360c8=0xbe2d4 *0xc820136100=0xbe2d4 *0xc820136138=0xbe2d4 *0xc820136170=0xbe2d4 *0xc8201361a8=0xbe2d4 *0xc8201361e0=0xbe2d4 *0xc820136218=0xbe2d4 *0xc820136250=0xbe2d4 *0xc820136288=0xbe2d4 *0xc8201362c0=0xbe2d4 *0xc8201362f8=0xbe2d4 *0xc820136330=0xbe2d4 *0xc820136368=0xbe2d4 *0xc8201363a0=0xbe2d4 *0xc8201363d8=0xbe2d4 *0xc820136410=0xbe2d4 *0xc820136448=0xbe2d4 *0xc820136480=0xbe2d4 *0xc8201364b8=0xbe2d4 *0xc8201364f0=0xbe2d4 *0xc820136528=0xbe2d4 *0xc820136560=0xbe2d4 *0xc820136598=0xbe2d4 *0xc8201365d0=0xbe2d4 *0xc820136608=0xbe2d4 *0xc820136640=0xbe2d4 *0xc820136678=0xbe2d4 *0xc8201366b0=0xbe2d4 *0xc8201366e8=0xbe2d4 *0xc820136720=0xbe2d4 *0xc820136758=0xbe2d4 *0xc820136790=0xbe2d4 *0xc8201367c8=0xbe2d4 *0xc820136800=0xbe2d4 *0xc820136838=0xbe2d4 *0xc820136870=0xbe2d4 *0xc8201368a8=0xbe2d4 *0xc8201368e0=0xbe2d4 *0xc820136918=0xbe2d4 *0xc820136950=0xbe2d4 *0xc820136988=0xbe2d4 *0xc8201369c0=0xbe2d4 *0xc8201369f8=0xbe2d4 *0xc820136a30=0xbe2d4 *0xc820136a68=0xbe2d4 *0xc820136aa0=0xbe2d4 *0xc820136ad8=0xbe2d4 *0xc820136b10=0xbe2d4 *0xc820136b48=0xbe2d4 *0xc820136b80=0xbe2d4 *0xc820136bb8=0xbe2d4 *0xc820136bf0=0xbe2d4 *0xc820136c28=0xbe2d4 *0xc820136c60=0xbe2d4 *0xc820136c98=0xbe2d4 *0xc820136cd0=0xbe2d4 *0xc820136d08=0xbe2d4 *0xc820136d40=0xbe2d4 *0xc820136d78=0xbe2d4 *0xc820136db0=0xbe2d4 *0xc820136de8=0xbe2d4 *0xc820136e20=0xbe2d4 *0xc820136e58=0xbe2d4 *0xc820136e90=0xbe2d4 *0xc820136ec8=0xbe2d4 *0xc820136f00=0xbe2d4 *0xc820136f38=0xbe2d4 *0xc820136f70=0xbe2d4 *0xc820136fa8=0xbe2d4 *0xc820136fe0=0xbe2d4 *0xc820137018=0xbe2d4 *0xc820137050=0xbe2d4 *0xc820137088=0xbe2d4 *0xc8201370c0=0xbe2d4 *0xc8201370f8=0xbe2d4 *0xc820137130=0xbe2d4 *0xc820137168=0xbe2d4 *0xc8201371a0=0xbe2d4 *0xc8201371d8=0xbe2d4 *0xc820137210=0xbe2d4 *0xc820137248=0xbe2d4 *0xc820137280=0xbe2d4 *0xc8201372b8=0xbe2d4 *0xc8201372f0=0xbe2d4 *0xc820137328=0xbe2d4 *0xc820137360=0xbe2d4 *0xc820137398=0xbe2d4 *0xc8201373d0=0xbe2d4 *0xc820137408=0xbe2d4 *0xc820137440=0xbe2d4 *0xc820137478=0xbe2d4 *0xc8201374b0=0xbe2d4 *0xc8201374e8=0xbe2d4 *0xc820137520=0xbe2d4 *0xc820137558=0xbe2d4 *0xc820137590=0xbe2d4 *0xc8201375c8=0xbe2d4 *0xc820137600=0xbe2d4 @@@ ==> *0xc820137638=0xbe2d4 *0xc820137670=0xbe2d4 *0xc8201376a8=0xbe2d4 *0xc8201376e0=0xbe2d4 *0xc820137718=0xbe2d4 *0xc820137750=0xbe2d4 *0xc820137788=0xbe2d4 *0xc8201377c0=0xbe2d4 *0xc8201377f8=0xbe2d4 *0xc820137830=0xbe2d4 *0xc820137868=0xbe2d4 *0xc8201378a0=0xbe2d4 *0xc8201378d8=0xbe2d4 *0xc820137910=0xbe2d4 *0xc820137948=0xbe2d4 *0xc820137980=0xbe2d4 *0xc8201379b8=0xbe2d4 *0xc8201379f0=0xbe2d4 *0xc820137a28=0xbe2d4 *0xc820137a60=0xbe2d4 *0xc820137a98=0xbe2d4 *0xc820137ad0=0xbe2d4 *0xc820137b08=0xbe2d4 *0xc820137b40=0xbe2d4 *0xc820137b78=0xbe2d4 *0xc820137bb0=0xbe2d4 *0xc820137be8=0xbe2d4 *0xc820137c20=0xbe2d4 *0xc820137c58=0xbe2d4 *0xc820137c90=0xbe2d4 *0xc820137cc8=0xbe2d4 *0xc820137d00=0xbe2d4 *0xc820137d38=0xbe2d4 *0xc820137d70=0xbe2d4 *0xc820137da8=0xbe2d4 *0xc820137de0=0xbe2d4 *0xc820137e18=0xbe2d4 *0xc820137e50=0xbe2d4 *0xc820137e88=0xbe2d4 *0xc820137ec0=0xbe2d4 *0xc820137ef8=0xbe2d4 *0xc820137f30=0xbe2d4 *0xc820137f68=0xbe2d4 *0xc820137fa0=0xbe2d4 *0xc820137fd8=0xbe2d4 *0xc820138010=0xbe2d4 *0xc820138048=0xbe2d4 *0xc820138080=0xbe2d4 *0xc8201380b8=0xbe2d4 *0xc8201380f0=0xbe2d4 *0xc820138128=0xbe2d4 *0xc820138160=0xbe2d4 *0xc820138198=0xbe2d4 *0xc8201381d0=0xbe2d4 *0xc820138208=0xbe2d4 *0xc820138240=0xbe2d4 *0xc820138278=0xbe2d4 *0xc8201382b0=0xbe2d4 *0xc8201382e8=0xbe2d4 *0xc820138320=0xbe2d4 *0xc820138358=0xbe2d4 *0xc820138390=0xbe2d4 *0xc8201383c8=0xbe2d4 *0xc820138400=0xbe2d4 *0xc820138438=0xbe2d4 *0xc820138470=0xbe2d4 *0xc8201384a8=0xbe2d4 *0xc8201384e0=0xbe2d4 *0xc820138518=0xbe2d4 *0xc820138550=0xbe2d4 *0xc820138588=0xbe2d4 *0xc8201385c0=0xbe2d4 *0xc8201385f8=0xbe2d4 *0xc820138630=0xbe2d4 *0xc820138668=0xbe2d4 *0xc8201386a0=0xbe2d4 *0xc8201386d8=0xbe2d4 *0xc820138710=0xbe2d4 *0xc820138748=0xbe2d4 *0xc820138780=0xbe2d4 *0xc8201387b8=0xbe2d4 *0xc8201387f0=0xbe2d4 *0xc820138828=0xbe2d4 *0xc820138860=0xbe2d4 *0xc820138898=0xbe2d4 *0xc8201388d0=0xbe2d4 *0xc820138908=0xbe2d4 *0xc820138940=0xbe2d4 *0xc820138978=0xbe2d4 *0xc8201389b0=0xbe2d4 *0xc8201389e8=0xbe2d4 *0xc820138a20=0xbe2d4 *0xc820138a58=0xbe2d4 *0xc820138a90=0xbe2d4 *0xc820138ac8=0xbe2d4 *0xc820138b00=0xbe2d4 *0xc820138b38=0xbe2d4 *0xc820138b70=0xbe2d4 *0xc820138ba8=0xbe2d4 *0xc820138be0=0xbe2d4 *0xc820138c18=0xbe2d4 *0xc820138c50=0xbe2d4 *0xc820138c88=0xbe2d4 *0xc820138cc0=0xbe2d4 *0xc820138cf8=0xbe2d4 *0xc820138d30=0xbe2d4 *0xc820138d68=0xbe2d4 *0xc820138da0=0xbe2d4 *0xc820138dd8=0xbe2d4 *0xc820138e10=0xbe2d4 *0xc820138e48=0xbe2d4 *0xc820138e80=0xbe2d4 *0xc820138eb8=0xbe2d4 *0xc820138ef0=0xbe2d4 *0xc820138f28=0xbe2d4 *0xc820138f60=0xbe2d4 *0xc820138f98=0xbe2d4 *0xc820138fd0=0xbe2d4 *0xc820139008=0xbe2d4 *0xc820139040=0xbe2d4 *0xc820139078=0xbe2d4 *0xc8201390b0=0xbe2d4 *0xc8201390e8=0xbe2d4 *0xc820139120=0xbe2d4 *0xc820139158=0xbe2d4 *0xc820139190=0xbe2d4 *0xc8201391c8=0xbe2d4 *0xc820139200=0xbe2d4 *0xc820139238=0xbe2d4 *0xc820139270=0xbe2d4 *0xc8201392a8=0xbe2d4 *0xc8201392e0=0xbe2d4 *0xc820139318=0xbe2d4 *0xc820139350=0xbe2d4 *0xc820139388=0xbe2d4 *0xc8201393c0=0xbe2d4 *0xc8201393f8=0xbe2d4 *0xc820139430=0xbe2d4 *0xc820139468=0xbe2d4 *0xc8201394a0=0xbe2d4 *0xc8201394d8=0xbe2d4 *0xc820139510=0xbe2d4 *0xc820139548=0xbe2d4 *0xc820139580=0xbe2d4 *0xc8201395b8=0xbe2d4 *0xc8201395f0=0xbe2d4 *0xc820139628=0xbe2d4 *0xc820139660=0xbe2d4 *0xc820139698=0xbe2d4 *0xc8201396d0=0xbe2d4 *0xc820139708=0xbe2d4 *0xc820139740=0xbe2d4 *0xc820139778=0xbe2d4 *0xc8201397b0=0xbe2d4 *0xc8201397e8=0xbe2d4 *0xc820139820=0xbe2d4 *0xc820139858=0xbe2d4 *0xc820139890=0xbe2d4 *0xc8201398c8=0xbe2d4 *0xc820139900=0xbe2d4 *0xc820139938=0xbe2d4 *0xc820139970=0xbe2d4 *0xc8201399a8=0xbe2d4 *0xc8201399e0=0xbe2d4 *0xc820139a18=0xbe2d4 *0xc820139a50=0xbe2d4 *0xc820139a88=0xbe2d4 *0xc820139ac0=0xbe2d4 *0xc820139af8=0xbe2d4 *0xc820139b30=0xbe2d4 *0xc820139b68=0xbe2d4 *0xc820139ba0=0xbe2d4 *0xc820139bd8=0xbe2d4 *0xc820139c10=0xbe2d4 *0xc820139c48=0xbe2d4 *0xc820139c80=0xbe2d4 *0xc820139cb8=0xbe2d4 *0xc820139cf0=0xbe2d4 *0xc820139d28=0xbe2d4 *0xc820139d60=0xbe2d4 *0xc820139d98=0xbe2d4 *0xc820139dd0=0xbe2d4 *0xc820139e08=0xbe2d4 *0xc820139e40=0xbe2d4 *0xc820139e78=0xbe2d4 *0xc820139eb0=0xbe2d4 *0xc820139ee8=0xbe2d4 *0xc820139f20=0xbe2d4 *0xc820139f58=0xbe2d4 *0xc820139f90=0xbe2d4 *0xc820139fc8=0xbe2d4 *0xc82013a000=0xbe2d4 *0xc82013a038=0xbe2d4 *0xc82013a070=0xbe2d4 *0xc82013a0a8=0xbe2d4 *0xc82013a0e0=0xbe2d4 *0xc82013a118=0xbe2d4 *0xc82013a150=0xbe2d4 *0xc82013a188=0xbe2d4 *0xc82013a1c0=0xbe2d4 *0xc82013a1f8=0xbe2d4 *0xc82013a230=0xbe2d4 *0xc82013a268=0xbe2d4 *0xc82013a2a0=0xbe2d4 *0xc82013a2d8=0xbe2d4 *0xc82013a310=0xbe2d4 *0xc82013a348=0xbe2d4 *0xc82013a380=0xbe2d4 *0xc82013a3b8=0xbe2d4 *0xc82013a3f0=0xbe2d4 *0xc82013a428=0xbe2d4 *0xc82013a460=0xbe2d4 *0xc82013a498=0xbe2d4 *0xc82013a4d0=0xbe2d4 *0xc82013a508=0xbe2d4 *0xc82013a540=0xbe2d4 *0xc82013a578=0xbe2d4 *0xc82013a5b0=0xbe2d4 *0xc82013a5e8=0xbe2d4 *0xc82013a620=0xbe2d4 *0xc82013a658=0xbe2d4 *0xc82013a690=0xbe2d4 *0xc82013a6c8=0xbe2d4 *0xc82013a700=0xbe2d4 *0xc82013a738=0xbe2d4 *0xc82013a770=0xbe2d4 *0xc82013a7a8=0xbe2d4 *0xc82013a7e0=0xbe2d4 *0xc82013a818=0xbe2d4 *0xc82013a850=0xbe2d4 *0xc82013a888=0xbe2d4 *0xc82013a8c0=0xbe2d4 *0xc82013a8f8=0xbe2d4 *0xc82013a930=0xbe2d4 *0xc82013a968=0xbe2d4 *0xc82013a9a0=0xbe2d4 *0xc82013a9d8=0xbe2d4 *0xc82013aa10=0xbe2d4 *0xc82013aa48=0xbe2d4 *0xc82013aa80=0xbe2d4 *0xc82013aab8=0xbe2d4 *0xc82013aaf0=0xbe2d4 *0xc82013ab28=0xbe2d4 *0xc82013ab60=0xbe2d4 *0xc82013ab98=0xbe2d4 *0xc82013abd0=0xbe2d4 *0xc82013ac08=0xbe2d4 *0xc82013ac40=0xbe2d4 *0xc82013ac78=0xbe2d4 *0xc82013acb0=0xbe2d4 *0xc82013ace8=0xbe2d4 *0xc82013ad20=0xbe2d4 *0xc82013ad58=0xbe2d4 *0xc82013ad90=0xbe2d4 *0xc82013adc8=0xbe2d4 *0xc82013ae00=0xbe2d4 *0xc82013ae38=0xbe2d4 *0xc82013ae70=0xbe2d4 *0xc82013aea8=0xbe2d4 *0xc82013aee0=0xbe2d4 *0xc82013af18=0xbe2d4 *0xc82013af50=0xbe2d4 *0xc82013af88=0xbe2d4 *0xc82013afc0=0xbe2d4 *0xc82013aff8=0xbe2d4 *0xc82013b030=0xbe2d4 *0xc82013b068=0xbe2d4 *0xc82013b0a0=0xbe2d4 *0xc82013b0d8=0xbe2d4 *0xc82013b110=0xbe2d4 *0xc82013b148=0xbe2d4 *0xc82013b180=0xbe2d4 *0xc82013b1b8=0xbe2d4 *0xc82013b1f0=0xbe2d4 *0xc82013b228=0xbe2d4 *0xc82013b260=0xbe2d4 *0xc82013b298=0xbe2d4 *0xc82013b2d0=0xbe2d4 *0xc82013b308=0xbe2d4 *0xc82013b340=0xbe2d4 *0xc82013b378=0xbe2d4 *0xc82013b3b0=0xbe2d4 *0xc82013b3e8=0xbe2d4 *0xc82013b420=0xbe2d4 *0xc82013b458=0xbe2d4 *0xc82013b490=0xbe2d4 *0xc82013b4c8=0xbe2d4 *0xc82013b500=0xbe2d4 *0xc82013b538=0xbe2d4 *0xc82013b570=0xbe2d4 *0xc82013b5a8=0xbe2d4 *0xc82013b5e0=0xbe2d4 *0xc82013b618=0xbe2d4 *0xc82013b650=0xbe2d4 *0xc82013b688=0xbe2d4 *0xc82013b6c0=0xbe2d4 *0xc82013b6f8=0xbe2d4 *0xc82013b730=0xbe2d4 *0xc82013b768=0xbe2d4 *0xc82013b7a0=0xbe2d4 *0xc82013b7d8=0xbe2d4 *0xc82013b810=0xbe2d4 *0xc82013b848=0xbe2d4 *0xc82013b880=0xbe2d4 *0xc82013b8b8=0xbe2d4 *0xc82013b8f0=0xbe2d4 *0xc82013b928=0xbe2d4 *0xc82013b960=0xbe2d4 *0xc82013b998=0xbe2d4 *0xc82013b9d0=0xbe2d4 *0xc82013ba08=0xbe2d4 *0xc82013ba40=0xbe2d4 *0xc82013ba78=0xbe2d4 *0xc82013bab0=0xbe2d4 *0xc82013bae8=0xbe2d4 *0xc82013bb20=0xbe2d4 *0xc82013bb58=0xbe2d4 *0xc82013bb90=0xbe2d4 *0xc82013bbc8=0xbe2d4 *0xc82013bc00=0xbe2d4 *0xc82013bc38=0xbe2d4 *0xc82013bc70=0xbe2d4 *0xc82013bca8=0xbe2d4 *0xc82013bce0=0xbe2d4 *0xc82013bd18=0xbe2d4 *0xc82013bd50=0xbe2d4 *0xc82013bd88=0xbe2d4 *0xc82013bdc0=0xbe2d4 *0xc82013bdf8=0xbe2d4 *0xc82013be30=0xbe2d4 *0xc82013be68=0xbe2d4 *0xc82013bea0=0xbe2d4 *0xc82013bed8=0xbe2d4 *0xc82013bf10=0xbe2d4 *0xc82013bf48=0xbe2d4 *0xc82013bf80=0xbe2d4 *0xc82013bfb8=0xbe2d4 *0xc82013bff0=0xbe2d4 *0xc82013c028=0xbe2d4 *0xc82013c060=0xbe2d4 *0xc82013c098=0xbe2d4 *0xc82013c0d0=0xbe2d4 *0xc82013c108=0xbe2d4 *0xc82013c140=0xbe2d4 *0xc82013c178=0xbe2d4 *0xc82013c1b0=0xbe2d4 *0xc82013c1e8=0xbe2d4 *0xc82013c220=0xbe2d4 *0xc82013c258=0xbe2d4 *0xc82013c290=0xbe2d4 *0xc82013c2c8=0xbe2d4 *0xc82013c300=0xbe2d4 *0xc82013c338=0xbe2d4 *0xc82013c370=0xbe2d4 *0xc82013c3a8=0xbe2d4 *0xc82013c3e0=0xbe2d4 *0xc82013c418=0xbe2d4 *0xc82013c450=0xbe2d4 *0xc82013c488=0xbe2d4 *0xc82013c4c0=0xbe2d4 *0xc82013c4f8=0xbe2d4 *0xc82013c530=0xbe2d4 *0xc82013c568=0xbe2d4 *0xc82013c5a0=0xbe2d4 *0xc82013c5d8=0xbe2d4 *0xc82013c610=0xbe2d4 *0xc82013c648=0xbe2d4 *0xc82013c680=0xbe2d4 *0xc82013c6b8=0xbe2d4 *0xc82013c6f0=0xbe2d4 *0xc82013c728=0xbe2d4 *0xc82013c760=0xbe2d4 *0xc82013c798=0xbe2d4 *0xc82013c7d0=0xbe2d4 *0xc82013c808=0xbe2d4 *0xc82013c840=0xbe2d4 *0xc82013c878=0xbe2d4 *0xc82013c8b0=0xbe2d4 *0xc82013c8e8=0xbe2d4 *0xc82013c920=0xbe2d4 *0xc82013c958=0xbe2d4 *0xc82013c990=0xbe2d4 *0xc82013c9c8=0xbe2d4 *0xc82013ca00=0xbe2d4 *0xc82013ca38=0xbe2d4 *0xc82013ca70=0xbe2d4 *0xc82013caa8=0xbe2d4 *0xc82013cae0=0xbe2d4 *0xc82013cb18=0xbe2d4 *0xc82013cb50=0xbe2d4 *0xc82013cb88=0xbe2d4 *0xc82013cbc0=0xbe2d4 *0xc82013cbf8=0xbe2d4 *0xc82013cc30=0xbe2d4 *0xc82013cc68=0xbe2d4 *0xc82013cca0=0xbe2d4 *0xc82013ccd8=0xbe2d4 *0xc82013cd10=0xbe2d4 *0xc82013cd48=0xbe2d4 *0xc82013cd80=0xbe2d4 *0xc82013cdb8=0xbe2d4 *0xc82013cdf0=0xbe2d4 *0xc82013ce28=0xbe2d4 *0xc82013ce60=0xbe2d4 *0xc82013ce98=0xbe2d4 *0xc82013ced0=0xbe2d4 *0xc82013cf08=0xbe2d4 *0xc82013cf40=0xbe2d4 *0xc82013cf78=0xbe2d4 *0xc82013cfb0=0xbe2d4 *0xc82013cfe8=0xbe2d4 *0xc82013d020=0xbe2d4 *0xc82013d058=0xbe2d4 *0xc82013d090=0xbe2d4 *0xc82013d0c8=0xbe2d4 *0xc82013d100=0xbe2d4 *0xc82013d138=0xbe2d4 *0xc82013d170=0xbe2d4 *0xc82013d1a8=0xbe2d4 *0xc82013d1e0=0xbe2d4 *0xc82013d218=0xbe2d4 *0xc82013d250=0xbe2d4 *0xc82013d288=0xbe2d4 *0xc82013d2c0=0xbe2d4 *0xc82013d2f8=0xbe2d4 *0xc82013d330=0xbe2d4 *0xc82013d368=0xbe2d4 *0xc82013d3a0=0xbe2d4 *0xc82013d3d8=0xbe2d4 *0xc82013d410=0xbe2d4 *0xc82013d448=0xbe2d4 *0xc82013d480=0xbe2d4 *0xc82013d4b8=0xbe2d4 *0xc82013d4f0=0xbe2d4 *0xc82013d528=0xbe2d4 *0xc82013d560=0xbe2d4 *0xc82013d598=0xbe2d4 *0xc82013d5d0=0xbe2d4 *0xc82013d608=0xbe2d4 *0xc82013d640=0xbe2d4 *0xc82013d678=0xbe2d4 *0xc82013d6b0=0xbe2d4 *0xc82013d6e8=0xbe2d4 *0xc82013d720=0xbe2d4 *0xc82013d758=0xbe2d4 *0xc82013d790=0xbe2d4 *0xc82013d7c8=0xbe2d4 *0xc82013d800=0xbe2d4 *0xc82013d838=0xbe2d4 *0xc82013d870=0xbe2d4 *0xc82013d8a8=0xbe2d4 *0xc82013d8e0=0xbe2d4 *0xc82013d918=0xbe2d4 *0xc82013d950=0xbe2d4 *0xc82013d988=0xbe2d4 *0xc82013d9c0=0xbe2d4 *0xc82013d9f8=0xbe2d4 *0xc82013da30=0xbe2d4 *0xc82013da68=0xbe2d4 *0xc82013daa0=0xbe2d4 *0xc82013dad8=0xbe2d4 *0xc82013db10=0xbe2d4 *0xc82013db48=0xbe2d4 *0xc82013db80=0xbe2d4 *0xc82013dbb8=0xbe2d4 *0xc82013dbf0=0xbe2d4 *0xc82013dc28=0xbe2d4 *0xc82013dc60=0xc0aac *0xc82013dc98=0xbc378 *0xc82013dcc8=0xbe1f0 *0xc82013de30=0xac23c *0xc82013df90=0x8ff2c]
fatal error: missed stack barrier

    goroutine 0 [idle]:
    runtime.throw(0x1b6913, 0x14)
        /home/a/src/git/go-official/src/runtime/panic.go:566 +0xc0
    runtime.gentraceback(0xbe2ec, 0xc820137670, 0xbe2d4, 0xc820000f00, 0x0, 0xc8200ad8f8, 0x40, 0x0, 0x0, 0x6, ...)
        /home/a/src/git/go-official/src/runtime/traceback.go:296 +0x1778
    runtime.sigprof(0xbe2ec, 0xc820137670, 0xbe2d4, 0xc820000f00, 0xc82016ac00)
        /home/a/src/git/go-official/src/runtime/proc.go:3078 +0x6e8
    runtime.sighandler(0xc80000001d, 0xc8200adce8, 0xc8200add68, 0xc820000f00)
        /home/a/src/git/go-official/src/runtime/signal_mips64x.go:64 +0x104
    runtime.sigtrampgo(0x1d, 0xc8200adce8, 0xc8200add68)
        /home/a/src/git/go-official/src/runtime/signal_sigtramp.go:48 +0x140
    runtime.sigtramp(0xc82004e310, 0x0, 0x800000001d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1f64c, ...)
        /home/a/src/git/go-official/src/runtime/sys_linux_mips64x.s:249 +0x48

    goroutine 6 [running]:

FAIL

See also e.g. http://build.golang.org/log/a97d7fbc0505b36b32e5c4865503993ce497a914

@cherrymui
Copy link
Member Author

@aclements

@bradfitz bradfitz added this to the Unplanned milestone Apr 15, 2016
@aclements
Copy link
Member

This is most likely the same issue as the TestStackBarrierProfiling failures in #15138.

@gopherbot
Copy link

CL https://golang.org/cl/23291 mentions this issue.

@cherrymui
Copy link
Member Author

Update:
with the current tip (1f8d276), the failure is even rarer, only 1 failure over 2000 runs, and it looks a different failure:

=== RUN   TestStackBarrierProfiling
--- FAIL: TestStackBarrierProfiling (1.94s)
    pprof_test.go:393: subprocess failed with exit status 2:
        runtime: invalid pc-encoded table f=runtime.asminit pc=0x90c00 targetpc=0x90c00 tab=[0/0]0x0
            value=0 until pc=0x90c00
        fatal error: invalid runtime symbol table

        goroutine 0 [idle]:
        runtime.throw(0x1b5f2e, 0x1c)
            /home/a/src/git/go-official/src/runtime/panic.go:566 +0xb8 fp=0xc42005f360 sp=0xc42005f348
        runtime.pcvalue(0x1f7140, 0x2d439, 0x90c00, 0xc42005f638, 0x80301, 0x90c00)
            /home/a/src/git/go-official/src/runtime/symtab.go:472 +0x5a8 fp=0xc42005f528 sp=0xc42005f360
        runtime.funcspdelta(0x1f7140, 0x90c00, 0xc42005f638, 0x200)
            /home/a/src/git/go-official/src/runtime/symtab.go:511 +0x64 fp=0xc42005f590 sp=0xc42005f528
        runtime.gentraceback(0x7697c, 0xc420036ac8, 0x766a4, 0xc420001040, 0x0, 0xc42005f8b0, 0x40, 0x0, 0x0, 0x6, ...)
            /home/a/src/git/go-official/src/runtime/traceback.go:263 +0xbe0 fp=0xc42005f830 sp=0xc42005f590
        runtime.sigprof(0x7697c, 0xc420036ac8, 0x766a4, 0xc420001040, 0xc42002c800)
            /home/a/src/git/go-official/src/runtime/proc.go:3123 +0x8b4 fp=0xc42005fb58 sp=0xc42005f830
        runtime.sighandler(0xc40000001d, 0xc42005fce8, 0xc42005fd68, 0xc420001040)
            /home/a/src/git/go-official/src/runtime/signal_mips64x.go:64 +0x104 fp=0xc42005fbf0 sp=0xc42005fb58
        runtime.sigtrampgo(0x1d, 0xc42005fce8, 0xc42005fd68)
            /home/a/src/git/go-official/src/runtime/signal_sigtramp.go:48 +0x140 fp=0xc42005fc88 sp=0xc42005fbf0
        runtime.sigtramp(0x0, 0x0, 0x800000001d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
            /home/a/src/git/go-official/src/runtime/sys_linux_mips64x.s:254 +0x5c fp=0xc42005fcd0 sp=0xc42005fc88

        goroutine 4 [running]:
        runtime.systemstack_switch()
            /home/a/src/git/go-official/src/runtime/asm_mips64x.s:171 +0x8 fp=0xc420041690 sp=0xc420041688
        runtime.markroot(0xc42001f228, 0xc400000006)
            /home/a/src/git/go-official/src/runtime/mgcmark.go:239 +0x704 fp=0xc420041718 sp=0xc420041690
        runtime.gcDrain(0xc42001f228, 0x5)
            /home/a/src/git/go-official/src/runtime/mgcmark.go:950 +0x3f4 fp=0xc420041760 sp=0xc420041718
        runtime.gcBgMarkWorker(0xc42001e000)
            /home/a/src/git/go-official/src/runtime/mgc.go:1453 +0x830 fp=0xc4200417d0 sp=0xc420041760
        runtime.goexit()
            /home/a/src/git/go-official/src/runtime/asm_mips64x.s:875 +0x4 fp=0xc4200417d0 sp=0xc4200417d0
        created by runtime.gcBgMarkStartWorkers
            /home/a/src/git/go-official/src/runtime/mgc.go:1342 +0xcc

        goroutine 1 [chan receive (scan)]:
        runtime.gopark(0x1c5a40, 0xc42005a2f8, 0x1b28cc, 0xc, 0x17, 0x3)
            /home/a/src/git/go-official/src/runtime/proc.go:265 +0x210 fp=0xc420077a58 sp=0xc420077a30
        runtime.goparkunlock(0xc42005a2f8, 0x1b28cc, 0xc, 0x25a717, 0x3)
            /home/a/src/git/go-official/src/runtime/proc.go:271 +0x6c fp=0xc420077a90 sp=0xc420077a58
        runtime.chanrecv(0x185280, 0xc42005a2a0, 0x0, 0xc420077b01, 0xb0000)
            /home/a/src/git/go-official/src/runtime/chan.go:496 +0x640 fp=0xc420077b18 sp=0xc420077a90
        runtime.chanrecv1(0x185280, 0xc42005a2a0, 0x0)
            /home/a/src/git/go-official/src/runtime/chan.go:378 +0x34 fp=0xc420077b48 sp=0xc420077b18
        testing.(*T).Run(0xc42007c0c0, 0x1b529d, 0x19, 0x1c5bf0, 0x25f01)
            /home/a/src/git/go-official/src/testing/testing.go:584 +0x468 fp=0xc420077c18 sp=0xc420077b48
        testing.RunTests.func1(0xc42007c000)
            /home/a/src/git/go-official/src/testing/testing.go:730 +0xe8 fp=0xc420077ca8 sp=0xc420077c18
        testing.tRunner(0xc42007c000, 0xc420077de8)
            /home/a/src/git/go-official/src/testing/testing.go:547 +0xf8 fp=0xc420077ce0 sp=0xc420077ca8
        testing.RunTests(0x1c5688, 0x253ca0, 0x9, 0x9, 0x1)
            /home/a/src/git/go-official/src/testing/testing.go:736 +0x4f4 fp=0xc420077e08 sp=0xc420077ce0
        testing.(*M).Run(0xc420077f38, 0x6f01f2cb31cc7202)
            /home/a/src/git/go-official/src/testing/testing.go:680 +0xf8 fp=0xc420077e90 sp=0xc420077e08
        main.main()
            runtime/pprof/_test/_testmain.go:72 +0x15c fp=0xc420077f88 sp=0xc420077e90
        runtime.main()
            /home/a/src/git/go-official/src/runtime/proc.go:189 +0x3f8 fp=0xc420077fd8 sp=0xc420077f88
        runtime.goexit()
            /home/a/src/git/go-official/src/runtime/asm_mips64x.s:875 +0x4 fp=0xc420077fd8 sp=0xc420077fd8

        goroutine 2 [force gc (idle)]:
        runtime.gopark(0x1c5a40, 0x259cc0, 0x1b30c7, 0xf, 0x14, 0x1)
            /home/a/src/git/go-official/src/runtime/proc.go:265 +0x210 fp=0xc420026370 sp=0xc420026348
        runtime.goparkunlock(0x259cc0, 0x1b30c7, 0xf, 0xc420000114, 0x1)
            /home/a/src/git/go-official/src/runtime/proc.go:271 +0x6c fp=0xc4200263a8 sp=0xc420026370
        runtime.forcegchelper()
            /home/a/src/git/go-official/src/runtime/proc.go:230 +0x134 fp=0xc4200263d8 sp=0xc4200263a8
        runtime.goexit()
            /home/a/src/git/go-official/src/runtime/asm_mips64x.s:875 +0x4 fp=0xc4200263d8 sp=0xc4200263d8
        created by runtime.init.4
            /home/a/src/git/go-official/src/runtime/proc.go:219 +0x4c

        goroutine 3 [runnable]:
        runtime.Gosched()
            /home/a/src/git/go-official/src/runtime/proc.go:244 +0x20 fp=0xc420042798 sp=0xc420042788
        runtime.bgsweep(0xc42004e000)
            /home/a/src/git/go-official/src/runtime/mgcsweep.go:68 +0x154 fp=0xc4200427d0 sp=0xc420042798
        runtime.goexit()
            /home/a/src/git/go-official/src/runtime/asm_mips64x.s:875 +0x4 fp=0xc4200427d0 sp=0xc4200427d0
        created by runtime.gcenable
            /home/a/src/git/go-official/src/runtime/mgc.go:195 +0x78

        goroutine 5 [finalizer wait]:
        runtime.gopark(0x1c5a40, 0x272f38, 0x1b2e15, 0xe, 0x14, 0x1)
            /home/a/src/git/go-official/src/runtime/proc.go:265 +0x210 fp=0xc420025b30 sp=0xc420025b08
        runtime.goparkunlock(0x272f38, 0x1b2e15, 0xe, 0x14, 0x1)
            /home/a/src/git/go-official/src/runtime/proc.go:271 +0x6c fp=0xc420025b68 sp=0xc420025b30
        runtime.runfinq()
            /home/a/src/git/go-official/src/runtime/mfinal.go:158 +0xf4 fp=0xc420025bd8 sp=0xc420025b68
        runtime.goexit()
            /home/a/src/git/go-official/src/runtime/asm_mips64x.s:875 +0x4 fp=0xc420025bd8 sp=0xc420025bd8
        created by runtime.createfing
            /home/a/src/git/go-official/src/runtime/mfinal.go:139 +0xc0

        goroutine 6 [GC assist wait]:
        runtime.gopark(0x1c5a40, 0x25a3d8, 0x1b2d27, 0xe, 0x25f14, 0x2)
            /home/a/src/git/go-official/src/runtime/proc.go:265 +0x210 fp=0xc4200afa28 sp=0xc4200afa00
        runtime.goparkunlock(0x25a3d8, 0x1b2d27, 0xe, 0x25a314, 0x2)
            /home/a/src/git/go-official/src/runtime/proc.go:271 +0x6c fp=0xc4200afa60 sp=0xc4200afa28
        runtime.gcAssistAlloc(0xc4200011e0)
            /home/a/src/git/go-official/src/runtime/mgcmark.go:553 +0x464 fp=0xc4200afae0 sp=0xc4200afa60
        runtime.mallocgc(0x400, 0x184d00, 0x25e01, 0x25fd0)
            /home/a/src/git/go-official/src/runtime/malloc.go:563 +0x1b8 fp=0xc4200afbb0 sp=0xc4200afae0
        runtime.makeslice(0x184d00, 0x400, 0x400, 0x0, 0x0, 0x0)
            /home/a/src/git/go-official/src/runtime/slice.go:57 +0x128 fp=0xc4200afc00 sp=0xc4200afbb0
        runtime/pprof_test.deepStack(0x78, 0xc42007e400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:419 +0x78 fp=0xc4200afc38 sp=0xc4200afc00
        runtime/pprof_test.deepStack(0x79, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afc70 sp=0xc4200afc38
        runtime/pprof_test.deepStack(0x7a, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afca8 sp=0xc4200afc70
        runtime/pprof_test.deepStack(0x7b, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afce0 sp=0xc4200afca8
        runtime/pprof_test.deepStack(0x7c, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afd18 sp=0xc4200afce0
        runtime/pprof_test.deepStack(0x7d, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afd50 sp=0xc4200afd18
        runtime/pprof_test.deepStack(0x7e, 0xc42007e000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afd88 sp=0xc4200afd50
        runtime/pprof_test.deepStack(0x7f, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afdc0 sp=0xc4200afd88
        runtime/pprof_test.deepStack(0x80, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afdf8 sp=0xc4200afdc0
        runtime/pprof_test.deepStack(0x81, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afe30 sp=0xc4200afdf8
        runtime/pprof_test.deepStack(0x82, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afe68 sp=0xc4200afe30
        runtime/pprof_test.deepStack(0x83, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afea0 sp=0xc4200afe68
        runtime/pprof_test.deepStack(0x84, 0xc42007e400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afed8 sp=0xc4200afea0
        runtime/pprof_test.deepStack(0x85, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200aff10 sp=0xc4200afed8
        runtime/pprof_test.deepStack(0x86, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200aff48 sp=0xc4200aff10
        runtime/pprof_test.deepStack(0x87, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200aff80 sp=0xc4200aff48
        runtime/pprof_test.deepStack(0x88, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200affb8 sp=0xc4200aff80
        runtime/pprof_test.deepStack(0x89, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200afff0 sp=0xc4200affb8
        runtime/pprof_test.deepStack(0x8a, 0xc42007e000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0028 sp=0xc4200afff0
        runtime/pprof_test.deepStack(0x8b, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0060 sp=0xc4200b0028
        runtime/pprof_test.deepStack(0x8c, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0098 sp=0xc4200b0060
        runtime/pprof_test.deepStack(0x8d, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b00d0 sp=0xc4200b0098
        runtime/pprof_test.deepStack(0x8e, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0108 sp=0xc4200b00d0
        runtime/pprof_test.deepStack(0x8f, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0140 sp=0xc4200b0108
        runtime/pprof_test.deepStack(0x90, 0xc42007e400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0178 sp=0xc4200b0140
        runtime/pprof_test.deepStack(0x91, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b01b0 sp=0xc4200b0178
        runtime/pprof_test.deepStack(0x92, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b01e8 sp=0xc4200b01b0
        runtime/pprof_test.deepStack(0x93, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0220 sp=0xc4200b01e8
        runtime/pprof_test.deepStack(0x94, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0258 sp=0xc4200b0220
        runtime/pprof_test.deepStack(0x95, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0290 sp=0xc4200b0258
        runtime/pprof_test.deepStack(0x96, 0xc42007e000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b02c8 sp=0xc4200b0290
        runtime/pprof_test.deepStack(0x97, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0300 sp=0xc4200b02c8
        runtime/pprof_test.deepStack(0x98, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0338 sp=0xc4200b0300
        runtime/pprof_test.deepStack(0x99, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0370 sp=0xc4200b0338
        runtime/pprof_test.deepStack(0x9a, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b03a8 sp=0xc4200b0370
        runtime/pprof_test.deepStack(0x9b, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b03e0 sp=0xc4200b03a8
        runtime/pprof_test.deepStack(0x9c, 0xc42007e400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0418 sp=0xc4200b03e0
        runtime/pprof_test.deepStack(0x9d, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0450 sp=0xc4200b0418
        runtime/pprof_test.deepStack(0x9e, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0488 sp=0xc4200b0450
        runtime/pprof_test.deepStack(0x9f, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b04c0 sp=0xc4200b0488
        runtime/pprof_test.deepStack(0xa0, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b04f8 sp=0xc4200b04c0
        runtime/pprof_test.deepStack(0xa1, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0530 sp=0xc4200b04f8
        runtime/pprof_test.deepStack(0xa2, 0xc42007e000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0568 sp=0xc4200b0530
        runtime/pprof_test.deepStack(0xa3, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b05a0 sp=0xc4200b0568
        runtime/pprof_test.deepStack(0xa4, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b05d8 sp=0xc4200b05a0
        runtime/pprof_test.deepStack(0xa5, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0610 sp=0xc4200b05d8
        runtime/pprof_test.deepStack(0xa6, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0648 sp=0xc4200b0610
        runtime/pprof_test.deepStack(0xa7, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0680 sp=0xc4200b0648
        runtime/pprof_test.deepStack(0xa8, 0xc42007e400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b06b8 sp=0xc4200b0680
        runtime/pprof_test.deepStack(0xa9, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b06f0 sp=0xc4200b06b8
        runtime/pprof_test.deepStack(0xaa, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0728 sp=0xc4200b06f0
        runtime/pprof_test.deepStack(0xab, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0760 sp=0xc4200b0728
        runtime/pprof_test.deepStack(0xac, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0798 sp=0xc4200b0760
        runtime/pprof_test.deepStack(0xad, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b07d0 sp=0xc4200b0798
        runtime/pprof_test.deepStack(0xae, 0xc42007e000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0808 sp=0xc4200b07d0
        runtime/pprof_test.deepStack(0xaf, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0840 sp=0xc4200b0808
        runtime/pprof_test.deepStack(0xb0, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0878 sp=0xc4200b0840
        runtime/pprof_test.deepStack(0xb1, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b08b0 sp=0xc4200b0878
        runtime/pprof_test.deepStack(0xb2, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b08e8 sp=0xc4200b08b0
        runtime/pprof_test.deepStack(0xb3, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0920 sp=0xc4200b08e8
        runtime/pprof_test.deepStack(0xb4, 0xc42007e400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0958 sp=0xc4200b0920
        runtime/pprof_test.deepStack(0xb5, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0990 sp=0xc4200b0958
        runtime/pprof_test.deepStack(0xb6, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b09c8 sp=0xc4200b0990
        runtime/pprof_test.deepStack(0xb7, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0a00 sp=0xc4200b09c8
        runtime/pprof_test.deepStack(0xb8, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0a38 sp=0xc4200b0a00
        runtime/pprof_test.deepStack(0xb9, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0a70 sp=0xc4200b0a38
        runtime/pprof_test.deepStack(0xba, 0xc42007e000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0aa8 sp=0xc4200b0a70
        runtime/pprof_test.deepStack(0xbb, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0ae0 sp=0xc4200b0aa8
        runtime/pprof_test.deepStack(0xbc, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0b18 sp=0xc4200b0ae0
        runtime/pprof_test.deepStack(0xbd, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0b50 sp=0xc4200b0b18
        runtime/pprof_test.deepStack(0xbe, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0b88 sp=0xc4200b0b50
        runtime/pprof_test.deepStack(0xbf, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0bc0 sp=0xc4200b0b88
        runtime/pprof_test.deepStack(0xc0, 0xc42007e400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0bf8 sp=0xc4200b0bc0
        runtime/pprof_test.deepStack(0xc1, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0c30 sp=0xc4200b0bf8
        runtime/pprof_test.deepStack(0xc2, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0c68 sp=0xc4200b0c30
        runtime/pprof_test.deepStack(0xc3, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0ca0 sp=0xc4200b0c68
        runtime/pprof_test.deepStack(0xc4, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0cd8 sp=0xc4200b0ca0
        runtime/pprof_test.deepStack(0xc5, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0d10 sp=0xc4200b0cd8
        runtime/pprof_test.deepStack(0xc6, 0xc42007e000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0d48 sp=0xc4200b0d10
        runtime/pprof_test.deepStack(0xc7, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0d80 sp=0xc4200b0d48
        runtime/pprof_test.deepStack(0xc8, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0db8 sp=0xc4200b0d80
        runtime/pprof_test.deepStack(0xc9, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0df0 sp=0xc4200b0db8
        runtime/pprof_test.deepStack(0xca, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0e28 sp=0xc4200b0df0
        runtime/pprof_test.deepStack(0xcb, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0e60 sp=0xc4200b0e28
        runtime/pprof_test.deepStack(0xcc, 0xc42007e400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0e98 sp=0xc4200b0e60
        runtime/pprof_test.deepStack(0xcd, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0ed0 sp=0xc4200b0e98
        runtime/pprof_test.deepStack(0xce, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0f08 sp=0xc4200b0ed0
        runtime/pprof_test.deepStack(0xcf, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0f40 sp=0xc4200b0f08
        runtime/pprof_test.deepStack(0xd0, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0f78 sp=0xc4200b0f40
        runtime/pprof_test.deepStack(0xd1, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0fb0 sp=0xc4200b0f78
        runtime/pprof_test.deepStack(0xd2, 0xc42007e000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b0fe8 sp=0xc4200b0fb0
        runtime/pprof_test.deepStack(0xd3, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b1020 sp=0xc4200b0fe8
        runtime/pprof_test.deepStack(0xd4, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b1058 sp=0xc4200b1020
        runtime/pprof_test.deepStack(0xd5, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b1090 sp=0xc4200b1058
        runtime/pprof_test.deepStack(0xd6, 0x400)
            /home/a/src/git/go-official/src/runtime/pprof/pprof_test.go:420 +0xd8 fp=0xc4200b10c8 sp=0xc4200b1090
        ...additional frames elided...
        created by testing.(*T).Run
            /home/a/src/git/go-official/src/testing/testing.go:583 +0x43c

        goroutine 7 [syscall]:
        runtime.notetsleepg(0xfff7154008, 0xffffffffffffffff, 0x0)
            /home/a/src/git/go-official/src/runtime/lock_futex.go:205 +0x78 fp=0xc420040638 sp=0xc420040610
        runtime.(*cpuProfile).getprofile(0xfff7154000, 0x0, 0x0, 0x0)
            /home/a/src/git/go-official/src/runtime/cpuprof.go:341 +0x79c fp=0xc420040740 sp=0xc420040638
        runtime.CPUProfile(0x0, 0x0, 0x0)
            /home/a/src/git/go-official/src/runtime/cpuprof.go:421 +0x50 fp=0xc420040768 sp=0xc420040740
        runtime/pprof.profileWriter(0x2412c0, 0xc42007a000)
            /home/a/src/git/go-official/src/runtime/pprof/pprof.go:617 +0x34 fp=0xc4200407c8 sp=0xc420040768
        runtime.goexit()
            /home/a/src/git/go-official/src/runtime/asm_mips64x.s:875 +0x4 fp=0xc4200407c8 sp=0xc4200407c8
        created by runtime/pprof.StartCPUProfile
            /home/a/src/git/go-official/src/runtime/pprof/pprof.go:611 +0x1d8

        goroutine 8 [syscall]:
        runtime.notetsleepg(0x259f78, 0x129e163a7, 0x0)
            /home/a/src/git/go-official/src/runtime/lock_futex.go:205 +0x78 fp=0xc420024b50 sp=0xc420024b28
        runtime.timerproc()
            /home/a/src/git/go-official/src/runtime/time.go:209 +0x154 fp=0xc420024bd8 sp=0xc420024b50
        runtime.goexit()
            /home/a/src/git/go-official/src/runtime/asm_mips64x.s:875 +0x4 fp=0xc420024bd8 sp=0xc420024bd8
        created by runtime.addtimerLocked
            /home/a/src/git/go-official/src/runtime/time.go:116 +0x24c
FAIL

@aclements
Copy link
Member

Interesting. This new failure does look different, though it's also in the stack jumping code, just like the problem that caused the unexpected stack barrier turned out to be. Perhaps the missing frame.sp isn't the only issue that sequence has on LR machines.

@aclements aclements modified the milestones: Go1.7Maybe, Unplanned May 23, 2016
@aclements
Copy link
Member

Actually, I take that back. The failing line in gentraceback is right after the stack jump code, but I don't think the stack jump triggered in this case, since the error is saying that f is asminit, but it would have had to be systemstack for the jump to trigger.

gopherbot pushed a commit that referenced this issue May 24, 2016
When gentraceback starts on a system stack in sigprof, it is
configured to jump to the user stack when it reaches the end of the
system stack. Currently this updates the current frame's FP, but not
its SP. This is okay on non-LR machines (x86) because frame.sp is only
used to find defers, which the bottom-most frame of the user stack
will never have.

However, on LR machines, we use frame.sp to find the saved LR. We then
use to resolve the function of the next frame, which is used to
resolved the size of the next frame. Since we're not updating frame.sp
on a stack jump, we read the saved LR from the system stack instead of
the user stack and wind up resolving the wrong function and hence the
wrong frame size for the next frame.

This has had remarkably few ill effects (though the resulting profiles
must be wrong). We noticed it because of a bad interaction with stack
barriers. Specifically, once we get the next frame size wrong, we also
get the location of its LR wrong. If we happen to get a stack slot
that contains a stale stack barrier LR (for a stack barrier we already
hit) and hasn't been overwritten with something else as we re-grew the
stack, gentraceback will fail with a "found next stack barrier at ..."
error, pointing at the slot that it thinks is an LR, but isn't.

Fixes #15138.

Updates #15313 (might fix it).

Change-Id: I13cfa322b44c0c2f23ac2b3d03e12631e4a6406b
Reviewed-on: https://go-review.googlesource.com/23291
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
@rsc
Copy link
Contributor

rsc commented May 27, 2016

It's got to be the systemstack switch. That's the running goroutine on the m that is doing the profiling. It's the only stack that would be being traced. And we just found a bug in the code. Obviously f=asminit means that we didn't just go over the switch. But maybe we went over the switch one frame earlier and got out of sync, causing the next frame to pick up a reference to asminit from somewhere. It's equally clear nothing relevant is actually calling asminit, which returns immediately and never makes any calls. asminit's PC shouldn't even appear on the stack (and it only runs on thread startup anyway).

@ianlancetaylor
Copy link
Contributor

Postponing to 1.8.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.8Early, Go1.7Maybe Jul 7, 2016
@quentinmit quentinmit added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 30, 2016
@rsc rsc modified the milestones: Go1.8, Go1.8Early Oct 20, 2016
@rsc
Copy link
Contributor

rsc commented Oct 27, 2016

@cherrymui can you still reproduce this after your recent change to save LR before allocating the stack frame?

@aclements
Copy link
Member

FWIW, I'm about to disable stack barriers by default, making this test even more of a moot point.

@cherrymui
Copy link
Member Author

I have been running the test in a loop for 10+ hours, without seeing any failure (with yesterday's tip). So it's probably ok now.

@akutz
Copy link

akutz commented Dec 30, 2016

It's still failing for me with Go 1.8beta2 on linux/amd64 https://gist.github.com/akutz/ef7f430239a4e2a540c006b26ba33015

@aclements
Copy link
Member

@akutz, this bug is about linux/mips64le. Please open a new bug with details about your system (including the value of CONFIG_HIGH_RES_TIMERS from /proc/config or /proc/config.gz or /boot/config or /boot/config-$(uname -r)). That said, I wouldn't worry about this failure unless it's actually blocking you. This is a weird test run in a weird debugging mode that's already basically irrelevant in Go 1.8 and will go away entirely in Go 1.9.

@akutz
Copy link

akutz commented Jan 6, 2017

So...open the issue or not? It was blocking me in that I couldn't compile Go 1.8. To do so I skipped the test. If it's going away then I won't worry about it. Thanks!

@aclements
Copy link
Member

Since it was just blocking all.bash from passing, skipping it is a fine workaround and I wouldn't worry about filing a bug.

@golang golang locked and limited conversation to collaborators Jan 6, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

8 participants