Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/gerritbot: deployed version crashes sometimes in GerritCL.Footer #27536

Closed
dmitshur opened this issue Sep 6, 2018 · 5 comments
Closed

x/build/cmd/gerritbot: deployed version crashes sometimes in GerritCL.Footer #27536

dmitshur opened this issue Sep 6, 2018 · 5 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Sep 6, 2018 •
edited

gerritbot has crashed a few times in production. The latest log ended with:

...
2018/08/26 18:54:12 Reloaded data from log *maintner.netMutSource.
2018/08/26 18:54:12 Starting corpus update loop ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x88 pc=0x8dbc5c]
goroutine 139 [running]:
golang.org/x/build/maintner.(*GerritCL).Footer(0xc46dc8efd0, 0xacaf9e, 0x14, 0x0, 0x0)
        /go/src/golang.org/x/build/maintner/gerrit.go:418 +0x4c
main.(*bot).checkPullRequests.func1.1(0xc46dc8efd0, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:308 +0x4d
golang.org/x/build/maintner.(*GerritProject).ForeachOpenCL(0xc42a6173b0, 0xc420220d90, 0xc42bc390f4, 0x2)
        /go/src/golang.org/x/build/maintner/gerrit.go:189 +0x246
main.(*bot).checkPullRequests.func1(0xc42a6173b0, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:307 +0xa2
golang.org/x/build/maintner.(*Gerrit).ForeachProjectUnsorted(0xc42f0512c0, 0xc420220e88, 0xc4201f2240, 0xc420220ef0)
        /go/src/golang.org/x/build/maintner/gerrit.go:84 +0xa1
main.(*bot).checkPullRequests(0xc4201f2230)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:302 +0xee
main.(*bot).corpusUpdateLoop(0xc4201f2230, 0xb359e0, 0xc42009a020)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:276 +0x75
created by main.main
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:64 +0x264

It looks like gerritbot was built a while ago (month or two), and there have been changes to maintner since then (that might help resolve the issue).

I'm guessing line 418 in gerrit.go refers to this old commit:

https://github.com/golang/build/blob/dd35151f24e38c0c3b2829571908d18ce9388c1c/maintner/gerrit.go#L418

First step will be to redeploy it with the current version of code, and see if that resolves the issue. There's a good chance it might. If not, the next step is to investigate how that value ended up being nil when it shouldn't be.

/cc @andybons
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 6, 2018
@dmitshur dmitshur self-assigned this Sep 6, 2018
@gopherbot gopherbot added this to the Unreleased milestone Sep 6, 2018
@dmitshur
Copy link
Contributor Author

dmitshur commented Sep 6, 2018

Redeployed now with current version. I'll check back in a week and see if it re-occurs.

@dmitshur
Copy link
Contributor Author

Haven't seen any crashes since then. I'll consider this resolved until there's evidence otherwise (we can re-open this).

@dmitshur
Copy link
Contributor Author

Seeing a few more instances of the same crash. This time it should be easier to debug because the deployed version is known/latest:

2018/09/15 16:16:52 Reloaded data from log *maintner.netMutSource.
2018/09/15 16:16:52 Starting corpus update loop ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x88 pc=0x9033e3]

goroutine 123 [running]:
golang.org/x/build/maintner.(*GerritCL).Footer(0xc04aff1130, 0xaeac4b, 0x14, 0x0, 0x0)
        /go/src/golang.org/x/build/maintner/gerrit.go:441 +0x43
main.(*bot).checkPullRequests.func1.1(0xc04aff1130, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:309 +0x4d
golang.org/x/build/maintner.(*GerritProject).ForeachOpenCL(0xc0039aab90, 0xc0001cfd90, 0xc0067d8234, 0x2)
        /go/src/golang.org/x/build/maintner/gerrit.go:189 +0x23c
main.(*bot).checkPullRequests.func1(0xc0039aab90, 0x0, 0x0)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:308 +0xa8
golang.org/x/build/maintner.(*Gerrit).ForeachProjectUnsorted(0xc00f7b7c20, 0xc0001cfe88, 0xc0001e8150, 0xc0000a4000)
        /go/src/golang.org/x/build/maintner/gerrit.go:84 +0x9f
main.(*bot).checkPullRequests(0xc0001e8140)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:303 +0xf1
main.(*bot).corpusUpdateLoop(0xc0001e8140, 0xb85a60, 0xc00009a020)
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:277 +0x75
created by main.main
        /go/src/golang.org/x/build/cmd/gerritbot/gerritbot.go:64 +0x250

@dmitshur dmitshur reopened this Sep 17, 2018
@dmitshur dmitshur removed their assignment Sep 17, 2018
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 17, 2018
@dmitshur dmitshur self-assigned this Sep 17, 2018
@dmitshur
Copy link
Contributor Author

I see a bug in maintner code. It's likely the bug causing this crash, but I don't have full information to say that conclusively.

I'll start by fixing it and see if anything more needs to be done. Will send a CL soon.

@gopherbot
Copy link

Change https://golang.org/cl/135677 mentions this issue: maintner: skip incomplete CLs in ForeachOpenCL

@dmitshur dmitshur mentioned this issue Sep 21, 2018
Closed
@golang golang locked and limited conversation to collaborators Sep 18, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@dmitshur @gopherbot