Change https://golang.org/cl/372794 mentions this issue: net/http/pprof: skip TestDeltaProfile on openbsd/arm

greplogs --dashboard -md -l -e 'FAIL: TestDeltaProfile ' --since=2021-12-17

2022-02-04T22:34:05-f9763a6/android-arm64-corellium

Given the android/arm64 failure, this does not seem specific to OpenBSD.
(CC @cherrymui)

Another android/arm64 today:

greplogs --dashboard -md -l -e 'FAIL: TestDeltaProfile ' --since=2022-02-05

2022-02-07T21:00:02-7db75b3/android-arm64-corellium

Change https://go.dev/cl/383997 mentions this issue: net/http/pprof: skip TestDeltaProfile on all arm and arm64 architectures

I suspect it may be possible to use something along the lines of the awaitBlockedGoroutine function from CL 384534 to eliminate the timing-dependence of this test.

(I do not plan to follow up on that myself.)

Marking as release-blocker for Go 1.19 due to the high failure rate on the builders (CC @golang/release).

If we don't care about this test, we can merge CL 383997 (mailed ~3 weeks ago) to encode that decision as a test-skip. Otherwise, this test failure needs to be addressed.

greplogs --dashboard -md -l -e 'FAIL: TestDeltaProfile ' --since=2022-02-08

2022-03-01T20:52:30-f4722d8/android-arm-corellium
2022-02-28T21:56:43-f04d5c1/android-arm-corellium

@golang/runtime needs to decide whether to fix and unskip this test, or to drop the test.

We should decide what to do here. I'll take a look at this.

@mknyszek Have you had a chance to revisit this decision? Do we need to make it before beta 1 (aiming for next week)?

Sorry, I didn't get around to it. I'll do it right now.

So, this hasn't happened since March. I have a feeling this is related to a few things that were fixed with respect to profiles recently. Curiously, the failures stopped happening on March 7th, and on March 8th https://go.dev/cl/384239 landed to fix #50996. There aren't any other pprof or profile related CLs around this one. The previous one landed in February, and the following one in May. That CL, I think, only affected proto output, but then again that's exactly what the net/http/pprof test is checking, because it obtains the profile via query.

This test is checking a mutex profile, but I imagine it uses the same stack frame information that anything else does. I think it's possible that what's happening in this test is call frames are dropped because these NOPs are actually (created when inlining for a "fake" PC for the inlined function) weren't handled correctly.

However, I'm not totally clear on what the failure mode for #50996 is. Specifically, neither mutexHop1 nor mutexHog2 are inline-able, but they have a few calls that are (like the Lock calls). So is it the parent frame that could get lost? Just the inlined call? The entire stack? I suspect it's the entire stack because if we land on one of these NOPs and don't handle them correctly (but only sometimes?). It's also not clear whether these NOPs play a role here. They probably play some role, given that this is a mutex profile and the Lock and Unlock calls are inlined.

I'm inclined to call this fixed. I haven't fully figured out the details, but I think there is a plausible connection here.

CC @rhysh @prattmic @cherrymui who were involved to varying degrees in https://go.dev/cl/384239. It might be clearer to y'all what exactly was the problem here and whether that CL fixed this particular issue. What's also puzzling me is, if there was a problem handling these NOPs, the failure should intuitively be more deterministic for a mutex profile, but I'm also not terribly familiar with how the mutex profile works.

At the very least, I'm moving this into the backlog optimistically to keep release progress moving.

I'm inclined to call this fixed. I haven't fully figured out the details, but I think there is a plausible connection here.

In that case, shall we close the issue until / unless it recurs?

I'm not opposed, I wanted to give others a chance to chime in, but it's probably less work to just close it and move on. :) If it happens again we can take some of this information forward.

If I recall correctly, https://go.dev/cl/384239 shouldn't affect this failure. In that case, the failure was that you'd end up with multiple Locations for things that should be the same Location. e.g., one Location including all of the inlined frames, and 2 separate Locations with one containing the inlined frame and the other containing the parent frame.

But all of the frames should still appear in the profile. In this case, the mutexHog2 frame is missing entirely, so this seems different.

Still, I think it is OK to wait and see if this occurs again.

https://go.dev/cl/383997 landed on March 8 to skip the test on arm and arm64. So the bug is probably not fixed.

Strange that mutexHog2 doesn't show up even once (in the non-delta profile printed at the end of each failure), since it must run at least once in order for that goroutine to check the done channel. But, if there's something that causes a function to not be reported some of the time (as the comments about gcc's optimizations hint), we wouldn't see all the times mutexHog1 is missing because those cause the test to skip.

Ha, OK. I missed that when I was walking back over the commit log. Thanks @rhysh. I'm going to reopen this. Still not certain this should block the release at this point, but it looks like there's a real problem and it's currently being masked.