Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crypto/x509: TestPlatformVerifier failures on macOS due to mismatched error text #65461

Closed
gopherbot opened this issue Feb 2, 2024 · 19 comments
Closed

crypto/x509: TestPlatformVerifier failures on macOS due to mismatched error text #65461

gopherbot opened this issue Feb 2, 2024 · 19 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin Security

Comments

@gopherbot
Copy link

gopherbot commented Feb 2, 2024 •
edited by cherrymui 

#!watchflakes
post <- pkg == "crypto/x509" && test ~ `TestPlatformVerifier` && `x509: “valid.testing.golang.invalid” certificate is not trusted`

Issue created automatically to collect these failures.

Example (log):

=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

— watchflakes
@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 2, 2024
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "crypto/x509" && test == "TestPlatformVerifier"
2024-02-01 21:42 gotip-darwin-amd64_14 go@c9d88ea2 crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)
2024-02-01 22:57 gotip-darwin-amd64-nocgo go@6d3c1ce8 crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)
2024-02-01 22:59 gotip-darwin-amd64-nocgo go@117164f9 crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.04s)

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "crypto/x509" && test == "TestPlatformVerifier"
2024-02-02 18:20 gotip-darwin-amd64-nocgo go@6b0309ce crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.01s)

— watchflakes

@bcmills
Copy link
Contributor

bcmills commented Feb 5, 2024

(attn @rolandshoemaker)

@bcmills bcmills changed the title crypto/x509: TestPlatformVerifier failures crypto/x509: TestPlatformVerifier failures on macOS due to mismatched error text Feb 5, 2024
@rolandshoemaker
Copy link
Member

I took a quick look at this, and I have zero clue how this could be flakey. The pattern of failures don't provide any particularly useful insight, the error returned is completely useless and I'm unable to repro locally.

Will think about this a little more, but for now I'm stumped.

@bcmills
Copy link
Contributor

bcmills commented Feb 5, 2024

Huh. I wonder if someone was changing the LUCI builder configuration during these runs? (Maybe someone on @golang/release would have some insight?)

@dmitshur
Copy link
Contributor

dmitshur commented Feb 5, 2024 •
edited

@prattmic might know if there've been changes to macOS builders on LUCI that would be relevant to this.

Something to check might be whether the builders running into this all have the root from #52108 added. But given the test is running (and not skipped with "test root is not in trust store"), I think it must be.

@prattmic
Copy link
Member

prattmic commented Feb 5, 2024

Every failing instance linked above was on the same bot, darwin-amd64-14--c1f58526-d1c4-4792-bef1-06d6c16b49e4.golang.cc.macservice.goog. I imagine that is related, but it is still odd because all of the bots use identical macOS images.

@rolandshoemaker
Copy link
Member

Bit flip somewhere in the root store? 🤷

@prattmic
Copy link
Member

prattmic commented Feb 5, 2024

I pulled up the test history on this bot. The bot has been running since 2024-01-09, and the test passed 100% of the time until a run starting 2024-02-01 11:30:06.033 America/Los_Angeles (https://cr-buildbucket.appspot.com/build/8757267625331049905). It has been failing 100% of the time since then.

@prattmic
Copy link
Member

prattmic commented Feb 5, 2024

I've also verified that this is the only bot that has had any failures.

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "crypto/x509" && test == "TestPlatformVerifier" && `x509: “valid.testing.golang.invalid” certificate is not trusted`
2024-02-05 14:52 gotip-darwin-amd64-nocgo go@e8d87728 crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)
2024-02-05 15:30 gotip-darwin-amd64-nocgo go@76ff0caa crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.03s)
2024-02-05 20:59 gotip-darwin-amd64-nocgo go@6076edc5 crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.03s)

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "crypto/x509" && test == "TestPlatformVerifier" && `x509: “valid.testing.golang.invalid” certificate is not trusted`
2024-02-02 16:26 go1.21-darwin-amd64-nocgo release-branch.go1.21@2fdad8af crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)
2024-02-06 12:43 gotip-darwin-amd64_14 go@6f44cc88 crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

— watchflakes

@prattmic
Copy link
Member

prattmic commented Feb 6, 2024

The cluster this instance was running in had an infrastructure issue that seems to have corrupted it. I have killed this instance.

@prattmic prattmic closed this as completed Feb 6, 2024
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "crypto/x509" && test == "TestPlatformVerifier" && `x509: “valid.testing.golang.invalid” certificate is not trusted`
2024-02-02 16:17 go1.22-darwin-amd64_14 release-branch.go1.22@b0957cfc crypto/x509.TestPlatformVerifier (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

— watchflakes

@dmitshur dmitshur mentioned this issue Feb 10, 2024
Open
@gopherbot gopherbot reopened this Feb 20, 2024
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "crypto/x509" && test ~ `TestPlatformVerifier` && `x509: “valid.testing.golang.invalid” certificate is not trusted`
2024-02-20 16:10 gotip-darwin-amd64_13 go@ff4e45fb crypto/x509.TestPlatformVerifier/non-nested_KU (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:245: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.07s)
2024-02-20 17:18 gotip-darwin-amd64_13 go@02785362 crypto/x509.TestPlatformVerifier/non-nested_KU (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:243: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "crypto/x509" && test ~ `TestPlatformVerifier` && `x509: “valid.testing.golang.invalid” certificate is not trusted`
2024-02-20 22:29 gotip-darwin-amd64_13 go@5428cc4f crypto/x509.TestPlatformVerifier/non-nested_KU (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:243: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "crypto/x509" && test ~ `TestPlatformVerifier` && `x509: “valid.testing.golang.invalid” certificate is not trusted`
2024-02-21 18:47 gotip-darwin-amd64_13 go@c07b9b00 crypto/x509.TestPlatformVerifier/non-nested_KU (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:243: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)
2024-02-21 20:35 gotip-darwin-amd64_13 go@b27d02c0 crypto/x509.TestPlatformVerifier/non-nested_KU (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:243: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)
2024-02-22 22:21 gotip-darwin-amd64_13 go@d892cb49 crypto/x509.TestPlatformVerifier/non-nested_KU (log) 
=== RUN   TestPlatformVerifier/non-nested_KU
=== PAUSE TestPlatformVerifier/non-nested_KU
=== CONT  TestPlatformVerifier/non-nested_KU
    platform_test.go:243: unexpected verification error: got "x509: “valid.testing.golang.invalid” certificate is not trusted", want "x509: “valid.testing.golang.invalid” certificate is not permitted for this usage"
--- FAIL: TestPlatformVerifier/non-nested_KU (0.00s)

— watchflakes

@rolandshoemaker
Copy link
Member

@prattmic it is possible we have another broken instance?

@prattmic
Copy link
Member

Quite possible, I was doing a sweep of potentially messed up instances (https://go.dev/cl/562399 has helped a lot, but some are still slipping through), and as part of that I've killed the instance that these tests failed on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin Security
Projects
Test Flakes
Status: Done
Milestone
No milestone
Development

No branches or pull requests
5 participants
@prattmic @dmitshur @rolandshoemaker @bcmills @gopherbot