Skip to content

x/tools/go/packages: TestJSON may be non-deterministic/flaky #27594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dmitshur opened this issue Sep 10, 2018 · 5 comments
Closed

x/tools/go/packages: TestJSON may be non-deterministic/flaky #27594

dmitshur opened this issue Sep 10, 2018 · 5 comments
Labels
FrozenDueToAge Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

I can't reproduce this locally on darwin/amd64, but I've seen TestJSON fail many times with the same error message on various less common environments:

--- FAIL: TestJSON (0.67s)
	packages_test.go:1205: wrong JSON: got <<{
			"ID": "a",
			"Name": "a",
			"PkgPath": "a",
			"GoFiles": [
				"a.go"
			],
			"CompiledGoFiles": [
				"a.go"
			]
		}
		{
			"ID": "b",
			"Name": "b",
			"PkgPath": "b",
			"GoFiles": [
				"b.go"
			],
			"CompiledGoFiles": [
				"b.go"
			],
			"Imports": {
				"a": "a"
			}
		}
		{
			"ID": "d",
			"Name": "d",
			"PkgPath": "d",
			"GoFiles": [
				"d.go"
			],
			"CompiledGoFiles": [
				"d.go"
			],
			"Imports": {
				"b": "b"
			}
		}
		{
			"ID": "c",
			"Name": "c",
			"PkgPath": "c",
			"GoFiles": [
				"c.go"
			],
			"CompiledGoFiles": [
				"c.go"
			],
			"Imports": {
				"b": "b"
			}
		}
		>>, want <<{
			"ID": "a",
			"Name": "a",
			"PkgPath": "a",
			"GoFiles": [
				"a.go"
			],
			"CompiledGoFiles": [
				"a.go"
			]
		}
		{
			"ID": "b",
			"Name": "b",
			"PkgPath": "b",
			"GoFiles": [
				"b.go"
			],
			"CompiledGoFiles": [
				"b.go"
			],
			"Imports": {
				"a": "a"
			}
		}
		{
			"ID": "c",
			"Name": "c",
			"PkgPath": "c",
			"GoFiles": [
				"c.go"
			],
			"CompiledGoFiles": [
				"c.go"
			],
			"Imports": {
				"b": "b"
			}
		}
		{
			"ID": "d",
			"Name": "d",
			"PkgPath": "d",
			"GoFiles": [
				"d.go"
			],
			"CompiledGoFiles": [
				"d.go"
			],
			"Imports": {
				"b": "b"
			}
		}
		>>
	packages_test.go:1244: Package 2 has ID "d" want "c"
	packages_test.go:1247: Package "d" has Name "d" want "c"
	packages_test.go:1244: Package 3 has ID "c" want "d"
	packages_test.go:1247: Package "c" has Name "c" want "d"
FAIL
FAIL	golang.org/x/tools/go/packages	21.562s

For example, it failed just now on freebsd-amd64-10_3 builder for Go 1.10. See https://build.golang.org/log/b5fd57cad07f4cb3e1e3c9e1654a21fc23d7c025.

The diff in JSON is simply that the last two packages c and d are swapped.

Looking at the source code of TestJSON, packages c and d both depend on b only, so unless go/packages API is meant to guarantee a stable package visiting order, d being visited before c seems valid. Perhaps on those unusual environments, filesystem directory listing is not deterministic, causing d sometimes to be visited before c.

/cc @ianthehat @matloob
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 10, 2018
@gopherbot gopherbot added this to the Unreleased milestone Sep 10, 2018
@dmitshur
Copy link
Contributor Author

It happened again on darwin-amd64-10_11 builder on go1.10 branch:

https://build.golang.org/log/c5f6c6b6e6863169c92564c528adbb0cbc163b23

So it's not just FreeBSD that's affected. Just providing another data point on the issue.

@dominikh
Copy link
Member

Another data point: I've run into this issue locally as well, Go 1.10.3 on Linux. I've only hit that bug once so far, after dozens of times of running the tests (as part of my normal work, not specifically testing for this issue)

@ianthehat ianthehat self-assigned this Oct 24, 2018
@ianthehat ianthehat added Testing An issue that has been verified to require only test changes, not just a test failure. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 24, 2018
@ianthehat
Copy link

c and d are the roots, so this shows the roots of the graph are incorrectly ordered. The Visit function does guarantee a stable order when traversing the dependencies of the roots.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/143737 mentions this issue: go/packages: sort root list in Load

@dmitshur
Copy link
Contributor Author

dmitshur commented Nov 6, 2018

This seems to be re-occurring but for Go 1.10 only, see #28609.

@golang golang locked and limited conversation to collaborators Nov 6, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@dominikh @ianthehat @dmitshur @gopherbot @matloob