Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/vet: warn about concurrent modification of a map (values added in a for loop) #46097

Open
rokusei opened this issue May 11, 2021 · 7 comments
Labels
Analysis Issues related to static analysis (vet, x/tools/go/analysis) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@rokusei
Copy link

rokusei commented May 11, 2021

(using #43698 as a template)
Related issues: #9926, #35239

Per the Go specification of for loops, within the For statements with range clause section, it states:

If a map entry is created during iteration, that entry may be produced during the iteration or may be skipped. The choice may vary for each entry created and from one iteration to the next.

I expect that package authors might not be expecting the non-deterministic behavior that comes with concurrent modification. This can lead to unexpected bugs and behavior. I believe Java goes so far to even throw a runtime exception.

Playground example
Real-world example

I propose that cmd/vet should emit a warning when:

  • a map is iterated over using a for statement
  • the map has elements added during iteration
@mvdan
Copy link
Member

mvdan commented May 11, 2021

How about doing it at run-time? Go will already panic if many concurrent reads and writes are done on a map. This wouldn't be a data race per se, but it could be possible to catch writes in the middle of a range at run-time.

@heschi heschi added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 11, 2021
@heschi
Copy link
Contributor

heschi commented May 11, 2021

Given that the behavior is specified, a runtime failure (even just in race mode) seems inappropriate to me.

As I understand it, vet checks are intended to have essentially no false positives. As such, someone would need to make a case that this is a bug, not just some of the time, but all the time.

cc @alandonovan

@heschi heschi added this to the Backlog milestone May 11, 2021
@mvdan
Copy link
Member

mvdan commented May 11, 2021

Given that the behavior is specified, a runtime failure (even just in race mode) seems inappropriate to me.

Right, but then the same applies to vet :) I guess I'm saying that if this is definitely wrong, then it would be ideal to panic at run-time.

@ianlancetaylor
Copy link
Contributor

It's not definitely wrong. It's entirely reasonable to add elements to a map during an iteration, if the new elements have some characteristic that allows the iteration to skip them.

I don't think vet should warn about this. Perhaps other static analyzers could warn about this case, but I don't see it as appropriate for vet.

That said, look at real code. Try writing the vet warning and running it over a bunch of packages written by different people (e.g., all Kubernetes packages including third party imported packages). If the new check never fires, it's probably not helpful and we shouldn't add it. If the new check only issues warnings on correct code, we shouldn't add it. If the new check only issues warnings on code that turns out to be incorrect, we should add it. Other cases (warnings on both correct and incorrect code) require a judgement call.

@bcmills
Copy link
Contributor

bcmills commented May 11, 2021

This pattern isn't always wrong. Consider, say, a program that constructs a map containing the transitive closure of nodes in a graph by iterating until no new nodes are added. Such a program is guaranteed to converge on a map with deterministic contents, even though the set of nodes scanned (and added) in any given iteration may vary.

To detect real bugs, perhaps it would suffice to make the order of iteration more aggressive under some configuration. For example, in -race mode we could randomly choose between producing every new entry and no new entries (similar to what is proposed for #35128, or the existing scheduler randomization). Then the real bugs could be detected during testing, perhaps by fuzz tests (#44551).

@timothy-king
Copy link
Contributor

timothy-king commented May 11, 2021

One criteria for the check is that there needs to be a path back to the range statement, e.g. do not report inserting and then breaking/returning. I suspect there will be too many false positives otherwise.

That said, look at real code.

+1. The background rate of people misunderstanding "skippable" insertions vs. getting it right will determine how reasonable adding this check is.

Consider, say, a program that constructs a map containing the transitive closure of nodes in a graph by iterating until no new nodes are added.

@bcmills I don't understand the example you have in mind. Can you elaborate?

To detect real bugs, perhaps it would suffice to make the order of iteration more aggressive under some configuration.

Why not do this on regular go test? (Overhead?)

@bcmills
Copy link
Contributor

bcmills commented May 11, 2021

I don't understand the example you have in mind. Can you elaborate?

Sure: https://play.golang.org/p/Aj5Oafuk8-I

@adonovan adonovan added the Analysis Issues related to static analysis (vet, x/tools/go/analysis) label Apr 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Analysis Issues related to static analysis (vet, x/tools/go/analysis) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

7 participants