Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: NTFS deduped file changed from regular to irregular [1.21 backport] #63764

Closed
gopherbot opened this issue Oct 27, 2023 · 2 comments
Closed
Labels
CherryPickApproved Used during the release process for point releases
Milestone

Comments

@gopherbot
Copy link

@bcmills requested issue #63429 to be considered for backport to the next 1.21 minor release.

@gopherbot, please backport to Go 1.21. This was a behavior change in Go 1.21 impacting users of the os package, and its impact was not realized at the time.

@gopherbot gopherbot added the CherryPickCandidate Used during the release process for point releases label Oct 27, 2023
@gopherbot gopherbot added this to the Go1.21.4 milestone Oct 27, 2023
@gopherbot
Copy link
Author

Change https://go.dev/cl/538218 mentions this issue: [release-branch.go1.21] os: report IO_REPARSE_TAG_DEDUP files as regular in Stat and Lstat

@cherrymui cherrymui added the CherryPickApproved Used during the release process for point releases label Nov 1, 2023
@gopherbot gopherbot removed the CherryPickCandidate Used during the release process for point releases label Nov 1, 2023
@gopherbot gopherbot modified the milestones: Go1.21.4, Go1.21.5 Nov 7, 2023
gopherbot pushed a commit that referenced this issue Nov 8, 2023
…lar in Stat and Lstat

Prior to CL 460595, Lstat reported most reparse points as regular
files. However, reparse points can in general implement unusual
behaviors (consider IO_REPARSE_TAG_AF_UNIX or IO_REPARSE_TAG_LX_CHR),
and Windows allows arbitrary user-defined reparse points, so in
general we must not assume that an unrecognized reparse tag represents
a regular file; in CL 460595, we began marking them as irregular.

As it turns out, the Data Deduplication service on Windows Server runs
an Optimization job that turns regular files into reparse files with
the tag IO_REPARSE_TAG_DEDUP. Those files still behave more-or-less
like regular files, in that they have well-defined sizes and support
random-access reads and writes, so most programs can treat them as
regular files without difficulty. However, they are still reparse
files: as a result, on servers with the Data Deduplication service
enabled, files could arbitrarily change from “regular” to “irregular”
without explicit user intervention.

Since dedup files are converted in the background and otherwise behave
like regular files, this change adds a special case to report DEDUP
reparse points as regular.

Fixes #63764.
Updates #63429.

No test because to my knowledge we don't have any Windows builders
that have the deduplication service enabled, nor do we have a way to
reliably guarantee the existence of an IO_REPARSE_TAG_DEDUP file.

(In theory we could add a builder with the service enabled on a
specific volume, write a test that encodes knowledge of that volume,
and use the GO_BUILDER_NAME environment variable to run that test only
on the specially-configured builders. However, I don't currently have
the bandwidth to reconfigure the builders in this way, and given the
simplicity of the change I think it is unlikely to regress
accidentally.)

Change-Id: I649e7ef0b67e3939a980339ce7ec6a20b31b23a1
Cq-Include-Trybots: luci.golang.try:go1.21-windows-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/538218
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Reviewed-by: David Chase <drchase@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@gopherbot
Copy link
Author

Closed by merging f26fa05 to release-branch.go1.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CherryPickApproved Used during the release process for point releases
Projects
None yet
Development

No branches or pull requests

2 participants