In go1.21 it's false and it's considered a symlink.

I don't think that's accurate. A symlink would have the ModeSymlink bit set, but as of https://go.dev/cl/460595 I believe you will file that the ModeSymlink bit remains unset, and the ModeIrregular bit is what is set for dedup files. The two bits mean different things.

POSIX defines a “regular file” as “a randomly accessible sequence of bytes, with no further structure imposed by the system.” Under that definition (which I think is the one the Go os package should use), it isn't clear to me which reparse points (if any) should be considered regular files.

Reparse points by definition do have “further structure imposed by the system” — namely, they have reparse tags and are handled by file system filters associated with those reparse tags. And some kinds of reparse points, such as IO_REPARSE_TAG_MOUNT_POINT and IO_REPARSE_TAG_LX_FIFO, certainly do not represent a randomly accessible sequence of bytes.

On the other hand, I would expect that at least some kinds of reparse points do represent “a randomly accessible sequence of bytes” from the perspective of an application. If we can explicitly enumerate which reparse points have that property, perhaps it would be ok to treat them as regular files? But they do still have “further structure imposed by the system”, so maybe not — I'm really not sure.

In the particular case of syncthing, perhaps there is some property other than regularity that syncthing could use instead?

(CC @golang/windows)

Compare #61893, #42184.

Change https://go.dev/cl/536655 mentions this issue: os,path/filepath: treat all non-symlink reparse points as irregular files

It is probably expecting too much from os to special-case reparse points other than IO_REPARSE_TAG_SYMLINK, The number of possible reparse points is unbounded, not only Microsoft can define them, also vendors and third parties. Also, other might be interested on special-casing IO_REPARSE_TAG_DEDUP instead of treating it is a regular file.

A good compromise could be to provide an easy way to access the reparse tag that is currently unexported in fs.FileInfo, so users can decide what to do with irregular files without having to open the file again and stating it using raw syscall functions. Maybe something like this:

package fs

type ReparseTagFileInfo interface {
	FileInfo
	ReparseTag() uint32
}

package os

func (fs *fileStat) ReparseTag() uint32 {
	return fs._ReparseTag
}

Which would be used like this:

var fi, err = os.Lstat(name)
if fi.Mode()&fs.ModeIrregular {
        if fix, ok := fi.(fs.ReparseTagFileInfo); ok {
	        switch fi.ReparseTag() {
                case syscall.IO_REPARSE_TAG_DEDUP:
                // custom logic
                }
        }
}

That would also be useful for us indeed - we currently do just what you say, we use syscalls on every irregular file to detect reparse points to treat them like regular ones.

I agree with the Boost developers that turning on NTFS deduplication should not make ordinary files look "irregular". People will turn this on to save some disk space, and then Go programs that use .IsRegular() to avoid opening printer devices and such will stop processing their files. That's not right for a portable os layer.

Whatever we decide for #61893, I think we should special-case the dedup reparse points and report them as regular files. If the behavior is new in Go 1.21 (wasn't present in Go 1.20), I think we should probably treat this as a critical bug fix and backport it.

Change https://go.dev/cl/537915 mentions this issue: os: report IO_REPARSE_TAG_DEDUP files as regular in Stat and Lstat

@imsodin, I have a proposed patch in https://go.dev/cl/537915 but I can't easily test it. (I don't have a Windows Server instance with deduplication enabled.)

If you can build that commit from source (https://go.dev/doc/install/source) and confirm that it fixes the problem for you or your users, I will request a backport to Go 1.21.

(Note that Gerrit has a “Download patch” option in the ⋮ menu to download a pending CL.)

@bcmills Your CL is equivalent to the workaround we added after upgrading to go1.21 and encountering the issue, and the user already confirmed it did fix the issue for them: syncthing/syncthing@9553365.
I hope that's good enough?

I agree that the approach is equivalent — mostly I want to rule out bugs in my implementation of it. 😅

But I'll go ahead and start the backport process based on that information.

@gopherbot, please backport to Go 1.21. This was a behavior change in Go 1.21 impacting users of the os package, and its impact was not realized at the time.

Backport issue(s) opened: #63764 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

Change https://go.dev/cl/538218 mentions this issue: [release-branch.go1.21] os: report IO_REPARSE_TAG_DEDUP files as regular in Stat and Lstat

Change https://go.dev/cl/565136 mentions this issue: os: don't treat mount points as symbolic links