Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: os: fast, comprehensive directory API #41265

Closed
networkimprov opened this issue Sep 7, 2020 · 10 comments
Closed

proposal: os: fast, comprehensive directory API #41265

networkimprov opened this issue Sep 7, 2020 · 10 comments

Comments

@networkimprov
Copy link

networkimprov commented Sep 7, 2020

This is an alternative to #41188, which seems impractical. It offers a replacement for os.Readdir() & os.Readdirnames(). We need a new API to address performance problems and missing features in the current one.

Previous discussion...

The API below meets all the requirements discussed. It provides explicit lazy-loading, ensures that all accessible fields have data from the same retrieval, and avoids an error check after every field access.

Use cases: When you need...
a) certain fields for every item, request them in ReadDirItems().
b) certain fields for some items, request them in DirItem.Load().
c) any fields of the OS defaults that aren't universal, check for them with DirItem.Has() before access.
d) implementation-specific fields, request them as above, then check for them with DirItem.Has() before access.
e) the latest data for an item, request it with DirItem.Load().

package os

// fields uint64 is a set of |'d fieldnames
// Platforms may define their own fieldnames outside a reserved range
// ReadDirItems() & DirItem.Load() ignore fieldnames not available on this OS
// OS default fields are always accessible in DirItem

type DirItem struct { ... }
func (d *DirItem) Has(fields uint64) bool   // indicates whether fields are loaded
func (d *DirItem) Load(fields uint64) error // (re-)loads fields
func (d *DirItem) Name() string             // always succeeds
func (d *DirItem) IsDir() bool              // this and other getters panic if field not loaded (a code error)
func (d *DirItem) Id() FileId
func (d *DirItem) Size() int64
func (d *DirItem) Mode() FileMode
func (d *DirItem) ModTime() time.Time
func (d *DirItem) CreTime() time.Time       // support OS-specific fields
func (d *DirItem) SameAs(*DirItem) bool     // calls .Id() for both objects; counterpart to os.SameFile

type FileId struct { ... }          // unix inode or winapi fileId; may include device info
func (id *FileId) String() string

func ReadDirItems(name string, opt uint64) ([]DirItem, error) // opt: fields | SortXyz
                                                              // if sorted, fieldnames must include sort key
func GetDirItem(name string, resolve bool) (*DirItem, error)  // does stat() or lstat()

func (f *File) ReadDirItems(n int, fields uint64) ([]DirItem, error)
func (f *File) GetDirItem() (*DirItem, error)

const (
   ItemOS    uint64 = 0 // fieldname for OS default fields
   ItemIsdir uint64 = 1 << iota
   ItemId
   ItemSize
   ItemMode
   ItemModTime
   ItemCreTime
)

const (
   SortDown uint64 = ... // descending switch
   SortName uint64 = ...
   SortId
   SortSize
   SortModTime
   SortCreTime
)

cc @robpike @rsc

@rsc
Copy link
Contributor

rsc commented Sep 11, 2020

Redesigning the os.File interface is explicitly out of scope for the io/fs proposal.
We are only making a simple interface for the current world, not building a new world.

@networkimprov
Copy link
Author

I'll retarget this proposal to the os package.

@networkimprov networkimprov changed the title proposal: io/fs: replace Readdir/FileInfo proposal: os: fast, comprehensive directory API Sep 14, 2020
@rsc
Copy link
Contributor

rsc commented Sep 15, 2020

Please stop. We are not redesigning the os package either.

@networkimprov
Copy link
Author

networkimprov commented Sep 16, 2020

Please stop.

Apologies if I've given offense somehow. #41188 looks like a non-starter. Could a better directory API land in the x/ repo?

(I completed the revisions I'd started above, dropping the io/fs references.)

@rsc
Copy link
Contributor

rsc commented Sep 16, 2020

Every person and every project has limited bandwidth for making decisions well, especially given the large amounts of time it can take to evaluate an idea fully. Redesigning the os.File API is not on our priority list right now. I'm sorry if that priority conflicts with yours, but again we have limited time and need to focus on higher-impact things.

It is of course fine to build your own APIs in a third-party package, as always.

@rsc rsc moved this from Incoming to Active in Proposals (old) Sep 16, 2020
@networkimprov
Copy link
Author

Then would you tag this "Proposal-Hold" instead of closing? You're gonna need it eventually :-)

@rsc
Copy link
Contributor

rsc commented Sep 18, 2020

A few things.

  1. Proposal-Hold is not for "gonna need eventually".
  2. It's far from clear that we're ever going to need to completely redo the entire directory API as proposed in this issue.
  3. If we did need a new API, the API here is not what we would do. If we were going to redo the API, it would be to simplify it further, not make it significantly more complex. Any new API proposal that increases complexity by double or more is a non-starter, and this looks like about 5-10X.

@rsc
Copy link
Contributor

rsc commented Sep 23, 2020

Based on the discussion above, this seems like a likely decline.

@rsc rsc moved this from Active to Likely Accept in Proposals (old) Sep 23, 2020
@rsc rsc moved this from Likely Accept to Likely Decline in Proposals (old) Sep 23, 2020
@networkimprov
Copy link
Author

We are not redesigning the os package either

Thankfully that didn't turn out to be true :-)

@rsc rsc moved this from Likely Decline to Declined in Proposals (old) Sep 30, 2020
@rsc
Copy link
Contributor

rsc commented Sep 30, 2020

No change in consensus, so declined (and retracted).

@golang golang locked and limited conversation to collaborators Sep 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

3 participants