Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: os/user: add iterators over users and groups #47907

Closed
mjonaitis1 opened this issue Aug 23, 2021 · 15 comments
Closed

proposal: os/user: add iterators over users and groups #47907

mjonaitis1 opened this issue Aug 23, 2021 · 15 comments

Comments

@mjonaitis1
Copy link

mjonaitis1 commented Aug 23, 2021

Currently os/user package supports users and groups lookup via name or id. Some applications might need functionality to iterate over all available users/groups in the system. On unix there is getent command which can retrieve users or groups by calling getent passwd or getent group respectively. However, this is not very convenient. A go-native functionality could remove the need of calling external commands.

Update: use slice returning function based api instead of callback based iterator api

After receiving some internal feedback, I was recommended to not use callback based iterator API. Since callback based api introduces a lot of unnecessary complexity, slice returning function API might be more reasonable.

Updated public facing API:

package user

// ListUsers returns a slice of all available user entries.
func ListUsers() ([]*User, error) {
	return allUsers()
}

// ListGroups returns a slice of all available group entries.
func ListGroups() ([]*Group, error) {
	return allGroups()
}

Example usage:

// Users
users, err := user.ListUsers()
for _, u := range users {
	fmt.Printf("Username: %s\n", u.Username)
}
if err != nil {
	fmt.Printf("error encountered while iterating users database: %v", err)
}

// Groups
groups, err := user.ListGroups()
for _, g := range groups {
	fmt.Printf("Groupname: %s\n", g.Name)
}
if err != nil {
	fmt.Printf("error encountered while iterating groups database: %v", err)
}

However, more feedback from go maintainers is needed in order to proceed.

Initial callback based proposal

Here is a proposed user facing public api in os/user:

package user

// NextUserFunc is used in users iteration process. It receives *User for each user record.
// If non-nil error is returned from NextUserFunc - iteration process is terminated.
type NextUserFunc func(*User) error

// NextGroupFunc is used in groups iteration process. It receives *Group for each group record.
// If non-nil error is returned from NextGroupFunc - iteration process is terminated.
type NextGroupFunc func(*Group) error

// IterateUsers iterates over user entries. For each retrieved *User entry provided NextUserFunc is called.
//
// On UNIX, if CGO is enabled, getpwent is used in the underlying implementation. Since getpwent is not thread-safe,
// locking is strongly advised.
func IterateUsers(n NextUserFunc) error {
	return iterateUsers(n)
}

// IterateGroups iterates over group entries. For each retrieved *Group entry provided NextGroupFunc is called.
//
// On UNIX, if CGO is enabled, getgrent is used in the underlying implementation. Since getgrent is not thread-safe,
// locking is strongly advised.
func IterateGroups(n NextGroupFunc) error {
	return iterateGroups(n)
}

Example usage:

// Iterating all users
err := user.IterateUsers(func(u *user.User) error {
	fmt.Println(u.Name)
	return nil
})
if err != nil {
	log.Fatalf("error occurred while iterating users: %v \n", err)
}

// Iterating all groups
err := user.IterateGroups(func(g *user.Group) error {
	fmt.Println(g.Name)
	return nil
})
if err != nil {
	log.Fatalf("error occurred while iterating groups: %v \n", err)
}	
@gopherbot gopherbot added this to the Proposal milestone Aug 23, 2021
@seankhliao seankhliao changed the title proposal: add users/groups iteration functionality to os/user proposal: os/user: add iterators over users and groups Aug 23, 2021
@seankhliao
Copy link
Member

cc @bradfitz @kevinburke

@gopherbot
Copy link

Change https://golang.org/cl/345592 mentions this issue: os/user: add users and groups iterators

@neild
Copy link
Contributor

neild commented Aug 31, 2021

How does this deal with groups in NIS/AD/other non-local sources?

Looking at the CL, iteration on Unix seems to only consult /etc/passwd and /etc/group.

@mjonaitis1
Copy link
Author

mjonaitis1 commented Sep 1, 2021

@neild when cgo is enabled iterate_cgo.go will be used for unix. This file contains logic that utilises getpwent and getgrent libc routines which deal with fetching users or groups from other than /etc/* sources. These routines can also fetch data through name service switch.

@bjorndm
Copy link

bjorndm commented Sep 3, 2021

@mjonaitis1 I think it would be better to port getpwent and getgrent from C to Go, otherwise pure-Go programs will work differently with regards to this functionality.

@motiejus
Copy link
Contributor

motiejus commented Sep 3, 2021

@mjonaitis1 I think it would be better to port getpwent and getgrent from C to Go, otherwise pure-Go programs will work differently with regards to this functionality.

getpwent/getgrent are part of POSIX.1-2001, and it is the recommended way to get users/groups out of the system. If one wants a reasonable way to retrieve system's users/groups, then they should use the system interfaces. In fact, exactly like golang stdlib does today with Lookup* calls.

The most common and desired case (in fact, the reason why we made this pull request) — the glibc's getpwent cannot be ported to pure go, since it requires dynamically loading C shared libraries; this is just how getpwent of glibc works.

@nightlyone
Copy link
Contributor

There is no POSIX api for such iteration and the implementation might trigger a huge number of element lookups when applied to a corporate user/group database with 100000 and more entries.

I would prefer to call a more specialized user directory function in that case and work with libraries better suited for such lookups and filtering and did so when I needed it.

@rsc
Copy link
Contributor

rsc commented Sep 15, 2021

Why does this need to be in the standard library?

@mjonaitis1
Copy link
Author

Why does this need to be in the standard library?

For an internal project, we needed an API to iterate over system users and groups. Go standard library package os/user already handles lookup, but not listing or iteration. While a third-party package for iteration can be done, code will need to be copied, since most of the machinery is private to os/user package. We think it may be useful to have iteration/listing to the broader community.

Furthermore, we have some plans to open source an nss library in Golang. Since our library supports iteration (i.e. backs glibc’s getgrnam(3), getgrgid(3), getpwnam(3), getpwuid(3)), it would be helpful to have an API to piggy-back to; a standard way to iterate over users and groups would allow the library to support a single api and be a drop-in extension.

@mjonaitis1
Copy link
Author

@mjonaitis1 I think it would be better to port getpwent and getgrent from C to Go, otherwise pure-Go programs will work differently with regards to this functionality.

Even if getpwent could be ported to pure Go, it might not be a good idea. The setpwent/getpwnent/endpwent appear to have been designed without much regard to multi-threading. For example, if two threads want to iterate and not be exposed to non-deterministic gaps, they must agree to coordinate their iteration sessions through some mutex. This is cumbersome and slow. So instead of porting these functions to go, we can simply reuse them since they are posix compliant.

@rsc
Copy link
Contributor

rsc commented Oct 20, 2021

I don't think Damien's comment has really been answered.
In sites where these entries are provided via network protocol, it may not even be possible to enumerate users and groups.
It's very difficult to say much about how well any of that will work.
And then there is Windows.

I also don't think #47907 (comment) has really been answered. It seems like this can easily live outside the standard library for the few programs that need it.

@rsc
Copy link
Contributor

rsc commented Oct 20, 2021

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals (old) Oct 20, 2021
@rsc
Copy link
Contributor

rsc commented Oct 27, 2021

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Decline in Proposals (old) Oct 27, 2021
@mjonaitis1
Copy link
Author

I don't think Damien's comment has really been answered. In sites where these entries are provided via network protocol, it may not even be possible to enumerate users and groups. It's very difficult to say much about how well any of that will work. And then there is Windows.

My knowledge about nss is limited, but I assume that libc routines like getpwent take care of the external sources, so iteration of network provided entries should be possible with cgo.

I also don't think #47907 (comment) has really been answered. It seems like this can easily live outside the standard library for the few programs that need it.

The main argument why it should be in the standard library is that there is already similar functionality in os/user package.

@rsc rsc moved this from Likely Decline to Declined in Proposals (old) Nov 3, 2021
@rsc
Copy link
Contributor

rsc commented Nov 3, 2021

No change in consensus, so declined.
— rsc for the proposal review group

@rsc rsc closed this as completed Nov 3, 2021
@golang golang locked and limited conversation to collaborators Nov 3, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

8 participants