Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: iter: ChunkIter & ChunkIter2 to chunk iterators #71472

Closed
vegas503 opened this issue Jan 29, 2025 · 1 comment
Closed

proposal: iter: ChunkIter & ChunkIter2 to chunk iterators #71472

vegas503 opened this issue Jan 29, 2025 · 1 comment
Labels
Milestone

Comments

@vegas503
Copy link

vegas503 commented Jan 29, 2025

Proposal Details

Proposal details

slices package provides useful Chunk function that batches a slice into chunks.

A fellow iter alternative would be extremely helpful to work with iterators.
Below is a sample implementation that I'm using in my projects (rewritten to fit into iter package):

package iter

// ...

// ChunkIter works like slices.Chunk but with iterators,
// i. e. iterates over it and yields batches of size n.
// Panics if n is less than 1.
func ChunkIter[I Seq[V], V any](it I, n int) Seq[[]V] {
	if n < 1 {
		panic("cannot be less than 1")
	}

	return func(yield func([]V) bool) {
		batch := []V{}
		for v := range it {
			batch = append(batch, v)
			if len(batch) == n {
				if !yield(batch) {
					return
				}

				batch = []V{}
			}
		}

		if len(batch) > 0 {
			yield(batch)
		}
	}
}

// ChunkIter2 works like slices.ChunkIter,
// but yields pairs of batches of size n.
// Panics if n is less than 1.
func ChunkIter2[I Seq2[K, V], K any, V any](it I, n int) Seq2[[]K, []V] {
	if n < 1 {
		panic("cannot be less than 1")
	}

	return func(yield func([]K, []V) bool) {
		kbatch := []K{}
		vbatch := []V{}

		for k, v := range it {
			kbatch = append(kbatch, k)
			vbatch = append(vbatch, v)

			if len(kbatch) == n {
				if !yield(kbatch, vbatch) {
					return
				}

				kbatch = []K{}
				vbatch = []V{}
			}
		}

		if len(kbatch) > 0 {
			yield(kbatch, vbatch)
		}
	}
}

Even though iterators are great, I would argue that implementing them results in code that is rather hard to read. Thus I find this snippet complex enough to be considered for addition into iter package.

Adding playground link that demonstrates and tests this implementation.

This is my first proposal for the gopher community, so please feel free to let me know if I should provide more details or update anything.

EDIT: After some consideration, I think ChunkSeq and ChunkSeq2 would probably be more appropriate than ChunkIter / ChunkIter2.

@gopherbot gopherbot added this to the Proposal milestone Jan 29, 2025
@seankhliao
Copy link
Member

there's some recent discussion of chunking in #61898, let's keep it in there for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants