Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

encoding/base32: decoder output depends on chunking of underlying reader #38657

Closed
AxbB36 opened this issue Apr 25, 2020 · 2 comments
Closed

encoding/base32: decoder output depends on chunking of underlying reader #38657

AxbB36 opened this issue Apr 25, 2020 · 2 comments
Labels
FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@AxbB36
Copy link

AxbB36 commented Apr 25, 2020

#31626 is about a problem that affects encoding/base64. A similar problem affects encoding/base32.

A decoder created by NewDecoder is sensitive to how its input is split into pieces, and it should not be. Whether the Decoder interprets the input as valid may depend on whether the underlying Reader yields one big byte slice, or two smaller byte slices, for example.

What version of Go are you using (go version)?

$ go version
go version go1.14.2 linux/amd64

Does this issue reproduce with the latest release?

Yes, with go1.14.2 on play.golang.org.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"

What did you do?

https://play.golang.org/p/KV4XPizhYj9

package main

import (
	"encoding/base32"
	"fmt"
	"io"
	"io/ioutil"
)

func test(chunks [][]byte) {
	pr, pw := io.Pipe()
	go func() {
		for _, chunk := range chunks {
			pw.Write(chunk)
		}
		pw.Close()
	}()
	output, err := ioutil.ReadAll(base32.NewDecoder(base32.StdEncoding, pr))
	fmt.Printf("%+q -> %+q %v\n", chunks, output, err)
}

func main() {
	input := []byte("I4======N4======")
	// input := []byte("I5XQ===============")
	for i := 0; i < len(input)+1; i++ {
		test([][]byte{input[:i], input[i:]})
	}
}

You can also try this input to see inconsistent error offsets (goes from index 4, to 0, back to 4):

	input := []byte("I5XQ===============")

What did you expect to see?

The decoder's output should be consistent, regardless of the chunking of the underlying reader.

["" "I4======N4======"] -> "" illegal base32 data at input byte 2
["I" "4======N4======"] -> "" illegal base32 data at input byte 2
["I4" "======N4======"] -> "" illegal base32 data at input byte 2
["I4=" "=====N4======"] -> "" illegal base32 data at input byte 2
["I4==" "====N4======"] -> "" illegal base32 data at input byte 2
["I4===" "===N4======"] -> "" illegal base32 data at input byte 2
["I4====" "==N4======"] -> "" illegal base32 data at input byte 2
["I4=====" "=N4======"] -> "" illegal base32 data at input byte 2
["I4======" "N4======"] -> "" illegal base32 data at input byte 2
["I4======N" "4======"] -> "" illegal base32 data at input byte 2
["I4======N4" "======"] -> "" illegal base32 data at input byte 2
["I4======N4=" "====="] -> "" illegal base32 data at input byte 2
["I4======N4==" "===="] -> "" illegal base32 data at input byte 2
["I4======N4===" "==="] -> "" illegal base32 data at input byte 2
["I4======N4====" "=="] -> "" illegal base32 data at input byte 2
["I4======N4=====" "="] -> "" illegal base32 data at input byte 2
["I4======N4======" ""] -> "" illegal base32 data at input byte 2

What did you see instead?

["" "I4======N4======"] -> "" illegal base32 data at input byte 2
["I" "4======N4======"] -> "" illegal base32 data at input byte 2
["I4" "======N4======"] -> "" illegal base32 data at input byte 2
["I4=" "=====N4======"] -> "" illegal base32 data at input byte 2
["I4==" "====N4======"] -> "" illegal base32 data at input byte 2
["I4===" "===N4======"] -> "" illegal base32 data at input byte 2
["I4====" "==N4======"] -> "" illegal base32 data at input byte 2
["I4=====" "=N4======"] -> "" illegal base32 data at input byte 2
["I4======" "N4======"] -> "Go" <nil>
["I4======N" "4======"] -> "Go" <nil>
["I4======N4" "======"] -> "Go" <nil>
["I4======N4=" "====="] -> "Go" <nil>
["I4======N4==" "===="] -> "Go" <nil>
["I4======N4===" "==="] -> "Go" <nil>
["I4======N4====" "=="] -> "Go" <nil>
["I4======N4=====" "="] -> "Go" <nil>
["I4======N4======" ""] -> "" illegal base32 data at input byte 2
@ianlancetaylor ianlancetaylor added help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 25, 2020
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Apr 25, 2020
@pelletier197
Copy link

I encountered the same issue with base64 decoder. I sadly have a use-case where I have a gigantic base64 string split in multiple parts, that I put back together and decode using the decoder (for memory issues), but in some cases, the output decoded is missing some bytes.

Had to go with the "load all the string in memory and decode it one shot", which is way more memory consuming.

@seankhliao seankhliao mentioned this issue Jan 22, 2022
Closed
@teivah teivah mentioned this issue Apr 30, 2022
Closed
@gopherbot
Copy link

Change https://go.dev/cl/403315 mentions this issue: encoding/base32: decoder output depends on chunking of underlying reader

@golang golang locked and limited conversation to collaborators May 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

Successfully merging a pull request may close this issue.

encoding/base32: decoder output depends on chunking of underlying reader teivah/go
4 participants
@ianlancetaylor @gopherbot @AxbB36 @pelletier197