Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mime/multipart: unable to process bigger parts #33666

Closed
danielkucera opened this issue Aug 15, 2019 · 4 comments
Closed

mime/multipart: unable to process bigger parts #33666

danielkucera opened this issue Aug 15, 2019 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@danielkucera
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.10.4 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/u/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/yw674/go"
GORACE=""
GOROOT="/usr/lib/go-1.10"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build154661289=/tmp/go-build -gno-record-gcc-switches"

What did you do?

func readStream(){
        resp, err := http.Get(device.Url)
        if err != nil {
                // handle error
        }
        defer resp.Body.Close()
        log.Printf("%v+", resp.Header)

        contentType := resp.Header.Get("Content-Type");
        if len(contentType) < 1 {
                return
        }

        mediaType, mediaTypeParams, err := mime.ParseMediaType(contentType)
        fmt.Println("parseMessageWithHeader", mediaType)
        if err != nil {
                return
        }
        
        boundary := mediaTypeParams["boundary"]
        log.Printf("boundary %s", boundary)

        mr := multipart.NewReader(resp.Body, boundary)
        for {
                p, err := mr.NextPart()
                if err == io.EOF {
                        return
                }
                if err != nil {
                        log.Fatal("np: ", err)
                }
                slurp, err := ioutil.ReadAll(p)
                if err != nil {
                        log.Fatal("io: ", err)
                }
                fmt.Printf("Part %q: %q\n", p.Header.Get("Foo"), slurp)
        }
}

What did you expect to see?

MJPEG chunks to be read to slurp.

What did you see instead?

2019/08/15 13:30:16 map[Content-Type:[multipart/x-mixed-replace; boundary=--myboundary] Date:[Thu, 15 Aug 2019 11:30:16 GMT]]+
parseMessageWithHeader multipart/x-mixed-replace
2019/08/15 13:30:16 boundary --myboundary
2019/08/15 13:30:16 np: multipart: NextPart: bufio: buffer full

I suspect this is due to peekBufferSize is only 4096 bytes:

const peekBufferSize = 4096

And the single parts of multipart are much longer (few hundreds of kBs)
Would it be possible to make this value configurable for example by adding NewReader method with configurable buffer size?
func NewReader(r io.Reader, boundary string) *Reader {

Shall I create a PR?

@danielkucera
Copy link
Author

something like this:
69cb30b

@andybons
Copy link
Member

@bradfitz @minux

@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 15, 2019
@andybons andybons added this to the Unplanned milestone Aug 15, 2019
@danielkucera danielkucera changed the title mime/multipart: unable to process MJPEG stream mime/multipart: unable to process bigger parts Aug 19, 2019
@danielkucera
Copy link
Author

My bad, the boundary returned from mime.ParseMediaType

boundary := mediaTypeParams["boundary"]

contains leading dashes:

2019/08/15 13:30:16 boundary --myboundary

After striping them out, everything works ok.

@andybons
Copy link
Member

@danielkucera glad you found the issue. Thanks for following up.

@golang golang locked and limited conversation to collaborators Aug 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

3 participants