Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mime/multipart: part.Reader hangs on an open stream even if entire multipart data is available #15431

Closed
blixt opened this issue Apr 24, 2016 · 5 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@blixt
Copy link
Contributor

blixt commented Apr 24, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    go1.6.2
  2. What operating system and processor architecture are you using (go env)?
    darwin/amd64 (Mac OS/X 10.11.4)
  3. What did you do?
    http://play.golang.org/p/D5so_L8JDD
  4. What did you expect to see?
    At the very least, all available parts successfully read before the reader hangs on more data from the socket. In this particular example the multipart reader can even reach "EOF" because once it reaches the postamble, it's impossible for there to be any more parts.
  5. What did you see instead?
    The multipart reader hangs before even reading the first part. Presumably because it's trying to peek ahead but the socket doesn't have enough data available for the peek buffer.
@blixt
Copy link
Contributor Author

blixt commented Apr 25, 2016

For some background, the reason I want to use multipart.Reader with an open stream is because of Amazon's new AVS API which involves creating a long-lived HTTP/2 request that progressively receives data in the form of multipart parts.

I did some experiments and the multipart.Part's Read method seems to block on bufio.Reader's Peek method. It doesn't seem possible to make bufio work with open streams that are shorter than the buffer length, so this might be tricky to fix. I even tried making a non-blocking reader to see how it would behave, but bufio.Reader just errors out.

@blixt
Copy link
Contributor Author

blixt commented Apr 26, 2016

For anyone else who runs into this issue, here's a working reimplementation of mime/multipart which works on both open and closed streams: https://github.com/rogertalk/go-avs/tree/master/multipart2

A few caveats:

  • I didn't implement quoted-printable support
  • It won't be as forgiving for incorrect implementations (e.g., whitespace after boundary)
  • It uses a bufio.Reader on top of a private implementation similar to bufio which wastes some memory (the reason is because textproto.NewReader(…) requires *bufio.Reader instead of io.Reader)
  • Probably a few other things are missing too since I only needed this for a specific use case

@bradfitz bradfitz added this to the Go1.7 milestone Apr 26, 2016
@bradfitz bradfitz added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 20, 2016
@rsc rsc modified the milestones: Go1.8, Go1.7 May 27, 2016
@quentinmit quentinmit assigned quentinmit and unassigned adg Oct 24, 2016
@gopherbot
Copy link

CL https://golang.org/cl/32032 mentions this issue.

@quentinmit quentinmit added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 25, 2016
@blixt
Copy link
Contributor Author

blixt commented Oct 29, 2016

This is fixed by CL https://golang.org/cl/32092 instead so I'm closing this issue.

@blixt blixt closed this as completed Oct 29, 2016
@quentinmit
Copy link
Contributor

We still need to add a test.

@quentinmit quentinmit reopened this Oct 30, 2016
@golang golang locked and limited conversation to collaborators Nov 7, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

6 participants