Skip to content

mime/multipart: Reader returns part that contains boundary string #14085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tleyden opened this issue Jan 24, 2016 · 16 comments
Closed

mime/multipart: Reader returns part that contains boundary string #14085

tleyden opened this issue Jan 24, 2016 · 16 comments

Comments

@tleyden
Copy link

tleyden commented Jan 24, 2016

I'm seeing unexpected behavior with the Go HTTP MultipartReader when doing multipart/related PUT requests to a REST api which contain 1 JSON part and 3 image parts. When calling reader.NextPart(), it seems to be returning data for both the 3rd and the 4th parts rather than just the 3rd part. After digging into it further, I noticed that the data it returns for part 3 contains the boundary string itself, which definitely seems like it's breaking the parsing.

I've only seen this happen rarely, so I think it's dependent on the actual content of the data.

  • Go version: go1.5.1 darwin/amd64
  • OS/Arch: Centos 7 Linux 64-bit + OSX 64-bit

Steps to reproduce:

  • Run this unit test
  • Actual: test fails because it finds a part that contains a boundary string
  • Expected: no part should returned from reader.NextPart() contain the boundary string
  • Expected: the multipart reader should be seeing 4 parts, but it's only seeing 3. Should match up with what was seen in wireshark for the request.

A hypothesis:

I did notice that this particular image data happens to contain the byte sequence for \r\n:

hexdump

Could that be causing this line in multipart.go to incorrectly segment it?

line, err := r.bufReader.ReadSlice('\n')

I'd be more than happy to try to distill this down to a repeatable unit test with no external dependencies.

@bradfitz
Copy link
Contributor

This was fixed in Go 1.5.2: https://golang.org/doc/devel/release.html#go1.5.minor

(or at least, an identical-sounding bug was fixed. If you still see it with Go 1.5.2 or Go 1.5.3, let us know)

@tleyden
Copy link
Author

tleyden commented Jan 24, 2016

Thanks! I'll retest it w/ go 1.5.3

@tleyden
Copy link
Author

tleyden commented Jan 24, 2016

@bradfitz upgrading from 1.5.1 to 1.5.3 didn't change the outcome of the unit test -- the go multipart module still can't correctly parse that particular multipart request.

It might be caused by some subtlety about the way the original ObjectiveC-based client is sending the request, because in a second unit test I tried using a go http client to reproduce the issue by sending a multipart/related request with approximately the same content, and couldn't reproduce any issues in either go 1.5.1 or 1.5.3.

By "approximately the same content", I mean:

  • The json part wasn't gzipped as it was in the original problematic request
  • The boundary was different, -9PWGbT-1_r1QpgvW0I-4E3 in failing vs 1174c809f7f3b164d57df5f5f23c652994408f51d135352b26e26a96ff6e in passing
  • The image attachment parts were identical and in the same order

@bradfitz bradfitz reopened this Jan 24, 2016
@bradfitz bradfitz changed the title MultipartReader returns part that contains boundary string mime/multipart: Reader returns part that contains boundary string Jan 24, 2016
@bradfitz bradfitz added this to the Go1.6Maybe milestone Jan 24, 2016
@bradfitz
Copy link
Contributor

@adg, you have time to look at this? You'll be working a bit before me, otherwise I can look (my) Monday.

@tleyden
Copy link
Author

tleyden commented Jan 24, 2016

@bradfitz @adg I managed to reproduce it via the go http client in a third unit test -- the only difference from the second unit test is that it explicitly sets the boundary string to -9PWGbT-1_r1QpgvW0I-4E3 in the client side mulitpart writer.

Test output

@bradfitz
Copy link
Contributor

@tleyden, this bug report is already way above normal quality-wise, but if I could be greedy: can you whittle it down to a single file with no external data dependencies, suitable for writing a TestFoo(t *testing.T) function out of? Or even something that would work in play.golang.org? That's 95% of the work of fixing bugs, usually, writing a reproducible small test case.

@tleyden
Copy link
Author

tleyden commented Jan 24, 2016

Yeah np, I'll give it a shot and post to the issue if I can come up with something.

@tleyden
Copy link
Author

tleyden commented Jan 24, 2016

OK I managed to remove the external file dependencies and replaced them with zero'd out byte slices, and it's still reproducing the issue. Admittedly it's still a bit clunky since it starts an http listener and goes through a client round trip.

@adg
Copy link
Contributor

adg commented Jan 25, 2016

@tleyden I tried running your TestReadMultipartBody4 against Go commit 801bebe and the test passed correctly. What version are you testing against?

=== RUN   TestReadMultipartBody4
2016/01/25 14:45:48 Writer using boundary: -9PWGbT-1_r1QpgvW0I-4E3
2016/01/25 14:45:48 attrs: map[boundary:-9PWGbT-1_r1QpgvW0I-4E3]
2016/01/25 14:45:48 Part: &{map[Content-Type:[application/json]] 0xc8200c82a0 0xc820948180 0  map[] {0xc8209a81e0}}
2016/01/25 14:45:48 Part data length: 2
2016/01/25 14:45:48 Part: &{map[Content-Disposition:[attachment; filename="result_image".]] 0xc8200f4070 0xc820948180 0  map[] {0xc820010230}}
2016/01/25 14:45:48 Part data length: 68881
2016/01/25 14:45:48 Part: &{map[Content-Disposition:[attachment; filename="source_image".]] 0xc8200f40e0 0xc820948180 0  map[] {0xc8200102d0}}
2016/01/25 14:45:48 Part data length: 97655
2016/01/25 14:45:48 Part: &{map[Content-Disposition:[attachment; filename="style_image".]] 0xc8200f4150 0xc820948180 0  map[] {0xc8200105f0}}
2016/01/25 14:45:48 Part data length: 85053
2016/01/25 14:45:48 err getting nextpart: EOF
2016/01/25 14:45:48 response status code: 200
--- PASS: TestReadMultipartBody4 (0.00s)

@tleyden
Copy link
Author

tleyden commented Jan 25, 2016

I'm on go 1.5.3 on OSX:

$ go version
go version go1.5.3 darwin/amd64

I downloaded it from here and built it from source using make.bash on top of my previous go version (1.5.1)

@adg
Copy link
Contributor

adg commented Jan 25, 2016

I'm able to reproduce the issue at go1.5.3, but the test passes against HEAD (~go1.6).

The question is whether it is worth fixing and releasing Go 1.5.4, or whether we should just recommend upgrading to go1.6, which should be out in a couple of weeks.

@bradfitz
Copy link
Contributor

I don't remember fixes to this code this cycle. Did you narrow down which commit is responsible?

@adg
Copy link
Contributor

adg commented Jan 25, 2016

I'll try bisecting.

@adg
Copy link
Contributor

adg commented Jan 25, 2016

Looks like the fix happened in commit 821b549

@adg
Copy link
Contributor

adg commented Jan 25, 2016

Since this is fixed at HEAD, and Go 1.6 will be released soon, I'll close this issue.

@adg adg closed this as completed Jan 25, 2016
@tleyden
Copy link
Author

tleyden commented Jan 25, 2016

Works for me. Thanks for getting to the bottom of this!

@golang golang locked and limited conversation to collaborators Jan 24, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants