Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/mail: characters allowed in RFC 5322 are invalid while parsing email header #58862

Closed
iredmail opened this issue Mar 4, 2023 · 11 comments
Closed
Assignees
Labels
NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@iredmail
Copy link

iredmail commented Mar 4, 2023

What version of Go are you using (go version)?

$ go version
go version go1.20.1 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What did you do?

Create an empty directory and save code below in main.go, run go mod tidy && go run ..
Note: it contains header Custom/Header: v.

package main

import (
	"log"
	"net/mail"
	"strings"
)

func main() {
	msg := `Date: Mon, 23 Jun 2015 11:40:36 -0400
From: Gopher <from@example.com>
To: Another Gopher <to@example.com>
Subject: Gophers at Gophercon
Custom/Header: v

Message body
`

	r := strings.NewReader(msg)
	m, err := mail.ReadMessage(r)
	if err != nil {
		log.Fatal(err)
	}

	println(m.Header.Get("Custom/Header"))
}

What did you expect to see?

Print value of email header Custom/Header: v

What did you see instead?

2023/03/04 20:57:52 malformed MIME header line: Custom/Header: v
exit status 1

Function mail.ReadMessage() calls textproto.ReadMIMEHeader(), it calls internal function validHeaderValueByte() (in net/textproto) to validate characters in email header field, / in header field is considered as invalid.

The problem is, validHeaderValueByte() is designed to validate characters in HTTP header, not email header.

According to RFC 5322 "Internet Message Format", section "2.2 Header fields":

   A field name MUST be composed of printable US-ASCII characters (i.e.,
   characters that have values between 33 and 126, inclusive), except
   colon.

Here's full list of printable US-ASCII characters:
https://www.ascii-code.com/characters/printable-characters

For example, characters /, *, [, ] are allowed in email header field according to RFC doc, but it won't pass current validHeaderFieldByte() because it's disallowed in http header field. This causes parse error as shown in above sample code. I believe there're some similar issues caused by validHeaderValueByte() too.

We may need a new exported function like ReadMIMEHeader but for email (maybe name it as ReadEmailMIMEHeader()), and allow characters as defined in RFC 5322, but other code in net/textproto may need some changes too.

@iredmail iredmail changed the title affected/package: net/mail net/mail: characters allowed in RFC are invalid in mail header field Mar 4, 2023
@iredmail iredmail changed the title net/mail: characters allowed in RFC are invalid in mail header field net/mail: characters allowed in RFC 5322 are invalid while parsing email header Mar 4, 2023
@ianlancetaylor ianlancetaylor added help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 6, 2023
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Mar 6, 2023
@bcmills
Copy link
Contributor

bcmills commented Mar 6, 2023

https://dev.golang.org/owners lists @bradfitz as the maybe-owner of net/mail?

@mengzhuo
Copy link
Contributor

mengzhuo commented Mar 7, 2023

Duplicate #39591

@iredmail
Copy link
Author

iredmail commented Mar 7, 2023

Duplicate #39591

Not a duplicate. This issue is for email header field, but #39591 is for header value.

@180909
Copy link
Contributor

180909 commented Mar 8, 2023

It's working on version 1.19

@iredmail
Copy link
Author

iredmail commented Mar 8, 2023

It's working on version 1.19

Yes it works in Go v1.19, but changed in v1.20 due to the fix in this issue: #53188

Again, net/textproto is (mostly) designed for http, not for email.

@seankhliao
Copy link
Member

cc @neild

@neild neild self-assigned this Mar 8, 2023
@mediumdaver
Copy link

mediumdaver commented Apr 17, 2023

The problem code in net/mail/message.go is:

    53  func ReadMessage(r io.Reader) (msg *Message, err error) {
    54  	tp := textproto.NewReader(bufio.NewReader(r))
    55  
    56  	hdr, err := tp.ReadMIMEHeader()
    57  	if err != nil {
    58  		return nil, err
    59  	}
    60  
    61  	return &Message{
    62  		Header: Header(hdr),
    63  		Body:   tp.R,
    64  	}, nil
    65  }

Using textproto.ReadMIMEHeader() is incorrect, as a valid header in textproto is, per the comment in that code:

   655  // validHeaderFieldByte reports whether c is a valid byte in a header
   656  // field name. RFC 7230 says:
   657  //
   658  //	header-field   = field-name ":" OWS field-value OWS
   659  //	field-name     = token
   660  //	tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
   661  //	        "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA
   662  //	token = 1*tchar

which is incorrect for a mail header, RFC 5322 says a valid header name is:

Header fields are lines beginning with a field name, followed by a
colon (":"), followed by a field body, and terminated by CRLF.  A
field name MUST be composed of printable US-ASCII characters (i.e.,
characters that have values between 33 and 126, inclusive), except
colon.

Can someone please fix it? ....please....

@ianlancetaylor
Copy link
Contributor

@gopherbot Please open backport to 1.20.

This stopped working due to the changes in #53188, but those changes don't apply to net/mail.

@gopherbot
Copy link

Change https://go.dev/cl/504416 mentions this issue: net/mail: permit more characters in mail headers

@gopherbot
Copy link

Backport issue(s) opened: #60875 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 19, 2023
@dmitshur dmitshur modified the milestones: Backlog, Go1.21 Jun 19, 2023
@gopherbot
Copy link

Change https://go.dev/cl/504881 mentions this issue: [release-branch.go1.20] net/mail: permit more characters in mail headers

gopherbot pushed a commit that referenced this issue Jun 21, 2023
We parse mail messages using net/textproto. For #53188, we tightened
up the bytes permitted by net/textproto to match RFC 7230.
However, this package uses RFC 5322 which is more permissive.
Restore the permisiveness we used to have, so that older code
continues to work.

For #58862
For #60332
Fixes #60874
Fixes #60875

Change-Id: I5437f5e18a756f6ca61c13c4d8ba727be73eff9a
Reviewed-on: https://go-review.googlesource.com/c/go/+/504881
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
bradfitz pushed a commit to tailscale/go that referenced this issue Jul 15, 2023
We parse mail messages using net/textproto. For golang#53188, we tightened
up the bytes permitted by net/textproto to match RFC 7230.
However, this package uses RFC 5322 which is more permissive.
Restore the permisiveness we used to have, so that older code
continues to work.

Fixes golang#58862
Fixes golang#60332

Change-Id: I5437f5e18a756f6ca61c13c4d8ba727be73eff9a
Reviewed-on: https://go-review.googlesource.com/c/go/+/504416
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

10 participants