Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: /etc/hosts parsing truncates to 4KB (os.GetPagesize) #21674

Closed
alexzorin opened this issue Aug 28, 2017 · 5 comments
Closed

net: /etc/hosts parsing truncates to 4KB (os.GetPagesize) #21674

alexzorin opened this issue Aug 28, 2017 · 5 comments

Comments

@alexzorin
Copy link

Due to the way net/file,open is implemented, pure-Go DNS resolution cannot handle /etc/hosts files that are larger than the OS page size, which is 4KB by default on Linux platforms.

Go silently truncates the data after 4KB and does not parse lines on the threshold, since EOL is never reached for them.

The cgo equivalents for this functionality handle these files properly, and so should Go.

We often see these large hosts files in the web hosting environment (e.g. cPanel), when servers are behind NAT and require large hosts files.

Env: Go1.9, linux/amd64

I have also attached a Dockerfile tarball that reproduces the issue fully - go-4kb-hosts.tar.gz

@odeke-em
Copy link
Member

Interesting! Thank you @alexzorin for reporting this issue. Just for easier lookup in the future, here is the permalink to the code you referred to

go/src/net/parse.go

Lines 67 to 72 in 93471a1

func open(name string) (*file, error) {
fd, err := os.Open(name)
if err != nil {
return nil, err
}
return &file{fd, make([]byte, 0, os.Getpagesize()), false}, nil
.

@odeke-em odeke-em added this to the Go1.10 milestone Aug 29, 2017
@odeke-em
Copy link
Member

@rsc I suppose that the seemingly arbitrary os.Getpagesize() limit was set because we wanted a performance hit from reads on page boundaries or that we thought that size was sufficient? I don't see a file limit for a hosts file in either of the below:

@gopherbot
Copy link

Change https://golang.org/cl/65671 mentions this issue: net: parse file lines longer than one page

@alexzorin
Copy link
Author

alexzorin commented Nov 17, 2017

Hello,

I've been tracking the CL and I'm not sure how to add my input, sorry if I'm doing it wrong.

I saw a comment by @rsc that worries me a little:

Or at least I'm OK with not seeing lines > 4kB

I originally ran into this issue with customers exactly because of entries on specific lines that exceeded 4kB (big NAT-ted cPanel servers). If the system resolver deals with it without a problem I would not expect Go to introduce such a surprising and arbitrary limitation here.

I apologise if the original issue made it look like the only concern was the remainder of the file - properly parsing long lines is equally important to my application.

@gopherbot
Copy link

Change https://golang.org/cl/79516 mentions this issue: net: accept 64 kB lines in /etc/hosts

@golang golang locked and limited conversation to collaborators Nov 28, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants