proposal: net/http: optimize http.Dir when files are not modified and add GODEBUG to preserve old behavior #64262
Comments
…UG to preserve old behavior Added a new GODEBUG environment variable. If GODEBUG=httpdircache=1, cache *os.File and []byte used by http.Dir.Open if the open file modification time is not changed. If GODEBUG=httpdircache=0, the old behavior is retained. go1.23 Default GODEBUG=httpdircache=1 . Fixes golang#64262
|
Change https://go.dev/cl/543636 mentions this issue: |
…UG to preserve old behavior Added a new GODEBUG environment variable. If GODEBUG=httpdircache=1, cache *os.File and []byte used by http.Dir.Open if the open file modification time is not changed. If GODEBUG=httpdircache=0, the old behavior is retained. go1.23 Default GODEBUG=httpdircache=1 . Fixes golang#64262
|
This significantly changes the memory propfile of a server using |
one A path should cache only one copy of *os.File and []byte into sync.Map. If this uses too much memory, I can modify the proposal and CL to cache only small files. |
|
As @seankhliao noted, caching can consume potentially unbounded memory. In addition, clients may want to be able to configure the cache policy, like how many files are cached, which files are cached, how long they are cached for, etc. It does not make sense for the standard library to impose one solution for the entire binary, nor does it make sense for the existing un-cached behavior to essentially be treated as deprecated, both of which are implied by the use of This should be a separate |
Proposal: net/http: optimize http.Dir when files are not modified and add GODEBUG to preserve old behavior
Author(s): @qiualidongfeng
Last updated: 2023-11-19
Abstract
If there are no errors, http.Dir calls os.Open once every time Open is called. If the file has not changed, performance can be improved by reducing system calls by caching *os.File and []byte read from *os.File.
Background
Background as described above.
Proposals to optimize http.Dir.
Proposal
http.Dir.Open When the file modification time does not change, use the cached *os.FIle and []byte.
Add the GODEBUG environment variable httpdircache, go1.23 defaults to 1, enabling the behavior after the proposal implementation. Setting the value to 0 restores the previous behavior.
Rationale
1, directly change the behavior, when the file modification time is unchanged but the content changes, backward incompatible.
2, Third-party libraries, such as https://github.com/qiulaidongfeng/cachefs , This is a prototype written in a third-party library before sending CL to the standard library. The advantage is that it can be used less than go1.23, and the disadvantage is that the code needs to be manually modified to use.
Compatibility
100% backward compatibility with godebug=httpdircache=0.
Implementation
I will send CL to implement the proposal, which is too close to go1.22 code freezing, so this proposal is for go1.23.
The text was updated successfully, but these errors were encountered: