Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7.3 linux/amd64

What did you do?

net/http has support to honor If-Modified-Since headers in http.ServeFile().

If I downgrade an application using a package manager (e.g. via rpm), the If-Modified-Since would always result in a 304 Not Modified.

src/net/http/fs.go contains the following:

if t, err := time.Parse(TimeFormat, r.Header.Get("If-Modified-Since")); err == nil && modtime.Before(t.Add(1*time.Second)) {
    ...
}

My proposal would be to have a configuration variable to only respond with 304 Not Modified on an exact match.

The exact match would be something like modtime.Truncate(time.Second).Equal(t.Truncate(time.Second))

As an example for another http server, nginx allows to serve files with "exact" match or "before" match and defaults to "exact" match:
http://nginx.org/en/docs/http/ngx_http_core_module.html#if_modified_since