New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: time.Sleep not monotonic if you import os/user & have used setcap to give binding capabilities #42150
Comments
perhaps CGo related due to os/user import but I can't reproduce 19:32:08 ~/testrepo-199 0:00:00
main » go build .
19:32:16 ~/testrepo-199 0:00:01
main » sudo setcap cap_net_bind_service=+ep testrepo-199
19:32:20 ~/testrepo-199 0:00:00
main » ./testrepo-199
2020-10-22 19:32:27.263129711 +0200 CEST m=+0.000057595 Uptime: 154ns
2020-10-22 19:32:32.263498017 +0200 CEST m=+5.000426013 Uptime: 5.000368853s
2020-10-22 19:32:37.263981624 +0200 CEST m=+10.000909642 Uptime: 10.000852462s
2020-10-22 19:20:00.410276497 +0200 CEST m=+15.001422131 Uptime: 15.001364944s
2020-10-22 19:20:05.410748758 +0200 CEST m=+20.001894382 Uptime: 20.001837211s
2020-10-22 19:20:10.411244842 +0200 CEST m=+25.002390468 Uptime: 25.002333275s
^C
19:20:11 ~/testrepo-199 0:12:16
main » cat main.go
package main
import (
"fmt"
"os/user"
"time"
)
func main() {
start := time.Now()
for {
fmt.Printf("%v Uptime: %v\n", time.Now(), time.Now().Sub(start))
time.Sleep(5 * time.Second)
}
cu := user.Current
fmt.Printf("never get here %v", cu)
} while the program was running: 19:32:23 ~ 0:00:00
» sudo timedatectl set-ntp 0
19:32:30 ~ 0:00:00
» sudo timedatectl set-time 19:20:00 |
If you import os/user, and set the capability, and run the program, and set the clock backward, you say that the output stops? Do you mean that the program keeps running, it just doesn't print anything? What happens if you wait until the clock catches up? A call to It might help to run your program, in the failing case, under |
In this incarnation we don't return from the sleep. When this was originally encountered on a full sized application every sleep that was in progress also waited for (original sleep time + time we set the clock back). We did still see inbound web requests being served.. so the whole application wasn't stuck.. just things stuck on sleep. I should add that while we didn't reproduce this in the sample code provided.. a ticker also suffered the lack of monotomic clock-ness.
We were able to reproduce this in: go version go1.12.7 linux/amd64 If that helps isolate. So far we can reproduce this on our embedded device plus a vm that is running a standard ubuntu (18.04?).. while another ubuntu box doesn't exhibit the problem. Very much feels like a "per system" problem. I'm not sure how go determines if a monotomic clock is available..etc.. maybe that is busted up.
Let me do that. |
Well.. the plot thickens. Doing a:
The problem goes away. Tried it several times with several pairs of eyes. |
I do see a potential problem here. The Go runtime will check for the VDSO Interestingly, this has been fixed on tip for the future 1.16 release, by https://golang.org/cl/257982. So while I don't know what is going wrong, it's possible that this will be fixed in 1.16. You might try applying that patch to your 1.15 sources to see if it helps. |
I do remember something about setcap possibly mucking with LD_LIBRARY_PATH in an attempt to avoid escalation -- but my skillset didn't let me dig deeper into that angle. |
Either release-branch.go1.15 with https://golang.org/cl/257982 applied on top, or just plain master. Thanks. |
Master does indeed rectify the problem. Is there anything else I can do to help out? It is very puzzling how os/user works into things.. It looks like our workarounds are:
|
FWIW, I dug into the kernel a bit and verified that it does unconditionally include the VDSO auxv entry, so it shouldn't be missing at the time of exec. It seems most likely then that the dynamic linker would be removing / changing the entry, though I've never heard of such behavior (some sort of "protection" for "setuid" binaries?). Do you know which dynamic linker your system is using? I looked at glibc but didn't see anything along these lines at a quick glance. This would align with importing "os/user", as that will cause the program to become dynamically linked by default. |
And to your question, it seems like static linking (by disabling cgo, I assume that is what you mean?) seems like the cleanest workaround to me, provided the pure Go versions of "os/user" and "net" are sufficient for your application. |
Since this will be fixed in the upcoming Go 1.16 release, I'm going to close this issue. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
If I compile that into a binary, and give the bind capability to the binary via:
And then I set the clock backwards 5minutes.
When I took out the import to os/user -- the problem goes away.
What did you expect to see?
I would expect to continue to see the "Uptime" messages to be printed every 5 seconds
What did you see instead?
Output stopped
The text was updated successfully, but these errors were encountered: