Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/tls: set timeout so that Close does not block indefinitely #31224

Closed
NWilson opened this issue Apr 3, 2019 · 35 comments
Closed

crypto/tls: set timeout so that Close does not block indefinitely #31224

NWilson opened this issue Apr 3, 2019 · 35 comments
Labels
FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@NWilson
Copy link

NWilson commented Apr 3, 2019

What did you do?

Reading the documentation for tls.Conn (https://golang.org/pkg/crypto/tls/) the deadline behaviour is not clearly explained.

  • It's not obvious that Read calls may block forever unless the write deadline is set to a non-zero time. (This is because handshaking is done internally, which writes.) Since this is different to net.TLSConn, it should be called out.
  • It's reasonable to understand that Handshake calls need to have both read and write deadline set... but could be called out explicitly.
  • But the killer: calling Close on a tls.Conn risks blocking forever, unless the write deadline has been set! (It writes out a close alert.)

This last one is the worst. It would be very easy to have a defer'ed call to Close, and assume that it will always return promptly, when in fact it may be possible for a malicious client to fill up the socket buffers with handshaking data, so that writing the close alert blocks (on an OS with tiny socket buffers?).

I have (or had) some code that does the following:

for {
  conn, _ := listener.Accept() // error handling omitted for clarity
  if tooManyConns() {
    conn.Close() // oops, may block!
    continue
  }
  go handleConn(conn)
}

I have now carefully gone and wrapped all calls to Close with a safeClose wrapper method that calls SetWriteDeadline first!

Perhaps Close should actually call SetWriteDeadline on the underlying connection with a sane value (eg time.Now().Add(100*time.Millisecond)) to ensure that no-one else is caught out.

Thankfully, http.conn.serve does call SetWriteDeadline as soon as possible for TLS connections, and never sets a zero write deadline, so that code at least is safe, and can't call Close with a zero deadline.

What did you expect to see?

More mention of what deadlines need to set, to prevent each method blocking.

@bradfitz bradfitz changed the title tls.Conn.Close documentation should explain blocking crypto/tls: Conn.Close documentation should explain blocking Apr 3, 2019
@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 3, 2019
@andybons andybons added this to the Unplanned milestone Apr 3, 2019
@andybons
Copy link
Member

andybons commented Apr 3, 2019

@FiloSottile

@networkimprov
Copy link

networkimprov commented Apr 5, 2019

Assuming the above points are all correct, a lot of folks use the tls API incorrectly. I have a TLS+TCP server that calls SetReadDeadline() and Close(). I don't recall mention of SetWriteDeadline() in any example code I read.

Maybe this calls for additional APIs, e.g. SetTLSReadDeadline() & CloseTLS()? Or a go vet item?

@networkimprov
Copy link

Ping @FiloSottile

@networkimprov
Copy link

@andybons, could this be milestoned for 1.13 and made release-blocker?

Correct use of TLS is a security concern, and one of the suggested fixes is to change .Close() behavior.

@FiloSottile
Copy link
Contributor

I agree this needs addressing, but it does not warrant changing subtle behavior while in deep feature freeze. The only fix for Go 1.13 will be probably in documentation, and we can discuss in 1.14 with more flexibility.

@FiloSottile FiloSottile modified the milestones: Unplanned, Go1.13 Jun 26, 2019
@networkimprov
Copy link

Thanks, happy to give feedback on the CL when ready...

@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@networkimprov
Copy link

networkimprov commented Jul 16, 2019

@FiloSottile, you tagged this for 1.13. It's now been re-tagged for 1.14; was that a mistake?

@networkimprov
Copy link

@FiloSottile we might want to tackle this one early in 1.14 cycle...

@gopherbot add release-blocker

@FiloSottile
Copy link
Contributor

I'm thinking of introducing a default deadline on the close notify send, and document in Read and Write, where they say they call Handshake, that that means they need both read and write deadlines.

@networkimprov
Copy link

Re setting a default deadline, is there any conflict with http.Server/Client? See #21133 (comment)

@FiloSottile
Copy link
Contributor

Reusing a net.Conn after a tls.Close is weird but I suppose not impossible? Sure, we can just reset it before leaving Close.

@networkimprov
Copy link

Just wondering whether the implicit deadlines that http uses might conflict with an implicit deadline in tls.Close?

@FiloSottile
Copy link
Contributor

Ah, well by the time tls.Close is called it doesn't really matter, does it? We are tearing down, so we can just override them.

@networkimprov
Copy link

@bradfitz any thoughts?

@bradfitz
Copy link
Contributor

bradfitz commented Oct 2, 2019

I haven't looked into this issue, but from a quick skim of this thread, it does indeed seem like things are too complicated/surprising for users.

@networkimprov
Copy link

@bradfitz any comment re a new implicit timer in tls.Close? Could that conflict with implicit timers in http?

@bradfitz
Copy link
Contributor

bradfitz commented Oct 2, 2019

A new timer in tls.Close sounds good. If crypto/tls doesn't and net/http doesn't already(?), then it seems that net/http should work around any such blocking and close the underlying net.Conn directly. But it might be already. I'd have to go refresh my memory of the code.

In any case, crypto/tls doing it (regardless of whether a higher layer is also doing it) sounds good.

@ianlancetaylor
Copy link
Contributor

Looks like no change was made for 1.14, so retargeting for 1.15. Also looks like the current thinking is to change the code, so dropping the documentation label.

@ianlancetaylor ianlancetaylor changed the title crypto/tls: Conn.Close documentation should explain blocking crypto/tls: set timeout so that Close does not block indefinitely Dec 5, 2019
@ianlancetaylor ianlancetaylor removed this from the Go1.14 milestone Dec 5, 2019
@networkimprov
Copy link

networkimprov commented Oct 17, 2020

Could we target this for 1.16?

cc @odeke-em in case it's of interest...

@FiloSottile
Copy link
Contributor

The changes here are

  • set a Deadline in Close of 5s before sending and trying to read the closeNotify
  • call attention to the fact that Read?Write call Handshake so need both deadlines in their docs

@FiloSottile FiloSottile modified the milestones: Backlog, Go1.16 Oct 20, 2020
@katiehockman katiehockman self-assigned this Oct 28, 2020
@gopherbot
Copy link

Change https://golang.org/cl/266037 mentions this issue: crypto/tls: set Deadline before sending close notify alert

@networkimprov
Copy link

Could the docs also give guidance on the deadline durations for Read & Write?

@katiehockman
Copy link
Contributor

I think that can be in a separate issue/CL. If we were to provide guidance about an appropriate deadline, then that would be more appropriate in the godoc for SetDeadline, SetReadDeadline, and/or SetWriteDeadline, not in Read or Write.
We would also need to determine what a sensible deadline would be, or how to provide guidance for that.

@networkimprov
Copy link

Since you have to set a write deadline on Read, and a read deadline on Write, it would be helpful to know if there's a minimum value necessary in TLS.

@katiehockman
Copy link
Contributor

Sure. Feel free to file another issue. That is out of scope for this one.

@kozlovic
Copy link

kozlovic commented Mar 4, 2021

@katiehockman @FiloSottile This is a bit unfortunate. I knew it could block, so I was setting my own deadline of 100ms (did not really care if the alert is sent or not, but I did not want it to block). With this change, my code now will block up to 5 seconds. Anything I can do to close the low level connection without blocking (up to 5sec) here?

@networkimprov
Copy link

Can I ask what services or clients are stalling on close often enough to cause problems?

@kozlovic
Copy link

kozlovic commented Mar 4, 2021

@networkimprov This is in context of NATS Server. When a client is closed, I try to flush and close the connection. Because of TLS, I was setting a low deadline. I have a test that produced the condition that may cause the TLS close to stall and they started to fail, and I realize that this is because of Go overriding my own deadline: https://github.com/nats-io/nats-server/blob/f82ba22be252755aefbf33af8d66db8cf7421b53/server/client.go#L4352
Here is one of the test that checks for that: https://github.com/nats-io/nats-server/blob/f82ba22be252755aefbf33af8d66db8cf7421b53/server/routes_test.go#L1185

@networkimprov
Copy link

Could you open a new issue to suggest that TLS close not override an existing deadline, or that a tls.Config field be added to let you specify the close deadline? And then mention that issue here. Thanks!

@networkimprov
Copy link

Ping @kozlovic re new issue...

@kozlovic
Copy link

@networkimprov Sorry for delay, took some time off. I created the issue #45162.

jkralik added a commit to plgd-dev/go-coap that referenced this issue May 12, 2021
jkralik added a commit to plgd-dev/go-coap that referenced this issue May 13, 2021
jkralik added a commit to plgd-dev/go-coap that referenced this issue May 13, 2021
jkralik added a commit to plgd-dev/go-coap that referenced this issue May 13, 2021
jkralik added a commit to plgd-dev/hub that referenced this issue May 14, 2021
jkralik added a commit to plgd-dev/hub that referenced this issue May 14, 2021
@golang golang locked and limited conversation to collaborators Mar 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

9 participants