New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
net/http/cookiejar: Secured, HttpOnly Cookie values are not persisted in http.Client.Jar #27138
Comments
@sadhasivam in you example I see error https://play.golang.org/p/Lhi2ZD_95AJ:
|
@andriisoldatenko thanks for trying out. we may need to |
any thoughts on this issue please ? |
I can't remember the details (maybe @vdobler does), but the Go cookie jar intentionally doesn't return all of the fields that were put into it. https://golang.org/src/net/http/cookiejar/jar.go#L219 is:
In other words, we explicitly return only Name and Value, not all of a Cookie's fields (including Secure and HttpOnly, but also e.g. Path, Expires, MaxAge and Raw). As the CookieJar.Cookies doc comment says, the method "returns the cookies to send in a request". When sending cookies in a request, you don't need the Path, Secure or other fields, only the Name and Value fields. Stepping back, this sounds a bit like an XY problem. You say, "I am trying to automate certain website user behavior using default HTTP/net package". You should be able automate user behavior (e.g. send HTTP requests) by sending only Name and Value. HTTP servers shouldn't need the HttpOnly or Secure fields. What sort of HTTP request isn't working? What sort of error are you getting? For example, if I point my browser (with dev tools enabled) at https://www.chicos.com/, while the cookies that the browser receives have the HTTP and Secure fields set, it looks like the cookies the browser sends in follow-up HTTP requests do not. |
Well, HTTPOnly Cookies are a more secure form of Cookie. ( https://www.owasp.org/index.php/HttpOnly ) . Here is my Use case: |
As that page you linked to says, HttpOnly is part of Set-Cookie HTTP header, which means it's part of a HTTP Response. HttpOnly is not part of the Cookie HTTP header, which would be a part of the HTTP Request. See https://en.wikipedia.org/wiki/List_of_HTTP_header_fields
You say that the HttpOnly and Secure fields are not getting persisted. I don't think that this matters. As I said, HTTP requests don't have these fields, so I don't think their lack, coming out of the cookie jar, is affecting the orchestration. As I said, it doesn't look like browsers set HttpOnly or Secure in their requests. With your https://www.chicos.com code example, you could hard-code re-adding the fields (like the XXX lines in https://play.golang.org/p/po4mqXhTUWv) before passing those cookies on to whatever makes your HTTP requests. I'm not saying that's a general solution, but it would indicate whether changing the cookie jar behavior would fix your underlying problem. But I suspect that it won't. |
It is. The relevant test is in https://golang.org/src/net/http/cookiejar/jar_test.go#L628 the cookie d=4. As Nigel explained in detail only the name=value pairs are sent and not the individual flags in a HTTP request Cookie header.
This seems to imply that you want to do more than net/http.CookieJar and its implementation *net/http/cookiejar.Jar can do: Manipulate cookies through anything else than HTTP requests/responses. You can manipulate cookies in a browser by two means:
The cookiejar implementation in the stdlib implements the rules for setting and reading back cookies via HTTP responses and request only and it is incapable of implementing the needed security mechanism for JavaScript access. Wrapping the stdlib implementation and adding the necessary access control (like same origin policies, or honoring the HttpOnly flag) is impossible as you cannot inspect the cookiejar content. If you need to do that you will have to use a fork of the stdlib implementation which exports the whole content of the jar. net/http.CookieJar.Cookies is not capable of doing so. |
I think this is not an issue, everything is working as intended and this issue should be closed. |
I am also facing this issue, and am not sure if it is really "working as intended". The problem is here: Here all In my opinion, the restriction is overly conservative. If a server explicitly requests ... && (https || !e.Secure || e.HttpOnly) |
„Secure“ means „send over HTTPS only“ so this works correctly. The option HttpOnly has nothing to do with allowing to transfer a Secure cookie over HTTP: put roughly it prevents accessing the cookie via JavaScript. Works as intended. |
@vdobler here is my understanding about cookie jar. Cookie jar should persist all types of cookies for headless browser state management. In the context of GoLang headless browser is HttpRequest, HttpResponse. Based on my understanding, if I maintain an HTTP orchestration to engage a session dialogue with server and server URL sets HttpOnly cookie. I can't continue further Http request conversation to the server because server excepts that cookie to share information. |
So you're saying I have to hack around the behavior while talking to HTTP servers from major enterprise vendors. Can't say I'm thrilled... |
@corani No I am not saying you should do some hack. I‘m saying you Must not return cookies with Secure set over HTTP. If a HTTP server sends you a Secure cookie it does NOT want it back over plain http. Seems like you are trying to violate the security offered by setting Secure. Dont do that. |
@vdobler well at least in my case we are using https connection and using all SSL certificates for the handshake. Change here is instead of browser we are programmatically orchestrating network calls. In the world of HTML scraping, this is a common requirement. Python, Ruby, Java, Node languages provide full-fledged Cookie Jar functionality out of the box. Selenium is another framework that provides proper context to this use-case. |
@sadhasivam I have to admit I do not understand the problem you are struggling with. |
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version
)?go version go1.10.3 darwin/amd64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env
)?What did you do?
I am trying to automate certain website user behavior using default HTTP/net package. I have used http.Client with CookieJar to ensure Cookies are persisted between request/response. Majority of Use cases worked a charm. But if a website using HttpOnly: true, Secured: true cookie values somehow cookies are not persisted and not authenticated properly. Original Requests, Response contains proper cookie value but not client.CookieJar values for next Request to work on.
Ref: https://play.golang.org/p/Lhi2ZD_95AJ
Program prints Http.Response Cookies and Client.Jar.Cookies
What did you expect to see?
I expect client.CookieJar Cookie values should reflect current response cookie values.
What did you see instead?
client.CookieJar Cookie value not storing HttpOnly, Secured values properly.
The text was updated successfully, but these errors were encountered: