Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http/cookiejar: Secured, HttpOnly Cookie values are not persisted in http.Client.Jar #27138

Closed
sadhasivam opened this issue Aug 22, 2018 · 18 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@sadhasivam
Copy link

sadhasivam commented Aug 22, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/username/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/username/Repositories/go_repo:/Users/username/selfie/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10.3/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/gq/h95qp68n6j57y0z5qlsmypb80000gp/T/go-build835835324=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I am trying to automate certain website user behavior using default HTTP/net package. I have used http.Client with CookieJar to ensure Cookies are persisted between request/response. Majority of Use cases worked a charm. But if a website using HttpOnly: true, Secured: true cookie values somehow cookies are not persisted and not authenticated properly. Original Requests, Response contains proper cookie value but not client.CookieJar values for next Request to work on.

Ref: https://play.golang.org/p/Lhi2ZD_95AJ

Program prints Http.Response Cookies and Client.Jar.Cookies

What did you expect to see?

I expect client.CookieJar Cookie values should reflect current response cookie values.

What did you see instead?

client.CookieJar Cookie value not storing HttpOnly, Secured values properly.

@meirf meirf changed the title Secured, HttpOnly Cookie values are not persisted in http.Client.Jar net/http/cookiejar: Secured, HttpOnly Cookie values are not persisted in http.Client.Jar Aug 22, 2018
@andybons
Copy link
Member

andybons commented Sep 4, 2018

@bradfitz

@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 4, 2018
@andybons andybons added this to the Unplanned milestone Sep 4, 2018
@andriisoldatenko
Copy link
Contributor

@sadhasivam in you example I see error https://play.golang.org/p/Lhi2ZD_95AJ:

prog.go:10:2: cannot find package "golang.org/x/net/publicsuffix" in any of:
	/usr/local/go/src/golang.org/x/net/publicsuffix (from $GOROOT)
	/go/src/golang.org/x/net/publicsuffix (from $GOPATH)

@sadhasivam
Copy link
Author

@andriisoldatenko thanks for trying out. we may need to
go get golang.org/x/net/publicsuffix to get the program working.

@sadhasivam
Copy link
Author

any thoughts on this issue please ?

@bradfitz
Copy link
Contributor

bradfitz commented Oct 1, 2018

I defer to the cookie people: @vdobler @nigeltao. I don't generally handle cookie-related things.

@nigeltao
Copy link
Contributor

nigeltao commented Oct 5, 2018

I can't remember the details (maybe @vdobler does), but the Go cookie jar intentionally doesn't return all of the fields that were put into it. https://golang.org/src/net/http/cookiejar/jar.go#L219 is:

for _, e := range selected {
	cookies = append(cookies, &http.Cookie{Name: e.Name, Value: e.Value})
}

In other words, we explicitly return only Name and Value, not all of a Cookie's fields (including Secure and HttpOnly, but also e.g. Path, Expires, MaxAge and Raw).

As the CookieJar.Cookies doc comment says, the method "returns the cookies to send in a request". When sending cookies in a request, you don't need the Path, Secure or other fields, only the Name and Value fields.


Stepping back, this sounds a bit like an XY problem. You say, "I am trying to automate certain website user behavior using default HTTP/net package". You should be able automate user behavior (e.g. send HTTP requests) by sending only Name and Value. HTTP servers shouldn't need the HttpOnly or Secure fields. What sort of HTTP request isn't working? What sort of error are you getting?

For example, if I point my browser (with dev tools enabled) at https://www.chicos.com/, while the cookies that the browser receives have the HTTP and Secure fields set, it looks like the cookies the browser sends in follow-up HTTP requests do not.

@sadhasivam
Copy link
Author

Well, HTTPOnly Cookies are a more secure form of Cookie. ( https://www.owasp.org/index.php/HttpOnly ) . Here is my Use case:
I am trying to simulate a Headless version of the browser and perform automated monitoring what Selenium, Cypress, Python[requests package], etc. Problem is with CookieJar implementation. In a trail of website page navigation, CookieJar takes care of proper cookie management and authorize request correctly. Just that HttpOnly, Secure Cookie is not getting persisted and could not perform the request/response orchestration.

@nigeltao
Copy link
Contributor

nigeltao commented Oct 5, 2018

HTTPOnly Cookies are a more secure form of Cookie. ( https://www.owasp.org/index.php/HttpOnly ) .

As that page you linked to says, HttpOnly is part of Set-Cookie HTTP header, which means it's part of a HTTP Response. HttpOnly is not part of the Cookie HTTP header, which would be a part of the HTTP Request.

See https://en.wikipedia.org/wiki/List_of_HTTP_header_fields

I am trying to simulate a Headless version of the browser and perform automated monitoring what Selenium, Cypress, Python[requests package], etc. Problem is with CookieJar implementation. In a trail of website page navigation, CookieJar takes care of proper cookie management and authorize request correctly. Just that HttpOnly, Secure Cookie is not getting persisted and could not perform the request/response orchestration.

You say that the HttpOnly and Secure fields are not getting persisted. I don't think that this matters. As I said, HTTP requests don't have these fields, so I don't think their lack, coming out of the cookie jar, is affecting the orchestration. As I said, it doesn't look like browsers set HttpOnly or Secure in their requests.

With your https://www.chicos.com code example, you could hard-code re-adding the fields (like the XXX lines in https://play.golang.org/p/po4mqXhTUWv) before passing those cookies on to whatever makes your HTTP requests. I'm not saying that's a general solution, but it would indicate whether changing the cookie jar behavior would fix your underlying problem. But I suspect that it won't.

@vdobler
Copy link
Contributor

vdobler commented Oct 8, 2018

that HttpOnly, Secure Cookie is not getting persisted

It is. The relevant test is in https://golang.org/src/net/http/cookiejar/jar_test.go#L628 the cookie d=4.
These values are persisted and are used in determining whether to send or not to send the cookie in a subsequent HTTP request.

As Nigel explained in detail only the name=value pairs are sent and not the individual flags in a HTTP request Cookie header.

I am trying to simulate a Headless version of the browser ...

This seems to imply that you want to do more than net/http.CookieJar and its implementation *net/http/cookiejar.Jar can do: Manipulate cookies through anything else than HTTP requests/responses. You can manipulate cookies in a browser by two means:

  1. Through a Set-Cookie header in a HTTP response and
  2. Through JavaScript via document.cookie.

The cookiejar implementation in the stdlib implements the rules for setting and reading back cookies via HTTP responses and request only and it is incapable of implementing the needed security mechanism for JavaScript access. Wrapping the stdlib implementation and adding the necessary access control (like same origin policies, or honoring the HttpOnly flag) is impossible as you cannot inspect the cookiejar content. If you need to do that you will have to use a fork of the stdlib implementation which exports the whole content of the jar. net/http.CookieJar.Cookies is not capable of doing so.

@sadhasivam
Copy link
Author

Thanks a lot, @vdobler @nigeltao @bradfitz Appreciate your explanation. Got the Objective of Cookie/Jar implementation. let me hash my brain for some internal implementation.

@vdobler
Copy link
Contributor

vdobler commented Apr 9, 2019

I think this is not an issue, everything is working as intended and this issue should be closed.

@bradfitz bradfitz closed this as completed Apr 9, 2019
@corani
Copy link

corani commented Jul 9, 2019

I am also facing this issue, and am not sure if it is really "working as intended".

The problem is here:
https://github.com/golang/go/blob/master/src/net/http/cookiejar/jar.go#L120

Here all Secure cookies are filtered out when the protocol is HTTP. However, according to https://tools.ietf.org/html/rfc6265#section-4.1.2.6 having attributes HttpOnly and Secure is a valid combination.

In my opinion, the restriction is overly conservative. If a server explicitly requests HttpOnly and Secure, I would expect the cookie to be sent even if the protocol is HTTP. So something like:

  ... && (https || !e.Secure || e.HttpOnly)

@vdobler @nigeltao @bradfitz

@vdobler
Copy link
Contributor

vdobler commented Jul 11, 2019

„Secure“ means „send over HTTPS only“ so this works correctly. The option HttpOnly has nothing to do with allowing to transfer a Secure cookie over HTTP: put roughly it prevents accessing the cookie via JavaScript.

Works as intended.

@sadhasivam
Copy link
Author

sadhasivam commented Jul 11, 2019

@vdobler here is my understanding about cookie jar. Cookie jar should persist all types of cookies for headless browser state management. In the context of GoLang headless browser is HttpRequest, HttpResponse.

Based on my understanding, if I maintain an HTTP orchestration to engage a session dialogue with server and server URL sets HttpOnly cookie. I can't continue further Http request conversation to the server because server excepts that cookie to share information.

@corani
Copy link

corani commented Jul 12, 2019

Works as intended.

So you're saying I have to hack around the behavior while talking to HTTP servers from major enterprise vendors. Can't say I'm thrilled...

@vdobler
Copy link
Contributor

vdobler commented Jul 12, 2019

@corani No I am not saying you should do some hack. I‘m saying you Must not return cookies with Secure set over HTTP. If a HTTP server sends you a Secure cookie it does NOT want it back over plain http. Seems like you are trying to violate the security offered by setting Secure. Dont do that.

@sadhasivam
Copy link
Author

sadhasivam commented Jul 12, 2019

@vdobler well at least in my case we are using https connection and using all SSL certificates for the handshake. Change here is instead of browser we are programmatically orchestrating network calls. In the world of HTML scraping, this is a common requirement. Python, Ruby, Java, Node languages provide full-fledged Cookie Jar functionality out of the box.

Selenium is another framework that provides proper context to this use-case.

@vdobler
Copy link
Contributor

vdobler commented Jul 13, 2019

@sadhasivam I have to admit I do not understand the problem you are struggling with.

@golang golang locked and limited conversation to collaborators Jul 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

8 participants