Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reflect: add func (Value) UnsafePointer() unsafe.Pointer #40592

Closed
mdempsky opened this issue Aug 5, 2020 · 15 comments
Closed

reflect: add func (Value) UnsafePointer() unsafe.Pointer #40592

mdempsky opened this issue Aug 5, 2020 · 15 comments

Comments

@mdempsky
Copy link
Member

mdempsky commented Aug 5, 2020

[Splitting out separately from my comment on #40481, per @rsc's suggestion.]

This proposal is to add a method

func (v Value) UnsafePointer() unsafe.Pointer { return unsafe.Pointer(v.Pointer()) }

This would allow callers to migrate from Value.Addr and Value.Pointer so that the later could be deprecated (fulfilling their TODOs [1], [2]), and eliminating a class of possible misuse of unsafe.Pointer.

Optionally, we can then also have go fix rewrite:

  1. unsafe.Pointer(v.Pointer()) -> v.UnsafePointer()
  2. unsafe.Pointer(v.UnsafeAddr()) -> v.Addr().UnsafePointer()
  3. v.Pointer() -> uintptr(v.UnsafePointer())
  4. v.UnsafeAddr() -> uintptr(v.Addr().UnsafePointer())

Replacing v.UnsafeAddr() with v.Addr().UnsafePointer() has a slight added cost v.UnsafeAddr() was able to simply directly return the address pointer, while Addr() potentially has to compute the resulting pointer's type. However, (*rtype).ptrTo already uses caching, and the compiler also already eagerly generates *T types for most types, so I expect in practice this to have negligible overhead.

--

Tangentially, the InterfaceData method also has a TODO to deprecate. I don't think there's anything preventing this from being deprecated today. Callers should probably just use Elem() to get a Value representing the concrete value, and then do something based on its Kind().

Within Google's internal Go code base, the only two uses of reflect.Value.InterfaceData appear to be:

  1. https://github.com/hashicorp/terraform/blob/master/dag/marshal.go#L205, which looks like dead code anyway: reflect.ValueOf(...).Kind() never returns reflect.Interface.

  2. https://fuchsia.googlesource.com/fuchsia.git/+/refs/heads/master/src/connectivity/network/netstack/socket_encode.go#41, which needs to be fixed anyway since it's using reflect.SliceHeader incorrectly too.

@gopherbot gopherbot added this to the Proposal milestone Aug 5, 2020
@ianlancetaylor
Copy link
Contributor

My recollection is that one of the reasons that Value.Pointer returns uintptr is that we wanted to prevent code like

p := v.Pointer()

from giving p the type unsafe.Pointer in a package that does not import unsafe. Of course you can do this already using your own packages, but we didn't want to encourage it in the standard library.

(This also gets us to a discussion about type based alias analysis. In ordinary Go code a pointer to *byte and a pointer to *int must always point to different locations, so changing memory through one pointer can't affect memory changed through the other pointer. But that statement is no longer true if either value was set via a conversion through unsafe.Pointer. Does this mean that the compiler can never assume that they point to different locations, or can we say that that assumption changes based on whether the package imports unsafe?)

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Aug 5, 2020
@bcmills
Copy link
Contributor

bcmills commented Aug 5, 2020

Re type-based alias analysis: even if the package does not import unsafe we can only assume that variables do not alias if they were allocated from within the package. (A caller can always supply pre-punned pointers as arguments.)

So I think if we want to do any deep reasoning about aliasing we would generally need whole-program analysis either way.

@mdempsky
Copy link
Member Author

mdempsky commented Aug 5, 2020

My recollection is that one of the reasons that Value.Pointer returns uintptr is that we wanted to prevent code like p := v.Pointer() from giving p the type unsafe.Pointer in a package that does not import unsafe.

Yes, that reason was justifiable back when Go supported "safe" mode, but we removed that back in 2018. There are many better ways to run untrusted Go programs today (e.g., within a process sandbox like gVisor or by compiling them to wasm).

This also gets us to a discussion about type based alias analysis.

This seems like an orthogonal concern to me. The only way to safely use reflect.Value.Pointer and reflect.Value.UnsafeAddr as pointer values today is to immediately convert them to unsafe.Pointer anyway.

@ianlancetaylor
Copy link
Contributor

It's somewhat orthogonal but perhaps not entirely negligible: see #26070.

@mdempsky
Copy link
Member Author

mdempsky commented Aug 5, 2020

I don't see any mention of type-based alias analysis in #26070. Can you elaborate on how they're related?

@ianlancetaylor
Copy link
Contributor

The connection I see here is what should happen when people write p := v.UnsafePointer() in a package that doesn't import unsafe, if we adopt #26070. Will it be confusing that they can use := to get an unsafe.Pointer that they can't use?

Perhaps this concern is not important.

@mdempsky
Copy link
Member Author

mdempsky commented Aug 5, 2020

I see. I wrote "This seems like an orthogonal concern to me," with "this" referring to type-based alias analysis. When you wrote "It's somewhat orthogonal," I thought you were still talking about type-based alias analysis too.

Assuming both #26070 and this proposal were accepted, I expect users would adapt by writing import _ "unsafe" // for unsafe.Pointer conversion analogous to how they write import _ "unsafe" // for go:linkname today. (Though as I've commented on #26070, I disagree with that proposal.)

@rsc
Copy link
Contributor

rsc commented Oct 29, 2020

#26070 was retracted.
As I remember it, we were picky about importing "unsafe" to get an unsafe.Pointer (and not providing ways to get one otherwise) for two reasons:

  1. It's good practice to call out the unsafe parts of the code clearly, and the import does that.
  2. App Engine needed to be kept from using unsafe.

(2) is no longer a concern, and (1) seems well enough handled by the name v.UnsafePointer, which seems just as clearly "unsafe" as import "unsafe".

So overall this seems like a fine change. It might be worth thinking about for Go 1.17 along with the other unsafe/pointer cleanup.

@rsc rsc moved this from Incoming to Active in Proposals (old) Nov 4, 2020
@rsc
Copy link
Contributor

rsc commented Nov 11, 2020

Based on the discussion above, this seems like a likely accept (for Go 1.17).

@rsc rsc moved this from Active to Likely Accept in Proposals (old) Nov 11, 2020
@rsc
Copy link
Contributor

rsc commented Nov 18, 2020

No change in consensus, so accepted.

@rsc rsc moved this from Likely Accept to Accepted in Proposals (old) Nov 19, 2020
@rsc rsc changed the title proposal: reflect: add func (Value) UnsafePointer() unsafe.Pointer reflect: add func (Value) UnsafePointer() unsafe.Pointer Nov 19, 2020
@rsc rsc modified the milestones: Proposal, Backlog Nov 19, 2020
@erincandescent
Copy link

Replacing v.UnsafeAddr() with v.Addr().UnsafePointer() has a slight added cost v.UnsafeAddr() was able to simply directly return the address pointer, while Addr() potentially has to compute the resulting pointer's type. However, (*rtype).ptrTo already uses caching, and the compiler also already eagerly generates *T types for most types, so I expect in practice this to have negligible overhead.

I'm not so sure about this. I've spent some time optimising a reflect based encoder. Starting with a per-field loop

for _, f := range fields {
    f.codec.encode(writer, v.Field(f.index)) 
}

I ended up with 22% of CPU time on a moderately simple benchmark spent in Value.Field

I decided to try and optimize it using (as a first pass) Value.UnsafeAddr() and reflect.NewAt(t, p)

p := unsafe.Pointer(v.UnsafeAddr())
for _, f := range fields {
    f.codec.encode(writer, reflect.NewAt(f.ty, unsafe.Pointer(uintptr(p) + f.offset)).Elem()) 
}

The result was... 33% of CPU time being spent in reflect.NewAt (and another 11% in Value.Elem()). This CPU time is all spent in ptrTo/typeOff (and it only gets worse in cases where we fall off of that path into the sync.Map cache)

This is probably not a big deal in the grand scheme of things; the overhead is only really notable because I've spent a bunch of effort optimizing other aspects (and I have ways of working around it in the general case), and there's still optimizations which can be done to avoid the reflect overhead in many cases, but it makes these kinds of optimizations a bit trickier (in that you're more likely to bump into situations where you've accidentally made things slower)

So if I were to make a couple of requests, they'd be:

  1. An equivalent of UnsafeAddr, and
  2. An inverse, i.e. a function equivalent to reflect.NewAt(t, p).Elem() (failing that: a variant of reflect.NewAt where the type parameter is already the pointer?)

@gopherbot
Copy link

Change https://golang.org/cl/350691 mentions this issue: reflect: add Value.UnsafePointer

@gopherbot
Copy link

Change https://golang.org/cl/356252 mentions this issue: reflect: add test that method values have the same code pointers

gopherbot pushed a commit that referenced this issue Oct 18, 2021
Updates #40592

Change-Id: I16252dd57aceb5c49ddc11d8c12c601ca87ca902
Reviewed-on: https://go-review.googlesource.com/c/go/+/356252
Trust: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/356255 mentions this issue: doc: document new reflect.UnsafePointer function

gopherbot pushed a commit that referenced this issue Oct 23, 2021
Updates #40592

Change-Id: If66629e47ca9859128ee3ad8fb584e022d7a6982
Reviewed-on: https://go-review.googlesource.com/c/go/+/356255
Trust: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/360855 mentions this issue: encoding/json: use reflect.Value.UnsafePointer over Pointer

gopherbot pushed a commit that referenced this issue Mar 2, 2022
The latter returns a uintptr, while the former returns a unsafe.Pointer.
A uintptr is unsafe if Go ever switches to a moving GC,
while a unsafe.Pointer will be properly tracked by the GC.

We do not use unsafe.Pointer for any unsafe type conversions,
and only use it for comparability purposes, which is relatively safe.

Updates #40592

Change-Id: I813e218668704b63a3043acda4331205a3835a66
Reviewed-on: https://go-review.googlesource.com/c/go/+/360855
Trust: Joseph Tsai <joetsai@digital-static.net>
Trust: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Joseph Tsai <joetsai@digital-static.net>
TryBot-Result: Gopher Robot <gobot@golang.org>
@dmitshur dmitshur modified the milestones: Backlog, Go1.18 Mar 4, 2022
@golang golang locked and limited conversation to collaborators Mar 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

7 participants