Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: spec: byte view: type that can represent a []byte or string #5376

Open
bradfitz opened this issue Apr 30, 2013 · 12 comments
Open

proposal: spec: byte view: type that can represent a []byte or string #5376

bradfitz opened this issue Apr 30, 2013 · 12 comments
Labels
LanguageChange Proposal-Hold v2 A language change or incompatible library change
Milestone

Comments

@bradfitz
Copy link
Contributor

Go has no built-in type which can represent a read-only view of possibly-changing bytes.

We have two current types which internally are a byte pointer and a length:  (the first
also has a capacity, irrelevant to this issue) 


[]byte

   Allowance: "you can mess with this memory"
   Restriction: "this memory isn't necessarily forever; anybody else might be changing it now (race) or later (iterator becoming invalidated after this call, etc)"

string

   Allowance: "if you have a reference to this, it's good forever, and nobody will ever mess with it".
   Restriction: "you can NOT change"


And because of their conflicting allowances and restrictions, any conversion from
string->[]byte or []byte->string necessarily has to make a copy and often
generates garbage. 

Both are great, but there's a missing piece:

Many places in Go's API want a type with *neither* of those allowances, and are happy
with both of those restrictions:

-- much of pkg strings
-- much of pkg bytes
-- all the strconv.Parse* functions (issue #2632)
-- most of the string arguments to system calls
-- the io.Writer interface's argument (no need then for io.WriteString)
-- leveldb's Key/Value accessors.

Look at leveldb's Iterator:

http://godoc.org/code.google.com/p/leveldb-go/leveldb/db#Iterator

It has to go out of its way to say "please do not corrupt my memory".  If
somebody uses memdb directly
(http://godoc.org/code.google.com/p/leveldb-go/leveldb/memdb) and misuses the Iterator
type, all the sudden the memory corruption and stack traces implicate the memdb package,
even though it's a caller of memdb who violated the db.Iterator contract.

All leveldb really wants is to give callers a view of the bytes.

That is, in addition to "string" with its promise A (handle for life) and
"[]byte" with its promise B (you can mutate), we need a a common type to both
of those with neither promise, what is constant-time assignable from a string or a
[]byte.

s := "string"
b := []byte(s)
var v byteview = s // constant time assignment
var v byteview = b // constant time assignment

A byteview would be represented in memory just like a string (*byte, int), but the
compiler would prevent mutations or addressing (like string), and callers would always
treat its backing data as ephemeral, like a []byte, unless negotiated otherwise.

Obviously this isn't a Go 1.n item, considering the impact it would have on the standard
library.

A good name for byteview would be "bytes", but we have a package
"bytes" already. Maybe we can get rid of that package.
@bradfitz
Copy link
Contributor Author

Comment 1:

Status changed to LongTerm.

@bradfitz
Copy link
Contributor Author

@rsc
Copy link
Contributor

rsc commented Nov 27, 2013

Comment 3:

Labels changed: added go1.3maybe.

@rsc
Copy link
Contributor

rsc commented Dec 4, 2013

Comment 4:

Labels changed: added release-none, removed go1.3maybe.

@rsc
Copy link
Contributor

rsc commented Dec 4, 2013

Comment 5:

Labels changed: added repo-main.

@cespare
Copy link
Contributor

cespare commented Oct 28, 2015

The discussion of this is at https://groups.google.com/d/msg/golang-dev/Y7j4B2r_eDw/Ovu7WbSzlOUJ.

@rsc rsc changed the title spec: byte view: type that can represent a []byte or string proposal: spec: byte view: type that can represent a []byte or string Jun 20, 2017
@rsc rsc added the v2 A language change or incompatible library change label Jun 20, 2017
@ianlancetaylor
Copy link
Contributor

Certain implementations of generics (#15292) would provide this. But not all--it's possible to imagine generics proposals that do not support this.

@ianlancetaylor ianlancetaylor added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed GarbageCollector LongTerm Performance NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Dec 5, 2017
@ianlancetaylor
Copy link
Contributor

Putting on hold until we have some more experience with generics.

@ianlancetaylor ianlancetaylor removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 17, 2020
@bradfitz
Copy link
Contributor Author

Update: I wrote go4.org/mem which I'm now happily using in a number of places.

@navytux
Copy link
Contributor

navytux commented Apr 22, 2020

Update: I wrote go4.org/mem which I'm now happily using in a number of places.

Everyone was forced to do something like this for ages...

https://godoc.org/lab.nexedi.com/kirr/go123/mem#String
https://godoc.org/lab.nexedi.com/kirr/go123/mem#Bytes

@bradfitz
Copy link
Contributor Author

bradfitz commented May 12, 2020

@navytux, the difference is that go4.org/mem doesn't let callers do unsafe things.

@navytux
Copy link
Contributor

navytux commented May 13, 2020

@bradfitz, indeed, thanks for feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
LanguageChange Proposal-Hold v2 A language change or incompatible library change
Projects
None yet
Development

No branches or pull requests

5 participants