Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: spec: add support for int128 and uint128 #9455

Open
mei-rune opened this issue Dec 27, 2014 · 144 comments
Open

proposal: spec: add support for int128 and uint128 #9455

mei-rune opened this issue Dec 27, 2014 · 144 comments
Labels
LanguageChange v2 A language change or incompatible library change
Milestone

Comments

@mei-rune
Copy link

No description provided.

@mikioh mikioh changed the title can add int128 and uint128 support spec: add support for int128 and uint128 Dec 27, 2014
@ianlancetaylor
Copy link
Contributor

Can you provide a real-life use case?

@twoleds
Copy link

twoleds commented Feb 12, 2015

it's good for UUID, IPv6, hashing (MD5) etc, we can store IPv6 into uint128 instead byte slice and do some mathematics with subnetworks, checking range of IP addresses

@minux
Copy link
Member

minux commented Feb 12, 2015

These use cases are not strong enough to justify adding 128-bit types,
which is a big task to emulate it on all targets.

  1. MD5 is not secure anymore, so there is little benefit adding types to
    store its result.
  2. How often do you need to manipulate a UUID as a number rather than a
    byte slice (or a string)?
  3. The other use cases can be done with math/big just as easy.

Also note that GCC doesn't support __int128 on 32-bit targets and Go do
want consistent language features across all supported architectures.

@twoleds
Copy link

twoleds commented Feb 13, 2015

I agree with you there aren't a lot of benefits for int128/uint128, maybe a little better performance for comparing and hashing in maps when we use uint128 for storing UUID/IPv6 because for byte slices or string we need do some loops and extra memory but it isn't important I think

@mei-rune
Copy link
Author

I stat all interface flux of a device in one day.

@rsc rsc changed the title spec: add support for int128 and uint128 proposal: spec: add support for int128 and uint128 Jun 20, 2017
@rsc rsc added the v2 A language change or incompatible library change label Jun 20, 2017
@the80srobot
Copy link

In addition to crypto, UUID and IPv6, int128 would be enormously helpful for volatile memory analysis, by giving you a safe uintptr diff type.

@iMartyn
Copy link

iMartyn commented Oct 2, 2017

It also just makes code that much more readable if you have to deal with large IDs e.g. those you get back from google directory API amongst others (effectively they're uuids encoded as uint128).
Obviously you can use math/big but it makes the code much harder to reason about because you have to parse the code mentally first, distracting you from reading the code.

@ericlagergren
Copy link
Contributor

Adding a data point: ran into a situation with a current project where I need to compute (x * y) % m where x*y can possibly overflow and require a 128-bit integer. Doing the modulus by hand for the high and low halves is needlessly complicated.

@jfesler
Copy link

jfesler commented Jan 6, 2018

Another +1 for both IPv6 and UUID cases.

@ianlancetaylor
Copy link
Contributor

The examples of UUID and IPv6 are not convincing to me. Those types can be done as a struct just as easily.

It's not clear that this is worth doing if processors do not have hardware support for the type; are there processors with 128-bit integer multiply and divide instructions?

See also #19623.

@ianlancetaylor ianlancetaylor added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Jan 9, 2018
@ericlagergren
Copy link
Contributor

@ianlancetaylor I do not think so. GCC seems to use the obvious 6 instructions for mul, 4 for add and sub, and a more involved routine for quo. I'm not how anybody could emulate mul, add, or sub that precisely (in Go) without assembly, but that prohibits inlining and adds function call overhead.

@ianlancetaylor
Copy link
Contributor

The fact that the current tools can't yet inline asm code is not in itself an argument for changing the language. We would additionally need to see a significant need for efficient int128 arithmetic.

If there were hardware support, that in itself would suggest a need, since presumably the processor manufacturers would only add such instructions if people wanted them.

@ericlagergren
Copy link
Contributor

ericlagergren commented Jan 10, 2018

If there were hardware support, that in itself would suggest a need

A need that—presumably—compilers couldn't meet by adding their own 128-bit types, which they have. I mean, for all but division it's a couple extra instructions. For most cases that's been sufficient.

I confess I'm not an expert on CPU characteristics, but my understanding is much of the driving force behind adding larger sizes was the ability to address more memory. That makes me think general 128-bit support is rather unlikely.

Yet major compilers have added support (GCC, Clang, ICC, ...) for C and C++. Rust has them because of LLVM. Julia has them as well.

Other languages and compilers having support isn't sufficient reason to make a language change, sure. But it's evidence there exists a need other than simply UUIDs.

Their domain seems to lie in cryptography and arbitrary-precision calculations, for now.

@FlorianUekermann
Copy link
Contributor

Additional usecases are timestamps, cryptographic nonces and database keys.

Examples like database keys, nonces and UUID represent a pretty large collection of applications where keys/handles can't ever be reused or number ranges can't overlap.

@ianlancetaylor
Copy link
Contributor

@FlorianUekermann People keep saying UUID, but I see no reason that a UUID could not be implemented using a struct. It's not like people use arithmetic on a UUID once it has been created. The only reason to add int128 to the language is if people are going to use arithmetic on values of that type.

@FlorianUekermann
Copy link
Contributor

FlorianUekermann commented Jan 11, 2018

It's not like people use arithmetic on a UUID once it has been created

They do. UUIDs don't have to be random. Sequential UUIDs are common in databases for example. Combine sequential UUIDs with some range partitioning and you'll wish for integer ops in practice.

Still, timestamps seem like the most obvious example to me, where 64bit is not sufficient and the full range of arithmetic operations is obviously meaningful. Had it been available, I would expect that the time package contained some examples.

How big of an undertaking is the implementation of div? The rest seems rather straightforward.

@ericlagergren
Copy link
Contributor

How big of an undertaking is the implementation of div?

The code for naïve 128-bit division exists in the stdlib already (math/big). The PowerPC Compiler Writer’s Guide has a 32-bit implementation of 64-bit division (https://cr.yp.to/2005-590/powerpc-cwg.pdf, page 82) that can be translated upwards.

@josharian
Copy link
Contributor

Use case: [u]int128 can be used to check for overflow of [u]int64 operations in a natural way. Yes, this could make you want int256, but since int64 is the word size of many machines, this particular overflow matters a lot. See e.g. #21588. Other obvious options to address this use case are math/bits and
#19623.

Somewhat related use case: #21835 (comment).

@robpike
Copy link
Contributor

robpike commented Apr 26, 2022

@ethindp That's not accurate. Much of the standard library would need updating: fmt, reflect, most encoders and decoders, crypto, and many more, while other swathes of the library might want to be updated as well. Most packages in the open source world that do similar things will also need updating. And there will then be compatibility cascades resulting from those changes.

Adding a core type to a language is not a "zero change" thing to do, even for "people who don't want to use it".

I'm not arguing for or against the change, at least not in this message, but your implication that it's essentially free to do this is just not true.

@0xN0x
Copy link

0xN0x commented May 26, 2022

@ethindp That's not accurate. Much of the standard library would need updating: fmt, reflect, most encoders and decoders, crypto, and many more, while other swathes of the library might want to be updated as well. Most packages in the open source world that do similar things will also need updating. And there will then be compatibility cascades resulting from those changes.

Adding a core type to a language is not a "zero change" thing to do, even for "people who don't want to use it".

And this is why this should have been done long ago, this issue is already from 2014. The longer it takes to be accepted, the more complicated and time-consuming it will be to set it up.

@beoran
Copy link

beoran commented Jun 23, 2022

#53171 seems relevant here. If we add a (u)int128 type, on several architectures these can be handled with SIMD it similar instructions, and get the benefit of optimization without having to write assembly.

The idea of making int a flexible integer type, which can grow automatically, is also great, for example, Ruby has this. But it is a separate issue, so maybe @robpike could open a new issue for this?

@josharian
Copy link
Contributor

making int a flexible integer type

That's #19623.

@josharian
Copy link
Contributor

Generics provides a potential answer to the strconv problem.

The strconv problem (also present in reflect and any number of other places) is functions and methods parameterized across int types. That parameterization happens in several ways: by function name, by parameter, or by returning the largest possible type (which still requires further parameterization for signed vs unsigned). In all instances, adding a new int type requires a large, messy API addition.

This type parameterization would probably be better accomplished with generics. (As anyone who has written much generics code quickly discovers, the non-generic stdlib is a significant stumbling block, often requiring reflect-based adapters, so there's a good reason to do this anyway.)

And once you have type parameterization, adding a new, larger int type becomes much lower impact.

It'd still be good to take a thorough pass through and make sure this would handle all the necessary cases. But this at least seems to me to be a promising and principled general approach.

@thockin
Copy link

thockin commented Mar 15, 2023

I just wanted to +1 the idea of 128 bit ints as a core feature. We have a path where we represent the size of an IP range, and IPv6 ranges can be larger than 64 bits. uint128 would have fit the bill. We can work around it with big.Int or something, but +1 to it just being a standard thing.

@lfaoro
Copy link

lfaoro commented Mar 16, 2023

I don't understand why it's such a big deal to just do it, you have 128 comments on this issue, it's been open since 2014, other mainstream languages already complied w/ the users need.

What else do you need to consider support for int128? We don't want to use big.Int anymore.

@gophun
Copy link

gophun commented Mar 16, 2023

@lfaoro It's been explained why it would be a big deal: #9455 (comment)

@phuclv90
Copy link

We have a path where we represent the size of an IP range, and IPv6 ranges can be larger than 64 bits. uint128 would have fit the bill.

@thockin an IPv6 address isn't a number and you don't do arithmetic on it. Only bitwise operations are needed. Therefore that's not a good reason to add the type. There are already lots of comments regarding IPv6 above

@thockin
Copy link

thockin commented Mar 16, 2023 via email

@chiro-hiro
Copy link

If anyone try to build a WebAssembly runtime with Go, int128 would help.

@JesseCoretta
Copy link

JesseCoretta commented May 11, 2023

Hi!

I have read the comments on this thread, and for the most part I see both points of view. But one thing occurs to me, and while it is somewhat related to UUIDs, unfortunately this can of worms is a little deeper when you consider ASN.1 Object Identifiers.

Per [X.667], there is precedent for UUIDs to be interpreted as actual integers in certain Object Identifiers. One that exists in the wild, just for those who are curious, is:

// {joint-iso-itu-t(2) uuid(25) ans(987895962269883002155146617097157934)}
2.25.987895962269883002155146617097157934

Now, many of you can look at that leaf arc and see quite clearly that it will overflow not just int64, but uint64 as well. By quite a bit, actually:

  • 340,282,366,920,938,463,463,374,607,431,768,211,455 for uint128
  • 18,446,744,073,709,551,615 for uint64
  • 9,223,372,036,854,775,807 for int64

Now, I know some of you might be biting your tongues, thinking "oh no ... if she's right, that would mean the encoding/asn1-provided ObjectIdentifier type is basically invalid too, right?".

Well, in a way it always been invalid (at least in my point of view). No offense to the Go team, its just that because int is used as the slice type, and int allows negative values -- something that should never, ever appear in an OID -- that's something else I need to account for manually in my code. And this isn't even taking the overflow issue into account.

That said, I'm not here to bash the asn1 package, nor am I directly requesting any changes to it. I know how the Go team feels about this package, but quite frankly I love ASN.1 and I am just grateful it is supported via my favorite language.

But I am voting in favor of uint128 support (and, I suppose, int128 by necessity), just from my obscure PoV. I can easily make my own OID type (e.g.: []uint128) and do what I need to do, never worrying about overflows involving legal UUID-based OIDs. So I guess, from where I stand, its sort of a compromise.

One counter-argument that I can foresee:

"OIDs have no set limit on the magnitude of individual arcs, so conceivably this would be an issue when someone registers an OID that has some ungodly long number that would overflow the largest supported uintXXX-type in the universe."

This is true. One can only do so much. I'm also certain that OID RAs (Registration Authorities) look for such questionable moves and perhaps mitigate them. But you're not wrong. And yet, [X.667] exists for a reason, so I would cite that as motivation for supporting uint128. At least those registrations are legitimate, if a little odd ...

Thank you for listening

Jesse 💜❤️

@Bjohnson131
Copy link

Is the team still looking for "strong use cases" for uint128?

@josharian
Copy link
Contributor

@Bjohnson131 more strong use cases are welcome. Please "load hidden" on the comments on this issue first, though, and take a pass through--it's easy to miss things that have already been discussed above.

I suspect that the most helpful thing at this moment might be to write a complete doc, paying particular attention to what packages might be impacted (search for the text "strconv" and "generics" in comments after loading all hidden), and what the answer there is. But that's a lot of work, with no guarantee of impact.

(It'd also be helpful to update the original post with a list of use cases proposed in the comments.)

@phuclv90
Copy link

phuclv90 commented Jul 19, 2023

FWIW .NET added support for 128-bit int not so long ago and you may want to check out the discussions

@josharian
Copy link
Contributor

@phuclv90 thanks! That conversation is strikingly similar to this one, including (perhaps) the denouement of using generics once they were available to solve the strconv problem.

@Bjohnson131
Copy link

Bjohnson131 commented Oct 15, 2023

I think that there's 2 things that people have not mentioned. (u)int128s allow us to write cleaner, more error-free software. period.

There's an air among communities that this isn't a valid reason, as people should write good code. Often though, (and we all know we're all guilty of this) our code isn't good or completely readable. (u)int128s would help everybody in this regaurd every time they're used to avoid messy code as well as code with bugs.

Intrinsically, this value-add should be obvious. more people writing more better bug-free code is good for the entire ecosystem.

@c-robinson
Copy link

It looks like the Go source indicates that it might be useful to have at least a uint128. There are multiple references to uint128 peppered around the crypto/aes package, as well as a type in crypto/internal/edwards25519 for uses that are waaaaay over my head:

// uint128 holds a 128-bit number as two 64-bit limbs, for use with the
// bits.Mul64 and bits.Add64 intrinsics.
type uint128 struct {
	lo, hi uint64
}

and then net/netip defines the same type with the same name, but adds a bunch of methods... so much so that the netip implementation seems like it would be entirely usable as a replacement for, say, a uint128-sized value stored in big.Int.

@minux's initial counter-argument to this proposal hinged on the statement that

The other use cases can be done with math/big just as easy.

But that would seem to be in conflict with the netip approach.

@Eisenwave
Copy link

I have written a proposal P3140: std::int_least128_t for C++.

The Motivation in that proposal is largely language-agnostic, so it might be helpful in this discussion.

@andreyvit
Copy link

andreyvit commented Apr 2, 2024

+1. My use cases: arithmetics — 128-bit fixed-point numerics (high-precision monetary values), bitwise ops — hashing, identifiers, larger bitmasks. Like others have noted, code simplification for bitwise operations is a noticeable benefit.

However, if the team wants an alternative proposal, we could allow arithmetics and bitwise ops on [n]uint64 types. This would take care of uint128, uint256 and similar, without extending the list of primary types. Then extending fmt, strconv, json and other packages would be an optional effort that doesn't have to happen in lockstep with compiler changes (or maybe doesn't need to happen at all).

@sylr
Copy link

sylr commented Apr 2, 2024

+1. My use cases: arithmetics — 128-bit fixed-point numerics (high-precision monetary values)

Same here, an int128 based fixed-point numeric could allow to be precise enough for cryptocurrencies such as bitcoin and be "large" enough to be also used for market capitalizations with fiat currencies.

@andreyvit
Copy link

I went through my codebase, and I'm more in favor of [2]uint64 proposal now (i.e. allow treating [n]uint64 as integers for arithmetics, bitwise ops, comparisons, maybe even literal initializers) than simply an uint128 type. This would be assigning new semantics to something that is already 99% used for bignums, won't break any existing code, and won't require any library changes (we can add extra strconv functions for these, but, most importantly perhaps, no sweeping reflection changes across all serialization libs).

People can add MarshalText etc on top of these as appropriate for their domains (which will probably produce hex, base64 or even dashed UUID formatting).

The reasoning for [n]uint64 over int128 etc is:

  • in at least two cases, I really want an uint256, not uint128
  • formatting and parsing is domain-specific (and materially differs in different places in my code)
  • I bet at some point in the future people will start asking for uint512, uint1024, etc
  • really the annoying part making code hard to read is operators; if these are solved by the compiler, everything else can be added trivially

Should I open a separate ticket for discussion, or is this a non-starter @ianlancetaylor @robpike?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
LanguageChange v2 A language change or incompatible library change
Projects
None yet
Development

No branches or pull requests