code review 6550058: build: make int 64 bits on amd64

build: make int 64 bits on amd64

The assembly offsets were converted mechanically using
code.google.com/p/rsc/cmd/asmlint. The instruction
changes were done by hand.

Fixes issue 2188.

[rsc, 2012-09-24 19:10:21]

Hello iant, r (cc: golang-dev@googlegroups.com),

I'd like you to review this change to
https://go.googlecode.com/hg/

[bradfitz, 2012-09-24 19:15:12]

How do benchmarks look?

[remyoudompheng, 2012-09-24 19:18:29]

http://codereview.appspot.com/6550058/diff/6001/src/pkg/bytes/asm_amd64.s
File src/pkg/bytes/asm_amd64.s (right):

http://codereview.appspot.com/6550058/diff/6001/src/pkg/bytes/asm_amd64.s#new...
src/pkg/bytes/asm_amd64.s:97: CMPL	BX, CX
this looks wrong

[rsc, 2012-09-24 19:21:33]

On 2012/09/24 19:15:12, bradfitz wrote:
> How do benchmarks look?

Dunno.

[rsc, 2012-09-24 19:21:42]

On 2012/09/24 19:18:29, remyoudompheng wrote:
> http://codereview.appspot.com/6550058/diff/6001/src/pkg/bytes/asm_amd64.s
> File src/pkg/bytes/asm_amd64.s (right):
> 
>
http://codereview.appspot.com/6550058/diff/6001/src/pkg/bytes/asm_amd64.s#new...
> src/pkg/bytes/asm_amd64.s:97: CMPL	BX, CX
> this looks wrong

Thanks. Fixed.

[iant, 2012-09-24 20:06:17]

LGTM

wait for r

[r, 2012-09-24 20:48:40]

i would like this CL to include at least the key pieces of associated
documentation, which include updates to

install  (should mention the different in size of int)
install-source.html (should mention the different in size of int)
go_faq.html

in the CL as it stands, nowhere does it mention that int is 64 bits on amd64. it
kinda doesn't say anything about it anywhere i can find, 32 bits before or 64
bits now. please fix

[rsc, 2012-09-25 00:15:38]

PTAL

On Mon, Sep 24, 2012 at 4:48 PM,  <r@golang.org> wrote:
> i would like this CL to include at least the key pieces of associated
> documentation, which include updates to
>
> install  (should mention the different in size of int)
> install-source.html (should mention the different in size of int)
> go_faq.html

I updated the FAQ. I could not figure out where to work the int size
into the install docs. It's unclear to me that that level of detail
belongs there: we don't talk about pointer sizes, for example.
Instead, I created a new placeholder for the Go 1.1 release notes
(go1.1.html) and wrote the int section. I can dial back the detail if
necessary.

> in the CL as it stands, nowhere does it mention that int is 64 bits on
> amd64. it kinda doesn't say anything about it anywhere i can find, 32
> bits before or 64 bits now. please fix

I think the release notes are the right canonical place for this
detail, at least in the absence of a new, detailed document about the
implementations.

Russ

[r, 2012-09-25 00:26:04]

LGTM
thanks.

please run tidy on the new doc to double-check formatting. here's my script

tmp=/tmp/tidy.$USER.$$
tidy -quiet -utf8 -e $file 2>$tmp

cat $tmp | 9 grep -v '^Info:' | 9 sed '
	s|line ([0-9]+) column [0-9]+ - |'$file:'\1 |
'

[rsc, 2012-09-25 00:57:49]

*** Submitted as http://code.google.com/p/go/source/detail?r=ad78acd6f0bb ***

build: make int 64 bits on amd64

The assembly offsets were converted mechanically using
code.google.com/p/rsc/cmd/asmlint. The instruction
changes were done by hand.

Fixes issue 2188.

R=iant, r, bradfitz, remyoudompheng
CC=golang-dev
http://codereview.appspot.com/6550058

[dave_cheney.net, 2012-09-25 01:01:17]

Wow. I'm amazed at how painless this was. I was honestly not expecting
this til the next major version of Go.

On Tue, Sep 25, 2012 at 10:57 AM,  <rsc@golang.org> wrote:
> *** Submitted as
> http://code.google.com/p/go/source/detail?r=ad78acd6f0bb ***
>
>
> build: make int 64 bits on amd64
>
> The assembly offsets were converted mechanically using
> code.google.com/p/rsc/cmd/asmlint. The instruction
> changes were done by hand.
>
> Fixes issue 2188.
>
> R=iant, r, bradfitz, remyoudompheng
> CC=golang-dev
> http://codereview.appspot.com/6550058
>
>
> http://codereview.appspot.com/6550058/

[mtj1, 2012-09-25 01:08:24]

Running extensive integer application benchmarks now.

On Tue, Sep 25, 2012 at 6:31 AM, Dave Cheney <dave@cheney.net> wrote:

> Wow. I'm amazed at how painless this was. I was honestly not expecting
> this til the next major version of Go.
>
> On Tue, Sep 25, 2012 at 10:57 AM,  <rsc@golang.org> wrote:
> > *** Submitted as
> > http://code.google.com/p/go/source/detail?r=ad78acd6f0bb ***
> >
> >
> > build: make int 64 bits on amd64
> >
> > The assembly offsets were converted mechanically using
> > code.google.com/p/rsc/cmd/asmlint. The instruction
> > changes were done by hand.
> >
> > Fixes issue 2188.
> >
> > R=iant, r, bradfitz, remyoudompheng
> > CC=golang-dev
> > http://codereview.appspot.com/6550058
> >
> >
> > http://codereview.appspot.com/6550058/
>



-- 
Michael T. Jones | Chief Technology Advocate  | mtj@google.com |  +1
650-335-5765

[tru64ufs, 2012-09-25 05:20:17]

On 2012/09/24 19:10:21, rsc wrote:
> Hello iant, r (cc: mailto:golang-dev@googlegroups.com),
> 
> I'd like you to review this change to
> https://go.googlecode.com/hg/


I was wondering if this is ok by everybody.
All the codes that supposed to take up 32bit now occupies 64bit unnecessarily.

This is sort of like the change we saw when 32bit application complied on 64bit:
now every pointer becomes 64bit, hence enlarging memory footprint, resident set
size, thus sometimes slows down computation.

I think people have been using int64 if they want 64 bit integer, and we should
use explicit int64 when we want 64bit integer (long long).

[mtj1, 2012-09-25 05:26:21]

The natural size of a 64-bit machine with 64-bit pointers and 64-bit
file system addresses is arguably 64-bits.

On Tue, Sep 25, 2012 at 10:50 AM,  <tru64ufs@gmail.com> wrote:
> On 2012/09/24 19:10:21, rsc wrote:
>>
>> Hello iant, r (cc: mailto:golang-dev@googlegroups.com),
>
>
>> I'd like you to review this change to
>> https://go.googlecode.com/hg/
>
>
>
> I was wondering if this is ok by everybody.
> All the codes that supposed to take up 32bit now occupies 64bit
> unnecessarily.
>
> This is sort of like the change we saw when 32bit application complied
> on 64bit: now every pointer becomes 64bit, hence enlarging memory
> footprint, resident set size, thus sometimes slows down computation.
>
> I think people have been using int64 if they want 64 bit integer, and we
> should use explicit int64 when we want 64bit integer (long long).
>
>
>
> http://codereview.appspot.com/6550058/



-- 
Michael T. Jones | Chief Technology Advocate  | mtj@google.com |  +1
650-335-5765

[tru64ufs, 2012-09-25 05:31:35]

Here's a link to LP64 vs. ILP64

http://www.unix.org/version2/whatsnew/lp64_wp.html

The problem is that you will loose 64 -> 32bit backward compatibility.
When it works great on 64bit machine, now the same code may be broken for 32bit
machine.


On 2012/09/25 05:26:21, mtj1 wrote:
> The natural size of a 64-bit machine with 64-bit pointers and 64-bit
> file system addresses is arguably 64-bits.
> 
> On Tue, Sep 25, 2012 at 10:50 AM,  <mailto:tru64ufs@gmail.com> wrote:
> > On 2012/09/24 19:10:21, rsc wrote:
> >>
> >> Hello iant, r (cc: mailto:golang-dev@googlegroups.com),
> >
> >
> >> I'd like you to review this change to
> >> https://go.googlecode.com/hg/
> >
> >
> >
> > I was wondering if this is ok by everybody.
> > All the codes that supposed to take up 32bit now occupies 64bit
> > unnecessarily.
> >
> > This is sort of like the change we saw when 32bit application complied
> > on 64bit: now every pointer becomes 64bit, hence enlarging memory
> > footprint, resident set size, thus sometimes slows down computation.
> >
> > I think people have been using int64 if they want 64 bit integer, and we
> > should use explicit int64 when we want 64bit integer (long long).
> >
> >
> >
> > http://codereview.appspot.com/6550058/
> 
> 
> 
> -- 
> Michael T. Jones | Chief Technology Advocate  | mailto:mtj@google.com |  +1
> 650-335-5765

[dave_cheney.net, 2012-09-25 05:33:15]

> This is sort of like the change we saw when 32bit application complied
> on 64bit: now every pointer becomes 64bit, hence enlarging memory
> footprint, resident set size, thus sometimes slows down computation.

Pointers in Go, uintptr, have always been the native word size. This
change affects only the int type, which is now effectively an alias
for uintptr, rather than int32. Being the default type for constant
expressions and used in indexing and slicing operations, this allows
for slices and arrays greater than 2Gb. This also allows for 64bit
processors to operate naturally on 64 bit quantities without shifting
or masking, which may lead to less pipeline stalls and faster
execution.

You are correct that _some_ structures will have a larger memory
footprint (but don't forget about padding that may have previously
been added), but this can be mitigated by authors using the correct
integer type, right down to uint8 if they prefer.

Cheers

Dave

* note for the pedantic: discussion of uint omitted for brevity.

[mtj1, 2012-09-25 05:35:36]

Yes, I suffered through all of this at SGI when we moved from 32 to 64
machines and all the same concerns. Once we finally made the switch we
and our customers never wanted to go back. That was 1998. ;-)

On Tue, Sep 25, 2012 at 11:01 AM,  <tru64ufs@gmail.com> wrote:
> Here's a link to LP64 vs. ILP64
>
> http://www.unix.org/version2/whatsnew/lp64_wp.html
>
> The problem is that you will loose 64 -> 32bit backward compatibility.
> When it works great on 64bit machine, now the same code may be broken
> for 32bit machine.

-- 
Michael T. Jones | Chief Technology Advocate  | mtj@google.com |  +1
650-335-5765

[tru64ufs, 2012-09-25 05:40:23]

I understand. I was at DEC when that happens as well ;)

anyways, the problem is we still have 32bit Linux running on many servers at
IDC, while there exists many 64bit Linux machine as well.

When we push code to these machine, the build happens on that machine.
We need to be really careful on 32/64bit interoperability.
Now we need to make sure that we run/test our code on 32bit machine to 
ensure the code works correctly, not the way around. 


Could you reconsider this change, please?


On 2012/09/25 05:35:36, mtj1 wrote:
> Yes, I suffered through all of this at SGI when we moved from 32 to 64
> machines and all the same concerns. Once we finally made the switch we
> and our customers never wanted to go back. That was 1998. ;-)
> 
> On Tue, Sep 25, 2012 at 11:01 AM,  <mailto:tru64ufs@gmail.com> wrote:
> > Here's a link to LP64 vs. ILP64
> >
> > http://www.unix.org/version2/whatsnew/lp64_wp.html
> >
> > The problem is that you will loose 64 -> 32bit backward compatibility.
> > When it works great on 64bit machine, now the same code may be broken
> > for 32bit machine.
> 
> -- 
> Michael T. Jones | Chief Technology Advocate  | mailto:mtj@google.com |  +1
> 650-335-5765

[mtj1, 2012-09-25 05:44:26]

It is not my call. The core Go team has been talking about it for
quite some time... (That does not make it "right" in some sense, but
it would probably have been good  to bring it up before.) If you've
been watching the change lists the magic "switch" from 32 to 64 was
reduced to a single small change. I suppose there could be a "Go for
small systems" build mode. That's what I suggest you ask for rather
than a repeal of what feels like the future.

On Tue, Sep 25, 2012 at 11:10 AM,  <tru64ufs@gmail.com> wrote:
>
> I understand. I was at DEC when that happens as well ;)
>
> anyways, the problem is we still have 32bit Linux running on many
> servers at IDC, while there exists many 64bit Linux machine as well.
>
> When we push code to these machine, the build happens on that machine.
> We need to be really careful on 32/64bit interoperability.
> Now we need to make sure that we run/test our code on 32bit machine to
> ensure the code works correctly, not the way around.
>
>
> Could you reconsider this change, please?
>
>
>
> On 2012/09/25 05:35:36, mtj1 wrote:
>>
>> Yes, I suffered through all of this at SGI when we moved from 32 to 64
>> machines and all the same concerns. Once we finally made the switch we
>> and our customers never wanted to go back. That was 1998. ;-)
>
>
>> On Tue, Sep 25, 2012 at 11:01 AM,  <mailto:tru64ufs@gmail.com> wrote:
>> > Here's a link to LP64 vs. ILP64
>> >
>> > http://www.unix.org/version2/whatsnew/lp64_wp.html
>> >
>> > The problem is that you will loose 64 -> 32bit backward
>
> compatibility.
>>
>> > When it works great on 64bit machine, now the same code may be
>
> broken
>>
>> > for 32bit machine.
>
>
>> --
>>
>> Michael T. Jones | Chief Technology Advocate  | mailto:mtj@google.com
>
> |  +1
>>
>> 650-335-5765
>
>
>
>
> http://codereview.appspot.com/6550058/



-- 
Michael T. Jones | Chief Technology Advocate  | mtj@google.com |  +1
650-335-5765

[iant2, 2012-09-25 05:53:17]

On Mon, Sep 24, 2012 at 10:20 PM,  <tru64ufs@gmail.com> wrote:
>
> I was wondering if this is ok by everybody.
> All the codes that supposed to take up 32bit now occupies 64bit
> unnecessarily.
>
> This is sort of like the change we saw when 32bit application complied
> on 64bit: now every pointer becomes 64bit, hence enlarging memory
> footprint, resident set size, thus sometimes slows down computation.
>
> I think people have been using int64 if they want 64 bit integer, and we
> should use explicit int64 when we want 64bit integer (long long).

Go uses int as the size of objects, e.g., it is the return type of
len, cap, and unsafe.Sizeof.  This is generally convenient and
appropriate.  However, it means that on a 64-bit system with a 32-bit
int it is impossible to have objects larger than 2G.  There are
various different ways to address that.  Making int 64-bits solves the
problem without requiring any new language concepts.  Since 64-bit
machines have 64-bit registers, it costs little at runtime.

You're quite right that structures with int fields are now larger.
This is notably true of slices and strings.  However, if you have
structures for which the size matters, then I think it would have been
appropriate to use int32 all along.  The language has always clearly
stated that the int type can be 64 bits.

Ian

[luriel, 2012-09-25 05:53:23]

> On Tue, Sep 25, 2012 at 11:10 AM,  <tru64ufs@gmail.com> wrote:
>> When we push code to these machine, the build happens on that machine.
>> We need to be really careful on 32/64bit interoperability.
>> Now we need to make sure that we run/test our code on 32bit machine to
>> ensure the code works correctly, not the way around.

I don't understand, this should not affect Go's excellent support for
cross-compilation, building with 8g on a 64bit systems should produce
binaries that use 32bit ints, just as always.

Code built with 6g will use 64bit ints, but only runs on 64bit systems anyway.

Uriel

[tru64ufs, 2012-09-25 11:03:00]

OK. Got it.
I just started following this list.
Thanks everybody for straighten up this issue.


On 2012/09/25 05:53:17, iant2 wrote:
> On Mon, Sep 24, 2012 at 10:20 PM,  <mailto:tru64ufs@gmail.com> wrote:
> >
> > I was wondering if this is ok by everybody.
> > All the codes that supposed to take up 32bit now occupies 64bit
> > unnecessarily.
> >
> > This is sort of like the change we saw when 32bit application complied
> > on 64bit: now every pointer becomes 64bit, hence enlarging memory
> > footprint, resident set size, thus sometimes slows down computation.
> >
> > I think people have been using int64 if they want 64 bit integer, and we
> > should use explicit int64 when we want 64bit integer (long long).
> 
> Go uses int as the size of objects, e.g., it is the return type of
> len, cap, and unsafe.Sizeof.  This is generally convenient and
> appropriate.  However, it means that on a 64-bit system with a 32-bit
> int it is impossible to have objects larger than 2G.  There are
> various different ways to address that.  Making int 64-bits solves the
> problem without requiring any new language concepts.  Since 64-bit
> machines have 64-bit registers, it costs little at runtime.
> 
> You're quite right that structures with int fields are now larger.
> This is notably true of slices and strings.  However, if you have
> structures for which the size matters, then I think it would have been
> appropriate to use int32 all along.  The language has always clearly
> stated that the int type can be 64 bits.
> 
> Ian