code review 88300045: build: remove tmp dir names from objects, support GOROO...

build: remove tmp dir names from objects, support GOROOT_FINAL again

If we compile a generated file stored in a temporary
directory - let's say /tmp/12345/work/x.c - then by default
6c stores the full path and then the pcln table in the
final binary includes the full path. This makes repeated builds
(using different temporary directories) produce different
binaries, even if the inputs are the same.

In the old 'go tool pack', the P flag specified a prefix to remove
from all stored paths (if present), and cmd/go invoked
'go tool pack grcP $WORK' to remove references to the
temporary work directory.

We've changed the build to avoid pack as much as possible,
under the theory that instead of making pack convert from
.6 to .a, the tools should just write the .a directly and save a
round of I/O.

Instead of going back to invoking pack always, define a common
flag -trimpath in the assemblers, C compilers, and Go compilers,
implemented in liblink, and arrange for cmd/go to use the flag.
Then the object files being written out have the shortened paths
from the start.

While we are here, reimplement pcln support for GOROOT_FINAL.
A build in /tmp/go uses GOROOT=/tmp/go, but if GOROOT_FINAL=/usr/local/go
is set, then a source file named /tmp/go/x.go is recorded instead as
/usr/local/go/x.go. We use this so that we can prepare distributions
to be installed in /usr/local/go without actually working in that
directory. The conversion to liblink deleted all the old file name
handling code, including the GOROOT_FINAL translation.
Bring the GOROOT_FINAL translation back.

Before this CL, using GOROOT_FINAL=/goroot make.bash:

        g% strings $(which go) | grep -c $TMPDIR
        6
        g% strings $(which go) | grep -c $GOROOT
        793
        g%

After this CL:

        g% strings $(which go) | grep -c $TMPDIR
        0
        g% strings $(which go) | grep -c $GOROOT
        0
        g%

(The references to $TMPDIR tend to be cgo-generated source files.)

Adding the -trimpath flag to the assemblers required converting
them to the new Go-semantics flag parser. The text in go1.3.html
is copied and adjusted from go1.1.html, which is when we applied
that conversion to the compilers and linkers.

Fixes issue 6989.

[rsc, 2014-04-15 23:47:02]

Hello r (cc: golang-codereviews@googlegroups.com),

I'd like you to review this change to
https://code.google.com/p/go/

[r, 2014-04-15 23:56:26]

i would expect to see changes to the flag parser too

https://codereview.appspot.com/88300045/diff/30001/src/liblink/obj.c
File src/liblink/obj.c (right):

https://codereview.appspot.com/88300045/diff/30001/src/liblink/obj.c#newcode91
src/liblink/obj.c:91: // That is, does s == t or s begins with t followed by a
slash?
or does s begin
(you have a number problem)

[rsc, 2014-04-16 00:03:10]

On Tue, Apr 15, 2014 at 7:56 PM, <r@golang.org> wrote:

> i would expect to see changes to the flag parser too
>

the flag parser is fine. 5a/lex.c, 6a/lex.c, 8a/lex.c were using
ARGBEGIN/ARGEND and now use flagparse instead (what 6c, 6g, 6l already use).

[rsc, 2014-04-16 00:05:13]

For anyone who finds this CL later, if you want to do more aggressive file
path rewriting, I wrote
http://godoc.org/code.google.com/p/rsc/cmd/editpclnas a standalone
'pack P' replacement before realizing it was the wrong
approach.

[rsc, 2014-04-16 00:06:44]

PTAL

Fixed the grammar in obj.c.

// Does s have t as a path prefix?
// That is, does s == t or does s begin with t followed by a slash?
// For portability, we allow ASCII case folding, so that
haspathprefix("a/b/c", "A/B") is true.
// Similarly, we allow slash folding, so that haspathprefix("a/b/c",
"a\\b") is true.

[iant, 2014-04-16 00:26:28]

LGTM

[rsc, 2014-04-16 00:46:51]

*** Submitted as https://code.google.com/p/go/source/detail?r=e5edcb7742c7 ***

build: remove tmp dir names from objects, support GOROOT_FINAL again

If we compile a generated file stored in a temporary
directory - let's say /tmp/12345/work/x.c - then by default
6c stores the full path and then the pcln table in the
final binary includes the full path. This makes repeated builds
(using different temporary directories) produce different
binaries, even if the inputs are the same.

In the old 'go tool pack', the P flag specified a prefix to remove
from all stored paths (if present), and cmd/go invoked
'go tool pack grcP $WORK' to remove references to the
temporary work directory.

We've changed the build to avoid pack as much as possible,
under the theory that instead of making pack convert from
.6 to .a, the tools should just write the .a directly and save a
round of I/O.

Instead of going back to invoking pack always, define a common
flag -trimpath in the assemblers, C compilers, and Go compilers,
implemented in liblink, and arrange for cmd/go to use the flag.
Then the object files being written out have the shortened paths
from the start.

While we are here, reimplement pcln support for GOROOT_FINAL.
A build in /tmp/go uses GOROOT=/tmp/go, but if GOROOT_FINAL=/usr/local/go
is set, then a source file named /tmp/go/x.go is recorded instead as
/usr/local/go/x.go. We use this so that we can prepare distributions
to be installed in /usr/local/go without actually working in that
directory. The conversion to liblink deleted all the old file name
handling code, including the GOROOT_FINAL translation.
Bring the GOROOT_FINAL translation back.

Before this CL, using GOROOT_FINAL=/goroot make.bash:

        g% strings $(which go) | grep -c $TMPDIR
        6
        g% strings $(which go) | grep -c $GOROOT
        793
        g%

After this CL:

        g% strings $(which go) | grep -c $TMPDIR
        0
        g% strings $(which go) | grep -c $GOROOT
        0
        g%

(The references to $TMPDIR tend to be cgo-generated source files.)

Adding the -trimpath flag to the assemblers required converting
them to the new Go-semantics flag parser. The text in go1.3.html
is copied and adjusted from go1.1.html, which is when we applied
that conversion to the compilers and linkers.

Fixes issue 6989.

LGTM=iant
R=r, iant
CC=golang-codereviews
https://codereview.appspot.com/88300045

[gobot, 2014-04-16 01:12:13]

This CL appears to have broken the openbsd-386-rootbsd builder.
See http://build.golang.org/log/c61b33ce505a47bb603c6f1ba6b18cf70656b2c1

[dave_cheney.net, 2014-04-16 01:18:31]

Looks real,

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x18 pc=0x805aa29]

runtime stack:
runtime.throw(0x8220e2a)
/usr/local/go/src/pkg/runtime/panic.c:520 +0x5f
runtime.sigpanic()
/usr/local/go/src/pkg/runtime/os_openbsd.c:218 +0x3a
acquirep(0x0)
/usr/local/go/src/pkg/runtime/proc.c:2422 +0x49
stopm()
/usr/local/go/src/pkg/runtime/proc.c:947 +0x123
findrunnable()
/usr/local/go/src/pkg/runtime/proc.c:1247 +0x2fc
schedule()
/usr/local/go/src/pkg/runtime/proc.c:1330 +0xe4
park0(0x18402640)
/usr/local/go/src/pkg/runtime/proc.c:1395 +0xd8
runtime.mcall(0x806f11b)
/usr/local/go/src/pkg/runtime/asm_386.s:190 +0x3c

On Wed, Apr 16, 2014 at 11:12 AM,  <gobot@golang.org> wrote:
> This CL appears to have broken the openbsd-386-rootbsd builder.
> See http://build.golang.org/log/c61b33ce505a47bb603c6f1ba6b18cf70656b2c1
>
>
> https://codereview.appspot.com/88300045/
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-codereviews" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-codereviews+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[rsc, 2014-04-16 01:50:01]

didn't happen on the next build. openbsd-386 seems flaky.