cmd/compile: global variable initialization done in unexpected order #51913

jcp19 · 2022-03-24T12:31:19Z

What version of Go are you using (`go version`)?

$ go version
go version go1.16.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/joao/Library/Caches/go-build"
GOENV="/Users/joao/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/joao/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/joao/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.16.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.16.4/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/2d/wcw2b3c57jz69cl5tg_s2fx00000gn/T/go-build3928827666=/tmp/go-build -gno-record-gcc-switches -fno-common"
GOROOT/bin/go version: go version go1.16.4 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.16.4
uname -v: Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64
ProductName:	macOS
ProductVersion:	12.2.1
BuildVersion:	21D62
lldb --version: lldb-1103.0.22.10
Apple Swift version 5.2.4 (swiftlang-1103.0.32.9 clang-1103.0.32.53)

What did you do?

I have a package consisting of the following two files:

f1.go

package main    
   
var A int = 3    
var B int = A + 1    
var C int = A

f2.go

package main    
   
import "fmt"    
                     
var D = f()      
   
func f() int {    
  A = 1    
  return 1    
}    
   
func main() {    
  fmt.Println(A, B, C)    
}

What did you expect to see?

According to the Go language specification, "package-level variable initialization proceeds stepwise, with each step selecting the variable earliest in declaration order which has no dependencies on uninitialized variables".

As such, I would expect two possible orders in which the global variables can be initialized:

A < B < C < D - happens when you compile the project by passing f1.go first to the compiler, followed by f2.go . In this case, the output is "1 4 3"
A < D < B < C - happens when f2.go is passed first to the compiler. In this case, the expected output would be "1 2 1".

What did you see instead?

For the second case (when f2.go is passed first), the actual output is "1 2 3". If instead I rewrite file f1.go to the following, I get the expected output for case 2.

Rewritten f2.go

package main    
   
import "fmt"    
   
var A int = initA()    
var B int = initB()    
var C int = initC()    
     
func initA() int {    
  fmt.Println("Init A")    
  return 3    
}    
     
func initB() int {    
  fmt.Println("Init B")    
  return A + 1    
}    
 
func initC() int {    
  fmt.Println("Init C")    
  return A    
}

Output

Init A
Init B
Init C
1 2 1

Additional Information

This issue was first discussed in the golang-nuts Google Group (link).

The text was updated successfully, but these errors were encountered:

go101 · 2022-03-24T12:39:21Z

A little clarification: OP means the outputs are different between go run f1.go f2.go and go run f2.go f1.go. And after rewriting f1.go, things changes a bit.

By the specification, the outputs should be always 1 4 3.

candlerb · 2022-03-24T15:23:31Z

By the specification, the outputs should be always 1 4 3.

The two scenarios are:

f1.go f2.go: declaration order is A B C D
f2.go f1.go: declaration order is D A B C

In the first case it's simple:

A is set to 3
B is set to 4
C is set to 3
D is set to 1, and A is set to 1 as a side-effect

Expected result: 1 4 3

In the second case, I come to a different conclusion from reading the spec than you:

D is not ready because it depends on A
A is set to 3
D is now "the next package-level variable that is earliest in declaration order and ready for initialization", so it gets set to 1, and A is set to 1 as a side-effect
B is set to 2
C is set to 1

Expected result: 1 2 1

That is, I read "next" to mean "the next variable to be initialized", not "the next variable following sequentially after the one which was last initialized". Is this an incorrect reading?

mknyszek · 2022-03-24T15:50:07Z

CC @golang/runtime

For additional context, see #31292. Unclear if this is a bug yet, but since it also involves the spec, CC @griesemer who was central to #31292.

griesemer · 2022-03-24T17:37:46Z

Edited. There appears to be a bug in the compiler. See #51913 (comment).

~~This is working as intended.~~ Note that the spec also says:

The declaration order of variables declared in multiple files is determined by the order in which the files are presented to the compiler: Variables declared in the first file are declared before any of the variables declared in the second file, and so on.

We don't need multiple files, we can just arrange the variable declarations accordingly. In the first case:

package main

var A int = 3
var B int = A + 1
var C int = A
var D = f()

func f() int {
	A = 1
	return 1
}

func main() {
	println(A, B, C, D)
}

the output is

1 4 3 1

Here's the corresponding trace from the type checker's initialization order computation (this is the trace produced by types2 which is used by the compiler, but note that at the moment the compiler still uses its own init order computation and not the types2 computation - still they match):

Computing initialization order for package main ("main")

Object dependency graph:
        A has no dependencies
        B depends on
                A
        C depends on
                A
        D depends on
                f
        f depends on
                A
        main depends on
                A
                B
                C
                D

Transposed object dependency graph (functions eliminated):
        A depends on 0 nodes
                B is dependent
                C is dependent
                D is dependent
        C depends on 1 nodes
        B depends on 1 nodes
        D depends on 1 nodes

Processing nodes:
        A (src pos 1) depends on 0 nodes now
        B (src pos 2) depends on 0 nodes now
        C (src pos 3) depends on 0 nodes now
        D (src pos 4) depends on 0 nodes now

Initialization order:
        A = 3
        B = A + 1
        C = A
        D = f()

For the 2nd case:

package main

var D = f()

func f() int {
	A = 1
	return 1
}

func main() {
	println(A, B, C, D)
}

var A int = 3
var B int = A + 1
var C int = A

the output is

1 2 3 1

and the corresponding init computation trace is:

Computing initialization order for package main ("main")

Object dependency graph:
        D depends on
                f
        f depends on
                A
        main depends on
                A
                B
                C
                D
        A has no dependencies
        B depends on
                A
        C depends on
                A

Transposed object dependency graph (functions eliminated):
        A depends on 0 nodes
                D is dependent
                B is dependent
                C is dependent
        D depends on 1 nodes
        C depends on 1 nodes
        B depends on 1 nodes

Processing nodes:
        A (src pos 4) depends on 0 nodes now
        D (src pos 1) depends on 0 nodes now
        B (src pos 5) depends on 0 nodes now
        C (src pos 6) depends on 0 nodes now

Initialization order:
        A = 3
        D = f()
        B = A + 1
        C = A

Thus, in this case D gets initialized before B because it's before B in the source. This explains the difference.

Closing.

aarzilli · 2022-03-24T17:55:37Z

In the second case, if f runs before the assignemnt to C shouldn't that see the side effect of the call to f?

griesemer · 2022-03-24T18:18:28Z

@aarzilli Good catch, I totally glanced over this. Indeed go/types and types2 compute the correct initialization order (currently not used by the compiler), while the compiler still has a bug here, I think. Re-opening.

For the 1st case:

A is set to 3. Variable state is: A = 3, B = 0, C = 0, D = 0.
B is set to A + 1. Variable state is: A = 3, B = 4, C = 0, D = 1.
C is set to A. Variable state is: A = 3, B = 4, C = 3, D = 0.
D is set to 1 (result of f()), and A is set to 1. Variable state is: A = 1, B = 4, C = 3, D = 1.

For the 2nd case:

A is set to 3. Variable state is: A = 3, B = 0, C = 0, D = 0.
D is set to 1 (result of f()), and A is set to 1. Variable state is: A = 1, B = 0, C = 0, D = 1.
B is set to A + 1. Variable state is: A = 1, B = 2, C = 0, D = 1.
C is set to A. Variable state is: A = 1, B = 2, C = 1, D = 1.

griesemer · 2022-03-24T18:29:34Z

Related issue: #49150

griesemer · 2022-03-24T18:30:52Z

cc: @mdempsky It looks like cmd/compile may still have an issue with initialization order. Maybe it's time to switch to the types2-determined order?

cc: @ianlancetaylor for gccgo results for these two cases.

mdempsky · 2022-03-24T19:11:40Z

The compiler issue here is that we optimize var X = 3; var Y = X into var X = 3; var Y = 3, even when there might be user function calls that could modify X's value before the spec says Y is initialized.

Unfortunately, I think this is more complex than just switching to types2's initialization order. I think cmd/compile is already correctly sorting the initialization statements; it's just misapplying an optimization that isn't actually safe. (And it looks like gccgo has a similar issue.)

The easy fix is to just disable that optimization, but that might lead to more dynamic initialization.

The more complex fix would involve actually tracking when user-written calls are sequenced during initialization, and keeping track of which global variables they might clobber, and how that limits subsequent optimization opportunities.

mdempsky · 2022-03-24T19:35:37Z

Here's a minimal repro of the issue, btw:

package main

var _ = func() int {
	a = false
	return 0
}()

var a = true
var b = a

func main() {
	if b {
		panic("FAIL")
	}
}

The Go spec says this program should silently exit with success. But instead it currently panics when compiled with either cmd/compile or gccgo.

gopherbot · 2022-03-24T19:47:21Z

Change https://go.dev/cl/395541 mentions this issue: cmd/compile: disable unsafe staticinit optimization

ianlancetaylor · 2022-03-24T19:59:03Z

I changed gccgo to match gc's behavior because the runtime package requires it (https://go.dev/cl/245098). I see that CL 395541 keeps the optimizations only for the runtime package, so I guess I'll do the same in gccgo.

mdempsky · 2022-03-24T23:24:37Z

I changed gccgo to match gc's behavior because the runtime package requires it (https://go.dev/cl/245098).

Thanks for the reference. For what it's worth, it looks like replacing var maxSearchAddr = maxOffAddr with func maxSearchAddr() offAddr { return maxOffAddr } and then changing uses of maxSearchAddr to maxSearchAddr() seems to eliminate the need for special casing package runtime in CL 395541.

I don't feel strongly about which way to proceed here. In general, I prefer fewer special cases for package runtime, in hopes that there are fewer surprises for the runtime team when switching between writing Standard Go and Runtime Go. But package initialization is already inherently weird for package runtime, and it seems unlikely the runtime team is going to write any tricky initializers that would interfere with this optimization.

If anyone else leans one way or the other here, let me know.

/cc @aclements @mknyszek @prattmic

aclements · 2022-03-25T13:22:49Z

I would also prefer that we not special-case the runtime here, especially if there's only one problem right now.

Is the issue with maxSearchAddr that we currently statically initialize it, but with CL 395541, it now gets dynamically initialized but the runtime depends on its value before it calls runtime.init? I don't see any dynamic assignments to either maxSearchAddr or maxOffAddr.

(It's too bad we don't have const structs. Then there would be an easy solution for maxSearchAddr. :P)

ianlancetaylor · 2022-03-25T20:08:28Z

Yes, if maxSearchAddr is initialized dynamically then it is initialized when we run package initialization in the call to doInit in main in proc.go. But long before that the code will use maxSearchAddr in pageAlloc.init called via mallocinit and schedinit. In an ordinary package this problem would be handled, but the runtime package initializes things outside of init functions.

gopherbot · 2022-03-26T00:27:01Z

Change https://go.dev/cl/395994 mentions this issue: compiler: revert for package-scope "a = b; b = x" just set "a = x"``

go101 · 2022-03-26T04:25:08Z

About source file order in a package, will it be better to always sort files in a package before compiling, to remove some unspecified behaviors?

mdempsky · 2022-03-26T04:31:47Z

About source file order in a package, will it be better to always sort files in a package before compiling, to remove some unspecified behaviors?

The Go spec recommends build systems to do that, and cmd/go already does when invoking cmd/compile.

go101 · 2022-03-26T05:53:19Z

Why not compulsively?

ianlancetaylor · 2022-03-26T23:12:43Z

That doesn't seem like an aspect that should be determined in a language spec.

In any case no reason to change it now.

This avoids a dependency on the compiler statically initializing maxSearchAddr, which is necessary so we can disable the (overly aggressive and spec non-conforming) optimizations in cmd/compile and gccgo. Updates #51913. Change-Id: I424e62c81c722bb179ed8d2d8e188274a1aeb7b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/396194 Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Austin Clements <austin@google.com> Run-TryBot: Matthew Dempsky <mdempsky@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2022-05-11T19:58:47Z

Change https://go.dev/cl/405549 mentions this issue: runtime: avoid staticinit dependency with sigsetAllExiting

mdempsky · 2022-05-11T20:08:18Z

@gopherbot Please backport to Go 1.18. This is a silent miscompilation of valid (albeit unlikely) code.

gopherbot · 2022-05-11T20:08:31Z

Backport issue(s) opened: #52857 (for 1.18).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

ianlancetaylor · 2022-06-24T21:10:00Z

@mdempsky Is there anything else to do for this issue? Thanks.

aclements · 2022-07-12T17:22:33Z

It looks like we have a fix out for this that has been +2'd but not landed. Is this low-risk enough to land at this point, or should we kick this to 1.20 and land it when the tree opens?

mdempsky · 2022-07-12T18:35:28Z

We discussed this earlier today. We're going to punt this to 1.20. We're confident the fix is correct, but there are uncertainties about how that might affect users accidentally depending on the existing behavior. The issue has also been present for a long time (and was added to gccgo for compatibility with cmd/compile even). So there doesn't seem to be an urgency to fix it in 1.19.

Revert CL 245098. It caused incorrect initialization ordering. Adjust the runtime package to work even with the CL reverted. Original description of CL 245098: This avoids requiring an init function to initialize the variable. This can only be done if x is a static initializer. The go1.15rc1 runtime package relies on this optimization. The package has a variable "var maxSearchAddr = maxOffAddr". The maxSearchAddr variable is used by code that runs before package initialization is complete. For golang/go#51913 Change-Id: I07a896da3d97c278bd144d95238bdd3f98c9a1ab Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/395994 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Than McIntosh <thanm@google.com>

cherrymui · 2023-01-10T23:31:33Z

It is probably too late now to get this in for 1.20. Maybe we want to submit the CL for 1.21 early when the tree opens.

gopherbot · 2023-03-02T17:58:32Z

This issue is currently labeled as early-in-cycle for Go 1.21.
That time is now, so a friendly reminder to look at it again.

mknyszek · 2023-06-09T19:23:52Z

Seems like nothing happened here for Go 1.21. Given that this was already bumped once, moving to Backlog. Feel free to punt it to Go 1.22 if you disagree. Thanks!

gopherbot · 2023-07-20T20:33:50Z

This issue is currently labeled as early-in-cycle for Go 1.22.
That time is now, so a friendly reminder to look at it again.

Currently, package runtime runs `osinit` before dynamic initialization of package-scope variables; but on GOOS=linux, `osinit` involves mutating `sigsetAllExiting`. This currently works because cmd/compile and gccgo have non-spec-conforming optimizations that statically initialize `sigsetAllExiting`, but disabling that optimization causes `sigsetAllExiting` to be dynamically initialized instead. This in turn causes the mutations in `osinit` to get lost. This CL moves the initialization of `sigsetAllExiting` from `osinit` into its initialization expression, and then removes the special case for continuing to perform the static-initialization optimization for package runtime. Updates #51913. Change-Id: I3be31454277c103372c9701d227dc774b2311dad Reviewed-on: https://go-review.googlesource.com/c/go/+/405549 Auto-Submit: Matthew Dempsky <mdempsky@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

mknyszek changed the title ~~package-initialization: global variable initialization done in unexpected order~~ runtime: global variable initialization done in unexpected order Mar 24, 2022

mknyszek modified the milestones: backlog, Backlog Mar 24, 2022

mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 24, 2022

griesemer closed this as completed Mar 24, 2022

griesemer reopened this Mar 24, 2022

griesemer changed the title ~~runtime: global variable initialization done in unexpected order~~ cmd/compile: global variable initialization done in unexpected order Mar 24, 2022

griesemer modified the milestones: Backlog, Go1.19 Mar 24, 2022

gopherbot mentioned this issue May 11, 2022

cmd/compile: global variable initialization done in unexpected order [1.18 backport] #52857

Closed

prattmic assigned mdempsky May 20, 2022

prattmic added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 20, 2022

dmitshur added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 12, 2022

thanm mentioned this issue Jul 12, 2022

Go compiler and runtime meeting notes #43930

Open

aclements modified the milestones: Go1.19, Go1.20 Jul 19, 2022

cherrymui modified the milestones: Go1.20, Go1.21 Jan 10, 2023

dmitshur added the early-in-cycle A change that should be done early in the 3 month dev cycle. label Jan 11, 2023

mknyszek modified the milestones: Go1.21, Backlog Jun 9, 2023

mdempsky modified the milestones: Backlog, Go1.22 Jun 9, 2023

gopherbot closed this as completed in afa3f8e Sep 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/compile: global variable initialization done in unexpected order #51913

cmd/compile: global variable initialization done in unexpected order #51913

jcp19 commented Mar 24, 2022

go101 commented Mar 24, 2022 •

edited

candlerb commented Mar 24, 2022 •

edited

mknyszek commented Mar 24, 2022

griesemer commented Mar 24, 2022 •

edited

aarzilli commented Mar 24, 2022

griesemer commented Mar 24, 2022

griesemer commented Mar 24, 2022

griesemer commented Mar 24, 2022

mdempsky commented Mar 24, 2022

mdempsky commented Mar 24, 2022 •

edited

gopherbot commented Mar 24, 2022

ianlancetaylor commented Mar 24, 2022 •

edited

mdempsky commented Mar 24, 2022

aclements commented Mar 25, 2022

ianlancetaylor commented Mar 25, 2022

gopherbot commented Mar 26, 2022

go101 commented Mar 26, 2022

mdempsky commented Mar 26, 2022

go101 commented Mar 26, 2022

ianlancetaylor commented Mar 26, 2022

gopherbot commented May 11, 2022

mdempsky commented May 11, 2022

gopherbot commented May 11, 2022

ianlancetaylor commented Jun 24, 2022

aclements commented Jul 12, 2022

mdempsky commented Jul 12, 2022

cherrymui commented Jan 10, 2023

gopherbot commented Mar 2, 2023

mknyszek commented Jun 9, 2023

gopherbot commented Jul 20, 2023

cmd/compile: global variable initialization done in unexpected order #51913

cmd/compile: global variable initialization done in unexpected order #51913

Comments

jcp19 commented Mar 24, 2022

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

Additional Information

go101 commented Mar 24, 2022 • edited

candlerb commented Mar 24, 2022 • edited

mknyszek commented Mar 24, 2022

griesemer commented Mar 24, 2022 • edited

aarzilli commented Mar 24, 2022

griesemer commented Mar 24, 2022

griesemer commented Mar 24, 2022

griesemer commented Mar 24, 2022

mdempsky commented Mar 24, 2022

mdempsky commented Mar 24, 2022 • edited

gopherbot commented Mar 24, 2022

ianlancetaylor commented Mar 24, 2022 • edited

mdempsky commented Mar 24, 2022

aclements commented Mar 25, 2022

ianlancetaylor commented Mar 25, 2022

gopherbot commented Mar 26, 2022

go101 commented Mar 26, 2022

mdempsky commented Mar 26, 2022

go101 commented Mar 26, 2022

ianlancetaylor commented Mar 26, 2022

gopherbot commented May 11, 2022

mdempsky commented May 11, 2022

gopherbot commented May 11, 2022

ianlancetaylor commented Jun 24, 2022

aclements commented Jul 12, 2022

mdempsky commented Jul 12, 2022

cherrymui commented Jan 10, 2023

gopherbot commented Mar 2, 2023

mknyszek commented Jun 9, 2023

gopherbot commented Jul 20, 2023

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

go101 commented Mar 24, 2022 •

edited

candlerb commented Mar 24, 2022 •

edited

griesemer commented Mar 24, 2022 •

edited

mdempsky commented Mar 24, 2022 •

edited

ianlancetaylor commented Mar 24, 2022 •

edited