cmd/cgo: opaque struct pointers are broken since Go 1.15.3 #42032

aykevl · 2020-10-16T23:39:20Z

What version of Go are you using (`go version`)?

$ go version
go version go1.15.3 linux/amd64

Does this issue reproduce with the latest release?

Yes, it reproduces since 76a2c87 (I bisected to that commit), see #41432.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ayke/.cache/go-build"
GOENV="/home/ayke/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ayke/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ayke:/home/ayke:/home/ayke:/home/ayke:/home/ayke:/home/ayke:/home/ayke"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build700779300=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I ran the following program:

package main

import "tinygo.org/x/go-llvm"

type param struct {
	llvmType llvm.Type
	name     string
}

func main() {
	i8Type := llvm.Int8Type()
	println("i8Type:   ", i8Type.C)

	p := param{llvmType: i8Type}
	params := []param{p}
	params = append(params, p)

	println("params[0]:", params[0].llvmType.C)
	println("params[1]:", params[1].llvmType.C)
}

What did you expect to see?

In Go before 76a2c87 it gives output like this, with all pointers set:

i8Type:    0x900940
params[0]: 0x900940
params[1]: 0x900940

What did you see instead?

Since 76a2c87 (in Go 1.15.3), I get output like this:

i8Type:    0x239eab0
params[0]: 0x239eab0
params[1]: 0x0

The pointer values change each run, that's not the issue. The issue is that params[1] is nil. To be precise, it appears that values appended to the slice with append are nil.

Interestingly, when I remove the name field of the struct, the issue disappears.

While investigating this issue, I also found some other weird behavior that might be related:

package main

import "fmt"

// typedef struct OpaqueType* TypeRef;
// int a;
// TypeRef GetType(void) {
//     return (void*)&a;
// }
import "C"

func main() {
	fmt.Println(C.GetType())
}

Output:

panic: can't call pointer on a non-pointer Value

goroutine 1 [running]:
reflect.Value.pointer(...)
	/usr/local/go/src/reflect/value.go:95
reflect.Value.Pointer(0x4a8a00, 0x587438, 0x16, 0xc000045c70)
	/usr/local/go/src/reflect/value.go:1456 +0x1b6
fmt.(*pp).printValue(0xc000070750, 0x4a8a00, 0x587438, 0x16, 0x76, 0x0)
	/usr/local/go/src/fmt/print.go:876 +0x195f
fmt.(*pp).printArg(0xc000070750, 0x4a8a00, 0x587438, 0x76)
	/usr/local/go/src/fmt/print.go:716 +0x2b4
fmt.(*pp).doPrintln(0xc000070750, 0xc000045f68, 0x1, 0x1)
	/usr/local/go/src/fmt/print.go:1173 +0xb1
fmt.Fprintln(0x4de100, 0xc000010018, 0xc000045f68, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/fmt/print.go:264 +0x58
fmt.Println(...)
	/usr/local/go/src/fmt/print.go:274
main.main()
	/home/ayke/tmp/cgo-bug-reproducer/main.go:34 +0x7a
exit status 2

(The result is the same in Go 1.15.3 and in the commit before 76a2c87). I'm including this as this smells like a bug somewhere, although I can't really say where. Maybe the code (2nd snippet) is wrong, but it looks correct to me.

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2020-10-17T00:06:25Z

CC @randall77

Here llvm.Type is defined as

type Type struct {
	C C.LLVMTypeRef
}

LLVMTypeRef is defined, in C, as typedef struct LLVMOpaqueType *LLVMTypeRef; LLVMOpaqueType is not defined.

llvm.Int8Type is

func Int8Type() (t Type)  { t.C = C.LLVMInt8Type(); return }

I think the issue here is that because C.LLVMTypeRef is a pointer to an opaque struct, it is marked as notinheap. That is fine. However, when a field of a struct is notinheap, then the struct itself becomes notinheap. But here that is wrong.

This works around a bug in Go 1.15.3 by pinning an older version. See: golang/go#42032

aclements · 2020-10-22T18:13:06Z

@ianlancetaylor , how would that lead to params[1] being nil? I would expect an overzealous notinheap propagation to cause build failures, but wouldn't expect it to have a runtime effect like this.

randall77 · 2020-10-22T19:25:53Z

@aykevl Your second issue (the reflect.Value.pointer one) I'm tracking at #42076.

Here C.LLVMTypeRef, and thus Type and param, are not marked go:notinheap. Only LLVMOpaqueType would be. A pointer to a go:notinheap value is not itself marked go:notinheap.

Reproduction note: /sudo apt-get install llvm-dev for the header files.

gopherbot · 2020-10-22T20:16:23Z

Change https://golang.org/cl/264300 mentions this issue: cmd/compile: fix storeType to handle pointers to go:notinheap types

randall77 · 2020-10-22T20:17:14Z

@gopherbot please open a backport issue for 1.15.

gopherbot · 2020-10-22T20:17:26Z

Backport issue(s) opened: #42151 (for 1.15).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

ianlancetaylor · 2020-10-22T21:59:39Z

@aclements Yes, I was evidently wrong. Sorry for the misleading suggestion.

aykevl mentioned this issue Oct 16, 2020

cmd/cgo: jmethodID/jfieldID is not mapped to uintptr if building with the Android NDK [1.15 backport] #41432

Closed

aykevl changed the title ~~cgo: CGo is broken in Go 1.15.3~~ cgo: CGo opaque structs are broken since Go 1.15.3 Oct 16, 2020

aykevl changed the title ~~cgo: CGo opaque structs are broken since Go 1.15.3~~ cgo: opaque struct pointers are broken since Go 1.15.3 Oct 16, 2020

aykevl mentioned this issue Oct 16, 2020

main: support gdb debugging with AVR tinygo-org/tinygo#1057

Merged

ianlancetaylor changed the title ~~cgo: opaque struct pointers are broken since Go 1.15.3~~ cmd/cgo: opaque struct pointers are broken since Go 1.15.3 Oct 16, 2020

ianlancetaylor added NeedsInvestigation release-blocker labels Oct 17, 2020

ianlancetaylor added this to the Go1.16 milestone Oct 17, 2020

aykevl added a commit to tinygo-org/tinygo that referenced this issue Oct 17, 2020

ci: work around bug in Go 1.15.3

7a03aad

This works around a bug in Go 1.15.3 by pinning an older version. See: golang/go#42032

aykevl mentioned this issue Oct 17, 2020

ci: work around bug in Go 1.15.3 tinygo-org/tinygo#1443

Merged

deadprogram pushed a commit to tinygo-org/tinygo that referenced this issue Oct 17, 2020

ci: work around bug in Go 1.15.3

47dc108

This works around a bug in Go 1.15.3 by pinning an older version. See: golang/go#42032

subosito added a commit to bukalapak/drafter-go that referenced this issue Oct 20, 2020

pin to 1.15.2 due to issue golang/go#42032

40d3750

gopherbot mentioned this issue Oct 22, 2020

cmd/cgo: opaque struct pointers are broken since Go 1.15.3 [1.15 backport] #42151

Closed

randall77 mentioned this issue Oct 22, 2020

cmd/cgo: segfault, regression in go 1.15.3 #42085

Closed

gopherbot closed this as completed in 933721b Oct 27, 2020

This was referenced Nov 6, 2020

Go: restore the possibility of go getting tensorflow and fix tests tensorflow/tensorflow#44653

Closed

[Go] package go-gettable, update wrappers, fix tests tensorflow/tensorflow#44655

Closed

deadprogram mentioned this issue Nov 8, 2020

ci: update CircleCI, Azure, and Docker builds to Go 1.15.4 tinygo-org/tinygo#1487

Merged

galeone mentioned this issue Nov 14, 2020

Init golang module galeone/tfgo#37

Merged

dkegel-fastly mentioned this issue Mar 6, 2021

map[string]uint64 generates incorrect wasm? tinygo-org/tinygo#1686

Closed

golang locked and limited conversation to collaborators Oct 27, 2021

gopherbot added the FrozenDueToAge label Oct 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/cgo: opaque struct pointers are broken since Go 1.15.3 #42032

cmd/cgo: opaque struct pointers are broken since Go 1.15.3 #42032

aykevl commented Oct 16, 2020 •

edited

Loading

ianlancetaylor commented Oct 17, 2020

aclements commented Oct 22, 2020

randall77 commented Oct 22, 2020

gopherbot commented Oct 22, 2020

randall77 commented Oct 22, 2020

gopherbot commented Oct 22, 2020

ianlancetaylor commented Oct 22, 2020

cmd/cgo: opaque struct pointers are broken since Go 1.15.3 #42032

cmd/cgo: opaque struct pointers are broken since Go 1.15.3 #42032

Comments

aykevl commented Oct 16, 2020 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

ianlancetaylor commented Oct 17, 2020

aclements commented Oct 22, 2020

randall77 commented Oct 22, 2020

gopherbot commented Oct 22, 2020

randall77 commented Oct 22, 2020

gopherbot commented Oct 22, 2020

ianlancetaylor commented Oct 22, 2020

aykevl commented Oct 16, 2020 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?