Can you clarify what you are measuring when you measure a slowdown? It seems to me that you have code that fails the cgo check, and you are replacing it with code that passes the cgo check. The code that fails the cgo check won't run, so what are you timing?

@ianlancetaylor Forgive me for not explaining that clearly. The only time I have seen the cgo check fail is when it is passed a short []byte retrieved from the Bytes() method on bytes.Buffer, such that the slice address points into the array inside a bytes.Buffer object. Aside from this and other situations with similarly designed structs the code in my project, lmdb-go, does not trigger the cgo argument check panic (I thought I had already changed everything to comply!).

You should be able to see this. Changing the code I provided so that b := []byte("test"), or simply writing enough into the buf object to overflow the internal array.

That is what I am benchmarking in my lmdb-go project, byte slices that are unaffected by this issue. And after compensating for this corner case I see processing in the normal case degrading farther than I would like it to.

As I eluded to, I don't think the behavior here is captured well by the cgo change proposal, or any other documentation that I have seen about the restrictions (like the cmd/cgo docs). I was under the impression that the code example from the issue description was passing the address of a struct field (i.e. the byte array field inside bytes.Buffer), in which case the documentation states

When passing a pointer to a field in a struct, the Go memory in question is the memory occupied by the field, not the entire struct.

I don't think the documentation is very clear about the need to perform the indirection at the cgo call site (as was suggested in #14210 and demonstrated at http://play.golang.org/p/eX1pSIjmo4). It never says explicitly says my code should work but it doesn't say that it shouldn't work either.

Thanks for the additional details.

The documentation doesn't spell out the need to perform the indirection at the cgo call site because there is no intent that that be a permanent requirement. The intent is what you quote from the docs: the memory in question is the memory occupied by the field. The question is how to implement that.

Thanks for the explanation. I have probably read you say as much before. But I now finally understand it in this context. So thank you for your patience as well.

I will try a little more to improve my own change, and to reduce the performance penalty I am seeing. It's not clear to me how escape analysis works with cgo. Though it seems to me that with the new restrictions pointers passed through cgo would be considered non-escaping? (edit: obviously unless they could be determined to escape for other reasons)

I also am not sure what is escaping here. I think you can run go build -gcflags=-m to see escape analysis explanations.

Ah yes. I had forgotten about that flag. It certainly prints out a lot of stuff. I will poke around, see what I can do with it, and get back soon.

Previously I did try manually inlining the function which takes the []byte address. It did not seem to provide a visible performance improvement. But I will look again with this flag enabled and see if I did something dumb. Thanks.

I seem to have gotten the relevant benchmarks for my project slightly closer to the original version. But there is still a sizable gulf between performance of the original and fixed code on normal-case input, which do not trigger any panics.

In regards to escaping pointers, it seems like cgo may be assuming that all pointers passed to cgo escape. This is understandable because it cannot be verified but it does seem like that could be relaxed if cgo is to assume that people comply with the restrictions it states (though cannot enforce). Following is a simple example and the output of go build -gcflags=-m which shows a simple []byte escaping in a simple cgo call.

http://play.golang.org/p/gG3o96a4rD

# github.com/bmatsuo/lmdb-go/tmp/cgosimple
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:14: can inline _Cgo_ptr
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:14: leaking param: ptr to result ~r1 level=0
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:27: _cgo_runtime_cgocall assuming arg#2 is unsafe uintptr
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:30: _cgo_runtime_cmalloc assuming arg#1 is unsafe uintptr
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:33: _cgo_runtime_cgocallback assuming arg#3 is unsafe uintptr
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:41: leaking param: p
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:41: leaking param: args
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:53: p0 escapes to heap
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:50: leaking param: p0
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:54: p1 escapes to heap
/tmp/go-build670788717/github.com/bmatsuo/lmdb-go/tmp/cgosimple/_obj/_cgo_gotypes.go:51: _Cfunc_my_func &p0 does not escape
./cgosimple.go:18: unsafe.Pointer(&b[0]) escapes to heap
./cgosimple.go:18: &b[0] escapes to heap
./cgosimple.go:17: ([]byte)("test") escapes to heap
./cgosimple.go:18: ... argument escapes to heap
./cgosimple.go:18: b escapes to heap
./cgosimple.go:17: ([]byte)("test") escapes to heap

I think it's best to keep issues focused on a single problem. The problem of pointers escaping when passed to cgo is a different problem. See #10303.

It is unclear to me what this issue is about. Can someone summarize what remains?

I think that any fix for this is the same as #14210, so closing as a duplicate.