Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: runtime/cgo: move built-in C helpers from cmd/cgo to runtime/cgo #51134

Closed
qmuntal opened this issue Feb 10, 2022 · 7 comments
Closed

Comments

@qmuntal
Copy link
Contributor

qmuntal commented Feb 10, 2022

cmd/cgo generates on the fly several Go functions, namely C.GoString, C.GoStringN, C.GoBytes, C.CString, and C.CBytes.

These functions inherit most of cgo usability problems despite being plain Go functions, most notably they do not play well with autocompletion. Additionally, its difficult to learn how to use them, and even know they exist, as they are only documented at https://pkg.go.dev/cmd/cgo in the middle of a long guide about how to use cgo, when one would expect them to appear in the Functions section.

I propose that we stop treating these function in a special way by defining and implementing them in runtime/cgo.

In order to maintain backwards compatibility, the autogenerated C functions won't be removed but they will just be a wrapper for the new runtime/cgo functions.

The proposed API is almost the same as the one already generated by runtime/cgo, with the only difference that GoStringN and GoBytes accept an int instead of a C.int as length parameter.

// GoString converts the C string p into a Go string.
func GoString(p *C.char) string

// GoStringN converts the C data p with explicit length to a Go string.
func GoStringN(p *C.char, l int) string

// GoBytes converts the C data p with explicit length to a Go []byte.
func GoBytes(p unsafe.Pointer, l int) []byte 

// CString converts the Go string s to a C string.
//
// The C string is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling C.free (be sure to include stdlib.h
// if C.free is needed).
func CString(s string) *C.char

// CBytes converts the Go []byte slice b to a C array.
//
// The C array is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling C.free (be sure to include stdlib.h
// if C.free is needed).
func CBytes(b []byte) unsafe.Pointer
@gopherbot gopherbot added this to the Proposal milestone Feb 10, 2022
@ericlagergren
Copy link
Contributor

ericlagergren commented Feb 11, 2022

I like this idea. However, I believe it will require some sort of compiler magic since C.* types are different across package boundaries.

$ cat main.go
package main

/*
static inline const char* foo() {
	return "hello from C";
}
*/
import "C"

import "example.com/ctest/b"

func main() {
	p := C.foo()
	s := b.GoString(p)
	println(s)
}
$ cat b/b.go
package b

import "C"

func GoString(p *C.char) string {
	return "wat"
}
$ go run main.go
# command-line-arguments
./main.go:14:18: cannot use p (variable of type *_Ctype_char) as type *b._Ctype_char in argument to b.GoString

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Feb 16, 2022
@rsc
Copy link
Contributor

rsc commented Feb 16, 2022

Yes, we could put each name in both C.Foo and cgo.Foo, with cgo translating C.Foo to cgo.Foo.
But why?
Then we will have two names for each of these functions, and people will get told "don't use the C.Foo names anymore, they are deprecated" (even if they are not). And generally it will cause confusion.
Is there a reason in favor of moving them?

I am not worried about having to look in the cmd/cgo docs. There are many cgo details there. Might as well have them all together.

As for tab completion, it seems like gopls could be updated to take care of that issue without putting users through a shift in how they write cgo code. Better to adapt the tools to reality at this point.

@rsc
Copy link
Contributor

rsc commented Feb 16, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals (old) Feb 16, 2022
@qmuntal
Copy link
Contributor Author

qmuntal commented Feb 17, 2022

After reading

I like this idea. However, I believe it will require some sort of compiler magic since C.* types are different across package boundaries. @ericlagergren

That's a very good point, before posting this proposal I did a quick prototype to verify it was feasible, but did not validate calling cgo.CString from another package. Digging a little bit in the issues backlog I see it is related to the long-standing #13467. Until that is resolved there is no way to treat the proposed API as normal Go functions (aka no compiler trick involved), as they contain C types on its signature. We could also reformulate the API to accept unsafe.Pointer instead of *C.char, but that would be awkward.

Then we will have two names for each of these functions, and people will get told "don't use the C.Foo names anymore, they are deprecated" (even if they are not). And generally it will cause confusion.
Is there a reason in favor of moving them? @rsc

These are the features that are missing:

  • Discoverability: These functions do not exist until they are used, the Go code compiles and the CGO definitions can be generated. Moreover, it is not possible to reuse definitions from other packages due to the way cgo generates functions. This is hard to fit in current autocompletion tools without special-casing.

  • Referenceability (is this a word?): There is no way to link to the definition of these functions, i.e. one can navigate to io.Writer but not to https://pkg.go.dev/cmd/cgo#CString. This makes it hard for humans to link these functions in discussions or code reviews, and for tools in docs popups.

Additionally, and probably nitpicking, having Go functions (which accepts Go types) exposed in the C package goes against the principle of least surprise, as I would expect to find in that package pure C symbols.

Better to adapt the tools to reality at this point. @rsc

It's true that this ship might already sailed. Improving the tools and the documentation for these functions will also solve most of my issues.

I will file a proposal to special-case these functions in gopls and also submit some commits that I think will improve their documentation, p.e. instructing cmd/cgo to add meaningful comments to the autogenerated functions so it can be displayed by current tooling.

@gopherbot
Copy link

Change https://go.dev/cl/386414 mentions this issue: cmd/go: add comments to C.* special functions

@rsc
Copy link
Contributor

rsc commented Feb 23, 2022

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Decline in Proposals (old) Feb 23, 2022
@rsc rsc moved this from Likely Decline to Declined in Proposals (old) Mar 2, 2022
@rsc
Copy link
Contributor

rsc commented Mar 2, 2022

No change in consensus, so declined.
— rsc for the proposal review group

@rsc rsc closed this as completed Mar 2, 2022
gopherbot pushed a commit that referenced this issue Mar 31, 2022
Adding comments to these functions help IDE tooling to display
meaningful documentation, p.e. on hover.

Tested with gopls and vscode.

Updates #51134

Change-Id: Ie956f7cf192af0e828def4a141783f3a2589f77d
Reviewed-on: https://go-review.googlesource.com/c/go/+/386414
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Auto-Submit: Russ Cox <rsc@golang.org>
@golang golang locked and limited conversation to collaborators Mar 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

4 participants