/cc @ianlancetaylor

Can you show an example where it would be appropriate for cgo code to include Python.h?

Sure. Basically just before any generated include directive. E.g.:

Actual _cgo_export.c:

/* Code generated by cmd/cgo; DO NOT EDIT. */

#include <stdlib.h>
#include "_cgo_export.h"

#pragma GCC diagnostic ignored "-Wunknown-pragmas"
...

Desired _cgo_export.c:

/* Code generated by cmd/cgo; DO NOT EDIT. */

#include <Python.h>
#include <stdlib.h>
#include "_cgo_export.h"

#pragma GCC diagnostic ignored "-Wunknown-pragmas"
...

The same goes for any other generated file which may contain Go exported functions and/or their declaration.

A possible solution could be to remove #include <stdlib.h> keeping only #include "_cgo_export.h". Then in _cgo_export.h the preamble from import "C" comments should be injected before any other include (i.e. stdlib.h and stddef.h).

@philtay Thanks. I understand the change you want to make. What I'm hoping to see is an example of actual Go code that requires that change. Not the output of cgo, but the code that a user would write that requires this change.

@ianlancetaylor Ah, ok. Please find below a self contained example:

main.go

package main

/*
#cgo pkg-config: python3
#cgo CFLAGS: -std=c99
#include <Python.h>
*/
import "C"

//export foobar
func foobar() *C.PyObject {
	return nil
}

func main() {}

main.c

#include <Python.h>
#include "_cgo_export.h"

PyObject *
foobar_wrap(PyObject *self, PyObject *Py_UNUSED(ignored))
{
    return foobar();
}

Run main.go to get an error like this one:

In file included from /usr/include/python3.7m/Python.h:8,
                 from main.go:6,
                 from _cgo_export.c:4:
/usr/include/python3.7m/pyconfig.h:1522: warning: "_POSIX_C_SOURCE" redefined
 1522 | #define _POSIX_C_SOURCE 200809L
      | 
In file included from /usr/include/bits/libc-header-start.h:33,
                 from /usr/include/stdlib.h:25,
                 from _cgo_export.c:3:
/usr/include/features.h:295: note: this is the location of the previous definition
  295 | # define _POSIX_C_SOURCE 199506L
      | 

Thanks for the example.

I'm sorry to be so unclear, but that is still not what I am looking for. I'm trying to understand in what cases someone would want to write Go code that uses cgo that does a #include <Python.h>. What is the actual Go code where this case arises?

Looking at your example, why would someone write Go code that returns a *PyObject? That seems like a memory leak or dangling pointer waiting to happen. Why not instead write C code that calls Go code to do some operation and then do the Python conversion in C? That seems much more likely to get memory issues right across the three languages.

@ianlancetaylor Two full blown examples in the wild:

• https://github.com/asottile/dockerfile/blob/8c51f0c668a64153ef5cf6eefb417768773ad782/pylib/main.go#L4
• https://github.com/DataDog/go-python3/blob/0b25cc550560f4b3e49c89af907f7cfeffe94dac/bytes.go#L11

My code is just a minimal reproducible example.

I think the author of this blog post is on your team:

https://blog.filippo.io/building-python-modules-with-go-1-5/

It explains why someone would want to write cgo code that does a #include <Python.h>.

Thanks for the pointers.

/cc @FiloSottile

@ianlancetaylor I'd like to help with this issue. Can you point me in the right direction? Thanks.

At this point I don't know what the right direction is. Changes in this area have a history of breaking existing code.

What about a cgo directive? Something like:

// #cgo include <foo.h>

A #cgo directive seems like kind of a last resort. I would hate to have to write the documentation that explains when you should write #include and when you should write #cgo include. While cgo is not particularly simple, our goal as to always be to make it as simple as possible, and distinguishing #include and #cgo include cannot be described as simple.

Quite a tricky change to get into this cycle, thus I shall move it to Go1.16.

Moving to Unplanned, since currently we don't have a plan to address this.