Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/cgo: macro expansion producing 'defined' has undefined behavior #38876

Closed
Sr0121 opened this issue May 5, 2020 · 23 comments
Closed

runtime/cgo: macro expansion producing 'defined' has undefined behavior #38876

Sr0121 opened this issue May 5, 2020 · 23 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@Sr0121
Copy link

Sr0121 commented May 5, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

OS: macOS Catalina 10.15.4

go env Output
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/sr/Library/Caches/go-build"
GOENV="/Users/sr/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/sr/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/7t/mh0c8z9x6rn72vfl7jxrqh4r0000gn/T/go-build284435102=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Try to run this code:

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", indexHandler)
	log.Fatal(http.ListenAndServe(":9999", nil))
}

// handler echoes r.URL.Path
func indexHandler(w http.ResponseWriter, req *http.Request) {
	fmt.Fprintf(w, "URL.Path = %q\n", req.URL.Path)
}

What did you expect to see?

Expect the program to run.

What did you see instead?

# runtime/cgo
In file included from gcc_darwin_amd64.c:6:
/usr/local/include/pthread.h:331:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:2: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
/usr/local/include/pthread.h:331:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:34: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
/usr/local/include/pthread.h:332:5: error: declaration of built-in function 'pthread_create' requires inclusion of the header <pthread.h> [-Werror,-Wbuiltin-requires-header]
/usr/local/include/pthread.h:540:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:2: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
/usr/local/include/pthread.h:540:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:34: note: expanded from macro     ####'_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
@ALTree
Copy link
Member

ALTree commented May 5, 2020

This is a dup of #38552 but you provided a reproducer so let's keep this one and I'll close the other thread.

@ALTree ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 5, 2020
@ALTree ALTree added this to the Go1.15 milestone May 5, 2020
@ianlancetaylor
Copy link
Contributor

What do you see if you create a file foo.c containing

#include <string.h>
#include <pthread.h>

and then run

cc -c -Wall -Werror foo.c

?

@Sr0121
Copy link
Author

Sr0121 commented May 5, 2020

What do you see if you create a file foo.c containing

#include <string.h>
#include <pthread.h>

and then run

cc -c -Wall -Werror foo.c

?

@ianlancetaylor

In file included from foo.c:2:
/usr/local/include/pthread.h:331:6: error: macro expansion producing 'defined'
      has undefined behavior [-Werror,-Wexpansion-to-defined]
#if !_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT
     ^
/usr/local/include/pthread.h:200:2: note: expanded from macro
      '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
        defined(SWIFT_CLASS_EXTRA) && (!defined(SWIFT_SDK_OVERLAY_PTHREA...
        ^
/usr/local/include/pthread.h:331:6: error: macro expansion producing 'defined'
      has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:34: note: expanded from macro
      '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
        defined(SWIFT_CLASS_EXTRA) && (!defined(SWIFT_SDK_OVERLAY_PTHREA...
                                        ^
/usr/local/include/pthread.h:332:5: error: declaration of built-in function
      'pthread_create' requires inclusion of the header <pthread.h>
      [-Werror,-Wbuiltin-requires-header]
int pthread_create(pthread_t _Nullable * _Nonnull __restrict,
    ^
/usr/local/include/pthread.h:540:6: error: macro expansion producing 'defined'
      has undefined behavior [-Werror,-Wexpansion-to-defined]
#if !_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT
     ^
/usr/local/include/pthread.h:200:2: note: expanded from macro
      '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
        defined(SWIFT_CLASS_EXTRA) && (!defined(SWIFT_SDK_OVERLAY_PTHREA...
        ^
/usr/local/include/pthread.h:540:6: error: macro expansion producing 'defined'
      has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:34: note: expanded from macro
      '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
        defined(SWIFT_CLASS_EXTRA) && (!defined(SWIFT_SDK_OVERLAY_PTHREA...
                                        ^

@ianlancetaylor
Copy link
Contributor

Thanks. Seems to me that your C compiler is broken, or at least that it doesn't agree with your library.

Is there some option that we can pass to the compiler to disable this pointless error? Does it work if you run

cc -Wall -Werror -Wno-expansion-to-defined foo.c

?

@Sr0121
Copy link
Author

Sr0121 commented May 6, 2020

Thanks. Seems to me that your C compiler is broken, or at least that it doesn't agree with your library.

Is there some option that we can pass to the compiler to disable this pointless error? Does it work if you run

cc -Wall -Werror -Wno-expansion-to-defined foo.c

?

A new error has occurred.

In file included from foo.c:2:
/usr/local/include/pthread.h:332:5: error: declaration of built-in function
      'pthread_create' requires inclusion of the header <pthread.h>
      [-Werror,-Wbuiltin-requires-header]
int pthread_create(pthread_t _Nullable * _Nonnull __restrict,
    ^
1 error generated.

@ianlancetaylor
Copy link
Contributor

I really don't understand that one.

That said, why it is including from /usr/local/include? Is that where standard library header files live on Darwin? Or is there another <pthread.h> that it should be using instead? What happens if you temporarily remove /usr/local/include/pthread.h and compile foo.c?

@victorneuret
Copy link

Hi!
I already encountered an error like that one.
I've had to add CGO_CPPFLAGS=-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wbuiltin-requires-header to my env to make it work.
I think that it's just removing the compilation flags.

My program is the same than yours and I was able to make it work using this.
I'm using MacOs Catalina 10.15.4

@Sr0121
Copy link
Author

Sr0121 commented May 6, 2020

Thank you all for your help. I finally found out that my Xcode command line tools can not work as expectation. What I did is to run these lines in Terminal.

sudo rm -rf /Library/Developer/CommandLineTools
xcode-select --install

@Sr0121 Sr0121 closed this as completed May 6, 2020
@stephanGarland
Copy link

FWIW I had the same issue (different codebase, tried using OP's code and reproduced issue), and reinstalling xcode did NOT fix it.

Installing gcc with brew and then explicitly setting CC and CXX to the symlinked gcc binaries did fix it.

@erikterwiel
Copy link

Hi!
I already encountered an error like that one.
I've had to add CGO_CPPFLAGS=-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wbuiltin-requires-header to my env to make it work.
I think that it's just removing the compilation flags.

My program is the same than yours and I was able to make it work using this.
I'm using MacOs Catalina 10.15.4

This works for me except -Wbuiltin-requires-header -> -Wno-builtin-requires-header.
Then, what add to env is: CGO_CPPFLAGS=-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-builtin-requires-header

@codeBehindMe
Copy link

So i noticed this issue on MacOs Catalina 10.15.4 as well, out of the blue. I tried the solution that @Sr0121 suggested, didn't work, I also tried the reinstalling gcc as per @stephanGarland's comment, to no avail.

Finally I upgraded 10.15.5 and did the above as well, still didn't change anything. I had to add compiler flags as per @victorneuret .

Kind of annoying how this came out of the blue.
I also tried different go version as well.

@sharpSteff
Copy link

@erikterwiel That worked for me as well!
Thank you!

@sharpSteff
Copy link

found out.. I recently symlinked gcc headers into usr/local/include to get some other project running.
After removing these symlinks from usr/local/include, everything works fine again

@codeBehindMe
Copy link

Yea I think I did something similar to get it working.

@kangkang59812
Copy link

Hi!
I already encountered an error like that one.
I've had to add CGO_CPPFLAGS=-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wbuiltin-requires-header to my env to make it work.
I think that it's just removing the compilation flags.
My program is the same than yours and I was able to make it work using this.
I'm using MacOs Catalina 10.15.4

This works for me except -Wbuiltin-requires-header -> -Wno-builtin-requires-header.
Then, what add to env is: CGO_CPPFLAGS=-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-builtin-requires-header

where to add and how?

@kangkang59812
Copy link

symlinked

how to do it?

@emil-j-olsson
Copy link

@kangkang59812
export CGO_CPPFLAGS="-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-builtin-requires-header"

Then check your environment via go env.

@vietvudanh
Copy link

One more problem is that I got permission denied after using the CGO_CPPFLAGS flag:

go build runtime/cgo: copying /Users/hrbc/Library/Caches/go-build/63/63a132c80210b5c7b4f4ed9a902aaec712c42b15e23aa2c1b1778e2f85165d04-d: open /usr/local/go/pkg/darwin_amd64/runtime/cgo.a: permission denied

I need to grant permission to /usr/local/go for it to work. I think this should be handled by the installer?

@qxysg1273
Copy link

Hi!
I already encountered an error like that one.
I've had to add CGO_CPPFLAGS=-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wbuiltin-requires-header to my env to make it work.
I think that it's just removing the compilation flags.

My program is the same than yours and I was able to make it work using this.
I'm using MacOs Catalina 10.15.4

Thanks, this works fom me!

@chengengliu
Copy link

chengengliu commented Jun 16, 2021

Hi!
I have tried above mentioned methods, including adding -Wno-error .... and reinstalling command line tools. After that, running cc -c -Wall -Werror foo.c does not have any errors but I still get error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]. go env output:
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/Apple/Library/Caches/go-build"
GOENV="/Users/Apple/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/Apple/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/Apple/go"</br>
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.5"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS="-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-builtin-requires-header"
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 </br>-fdebug-prefix-map=/var/folders/7s/jyvyj40j623gz34m5f1pksxr0000gn/T/go-build3850003340=/tmp/go-build -gno-record-gcc-switches -fno-common".
MacOS11.4.

The error:

# runtime/cgo
In file included from gcc_darwin_amd64.c:6:
/usr/local/include/pthread.h:328:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:197:2: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'

@chengengliu
Copy link

Hi!
I have tried above mentioned methods, including adding -Wno-error .... and reinstalling command line tools. After that, running cc -c -Wall -Werror foo.c does not have any errors but I still get error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]. go env output:
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/Apple/Library/Caches/go-build"
GOENV="/Users/Apple/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/Apple/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/Apple/go"</br>
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.5"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS="-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-builtin-requires-header"
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 </br>-fdebug-prefix-map=/var/folders/7s/jyvyj40j623gz34m5f1pksxr0000gn/T/go-build3850003340=/tmp/go-build -gno-record-gcc-switches -fno-common".
MacOS11.4.

Now the still got the error:

# runtime/cgo
In file included from gcc_darwin_amd64.c:6:
/usr/local/include/pthread.h:328:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:197:2: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'

Problems solved. I reinstalled Xcode and reboot the machine. Now it works like a charm. I guess there must be something I played with gcc that broke it.

@yamasite
Copy link

yamasite commented Oct 2, 2021

@kangkang59812 export CGO_CPPFLAGS="-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-builtin-requires-header"

Then check your environment via go env.

I met the same issue when using gvm to install go1.16 in Big Sur. Your solution totally works!

mac@macdeMac-mini ~ % gvm install go1.16

Downloading Go source...

Installing go1.16...
 * Compiling...
ERROR: Failed to compile. Check the logs at /Users/mac/.gvm/logs/go-go1.16-compile.log
ERROR: Failed to use installed version
mac@macdeMac-mini ~ %
mac@macdeMac-mini ~ % cat /Users/mac/.gvm/logs/go-go1.16-compile.log
Building Go cmd/dist using /usr/local/go. (go1.17.1 darwin/amd64)
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
# runtime/cgo
In file included from gcc_darwin_amd64.c:6:
/usr/local/include/pthread.h:331:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:2: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
/usr/local/include/pthread.h:331:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:34: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
/usr/local/include/pthread.h:540:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:2: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
/usr/local/include/pthread.h:540:6: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined]
/usr/local/include/pthread.h:200:34: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
go tool dist: FAILED: /Users/mac/.gvm/gos/go1.16/pkg/tool/darwin_amd64/go_bootstrap install -gcflags=all= -ldflags=all= std cmd: exit status 2
mac@macdeMac-mini ~ % export CGO_CPPFLAGS="-Wno-error -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-builtin-requires-header"
mac@macdeMac-mini ~ % gvm install go1.16

Installing go1.16...
 * Compiling...
go1.16 successfully installed!

@chqrlie
Copy link

chqrlie commented May 8, 2022

This #define is even more bogus! The definition in <pthread.h> is this:

#define _PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT \
	defined(SWIFT_CLASS_EXTRA) && (!defined(SWIFT_SDK_OVERLAY_PTHREAD_EPOCH) || (SWIFT_SDK_OVERLAY_PTHREAD_EPOCH < 1))

The macro expansion is not properly parenthesized. The macro is used this way in the same file:

#if !_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT

Which expands to:

#if !defined(SWIFT_CLASS_EXTRA) && (!defined(SWIFT_SDK_OVERLAY_PTHREAD_EPOCH) || (SWIFT_SDK_OVERLAY_PTHREAD_EPOCH < 1))

The ! only applies to the first test defined(SWIFT_CLASS_EXTRA) not to the whole boolean expression.

This seems to be a bug in <pthread.h> for Apple to fix.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
None yet
Development

No branches or pull requests