cmd/go: cgo hang on a darwin machine #21178

jimrobinson · 2017-07-26T16:06:22Z

Please answer these questions before submitting your issue. Thanks!

What did you do?

I was running "go build" on a project, and saw the process not return for several minutes, which is unusual, a typical build takes 3-4 seconds. Looking at ps(1) output, the build appeared to be waiting on a "cgo" job within the github.com/confluentinc/confluent-kafka-go/kafka package. I finally killed the job, and subsequent builds worked w/o a problem.

What did you expect to see?

I would expect to see go build return success or return compile errors relatively quickly. The subsequent successful builds took 3-4 seconds, the hung process had been hanging for 2+ minutes.

What did you see instead?

A hung process. An lsof of the job showed nothing significant open by the offending job:

$ lsof -p 4651
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF      NODE NAME
cgo     4651 jimr  cwd    DIR                1,4     1122 151466918 /Users/jimr/Library/gocode/src/github.com/confluentinc/confluent-kafka-go/kafka
cgo     4651 jimr  txt    REG                1,4  3850160 174266652 /usr/local/go/pkg/tool/darwin_amd64/cgo
cgo     4651 jimr  txt    REG                1,4   698896 195136932 /usr/lib/dyld
cgo     4651 jimr    0r   CHR                3,2      0t0       304 /dev/null
cgo     4651 jimr    1   PIPE 0x9180c662b5ede611    16384           ->0x9180c662b5edbb51
cgo     4651 jimr    2   PIPE 0x9180c662b5ede611    16384           ->0x9180c662b5edbb51
cgo     4651 jimr    3   PIPE 0x9180c662b5ede6d1    16384           ->0x9180c662b5ede791
cgo     4651 jimr    4   PIPE 0x9180c662b5ede791    16384           ->0x9180c662b5ede6d1
cgo     4651 jimr    5   PIPE 0x9180c662b5ede3d1    16384           ->0x9180c662a6fa3511
cgo     4651 jimr    6   PIPE 0x9180c662a6fa3511    16384           ->0x9180c662b5ede3d1
cgo     4651 jimr    7   PIPE 0x9180c662a6fa29d1    16384           ->0x9180c662a6fa44d1
cgo     4651 jimr    8   PIPE 0x9180c662a6fa44d1    16384           ->0x9180c662a6fa29d1
cgo     4651 jimr    9   PIPE 0x9180c662a6fa2b51    16384           ->0x9180c662a6fa2a91
cgo     4651 jimr   10   PIPE 0x9180c662a6fa2a91    16384           ->0x9180c662a6fa2b51

I stupidly managed to lose the exact command being run, but I believe it was something like this one:

  501  5633  5606   0  9:02AM ttys000    0:00.09 /usr/local/go/pkg/tool/darwin_amd64/cgo -objdir /var/folders/jz/618480gn79q0flkfzpspqs8h0000gn/T/go-build476481432/github.com/confluentinc/confluent-kafka-go/kafka/_obj/ -importpath github.com/confluentinc/confluent-kafka-go/kafka -- -I/usr/local/Cellar/librdkafka/0.9.4/include -I /var/folders/jz/618480gn79q0flkfzpspqs8h0000gn/T/go-build476481432/github.com/confluentinc/confluent-kafka-go/kafka/_obj/ -g -O2 build_dynamic.go config.go consumer.go error.go event.go generated_errors.go handle.go kafka.go message.go metadata.go misc.go producer.go testhelpers.go

When I killed the offending process, I got the following output:

# github.com/confluentinc/confluent-kafka-go/kafka
clang produced no output
on input:
#line 23 "/Users/jimr/Library/gocode/src/github.com/confluentinc/confluent-kafka-go/kafka/metadata.go"

#include <stdlib.h>
#include <librdkafka/rdkafka.h>

struct rd_kafka_metadata_broker *_getMetadata_broker_element(struct rd_kafka_metadata *m, int i) {
  return &m->brokers[i];
}

struct rd_kafka_metadata_topic *_getMetadata_topic_element(struct rd_kafka_metadata *m, int i) {
  return &m->topics[i];
}

struct rd_kafka_metadata_partition *_getMetadata_partition_element(struct rd_kafka_metadata *m, int topic_idx, int partition_idx) {
  return &m->topics[topic_idx].partitions[partition_idx];
}

int32_t _get_int32_element (int32_t *arr, int i) {
  return arr[i];
}


#line 1 "cgo-generated-wrapper"

#line 1 "cgo-builtin-prolog"
#include <stddef.h> /* for ptrdiff_t and size_t below */

/* Define intgo when compiling with GCC.  */
typedef ptrdiff_t intgo;

typedef struct { char *p; intgo n; } _GoString_;
typedef struct { char *p; intgo n; intgo c; } _GoBytes_;
_GoString_ GoString(char *p);
_GoString_ GoStringN(char *p, int l);
_GoBytes_ GoBytes(void *p, int n);
char *CString(_GoString_);
void *CBytes(_GoBytes_);
void *_CMalloc(size_t);
#line 1 "not-declared"
void __cgo_f_1_1(void) { __typeof__(CString) *__cgo_undefined__; }
#line 1 "not-type"
void __cgo_f_1_2(void) { CString *__cgo_undefined__; }
#line 1 "not-const"
void __cgo_f_1_3(void) { enum { __cgo__undefined__ = (CString)*1 }; }
#line 2 "not-declared"
void __cgo_f_2_1(void) { __typeof__(GoString) *__cgo_undefined__; }
#line 2 "not-type"
void __cgo_f_2_2(void) { GoString *__cgo_undefined__; }
#line 2 "not-const"
void __cgo_f_2_3(void) { enum { __cgo__undefined__ = (GoString)*1 }; }
#line 3 "not-declared"
void __cgo_f_3_1(void) { __typeof__(RD_KAFKA_RESP_ERR_NO_ERROR) *__cgo_undefined__; }
#line 3 "not-type"
void __cgo_f_3_2(void) { RD_KAFKA_RESP_ERR_NO_ERROR *__cgo_undefined__; }
#line 3 "not-const"
void __cgo_f_3_3(void) { enum { __cgo__undefined__ = (RD_KAFKA_RESP_ERR_NO_ERROR)*1 }; }
#line 4 "not-declared"
void __cgo_f_4_1(void) { __typeof__(_getMetadata_broker_element) *__cgo_undefined__; }
#line 4 "not-type"
void __cgo_f_4_2(void) { _getMetadata_broker_element *__cgo_undefined__; }
#line 4 "not-const"
void __cgo_f_4_3(void) { enum { __cgo__undefined__ = (_getMetadata_broker_element)*1 }; }
#line 5 "not-declared"
void __cgo_f_5_1(void) { __typeof__(_getMetadata_partition_element) *__cgo_undefined__; }
#line 5 "not-type"
void __cgo_f_5_2(void) { _getMetadata_partition_element *__cgo_undefined__; }
#line 5 "not-const"
void __cgo_f_5_3(void) { enum { __cgo__undefined__ = (_getMetadata_partition_element)*1 }; }
#line 6 "not-declared"
void __cgo_f_6_1(void) { __typeof__(_getMetadata_topic_element) *__cgo_undefined__; }
#line 6 "not-type"
void __cgo_f_6_2(void) { _getMetadata_topic_element *__cgo_undefined__; }
#line 6 "not-const"
void __cgo_f_6_3(void) { enum { __cgo__undefined__ = (_getMetadata_topic_element)*1 }; }
#line 7 "not-declared"
void __cgo_f_7_1(void) { __typeof__(_get_int32_element) *__cgo_undefined__; }
#line 7 "not-type"
void __cgo_f_7_2(void) { _get_int32_element *__cgo_undefined__; }
#line 7 "not-const"
void __cgo_f_7_3(void) { enum { __cgo__undefined__ = (_get_int32_element)*1 }; }
#line 8 "not-declared"
void __cgo_f_8_1(void) { __typeof__(free) *__cgo_undefined__; }
#line 8 "not-type"
void __cgo_f_8_2(void) { free *__cgo_undefined__; }
#line 8 "not-const"
void __cgo_f_8_3(void) { enum { __cgo__undefined__ = (free)*1 }; }
#line 9 "not-declared"
void __cgo_f_9_1(void) { __typeof__(int) *__cgo_undefined__; }
#line 9 "not-type"
void __cgo_f_9_2(void) { int *__cgo_undefined__; }
#line 9 "not-const"
void __cgo_f_9_3(void) { enum { __cgo__undefined__ = (int)*1 }; }
#line 10 "not-declared"
void __cgo_f_10_1(void) { __typeof__(int32_t) *__cgo_undefined__; }
#line 10 "not-type"
void __cgo_f_10_2(void) { int32_t *__cgo_undefined__; }
#line 10 "not-const"
void __cgo_f_10_3(void) { enum { __cgo__undefined__ = (int32_t)*1 }; }
#line 11 "not-declared"
void __cgo_f_11_1(void) { __typeof__(int64_t) *__cgo_undefined__; }
#line 11 "not-type"
void __cgo_f_11_2(void) { int64_t *__cgo_undefined__; }
#line 11 "not-const"
void __cgo_f_11_3(void) { enum { __cgo__undefined__ = (int64_t)*1 }; }
#line 12 "not-declared"
void __cgo_f_12_1(void) { __typeof__(rd_kafka_metadata) *__cgo_undefined__; }
#line 12 "not-type"
void __cgo_f_12_2(void) { rd_kafka_metadata *__cgo_undefined__; }
#line 12 "not-const"
void __cgo_f_12_3(void) { enum { __cgo__undefined__ = (rd_kafka_metadata)*1 }; }
#line 13 "not-declared"
void __cgo_f_13_1(void) { __typeof__(rd_kafka_query_watermark_offsets) *__cgo_undefined__; }
#line 13 "not-type"
void __cgo_f_13_2(void) { rd_kafka_query_watermark_offsets *__cgo_undefined__; }
#line 13 "not-const"
void __cgo_f_13_3(void) { enum { __cgo__undefined__ = (rd_kafka_query_watermark_offsets)*1 }; }
#line 14 "not-declared"
void __cgo_f_14_1(void) { __typeof__(rd_kafka_topic_t) *__cgo_undefined__; }
#line 14 "not-type"
void __cgo_f_14_2(void) { rd_kafka_topic_t *__cgo_undefined__; }
#line 14 "not-const"
void __cgo_f_14_3(void) { enum { __cgo__undefined__ = (rd_kafka_topic_t)*1 }; }
#line 1 "completed"
int __cgo__1 = __cgo__2;

System details

go version go1.8.3 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jimr/Library/gocode:/Users/jimr/proj/github:/Users/jimr/proj/code/gocode/trunk"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/jz/618480gn79q0flkfzpspqs8h0000gn/T/go-build427121574=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
GOROOT/bin/go version: go version go1.8.3 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.8.3 X:framepointer
uname -v: Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.12.6
BuildVersion:	16G29
lldb --version: lldb-370.0.42
  Swift-3.1
gdb --version: GNU gdb (GDB) 7.12.1

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2017-07-26T17:50:44Z

Do I understand correctly that this happened once, and when you retried it everything was fine?

That is going to be hard to fix.

jimrobinson · 2017-07-26T17:55:12Z

Yes, it happened once, I re-ran using "go build -v -a" which worked fine, and every subsequent "go build" worked. Feel free to reject/boot the ticket since I know the details aren't very useful. If I had been building this on a linux machine I'd have tried to attach with strace, but the dtrace command doesn't seem to operate in a useful way on darwin. :( Are there commands you know of that I could have run to get more useful data?

…

On Wed, Jul 26, 2017 at 10:51 AM, Ian Lance Taylor ***@***.*** > wrote: Do I understand correctly that this happened once, and when you retried it everything was fine? That is going to be hard to fix. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#21178 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUAsmJWOajgLIR8ioTkHLDpmHxgzfVvks5sR3yxgaJpZM4OkJK4> .

bcmills · 2017-07-26T18:20:49Z

Are there commands you know of that I could have run to get more useful
data?

If you kill a Go program with SIGQUIT (^\) instead of SIGINT (^C), you should normally get a goroutine traceback, which might help to identify where the program is hung.

rsc · 2017-12-01T01:55:46Z

Not reproducible.

bradfitz added this to the Go1.10 milestone Jul 26, 2017

bradfitz added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 26, 2017

bradfitz changed the title ~~cgo hang on a darwin machine~~ cmd/go: cgo hang on a darwin machine Jul 26, 2017

bradfitz added OS-Darwin help wanted labels Jul 26, 2017

rsc closed this as completed Dec 1, 2017

golang locked and limited conversation to collaborators Dec 1, 2018

gopherbot added the FrozenDueToAge label Dec 1, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: cgo hang on a darwin machine #21178

cmd/go: cgo hang on a darwin machine #21178

jimrobinson commented Jul 26, 2017

ianlancetaylor commented Jul 26, 2017

jimrobinson commented Jul 26, 2017 via email

bcmills commented Jul 26, 2017

rsc commented Dec 1, 2017

cmd/go: cgo hang on a darwin machine #21178

cmd/go: cgo hang on a darwin machine #21178

Comments

jimrobinson commented Jul 26, 2017

What did you do?

What did you expect to see?

What did you see instead?

System details

ianlancetaylor commented Jul 26, 2017

jimrobinson commented Jul 26, 2017 via email

bcmills commented Jul 26, 2017

rsc commented Dec 1, 2017