Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: failed to debug while succeeding to compile and start server #58330

Closed
adzfolc opened this issue Feb 5, 2023 · 12 comments
Closed

cmd/link: failed to debug while succeeding to compile and start server #58330

adzfolc opened this issue Feb 5, 2023 · 12 comments
Assignees
@thanm
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@adzfolc
Copy link

adzfolc commented Feb 5, 2023 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.19.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes, I switch to latest go version by executing brew install go, and I still can't debug crdb while I can compile whole project.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
go env
GO111MODULE="on"
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/adzfolc/Library/Caches/go-build"
GOENV="/Users/adzfolc/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/adzfolc/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/adzfolc/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.com.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/adzfolc/go/src/github.com/cockroachdb/cockroach/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/7k/lh4pzfn52pzc6bg_n40__4l40000gn/T/go-build2652470123=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  • Procedure to reproduce
  1. Hardware and OS
HardWare: arm64
CPU: Apple M1 Max
OS: macOS 13.2 (22D49)
  1. Install go, download go1.19.4.darwin-arm64.pkg and install
  2. Git clone CRDB code and compile
  3. Build CRDB as follows by bazel
  • cd $GOPATH/src/github.com/cockroachdb/cockroach
  • ./dev gen go
  • ./dev gen bazel
  • ./dev gen cgo
  • ./dev build short
  1. Debug CRDB code in GoLand, conf like below, detaild Program arguments in GoLand is start-single-node --listen-addr=127.0.0.1:26257 --http-addr=127.0.0.1:8081 --store /Users/adzfolc/Workspace/crdbdata --insecure
    Pic without program arguments
  2. Debug failed, GoLand reports err like below
# github.com/cockroachdb/cockroach/pkg/cmd/cockroach
panic: unexpected 32-bit segment

goroutine 1 [running]:
cmd/link/internal/ld.machoCombineDwarf(0x102c35a08?, 0x140006a4570?, 0x14000627180, {0x1401d1af540?, 0x21?}, {0x1401cf675c0, 0x52})
	/usr/local/go/src/cmd/link/internal/ld/macho_combine_dwarf.go:218 +0xfb4
cmd/link/internal/ld.(*Link).hostlink(0x14000150000)
	/usr/local/go/src/cmd/link/internal/ld/lib.go:1830 +0x4540
cmd/link/internal/ld.Main(_, {0x10, 0x20, 0x1, 0x1f, 0x1e, 0x7c00000, {0x102b6c802, 0x14}, {0x102b7016d, ...}, ...})
	/usr/local/go/src/cmd/link/internal/ld/main.go:378 +0x1444
main.main()
	/usr/local/go/src/cmd/link/main.go:72 +0xd98

Compilation finished with exit code 2
  • Appendix
  1. I used to compile crdb with go installed by brew install go, and cockroach compile by go shows different go version from go version.
➜  cockroach git:(master) ✗ go version
go version go1.19.5 darwin/arm64
➜  cockroach git:(master) ✗ ./cockroach version
Build Tag:        v23.1.0-alpha.1-1527-gdc32242213-dirty
Build Time:       2023/02/02 02:48:02
Distribution:     CCL
Platform:         darwin arm64 (arm64-apple-darwin22.3.0)
Go Version:       go1.19.4
C Compiler:       Apple LLVM 14.0.0 (clang-1400.0.29.202)
Build Commit ID:  dc3224221324dcc4336861232003f2ec1aac4f9c
Build Type:       development
  1. I can start crdb server with compiled binary cockroach like below
➜  cockroach git:(master) ./cockroach-short start-single-node --listen-addr=127.0.0.1:26257 --http-addr=127.0.0.1:8081 --store /Users/adzfolc/Workspace/crdbdata --insecure 
*
* WARNING: ALL SECURITY CONTROLS HAVE BEEN DISABLED!
* 
* This mode is intended for non-production testing only.
* 
* In this mode:
* - Your cluster is open to any client that can access 127.0.0.1.
* - Intruders with access to your machine or network can observe client-server traffic.
* - Intruders can log in without password and read or write any data in the cluster.
* - Intruders can consume all your server's resources and cause unavailability.
*
*
* INFO: To start a secure server without mandating TLS for clients,
* consider --accept-sql-without-tls instead. For other options, see:
* 
* - https://go.crdb.dev/issue-v/53404/v23.1
* - https://www.cockroachlabs.com/docs/v23.1/secure-a-cluster.html
*
*
* WARNING: Running a server without --sql-addr, with a combined RPC/SQL listener, is deprecated.
* This feature will be removed in the next version of CockroachDB.
*
CockroachDB node starting at 2023-02-05 03:03:31.559729 +0000 UTC m=+1.257295668 (took 1.2s)
build:               CCL v23.1.0-alpha.1-1629-g8e24570fa3 @ 2023/02/05 02:37:21 (go1.19.4)
webui:               http://127.0.0.1:8081
sql:                 postgresql://root@127.0.0.1:26257/defaultdb?sslmode=disable
sql (JDBC):          jdbc:postgresql://127.0.0.1:26257/defaultdb?sslmode=disable&user=root
RPC client flags:    ./cockroach-short <client cmd> --host=127.0.0.1:26257 --insecure
logs:                /Users/adzfolc/Workspace/crdbdata/logs
temp dir:            /Users/adzfolc/Workspace/crdbdata/cockroach-temp2132907417
external I/O path:   /Users/adzfolc/Workspace/crdbdata/extern
store[0]:            path=/Users/adzfolc/Workspace/crdbdata
storage engine:      pebble
clusterID:           4413793c-7314-486c-98c6-8253b2bb6ac9
status:              restarted pre-existing node
nodeID:              1

And I can connect the server, and executing queries normally.

➜  ~ alias crdb
crdb='/Users/adzfolc/go/src/github.com/cockroachdb/cockroach/cockroach sql --certs-dir=certs --insecure --host=localhost:26257'
➜  ~ crdb
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v23.1.0-alpha.1-1629-g8e24570fa3 (arm64-apple-darwin22.3.0, built 2023/02/05 02:37:21, go1.19.4) (same version as client)
# Cluster ID: 4413793c-7314-486c-98c6-8253b2bb6ac9
#
# Enter \? for a brief introduction.
#
root@localhost:26257/defaultdb> select count(1) from t;                                                                                                                  
  count
---------
   1000
(1 row)

Time: 11ms total (execution 10ms / network 1ms)

root@localhost:26257/defaultdb>

What did you expect to see?

  1. I hope to debug crdb code normally as before, I'm not sure what is the real cause leading to the problem.

What did you see instead?

  1. GoLand debug crdb code failed, while I can start binary cockroach normally.
GOROOT=/usr/local/go #gosetup
GOPATH=/Users/adzfolc/go #gosetup
/usr/local/go/bin/go build -o /private/var/folders/7k/lh4pzfn52pzc6bg_n40__4l40000gn/T/GoLand/___crdb -gcflags all=-N -l . #gosetup
# github.com/knz/strtime
gmtime_r.c:47:1: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
# github.com/cockroachdb/cockroach/pkg/cmd/cockroach
panic: unexpected 32-bit segment

goroutine 1 [running]:
cmd/link/internal/ld.machoCombineDwarf(0x102c35a08?, 0x140006a4570?, 0x14000627180, {0x1401d1af540?, 0x21?}, {0x1401cf675c0, 0x52})
	/usr/local/go/src/cmd/link/internal/ld/macho_combine_dwarf.go:218 +0xfb4
cmd/link/internal/ld.(*Link).hostlink(0x14000150000)
	/usr/local/go/src/cmd/link/internal/ld/lib.go:1830 +0x4540
cmd/link/internal/ld.Main(_, {0x10, 0x20, 0x1, 0x1f, 0x1e, 0x7c00000, {0x102b6c802, 0x14}, {0x102b7016d, ...}, ...})
	/usr/local/go/src/cmd/link/internal/ld/main.go:378 +0x1444
main.main()
	/usr/local/go/src/cmd/link/main.go:72 +0xd98
  1. For more details, u can refer to [BugReport]Failed to print values by executing cockroach debug keys path --values cockroachdb/cockroach#96148. There are detailed chat history between me and crdb community members.
@adzfolc adzfolc changed the title linkers: Failed to debug while succeeding to compile linkers: Failed to debug while succeeding to compile and start server Feb 5, 2023
@adzfolc adzfolc mentioned this issue Feb 5, 2023
Closed
@adzfolc
Copy link
Author

adzfolc commented Feb 5, 2023

For detailed chat history between me and crdb community members, plz refer to cockroachdb/cockroach#96148. I'm looking forward to your replies sincerely!!!

@adzfolc
Copy link
Author

adzfolc commented Feb 5, 2023

In addition, I just tried to remote debug in GoLand which needs gops. I tried to install go ps and failed.

➜  ~ go install github.com/google/gops@latest
# github.com/google/gops
/usr/local/go/pkg/tool/darwin_arm64/link: /usr/local/go/pkg/tool/darwin_arm64/link: combining dwarf failed: no room to add dwarf info. Need at least 1032 padding bytes, found 508
➜  ~ go env                                  
GO111MODULE="on"
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/adzfolc/Library/Caches/go-build"
GOENV="/Users/adzfolc/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/adzfolc/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/adzfolc/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.com.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/7k/lh4pzfn52pzc6bg_n40__4l40000gn/T/go-build1152466667=/tmp/go-build -gno-record-gcc-switches -fno-common"
➜  ~

@seankhliao seankhliao changed the title linkers: Failed to debug while succeeding to compile and start server cmd/link: failed to debug while succeeding to compile and start server Feb 5, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Feb 5, 2023
@dr2chase
Copy link
Contributor

dr2chase commented Feb 6, 2023

The program you are trying to install is "gopls", go install golang.org/x/tools/gopls@latest, but gopls is not a debugger itself.

For debuggers, you want either delve or glv:

  • go install -v github.com/go-delve/delve/cmd/dlv@latest
  • go install github.com/aarzilli/gdlv@latest

They're somewhat more Go-savvy and Go-specific than gdb or lldb.

@adzfolc
Copy link
Author

adzfolc commented Feb 7, 2023

go install golang.org/x/tools/gopls@latest

Thanks for reply.
I tried to install delve and succeeded, and failed to install gdlv and gopls.

➜  cockroach git:(master) ✗ go install github.com/aarzilli/gdlv@latest
# github.com/aarzilli/gdlv
panic: unexpected 32-bit segment

goroutine 1 [running]:
cmd/link/internal/ld.machoCombineDwarf(0x100909a08?, 0x140006160c0?, 0x140005cc180, {0x14001464870?, 0x1?}, {0x14001ee2120, 0x53})
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/macho_combine_dwarf.go:218 +0xfb4
cmd/link/internal/ld.(*Link).hostlink(0x14000154000)
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/lib.go:1830 +0x4540
cmd/link/internal/ld.Main(_, {0x10, 0x20, 0x1, 0x1f, 0x1e, 0x7c00000, {0x100840842, 0x14}, {0x1008441ad, ...}, ...})
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/main.go:378 +0x1444
main.main()
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/main.go:72 +0xd98
➜  cockroach git:(master) ✗ go install golang.org/x/tools/gopls@latest
# golang.org/x/tools/gopls
panic: unexpected 32-bit segment

goroutine 1 [running]:
cmd/link/internal/ld.machoCombineDwarf(0x10094da08?, 0x1400000e0c0?, 0x14003c1c100, {0x14001ad42d0?, 0x1?}, {0x14001bea8a0, 0x53})
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/macho_combine_dwarf.go:218 +0xfb4
cmd/link/internal/ld.(*Link).hostlink(0x14000154000)
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/lib.go:1830 +0x4540
cmd/link/internal/ld.Main(_, {0x10, 0x20, 0x1, 0x1f, 0x1e, 0x7c00000, {0x100884842, 0x14}, {0x1008881ad, ...}, ...})
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/internal/ld/main.go:378 +0x1444
main.main()
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/cmd/link/main.go:72 +0xd98

Summary:

  1. I tried to remote debug in GoLand who noticed me to install gops.
    图片
    图片
# github.com/google/gops
/opt/homebrew/Cellar/go/1.19.5/libexec/pkg/tool/darwin_arm64/link: /opt/homebrew/Cellar/go/1.19.5/libexec/pkg/tool/darwin_arm64/link: combining dwarf failed: no room to add dwarf info. Need at least 1032 padding bytes, found 476
  1. I want to local debug cockroachdb, and failed like these
GOROOT=/opt/homebrew/Cellar/go/1.19.5/libexec #gosetup
GOPATH=/Users/adzfolc/go #gosetup
/opt/homebrew/Cellar/go/1.19.5/libexec/bin/go build -o /private/var/folders/7k/lh4pzfn52pzc6bg_n40__4l40000gn/T/GoLand/___crdb -gcflags all=-N -l . #gosetup
# github.com/knz/strtime
gmtime_r.c:47:1: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
# github.com/cockroachdb/cockroach/pkg/cmd/cockroach
/opt/homebrew/Cellar/go/1.19.5/libexec/pkg/tool/darwin_arm64/link: /opt/homebrew/Cellar/go/1.19.5/libexec/pkg/tool/darwin_arm64/link: combining dwarf failed: no room to add dwarf info. Need at least 1032 padding bytes, found 696

Compilation finished with exit code 2

@mknyszek mknyszek added this to the Backlog milestone Feb 8, 2023
@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 8, 2023
@thanm
Copy link
Contributor

thanm commented Feb 8, 2023

Thanks for the report. It would be helpful if you could relate which xcode version you are using.

@thanm
Copy link
Contributor

thanm commented Feb 8, 2023

OK, I have an arm64 mac with the right version of the OS installed, I made a few attempts to reproduce, but no luck so far.

Based the linker failure mode, my best guess would be that the C/C++ compiler is emitting something that is making the Go linker unhappy, so until I can set things up on my machine to use exactly that version of clang, I won't be able to repro. Please do post more info about what version of xcode you have installed, thanks.

@thanm thanm added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 8, 2023
@adzfolc
Copy link
Author

adzfolc commented Feb 9, 2023 •
edited

Thanks for the report. It would be helpful if you could relate which xcode version you are using.

xcode, gcc, g++, clang version as following

➜  ~ /usr/bin/xcodebuild -version
Xcode 14.2
Build version 14C18
➜  ~ gcc -v
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
➜  ~ g++ -v
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
➜  ~ clang -v     
Homebrew clang version 15.0.7
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin

@adzfolc
Copy link
Author

adzfolc commented Feb 9, 2023

go version and go env as following

➜  ~ go version
go version go1.19.5 darwin/arm64
➜  ~ go env
GO111MODULE="on"
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/adzfolc/Library/Caches/go-build"
GOENV="/Users/adzfolc/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/adzfolc/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/adzfolc/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/opt/homebrew/Cellar/go/1.19.5/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.19.5/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.19.5"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/7k/lh4pzfn52pzc6bg_n40__4l40000gn/T/go-build3757960505=/tmp/go-build -gno-record-gcc-switches -fno-common"

@thanm
Copy link
Contributor

thanm commented Feb 10, 2023

Well, I am stumped. I have set up an arm64 macbook so that it has the same configuration as the one you've reported, but I can't seem to reproduce the failure you're seeing at all (things build fine).

$ uname -m
arm64
$ sw_vers
ProductName: macOS
ProductVersion: 13.2
BuildVersion: 22D49
$ clang -v
Homebrew clang version 15.0.7
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin
$ gcc -v
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
$ go version
go version go1.19 darwin/arm64
$

Have you tried this on other machines with similar setups?

@adzfolc
Copy link
Author

adzfolc commented Feb 18, 2023

Well, I am stumped. I have set up an arm64 macbook so that it has the same configuration as the one you've reported, but I can't seem to reproduce the failure you're seeing at all (things build fine).

$ uname -m
arm64
$ sw_vers
ProductName: macOS
ProductVersion: 13.2
BuildVersion: 22D49
$ clang -v
Homebrew clang version 15.0.7
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin
$ gcc -v
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
$ go version
go version go1.19 darwin/arm64
$

Have you tried this on other machines with similar setups?

After re-init my Mac for several times, and re-install dev chain, finally I can debug now.Thanks for your reply!

@adzfolc adzfolc closed this as completed Feb 21, 2023
@zjwang001
Copy link

macho_combine_dwarf

这个问题怎么解决的呢 有谁知道吗

@adzfolc
Copy link
Author

adzfolc commented Jul 3, 2023

更新工具链 并 重试
Update your complier to appopriate version and retry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thanm thanm

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
6 participants
@mknyszek @dr2chase @gopherbot @thanm @zjwang001 @adzfolc