Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: install produces executable with invalid signature (macOS) #63997

Closed
koliyo opened this issue Nov 8, 2023 · 33 comments
Closed

cmd/go: install produces executable with invalid signature (macOS) #63997

koliyo opened this issue Nov 8, 2023 · 33 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@koliyo
Copy link

koliyo commented Nov 8, 2023

I can not launch gopls at all. It is immediately killed

$ gopls
[1]    6414 killed     gopls

I have looked in system console and it seems this is related to code signing:

default	08:59:00.573715+0100	kernel	CODE SIGNING: cs_invalid_page(0x102b24000): p=4159[gopls] final status 0x23020200, denying page sending SIGKILL
default	08:59:00.573730+0100	kernel	CODE SIGNING: process 4159[gopls]: rejecting invalid page at address 0x102b24000 from offset 0x4000 in file "/Users/nils/Work/go/bin/gopls" (cs_mtime:1699430007.975302102 == mtime:1699430007.975302102) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)
default	08:59:00.573996+0100	kernel	CODE SIGNING: process 4159[gopls]: rejecting invalid page at address 0x102b24000 from offset 0x4000 in file "/Users/nils/Work/go/bin/gopls" (cs_mtime:1699430007.975302102 == mtime:1699430007.975302102) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)
default	08:59:00.574052+0100	kernel	gopls[4159] Corpse allowed 1 of 5
default	08:59:00.938868+0100	ReportCrash	Formulating fatal 309 report for corpse[4159] gopls
default	08:59:00.940445+0100	osanalyticshelper	creating type 309 as /Users/nils/Library/Logs/DiagnosticReports/.gopls-2023-11-08-085900.ips
default	08:59:00.942224+0100	osanalyticshelper	Saved type '309(<private>)' report (7 of max 25) at /Users/nils/Library/Logs/DiagnosticReports/gopls-2023-11-08-085900.ips
default	08:59:00.942312+0100	osanalyticshelper	xpc log creation type 309 result success: /Users/nils/Library/Logs/DiagnosticReports/gopls-2023-11-08-085900.ips
default	08:59:00.942394+0100	ReportCrash	client log create type 309 result success: /Users/nils/Library/Logs/DiagnosticReports/gopls-2023-11-08-085900.ips
default	08:59:00.942452+0100	ReportCrash	no MetricKit for process gopls type 309 bundleId (null)

Not sure what to do about this

Here is an example of such ips file:

{"app_name":"gopls","timestamp":"2023-11-08 08:59:00.00 +0100","app_version":"","slice_uuid":"3513ca24-39ed-3d73-96b9-9b5acf281169","build_version":"","platform":1,"share_with_app_devs":1,"is_first_party":1,"bug_type":"309","os_version":"macOS 14.1.1 (23B81)","roots_installed":0,"incident_id":"AD4F6056-9001-4D59-9314-23DCE3A4461A","name":"gopls"}
{
  "uptime" : 230,
  "procRole" : "Unspecified",
  "version" : 2,
  "userID" : 501,
  "deployVersion" : 210,
  "modelCode" : "MacBookPro18,2",
  "coalitionID" : 585,
  "osVersion" : {
    "train" : "macOS 14.1.1",
    "build" : "23B81",
    "releaseType" : "User"
  },
  "captureTime" : "2023-11-08 08:59:00.5753 +0100",
  "codeSigningMonitor" : 1,
  "incident" : "AD4F6056-9001-4D59-9314-23DCE3A4461A",
  "pid" : 4159,
  "translated" : false,
  "cpuType" : "ARM-64",
  "roots_installed" : 0,
  "bug_type" : "309",
  "procLaunch" : "2023-11-08 08:59:00.5629 +0100",
  "procStartAbsTime" : 5667297332,
  "procExitAbsTime" : 5667563642,
  "procName" : "gopls",
  "procPath" : "\/Users\/USER\/*\/gopls",
  "parentProc" : "zsh",
  "parentPid" : 3868,
  "coalitionName" : "dev.warp.Warp-Stable",
  "crashReporterKey" : "365F932B-9741-7702-C2F7-EDB5266157E3",
  "responsiblePid" : 536,
  "responsibleProc" : "stable",
  "codeSigningID" : "a.out",
  "codeSigningTeamID" : "",
  "codeSigningFlags" : 587334144,
  "codeSigningValidationCategory" : 0,
  "codeSigningTrustLevel" : 4294967295,
  "instructionByteStream" : {"beforePC":"TtkAlB8gA9XgBwD54wMeqjayAZTgB0D50P\/\/FwAAAAAAAAAAAAAAAA==","atPC":"kAtA+f9jMOvJBQBU\/g8f+P2DH\/j9IwDRgAAAtOEDH6riAx+qCAAAFA=="},
  "sip" : "enabled",
  "vmRegionInfo" : "0x102b26320 is in 0x102b20000-0x103670000;  bytes after start: 25376  bytes before end: 11836639\n      REGION TYPE                    START - END         [ VSIZE] PRT\/MAX SHRMOD  REGION DETAIL\n      UNUSED SPACE AT START\n--->  __TEXT                      102b20000-103670000    [ 11.3M] r-x\/r-x SM=COW  ...\/USER\/*\/gopls\n      __DATA_CONST                103670000-1038bc000    [ 2352K] r--\/rw- SM=COW  ...\/USER\/*\/gopls",
  "exception" : {"codes":"0x0000000000000032, 0x0000000102b26320","rawCodes":[50,4340212512],"type":"EXC_BAD_ACCESS","signal":"SIGKILL (Code Signature Invalid)","subtype":"UNKNOWN_0x32 at 0x0000000102b26320"},
  "termination" : {"flags":0,"code":2,"namespace":"CODESIGNING","indicator":"Invalid Page"},
  "ktriageinfo" : "VM - (arg = 0x0) Waiting on busy page was interrupted\nVM - (arg = 0x0) Didn't get back data for this file\nVM - (arg = 0x0) Fault was interrupted\nVM - (arg = 0x0) A memory corruption was found in executable text\n",
  "vmregioninfo" : "0x102b26320 is in 0x102b20000-0x103670000;  bytes after start: 25376  bytes before end: 11836639\n      REGION TYPE                    START - END         [ VSIZE] PRT\/MAX SHRMOD  REGION DETAIL\n      UNUSED SPACE AT START\n--->  __TEXT                      102b20000-103670000    [ 11.3M] r-x\/r-x SM=COW  ...\/USER\/*\/gopls\n      __DATA_CONST                103670000-1038bc000    [ 2352K] r--\/rw- SM=COW  ...\/USER\/*\/gopls",
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":4,"task_for_pid":1},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":30274,"threadState":{"x":[{"value":4352059088,"symbolLocation":8744,"symbol":"typerel.*"},{"value":8752},{"value":8752},{"value":4352050336,"symbolLocation":0,"symbol":"runtime.types"},{"value":4354416168},{"value":4353086872,"symbolLocation":1036528,"symbol":"typerel.*"},{"value":4353087072,"symbolLocation":1036728,"symbol":"typerel.*"},{"value":4353086888,"symbolLocation":1036544,"symbol":"typerel.*"},{"value":3},{"value":0},{"value":0},{"value":3},{"value":1374390894608},{"value":3263713395},{"value":37858},{"value":7},{"value":1374390829984},{"value":1374390836384},{"value":0},{"value":1374390836488},{"value":125},{"value":3},{"value":4360589644,"symbolLocation":4,"symbol":"timebase"},{"value":6126694288},{"value":6126694352},{"value":6513804731},{"value":1374390836400},{"value":16},{"value":1374389543328}],"flavor":"ARM_THREAD_STATE64","lr":{"value":4340257956},"cpsr":{"value":536875008},"fp":{"value":1374390836248},"sp":{"value":1374390836256},"esr":{"value":2181038087,"description":"(Instruction Abort) Translation fault"},"pc":{"value":4340212512,"matchesCrashFrame":1},"far":{"value":4340212512}},"queue":"com.apple.main-thread","frames":[{"imageOffset":25376,"symbol":"*\/abi.Name.Name","symbolLocation":0,"imageIndex":0},{"imageOffset":70820,"symbol":"runtime.(*itab).init","symbolLocation":388,"imageIndex":0}]},{"id":30275,"frames":[{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":0},{"value":18446744073709551615},{"value":1},{"value":1},{"value":0},{"value":20000},{"value":10000000},{"value":3},{"value":60},{"value":7961593232,"symbolLocation":0,"symbol":"errno"},{"value":17},{"value":2150627323},{"value":2043},{"value":6135098880},{"value":2152726523},{"value":2043},{"value":6516675220,"symbolLocation":0,"symbol":"__error"},{"value":8112471192},{"value":0},{"value":6135098968},{"value":125},{"value":3},{"value":0},{"value":0},{"value":0},{"value":0},{"value":4354084080,"symbolLocation":1384,"symbol":"go:funcrel.*"},{"value":1},{"value":1374389545408}],"flavor":"ARM_THREAD_STATE64","lr":{"value":4340666120},"cpsr":{"value":1610616832},"fp":{"value":7358459582998995428},"sp":{"value":6135098880},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":4340666120},"far":{"value":0}}},{"id":30276,"frames":[{"imageOffset":20652,"symbol":"__psynch_cvwait","symbolLocation":8,"imageIndex":3},{"imageOffset":30204,"symbol":"_pthread_cond_wait","symbolLocation":1228,"imageIndex":4},{"imageOffset":484712,"symbol":"runtime.pthread_cond_wait_trampoline.abi0","symbolLocation":24,"imageIndex":0},{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":260},{"value":0},{"value":512},{"value":0},{"value":0},{"value":160},{"value":0},{"value":0},{"value":6143519624},{"value":0},{"value":1374389980256},{"value":2},{"value":0},{"value":0},{"value":0},{"value":0},{"value":305},{"value":8112471384},{"value":0},{"value":1374389980232},{"value":1374389980296},{"value":6143520992},{"value":0},{"value":0},{"value":512},{"value":513},{"value":768},{"value":18446744073709551520},{"value":1374389546656}],"flavor":"ARM_THREAD_STATE64","lr":{"value":6516938236},"cpsr":{"value":1610616832},"fp":{"value":6143519744},"sp":{"value":6143519600},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6516687020},"far":{"value":0}}},{"id":30277,"frames":[{"imageOffset":20652,"symbol":"__psynch_cvwait","symbolLocation":8,"imageIndex":3},{"imageOffset":30204,"symbol":"_pthread_cond_wait","symbolLocation":1228,"imageIndex":4},{"imageOffset":484712,"symbol":"runtime.pthread_cond_wait_trampoline.abi0","symbolLocation":24,"imageIndex":0},{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":260},{"value":0},{"value":256},{"value":0},{"value":0},{"value":160},{"value":0},{"value":0},{"value":6151957384},{"value":0},{"value":1374390584160},{"value":2},{"value":0},{"value":0},{"value":0},{"value":0},{"value":305},{"value":8112471384},{"value":0},{"value":1374390584136},{"value":1374390584200},{"value":6151958752},{"value":0},{"value":0},{"value":256},{"value":257},{"value":512},{"value":18446744073709551520},{"value":1374390591904}],"flavor":"ARM_THREAD_STATE64","lr":{"value":6516938236},"cpsr":{"value":1610616832},"fp":{"value":6151957504},"sp":{"value":6151957360},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6516687020},"far":{"value":0}}},{"id":30278,"frames":[{"imageOffset":20652,"symbol":"__psynch_cvwait","symbolLocation":8,"imageIndex":3},{"imageOffset":30204,"symbol":"_pthread_cond_wait","symbolLocation":1228,"imageIndex":4},{"imageOffset":484712,"symbol":"runtime.pthread_cond_wait_trampoline.abi0","symbolLocation":24,"imageIndex":0},{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":260},{"value":0},{"value":256},{"value":0},{"value":0},{"value":160},{"value":0},{"value":0},{"value":6160411736},{"value":0},{"value":1374391108448},{"value":2},{"value":0},{"value":0},{"value":0},{"value":0},{"value":305},{"value":8112471384},{"value":0},{"value":1374391108424},{"value":1374391108488},{"value":6160412896},{"value":0},{"value":0},{"value":256},{"value":257},{"value":512},{"value":18446744073709551520},{"value":1374391116192}],"flavor":"ARM_THREAD_STATE64","lr":{"value":6516938236},"cpsr":{"value":1610616832},"fp":{"value":6160411856},"sp":{"value":6160411712},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6516687020},"far":{"value":0}}},{"id":30279,"frames":[{"imageOffset":20652,"symbol":"__psynch_cvwait","symbolLocation":8,"imageIndex":3},{"imageOffset":30204,"symbol":"_pthread_cond_wait","symbolLocation":1228,"imageIndex":4},{"imageOffset":484712,"symbol":"runtime.pthread_cond_wait_trampoline.abi0","symbolLocation":24,"imageIndex":0},{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":260},{"value":0},{"value":0},{"value":0},{"value":0},{"value":160},{"value":0},{"value":0},{"value":6168833288},{"value":0},{"value":1374390585312},{"value":2},{"value":0},{"value":0},{"value":0},{"value":0},{"value":305},{"value":8112471384},{"value":0},{"value":1374390585288},{"value":1374390585352},{"value":6168834272},{"value":0},{"value":0},{"value":0},{"value":1},{"value":256},{"value":18446744073709551584},{"value":1374390592736}],"flavor":"ARM_THREAD_STATE64","lr":{"value":6516938236},"cpsr":{"value":1610616832},"fp":{"value":6168833408},"sp":{"value":6168833264},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6516687020},"far":{"value":0}}}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "arm64",
    "base" : 4340187136,
    "size" : 11862016,
    "uuid" : "3513ca24-39ed-3d73-96b9-9b5acf281169",
    "path" : "\/Users\/USER\/*\/gopls",
    "name" : "gopls"
  },
  {
    "size" : 0,
    "source" : "A",
    "base" : 0,
    "uuid" : "00000000-0000-0000-0000-000000000000"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 6513258496,
    "size" : 607000,
    "uuid" : "ec7a3ba0-f9bf-3ab8-a0f4-8622e5606b20",
    "path" : "\/usr\/lib\/dyld",
    "name" : "dyld"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 6516666368,
    "size" : 241648,
    "uuid" : "b7751381-1442-30b5-91b9-ad7be461bebe",
    "path" : "\/usr\/lib\/system\/libsystem_kernel.dylib",
    "name" : "libsystem_kernel.dylib"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 6516908032,
    "size" : 53236,
    "uuid" : "daf95373-5de6-39a1-a6ce-d87f3f0629cc",
    "path" : "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "name" : "libsystem_pthread.dylib"
  }
],
  "sharedCache" : {
  "base" : 6512508928,
  "size" : 4018896896,
  "uuid" : "80dd42b3-8a52-3caf-9848-54a1c2732864"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=1.0G resident=0K(0%) swapped_out_or_unallocated=1.0G(100%)\nWritable regions: Total=610.0M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=610.0M(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nKernel Alloc Once                   32K        1 \nMALLOC                           522.2M       12 \nMALLOC guard page                   96K        6 \nSTACK GUARD                         80K        5 \nStack                             48.1M        6 \nStack Guard                       56.0M        1 \nVM_ALLOCATE                        1.2G       36 \n__AUTH                             339K       65 \n__AUTH_CONST                      4053K      147 \n__DATA                            2466K      142 \n__DATA_CONST                      11.5M      150 \n__DATA_DIRTY                       363K       58 \n__LINKEDIT                       888.2M        2 \n__OBJC_RO                         70.8M        1 \n__OBJC_RW                         2156K        1 \n__TEXT                           131.1M      156 \ndyld private memory                272K        2 \nshared memory                       16K        1 \n===========                     =======  ======= \nTOTAL                              2.9G      792 \n",
  "legacyInfo" : {
  "threadTriggered" : {
    "queue" : "com.apple.main-thread"
  }
},
  "logWritingSignature" : "8dbd5b1ee96ca3011838b0adc040bf1d106b1d9d",
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "6297d96be2c9387df974efa4",
      "factorPackIds" : {

      },
      "deploymentId" : 240000014
    },
    {
      "rolloutId" : "6112d17137f5d11121dcd4e2",
      "factorPackIds" : {

      },
      "deploymentId" : 240000442
    }
  ],
  "experiments" : [
    {
      "treatmentId" : "a092db1b-c401-44fa-9c54-518b7d69ca61",
      "experimentId" : "64a844035c85000c0f42398a",
      "deploymentId" : 400000019
    }
  ]
}
}

gopls version

Not possible to check, using gopls -v version:

$ gopls -v version
[1]    5720 killed     gopls -v version

But I installed latest gopls version, using VSCode update functionality

go env

What did you do?

What did you expect to see?

What did you see instead?

Editor and settings

Logs

@koliyo koliyo added gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository. labels Nov 8, 2023
@gopherbot gopherbot added this to the Unreleased milestone Nov 8, 2023
@koliyo
Copy link
Author

koliyo commented Nov 8, 2023

I was able to work around this issue by resigning the binary:

codesign -f -s - /Users/nils/Work/go/bin/gopls

And now it works:

gopls -v version
Build info
----------
golang.org/x/tools/gopls v0.14.1
    golang.org/x/tools/gopls@v0.14.1 h1:XaTETpi7Q67XO8nftquJitcx+9c2bPclO8Kz2sBVvec=
    github.com/BurntSushi/toml@v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=
    github.com/google/go-cmp@v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/exp/typeparams@v0.0.0-20221212164502-fae10dda9338 h1:2O2DON6y3XMJiQRAS1UWU+54aec2uopH3x7MAiqGW6Y=
    golang.org/x/mod@v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY=
    golang.org/x/sync@v0.4.0 h1:zxkM55ReGkDlKSM+Fu41A+zmbZuaPVbGMzvvdUPznYQ=
    golang.org/x/sys@v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
    golang.org/x/telemetry@v0.0.0-20231011160506-788d5629a052 h1:1baVNneD/IRxmu8JQdBuki78zUqBtZxq8smZXQj0X2Y=
    golang.org/x/text@v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k=
    golang.org/x/tools@v0.14.1-0.20231026192422-8b5abd452b28 h1:5YgdZAe2w0x3Xrjv0+GXrI0jvm7qCQK/ySGFfiEHMfU=
    golang.org/x/vuln@v1.0.1 h1:KUas02EjQK5LTuIx1OylBQdKKZ9jeugs+HiqO5HormU=
    honnef.co/go/tools@v0.4.5 h1:YGD4H+SuIOOqsyoLOpZDWcieM28W47/zRO7f+9V3nvo=
    mvdan.cc/gofumpt@v0.4.0 h1:JVf4NN1mIpHogBj7ABpgOyZc65/UUOkKQFkoURsz4MM=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.21.3

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 8, 2023
@findleyr
Copy link
Contributor

findleyr commented Nov 8, 2023

I'm unfamiliar with code signing, but found #42684, which also relates to arm64. I don't see how this problem could be specific to gopls: are you able to install and run other go programs using go install? Are you able to run local binaries using go run?

(asking to confirm before I CC compiler folks)

@koliyo
Copy link
Author

koliyo commented Nov 9, 2023

I have not had any problems with go run, and I tested go install on a local project, and can launch that artifact without problem from the target bin path.

Not sure why I see this problem, I have another dev machine where I have not seen the same issue. Also, not sure I can reproduce the original problem now that I fixed it using codesign explicitly

@koliyo
Copy link
Author

koliyo commented Nov 9, 2023

Actually, I can reproduce it!

If I delete /Users/nils/Work/go/bin/gopls and then install gopls using Go: Install/Update tools in VSCode I get the original error.

$ ~/Work/go/bin/gopls 
[1]    98838 killed     ~/Work/go/bin/gopls

@findleyr
Copy link
Contributor

findleyr commented Nov 9, 2023

Thanks for following up. Can you please try installing gopls@v0.13.2 (an older version) to see if the outcome is different?

go install golang.org/x/tools/gopls@v0.13.2

Then can you try installing another arbitrary binary from x/tools, to see if it works? Say:

go install golang.org/x/tools/cmd/gonew@latest

Can you do gonew --help?

@koliyo
Copy link
Author

koliyo commented Nov 10, 2023

gopls@v0.13.2
-> Same error as before

gonew@latest
-> Works

gonew --help
usage: gonew srcmod[@version] [dstmod [dir]]
See https://pkg.go.dev/golang.org/x/tools/cmd/gonew.

@findleyr
Copy link
Contributor

Thanks @koliyo for confirming.

CC @golang/compiler -- does anyone have an idea why the gopls binary would require signing? It seems like this just started happening in the past couple weeks, and may be limited to arm64 based on the reports I've seen.

@cherrymui
Copy link
Member

macOS on ARM64 requires all binaries to be signed, gopls is not an exception. The go toolchain should already produce a signed binary. What is the output of codesign -vv <binary>? It should say it is valid if it is signed properly.

How is the binary installed? Do you by any chance overwrite an existing binary (like using the cp command to copy to an existing file)? The macOS kernel caches the code signature. If you overwrite an existing binary, the kernel still keeps the old signature cached, and when the binary is launched it sees a mismatch and kills the program. Deleting the old binary and creating a new one would fix. Rebooting the machine would also clear the cache. I think the go command always arranges it to create a new binary, and the go linker also calls mmap to invalidate the kernel cache. But if you copy the file to an existing file after build, that would not help.

Do you just run the go install command and run the binary right after? Or there is some other step? Thanks.

@koliyo
Copy link
Author

koliyo commented Nov 13, 2023

codesign -vv ~/Work/go/bin/gopls
/Users/nils/Work/go/bin/gopls: invalid signature (code or signature have been modified)
In architecture: arm64

This is a "fresh" binary, installed with go install golang.org/x/tools/gopls@latest

I do not do any other operation than the install command. Normally I install gotools using VSCode.

It is possible that I have used gopls from homebrew at some point in the past, not sure, but that was not recently, and should be another install directory

@findleyr
Copy link
Contributor

Hmm. I'm a bit out of my depth here, but here are a couple ideas to try based on @cherrymui's comment:

  • Can you try deleting the gopls binary before installing? (you may have already tried this).
  • Can you try installing into a different directory?
TEMP_GOBIN=$(mktemp -d)
GOBIN=$TEMP_GOBIN go install golang.org/x/tools/gopls@latest
$TEMP_GOBIN/gopls version

@cherrymui
Copy link
Member

/Users/nils/Work/go/bin/gopls: invalid signature (code or signature have been modified)

Interesting. If a fresh go install'd binary has invalid signature, that looks bad. The experiment @findleyr mentioned sounds like a good idea to try. Also, maybe try go clean -cache before building in case there is some corruption in the cache. What version of the Go toolchain are you using? Could you share go env output (in case there is something weird in CC or C compiler flags)? Thanks.

@suzmue suzmue added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 13, 2023
@kidandcat
Copy link

go clean -cache

I was getting the same issue, after cleaning the cache and doing a fresh install, it worked right. (Sorry, forgot to get go env before cleaning the cache)

PD: A complete computer restart did not fix the issue neither, and it happened with previous gopls versions too (I tested down to 0.13.0)

@findleyr
Copy link
Contributor

CC @bcmills, in case this is related to the build cache.

It sounds like gopls is not directly involved here: its contribution is being a common binary that gets (re)installed frequently.

@bcmills
Copy link
Contributor

bcmills commented Nov 17, 2023

My understanding is that code-signing happens either at link time or afterward.

So I'm not sure why clearing the build cache would do anything, beyond perhaps causing the program to be rebuilt again. Although, if you run go install on a binary target that is not actually stale cmd/go just updates its mtime instead of overwriting it:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/work/exec.go;l=1772-1809;drc=c75a617af02fcfa53effed30803f2a96388680d5
So that might explain why go install didn't fix the signing error on its own, but it doesn't give us much of a clue as to how the system got into that error state to begin with. 🤔

@koliyo
Copy link
Author

koliyo commented Nov 20, 2023

Tmp bin directory, same error:

TEMP_GOBIN=$(mktemp -d)
GOBIN=$TEMP_GOBIN go install golang.org/x/tools/gopls@latest
$TEMP_GOBIN/gopls version
[1]    34632 killed     $TEMP_GOBIN/gopls version

Clean cache does not seem to work for me:

go clean -cache
rm ~/Work/go/bin/gopls
go install golang.org/x/tools/gopls@latest
~/Work/go/bin/gopls
[1]    34445 killed     ~/Work/go/bin/gopls

Here is my env:

go env
GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/nils/Library/Caches/go-build'
GOENV='/Users/nils/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/nils/Work/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/nils/Work/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.4/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.4/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/7w/r3zd6mzj6jngg60zxf7q383h0000gn/T/go-build3163292121=/tmp/go-build -gno-record-gcc-switches -fno-common'

@findleyr findleyr removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 20, 2023
@adonovan adonovan changed the title x/tools/gopls: Can not run on MacOS due to code signing error cmd/go: install produces executable with invalid signature (macOS) Nov 28, 2023
@findleyr findleyr removed the gopls Issues related to the Go language server, gopls. label Nov 29, 2023
@findleyr
Copy link
Contributor

findleyr commented Nov 29, 2023

I don't think this is directly related to gopls, but we have seen an increase in these reports via the VS Code Go issue tracker. I do wonder what changed. @cherrymui @bcmills any idea how to proceed?

@bcmills
Copy link
Contributor

bcmills commented Nov 29, 2023

I don't know much about how macOS code-signing works, unfortunately. It would really help if we had some way to reproduce it.


For similar issues in the past I have requested that users tar up the contents of $(go env GOCACHE) and send it privately, along with the source code for their main module, the output of go env, and the output of the POSIX env command. It's pretty heavy-handed, but I was able to reproduce a failure due to a corrupted cache file that way. 😅

Unfortunately, in this case I suspect that codesign may produce a signature specific to the user's machine, so even that might not be enough to get any insight into what's going on. 😞

@cherrymui
Copy link
Member

I tried to go install gopls on my macOS ARM64 machine, but I couldn't reproduce. It generates an executable with valid signature and it runs fine.

What version of macOS do you use? And C toolchain (Xcode) version?
If you install gopls with cgo disabled, i.e. CGO_ENABLED=0 go install, does it still reproduce?
Thanks.

@koliyo
Copy link
Author

koliyo commented Dec 8, 2023

This does work!

CGO_ENABLED=0 go install

So it seems like there is some kind of system level caching of signatures that is messed up? Specifically on my machine.

XCode 15.0.1
macOS 14.0 (23A344)

@koliyo
Copy link
Author

koliyo commented Dec 8, 2023

Ok, I have found some additional hints!

I have been using zsh for a long time, and also when this bug was reported. But since then I have changed to fish.

The fix with CGO_ENABLED=0 is true for zsh, without it I still get the killed process.

But, what I discovered is that I can't even compile go code (with c includes) on my relatively unmodified fish environment:

❯ go install golang.org/x/tools/gopls@latest
_cgo_export.c:3:10: fatal error: 'stdlib.h' file not found

I have looked at my environment variables, but was unable to deduce the issue. I asked ChatGPT and was suggested the following:

set -gx SDKROOT (xcrun --sdk macosx --show-sdk-path)

And after this updated env I can compile gopls in fish shell. And what is even more interesting, it does produce a working executable, even without CGO_ENABLED=0!

I do not have SDKROOT defined in zsh.

Anyway there seem to be something going on with include/library paths at some level.

Ok update. I have ccache installed and have added to the front of PATH to activate ccache:

❯ which clang
/opt/homebrew/opt/ccache/libexec/clang

True for both zsh and fish

But ccache is only an indirection to the actual clang, via symlink, and I discovered that fish was using xcode clang while zsh was using homebrew clang

fish:

❯ clang --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

zsh:

$ clang --version
Homebrew clang version 17.0.6
Target: arm64-apple-darwin23.0.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin

Then I added opt/homebrew/opt/llvm to the fish PATH, and now both ccache and clang are the same executable in fish and zsh. But there are still differences. If I again remove SDKROOT from fish shell, I now get this new error:

❯ go install golang.org/x/tools/gopls@latest
# golang.org/x/tools/gopls
/opt/homebrew/Cellar/go/1.21.5/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
ld: library 'resolv' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)

And as before, SDKROOT was never defined in zsh.

And to reiterate, if I in fish define SDKROOT, I can now compile gopls which does verify and run properly, without the cgo definition.

Hope this give you guys some clues.

@koliyo
Copy link
Author

koliyo commented Dec 8, 2023

Discovered another difference between fish and zsh on my machine. Zsh had CC and CXX defined, which had an impact on go env

In fish:

go env
...
CC='cc'
CXX='c++'

in zsh:

CC='clang'
CXX='clang++'

if I export CC/CXX in fish I get the exact same results as zsh. I do not need to define SDKROOT, but the gopls binary is killed by invalid signature.

@cherrymui
Copy link
Member

Thanks for the investigation. I could imagine ccache causes problems.

I think generally Apple's C toolchain works better than a custom installed one. Could you try setting CC=/usr/bin/clang?

❯ go install golang.org/x/tools/gopls@latest
_cgo_export.c:3:10: fatal error: 'stdlib.h' file not found

This looks like the environment being mis-configured. Could you build any C program that #include's stdlib.h?

@koliyo
Copy link
Author

koliyo commented Dec 8, 2023

Yeah, this is interesting, simple hello world c program, fails with xcode clang, but not with /usr/bin/clang

❯ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang foo.c
foo.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
❯ /usr/bin/clang foo.c
# No error

@koliyo
Copy link
Author

koliyo commented Dec 8, 2023

They are the same version, but yet different:

❯ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
❯ /usr/bin/clang --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
❯ diff /usr/bin/clang /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
Binary files /usr/bin/clang and /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang differ

@cherrymui
Copy link
Member

What is the version of /usr/bin/clang? Usually it points to the Xcode clang. Maybe your Xcode installation is incomplete or corrupted? Could you try installing Xcode again?

@koliyo
Copy link
Author

koliyo commented Dec 8, 2023

/usr/bin/clang version was included above, here it is again

❯ /usr/bin/clang --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

It is not a symlink either

❯ ll /usr/bin/clang
-rwxr-xr-x  76 root  wheel   164K Nov 18 19:13 /usr/bin/clang*

Also, I have now removed ccache from path, with same result, so does not seem to be ccache dependent:

❯ which clang
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

@koliyo
Copy link
Author

koliyo commented Dec 8, 2023

NOTE: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin is currently ahead of /usr/local/bin in my $PATH

@cherrymui
Copy link
Member

I think /usr/bin/clang is usually not a symlink but a compiled wrapper program that invokes clang using xcrun. I'd suggest you try to install an unbroken Xcode, and set CC to its clang and see if you can still reproduce the Go issue. Thanks.

@koliyo
Copy link
Author

koliyo commented Dec 8, 2023

The problem seems to be the order of my shell paths (which are propagated into VSCode). As long as i keep /usr/bin before /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin the original issue reported is gone.

I will try and keep /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin out of path all together. It was because I had some issues with compiling swift+c FFI code at some point, which worked inside xcode, but not from command line with swift build, then I used xcodebuild -find clang and added this to PATH. Anyway, I think I have a better setup now, and do not have this issue anymore.

@cherrymui
Copy link
Member

Thanks. Sounds like we can close this issue. Feel free to reopen if there is anything that still needs to be addressed from the Go side. Thanks.

@cherrymui cherrymui closed this as not planned Won't fix, can't repro, duplicate, stale Dec 8, 2023
@findleyr
Copy link
Contributor

Thanks so much @koliyo for following up on this issue. We've seen similar reports from several users, and without your diligence it may have taken us a very long time to figure this out on our own.

Thank you @cherrymui for the sage advice and help investigating!

@ysmolski
Copy link
Member

In my case singing failed for gopls because of including custom llvm toolchain into PATH's:

export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"

I have remove those and problem was gone. Kudos to @koliyo !

@jinmel
Copy link

jinmel commented Feb 15, 2024

Ok, I have found some additional hints!

I have been using zsh for a long time, and also when this bug was reported. But since then I have changed to fish.

The fix with CGO_ENABLED=0 is true for zsh, without it I still get the killed process.

But, what I discovered is that I can't even compile go code (with c includes) on my relatively unmodified fish environment:

❯ go install golang.org/x/tools/gopls@latest
_cgo_export.c:3:10: fatal error: 'stdlib.h' file not found

I have looked at my environment variables, but was unable to deduce the issue. I asked ChatGPT and was suggested the following:

set -gx SDKROOT (xcrun --sdk macosx --show-sdk-path)

And after this updated env I can compile gopls in fish shell. And what is even more interesting, it does produce a working executable, even without CGO_ENABLED=0!

I do not have SDKROOT defined in zsh.

Anyway there seem to be something going on with include/library paths at some level.

Ok update. I have ccache installed and have added to the front of PATH to activate ccache:

❯ which clang
/opt/homebrew/opt/ccache/libexec/clang

True for both zsh and fish

But ccache is only an indirection to the actual clang, via symlink, and I discovered that fish was using xcode clang while zsh was using homebrew clang

fish:

❯ clang --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

zsh:

$ clang --version
Homebrew clang version 17.0.6
Target: arm64-apple-darwin23.0.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin

Then I added opt/homebrew/opt/llvm to the fish PATH, and now both ccache and clang are the same executable in fish and zsh. But there are still differences. If I again remove SDKROOT from fish shell, I now get this new error:

❯ go install golang.org/x/tools/gopls@latest
# golang.org/x/tools/gopls
/opt/homebrew/Cellar/go/1.21.5/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
ld: library 'resolv' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)

And as before, SDKROOT was never defined in zsh.

And to reiterate, if I in fish define SDKROOT, I can now compile gopls which does verify and run properly, without the cgo definition.

Hope this give you guys some clues.

My reason was this one. Be careful about what compiler toolchain you are using.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

9 participants