CL 111275 you linked to only disables DWARF combining for buildmode=exe, which is what gomobile's build_iosapp.go is generating. For c-archive, Xcode is supposed to split the DWARF from the archive during its build.
It could be that our __asm marker is incompatible with the DWARF we generate. Can you submit a binary without -w if bitcode is disabled? Perhaps you need to disable the __asm marker as well.

It could be that our __asm marker is incompatible with the DWARF we generate. Can you submit a binary without -w if bitcode is disabled? Perhaps you need to disable the __asm marker as well.

The issue occurs when I try to "recompile from bitcode" in Xcode's Organizer with an archived ipa.

If I disable __asm then I have to disable bitcode, in which case I can't attempt to "recompile from bitcode" anymore.

Yes, but your initial problem was the error email from Apple. I'm trying to work why -w is needed. If it is needed because of the __asm section and bitcode then we'll just do -w by default.

If not, then I'd like to figure out what else could cause the problem. Including DWARF in object files and archives shouldn't be a problem. The special Apple dwarf splitting happens for executables.

Since the apple email mentions app thinning, I'm pretty sure a bitcode=off build won't trigger it. But I will try uploading a build with bitcode off but __asm included to see what happens.

It would be nice to be able to emit debug information for the golang code. I'm guessing the dwarf format it expects for asm is slightly different? I will poke at the generated sections and see if I notice anything different between real assembled objects vs go.o

Stack traces on mobile is #22716.

I disabled bitcode on my project, removed the -w and tried to upload a build. It fails processing in Xcode and never got to the upload. The logs show a similar error:

2019-03-25 19:45:41 +0000  Running /Applications/Xcode.app/Contents/Developer/usr/bin/symbols '-noTextInSOD' '-noDaemon' '-arch' 'all' '-symbolsPackageDir' '/var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/XcodeDistPipeline.j3X/Symbols' '/var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/XcodeDistPipeline.j3X/MyApp.app.dSYM/Contents/Resources/Dwarf/MyApp'
2019-03-25 19:45:41 +0000  error: Could not load symbol information for B60F1C62-D42E-3E17-90DF-9F6ADE25D2CD.  Symbols file will contain no data.
2019-03-25 19:45:41 +0000  /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/XcodeDistPipeline.j3X/MyApp.app.dSYM/Contents/Resources/Dwarf/MyApp [arm64, 0.018700 seconds]:
2019-03-25 19:45:41 +0000  /Applications/Xcode.app/Contents/Developer/usr/bin/symbols exited with 1

It seems something about the combination of __asm and DWARF in the go.o object causes the entire app object to end up with broken debug symbols.

Previously I was using fastlane to upload my builds and not Xcode, so just to make sure I re-enabled bitcode (still without -w) and tried to upload from Xcode this time. Interestingly that combination gets further and uploads to App Store, but then processing fails like it did before.

The underlying problem seems the same, it just depends on where the symbol extraction is happening. If you force it locally with bitcode recompile or non-bitcode build you can see the error right away, but if you upload a bitcode build it happens on Apple's servers and errors out the same way there.

Unfortunately it seems like now that we're emitting __asm unconditionally, it's not possible to submit either bitcode or non-bitcode builds as long as DWARF generation for go.o is enabled.

For reference, here's what size shows for go.o:

libtest_ios_arm64.a(go.o):
Segment : 0x0 (vmaddr 0x0 fileoff 4096)
        Section (__TEXT, __text): 0x210478 (addr 0x0 offset 4096)
        Section (__DATA, __rodata): 0x14e3c9 (addr 0x210480 offset 2167936)
        Section (__DATA, __typelink): 0x2320 (addr 0x35e860 offset 3536992)
        Section (__DATA, __itablink): 0x838 (addr 0x360b80 offset 3545984)
        Section (__DATA, __gosymtab): 0x0 (addr 0x3613b8 offset 3548088)
        Section (__DATA, __gopclntab): 0x1a69e0 (addr 0x3613c0 offset 3548096)
        Section (__LLVM, __asm): 0x1 (addr 0x508000 offset 5279744)
        Section (__DATA, __noptrdata): 0x2a981 (addr 0x508020 offset 5279776)
        Section (__DATA, __mod_init_func): 0x8 (addr 0x5329a8 offset 5454248)
        Section (__DATA, __data): 0x9c50 (addr 0x5329c0 offset 5454272)
        Section (__DATA, __bss): 0x1e150 (addr 0x53c620 offset 0)
        Section (__DATA, __noptrbss): 0x3058 (addr 0x55a780 offset 0)
        Section (__DWARF, __debug_abbrev): 0x1e0 (addr 0x55e000 offset 5496832)
        Section (__DWARF, __debug_line): 0x9a93d (addr 0x55e1e0 offset 5497312)
        Section (__DWARF, __debug_frame): 0x501ec (addr 0x5f8b1d offset 6130461)
        Section (__DWARF, __debug_pubnames): 0xd283 (addr 0x648d09 offset 6458633)
        Section (__DWARF, __debug_pubtypes): 0x26f05 (addr 0x655f8c offset 6512524)
        Section (__DWARF, __debug_gdb_scri): 0x4c (addr 0x67ce91 offset 6672017)
        Section (__DWARF, __debug_info): 0x165dc5 (addr 0x67cedd offset 6672093)
        Section (__DWARF, __debug_loc): 0x1f6af7 (addr 0x7e2ca2 offset 8137890)
        Section (__DWARF, __debug_ranges): 0xb03b0 (addr 0x9d9799 offset 10196889)
        total 0xa89044
total 0x0

vs for 000011.o which is generated via clang's assembler:

libtest_ios_arm64.a(000011.o):
Segment : 0x1b3 (vmaddr 0x0 fileoff 704)
        Section (__TEXT, __text): 0x4c (addr 0x0 offset 704)
        Section (__LLVM, __asm): 0x1 (addr 0x4c offset 780)
        Section (__DWARF, __debug_info): 0xbd (addr 0x4d offset 781)
        Section (__DWARF, __debug_abbrev): 0x28 (addr 0x10a offset 970)
        Section (__DWARF, __debug_aranges): 0x30 (addr 0x132 offset 1010)
        Section (__DWARF, __debug_line): 0x51 (addr 0x162 offset 1058)
        total 0x1b3
total 0x1b3

It's a bit strange that go.o says total 0x0 at the end, although that's consistent even if I build go.o with -ldflags=-w:

libtest_ios_arm64.a(go.o):
Segment : 0x0 (vmaddr 0x0 fileoff 4096)
        Section (__TEXT, __text): 0x210478 (addr 0x0 offset 4096)
        Section (__DATA, __rodata): 0x14e3c9 (addr 0x210480 offset 2167936)
        Section (__DATA, __typelink): 0x2320 (addr 0x35e860 offset 3536992)
        Section (__DATA, __itablink): 0x838 (addr 0x360b80 offset 3545984)
        Section (__DATA, __gosymtab): 0x0 (addr 0x3613b8 offset 3548088)
        Section (__DATA, __gopclntab): 0x1a69e0 (addr 0x3613c0 offset 3548096)
        Section (__LLVM, __asm): 0x1 (addr 0x508000 offset 5279744)
        Section (__DATA, __noptrdata): 0x2a981 (addr 0x508020 offset 5279776)
        Section (__DATA, __mod_init_func): 0x8 (addr 0x5329a8 offset 5454248)
        Section (__DATA, __data): 0x9c50 (addr 0x5329c0 offset 5454272)
        Section (__DATA, __bss): 0x1e150 (addr 0x53c620 offset 0)
        Section (__DATA, __noptrbss): 0x3058 (addr 0x55a780 offset 0)
        total 0x55d4fb
total 0x0

Can you try removing the __asm generating code from Go just to make sure that's the reason -w doesn't work?

Can you try removing the __asm generating code from Go just to make sure that's the reason -w doesn't work?

Good idea! I removed just the .llvmasm change, disabled bitcode on my project and attempted to upload. It failed the same way:

with the logs showing:

2019-03-25 21:03:18 +0000  error: Could not load symbol information for 5DB22712-C769-3962-A5C1-E5995FE4F8A2.  Symbols file will contain no data.
2019-03-25 21:03:18 +0000  /Applications/Xcode.app/Contents/Developer/usr/bin/symbols exited with 1

So clearly this is not related to the __asm bit and something else is going on. I will try go1.12 next to see if it has the same behavior.

So clearly this is not related to the __asm bit and something else is going on. I will try go1.12 next to see if it has the same behavior.

Same thing, I can't get it working at all on go1.12.1 unless I add -ldflags=-w.

I also tried go1.11.6 and it has the same behavior.

Since this is my first time trying to use golang on iOS/tvOS I'm not sure if this ever worked in the past.

So here's a simple test that shows that the go.o generated by golang is not able to be parsed by /usr/bin/symbols:

$ go version
go version go1.11.6 darwin/amd64

$ cat test.go
package main

import "C"
import "log"

func main() {
    log.Printf("foo")
}

$ CGO_CFLAGS=-g go build -buildmode=c-archive -o test.a test.go
$ ar xv test.a
x - __.SYMDEF SORTED
x - go.o
x - 000000.o
x - 000001.o
x - 000002.o
x - 000003.o
x - 000004.o
x - 000005.o
x - 000006.o
x - 000007.o
x - 000008.o
x - 000009.o
x - 000010.o

$ size -x -l -m 000000.o
Segment : 0x98 (vmaddr 0x0 fileoff 528)
	Section (__TEXT, __text): 0x0 (addr 0x0 offset 528)
	Section (__DWARF, __apple_names): 0x24 (addr 0x0 offset 528)
	Section (__DWARF, __apple_objc): 0x24 (addr 0x24 offset 564)
	Section (__DWARF, __apple_namespac): 0x24 (addr 0x48 offset 600)
	Section (__DWARF, __apple_types): 0x2c (addr 0x6c offset 636)
	total 0x98
total 0x98

$ symbols 000000.o
000000.o [x86_64, 0.000155 seconds]:
    null-uuid                            /Users/tmm1/fancybits/bitcode/000000.o [OBJECT, FaultedFromDisk]
        0x0000000000000000 (    0x98)  SEGMENT
            0x0000000000000000 (    0x24) __DWARF __apple_names
            0x0000000000000024 (    0x24) __DWARF __apple_objc
            0x0000000000000048 (    0x24) __DWARF __apple_namespac
            0x000000000000006c (    0x2c) __DWARF __apple_types

$ size -x -l -m go.o
Segment : 0x0 (vmaddr 0x0 fileoff 4096)
	Section (__TEXT, __text): 0x9380c (addr 0x0 offset 4096)
	Section (__DATA, __rodata): 0x47809 (addr 0x93820 offset 608288)
	Section (__DATA, __typelink): 0xbec (addr 0xdb040 offset 901184)
	Section (__DATA, __itablink): 0x60 (addr 0xdbc30 offset 904240)
	Section (__DATA, __gosymtab): 0x0 (addr 0xdbc90 offset 904336)
	Section (__DATA, __gopclntab): 0x77f25 (addr 0xdbca0 offset 904352)
	Section (__DATA, __noptrdata): 0xcc5c (addr 0x154000 offset 1396736)
	Section (__DATA, __mod_init_func): 0x8 (addr 0x160c60 offset 1449056)
	Section (__DATA, __data): 0x6910 (addr 0x160c80 offset 1449088)
	Section (__DATA, __bss): 0x1c790 (addr 0x1675a0 offset 0)
	Section (__DATA, __noptrbss): 0x2598 (addr 0x183d40 offset 0)
	Section (__DWARF, __debug_abbrev): 0x1d3 (addr 0x187000 offset 1478656)
	Section (__DWARF, __debug_line): 0x25975 (addr 0x1871d3 offset 1479123)
	Section (__DWARF, __debug_frame): 0x14224 (addr 0x1acb48 offset 1633096)
	Section (__DWARF, __debug_pubnames): 0x812f (addr 0x1c0d6c offset 1715564)
	Section (__DWARF, __debug_pubtypes): 0xae69 (addr 0x1c8e9b offset 1748635)
	Section (__DWARF, __debug_gdb_scri): 0x45 (addr 0x1d3d04 offset 1793284)
	Section (__DWARF, __debug_info): 0x767d1 (addr 0x1d3d49 offset 1793353)
	Section (__DWARF, __debug_loc): 0x6bb48 (addr 0x24a51a offset 2278682)
	Section (__DWARF, __debug_ranges): 0x26030 (addr 0x2b6062 offset 2719842)
	total 0x2daeb4
total 0x0

$ symbols go.o
Unable to find file, pid, task or signature matching: go.o
WARNING: You currently do not have task_for_pid() privileges.

$ symbols -v
symbols version:			@(#)PROGRAM:symbols  PROJECT:SamplingTools-64460.8
CoreSymbolicationDT.framework version:	64460.7

I got a tip on twitter that the newly release Xcode 10.2 might help: https://twitter.com/MarkVillacampa/status/1110309609464455169?s=20

Unfortunately there was no difference. The behavior of /usr/bin/symbols is the same, even though the version number increased:

$ symbols -v
symbols version:			@(#)PROGRAM:symbols  PROJECT:SamplingTools-64490.33
CoreSymbolicationDT.framework version:	64490.26

and when I try uploading from Xcode I get the same "Symbols tool failed".

I don't see you're using GOARCH so it seems to me this won't even work for macOS programs; I suppose submitting to the macOS App Store will probably also fail without -w?
Pleasing symbolscould get us a long way on #22716, but I'm not sure where to begin figuring out what is upsetting it; the error is useless.

I don't see you're using GOARCH so it seems to me this won't even work for macOS programs

Correct, the issue affects all mach-o objects linked by golang regardless of architecture or OS.

Pleasing symbols could get us a long way on #22716, but I'm not sure where to begin figuring out what is upsetting it; the error is useless.

What's interesting is that symbols chokes on the object created with -ldflags=-w as well. It seems Xcode just avoids calling symbols in the case where no DWARF data is present, which is why the -w workaround is effective.

I'm not sure how best to proceed from here, either. I can try opening a radar with apple to see if they have any insights.

It might be worth trying to disable SIP and running lldb -- symbols to see if its attempting to open the file and where it gives up. My hunch is that its something about the mach-o headers that's out of place since the issue occurs with and without DWARF data present.

and when I try uploading from Xcode I get the same "Symbols tool failed".

Same here :(，have any luck with that ?

@tmm1 what are the downsides to using -ldflags=-w?

@tmm1 what are the downsides to using -ldflags=-w?

$ go tool link 2>&1 | grep -- "-w"
  -w    disable DWARF generation

You lose debug information. However its unclear if the debug symbols were being used in the first place, so there may be no practical downside.

@eliasnaur I was able to make symbols go.o work with the following patch:

diff --git a/src/cmd/link/internal/ld/macho.go b/src/cmd/link/internal/ld/macho.go
index d13857081a..3cd118942d 100644
--- a/src/cmd/link/internal/ld/macho.go
+++ b/src/cmd/link/internal/ld/macho.go
@@ -560,7 +560,7 @@ func Asmbmacho(ctxt *Link) {
                ms = newMachoSeg("", 40)
 
                ms.fileoffset = Segtext.Fileoff
-               if ctxt.Arch.Family == sys.ARM || ctxt.BuildMode == BuildModeCArchive {
+               if ctxt.Arch.Family == sys.ARM /*|| ctxt.BuildMode == BuildModeCArchive*/ {
                        ms.filesize = Segdata.Fileoff + Segdata.Filelen - Segtext.Fileoff
                } else {
                        ms.filesize = Segdwarf.Fileoff + Segdwarf.Filelen - Segtext.Fileoff

AFAICT this is all legacy workaround code and the whole if block can probably be removed.

Great! Does it fix #28997 as well?

Can you submit your change as a CL? As a test you can add the symbols go.o to cmd/link/dwarf_test.go.

Hmm, so with my patch above, /usr/bin/symbols can now parse go.o with and without -w.

However what's strange.. if I build with -w (i.e. no DWARF data), then symbols does show a list of symbols:

go.o [arm64, 0.002350 seconds]:
    null-uuid                            go.o [OBJECT, FaultedFromDisk]  
        0x0000000000000000 (0x57d000)  SEGMENT
            0x0000000000000000 (0x21d308) __TEXT __text
                0x0000000000000000 (    0x68) go.buildid [FUNC, NameNList, MangledNameNList, Merged, NList] 
                0x0000000000000068 (   0x110) sync/atomic.(*Value).Store [FUNC, NameNList, MangledNameNList, NList] 
                0x0000000000000178 (    0x10) sync/atomic.CompareAndSwapUintptr [FUNC, NameNList, MangledNameNList, NList] 
...

If I build without -w, then symbols finds nothing:

go.o [arm64, 0.001176 seconds]:
    null-uuid                            go.o [OBJECT, Empty]  
        0x0000000000000000 (0xac8398)  SEGMENT

The behavior is the opposite of what I would expect, and indicates that something is still not quite right in our mach-o generation.

Can you submit your change as a CL? As a test you can add the symbols go.o to cmd/link/dwarf_test.go.

I sent a CL so we can see how the trybots react to the change.

Change https://golang.org/cl/170377 mentions this issue: cmd/link/internal/ld: fix c-archive mach-o compatibility

Change https://golang.org/cl/170451 mentions this issue: cmd/link/internal/ld: ensure mach-o debug symbols are visible to App Store

However what's strange.. if I build with -w (i.e. no DWARF data), then symbols does show a list of symbols:
If I build without -w, then symbols finds nothing:

According to the symbols man page, it will look in several places for symbol data. If DWARF data is present, then that takes precedence over everything else. So I think what's happening is if we skip DWARF (i.e. -ldflags=-w), then it reads the basic symbol info out of the __TEXT segment and displays it. But when we build with DWARF, it ignores everything else but then fails to parse the DWARF data for some reason thus marking the object as "Empty" (even though the man page claims the different data sources are merged together).

I am at a loss as to why the DWARF data is not being parsed correctly (especially since dwarfdump has no problem), but figuring that out would help solve #22716

SOURCES OF SYMBOL INFORMATION
     symbols uses multiple sources of symbol information. All sources are queried, and the resulting data is merged.

     NLIST symbol information is found in the LINKEDIT segment. It is imprecise, and only contains a starting
     address, not a length or ending address. Nlist data is not required to (and typically does not) reference all
     symbols. Usually only external symbol information is available. Even that may be removed by use of strip(1)

     DEBUG MAP symbol information is multi-part. A per-symbol reference in the original file points to an external
     file containing additional debug information, usually DWARF.

     FUNCTION STARTS symbol information is found in the LC_FUNCTION_STARTS load command of the target. It is impre-
     cise, and only contains a starting address, not a length or ending address. Furthermore, it contains no name or
     mangled name information. The LC_FUNCTION_STARTS load command is optional, it may not be found in all targets.

     DWARF symbol information is true debug info. It is usually precise (but it is not required to be). When avail-
     able, dwarf information is treated as more reliable than any other information source, and conflicts are
     resolved in favor of dwarf data.

     DSYM. A dSYM is an external file containing DWARF and NLIST symbol information.

I have opened a technical support incident (TSI) with Apple about this issue, as well as detailed the problem in rdar://49599876. They are forwarding to the relevant teams/developers and hopefully we can get some insight into what's wrong with our generated mach-o objects.

I have opened a technical support incident (TSI) with Apple about this issue, as well as detailed the problem in rdar://49599876

I heard back on my TSI!!

Thanks for bringing this to our attention. Our Developer Tools team has analyzed your object files, in particular the one with DWARF, and identified potential improvements for how DWARF data is extracted from an object file by the symbols tool. Specifically, the tool is currently short-circuiting because it encountered a DW_TAG_inlined_subroutine abbreviation table that includes a pair of DW_AT_call_line, DW_FORM_udata. Using DW_FORM_udata is valid DWARF, so handling this better is something the team may consider for the future. We can’t guarantee if or when this may happen.

One change for you to try and make this better right now — could you try using DW_FORM_data1 or DW_FORM_data2 in place of DW_FORM_udata? If you do so, does your interaction with the symbols data improve to a workable level?

One change for you to try and make this better right now — could you try using DW_FORM_data1 or DW_FORM_data2 in place of DW_FORM_udata? If you do so, does your interaction with the symbols data improve to a workable level?

I confirmed that the following change makes symbols go.o work correctly:

diff --git a/src/cmd/internal/dwarf/dwarf.go b/src/cmd/internal/dwarf/dwarf.go
index df80039063..b0349a15bc 100644
--- a/src/cmd/internal/dwarf/dwarf.go
+++ b/src/cmd/internal/dwarf/dwarf.go
@@ -434,7 +434,7 @@ var abbrevs = [DW_NABRV]dwAbbrev{
                        {DW_AT_low_pc, DW_FORM_addr},
                        {DW_AT_high_pc, DW_FORM_addr},
                        {DW_AT_call_file, DW_FORM_data4},
-                       {DW_AT_call_line, DW_FORM_udata},
+                       //{DW_AT_call_line, DW_FORM_udata},
                },
        },

@@ -446,7 +446,7 @@ var abbrevs = [DW_NABRV]dwAbbrev{
                        {DW_AT_abstract_origin, DW_FORM_ref_addr},
                        {DW_AT_ranges, DW_FORM_sec_offset},
                        {DW_AT_call_file, DW_FORM_data4},
-                       {DW_AT_call_line, DW_FORM_udata},
+                       //{DW_AT_call_line, DW_FORM_udata},
                },
        },

I tried changing the udata to data1/data2 but that didn't work, and also made dwarfdump start throwing errors. I don't really understand enough about DWARF data to know the difference or what else is required here.

Can you open a new issue or amend #28997? You could also CC someone from the the Go team with DWARF knowledge.

For future people ending up here, the relevant commits are:
d2016af
a0c96a9
2ae793e
b56d1ba

@steeve i'm excited that symbols will now work properly based on that first commit, and bitcode support on the whole. Do we yet know when this will land in a release build?

@rudyrichter I believe everything will be in 1.13
In the mean time, we have a custom 1.11.10 binary distrib with the patches up there cherry-picked at https://github.com/znly/go/releases/tag/go1.11.10-znly-2

@steeve you will need to cherry-pick additional commits for bitcode support:

b4f3b8a
ba96564
0fe1986
277609f
eb00167
804a402

(also you should use cherry-pick -x when making picks so the original commit sha is noted)

@tmm1 hopefully they'll apply cleanly on top of 1.11.10! We haven't updated to 1.12 because we suspected a speed regression due to be10ad7 (not confirmed)
Thanks for the cherry-pick hint !