cmd/link: c-archive iOS App Store submissions fail without -w

[eliasnaur]

I've been using the new golang patches on master for a few days and have been able to run my golang code in the simulator and on tvOS/iOS devices. (Note that I am still using go directly to generate lib.a as outlined above instead of leveraging gomobile).

Now I'm trying to submit an app to the store and running into some weird issues. When submitting to the App Store, I get an email saying processing was stopped:

App Store Connect: Processing stopped for MyApp
While processing your iOS app errors occurred in the app thinning process, and your app couldn’t be thinned. If your app contains bitcode, bitcode processing may have failed. Because of these errors, this build of your app will not be able to be submitted for review or placed on the App Store. For information that may help resolve this issue, see Tech Note 2432.

I tried the debugging steps linked in the Tech Note to "rebuild from bitcode" in Xcode, and am getting a strange error about dSYMs missing in IDEDistributionPipeline.log:

ipatool failed with an exception: #<CmdSpec::NonZeroExcitException: Command exited with pid 71395 exit 1:
/Applications/Xcode.app/Contents/Developer/usr/bin/symbols -failIfMissingDsym -symbolsPackageDir /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app/MyApp
Stdout:
    ERROR: Did not find dSYM for MyApp
    /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app/MyApp [arm64, 0.001634 seconds]:

Stderr:
    error: Could not load symbol information for 22D519D5-0C78-3905-8681-93FE91325191.  Symbols file will contain no data.

>
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:248:in `run'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2473:in `block in CompileOrStripBitcodeInBundle'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2396:in `each'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2396:in `CompileOrStripBitcodeInBundle'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2628:in `ProcessIPA'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:3445:in `<main>'

I noticed this:

https://github.com/golang/mobile/blob/6621de06e1e9d52c38a8c542770bdbeac9ecfef6/cmd/gomobile/build_iosapp.go#L74-L75

So I tried again with -ldflags=-w, and that worked! I assumed that wasn't necessary anymore after bcdbd58 but I guess it is still required.

I am now able to submit builds up to App Store:

App Store Connect: Version 2019 (3.25.619) for MyApp has completed processing.

Originally posted by @tmm1 in #22395 (comment)

[eliasnaur]

CL 111275 you linked to only disables DWARF combining for buildmode=exe, which is what gomobile's build_iosapp.go is generating. For c-archive, Xcode is supposed to split the DWARF from the archive during its build.
It could be that our __asm marker is incompatible with the DWARF we generate. Can you submit a binary without -w if bitcode is disabled? Perhaps you need to disable the __asm marker as well.

[tmm1]

It could be that our __asm marker is incompatible with the DWARF we generate. Can you submit a binary without -w if bitcode is disabled? Perhaps you need to disable the __asm marker as well.

The issue occurs when I try to "recompile from bitcode" in Xcode's Organizer with an archived ipa.

If I disable __asm then I have to disable bitcode, in which case I can't attempt to "recompile from bitcode" anymore.

[eliasnaur]

Yes, but your initial problem was the error email from Apple. I'm trying to work why -w is needed. If it is needed because of the __asm section and bitcode then we'll just do -w by default.

If not, then I'd like to figure out what else could cause the problem. Including DWARF in object files and archives shouldn't be a problem. The special Apple dwarf splitting happens for executables.

[tmm1]

Since the apple email mentions app thinning, I'm pretty sure a bitcode=off build won't trigger it. But I will try uploading a build with bitcode off but __asm included to see what happens.

It would be nice to be able to emit debug information for the golang code. I'm guessing the dwarf format it expects for asm is slightly different? I will poke at the generated sections and see if I notice anything different between real assembled objects vs go.o

[eliasnaur]

Stack traces on mobile is #22716.

[tmm1]

I disabled bitcode on my project, removed the -w and tried to upload a build. It fails processing in Xcode and never got to the upload. The logs show a similar error:

2019-03-25 19:45:41 +0000  Running /Applications/Xcode.app/Contents/Developer/usr/bin/symbols '-noTextInSOD' '-noDaemon' '-arch' 'all' '-symbolsPackageDir' '/var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/XcodeDistPipeline.j3X/Symbols' '/var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/XcodeDistPipeline.j3X/MyApp.app.dSYM/Contents/Resources/Dwarf/MyApp'
2019-03-25 19:45:41 +0000  error: Could not load symbol information for B60F1C62-D42E-3E17-90DF-9F6ADE25D2CD.  Symbols file will contain no data.
2019-03-25 19:45:41 +0000  /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/XcodeDistPipeline.j3X/MyApp.app.dSYM/Contents/Resources/Dwarf/MyApp [arm64, 0.018700 seconds]:
2019-03-25 19:45:41 +0000  /Applications/Xcode.app/Contents/Developer/usr/bin/symbols exited with 1

It seems something about the combination of __asm and DWARF in the go.o object causes the entire app object to end up with broken debug symbols.

Previously I was using fastlane to upload my builds and not Xcode, so just to make sure I re-enabled bitcode (still without -w) and tried to upload from Xcode this time. Interestingly that combination gets further and uploads to App Store, but then processing fails like it did before.

The underlying problem seems the same, it just depends on where the symbol extraction is happening. If you force it locally with bitcode recompile or non-bitcode build you can see the error right away, but if you upload a bitcode build it happens on Apple's servers and errors out the same way there.

Unfortunately it seems like now that we're emitting __asm unconditionally, it's not possible to submit either bitcode or non-bitcode builds as long as DWARF generation for go.o is enabled.

[tmm1]

For reference, here's what size shows for go.o:

libtest_ios_arm64.a(go.o):
Segment : 0x0 (vmaddr 0x0 fileoff 4096)
        Section (__TEXT, __text): 0x210478 (addr 0x0 offset 4096)
        Section (__DATA, __rodata): 0x14e3c9 (addr 0x210480 offset 2167936)
        Section (__DATA, __typelink): 0x2320 (addr 0x35e860 offset 3536992)
        Section (__DATA, __itablink): 0x838 (addr 0x360b80 offset 3545984)
        Section (__DATA, __gosymtab): 0x0 (addr 0x3613b8 offset 3548088)
        Section (__DATA, __gopclntab): 0x1a69e0 (addr 0x3613c0 offset 3548096)
        Section (__LLVM, __asm): 0x1 (addr 0x508000 offset 5279744)
        Section (__DATA, __noptrdata): 0x2a981 (addr 0x508020 offset 5279776)
        Section (__DATA, __mod_init_func): 0x8 (addr 0x5329a8 offset 5454248)
        Section (__DATA, __data): 0x9c50 (addr 0x5329c0 offset 5454272)
        Section (__DATA, __bss): 0x1e150 (addr 0x53c620 offset 0)
        Section (__DATA, __noptrbss): 0x3058 (addr 0x55a780 offset 0)
        Section (__DWARF, __debug_abbrev): 0x1e0 (addr 0x55e000 offset 5496832)
        Section (__DWARF, __debug_line): 0x9a93d (addr 0x55e1e0 offset 5497312)
        Section (__DWARF, __debug_frame): 0x501ec (addr 0x5f8b1d offset 6130461)
        Section (__DWARF, __debug_pubnames): 0xd283 (addr 0x648d09 offset 6458633)
        Section (__DWARF, __debug_pubtypes): 0x26f05 (addr 0x655f8c offset 6512524)
        Section (__DWARF, __debug_gdb_scri): 0x4c (addr 0x67ce91 offset 6672017)
        Section (__DWARF, __debug_info): 0x165dc5 (addr 0x67cedd offset 6672093)
        Section (__DWARF, __debug_loc): 0x1f6af7 (addr 0x7e2ca2 offset 8137890)
        Section (__DWARF, __debug_ranges): 0xb03b0 (addr 0x9d9799 offset 10196889)
        total 0xa89044
total 0x0

vs for 000011.o which is generated via clang's assembler:

libtest_ios_arm64.a(000011.o):
Segment : 0x1b3 (vmaddr 0x0 fileoff 704)
        Section (__TEXT, __text): 0x4c (addr 0x0 offset 704)
        Section (__LLVM, __asm): 0x1 (addr 0x4c offset 780)
        Section (__DWARF, __debug_info): 0xbd (addr 0x4d offset 781)
        Section (__DWARF, __debug_abbrev): 0x28 (addr 0x10a offset 970)
        Section (__DWARF, __debug_aranges): 0x30 (addr 0x132 offset 1010)
        Section (__DWARF, __debug_line): 0x51 (addr 0x162 offset 1058)
        total 0x1b3
total 0x1b3

It's a bit strange that go.o says total 0x0 at the end, although that's consistent even if I build go.o with -ldflags=-w:

libtest_ios_arm64.a(go.o):
Segment : 0x0 (vmaddr 0x0 fileoff 4096)
        Section (__TEXT, __text): 0x210478 (addr 0x0 offset 4096)
        Section (__DATA, __rodata): 0x14e3c9 (addr 0x210480 offset 2167936)
        Section (__DATA, __typelink): 0x2320 (addr 0x35e860 offset 3536992)
        Section (__DATA, __itablink): 0x838 (addr 0x360b80 offset 3545984)
        Section (__DATA, __gosymtab): 0x0 (addr 0x3613b8 offset 3548088)
        Section (__DATA, __gopclntab): 0x1a69e0 (addr 0x3613c0 offset 3548096)
        Section (__LLVM, __asm): 0x1 (addr 0x508000 offset 5279744)
        Section (__DATA, __noptrdata): 0x2a981 (addr 0x508020 offset 5279776)
        Section (__DATA, __mod_init_func): 0x8 (addr 0x5329a8 offset 5454248)
        Section (__DATA, __data): 0x9c50 (addr 0x5329c0 offset 5454272)
        Section (__DATA, __bss): 0x1e150 (addr 0x53c620 offset 0)
        Section (__DATA, __noptrbss): 0x3058 (addr 0x55a780 offset 0)
        total 0x55d4fb
total 0x0

[eliasnaur]

Can you try removing the __asm generating code from Go just to make sure that's the reason -w doesn't work?

[tmm1]

Can you try removing the __asm generating code from Go just to make sure that's the reason -w doesn't work?

Good idea! I removed just the .llvmasm change, disabled bitcode on my project and attempted to upload. It failed the same way:

with the logs showing:

2019-03-25 21:03:18 +0000  error: Could not load symbol information for 5DB22712-C769-3962-A5C1-E5995FE4F8A2.  Symbols file will contain no data.
2019-03-25 21:03:18 +0000  /Applications/Xcode.app/Contents/Developer/usr/bin/symbols exited with 1

So clearly this is not related to the __asm bit and something else is going on. I will try go1.12 next to see if it has the same behavior.

[tmm1]

So clearly this is not related to the __asm bit and something else is going on. I will try go1.12 next to see if it has the same behavior.

Same thing, I can't get it working at all on go1.12.1 unless I add -ldflags=-w.

I also tried go1.11.6 and it has the same behavior.

Since this is my first time trying to use golang on iOS/tvOS I'm not sure if this ever worked in the past.

[tmm1]

So here's a simple test that shows that the go.o generated by golang is not able to be parsed by /usr/bin/symbols:

$ go version
go version go1.11.6 darwin/amd64

$ cat test.go
package main

import "C"
import "log"

func main() {
    log.Printf("foo")
}

$ CGO_CFLAGS=-g go build -buildmode=c-archive -o test.a test.go
$ ar xv test.a
x - __.SYMDEF SORTED
x - go.o
x - 000000.o
x - 000001.o
x - 000002.o
x - 000003.o
x - 000004.o
x - 000005.o
x - 000006.o
x - 000007.o
x - 000008.o
x - 000009.o
x - 000010.o

$ size -x -l -m 000000.o
Segment : 0x98 (vmaddr 0x0 fileoff 528)
	Section (__TEXT, __text): 0x0 (addr 0x0 offset 528)
	Section (__DWARF, __apple_names): 0x24 (addr 0x0 offset 528)
	Section (__DWARF, __apple_objc): 0x24 (addr 0x24 offset 564)
	Section (__DWARF, __apple_namespac): 0x24 (addr 0x48 offset 600)
	Section (__DWARF, __apple_types): 0x2c (addr 0x6c offset 636)
	total 0x98
total 0x98

$ symbols 000000.o
000000.o [x86_64, 0.000155 seconds]:
    null-uuid                            /Users/tmm1/fancybits/bitcode/000000.o [OBJECT, FaultedFromDisk]
        0x0000000000000000 (    0x98)  SEGMENT
            0x0000000000000000 (    0x24) __DWARF __apple_names
            0x0000000000000024 (    0x24) __DWARF __apple_objc
            0x0000000000000048 (    0x24) __DWARF __apple_namespac
            0x000000000000006c (    0x2c) __DWARF __apple_types

$ size -x -l -m go.o
Segment : 0x0 (vmaddr 0x0 fileoff 4096)
	Section (__TEXT, __text): 0x9380c (addr 0x0 offset 4096)
	Section (__DATA, __rodata): 0x47809 (addr 0x93820 offset 608288)
	Section (__DATA, __typelink): 0xbec (addr 0xdb040 offset 901184)
	Section (__DATA, __itablink): 0x60 (addr 0xdbc30 offset 904240)
	Section (__DATA, __gosymtab): 0x0 (addr 0xdbc90 offset 904336)
	Section (__DATA, __gopclntab): 0x77f25 (addr 0xdbca0 offset 904352)
	Section (__DATA, __noptrdata): 0xcc5c (addr 0x154000 offset 1396736)
	Section (__DATA, __mod_init_func): 0x8 (addr 0x160c60 offset 1449056)
	Section (__DATA, __data): 0x6910 (addr 0x160c80 offset 1449088)
	Section (__DATA, __bss): 0x1c790 (addr 0x1675a0 offset 0)
	Section (__DATA, __noptrbss): 0x2598 (addr 0x183d40 offset 0)
	Section (__DWARF, __debug_abbrev): 0x1d3 (addr 0x187000 offset 1478656)
	Section (__DWARF, __debug_line): 0x25975 (addr 0x1871d3 offset 1479123)
	Section (__DWARF, __debug_frame): 0x14224 (addr 0x1acb48 offset 1633096)
	Section (__DWARF, __debug_pubnames): 0x812f (addr 0x1c0d6c offset 1715564)
	Section (__DWARF, __debug_pubtypes): 0xae69 (addr 0x1c8e9b offset 1748635)
	Section (__DWARF, __debug_gdb_scri): 0x45 (addr 0x1d3d04 offset 1793284)
	Section (__DWARF, __debug_info): 0x767d1 (addr 0x1d3d49 offset 1793353)
	Section (__DWARF, __debug_loc): 0x6bb48 (addr 0x24a51a offset 2278682)
	Section (__DWARF, __debug_ranges): 0x26030 (addr 0x2b6062 offset 2719842)
	total 0x2daeb4
total 0x0

$ symbols go.o
Unable to find file, pid, task or signature matching: go.o
WARNING: You currently do not have task_for_pid() privileges.

$ symbols -v
symbols version:			@(#)PROGRAM:symbols  PROJECT:SamplingTools-64460.8
CoreSymbolicationDT.framework version:	64460.7

[tmm1]

I got a tip on twitter that the newly release Xcode 10.2 might help: https://twitter.com/MarkVillacampa/status/1110309609464455169?s=20

Unfortunately there was no difference. The behavior of /usr/bin/symbols is the same, even though the version number increased:

$ symbols -v
symbols version:			@(#)PROGRAM:symbols  PROJECT:SamplingTools-64490.33
CoreSymbolicationDT.framework version:	64490.26

and when I try uploading from Xcode I get the same "Symbols tool failed".

[eliasnaur]

I don't see you're using GOARCH so it seems to me this won't even work for macOS programs; I suppose submitting to the macOS App Store will probably also fail without -w?
Pleasing symbolscould get us a long way on #22716, but I'm not sure where to begin figuring out what is upsetting it; the error is useless.

[tmm1]

I don't see you're using GOARCH so it seems to me this won't even work for macOS programs

Correct, the issue affects all mach-o objects linked by golang regardless of architecture or OS.

Pleasing symbols could get us a long way on #22716, but I'm not sure where to begin figuring out what is upsetting it; the error is useless.

What's interesting is that symbols chokes on the object created with -ldflags=-w as well. It seems Xcode just avoids calling symbols in the case where no DWARF data is present, which is why the -w workaround is effective.

I'm not sure how best to proceed from here, either. I can try opening a radar with apple to see if they have any insights.

It might be worth trying to disable SIP and running lldb -- symbols to see if its attempting to open the file and where it gives up. My hunch is that its something about the mach-o headers that's out of place since the issue occurs with and without DWARF data present.

[Args]

and when I try uploading from Xcode I get the same "Symbols tool failed".

Same here :(，have any luck with that ?