syscall: darwin arm{,64} libSystem syscalls are not compliant with the App Store submissions

[s-s]

What version of Go are you using (go version)?

go version devel +bf870f3d54 Tue Nov 27 18:04:27 2018 +0300 darwin/amd64

Does this issue reproduce with the latest release?

Only master branch affected, go 1.11.2 and 1.10.5 are fine (as they use bare Syscall functions).

What operating system and processor architecture are you using (go env)?

go env Output

GOARCH="amd64"
GOBIN=""
GOCACHE="off"
GOEXE=""
GOFLAGS="-ldflags=-s -ldflags=-w"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="<...>/.build"
GOPROXY=""
GORACE=""
GOROOT="<...>/go"
GOTMPDIR=""
GOTOOLDIR="<...>/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build998597890=/tmp/go-build -gno-record-gcc-switches -fno-common"

Actually, I'm using gomobile, so GOARCHs are arm, arm64 (and 386, amd64 for simulator).

What did you do?

I'm building an iOS framework using gomobile and Go from master branch. Then I submit an app with this framework to the App Store Connect. After submission some (unknown) post processing happens on Apple's side.

What did you expect to see?

Binary should be processed successfully. (It is for go 1.11.2 and 1.10.5)

What did you see instead?

Binary is rejected for "Non-public API usage" - The app references non-public symbols: ___getdirentries64, _fstat64, _lstat64, _stat64

This happens because of these changes: a3b0144, c9762b8 in master branch

PS: I've solved it by patching syscall/mksyscall.pl - reverting back bare Syscall usage for mentioned symbols and generating updated zsyscall_darwin_{arm,arm64,386,amd64}.{go,s} using syscall/mksyscall.pl and syscall/mkasm_darwin.go.

[ianlancetaylor]

CC @randall77 @eliasnaur

[eliasnaur]

Because it is a regression, I took the liberty and made this issue a release blocker.

[eliasnaur]

According to the macOS stat(2) man page, platforms released after widening inodes to 64-bit do not have the *64 variants available, and the non-postfixed system calls are already 64-bit compliant:

"Platforms that were released after these updates only have the newer variants available to them. These platforms have the macro _DARWIN_FEATURE_ONLY_64_BIT_INODE defined."

I checked with a simple Xcode project that _DARWIN_FEATURE_ONLY_64_BIT_INODE is indeed defined for iOS builds, but not for macOS builds.

The fix for fstat, lstat, stat seems to be to drop the "64" postfix on iOS.

Getdirentries is more complicated. According to its man page,

"As of Mac OS X 10.6, getdirentries() is deprecated, and it is recommended that applications use readdir(3) rather than using
getdirentries() directly. Due to limitations with the system call, getdirentries() will not work with 64-bit inodes; in order to use
getdirentries(), _DARWIN_NO_64_BIT_INODE must be defined. See stat(2) for more information on _DARWIN_NO_64_BIT_INODE and its other
effects."

Since _DARWIN_NO_64_BIT_INODE must be defined to access getdirentries, and since iOS has _DARWIN_FEATURE_ONLY_64_BIT_INODE defined, it is likely that getdirentries is not (supposed to be) available on iOS in any form.

So it seems that getdirentries has to be replaced, at least on iOS. I don't know whether that's realistic. Perhaps getdirentries could be removed from package syscall on iOS altogether, even though it breaks the Go compatibility guarantee.

[eliasnaur]

I worked a bit on this; removing the "64" postfixes was easy, but getdirentries is a central system call for directory listing, and I couldn't think of a straightforward replacement for it. The man pages suggest the (fd)opendir/readdir/closedir API but those calls operate on directory stream pointers, not file descriptors. I'm leaving this for more capable hands.

[ianlancetaylor]

@eliasnaur Want to send the fixes you have? After we get those in, changing syscall.ReadDirent to not use Getdirentries on Darwin should be doable. I'm not too concerned about mobile programs calling syscall.Getdirentries directly.

[gopherbot]

Change https://golang.org/cl/151938 mentions this issue: syscall: avoid "64"-postfixed libSystem syscalls on iOS

[gopherbot]

Change https://golang.org/cl/153338 mentions this issue: runtime,os,syscall,internal/poll: replace getdirentries on iOS

[eliasnaur]

@s-s, can you please verify that the issue is fixed in go tip (or the beta release that is imminent)?