syscall: darwin arm{,64} libSystem syscalls are not compliant with the App Store submissions #28984

s-s · 2018-11-28T09:29:57Z

What version of Go are you using (`go version`)?

go version devel +bf870f3d54 Tue Nov 27 18:04:27 2018 +0300 darwin/amd64

Does this issue reproduce with the latest release?

Only master branch affected, go 1.11.2 and 1.10.5 are fine (as they use bare Syscall functions).

What operating system and processor architecture are you using (`go env`)?

go env Output

GOARCH="amd64"
GOBIN=""
GOCACHE="off"
GOEXE=""
GOFLAGS="-ldflags=-s -ldflags=-w"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="<...>/.build"
GOPROXY=""
GORACE=""
GOROOT="<...>/go"
GOTMPDIR=""
GOTOOLDIR="<...>/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build998597890=/tmp/go-build -gno-record-gcc-switches -fno-common"

Actually, I'm using gomobile, so GOARCHs are arm, arm64 (and 386, amd64 for simulator).

What did you do?

I'm building an iOS framework using gomobile and Go from master branch. Then I submit an app with this framework to the App Store Connect. After submission some (unknown) post processing happens on Apple's side.

What did you expect to see?

Binary should be processed successfully. (It is for go 1.11.2 and 1.10.5)

What did you see instead?

Binary is rejected for "Non-public API usage" - The app references non-public symbols: ___getdirentries64, _fstat64, _lstat64, _stat64

This happens because of these changes: a3b0144, c9762b8 in master branch

PS: I've solved it by patching syscall/mksyscall.pl - reverting back bare Syscall usage for mentioned symbols and generating updated zsyscall_darwin_{arm,arm64,386,amd64}.{go,s} using syscall/mksyscall.pl and syscall/mkasm_darwin.go.

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2018-11-28T13:54:36Z

CC @randall77 @eliasnaur

eliasnaur · 2018-11-28T15:17:53Z

Because it is a regression, I took the liberty and made this issue a release blocker.

eliasnaur · 2018-11-28T15:44:30Z

According to the macOS stat(2) man page, platforms released after widening inodes to 64-bit do not have the *64 variants available, and the non-postfixed system calls are already 64-bit compliant:

"Platforms that were released after these updates only have the newer variants available to them. These platforms have the macro _DARWIN_FEATURE_ONLY_64_BIT_INODE defined."

I checked with a simple Xcode project that _DARWIN_FEATURE_ONLY_64_BIT_INODE is indeed defined for iOS builds, but not for macOS builds.

The fix for fstat, lstat, stat seems to be to drop the "64" postfix on iOS.

Getdirentries is more complicated. According to its man page,

"As of Mac OS X 10.6, getdirentries() is deprecated, and it is recommended that applications use readdir(3) rather than using
getdirentries() directly. Due to limitations with the system call, getdirentries() will not work with 64-bit inodes; in order to use
getdirentries(), _DARWIN_NO_64_BIT_INODE must be defined. See stat(2) for more information on _DARWIN_NO_64_BIT_INODE and its other
effects."

Since _DARWIN_NO_64_BIT_INODE must be defined to access getdirentries, and since iOS has _DARWIN_FEATURE_ONLY_64_BIT_INODE defined, it is likely that getdirentries is not (supposed to be) available on iOS in any form.

So it seems that getdirentries has to be replaced, at least on iOS. I don't know whether that's realistic. Perhaps getdirentries could be removed from package syscall on iOS altogether, even though it breaks the Go compatibility guarantee.

eliasnaur · 2018-11-29T12:45:24Z

I worked a bit on this; removing the "64" postfixes was easy, but getdirentries is a central system call for directory listing, and I couldn't think of a straightforward replacement for it. The man pages suggest the (fd)opendir/readdir/closedir API but those calls operate on directory stream pointers, not file descriptors. I'm leaving this for more capable hands.

ianlancetaylor · 2018-11-29T21:11:40Z

@eliasnaur Want to send the fixes you have? After we get those in, changing syscall.ReadDirent to not use Getdirentries on Darwin should be doable. I'm not too concerned about mobile programs calling syscall.Getdirentries directly.

gopherbot · 2018-11-30T11:42:24Z

Change https://golang.org/cl/151938 mentions this issue: syscall: avoid "64"-postfixed libSystem syscalls on iOS

The stat(2) man page contain this comment about the 64-bit versions of the system file functions: "Platforms that were released after these updates only have the newer variants available to them. These platforms have the macro _DARWIN_FEATURE_ONLY_64_BIT_INODE defined." It turns out that on iOS the _DARWIN_FEATURE_ONLY_64_BIT_INODE is defined and that even though the "64"-postfixed versions are accessible they are deemed private. Apps that refer to private API are not admissible on App Store, and after the Go runtime started using libSystem instead of direct syscalls, the App Store submission checks reject apps built with Go tip. The fix is simple: use the non-postfixed versions on iOS. getdirentries(2) is not changed; it is not available at all on iOS and needs replacement. Updates #28984 Change-Id: Icb8d44e271456acaa1913ba486fcf5b569722fa9 Reviewed-on: https://go-review.googlesource.com/c/151938 Run-TryBot: Elias Naur <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Keith Randall <[email protected]>

gopherbot · 2018-12-09T06:02:01Z

Change https://golang.org/cl/153338 mentions this issue: runtime,os,syscall,internal/poll: replace getdirentries on iOS

eliasnaur · 2018-12-13T20:33:25Z

@s-s, can you please verify that the issue is fixed in go tip (or the beta release that is imminent)?

s-s · 2018-12-14T09:59:13Z

@eliasnaur, checked with tip version - everything is ok now! Thank you!
PS: go version devel +38e7177c94 Fri Dec 14 05:48:18 2018 +0000 darwin/amd64

gopherbot · 2018-12-18T02:37:14Z

Change https://golang.org/cl/154658 mentions this issue: unix: avoid "64"-postfixed libSystem calls on iOS

gopherbot · 2018-12-18T02:37:22Z

Change https://golang.org/cl/154659 mentions this issue: unix: remove Getdirentries on iOS

Parallel to CL 151938 for package x/sys/unix instead of syscall. iOS needs to use these functions without the "64" postfix. (The functions do exist, but the App Store bans their use.) Updates golang/go#28984 Change-Id: I6b82950700cc8a1afca612844b05fa007574e008 Reviewed-on: https://go-review.googlesource.com/c/154658 Reviewed-by: Brad Fitzpatrick <[email protected]>

This system call doesn't exist on iOS. Update golang/go#28984 Change-Id: I92eb6fd3eb263863a31338bc18c9826a2242434b Reviewed-on: https://go-review.googlesource.com/c/154659 Reviewed-by: Brad Fitzpatrick <[email protected]>

gopherbot · 2019-03-20T23:20:06Z

Change https://golang.org/cl/168479 mentions this issue: syscall: deprecate Getdirentries on darwin

Getdirentries is implemented with the __getdirentries64 function in libSystem.dylib. That function works, but it's on Apple's can't-be-used-in-an-app-store-application list. Implement Getdirentries using the underlying fdopendir/readdir_r/closedir. The simulation isn't faithful, and could be slow, but it should handle common cases. Don't use Getdirentries in the stdlib, use fdopendir/readdir_r/closedir instead (via (*os.File).readdirnames). Fixes #30933 Update #28984 RELNOTE=yes Change-Id: Ia6b5d003e5bfe43ba54b1e1d9cfa792cc6511717 Reviewed-on: https://go-review.googlesource.com/c/go/+/168479 Reviewed-by: Emmanuel Odeke <[email protected]> Reviewed-by: Brad Fitzpatrick <[email protected]> Run-TryBot: Emmanuel Odeke <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>

gopherbot · 2019-04-03T21:31:46Z

Change https://golang.org/cl/170640 mentions this issue: [release-branch.go1.12] syscall: avoid _getdirentries64 on darwin

Getdirentries is implemented with the __getdirentries64 function in libSystem.dylib. That function works, but it's on Apple's can't-be-used-in-an-app-store-application list. Implement Getdirentries using the underlying fdopendir/readdir_r/closedir. The simulation isn't faithful, and could be slow, but it should handle common cases. Don't use Getdirentries in the stdlib, use fdopendir/readdir_r/closedir instead (via (*os.File).readdirnames). (Incorporates CL 170837 and CL 170698, which were small fixes to the original tip CL.) Fixes #31244 Update #28984 RELNOTE=yes Change-Id: Ia6b5d003e5bfe43ba54b1e1d9cfa792cc6511717 Reviewed-on: https://go-review.googlesource.com/c/go/+/168479 Reviewed-by: Emmanuel Odeke <[email protected]> Reviewed-by: Brad Fitzpatrick <[email protected]> Run-TryBot: Emmanuel Odeke <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> (cherry picked from commit 9da6530) Reviewed-on: https://go-review.googlesource.com/c/go/+/170640 Run-TryBot: Brad Fitzpatrick <[email protected]>

zx2c4 · 2019-09-19T05:21:23Z

@eliasnaur does this need a corresponding commit in x/sys?

eliasnaur · 2019-09-19T08:10:01Z

Do you mean https://golang.org/cl/154659 or something else?

zx2c4 · 2019-09-19T10:09:52Z

@eliasnaur That CL is for iOS-only, right? The syscall change later got ported to macOS (x86, amd64) too, due to similar appstore restrictions there. But I don't see a corresponding x/sys change for macOS.

eliasnaur · 2019-09-19T11:22:50Z

On Thu Sep 19, 2019 at 3:12 AM Jason A. Donenfeld wrote: @eliasnaur That CL is for iOS-only, right? The syscall change later got ported to macOS (x86, amd64) too, due to similar appstore restrictions there. But I don't see a corresponding x/sys change for macOS.

I see. Then yes, I suppose a x/sys port is needed.

ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 28, 2018

ianlancetaylor added this to the Go1.12 milestone Nov 28, 2018

ianlancetaylor added the OS-Darwin label Nov 28, 2018

eliasnaur added the release-blocker label Nov 28, 2018

andybons assigned randall77 Nov 28, 2018

gopherbot closed this as completed in 9eb383e Dec 13, 2018

zx2c4 mentioned this issue Mar 19, 2019

go 1.12 binaries rejected from mac app store due to ___getdirentries64 #30933

Closed

halseth mentioned this issue May 2, 2019

x/mobile: apps built with go 1.12, gomobile and Xcode 10.2 are using non-public APIs #31628

Closed

eliasnaur reopened this Sep 19, 2019

eliasnaur closed this as completed Sep 19, 2019

zx2c4 mentioned this issue Sep 19, 2019

x/sys/unix: ___getdirentries64 results in app store rejections #34400

Closed

golang locked and limited conversation to collaborators Sep 18, 2020

gopherbot added the FrozenDueToAge label Sep 18, 2020

rsc unassigned randall77 Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

syscall: darwin arm{,64} libSystem syscalls are not compliant with the App Store submissions #28984

syscall: darwin arm{,64} libSystem syscalls are not compliant with the App Store submissions #28984

s-s commented Nov 28, 2018

ianlancetaylor commented Nov 28, 2018

eliasnaur commented Nov 28, 2018

eliasnaur commented Nov 28, 2018 •

edited

Loading

eliasnaur commented Nov 29, 2018 •

edited

Loading

ianlancetaylor commented Nov 29, 2018

gopherbot commented Nov 30, 2018

gopherbot commented Dec 9, 2018

eliasnaur commented Dec 13, 2018

s-s commented Dec 14, 2018

gopherbot commented Dec 18, 2018

gopherbot commented Dec 18, 2018

gopherbot commented Mar 20, 2019

gopherbot commented Apr 3, 2019

zx2c4 commented Sep 19, 2019

eliasnaur commented Sep 19, 2019 •

edited

Loading

zx2c4 commented Sep 19, 2019

eliasnaur commented Sep 19, 2019 via email

syscall: darwin arm{,64} libSystem syscalls are not compliant with the App Store submissions #28984

syscall: darwin arm{,64} libSystem syscalls are not compliant with the App Store submissions #28984

Comments

s-s commented Nov 28, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

ianlancetaylor commented Nov 28, 2018

eliasnaur commented Nov 28, 2018

eliasnaur commented Nov 28, 2018 • edited Loading

eliasnaur commented Nov 29, 2018 • edited Loading

ianlancetaylor commented Nov 29, 2018

gopherbot commented Nov 30, 2018

gopherbot commented Dec 9, 2018

eliasnaur commented Dec 13, 2018

s-s commented Dec 14, 2018

gopherbot commented Dec 18, 2018

gopherbot commented Dec 18, 2018

gopherbot commented Mar 20, 2019

gopherbot commented Apr 3, 2019

zx2c4 commented Sep 19, 2019

eliasnaur commented Sep 19, 2019 • edited Loading

zx2c4 commented Sep 19, 2019

eliasnaur commented Sep 19, 2019 via email

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

eliasnaur commented Nov 28, 2018 •

edited

Loading

eliasnaur commented Nov 29, 2018 •

edited

Loading

eliasnaur commented Sep 19, 2019 •

edited

Loading