build: dyld: Symbol not found: _fdopendir$INODE64 on OS X 10.9 Mavericks

[probonopd]

What version of Go are you using (go version)?

go1.13.3.darwin-amd64.tar.gz

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

OS X 10.9.5 Build 13F34

What did you do?

Running OS X 10.9.5 Build 13F34, downloaded go1.13.3.darwin-amd64.tar.gz. Trying to run it

What did you expect to see?

Working go command

What did you see instead?

sh-3.2# export PATH=$PATH:/usr/local/go/bin
sh-3.2# go
go             gobject-query  gofmt          
sh-3.2# go
dyld: Symbol not found: _fdopendir$INODE64
  Referenced from: /usr/local/go/bin/go
  Expected in: flat namespace

Trace/BPT trap: 5

Before this is closed with "too old OS version": One of the main advantages of Go imho is that it is statically compiled and hence should run independent from the libraries shipped with the OS. Hence it would be highly appreciated if this could be made work, or at least explained why it is not possible.

It used to work in

sh-3.2# go version
go version go1.10.8 darwin/amd64

What has changed to make this no longer work?

Could https://github.com/macports/macports-legacy-support be used to fix this, especially macports/macports-ports@abe46d0?

[ianlancetaylor]

According to https://github.com/golang/go/wiki/MinimumRequirements Go 1.13 only supports macOS 10.10 and higher.

[probonopd]

Yes. Why?

[ianlancetaylor]

See #23011 and #23122.

[probonopd]

• all: announce end of support for old macOS releases #23011 does not contain a technical reason for why OS X 10.9 cannot be supported any longer, if I have not overlooked something
• all: remove support for OS X 10.8 and 10.9 #23122 just implements the removal of support for OS X 10.9

Not everyone can run the latest versions of macOS for various reasons. I would like to know what is technically preventing the latest Go versions to work on older Mac OS X and OS X versions.

[ianlancetaylor]

You are correct: the decision to drop support for older macOS releases was not made for technical reasons. We have limited resources. We do not want to support releases for which we do not run builders. We do not want to run builders for releases that are not receiving security updates. So we chose to drop support.

The only thing I know of that technically prevents the latest Go versions from running on old macOS versions is this issue that you have just filed. If you want to analyze this issue and find a fix and send in the fix, we would probably consider it if it is not too invasive. But we do not consider ourselves responsible for fixing the issue. We prefer to focus our limited resources on newer versions. We aren't even running the old systems that we would need to run to analyze and test any fix.

Older versions of Go should continue to work fine on older versions of macOS.

I'm sorry that this is likely not the answer you want to hear, but I think it is better to be realistic about what we can accomplish.

[probonopd]

We prefer to focus our limited resources on newer versions.

Why? Wouldn't it be more reasonable to focus the limited resources on the oldest possible Mac OS X release and delegate the duty to keep things binary compatible to Apple (read: if they break compatibility, blame them)?

[ianlancetaylor]

Clearly Go must support current versions of the operating system. Evidence shows that each new macOS release requires adjustments by Go. For example, right now there are several bug reports for the Catalina release (#34935, #35177, #35247). We are going to have to investigate and fix those problems. It would be nice if we could say "oh, that's Apple's fault, ask them to fix it," but in reality that clearly won't work. Apple's interest in supporting programs that are not created by Apple approved toolchains is very small. We have to accommodate them. They are unlikely to accommodate us.

Since we must spend time on macOS support, I think it is clearly better to spend that time on current macOS versions rather than old ones.

[probonopd]

Argh.

In any case, https://github.com/macports/macports-legacy-support seems to add the missing fopendir. Could macports-legacy-support be used to make Go work on less-than-recent systems? cc @kencu

[ianlancetaylor]

I'm likely missing something but I don't see how the macports-legacy-support repo would help. That is for C programs, not Go programs.

Have you tried downloading the source code for Go and building it yourself as described at https://golang.org/doc/install/source, instead of using the binary download?

[probonopd]

I'm likely missing something but I don't see how the macports-legacy-support repo would help. That is for C programs, not Go programs.

From looking at its description I was under the impression that macports-legacy-support contains implementations of some symbols that older Mac OS X, Mac OS, and macOS versions are lacking, in order to enable software built for later versions to run on earlier versions of the system. Hence I thought it might be a way to allow Go to run on earlier versions without having to be built on earlier versions (something the Go project wants to avoid due to the additional work).

I have not tried to build Go myself on the Mac, since my use case is a bit different: I would like to build Go applications on Linux, and run them on OS X 10.9.

[kencu]

Hi. Yes -- in MacPorts, one of our members, @Ionic , has been working with this with some success. https://github.com/macports/macports-ports/blob/52f3f71c33a430c6046efdddf9cce347ae2de3db/lang/go/Portfile#L68 . It currently does require a manual step to enable linking in the legacysupport library that contains the extra functions that perhaps we might automate one day.

@Ionic has also been working on an enhanced version of this of late, https://trac.macports.org/ticket/58935 that we are / I am just trying to fully wrap my head around. But it is do-able, it appears.

[Ionic]

Yes, MP legacy support can be used to get go working again.

Yes, MP legacy support is currently "broken" because it doesn't generate the needed symbols (for newer go versions).

Yes, I have modified the build environment to generate them, based on architecture.

No, I haven't made a PR for MP legacy support yet because I a.) am lazy, b.) was waiting for some review for obvious errors, though I guess there aren't any really and c.) was occupied with other things in the meantime.

I guess that go is right in using the suffixed symbol versions, there doesn't seem to be a bug in go as such. It's just unfortunate that older OS X/macOS versions don't provide it.

MP legacy support can naturally also be used outside of MacPorts (I do find the name a bit unfortunate, but I understand that it originated there), but even with a working go build, users of go will need to build their programs with special flags on older operating system versions to get a working binary (or one in the first place).

[Ionic]

This said, I consider this a hack to make go work on older platforms - not something that could be reasonably incorporated into go itself.

The other point is that, if go actually needs the fdopendir function, it doesn't make sense to not limit the supported platforms to those that provide it out of the box. The alternative would be to avoid using it xor, if needed, emulate it within go itself, which would increase maintenance load. Probably not a smart decision for the upstream project.

I do, however, see it as a working solution for downstream package systems, even though not fully automatic/transparent to end users.

[kencu]

As a tiny bit of help, how would one add a link support/runtime library to a go build automatically? I assume go must have a list of libraries it links in to the build somewhere (and clearly I'm demonstrating my lack of go fu here). If we could have that -lLegacySupport library linked in without adding in an external step, that would be helpful.

[ianlancetaylor]

It's complicated. On macOS the imported library is the use of /usr/lib/libsystem.B.dylib in a number of go:cgo_import_dynamic comments in runtime/sys_darwin.go. That would be easy enough to patch on your own system, but hard to do it in a general way.

For that matter, if you want to patch your own system, you could just change machogenasmsym in cmd/link/internal/ld/macho.go to not add the suffix.

[Ionic]

The issue isn't the suffix. (Or to be more precise: the missing suffix is only an implementation issue in MacportsLegacySupport. That's a solved issue, I just need to follow up on that.)

The issue is that the function is not provided on older Darwin versions at all.

What Ken was asking for was a way to specify other external libraries to be linked to for all operations by go itself.

My current workaround forces an external linker and passing custom external linker flags (essentially -L... and -lLegacySupport), but that's not "automagic" or transparent. Any go user would need to explicitly enable external linking and pass these flags manually in order to obtain working go programs - even after the binary has been built and installed successfully.

It was, however, the only way to do something like this I could find. Maybe there's a better way, though. One which works automatically, even after go was built/installed and particularly one that doesn't require an external linker.

[ianlancetaylor]

You could perhaps set the required flags in the GOFLAGS environment variable.

Perhaps syscall.Getdirentries could be rewritten to not call fdopendir.

[tklauser]

Perhaps syscall.Getdirentries could be rewritten to not call fdopendir.

It could maybe be changed to fall back to the raw syscall version, like we do for old Go releases in x/sys/unix, see https://golang.org/cl/196478

[Ionic]

I'm not advocating replacing the fdopendir usage. That's fine, and besides, go will likely be using more standard functionality that older systems don't provide but new ones do in the future. Working around such things will only make maintenance much, much harder without a clear gain - especially since Apple officially only supports the current OS release and at most the two most recent previous ones, or something along those lines.

Using the raw syscall value would likewise not help us. If only it might only make things more complicated. I'm not sure if a syscall that is unknown to the kernel and C library could be resolved by a different library. Additionally, syscall(2) is deprecated anyway and Apple does not intend to provide proper support for it (or any actual syscalls, for that matter - applications are supposed to link against functions directly without taking the syscall route).

Yes, I know that I could pass ldflags via GOFLAGS, but the documentation doesn't offer any way to specify additional system libraries to link against and even less a way to actually record something like that for automatic usage (i.e., . Enabling external link mode and specifying external linker flags was the only way I found - and even this has to be specified explicitly by users for own code even after go itself was built successfully. I essentially want to tell the go build system to take a specific library and always link to that, without users having to specify it ever again - while bootstrapping go itself, while rebuilding go itself and while building user programs - just like it already does with the system C library/libraries.

[ianlancetaylor]

There is no way to add a required library, and there is no simple way of adding that functionality. As I suggested earlier in #35269 (comment), there is no list of libraries to link against. There is a list of specific symbols and the libraries from which they come.

I'm inclined to close this issue since I don't think the Go team is going to take any action here.

[probonopd]

without a clear gain - especially since Apple officially only supports the current OS release and at most the two most recent previous ones, or something along those lines

Makes a big difference for me personally: The difference is whether my 2007 Mac is a feasible system or a pile of junk. Because Apple stopped supporting it 4 or so macOS releases ago. Please be mindful of our environment and don't add to this kind of forced electronic waste.

And wait, didn't Apple Marketing brag that macOS was fully UNIX 03-compliant? Apparently, Apple obtained UNIX 03 certification with OS X 10.5. So, if Go uses that, wouldn't it have to work? Apple can't just deprecate stuff at will.

Is this certification worth anything or is it just a worthless piece of paper?

[Ionic]

Makes a big difference for me personally: The difference is whether my 2007 Mac is a feasible system or a pile of junk. Because Apple stopped supporting it 4 or so macOS releases ago. Please be mindful of our environment and don't add to this kind of forced electronic waste.

That's not upstream's job, though. I can speak with both upstream and downstream hats on. Supporting legacy, unsupported (by their vendor), insecure operating systems with missing functionality is a pain for upstream developers.

That's why downstream package management systems exist. They take upstream's base and modify it to work on their (and their user's) systems. This is also true for MacPorts, which patches software (like go) for such older systems on a voluntary(!) basis to also support legacy systems, if reasonably possible.

And wait, didn't Apple Marketing brag that macOS was fully UNIX 03-compliant? Apparently, Apple obtained UNIX 03 certification with OS X 10.5. So, if Go uses that, wouldn't it have to work? Apple can't just deprecate stuff at will.

Yes, they went through the process. But neither the syscall function nor the fdopendir function are part of SUSv3. SUS defines functions, but leaves the details (e.g., as syscalls xor library functions) to the implementation.

fdopendir made its debut in SUSv4, so it's not surprising that older OS X systems don't support it.

The only thing Apple did deprecate was internal implementation details - and that's totally fine.

Systems and specifications evolve over time.

There is no way to add a required library, and there is no simple way of adding that functionality.

That's unfortunate, but...

As I suggested earlier in #35269 (comment), there is no list of libraries to link against. There is a list of specific symbols and the libraries from which they come.

I guess I could try to patch this instead and pull in the support library. Would need to test this, but I figure that could be a lot more complicated then it initially sounds and downright impossible.

I'm inclined to close this issue since I don't think the Go team is going to take any action here.

Honestly, sounds fine to me.

[kzc]

go 1.15 and programs built with it run fine on OSX 10.9 Mavericks using the latest macports-legacy-support library and DYLD_INSERT_LIBRARIES: evanw/esbuild#81 (comment). It might work with older OS versions, but I haven't tried.

[probonopd]

Thanks for the heads-up @ksc, this is excellent to know.

Are there any technical reasons that are preventing Go from working this way by default?

I doubt many developers will go through the extra hassle to do extra steps, so a solution that would solve this "out of the box" for everyone would be ideal.

[ianlancetaylor]

@probonopd Working which way by default? I'm not sure I understand what the Go project could do here. (And, of course, bear in mind the earlier discussion on this issue; the Go project does not support macOS 10.9.)

[kencu]

the simplest thing to do is probably to automatically wrap go binaries in a launch script that sets the proper environment variables. We already have a number of workarounds that work this way on MacPorts. If this is reliable enough to do generally, we'll open a PR for it, using one of our other similar workarounds as a template.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)