JNA cannot load system library on Mac with OpenJDK 14

[dmitry-timofeev]

JNA cannot load a system library on Mac with OpenJDK 14 (but works fine with 11 and 13).

The library is installed via Homebrew, and is accessible from /usr/local/lib:

ls -lah /usr/local/lib/libsodium.dylib 
lrwxr-xr-x  1 user  admin    48B Nov 28 14:30 /usr/local/lib/libsodium.dylib -> ../Cellar/libsodium/1.0.18_1/lib/libsodium.dylib

Logs of JNA load attempts:

On 14 (failing):

???. 06, 2020 5:46:27 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from jdk.internal.loader.ClassLoaders$AppClassLoader@55054057 for /com/sun/jna/darwin/libjnidispatch.jnilib
???. 06, 2020 5:46:27 PM com.sun.jna.Native extractFromResourcePath
INFO: Found library resource at jar:file:/Users/user/.m2/repository/net/java/dev/jna/jna/5.5.0/jna-5.5.0.jar!/com/sun/jna/darwin/libjnidispatch.jnilib
???. 06, 2020 5:46:27 PM com.sun.jna.Native extractFromResourcePath
INFO: Extracting library to /Users/user/Library/Caches/JNA/temp/jna3004104893156453390.tmp
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for library 'sodium'
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding paths from jna.library.path: null
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying libsodium.dylib
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Loading failed with message: dlopen(libsodium.dylib, 9): image not found
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding system paths: [/usr/lib, /usr/lib]
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying libsodium.dylib
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Loading failed with message: dlopen(libsodium.dylib, 9): image not found
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for matching frameworks
???. 06, 2020 5:46:27 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from jdk.internal.loader.ClassLoaders$AppClassLoader@55054057 for sodium
???. 06, 2020 5:46:27 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Loading failed with message: Native library (darwin/libsodium.dylib) not found in resource path (/Users/user/Documents/jna-lib-loading-test/target/test-classes:/Users/user/Documents/jna-lib-loading-test/target/classes:/Users/user/.m2/repository/net/java/dev/jna/jna/5.5.0/jna-5.5.0.jar:/Users/user/.m2/repository/org/junit/jupiter/junit-jupiter/5.6.1/junit-jupiter-5.6.1.jar:/Users/user/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.6.1/junit-jupiter-api-5.6.1.jar:/Users/user/.m2/repository/org/apiguardian/apiguardian-api/1.1.0/apiguardian-api-1.1.0.jar:/Users/user/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar:/Users/user/.m2/repository/org/junit/platform/junit-platform-commons/1.6.1/junit-platform-commons-1.6.1.jar:/Users/user/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.6.1/junit-jupiter-params-5.6.1.jar:/Users/user/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.6.1/junit-jupiter-engine-5.6.1.jar:/Users/user/.m2/repository/org/junit/platform/junit-platform-engine/1.6.1/junit-platform-engine-1.6.1.jar:)
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.13 s <<< FAILURE! - in org.example.SodiumTest
[ERROR] testInit  Time elapsed: 0.12 s  <<< ERROR!
java.lang.UnsatisfiedLinkError: 
Unable to load library 'sodium':
dlopen(libsodium.dylib, 9): image not found
dlopen(libsodium.dylib, 9): image not found
Native library (darwin/libsodium.dylib) not found in resource path (/Users/user/Documents/jna-lib-loading-test/target/test-classes:/Users/user/Documents/jna-lib-loading-test/target/classes:/Users/user/.m2/repository/net/java/dev/jna/jna/5.5.0/jna-5.5.0.jar:/Users/user/.m2/repository/org/junit/jupiter/junit-jupiter/5.6.1/junit-jupiter-5.6.1.jar:/Users/user/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.6.1/junit-jupiter-api-5.6.1.jar:/Users/user/.m2/repository/org/apiguardian/apiguardian-api/1.1.0/apiguardian-api-1.1.0.jar:/Users/user/.m2/repository/org/opentest4j/opentest4j/1.2.0/opentest4j-1.2.0.jar:/Users/user/.m2/repository/org/junit/platform/junit-platform-commons/1.6.1/junit-platform-commons-1.6.1.jar:/Users/user/.m2/repository/org/junit/jupiter/junit-jupiter-params/5.6.1/junit-jupiter-params-5.6.1.jar:/Users/user/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.6.1/junit-jupiter-engine-5.6.1.jar:/Users/user/.m2/repository/org/junit/platform/junit-platform-engine/1.6.1/junit-platform-engine-1.6.1.jar:)
        at org.example.SodiumTest.testInit(SodiumTest.java:12)

On 11/13 (passing):

???. 06, 2020 5:39:47 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27 for /com/sun/jna/darwin/libjnidispatch.jnilib
???. 06, 2020 5:39:47 PM com.sun.jna.Native extractFromResourcePath
INFO: Found library resource at jar:file:/Users/user/.m2/repository/net/java/dev/jna/jna/5.5.0/jna-5.5.0.jar!/com/sun/jna/darwin/libjnidispatch.jnilib
???. 06, 2020 5:39:47 PM com.sun.jna.Native extractFromResourcePath
INFO: Extracting library to /Users/user/Library/Caches/JNA/temp/jna5613620695150474271.tmp
???. 06, 2020 5:39:47 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for library 'sodium'
???. 06, 2020 5:39:47 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding paths from jna.library.path: null
???. 06, 2020 5:39:47 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying libsodium.dylib
???. 06, 2020 5:39:47 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Found library 'sodium' at libsodium.dylib

Provide complete information about the problem

1. Version of JNA and related jars: 5.5.0
2. Version and vendor of the java virtual machine:

java -version
openjdk version "14" 2020-03-17
OpenJDK Runtime Environment (build 14+36-1461)
OpenJDK 64-Bit Server VM (build 14+36-1461, mixed mode, sharing)

3. Operating system: macOS
4. System architecture (CPU type, bitness of the JVM): x64
5. Steps to reproduce:

• Clone https://github.com/dmitry-timofeev/jna-load-test
• Install libsodium via homebrew: brew install libsodium
• Run mvn test with OpenJDK 14

[matthiasblaesing]

I don't see a JNA issue - JNA itself loads its own dispatch library first and it fails when loading libsodium. There is a reason the issue templates asks you to take your questions to the mailing list, then more people can help.

I reduced the problem to the minimum and just try to load the native sodium library:

Loading `sodium` works on JDK 8

Matthiass-MacBook-Pro:Home matthias$ jrunscript -cp /Users/matthias/Downloads/jna-5.5.0.jar -Djna.debug_load=true  -e 'com.sun.jna.NativeLibrary.getInstance("sodium")' 
Apr 06, 2020 12:37:23 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from java.net.URLClassLoader@42a57993 for /com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:37:23 PM com.sun.jna.Native extractFromResourcePath
INFO: Found library resource at jar:file:/Users/matthias/Downloads/jna-5.5.0.jar!/com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:37:23 PM com.sun.jna.Native extractFromResourcePath
INFO: Extracting library to /Users/matthias/Library/Caches/JNA/temp/jna725419789810539695.tmp
Apr 06, 2020 12:37:23 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for library 'sodium'
Apr 06, 2020 12:37:23 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding paths from jna.library.path: null
Apr 06, 2020 12:37:23 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying libsodium.dylib
Apr 06, 2020 12:37:23 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Found library 'sodium' at libsodium.dylib
Matthiass-MacBook-Pro:Home matthias$ 

Loading `sodium` fails on JDK 14

Matthiass-MacBook-Pro:Home matthias$ bin/jrunscript -cp /Users/matthias/Downloads/jna-5.5.0.jar -Djna.debug_load=true  -e 'com.sun.jna.NativeLibrary.getInstance("sodium")' 
Warning: Nashorn engine is planned to be removed from a future JDK release
Apr 06, 2020 12:37:58 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from java.net.URLClassLoader@7adf9f5f for /com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:37:58 PM com.sun.jna.Native extractFromResourcePath
INFO: Found library resource at jar:file:/Users/matthias/Downloads/jna-5.5.0.jar!/com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:37:58 PM com.sun.jna.Native extractFromResourcePath
INFO: Extracting library to /Users/matthias/Library/Caches/JNA/temp/jna16791595602557599137.tmp
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for library 'sodium'
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding paths from jna.library.path: null
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying libsodium.dylib
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Loading failed with message: dlopen(libsodium.dylib, 9): image not found
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding system paths: [/usr/lib, /usr/lib]
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying libsodium.dylib
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Loading failed with message: dlopen(libsodium.dylib, 9): image not found
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for matching frameworks
Apr 06, 2020 12:37:58 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from java.net.URLClassLoader@7adf9f5f for sodium
Apr 06, 2020 12:37:58 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Loading failed with message: Native library (darwin/libsodium.dylib) not found in resource path ([file:/Users/matthias/Downloads/jna-5.5.0.jar])
Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'sodium':
dlopen(libsodium.dylib, 9): image not found
dlopen(libsodium.dylib, 9): image not found
Native library (darwin/libsodium.dylib) not found in resource path ([file:/Users/matthias/Downloads/jna-5.5.0.jar])
	at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
	at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
	at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:397)
	at jdk.scripting.nashorn.scripts/jdk.nashorn.internal.scripts.Script$1$\^string\_/0x0000000800c5a840.:program(<string>:1)
	at jdk.scripting.nashorn/jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:655)
	at jdk.scripting.nashorn/jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:513)
	at jdk.scripting.nashorn/jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:527)
	at jdk.scripting.nashorn/jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:456)
	at jdk.scripting.nashorn/jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:413)
	at jdk.scripting.nashorn/jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:409)
	at jdk.scripting.nashorn/jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:162)
	at java.scripting/javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
	at java.scripting/com.sun.tools.script.shell.Main.evaluateString(Main.java:302)
	at java.scripting/com.sun.tools.script.shell.Main.evaluateString(Main.java:323)
	at java.scripting/com.sun.tools.script.shell.Main$3.run(Main.java:221)
	at java.scripting/com.sun.tools.script.shell.Main.main(Main.java:48)
	Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libsodium.dylib, 9): image not found
		at com.sun.jna.Native.open(Native Method)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:191)
		... 15 more
	Suppressed: java.lang.UnsatisfiedLinkError: dlopen(libsodium.dylib, 9): image not found
		at com.sun.jna.Native.open(Native Method)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:204)
		... 15 more
	Suppressed: java.io.IOException: Native library (darwin/libsodium.dylib) not found in resource path ([file:/Users/matthias/Downloads/jna-5.5.0.jar])
		at com.sun.jna.Native.extractFromResourcePath(Native.java:1095)
		at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:276)
		... 15 more
Matthiass-MacBook-Pro:Home matthias$ 

Loading `/usr/local/lib/libsodium.dylib` works on JDK 8

Matthiass-MacBook-Pro:Home matthias$ jrunscript -cp /Users/matthias/Downloads/jna-5.5.0.jar -Djna.debug_load=true  -e 'com.sun.jna.NativeLibrary.getInstance("/usr/local/lib/libsodium.dylib")' 
Apr 06, 2020 12:36:12 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from java.net.URLClassLoader@42a57993 for /com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:36:12 PM com.sun.jna.Native extractFromResourcePath
INFO: Found library resource at jar:file:/Users/matthias/Downloads/jna-5.5.0.jar!/com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:36:12 PM com.sun.jna.Native extractFromResourcePath
INFO: Extracting library to /Users/matthias/Library/Caches/JNA/temp/jna7784542883159459561.tmp
Apr 06, 2020 12:36:12 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for library '/usr/local/lib/libsodium.dylib'
Apr 06, 2020 12:36:12 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding paths from jna.library.path: null
Apr 06, 2020 12:36:12 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying /usr/local/lib/libsodium.dylib
Apr 06, 2020 12:36:12 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Found library '/usr/local/lib/libsodium.dylib' at /usr/local/lib/libsodium.dylib
Matthiass-MacBook-Pro:Home matthias$

Loading `/usr/local/lib/libsodium.dylib` works on JDK 14

Matthiass-MacBook-Pro:Home matthias$ bin/jrunscript -cp /Users/matthias/Downloads/jna-5.5.0.jar -Djna.debug_load=true  -e 'com.sun.jna.NativeLibrary.getInstance("/usr/local/lib/libsodium.dylib")' 
Warning: Nashorn engine is planned to be removed from a future JDK release
Apr 06, 2020 12:36:30 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from java.net.URLClassLoader@7adf9f5f for /com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:36:30 PM com.sun.jna.Native extractFromResourcePath
INFO: Found library resource at jar:file:/Users/matthias/Downloads/jna-5.5.0.jar!/com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:36:30 PM com.sun.jna.Native extractFromResourcePath
INFO: Extracting library to /Users/matthias/Library/Caches/JNA/temp/jna15131522222744173067.tmp
Apr 06, 2020 12:36:30 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for library '/usr/local/lib/libsodium.dylib'
Apr 06, 2020 12:36:30 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding paths from jna.library.path: null
Apr 06, 2020 12:36:30 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying /usr/local/lib/libsodium.dylib
Apr 06, 2020 12:36:30 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Found library '/usr/local/lib/libsodium.dylib' at /usr/local/lib/libsodium.dylib
Matthiass-MacBook-Pro:Home matthias$ 

So the sodium library can be loaded if the full path is specified. My gut feeling is, that something in the java launcher was changed (maybe searching libraries from /usr/local/lib was dropped).

This is outside the scope of JNA and the search patch for libaries needs to be setup by the users:

Loading `sodium` on JDK14 works when `jna.library.path` is correctly setup

Matthiass-MacBook-Pro:~ matthias$ ./Downloads/jdk-14.jdk/Contents/Home/bin/jrunscript -cp /Users/matthias/Downloads/jna-5.5.0.jar -Djna.debug_load=true -Djna.library.path=/usr/local/lib  -e 'com.sun.jna.NativeLibrary.getInstance("sodium")'
Warning: Nashorn engine is planned to be removed from a future JDK release
Apr 06, 2020 12:58:51 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from java.net.URLClassLoader@7adf9f5f for /com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:58:51 PM com.sun.jna.Native extractFromResourcePath
INFO: Found library resource at jar:file:/Users/matthias/Downloads/jna-5.5.0.jar!/com/sun/jna/darwin/libjnidispatch.jnilib
Apr 06, 2020 12:58:51 PM com.sun.jna.Native extractFromResourcePath
INFO: Extracting library to /Users/matthias/Library/Caches/JNA/temp/jna17138539137708730107.tmp
Apr 06, 2020 12:58:51 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Looking for library 'sodium'
Apr 06, 2020 12:58:51 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Adding paths from jna.library.path: /usr/local/lib
Apr 06, 2020 12:58:51 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Trying /usr/local/lib/libsodium.dylib
Apr 06, 2020 12:58:51 PM com.sun.jna.NativeLibrary loadLibrary
INFO: Found library 'sodium' at /usr/local/lib/libsodium.dylib
Matthiass-MacBook-Pro:~ matthias$ 

[dmitry-timofeev]

Hi @matthiasblaesing , thank you for taking a look at the issue! It appeared as a JNA issue, as library loading by its short name works on all the previous JVM versions but not on 14, and is performed by JNA code. Do I understand correctly that the mechanism of library location detection is environment-specific and hence not supported?

---

Also checked the system properties of JVM 11 and 14, and there are no differences in paths.

[matthiasblaesing]

@dmitry-timofeev JNA has a multiple levels/options how to load a native library. The interesting code is at:

https://github.com/java-native-access/jna/blob/master/src/com/sun/jna/NativeLibrary.java#L153

The code searches for the native library - if it is found, it will be loaded by Native#open - that is a native call to:

https://github.com/java-native-access/jna/blob/master/native/dispatch.c#L2216

That function calls either LoadLibraryExW/GetModuleHandleW (Windows) or dlopen (all other platforms). So that behaviour is platform dependend.

If loading via the normal path fails, JNA will try to extract the library from the archive and/or finally load the library via System#loadLibrary. That latter is what I assume behaves differently on JDK14.

I hope that clears it a bit more.

[dmitry-timofeev]

Thanks for clarifications!

I stepped in the debugger through the loading procedure to see the difference, and it appears that on JDK 11 the very first Native#open succeeds (at line 191), with an empty searchPath and libraryPath = "libsodium.dylib". That is, on JDK 11 the native dlopen invocation works without any extra paths, that are tried by JNA subsequently. With the same context, this invocation fails on Java 14.

I noticed the following difference between JDK 11 and JDK 14 distribution: JDK 14 executable files have extended attributes set. E.g., here are ones for java:

dmitry-mac:bin user$ xattr -l java
com.apple.quarantine: 0181;5e746b66;Homebrew\x20Cask;F0450437-3DAE-4434-BBC5-E6C02C5D001C

I haven't dug deeper, but it seems the com.apple.quarantine attribute affects the dlopen behaviour.

[tmair-red]

@dmitry-timofeev We also had a similar issue and it seems that the semantics of dlopen change when running with hardened runtime support enabled (codesigned with entitlements).

From the dlopen manpage:

Note: If the main executable is a set[ug]id binary or codesigned with entitlements, then all environment variables are ignored, and only a full path can beused.

So the difference in your tests was probably running with hardened runtime enabled or disabled.