[vm/ffi] Native asset resolution by the Dart VM for `FfiNative`s

[dcharkes]

Given the following Dart code.

@FfiNative<Int64 Function(Int64, Int64)>('sum', asset: 'mylib/mylib')
external int sum(int a, int b);

Update 2022-10-19: We're taking a different approach. Below is no longer applicable.

The embedder can register a resolver function to look up symbols in assets.

/**
 * FFI native asset C function pointer resolver callback.
 *
 * \param asset The name of the asset.
 * \param symbol The name of the symbol.
 * \param error Returns NULL if lookup is successful, an error message
 *   otherwise. The caller is responsible for calling free() on the error
 *   message.
 *
 * \return Looks up the symbol in the asset and returns its address in memory.
 *         Returns NULL if the asset or symbol doesn't exist.
 */
typedef const void* (*Dart_NativeAssetResolver)(const char* asset,
                                                const char* symbol,
                                                char** error);

/**
 * Sets the callback used to resolve FFI native functions in assets for an
 * isolate group.
 *
 * The resolved functions are expected to be a C function pointer of the
 * correct signature (as specified in the `@FfiNative<NFT>()` function
 * annotation in Dart code).
 *
 * NOTE: This is an experimental feature and might change in the future.
 *
 * \param isolate_group An isolate group.
 * \param resolver A native function resolver.
 */
DART_EXPORT void Dart_SetNativeAssetResolver(
    Dart_IsolateGroup isolate_group,
    Dart_NativeAssetLookup resolver);

The Dart standalone embedder can receive a mapping from asset to path through a flag. (As follow up, this mapping will be provided automatically by packages which want to bundle native assets.)

Internal design doc

[dcharkes]

Backend Implementation

@mraleph suggested to embed the resolution in kernel files and the Dart VM rather than let embedders do the resolution. This way we don't have to reimplement resolution in each embedder. Each embedder needs to provide the mapping as input to the kernel compiler instead. (This makes the first post obsolete.)

internal design doc

Dart API

@mraleph suggested to let assetId's default to library name.

If we combine that with our wish to make @Ffinatives more concise (#50097), and to do it in a non-breaking way, the Dart API would be as follows:

/// Annotation to be used for marking an external function as FFI native.
///
/// Example:
///
/// ```dart template:top
/// @FfiNative<Int64 Function(Int64, Int64)>('FfiNative_Sum', isLeaf:true)
/// external int sum(int a, int b);
/// ```
///
/// Calling such functions will throw an exception if no resolver
/// was set on the library or the resolver failed to resolve the name.
///
/// See `Dart_SetFfiNativeResolver` in `dart_api.h`
///
/// NOTE: This is an experimental feature and may change in the future.
@Since('2.14')
@Deprecated('Use Native instead.')
class FfiNative<T> {
  final String nativeName;
  final bool isLeaf;
  const FfiNative(this.nativeName, {this.isLeaf: false});
}

/// Annotation to be used for marking an external function as native.
///
/// Example:
///
/// ```dart template:top
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Calling such function will try to resolve the [symbol] in
/// 1. the provided [asset],
/// 2. the native resolver set with `Dart_SetFfiNativeResolver` in
///    `dart_api.h`, and
/// 3. the current process.
///
/// Calling such function will throw an exception if the symbol fails to
/// resolve in all cases.
///
/// NOTE: This is an experimental feature and may change in the future.
@Since('2.19')
class Native<T> {
  /// The [symbol] to be resolved.
  ///
  /// If not specified, defaults to the annotated function name.
  ///
  /// Example:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(symbol: 'sum')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above two examples are identical.
  ///
  /// Prefer omitting [symbol].
  final String? symbol;

  /// The [asset] in which [symbol] is resolved.
  ///
  /// If not specified, defaults to [Asset] annotation on the `library`.
  /// If [Asset] on library is omitted, defaults to library name.
  ///
  /// Example (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2 (file `package:a/b.dart`):
  ///
  /// ```dart
  /// @Asset('package:a/a.dart')
  /// library;
  ///
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 3 (file `package:a/b.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(asset: 'package:a/a.dart')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above three examples are identical.
  ///
  /// Prefer using the library name as `Asset` over specifying it.
  /// Prefer specifying [Asset] on `library` over specifying on [Native].
  final String? asset;

  /// Native function does not call back into Dart code or the Dart VM.
  ///
  /// Leaf calls are faster than non-leaf calls.
  final bool isLeaf;

  const Native({
    this.asset,
    this.isLeaf: false,
    this.symbol,
  });
}

/// Annotation to be used for specifying the default asset on a library.
///
/// If no annotation is provided, defaults to library name.
///
/// Example (file `package:a/a.dart`):
///
/// ```dart
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Example 2 (file `package:a/b.dart`):
///
/// ```dart
/// @Asset('package:a/a.dart')
/// library;
///
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// The above two examples are identical.
///
/// Prefer using the library name as `Asset` over specifying it.
@Since('2.19')
class Asset {
  final String asset;

  const Asset(
    this.asset,
  );
}

cc @lrhn @mraleph @mkustermann @mit-mit

Some questions to answer:

1. Do we want the optional asset on Native? If not, we force people to write an extra file if they want to resolve in multiple assets.
2. Do things need different names for annotations or fields?

[mkustermann]

Do we want the optional asset on Native? If not, we force people to write an extra file if they want to resolve in multiple assets.

It's probably beneficial to allow one library to use natives from multiple native libraries.

@lrhn Do you think the name "asset" is too generic here?

[lrhn]

I have absolutely no idea what "asset" means. And that's after reading the documentation. So yes, too generic.

I think an asset is a string. But it can also be a library name (there are no examples of that, and I wouldn't recommend people using library names for anything).
It can be used to resolve a symbol in. Which seems to be a native thing.

I really need the ELI5 version of what's going on 😉.

Either describe what "asset" means in the documentation, or pick a name that describes it more precisely (if one exists).

[dcharkes]

Later, these assets will be coming from the bin/native.dart CLI and be included by Dart Dev (dart run / dart compile).

En example of a dart_app that depends on two packages which provide native assets:

dart_app$ dart run native list
Building package executable... (1.2s)
Built native:native.

- name: mylib_dylib # Can be a library name including `package:`.
  packaging: dynamic
  path:
    path_type: absolute
    uri: /usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/mylib_dylib/native/lib/linux_x64/libmylib_dylib.so
  target: linux_x64
- name: mylib_source # Can be a library name including `package:`.
  packaging: dynamic
  path:
    path_type: absolute
    uri: /usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/dart_app/.dart_tool/native/mylib_source/linux_x64/libmylib_source.so
  target: linux_x64

When doing

/// @Native<Int64 Function(Int64, Int64)>(asset: `mylib_source`)
/// external int sum(int a, int b)

calling sum will result in looking up sum in /usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/dart_app/.dart_tool/native/mylib_source/linux_x64/libmylib_source.so in dart run.

I have no idea how to refer to that in the dart:ffi Natives documentation if we don't land the CLI and dartdev support in the same CL. Unfortunately, the idea was to land the backend logic separately. Dartdev would convert the CLI output to .dart_tools/native_assets.dart:

@pragma('vm:entry-point')
@pragma('vm:native-assets', {
  "linux_x64": {
    "mylib_dylib": [
      "absolute",
      "/usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/mylib_dylib/native/lib/linux_x64/libmylib_dylib.so"
    ],
    "mylib_source": [
      "absolute",
      "/usr/local/google/home/dacoharkes/ffi-samples/dacoharkes/native_lib_distribution/dart_app/.dart_tool/native/mylib_source/linux_x64/libmylib_source.so"
    ]
  }
})
class _NativeAssets {}

but this is an implementation detail that users should never see.

Any suggestions for what to put in the implementation for when dartdev and CLI support have not been landed?

[dcharkes]

I mean, we could add documentation on passing in a library named .dart_tools/native_assets.dart containing the above pragma. But do we want to expose users to that low level VM API?

[lrhn]

So, an asset is a named abstraction over a native library (which is itself a can of worms to define in a consistent cross-platform way). You can refer to the native library by its asset name.

The compiler and/or runtime provides a binding from asset name to native library, which can (and probably does) depend on the target platform. The compiler/runtme can then resolve/lookup identifiers against the native library, and use it to bind external Dart function declarations to native functions as their implementation.

The name can be any string.
Defaults include the current library's file name without the trailing .dart?

• "# Can be a library name including package:." - That's not a "library name". A library name is what comes after library in a library declaration, like library foo.bar.baz;. Do you mean the package URI, the package URI's path or the library's file name (minus extension) only? (Make sure to define it even if the library file name does not end in just .dart, or in .dart at all.)

[dcharkes]

So, an asset is a named abstraction over a native library (which is itself a can of worms to define in a consistent cross-platform way). You can refer to the native library by its asset name.

The compiler and/or runtime provides a binding from asset name to native library, which can (and probably does) depend on the target platform. The compiler/runtme can then resolve/lookup identifiers against the native library, and use it to bind external Dart function declarations to native functions as their implementation.

Correct. You are good with words! 😄

However, we also support looking up symbols in the process or executable. In that case asset maps not really to a native library. So one can refer to a native library, or process, or executable by asset name.

The binding part is correct verbatim.

The name can be any string. Defaults include the current library's file name without the trailing .dart?

My bad, it should be "library uri", which includes package: and .dart (if the file name has that extension).

• "# Can be a library name including package:." - That's not a "library name". A library name is what comes after library in a library declaration, like library foo.bar.baz;. Do you mean the package URI, the package URI's path or the library's file name (minus extension) only? (Make sure to define it even if the library file name does not end in just .dart, or in .dart at all.)

It should be URI indeed. Though that raises the question, what is the uri of a file outside lib/?

What is the difference between a package uri and library uri? Or does "library uri" not exist?

Take two:

/// A [Native] external function is implemented in native code.
///
/// The native external function is resolved in the native library bound to
/// [asset].
///
/// The compiler and/or runtime provides a binding from [asset] name to native
/// library, which depends on the target platform. The compiler/runtme can then
/// resolve/lookup identifiers against the native library, and use it to bind
/// external Dart function declarations to native functions as their implementation.
///
/// Use this class as annotation on `external` functions, to make them resolve
/// to native functions.
///
/// Example:
///
/// ```dart template:top
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Calling such function will try to resolve the [symbol] in
/// 1. the provided [asset],
/// 2. the native resolver set with `Dart_SetFfiNativeResolver` in
///    `dart_api.h`, and
/// 3. the current process.
///
/// Calling such function will throw an exception if the symbol fails to
/// resolve in all cases.
///
/// NOTE: This is an experimental feature and may change in the future.
@Since('2.19')
class Native<T> {
  /// The [symbol] to be resolved.
  ///
  /// If not specified, defaults to the annotated function name.
  ///
  /// Example:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(symbol: 'sum')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above two examples are identical.
  ///
  /// Prefer omitting [symbol].
  final String? symbol;

  /// The [asset] in which [symbol] is resolved.
  ///
  /// If not specified, defaults to [Asset] annotation on the `library`.
  /// If [Asset] on library is omitted, defaults to package uri.
  ///
  /// Example (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2 (file `package:a/b.dart`):
  ///
  /// ```dart
  /// @Asset('package:a/a.dart')
  /// library;
  ///
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 3 (file `package:a/b.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(asset: 'package:a/a.dart')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above three examples are identical.
  ///
  /// Prefer using the package uri as `Asset` over specifying it.
  /// Prefer specifying [Asset] on `library` over specifying on [Native].
  final String? asset;

  /// Native function does not call back into Dart code or the Dart VM.
  ///
  /// Leaf calls are faster than non-leaf calls.
  final bool isLeaf;

  const Native({
    this.asset,
    this.isLeaf: false,
    this.symbol,
  });
}

/// An [Asset] is a named abstraction over a native library.
///
/// The compiler and/or runtime provides a binding from asset name to native
/// library, which depends on the target platform. The compiler/runtme can then
/// resolve/lookup identifiers against the native library, and use it to bind
/// external Dart function declarations to native functions as their implementation.
///
/// Use this class as annotation on a `library` to make all [Native]s in this
/// library use the specified [asset] (unless overwrittten with [Native.asset]).
///
/// If no annotation is provided, defaults to package uri.
///
/// Example (file `package:a/a.dart`):
///
/// ```dart
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Example 2 (file `package:a/b.dart`):
///
/// ```dart
/// @Asset('package:a/a.dart')
/// library;
///
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// The above two examples are identical.
///
/// Prefer using the package uri as `Asset` over specifying it.
@Since('2.19')
class Asset {
  final String asset;

  const Asset(
    this.asset,
  );
}

[lrhn]

It should be URI indeed. Though that raises the question, what is the uri of a file outside lib/?

It's a file:-schemed URI then.

What is the difference between a package uri and library uri? Or does "library uri" not exist?

A "package URI" is a shorthand sometimes used for "URI whose scheme is package".
A "library URI" is (typically, in a Dart context) the URI used to denote a Dart library.

A library URI is usually, but not always, a package URI. The counter examples are the libraries in bin/ or /test, which do not have package URIs denoting them, and are therefore always denoted by file: URIs (or maybe http: URIs, if we serve the tests over HTTP for some reason). Those are rarely referred to except by nearby files, using relative paths.

---

My attempt:

/// Annotation specifying how to bind an external function to native code.
///
/// The annotation applies only to `external` function declarations.
///
/// A [Native]-annotated `external` function is implemented by native code.
/// The implementation is found in the native library denoted by an asset name.
///
/// The compiler and/or runtime provides a binding from asset name to native
/// library, which depends on the target platform.
/// The compiler/runtime can then resolve/lookup symbols (identifiers)
/// against the native library, to find a native function,
/// and bind an `external` Dart function declaration to that native function.
///
/// Use this annotation on `external` functions to specify that they
/// are resolved against an asset, and to, optionally, provide overrides
/// of the default symbol and asset names.
///
/// The type argument to the [Native] annotation must be a function type
/// representing the native function's parameter and return types.
///
/// Example:
///
/// ```dart template:top
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Calling such function will try to resolve the [symbol] in (in that order)
/// 1. the provided or default [asset],
/// 2. the native resolver set with `Dart_SetFfiNativeResolver` in
///    `dart_api.h`, and
/// 3. the current process.
///
/// At least one of those three *must* provide a binding for the symbol,
/// otherwise the method call fails.
///
/// NOTE: This is an experimental feature and may change in the future.
@Since('2.19')
class Native<T> {
  /// The [symbol] to be resolved, if not using the default.
  ///
  /// If not specified, the default symbol used for native function lookup
  /// is the annotated function's name.
  ///
  /// Example:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2:
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(symbol: 'sum')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above two examples are equivalent.
  ///
  /// Prefer omitting the [symbol] when possible.
  final String? symbol;

  /// The asset name in which [symbol] is resolved, if not using the default.
  ///
  /// If no asset name is specified,
  /// the default is to use an asset name specified using
  /// an [Asset] annotation on the current library's `library` declaration,
  /// and if there is no [Asset] annotation on the current library,
  /// the library's URI, as a string is used instead.
  ///
  /// Example (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 2 (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Asset('package:a/a.dart')
  /// library;
  ///
  /// @Native<Int64 Function(Int64, Int64)>()
  /// external int sum(int a, int b);
  /// ```
  ///
  /// Example 3 (file `package:a/a.dart`):
  ///
  /// ```dart
  /// @Native<Int64 Function(Int64, Int64)>(asset: 'package:a/a.dart')
  /// external int sum(int a, int b);
  /// ```
  ///
  /// The above three examples are all equivalent.
  ///
  /// Prefer using the library URI as an asset name over specifying it.
  /// Prefer using an [Asset] on the `library` declaration
  /// over specifying the asset name in a [Native] annotation.
  final String? asset;

  /// Whether the function is a leaf function.
  ///
  /// A leaf function must not run Dart code or call back into the Dart VM.
  ///
  /// Leaf calls are faster than non-leaf calls.
  final bool isLeaf;

  const Native({
    this.asset,
    this.isLeaf: false,
    this.symbol,
  });
}

/// Annotation specifying the default asset name for the current library.
///
/// The annotation applies only to `library` declarations.
///
/// The compiler and/or runtime provides a binding from _asset name_ to native
/// library, which depends on the target platform and architecture.
/// The compiler/runtime can resolve identifiers (symbols)
/// against the native library, looking up native function implementations
/// which are then used as the implementation
/// of `external` Dart function declarations
///
/// If used as annotation on a `library` declaration, all [Native]-annotated
/// external functions in this library will use the specified [asset] name
/// for native function resolution (unless overridden by [Native.asset]).
///
/// If no [Asset] annotation is provided, the current library's URI
/// is the default asset name for [Native]-annotated external functions.
///
/// Example (file `package:a/a.dart`):
///
/// ```dart
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// Example 2 (file `package:a/a.dart`):
///
/// ```dart
/// @Asset('package:a/a.dart')
/// library;
///
/// @Native<Int64 Function(Int64, Int64)>()
/// external int sum(int a, int b);
/// ```
///
/// The above two examples are equivalent.
///
/// Prefer using the library URI as asset name when possible.
@Since('2.19')
class Asset {
  /// The default asset name for [Native] external functions in this library.
  final String asset;

  const Asset(
    this.asset,
  );
}

(@dcharkes: edited typo and punctuation)

[dcharkes]

Thanks @lrhn!

Should we have "(file package:a/b.dart)" instead of "(file package:a/a.dart)" in the examples where the asset is specified in the annotations as 'package:a/a.dart'?