From 43950e286968d96a74e47585408aba06f0478a3d Mon Sep 17 00:00:00 2001 From: Jonas Finnemann Jensen Date: Wed, 24 Jan 2024 15:01:59 +0100 Subject: [PATCH 01/12] A draft at a build-hook specification --- .../doc/build-hook-specification.md | 171 ++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 pkgs/native_assets_builder/doc/build-hook-specification.md diff --git a/pkgs/native_assets_builder/doc/build-hook-specification.md b/pkgs/native_assets_builder/doc/build-hook-specification.md new file mode 100644 index 0000000000..c91aca8a19 --- /dev/null +++ b/pkgs/native_assets_builder/doc/build-hook-specification.md @@ -0,0 +1,171 @@ +Build Hook Specification +======================== + +**DISCLAIMER:** This is really just a native attempt, at writing a specification. + + +## Version 1: Currently implemented behind `--enable-experiment=native-assets` + + +### Concepts + + +#### Asset +A file that is identified by an `assetId`. There may be multiple files with the +same `assetId` with different characteristics such as `target` or `link_mode`. + +#### AssetId +An asset must have an `assetId`. Dart code that uses an asset, references the +asset using the `assetId`. + +A package MUST prefix all assetIds it defines with: `package:/`, this +ensures assets don't conflict between packages. + +Conventionally, an asset referenced from `lib/src/foo.dart` in `package:foo` +has `assetId = 'package:foo/src/foo.dart'`. + +#### Target + +A target specifies operating system and architecture. + +List of `platform` and `architecture` tuples supported by the +Dart and Flutter SDKs to be specified below: + * `linux_x64` + * TODO: enumerate full list of supported platform/architecture tuples. + +**Remark:** It may appear confusing that `link_mode` is not part of the target +(if so, that's probably because it is confusing). + + +### `build.dart` +A package with native assets MUST define a `build.dart` file which accepts +a `--config` option as follows: + +```console +$ dart build.dart --config +``` + +The `--config` option specifies location of the `build_config.yaml` file. + +The `build.dart` file MUST: + * Read the `build_config.yaml` file, + * Build assets using configuration from `build_config.yaml`. + * If `dry_run: true` in `build_config.yaml`, then this may be skipped. + * Write assets into `out_dir` from `build_config.yaml`. + * If `dry_run: true` in `build_config.yaml`, then this may be skipped. + * Filename are unrelated to `assetId`. + * Arbitrary file names are fine, `build_output.yaml` will map `assetId` to files. + * MUST avoid file name `build_output.yaml` (yes, this is poor design) + * Write `build_output.yaml` into `out_dir` (from `build_config.yaml`). + * This maps `assetId` to assets previous written in `out_dir`. + * There may be multiple assets for a given `assetId` depending on + characteristics like `target`, `link_mode`, etc. + * If `dry_run: true` in `build_config.yaml`, the list of assets that would be + generated must this be enumerated. Mapping `assetId`s to non-existing files + is expected. + + +### `build_config.yaml` +Configuration file passed to `build.dart` as the `--config` option. + +```yaml +# Build in dry-run mode. +# +# Running in dry-run mode `/build_output.yaml` must be written, but +# the files it references need not exist. +dry_run: true | false +# Build Mode. +# +# A hint `build.dart` can use to determined which optimizations to enable and +# whether or not to include debug symbols in the format relevant for the asset. +build_mode: release | debug +# Unspecified +dependency_metadata: {} +# Preferred link mode +link_mode_preference: dynamic | static | prefer-dynamic | prefer-static +# Path to output directory where assets should be placed. +# +# This is also where `build_output.yaml` should be written. +# +# Remark: Avoid using the name "build_output.yaml" for an asset file, this is +# forbidden. +out_dir: +# Name of the package that contains the `build.dart` +# +# Remark: This is entirely redundant since this is a config file specified to +# `build.dart`, and the author of `build.dart` probably knows the name of the +# package they are writing. +package_name: my_package_with_native_assets +# Path to root folder for the package that contains `build.dart`. +# +# This is useful if `build.dart` wishes to find source code files embedded in +# its own package and compile them to an asset. +package_root: $PUB_CACHE/hosted/.../my_package_with_native_assets/ +# Target architecture +# +# Combined with `target_os` this specifies the "target" for which assets +# should be built +target_architecture: x64 +# Target operating system +# +# Combined with `target_architecture` this specifies the "target" for which +# assets should be built. +# +# Remark: If this appears confusing, that's because it is. +target_os: linux +# Version of this file. +version: 1.0.0 +``` + +### `build_output.yaml` +Mapping from `assetId`s to files created by `build.dart`. + +```yaml +timestamp: +assets: + - id: + link_mode: dynamic | static | prefer-dynamic | prefer-static + path: + path_type: absolute | relative + uri: + target: + ... +dependencies: + - + ... +metadata: {} +version: 1.0.0 +``` + + +### `.dart_tool/native_assets.yaml` + +This file is internal to the Dart SDK and Flutter SDK. +It produced by `native_asset_builder` (as embedded in the SDKs). +For each `target` it maps from `assetId` to `url` +(with `urlKind` indicating if it's a relative or absolute path). + +```yaml +format-version: [1, 0, 0] +native-assets: + : + : [, ] + ... +``` + +This file is used by the embedded SDK to resolve `assetId`s when running in +development mode or build a deployable application (or standalone executable). + + + + +## Version 2: Proposal for a next iteration. + +### Concepts + +#### Asset +#### AssetId +#### ... + + + From d75f6ab7e754647bbba060307412c9717b1a7ddc Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Thu, 25 Jan 2024 09:31:32 +0100 Subject: [PATCH 02/12] Iterate on the v1 spec --- .../v1.md} | 123 ++++++++++++------ 1 file changed, 84 insertions(+), 39 deletions(-) rename pkgs/native_assets_builder/doc/{build-hook-specification.md => protocol/v1.md} (60%) diff --git a/pkgs/native_assets_builder/doc/build-hook-specification.md b/pkgs/native_assets_builder/doc/protocol/v1.md similarity index 60% rename from pkgs/native_assets_builder/doc/build-hook-specification.md rename to pkgs/native_assets_builder/doc/protocol/v1.md index c91aca8a19..55f7d13128 100644 --- a/pkgs/native_assets_builder/doc/build-hook-specification.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -1,19 +1,16 @@ Build Hook Specification ======================== -**DISCLAIMER:** This is really just a native attempt, at writing a specification. - - -## Version 1: Currently implemented behind `--enable-experiment=native-assets` - +Version 1: As of 2024-01-24 implemented behind `--enable-experiment=native-assets` ### Concepts - #### Asset A file that is identified by an `assetId`. There may be multiple files with the same `assetId` with different characteristics such as `target` or `link_mode`. +Note that in v1 there are _only_ native code assets, no data assets. + #### AssetId An asset must have an `assetId`. Dart code that uses an asset, references the asset using the `assetId`. @@ -33,8 +30,6 @@ Dart and Flutter SDKs to be specified below: * `linux_x64` * TODO: enumerate full list of supported platform/architecture tuples. -**Remark:** It may appear confusing that `link_mode` is not part of the target -(if so, that's probably because it is confusing). ### `build.dart` @@ -55,7 +50,7 @@ The `build.dart` file MUST: * If `dry_run: true` in `build_config.yaml`, then this may be skipped. * Filename are unrelated to `assetId`. * Arbitrary file names are fine, `build_output.yaml` will map `assetId` to files. - * MUST avoid file name `build_output.yaml` (yes, this is poor design) + * MUST avoid file name `build_output.yaml`. * Write `build_output.yaml` into `out_dir` (from `build_config.yaml`). * This maps `assetId` to assets previous written in `out_dir`. * There may be multiple assets for a given `assetId` depending on @@ -74,45 +69,65 @@ Configuration file passed to `build.dart` as the `--config` option. # Running in dry-run mode `/build_output.yaml` must be written, but # the files it references need not exist. dry_run: true | false + # Build Mode. # # A hint `build.dart` can use to determined which optimizations to enable and # whether or not to include debug symbols in the format relevant for the asset. +# +# Not provided on dry runs. build_mode: release | debug -# Unspecified -dependency_metadata: {} + +# Metadata as output from build.dart from direct dependencies. +# +# Not provided on dry runs. +dependency_metadata: + # package name of direct dependency. + some_package_name: + # key value pairs. + some_key: some_value + # Preferred link mode link_mode_preference: dynamic | static | prefer-dynamic | prefer-static + # Path to output directory where assets should be placed. # # This is also where `build_output.yaml` should be written. # # Remark: Avoid using the name "build_output.yaml" for an asset file, this is # forbidden. -out_dir: +out_dir: /absolute/path/to/out_dir/ + # Name of the package that contains the `build.dart` # # Remark: This is entirely redundant since this is a config file specified to # `build.dart`, and the author of `build.dart` probably knows the name of the # package they are writing. package_name: my_package_with_native_assets + # Path to root folder for the package that contains `build.dart`. # # This is useful if `build.dart` wishes to find source code files embedded in # its own package and compile them to an asset. -package_root: $PUB_CACHE/hosted/.../my_package_with_native_assets/ +# +# Note that this will be most likely a path in the pub cache when the package +# with a `build.dart` is a dependency of another package. +package_root: /absolute/path/to/my_package_with_native_assets + # Target architecture # # Combined with `target_os` this specifies the "target" for which assets -# should be built -target_architecture: x64 +# should be built. +# +# Not provided on dry runs. +target_architecture: x64 | ia32 | arm | arm64 | riscv32 | riscv64 + # Target operating system # # Combined with `target_architecture` this specifies the "target" for which # assets should be built. -# -# Remark: If this appears confusing, that's because it is. -target_os: linux +target_os: android | ios | linux | macos | windows + # Version of this file. version: 1.0.0 ``` @@ -121,19 +136,47 @@ version: 1.0.0 Mapping from `assetId`s to files created by `build.dart`. ```yaml -timestamp: +# The list of assets. +# +# In dry runs, must contain assets for each architecture for the requested os. assets: - - id: + - id: 'package:my_package_with_native_assets/src/foo.dart' link_mode: dynamic | static | prefer-dynamic | prefer-static path: - path_type: absolute | relative - uri: - target: + path_type: absolute | system | process | executable + # Only provided for path_type absolute and system. + # + # If path_type absolute: The absolute path to the file name. + # + # If path_type system: The path of the dynamic library as available on + # the target machine's PATH. + uri: /absolute/path/to/outdir/arbitrary_filename.whatever + target: linux_x64 ... + +# The files used by this build. +# +# If any of the files in [dependencies2] are modified after [timestamp], the +# build will be re-run. +# +# Not output on dry runs. dependencies: - - + - /absolute/path/to/my_package_with_native_assets/build.dart ... -metadata: {} + +# The time the build this output is for started. +# +# Must be before any of the files in dependencies are read. +timestamp: 2024-01-02 17:05:35.000 + +# Metadata usable for build.dart of packages directly depending on this package. +# +# Not output in dry runs. +metadata: + # Key value pairs. + some_key: some_value + +# Version of this file. version: 1.0.0 ``` @@ -146,26 +189,28 @@ For each `target` it maps from `assetId` to `url` (with `urlKind` indicating if it's a relative or absolute path). ```yaml +# Version of this file. format-version: [1, 0, 0] + +# A mapping from target and assetId to an asset. native-assets: - : - : [, ] + # A . + linux_x64: + # An . + : 'package:my_package_with_native_assets/src/foo.dart' + # A list of path_type and optionally a path. + - absolute | relative | system | process | executable + - # Only provided for absolute, relative, and system path types. ... ``` This file is used by the embedded SDK to resolve `assetId`s when running in development mode or build a deployable application (or standalone executable). +## Wishes to improve for v2 - - -## Version 2: Proposal for a next iteration. - -### Concepts - -#### Asset -#### AssetId -#### ... - - - +* Multiple asset types. Rename `Asset` to `NativeCodeAsset` and introduce `DataAsset`s. +* The path_types really only make sense for `link_mode: dynamic`. +* Fix the mismatch between `target` for output but `os` and `architecture` for config. + * Should the architecture field of native code assets be optional and be ommitted in dry runs? +* Introduce a `link.dart` protocol with a link input and link output. From 13e9ed69e13233573f614e5a04b674281d1975a4 Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Thu, 25 Jan 2024 12:52:22 +0100 Subject: [PATCH 03/12] Update pkgs/native_assets_builder/doc/protocol/v1.md Co-authored-by: Moritz --- pkgs/native_assets_builder/doc/protocol/v1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index 55f7d13128..11fb9757fb 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -156,7 +156,7 @@ assets: # The files used by this build. # -# If any of the files in [dependencies2] are modified after [timestamp], the +# If any of the files in [dependencies] are modified after [timestamp], the # build will be re-run. # # Not output on dry runs. From 6f599084a32031879cdf19d0f1dc7a277ed5944b Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Thu, 25 Jan 2024 12:52:30 +0100 Subject: [PATCH 04/12] Update pkgs/native_assets_builder/doc/protocol/v1.md Co-authored-by: Moritz --- pkgs/native_assets_builder/doc/protocol/v1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index 11fb9757fb..cf6e411583 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -6,7 +6,7 @@ Version 1: As of 2024-01-24 implemented behind `--enable-experiment=native-asset ### Concepts #### Asset -A file that is identified by an `assetId`. There may be multiple files with the +A reference to a file that is identified by an `assetId`. There may be multiple files with the same `assetId` with different characteristics such as `target` or `link_mode`. Note that in v1 there are _only_ native code assets, no data assets. From 4bc8cbd94889f0395b0ccb9e6d0f2dab420768cf Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Thu, 25 Jan 2024 12:52:39 +0100 Subject: [PATCH 05/12] Update pkgs/native_assets_builder/doc/protocol/v1.md Co-authored-by: Moritz --- pkgs/native_assets_builder/doc/protocol/v1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index cf6e411583..6d0addbe58 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -23,7 +23,7 @@ has `assetId = 'package:foo/src/foo.dart'`. #### Target -A target specifies operating system and architecture. +A target specifies the operating system and architecture of the targeted platform. List of `platform` and `architecture` tuples supported by the Dart and Flutter SDKs to be specified below: From 5e01f816185a22d2ad48e1475be5d435a661cc8a Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Thu, 25 Jan 2024 15:01:40 +0100 Subject: [PATCH 06/12] schema version --- pkgs/native_assets_builder/doc/protocol/v1.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index 6d0addbe58..b73b3269bc 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -128,7 +128,7 @@ target_architecture: x64 | ia32 | arm | arm64 | riscv32 | riscv64 # assets should be built. target_os: android | ios | linux | macos | windows -# Version of this file. +# Schema version of this file. version: 1.0.0 ``` @@ -176,7 +176,7 @@ metadata: # Key value pairs. some_key: some_value -# Version of this file. +# Schema version of this file. version: 1.0.0 ``` @@ -189,7 +189,7 @@ For each `target` it maps from `assetId` to `url` (with `urlKind` indicating if it's a relative or absolute path). ```yaml -# Version of this file. +# Schema version of this file. format-version: [1, 0, 0] # A mapping from target and assetId to an asset. From f745db4532624703690816d60e080b28f1384744 Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Thu, 25 Jan 2024 16:57:18 +0100 Subject: [PATCH 07/12] address more comments --- pkgs/native_assets_builder/doc/protocol/v1.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index b73b3269bc..459b145898 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -10,6 +10,8 @@ A reference to a file that is identified by an `assetId`. There may be multiple same `assetId` with different characteristics such as `target` or `link_mode`. Note that in v1 there are _only_ native code assets, no data assets. +Native code assets are dynamic libraries. +Native data assets will be blobs of data accessible as a byte list in Dart. #### AssetId An asset must have an `assetId`. Dart code that uses an asset, references the @@ -33,8 +35,8 @@ Dart and Flutter SDKs to be specified below: ### `build.dart` -A package with native assets MUST define a `build.dart` file which accepts -a `--config` option as follows: +To bundle native assets with a package, that package must define a `build.dart` +script which accepts a `--config` option as follows: ```console $ dart build.dart --config @@ -61,7 +63,8 @@ The `build.dart` file MUST: ### `build_config.yaml` -Configuration file passed to `build.dart` as the `--config` option. +The Dart and Flutter SDK invoke `build.dart` of all packages in the transitive +dependencies and pass a `build_config.yaml` as the `--config` option. ```yaml # Build in dry-run mode. From be58730e7d3c736cde2f41b7b210c89bfcf204c1 Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Thu, 25 Jan 2024 17:15:16 +0100 Subject: [PATCH 08/12] Add path_type concepts for native_assets.yaml --- pkgs/native_assets_builder/doc/protocol/v1.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index 459b145898..d241785aec 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -186,10 +186,21 @@ version: 1.0.0 ### `.dart_tool/native_assets.yaml` +(This is not part of the build.dart protocol.) + This file is internal to the Dart SDK and Flutter SDK. It produced by `native_asset_builder` (as embedded in the SDKs). -For each `target` it maps from `assetId` to `url` -(with `urlKind` indicating if it's a relative or absolute path). +For each `target` it maps from `assetId` to a path type with an optional path. + +These paths are used to resolve `@Native() external` functions in Dart code. + +The path types in this file are as follows: + +* `absolute` paths are absolute paths on the system where the Dart executable is running. This path type is used when running in JIT mode on a developers' host machine, or in an Android app where the root path is the Android bundle. +* `relative` paths are relative to the kernel or aot snapshot. This path type is used for shipping native assets in a directory together with a kernelsnapshot, aotsnapshot, or standalone executable (dartaotruntime+aotsnapshot). +* `system` paths are expected to resolve on the target machine PATH. +* `process` "paths" have no path, symbols are resolved in the current process. +* `executable` "paths" have no path, symbols are resolved in the current executable. ```yaml # Schema version of this file. @@ -200,7 +211,7 @@ native-assets: # A . linux_x64: # An . - : 'package:my_package_with_native_assets/src/foo.dart' + 'package:my_package_with_native_assets/src/foo.dart': # A list of path_type and optionally a path. - absolute | relative | system | process | executable - # Only provided for absolute, relative, and system path types. From e1e8754b1372ab474a0cbb883e727c8b398501af Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Thu, 25 Jan 2024 17:49:51 +0100 Subject: [PATCH 09/12] Elaborate concepts --- pkgs/native_assets_builder/doc/protocol/v1.md | 37 ++++++++++--------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index d241785aec..6fb58687d2 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -6,33 +6,34 @@ Version 1: As of 2024-01-24 implemented behind `--enable-experiment=native-asset ### Concepts #### Asset -A reference to a file that is identified by an `assetId`. There may be multiple files with the -same `assetId` with different characteristics such as `target` or `link_mode`. -Note that in v1 there are _only_ native code assets, no data assets. -Native code assets are dynamic libraries. -Native data assets will be blobs of data accessible as a byte list in Dart. +An asset is a file together with metadata that tells the Dart and Flutter SDK how to bundle such asset in an application bundle. -#### AssetId -An asset must have an `assetId`. Dart code that uses an asset, references the -asset using the `assetId`. +Currently, all such assets are dynamic libraries. -A package MUST prefix all assetIds it defines with: `package:/`, this -ensures assets don't conflict between packages. +An asset must have an `assetId`. Dart code that uses an asset, references the asset using the `assetId`. -Conventionally, an asset referenced from `lib/src/foo.dart` in `package:foo` -has `assetId = 'package:foo/src/foo.dart'`. +An asset must have a target os and target architecture. When accessing the asset at runtime, the asset for the current os and architecture is accessed. -#### Target +An asset must have a link mode. Currently, the only supported link mode is dynamic. (Static linking of native code will be added later.) -A target specifies the operating system and architecture of the targeted platform. +An asset must specify a path type for dynamic linking. -List of `platform` and `architecture` tuples supported by the -Dart and Flutter SDKs to be specified below: - * `linux_x64` - * TODO: enumerate full list of supported platform/architecture tuples. +* `absolute` paths are absolute paths in the output dir (see below). +* `system` paths are expected to resolve on the target machine PATH. In this case the asset is not a file but only metadata. +* `process` "paths" have no path, symbols are resolved in the current process. In this case the asset is not a file but only metadata. +* `executable` "paths" have no path, symbols are resolved in the current executable. In this case the asset is not a file but only metadata. + +#### AssetId +An asset must have an `assetId`. Dart code that uses an asset, references the asset using the `assetId`. +A package must prefix all assetIds it defines with: `package:/`, this ensures assets don't conflict between packages. +Conventionally, an asset referenced from `lib/src/foo.dart` in `package:foo` has `assetId = 'package:foo/src/foo.dart'`. + +#### Target os and architecture + +A target specifies the operating system and architecture of the targeted platform. ### `build.dart` To bundle native assets with a package, that package must define a `build.dart` From 7fb1fce1cba9119c70810c7394eb510c1e550420 Mon Sep 17 00:00:00 2001 From: Daco Harkes Date: Wed, 31 Jan 2024 11:35:02 +0100 Subject: [PATCH 10/12] Address a bunch of comments --- pkgs/native_assets_builder/doc/protocol/v1.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index 6fb58687d2..1a940193f4 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -19,10 +19,10 @@ An asset must have a link mode. Currently, the only supported link mode is dynam An asset must specify a path type for dynamic linking. -* `absolute` paths are absolute paths in the output dir (see below). -* `system` paths are expected to resolve on the target machine PATH. In this case the asset is not a file but only metadata. -* `process` "paths" have no path, symbols are resolved in the current process. In this case the asset is not a file but only metadata. -* `executable` "paths" have no path, symbols are resolved in the current executable. In this case the asset is not a file but only metadata. +* An `absolute` path is an absolute path in the output dir (see below). +* A `system` path is expected to resolve on the target machine PATH. In this case the asset is not a file but only metadata. +* A `process` "path" has no path, symbols are resolved in the current process. In this case the asset is not a file but only metadata. +* An `executable` "path" has no path, symbols are resolved in the current executable. In this case the asset is not a file but only metadata. #### AssetId An asset must have an `assetId`. Dart code that uses an asset, references the asset using the `assetId`. @@ -37,30 +37,30 @@ A target specifies the operating system and architecture of the targeted platfor ### `build.dart` To bundle native assets with a package, that package must define a `build.dart` -script which accepts a `--config` option as follows: +script which accepts a `--config ` command line option as follows: ```console $ dart build.dart --config ``` -The `--config` option specifies location of the `build_config.yaml` file. +The `--config` option specifies the location of the `build_config.yaml` file as an absolute path. The `build.dart` file MUST: * Read the `build_config.yaml` file, - * Build assets using configuration from `build_config.yaml`. - * If `dry_run: true` in `build_config.yaml`, then this may be skipped. - * Write assets into `out_dir` from `build_config.yaml`. - * If `dry_run: true` in `build_config.yaml`, then this may be skipped. - * Filename are unrelated to `assetId`. + * Build assets using the configuration from `build_config.yaml`. + * If the boolean parameter `dry_run` is set to true in the build configuration, then this may be skipped. + * Write asset files into output dir that was provided by `build_config.yaml`. + * If the boolean parameter `dry_run` is set to true in the build configuration, then this may be skipped. + * The file name inside `out_dir` doesn't really matter, because in Dart code you'll always access things by `assetId`. So in some sense the file name is irrelevant. * Arbitrary file names are fine, `build_output.yaml` will map `assetId` to files. * MUST avoid file name `build_output.yaml`. * Write `build_output.yaml` into `out_dir` (from `build_config.yaml`). - * This maps `assetId` to assets previous written in `out_dir`. + * This maps the `assetId`s to assets previously written into `out_dir`. * There may be multiple assets for a given `assetId` depending on characteristics like `target`, `link_mode`, etc. - * If `dry_run: true` in `build_config.yaml`, the list of assets that would be - generated must this be enumerated. Mapping `assetId`s to non-existing files - is expected. + * If the boolean parameter `dry_run` is set to true in the build configuration, + the list of assets must be output, but the asset files must not be written. + The asset file urls are expected to refer to non-existing files. ### `build_config.yaml` From ac5a7da8eb64eb8c45fc8b21d34385c7eb02151f Mon Sep 17 00:00:00 2001 From: Moritz Date: Fri, 2 Feb 2024 14:15:28 +0100 Subject: [PATCH 11/12] Suggestions on reformulating --- pkgs/native_assets_builder/doc/protocol/v1.md | 96 +++++++++---------- 1 file changed, 46 insertions(+), 50 deletions(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index 1a940193f4..ea78b74941 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -7,65 +7,33 @@ Version 1: As of 2024-01-24 implemented behind `--enable-experiment=native-asset #### Asset -An asset is a file together with metadata that tells the Dart and Flutter SDK how to bundle such asset in an application bundle. +An asset is data which is accessible from a Dart or Flutter application. To retrieve an asset at runtime, we associate with it a string as a unique identifier, the `assetId`. There are several example types of assets: +* Assets which designate symbols present in the target system, process, or executable. They are identified by their name. +* Dynamic libraries bundled into the application. -Currently, all such assets are dynamic libraries. +An application is compiled to run on a certain target OS and architecture. If different targets require different assets, the package developer must specify which asset to bundle for which target. -An asset must have an `assetId`. Dart code that uses an asset, references the asset using the `assetId`. +An asset has different ways of being accessible in the final application. It is either brought in "manually" by having the package developer specify a path of the asset on the current system, it can be part of the Dart or Flutter SDK, or it can be already present in the target system. If the asset is bundled "manually", the Dart or Flutter SDK will take care of copying the asset from its specified location on the current system into the application bundle. -An asset must have a target os and target architecture. When accessing the asset at runtime, the asset for the current os and architecture is accessed. - -An asset must have a link mode. Currently, the only supported link mode is dynamic. (Static linking of native code will be added later.) - -An asset must specify a path type for dynamic linking. - -* An `absolute` path is an absolute path in the output dir (see below). -* A `system` path is expected to resolve on the target machine PATH. In this case the asset is not a file but only metadata. -* A `process` "path" has no path, symbols are resolved in the current process. In this case the asset is not a file but only metadata. -* An `executable` "path" has no path, symbols are resolved in the current executable. In this case the asset is not a file but only metadata. +Assets are also called "native assets" to differentiate them from the Dart code also bundled with an application. #### AssetId -An asset must have an `assetId`. Dart code that uses an asset, references the asset using the `assetId`. - -A package must prefix all assetIds it defines with: `package:/`, this ensures assets don't conflict between packages. - -Conventionally, an asset referenced from `lib/src/foo.dart` in `package:foo` has `assetId = 'package:foo/src/foo.dart'`. +An asset must have a string identifier called `assetId`. Dart code that uses an asset, references the asset using this `assetId`. -#### Target os and architecture +A package must prefix all `assetId`s it defines with: `package:/`, `` being the current package's name. This ensures assets don't conflict between packages. -A target specifies the operating system and architecture of the targeted platform. - -### `build.dart` -To bundle native assets with a package, that package must define a `build.dart` -script which accepts a `--config ` command line option as follows: - -```console -$ dart build.dart --config -``` +Additionally, the convention is that an asset referenced from `lib/src/foo.dart` in `package:foo` has the `assetId` `'package:foo/src/foo.dart'`. -The `--config` option specifies the location of the `build_config.yaml` file as an absolute path. +#### Target -The `build.dart` file MUST: - * Read the `build_config.yaml` file, - * Build assets using the configuration from `build_config.yaml`. - * If the boolean parameter `dry_run` is set to true in the build configuration, then this may be skipped. - * Write asset files into output dir that was provided by `build_config.yaml`. - * If the boolean parameter `dry_run` is set to true in the build configuration, then this may be skipped. - * The file name inside `out_dir` doesn't really matter, because in Dart code you'll always access things by `assetId`. So in some sense the file name is irrelevant. - * Arbitrary file names are fine, `build_output.yaml` will map `assetId` to files. - * MUST avoid file name `build_output.yaml`. - * Write `build_output.yaml` into `out_dir` (from `build_config.yaml`). - * This maps the `assetId`s to assets previously written into `out_dir`. - * There may be multiple assets for a given `assetId` depending on - characteristics like `target`, `link_mode`, etc. - * If the boolean parameter `dry_run` is set to true in the build configuration, - the list of assets must be output, but the asset files must not be written. - The asset file urls are expected to refer to non-existing files. +A target specifies the operating system and architecture of the targeted platform of the application. +#### `build.dart` +If there are assets in a package which have to be "manually" bundled, i.e. they are not shipped as part of Dart or Flutter and are also not present on the target system by default, that package must define a `build.dart` script specifying these assets. This script takes as input information such as the target OS and architecture in a `build_config.yaml` file, and returns as output a list of assets to be bundled in a `build_output.yaml` file. ### `build_config.yaml` -The Dart and Flutter SDK invoke `build.dart` of all packages in the transitive -dependencies and pass a `build_config.yaml` as the `--config` option. + +This is the structure of the input to a `build.dart` script. ```yaml # Build in dry-run mode. @@ -137,7 +105,8 @@ version: 1.0.0 ``` ### `build_output.yaml` -Mapping from `assetId`s to files created by `build.dart`. + +This file is the output from running a `build.dart` script, and contains the list of assets to be bundled into the application. ```yaml # The list of assets. @@ -150,7 +119,8 @@ assets: path_type: absolute | system | process | executable # Only provided for path_type absolute and system. # - # If path_type absolute: The absolute path to the file name. + # If path_type absolute: The absolute path to the file name on the + # current machine. # # If path_type system: The path of the dynamic library as available on # the target machine's PATH. @@ -184,10 +154,36 @@ metadata: version: 1.0.0 ``` +### Running `build.dart` + +The input is passed as a absolute path to a YAML file `build_config.yaml` encoding the input information as follows: + +```console +$ dart build.dart --config +``` + +The Dart and Flutter SDK invoke `build.dart` of all packages in the transitive dependencies and pass a `build_config.yaml` as the `--config` option. + +If not in `dry_run` mode, the `build.dart` file MUST: + * Read the `build_config.yaml` file, + * Create assets using the configuration from `build_config.yaml`, if not already existing. + * Write asset files into the directory with path `out_dir` that was provided by `build_config.yaml`. + * MUST avoid file name `build_output.yaml`. + * Write `build_output.yaml` into the directory at `out_dir`. + * This maps the `assetId`s to assets previously written into `out_dir`. + * There may be multiple assets for a given `assetId` depending on + characteristics like `target`, `link_mode`, etc. + * If the boolean parameter `dry_run` is set to true in the build configuration, + the list of assets must be output, but the asset files must not be written. + The asset file urls are expected to refer to non-existing files. + +Notes: + * The file name inside `out_dir` is irrelevant, because in Dart code the asset will only be referenced by its `assetId`. + * If the boolean parameter `dry_run` is set to true in the build configuration, then the actual creation and writing of assets can be skipped, as long as the assets are still listed in the `build_output.yaml`. ### `.dart_tool/native_assets.yaml` -(This is not part of the build.dart protocol.) +**This is not part of the build.dart protocol.** This file is internal to the Dart SDK and Flutter SDK. It produced by `native_asset_builder` (as embedded in the SDKs). From 746a74db71ca0013a519d652a976e7bdf8f23d2b Mon Sep 17 00:00:00 2001 From: Moritz Date: Fri, 9 Feb 2024 14:34:52 +0100 Subject: [PATCH 12/12] Apply change request --- pkgs/native_assets_builder/doc/protocol/v1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkgs/native_assets_builder/doc/protocol/v1.md b/pkgs/native_assets_builder/doc/protocol/v1.md index ea78b74941..d75a4cfeed 100644 --- a/pkgs/native_assets_builder/doc/protocol/v1.md +++ b/pkgs/native_assets_builder/doc/protocol/v1.md @@ -29,7 +29,7 @@ Additionally, the convention is that an asset referenced from `lib/src/foo.dart` A target specifies the operating system and architecture of the targeted platform of the application. #### `build.dart` -If there are assets in a package which have to be "manually" bundled, i.e. they are not shipped as part of Dart or Flutter and are also not present on the target system by default, that package must define a `build.dart` script specifying these assets. This script takes as input information such as the target OS and architecture in a `build_config.yaml` file, and returns as output a list of assets to be bundled in a `build_output.yaml` file. +Any package which needs to access assets at runtime must define a `build.dart` script to specify these assets. If there are assets in a package which are not shipped as part of Dart or Flutter and also not present on the target system by default, those assets will be bundled into the final application by the Dart or Flutter SDK automatically. The `build.dart` script takes as input information such as the target OS and architecture in a `build_config.yaml` file, and returns as output a list of assets which can be accessed at runtime in a `build_output.yaml` file. ### `build_config.yaml`