Build hook spec (#955)

To be followed up by a v2.md

Author: dcharkes

pkgs/native_assets_builder/doc/protocol/v1.md

@@ -0,0 +1,227 @@
+Build Hook Specification
+========================
+
+Version 1: As of 2024-01-24 implemented behind `--enable-experiment=native-assets`
+
+### Concepts
+
+#### Asset
+
+An asset is data which is accessible from a Dart or Flutter application. To retrieve an asset at runtime, we associate with it a string as a unique identifier, the `assetId`. There are several example types of assets:
+* Assets which designate symbols present in the target system, process, or executable. They are identified by their name.
+* Dynamic libraries bundled into the application.
+
+An application is compiled to run on a certain target OS and architecture. If different targets require different assets, the package developer must specify which asset to bundle for which target.
+
+An asset has different ways of being accessible in the final application. It is either brought in "manually" by having the package developer specify a path of the asset on the current system, it can be part of the Dart or Flutter SDK, or it can be already present in the target system. If the asset is bundled "manually", the Dart or Flutter SDK will take care of copying the asset from its specified location on the current system into the application bundle.
+
+Assets are also called "native assets" to differentiate them from the Dart code also bundled with an application.
+
+#### AssetId
+An asset must have a string identifier called `assetId`. Dart code that uses an asset, references the asset using this `assetId`.
+
+A package must prefix all `assetId`s it defines with: `package:<package>/`, `<package>` being the current package's name. This ensures assets don't conflict between packages.
+
+Additionally, the convention is that an asset referenced from `lib/src/foo.dart` in `package:foo` has the `assetId` `'package:foo/src/foo.dart'`.
+
+#### Target
+
+A target specifies the operating system and architecture of the targeted platform of the application.
+
+#### `build.dart`
+Any package which needs to access assets at runtime must define a `build.dart` script to specify these assets. If there are assets in a package which are not shipped as part of Dart or Flutter and also not present on the target system by default, those assets will be bundled into the final application by the Dart or Flutter SDK automatically. The `build.dart` script takes as input information such as the target OS and architecture in a `build_config.yaml` file, and returns as output a list of assets which can be accessed at runtime in a `build_output.yaml` file. 
+
+### `build_config.yaml`
+
+This is the structure of the input to a `build.dart` script.
+
+```yaml
+# Build in dry-run mode.
+#
+# Running in dry-run mode `<out_dir>/build_output.yaml` must be written, but
+# the files it references need not exist.
+dry_run: true | false
+
+# Build Mode.
+#
+# A hint `build.dart` can use to determined which optimizations to enable and
+# whether or not to include debug symbols in the format relevant for the asset.
+#
+# Not provided on dry runs.
+build_mode: release | debug
+
+# Metadata as output from build.dart from direct dependencies.
+#
+# Not provided on dry runs.
+dependency_metadata:
+  # package name of direct dependency.
+  some_package_name:
+    # key value pairs.
+    some_key: some_value
+
+# Preferred link mode
+link_mode_preference: dynamic | static | prefer-dynamic | prefer-static
+
+# Path to output directory where assets should be placed.
+#
+# This is also where `build_output.yaml` should be written.
+#
+# Remark: Avoid using the name "build_output.yaml" for an asset file, this is
+#         forbidden.
+out_dir: /absolute/path/to/out_dir/
+
+# Name of the package that contains the `build.dart`
+#
+# Remark: This is entirely redundant since this is a config file specified to
+# `build.dart`, and the author of `build.dart` probably knows the name of the
+# package they are writing.
+package_name: my_package_with_native_assets
+
+# Path to root folder for the package that contains `build.dart`.
+#
+# This is useful if `build.dart` wishes to find source code files embedded in
+# its own package and compile them to an asset.
+#
+# Note that this will be most likely a path in the pub cache when the package
+# with a `build.dart` is a dependency of another package.
+package_root: /absolute/path/to/my_package_with_native_assets
+
+# Target architecture
+#
+# Combined with `target_os` this specifies the "target" for which assets
+# should be built.
+#
+# Not provided on dry runs.
+target_architecture: x64 | ia32 | arm | arm64 | riscv32 | riscv64
+
+# Target operating system
+#
+# Combined with `target_architecture` this specifies the "target" for which
+# assets should be built.
+target_os: android | ios | linux | macos | windows
+
+# Schema version of this file.
+version: 1.0.0
+```
+
+### `build_output.yaml`
+
+This file is the output from running a `build.dart` script, and contains the list of assets to be bundled into the application.
+
+```yaml
+# The list of assets.
+#
+# In dry runs, must contain assets for each architecture for the requested os.
+assets:
+  - id: 'package:my_package_with_native_assets/src/foo.dart'
+    link_mode: dynamic | static | prefer-dynamic | prefer-static
+    path:
+      path_type: absolute | system | process | executable
+      # Only provided for path_type absolute and system.
+      #
+      # If path_type absolute: The absolute path to the file name on the
+      # current machine.
+      #
+      # If path_type system: The path of the dynamic library as available on 
+      # the target machine's PATH.
+      uri: /absolute/path/to/outdir/arbitrary_filename.whatever
+    target: linux_x64
+  ...
+
+# The files used by this build.
+#
+# If any of the files in [dependencies] are modified after [timestamp], the
+# build will be re-run.
+#
+# Not output on dry runs.
+dependencies:
+  - /absolute/path/to/my_package_with_native_assets/build.dart
+  ...
+
+# The time the build this output is for started.
+#
+# Must be before any of the files in dependencies are read.
+timestamp: 2024-01-02 17:05:35.000
+
+# Metadata usable for build.dart of packages directly depending on this package.
+#
+# Not output in dry runs.
+metadata:
+  # Key value pairs.
+  some_key: some_value
+
+# Schema version of this file.
+version: 1.0.0
+```
+
+### Running `build.dart`
+
+The input is passed as a absolute path to a YAML file `build_config.yaml` encoding the input information as follows:
+
+```console
+$ dart build.dart --config <build_config.yaml>
+```
+
+The Dart and Flutter SDK invoke `build.dart` of all packages in the transitive dependencies and pass a `build_config.yaml` as the `--config` option.
+
+If not in `dry_run` mode, the `build.dart` file MUST:
+ * Read the `build_config.yaml` file,
+ * Create assets using the configuration from `build_config.yaml`, if not already existing.
+ * Write asset files into the directory with path `out_dir` that was provided by `build_config.yaml`.
+   * MUST avoid file name `build_output.yaml`.
+ * Write `build_output.yaml` into the directory at `out_dir`.
+   * This maps the `assetId`s to assets previously written into `out_dir`.
+   * There may be multiple assets for a given `assetId` depending on
+     characteristics like `target`, `link_mode`, etc.
+   * If the boolean parameter `dry_run` is set to true in the build configuration,
+     the list of assets must be output, but the asset files must not be written.
+     The asset file urls are expected to refer to non-existing files.
+
+Notes: 
+   * The file name inside `out_dir` is irrelevant, because in Dart code the asset will only be referenced by its `assetId`.
+   * If the boolean parameter `dry_run` is set to true in the build configuration, then the actual creation and writing of assets can be skipped, as long as the assets are still listed in the `build_output.yaml`.
+
+### `.dart_tool/native_assets.yaml`
+
+**This is not part of the build.dart protocol.**
+
+This file is internal to the Dart SDK and Flutter SDK.
+It produced by `native_asset_builder` (as embedded in the SDKs).
+For each `target` it maps from `assetId` to a path type with an optional path.
+
+These paths are used to resolve `@Native() external` functions in Dart code.
+
+The path types in this file are as follows:
+
+* `absolute` paths are absolute paths on the system where the Dart executable is running. This path type is used when running in JIT mode on a developers' host machine, or in an Android app where the root path is the Android bundle.
+* `relative` paths are relative to the kernel or aot snapshot. This path type is used for shipping native assets in a directory together with a kernelsnapshot, aotsnapshot, or standalone executable (dartaotruntime+aotsnapshot).
+* `system` paths are expected to resolve on the target machine PATH.
+* `process` "paths" have no path, symbols are resolved in the current process.
+* `executable` "paths" have no path, symbols are resolved in the current executable.
+
+```yaml
+# Schema version of this file.
+format-version: [1, 0, 0]
+
+# A mapping from target and assetId to an asset.
+native-assets:
+  # A <target>.
+  linux_x64:
+    # An <assetId>.
+    'package:my_package_with_native_assets/src/foo.dart':
+      # A list of path_type and optionally a path.
+      - absolute | relative | system | process | executable
+      - <url> # Only provided for absolute, relative, and system path types.
+  ...
+```
+
+This file is used by the embedded SDK to resolve `assetId`s when running in
+development mode or build a deployable application (or standalone executable).
+
+## Wishes to improve for v2
+
+* Multiple asset types. Rename `Asset` to `NativeCodeAsset` and introduce `DataAsset`s.
+* The path_types really only make sense for `link_mode: dynamic`.
+* Fix the mismatch between `target` for output but `os` and `architecture` for config.
+  * Should the architecture field of native code assets be optional and be ommitted in dry runs?
+* Introduce a `link.dart` protocol with a link input and link output.