Macro application annotation identification

[lrhn]

EDIT: Updated to allow designating a constant variable as a macro application annotation, and removed the attempt to recursively follow variables or type-aliases. Only the macro application declarations that are mentioned in the package configuration will trigger macros when used as annotations.

Compilers need to recognize annotations as macro application annotations, and they need to do so at a point in compilation where the program isn't yet complete (because macros haven't run), which means expression evaluation isn't necessarily possible. The existing source is not a Dart program, and the language only defines semantics for complete and valid Dart programs.

Since macro applications are annotations, and annotations are expressions, we need some way to recognize an expression as representing a macro, and preferably without needing to evaluate another annotation to mark the first annotation class as a macro application.

Here's an attempt to flesh out an idea, based on in-person discussions from the joint meeting week, of defining which declarations are macro application declarations in metadata, completely ouside of the source.

Pubspec/package-config-defined macro annotations

We want to recognize specific annotation expressions as macro application triggers.

We do that by singling out a macro application annotation class or macro application annotation variable, and then considering every annotation which constructs an instance of the class or is a reference to the variable, to be macro application triggers.

Further, we associate a macro implementation with the macro application. There are a number of ways we can do that, I'll mention some and leave that open for now.

The metadata specifying which class is a macro application trigger, and which macro implementation it executes, will be available in the .dart_tool/package_config.json file, as a non-breaking addition to the format. (The format for a program that doesn't depend on any macro-defining packages will be the same as today, and having a macro available will add extra information.)

The information in the package configuration is generated from similar information in the declaring package's pubspec.yaml, and written by the Pub tool along with the rest of the .dart_tool/package_config.json file.

Pubspec format

A package which introduces a macro application class can add (something like) the following to its pubspec.yaml:

macros:
  - 
    application:
       library: macros.dart
       name: MyMacro 
    implementation:
       library: src/macros/my_macro_impl.dart
       name: MyMacroImpl
  - 
     application:
       local_library: tools/test_macro.dart
       name: MyTest
     implementation:
       local_library: tools/test_macro_impl.dart
       name: testMacro

The library value is a path inside the current package's lib/ directory. That is, the path of a package: URI without the leading package:mypackagename/. This file can be referenced by other packages.

The local_library value is a file path relative to the pubspec.yaml file itself, which must not be inside the lib/ directory. It allows a package to declare a local macro for its own use only, for example a macro only used by its own tests.

The name value is an identifier, which should be the name of a top-level declaration inside that library.

Files inside lib/ must not depend on a local macro. (They can't without importing using a file: URI, so that should just come naturally.)

The implementation: (somehow) specifies how to run the implementation code.

Package-config format

This information is written into the package-config.json file for any package which depends on the macro-declaring package.

Local macros are only included for the current package, public macros are included for all packages in the program.

The format for a package: entry is changed to include a macros: [] field:

   {
      "name": "my_package",
      "rootUri": ".../pub_cache/hosted/my_package/1.2.3/",
      "packageUri": "lib/",
      "languageVersion": "3.7",
      "macros": [
        { 
          "application": {
            "library": "macros.dart",
            "name": "MyMacro"
          },
          "implementation": {
            "library": "src/macros/my_macro_impl.dart",
            "name": "MyMacroImpl"
          }
        }
      ]
    }

The library URI is relative to the surrounding package's package: URI root: package:my_package/.
Which means that both the macro application annotation and the macro implementation entry point must be in the same package. (They can always call code from other packages afterwards.)
It's not possible to designate an existing annotation from another package as a macro trigger. No making @override a macro trigger!

A local_library entry can be included for the current package, which has a path relative to the package_config.json file itself. (It can use non-relative file paths, but generally shouldn't. I guess it can technically use a package: URI as the implementation, but it should use library for that.)

  {
    "name": "my_package",
    "rootUri": "../",
    "packageUri": "lib/"
    "macros": [
      {
        "application": {
           "local_library": "../tools/test_macro.dart",
           "name": "MyTest"
        },
        "implementation": {
           "local_library": "../tools/test_macro_impl.dart",
           "name": "testMacro"
        }
      }
    ]
  }

No entry may have a library path for the application and local_library for the implementation.
The other direction is allowed, and is then considered a package local macro.

Tool usage

With this information available, a source processing tool (compiler, analyzer) can recognize a macro application class syntactically. It's an annotation referring to the declaration with the given name in the given library. Above the it's the MyMacro declaration of the package:my_macro/macros.dart library.

Since the program (and even the current library, if it has a macro appliication annotation) is not complete, the macro tool needs to have its own preliminary name resolution, which predicts which declarations, with which names, are exported by which libraries, and which then builds the preliminary import and declaration scopes for each library.
Then metadata annotations are resolved against these preliminary declarations. Since the program is incomplete, not being able to resolve a name is not an error.

The next step is then recognizing source annotations which are instances of that declaration.

To do that, the compiler (I'll use that to cover any program which may run macros) can directly recognize a metadata annotation whose constant expression is either an invocation of a constructor of a macro-application class declaration, or a direct reference to a macro-application constant variable declaration. That is, either:

• The expression is a constant object creation expression whose type clause (the clause before any .name, (args) and or type arguments in the constructor invocation) denotes a type declaration, and that type declaration's name and declaring library are marked as a macro application in the package configuration. Or,
• The expression is a (possibly qualified) identifier which resolves to a constant variable declaration, and that variable declaration's name and declaring library are marked as a macro application in the package configuration.

This is very limited, and deliberately so.
There is no conditional expression. You can't do const macro = isWeb ? MyMacro.web() : MyMacro.native(); to choose between two different macro implementations, rather the macro constant itself defines which implementation to use.
One can do const macro = MyMacro(isWeb); and handle the conditional behavior inside the macro implementation.

Notice that the "library and name" denoting a macro application declaration are checked against the resolved declaration's original library, not any library it might have been exported through. There has to be a declaration with the specified name in the specified library, not just in its export scope. (A type alias declaration suffices, the actual class can still be declared in another library.) This design is chosen so that import resolution doesn't have to remember the export path(s) that the name is imported through, only where the declaration was originally written.

Author support

This design requires a macro author to update the pubspec.yaml file every time they make a change to the naming or location of a macro application annotation class or a macro implementation.
That gets old fast.

I propose that the analysis/language server helps with this, by having a @macro annotation added to dart:core.
Then an author can annotate the macro application class as:

@macro("src/macro/impl.dart", "MyMacro")
class MyMacro {
  // ...
}

and get a diagnostic, with a quick-fix, if that doesn't match a macro listing in pubspec.yaml.

That is, the way to write a macro is to write:

@macro("src/macro/impl.dart", "MyMacroImpl") // Path relative to current file.
class MyMacro implements SomeMacroInterfaces {
  const MyMacro();
}

and then have a quick-fix insert the corresponding lines into pubspec.yaml and update package_config.json,
and have another quick-fix create a missing src/macro/impl.dart if it doesn't exist and add a class MyMacroImpl implements ProperMacroInterfaces {} to it.

Probably even allow the implementation to have a back-reference, with the creation quick-fix adding it automatically:

@macro.impl("/macro.dart", "MyMacro")

so that if just one of the @macro annotations get out of sync with declarations, they can be auto-fixed to the new name and location, based on finding the back-link.

Macro implementation execution

The macro implementation is run as a separate Dart program, in a context where it has access to the "macro execution context" that triggers the macro execution, and which allows introspection and code generation. That context is provided as an argument in each step where the macro is executed.

Macro implementation code is completely normal Dart code, with no new features. It may even contain macro applications itself.

Currently the macro implementation is defined a class declaration, which means that it's implicitly constructed by the macro execution context (it must have an unnamed constructor that can be called with zero arguments).
That can still work. There is some set-up that happens before macro execution starts, so it's not unreasonable to allow that set-up to be generated specifically for the macro class, so that it can be instantiated. (Or use dart:mirrors. Or not!)

Other options are:

• Point to a constant variable declaration which must hold the macro implementation object. Rather than relying on calling a default constructor to create the object, have the author create the instance themselves. Having to be constant also ensures the object isn't stateful, and doesn't keep information between stages. (Unless it cheats and uses global variables, but we can say that we don't promise that later stages are run in the same isolate. And then sometimes not do so.)
• Point to a not-necessarily-constant getter (can be a final variable, can be a real getter) which produces the instance. That allows an implementation object with state, but we can still say that each phase asks for a new instance, and may not do so in the same isolate, if we want to keep phases separate. (Which we do, because clever recompilation may be able to decide that earlier stages will generate the same result again, so there is no need to actually run them.)
• Point to any function that can be called with zero arguments, and call it to produce the macro implementation object. Basically the same as getter.

All of these approaches, including class-with-zero-argument-constructor, have the restriction that a macro implementation cannot do asynchronous initialization.
The solution to that is to delay that initialization until the start of phase 1. Probably not a problem in practice.

I suggest using the same restriction as for macro annotations: The name should be the name of a type with a constant unnamed zero-argument constructor, or a constant variable already containing a value.

Annotation Introspection

When the macro implementation runs, it can introspect on the macro application annotation. If that annotation refers to a variable, then the library and name of that variable may be all available information. That should be fine for macros that are not parameterized.

When introspecting on a constant constructor invocation, the macro may be able to get some information about type arguments and arguments, at least their syntax. For type arguments, their name may be enough to refer to the same type again. For argument expressions, some expression structures may be possible to reify as values in the macro implementation (for example something JSON-like, with only list, map and simple value literals). Other arguments may be completely opaque, at least until later phases of macro execution.

Nothing is finalized about this yet.

Post-macro-execution

The macro application annotation is not removed by macro execution, it must still (or finally) be valid as a constant expression in the completed program.

If macro generated declarations change the preliminary binding of a name, either one denoting a macro application declaration, or a name occurring in the arguments to a macro application constructor, then the macro subsystem decides whether that should be considered an error or not. The language doesn't consider the validity of partial programs, it does not have an opinion until the program has been completed by running all macros. That is: Macros do not add any new language features. It relies on the augmentations feature, but that is a separate feature. The macro feature is a tool feature, a Dart program pre-processor that runs on something that may not be a valid or complete Dart program, and therefore not something the Dart language has any opinion on. The macro preprocessor runs completely normal Dart code as macro implementation, and only when that's done and the program is expected to be complete, does the Dart language have any say about it, at which point it too should be a completely normal Dart program.

[johnniwinther]

We could be even more lax about what names can trigger macro application and allow any identifier exported from the library to trigger macro application. This would support application through const constructor invocation, like @Macro(), and constant references, like @macro where const macro = ....

[lrhn]

@johnniwinther SGTM, will rephrase. And drop the recursive checking through aliases and unmarked constants.
We're working with Dart code at a point before constant values can exist. It's OK to be a little pedantic about what names (might) mean.

[rrousselGit]

Editing the pubspec doesn't seem to be a requirement using the current dev branch. Would that be a new requirement?

In that case, would not relying on @macro for macros be a solution? If we were to instead use a separate syntax, such as #macro instead, would this remove the need for listing macros in the pubspec?

[lrhn]

If this approach is chosen, the tools (analysis/language server) should update the pubspec.yaml for you, if at all possible.

All you would have to do is write, fx, lib/src/macro.dart:

@macro('/src/macro/my_macro.dart', 'MyMacroImpl')
const myMacro = _MyMacroTrigger();

then tools should tell ask you if you want to update pubspec, and it'll automatically insert

macros:
  - 
    application:
       library: src/macro.dart
       name: myMacro 
    implementation:
       library: src/macros/my_macro.dart
       name: MyMacroImpl

which will probably trigger pub get to update .dart_tool/package_config.json.

Next thing it'll tell you that there is no src/macros/my_macro.dart with a MyMacroImpl class, and ask if it should create it.

That's obviously me being highly optimistic about the work of someone else. (Hi Analyzer people! You're doing great!)

Using a syntax like #macro (how?) seem like it will need a language change, which isn't needed otherwise.
(I prefer to not have macros introduce any changes to the language, which aren't usable without macros too.)

[davidmorgan]

Thanks Lasse! I filed #3847 for the annotation introspection issue since it's quite a deep one :)

Generally, looks good to me :)

My one suggestion would be maybe to generalize the first part away from macros: a mechanism to write structured information for the build tools in pubspec.yaml, to be passed through package_config.json, seems like it might be useful for other things. The mapping from @macro to pubspec.yaml enforced by a lint + quick fix could also apply more generally. Maybe some word like pragma, precompile, define, ...

With respect to macros it's possible we end up wanting to add to the structured data e.g. to pass hints related to optimization to the build tools. So even if it's just for macros a bit of leeway for additions would be nice.

Thanks :)

[davidmorgan]

@lrhn Another note related to "we might want to store more data this way", I believe the macro metadata proposal from @johnniwinther #3847 might include wanting to specify annotations that are not macros but are "more introspectable" by macros.

It would essentially mean that modular compiles store enough information to reconstruct the annotation as source, in addition to as a const value.

Possibly this is a fit for the same mechanism.

[lrhn]

Storing more data starts to worry me.
If we expect larger amounts of data, not just "~1 or less" per package, it might be better to have a separate file, and a general feature for producing metadata accessible to tools from pubspec.yaml entries of all the available packages.

Something like

  tool_data:
    macros:  # key to add entries to
      ... Arbitrary JSON-data values
  dev_tool_data: # only for same package
     macros: ...

Where all the data from all transitive dependencies, plus dev-data from the current package, is collected, put into lists by their key, expanded if they are already lists, and written to individual files per key, like .dart_tool/tool_data/macros.json, and then any tool can read what they need.
Maybe the file is sectioned by originating package. That's probably safer, then paths in the data can be package relative.

(Pub will have to decide when to GC that directory, which could be every time it regenerates it.)