Macro argument scopes (specifically, Code objects), and augmentation libraries

[jakemac53]

We want to be able to model the Macro output in terms of an augmentation library, and in general we want the code output by macros to not have special powers.

At the same time, we allow Code arguments to macro constructors, and we want the scope of that code to be essentially the same scope as the macro annotation. However, we also possibly want to be able to refer to local variables within those expressions.

Example:

abstract class Thing {
  int get x;
  int get y;
  
  @MemoizedVar(x + y) // `x` and `y` here are local variables. 
  int get z;
}

But what if z is a function, and it has parameters named x or y? We need to clarify whether the macro arguments scope is that of the body of the declaration it annotates, or the class, or something else more custom.

Some factors to be taken into consideration:

• It might be useful for a macro arguments to be able to access the arguments to a function.
• A macro on z can emit code for declarations outside the scope of the body of z (for instance it can add fields and methods to the surrounding class and/or library, or add whole new classes).
• Macros may annotate static methods/fields on classes.

There are a few possible solutions I have been considering to this:

Solution 1: No access to local variables or params in macro argument expressions

The scope is the lexical scope of the body of the class, static members of the class are visible (possibly without being qualified by ClassName.?), but no local variables.

This could be modeled by generating a static getter next to the original annotated declaration:

library augment 'original.dart';

augment class Thing {
  static  get _$someGeneratedName => x + y;
}

Pros

• Easiest to define.
• Valid anywhere in the library (may need some munging to add the qualified ClassName. in front, or just require it).

Cons

• Very limiting, can't access parameters or local variables.

Solution 2: Its own lexical scope, as if the expression was inside the body of a new declaration

We could model this in the same way as Solution 1, except the getter wouldn't always be static - it would only be static if the annotated declaration was static.

Pros

• I think this mostly behaves how users would expect, the scope is essentially the same as the annotated declaration.
• Allows access to local variables.

Cons

• Possibly a bit magical, might be unexpected that expressions in macro annotations on static fields have a different scope.
• These expressions cannot be used in parent scopes (unless the annotated declaration was static).

Solution 3: The scope is the same as the scope at the top of the body of the annotated declaration.

We could again model this in essentially same way, except with a local function at the top of the body of the annotated declaration.

library augment 'original.dart';

augment class Thing {
  int get z {
    // Note that the `Code` object in this case would have to be a function invocation not a reference to a getter.
    _$someGeneratedName() => x + y;
    // Rest of the generated code here (should contain an `augment super`).
  }
}

Pros

• Has access to parameters as well as local variables, could be useful.

Cons

• Breaks lexical scoping, the expression looks like normal code and lives outside the body of the declaration, so it likely isn't intuitive that the parameters are available.
• The expression is only usable within the annotated declaration (can't be used even on other declarations in the class).

[jakemac53]

My personal opinion at this time is to go with solution 2. I think its the least surprising, pretty easy to define and model in terms of augmentation libraries, and still provides a good amount of functionality.

[munificent]

Do we only support macro arguments that are used as expressions? I can imagine a macro like:

@addIntField(foo)
class C {
}

Which expands to:

class C {
  int foo;
}

In this case, the macro argument isn't treat as an expression, it's treated as an identifier that binds a name. That suggests another potential solution:

Solution 3: Macro arguments are unresolved

They are treated as pure syntax. It is only the resulting code where they are interpolated that gets resolved. In fact, the same macro argument could be interpolated into multiple different points in the generated code such that each resolves in different ways.

I don't know if this is the best solution, but I think it's a fairly simple one that's implementable.

[jakemac53]

We could make them unresolved too, but I think its a really nice affordance to have them be resolved. It means you can get code completion and everything else within those expressions.

In this case, the macro argument isn't treat as an expression, it's treated as an identifier that binds a name. That suggests another potential solution:

I think for this case I would probably just ask for a String that is the name? Most of the interesting cases will be more complex expressions, where code completion and error checking etc would be nice.

[munificent]

I think for this case I would probably just ask for a String that is the name?

We could, but that feels weird to me. The macro argument is already lived from being an expression to being a metaexpression that represents an identifier. Why the should macro user have to manually double lift the identifier into a string? If the macro needs an identifier which it will then use to declare something, isn't the natural syntax for that argument... an identifier?

[Levi-Lesches]

I'd argue that by passing an identifier into a macro, I'd expect the identifier to resolve and pass the value instead to the macro. This would make using and writing macros that take parameters more intuitive, since that's how constructors already work. If I wanted to pass in a literal name, it would be pretty obvious to me that I should be using a String, just like I would when passing literal text to a UI framework.

[jakemac53]

@Levi-Lesches ya that is basically my intuition. If I wanted to pass a variable definition as Dart code I would probably expect it to be an entire declaration anyways (final int foo or var foo) etc.

We had discussed only supporting expressions, which I still think is the right choice, but i don't see that codified in the spec yet so I will send a PR for that as well.

@tatumizer please read https://github.com/dart-lang/language/blob/master/working/macros/feature-specification.md#arguments

[jakemac53]

@tatumizer The issue is those wouldn't be resolved expressions (its back to strings) instead of actual Dart code. I get where you are coming from with it being a bit confusing - there is something to learn there. But that isn't in any case what this issue is about (although its related).

We also don't want to leak the macro apis (including the Code api) into app code.

[jakemac53]

We have one issue already discussing this generally (#1989), and in any case the syntax isn't related to the scope of those expressions, which is what I am trying to nail down here.

[TimWhiting]

Personally I think both unresolved and resolved syntax are useful. I appreciate the code-completion of the latter and the flexibility of the former.

Example:

macro MyMacro implements ... {
  MyMacro(LexedTokens tokens, {required Expression initializer});

  /// Implementation uses the resolved expression to get the type parameter for the field
  /// but uses an unresolved identifier for the first parameter.
}


/// Usage

late int x;
late int y;

@addLateField(foo, initializer: x + y);
class C {
}

/// expands to 
augment class C {
  late int foo = x + y;
}

The tokens for unresolved code cannot include a comma notably.

It encourages macro authors to use resolved subclasses of Code for the most part (because of being able to resolve types and have code completion for users), while allowing for simple usage for example of unresolved identifiers.

It is up to the author of the macro to return / throw an error if the first argument is not a simple identifier. But this way you don't have to worry about resolving identifiers in scope when they aren't intended to be in scope to begin with.

[Levi-Lesches]

I'd actually agree with @jakemac53 again here, that passing in x + y as a parameter feels wrong when there's a class definition right there you can put it into:

If I wanted to pass a variable definition as Dart code I would probably expect it to be an entire declaration anyways (final int foo or var foo) etc.

My logic is that since the macro does not know what the expression is, it can't do anything useful with it that you can't do by annotating the field. In other words, instead of:

late int x, y;

@MemoizedVar(x + y, "z")  // syntax here doesn't matter right now
class MyClass { }

// expands to: 
augment class MyClass {
  int? _z;
  int get z => _z ??= x + y;
}

I'd much rather see:

late int x, y;

class MyClass {
  @memoize
  final int z = x + y;
}

// expands to: 
augment class MyClass {
  final int? _z;

  @override
  int get z => _z ??= augment super;
}

Again, the macro's job isn't to introduce a field called z, initialize it with x + y, and cache it -- the macro's job is just to memoize any field. A line like @MemoizedVar(x + y) or @addLateField(foo, initializer: x + y) contains so much information that could easily just be in Dart syntax. I believe it would be better to just have the late final foo = x + y; line and annotate that with whatever generic macro functionality you need.

[jakemac53]

The @MemoizedVar example was definitely not a very useful real world example, I agree for that case I would rather see a macro on a field.

I actually was just struggling to come up with examples that were useful, so it might be better to start from those? Do we actually need this feature at all?

[Levi-Lesches]

Given that a parameter in the macro's constructor means "I need to operate on this expression but I don't know what it is", I think, generally speaking, it should always be possible to write that expression out as regular Dart code and apply the macro to that directly. Between external, abstract classes, get/set, late, and nullables, I have a hard time imagining a declaration that needs to have its definition in the same place.

[lrhn]

A problem here is that @SomeMacro(x + y) looks like a normal annotation, which creates a constant value.
The macro is instead using the expression as argument. (It can't do anything else, only constant values exists at compile-time, everything else is at most syntax.)

There is precedence for generating AST nodes from syntax in C#.
For example the LINQ SELECT operation desugars to calling the Select method on the underlying collection, but if the compiler statically determines whether that Select method expects an Expression<Func<T, bool>> instead of a Func<T, bool>, then it builds an AST of the expression instead.
It works everywhere. If the context type of an expression is Expression<T> then the expression has context type T, but it's compiled into an expression creating an AST of the expression, instead of into code evaluating the expression.
C# expressions might be grammatically simpler than Dart expression (not sure anymore, they've added a lot since then).

That might be way to do things in Dart too. If a macro annotation needs an expression, it can be typed as needing the expression, and then you can also pass values to the macro.