Pick a supported serialization format for host<->macro communication

[davidmorgan]

The dart_model exploration currently uses JSON but there is no particular reason to think that's a good choice :)

Even if we don't follow the dart_model direction, it seems likely we want to formalize the serialization format so that we can have stable serialization. Some well-supported system that allows specification of a schema is probably needed so we can actually have guarantees of compatibility.

It looks like macros often do only trivial computation and so they are going to spend most of their run time deserializing then serializing. The best way to optimize this is to do less work, to send less data; but even then it might be beneficial to also have the fastest possible serialization and deserialization.

Protocol buffers are an obvious possibility, flat buffers might be more interesting since they are explicitly for cases where deserialization/serialization are an important overhead. If we follow the dart_model direction then capability to quickly compute deltas and apply deltas will also be important.

Sending this straight to the backlog since the precise format doesn't matter too much yet.

@jakemac53 @scheglov FYI :)

[rrousselGit]

Would this be possibly related to macros sending information to the analyzer ; which the analyzer can then use to emit custom diagonistics?

[davidmorgan]

Macros are fundamentally built on serialization: when you call async methods on the macro API, the macro communicates with the host via serialized data. The hosts we have today are the analyzer and the CFE.

[davidmorgan]

I looked into this a little today; I don't like the flatbuffers schema format; one possibility that comes to mind is to use a JSON schema coupled with some more efficient way of serializing JSON, possibly we can do something like flatbuffers where data is accumulated into a buffer that is immediately ready to send.

[jakemac53]

I don't know anything about flat buffers, do you have a tldr? How does it compare with the adhoc format we have today?

[jakemac53]

Also, the idea of using FFI to enable multithreading + shared object memory with the host has been brought up previously. I have no idea what that concretely means or what it would look like though. @mraleph I heard maybe you would be a good reference here?

[jakemac53]

It sounds to me from some very quick reading that using FFI to write to a shared buffer (where?), and FlatBuffer to actually control those buffer formats, with Dart wrappers reading from those buffers directly, could be interesting.