Feature Request: Reader macros

[MasonProtter]

String macros are fine, but they really make the enclosed code feel like a second class citizen. Sometimes they make sense, but I think it would be really nice to have true reader macros. To copy-paste from my earlier issue #31449 which is apparently orthogonal to a pure-julia parser:

---

Having a Julia parser would make it possible to define a form of macro where code inside the macro scope is parsed according to user defined rules. These macros would essentially be string macros except you would no longer need to wrap the enclosed code in quotation marks, giving you truly arbitrary syntax inside a macro but retaining things like code highlighting, auto indentation, autocompletion, etc. Depending on the design of the Julia parser, it could also potentially offer more support when building reader macros than we currently offer for building string macros.

Reader macros wouldn't really allow for new features over string macros per se, but I think the fact that string macro code needs to be inside a string substantially hampers their adoption for many potential applications, especially DSL implementation.

Reader Macro Syntax

I'm not deadset on any particular syntax to mark a reader macro, but something like !@ could be used.

As an example Py2Jl.jl transpiler interface would go from

using Py2Jl

py2jl"""
def sum_by(f, seq):
    s = 0
    for e in seq:
        s = s + f(e)
    return s
"""

to

using Py2Jl

!@py2jl(
def sum_by(f, seq):
    s = 0
    for e in seq:
        s = s + f(e)
    return s
)

Obviously, there would have to be certain limitations put on reader macros, ie. if we use parens !@( ... ) to enclose code, then the parser for the macro must respect parenthesis balancing to make sure it doesn't leak out of its scope.

---

@JeffBezanson said in the earlier issue

I think the main objection people have is that they'd like to at least be able to parse julia code without a full julia runtime available.

I thought a bit about this and I think I'd respond that the parser could still support reader macros without having the full runtime available. We could just lower them to a string macro like representation, e.g. something like

Meta.parse("""
!@py2jl(
def sum_by(f, seq):
    s = 0
    for e in seq:
        s = s + f(e)
    return s
)
"""

#RESULTS 
quote
    !@py2jl"""
def sum_by(f, seq):
    s = 0
    for e in seq:
        s = s + f(e)
    return s
"""
end

[MasonProtter]

I guess I should also say that maybe an alternative approach would be to just change the way we do syntax highlighting so that it applies to the inside of a string macro, e.g. just highlight the code in

py2jl"""
def sum_by(f, seq):
    s = 0
    for e in seq:
        s = s + f(e)
    return s
"""

I must say, I'm not a big fan of using """ or " as a delimiter for reader macros, but it'd still be better than the current situation.

[mbauman]

Personally, I like the editor syntax highlighting solution, especially with the backtick @py_cmd versions. Packages like Weave.jl already have support for mixed languages in .jmd files in Atom; most editor syntaxes should be able to support the same. I don't think most editors make it easy for their syntax definitions to be extended, though — we'd probably have to hard-code in support for a handful of common interop packages. That'd be no different in the reader macro case, though.

py```
def sum_by(f, seq):
    s = 0
    for e in seq:
        s = s + f(e)
    return s
```

(ok, that's a cheat here)

[MasonProtter]

Honestly, I think it’s fine to just use Julia‘a own syntax highlighting on the code in a reader macro. It doesn’t need to be custom for the macro.

So if I wanted to advocate for changing the highlighting for string and cmd macros to just apply julia syntax highlighting to the enclosed code rather than the string highlighting, where would be the appropriate place to do that? The repos for various editor's julia support or is there a more central place the change can come from?

[JeffBezanson]

I just don't see !@x(...) as a significantly better delimiter than x""" ... """. Reader macros make much more sense in lisp, where they can be used to define meanings for [ ], { }, etc. If you need some longer delimiter, might as well not bother. I like the idea of allowing editors to recognize and format the interior of specific string macros.

[Keno]

I like the idea of allowing editors to recognize and format the interior of specific string macros.

The sublime julia syntax highlighting had this feature in 2013 ;).