ExprTools: Metaprogramming from reflection
2021-07-29 , Purple

Have you ever had a list of Methods, e.g. from the output of methodswith, and thought ”I just want to implement all of these, it would be great to use metaprogramming for that”?
ExprTools.jl has the parts to let you extract the info out of the method table, manipulate it, and then generate the AST you want for the new method you want to define.

Does this access undocumented Julia internals? Absolutely!
Is this well tested? Comprehensively!
Is this a good idea? Who knows!


Sometimes you want to generate definitions for many methods. Consider for example implementing there delegation pattern. Where you have a field of a different type, and you want to overload all methods that accept that field’s type to also accept this new object, and have them just delegate to calling the method on the field. Ideally this wouldn’t come up and you would just need to implement a small well documented set for an interface. But sometimes things can’t be ideal. Generating overloads from the method table is one way to take a jack-hammer to blast through the problem. But even outside that it can be useful as this talk will discuss.

ExprTools.jl was created to hold a more robust version of splitdef and combinedef from MacroTools.jl.
splitdef takes the AST for a method definition and outputs a dictionary of all the parts: name, args, whereparams, body etc. combinedef does the reverse: taking such a dictionary, and outputting an AST that declares the method.
splitdef is very useful since it both handles different equivalent syntax forms, and makes the key parts accessible in a consistent way.
This makes it easier to write function decorator macros, and also macros that let the user write something that looks like a function but is actually transformed into something else.
This dictionary is also useful, and it would be great if we could define it not from an AST but from a method that has already been defined. We could access all the information we need via reflection. This is exactly what the signature function provides.

The signature function takes in a Method object, which can be obtained from methods or methodswith, and returns a dictionary like splitdef would, except it excludes the body.
Excluding the body is generally not useful for this kind of generated code anyway since the user will generally want to fill the body with their own code that calls the method we are generating from. One exampl. Another example is generating overloaded operators for overloading-based reverse mode AD from ChainRules.jl’s rrule.

The main alternative for this kind of approach is something along the lines of Cassette.jl, which in effect allows the overloading of what it means to call a function. There are three key differences of an ExprTools-based generation from reflection approach over a Cassette-based overdubbing approach.

Overdubbing occurs in a specific dynamically scoped context, method generation applies globally.
A downside of method generation is that it will not detect new methods added after the generation is performed, overdubbing does.
An upside of method generation is that it is just plain julia code, so it doesn’t break the compiler’s ability to do type inference. The compiler is completely prepared to deal with julia-code. This is (sadly, but demonstrably) not true for Cassette right now.

This talk will spend ~3 minutes time covering the basics of ExprTools, with splitdef and combine. It will spend 4 minutes demonstrating signature and the generation of methods from the methods tables. It will spend ~1 minutes peeking under the covers as to how it works.

Lyndon White (@oxinabox) is a research software engineer at Invenia Labs (Cambridge, UK). He helps researchers use machine learning, constrained optimization, and generally tools from the technical computing domain to optimize the power grid. He get to do all the best parts of being a software developer and all the best parts of being a researcher, its great.
He works a lot on the Julia AutoDIff code, and is the leader of the ChainRules project.

This speaker also appears in: