Juliacon 2024

Julia confers great potential performance to users. However, the path from performant prototype to reliable-but-still performant system in deployment is paved with a growing list of methods to implement, function signatures that may need updating across many methods, and types where some method always seems to be missing but only fails when the code is run. Building large, composable, inter-operable systems in any language is difficult, and Julia is no exception. However, these problems may be especially frustrating within Julia as entire classes of problems may not be caught until runtime, potentially in production.

Often, an exhaustive test suite is seen as a panacea to production or robustness woes. Yet, without some standardization of how tests are performed, these tests can be the subject of almost as much maintenance time as the core code itself. For large projects where the size of the test suite grows, so too does the time to encounter failing tests from interface and method signature changes, rather than underlying correctness issues. As the maintenance burden and time to test increases, so too does the likelihood of missing a test and accidentally shipping broken code.

This talk proposes that focusing on the interfaces which comprise systems allows us to improve our confidence that code is correct, at least from the perspective of the type and dispatch system. We observe that numerical and algorithmic correctness is often not a rate-limiting problem the Julia community tends to face given its scientific and academic roots, but test suites can often overfit to this style of correctness. Crucially, the highly composable nature of Julia often leads to many implicit or accidental assumptions being held by developers which may not hold in general, resulting in runtime failure when types do not compose as expected or fully satisfy implicit interfaces.

Our thesis is that by leveraging tools to implement and enforce interfaces, a functional codestyle to ensure conformity, and building a uniform testing structure, we can more easily manage complexity in a large and growing codebase. These techniques improve productivity by informing developers of what a type implementation needs, simplifying refactor efforts, and catching interface violations which otherwise may slip through the cracks of an incomplete test suite.

We explore the idea of managing complexity through interface-driven design via two lenses: the theoretical approach of design patterns and processes with associated tooling, as well as a study of existing packages which showcase interface best practices.

From the design perspective, we highlight the existence of packages such as RequiredInterfaces.jl, BinaryTraits.jl, or Interfaces.jl to define, check, and potentially dispatch on interfaces or traits. We will discuss the implications of these packages on developer productivity as documentation and "onboarding" tools, as well as tools to improve trust in our Julia code to be better-behaved in production by reducing runtime errors strictly to logical errors. A discussion of functional codestyle, how it interoperates with interface-driven design, and advantages of a functional approach to logical correctness will be included. Traits and interfaces will be compared and contrasted, viewing traits as a dispatch tool relying on an interface specification. We will discuss the process that goes into developing good interfaces to capture the essence of a problem, chiefly separating the "what" from the "how" of functions.

For exemplar packages, we focus on two different approaches:

1. Implementing robust interface testers, such as ChainRulesTestUtils.jl which provides a tester function for rrules and frules
2. Implementing a powerful common "language" and interface for packages to inter-operate, as demonstrated by GeoInterface.jl

The implications of these packages and their offerings to improving the developer experience will be emphasized. ChainRulesTestUtils.jl not only verifies an interface implementation, but also checks for the numerical correctness as well. GeoInterface.jl utilizes interfaces to facilitate easy interoperability between packages which satisfy its interface specification.

We conclude by reiterating that while the performance of Julia the language is remarkable, the developer experience of building large and robust systems is fraught with difficulties. Yet, taking an interface-driven design approach to Julia leverages the strengths of the Julia language (multiple dispatch, specializing methods, composition of types) to arrive at code which avoids easy but subtle mistakes when working on projects of increasing complexity. By designing projects to prioritize managing complexity as a codebase grows, we hope to ease difficult parts of productionizing Julia codebases.