2024-07-12 –, REPL (2, main stage)
Many interactive applications will, at some point, need to move work out of the primary user loop, into a background process that operates asynchronously. To do this easily, scalably, efficiently and robustly is a non trivial problem. This talk introduces Horus.jl, a Julia package that makes it easy create and run background jobs in any Julia application.
Users expect interactive applications to always be responsive, even when doing large computations, or connecting to to unstable network services such as mail servers. So when an application needs heavy compute, or to perform action such as sending emails that don't need to be finished before a response is sent back to the user, a background job can be used to shift the work out of the main user loop.
This can be done somewhat trivially with a thread or even an async task. However, that does not account for some requirements of robust multiuser systems. What happens if the application server crashes before the background job is completed? How do you distribute the load if a large number of simultaneous request are received? What happens if the task depends on a service (say a mail server) that is down at the moment?
Horus.jl is a new Julia package that allows you to create background jobs easily. It uses Redis as a transactional queue, and that job will run successfully only once. It provides facilities for error handling and retries. It provides somewhat similar functionality, in Julia, to Python libraries such as Celery, or Ruby libraries like Sidekick. It is the missing piece to building large interactive applications in Julia.
Avik leads software engineering at JuliaHub, is a contributor to open source Julia and maintainer of several Julia packages. Avik is the author of Julia High Performance, co-founder of two artificial intelligence start-ups in the financial services sector and creator of large complex trading systems for the world's leading investment banks.