Using Elixir for scheduling cloud-native DAG workflows

Speaker:
Lucjan Suski


Abstract:

Current state-of-art DAG workflow schedulers (AirFlow, Luigi) do not seem to leverage the power of on-demand cloud computing instances. Therefore, Lucjan and his team used Elixir's distributed capabilities to write a powerful, yet simple, scheduler for their data processing pipeline. Lucjan will talk about the various approaches to running external processes in Elixir, monitoring and displaying progress UI in LiveView and testing and the challenges.