ElixirConf 2023 - Zack Kayzer and Stephanie Lane - Unleashing the Power of DAGs with Pacer Workflow

Comments at: https://elixirforum.com/t/elixirconf-...

Have you ever had a complex module that is challenging to comprehend, debug, and maintain due to data that has intricate interdependencies with logic stuck in numerous private functions? How do you sort out the dependencies while keeping the relationships between your data points clear and easy to reason about? Try a Directed Acyclic Graph (DAG)!

In this talk, we'll explore how we originally tried to solve the problem, how we settled on using a DAG to create a dependency graph, and what we were missing from a traditional DAG that led us to write an abstraction called Pacer.Workflow. Graphing libraries like Erlang's :digraph and the Hex package LibGraph provide functions and algorithms for working with DAGs like topological sorts and utilities for visualizing graphs out of the box; however, we wanted to build some more tooling and utilities on top of these features, such as the ability to collapse several nodes in the graph into a single node representation in certain situations (spoiler: we associate nodes in the graph with functions that we want to run, and in some cases, we want to run several functions in parallel if possible, thus the need to collapse multiple nodes into one).

While building out Pacer.Workflow, we focused heavily on the developer experience, starting from providing a number of compile-time checks and guarantees with descriptive messaging when constraints are violated, building in Telemetry events from day one, and a focus on documentation and example usage.