Jace Warren - Replacing a Billion+ Record per Day Cassandra Cluster With Elixir and SQLite

At Savi Solutions, our video surveillance platform needs to be able to process billions of records per day for customers including Burger King, Little Caesars, 5 Guys, and more. As a result, we need a low latency, high performance, and linear scalable database, so we chose Cassandra. However, as our Cassandra cluster began to exhibit performance issues, degrade, and grow in cost, it was evident that we needed to make a change. Thanks to SQLite, Elixir, and the Beam, we were able to completely remove Cassandra from our stack, allowing us to distribute all storage workloads across existing application servers with minimal overhead at no cost.

In this presentation I will cover the strategies and lessons learned in setting up Cassandra and migrating to SQLite such as:

Cassandra and SQLite schema design for video streaming
SQLite partitioning strategies
Interacting with multiple SQLite database files via Ecto
Executing zero-downtime migrations
By the end of this presentation, you will have a better understanding of trade offs while using Cassandra and SQLite and their ability to pair with Elixir in building distributed, scalable, and fault-tolerant applications.

About Jace
Jace currently works as a Principal Software Engineer at Savi Solutions. He first fell in love with software development at the age of 14 when he taught himself how to write his first application. He received his first series of programming contracts by building a Job Procurement Bot that would autonomously scrape the web and apply for hundreds of job opportunities per week. When he is not working, he enjoys spending time with his family, engaging in video game development, or being towed around the neighborhood on his longboard by his dog.