Advancing Spark - Developing Python Libraries with Databricks Repos

The addition of Databricks Repos changed a lot of our working processes around maintaining notebooks, but the process for building out our own python libraries hasn't changed much over the years. With "Files for Databricks Repos", we suddenly see a massive shift in how we can structure our library development, with some huge productivity boosts in there.

In this video, Simon talks through the process from the ground up - taking a simple dataframe transformation, turning it into a function, building that function into a wheel then replacing it with a direct reference inside Databricks Repos!

For more info on the new additions to Databricks Repos, check out https://docs.databricks.com/repos.htm...

As always, if you need help with our Data Lakehouse journey, stop by www.advancinganalytics.co.uk to see if we can help