Advancing Spark - Automated Data Quality with Lakehouse Monitoring

As data engineers, we've built countless ETL scripts that tracked data quality, over and over again. Wouldn't it be lovely if our systems just regularly polled the data and checked the DQ for us? Wouldn't it be great if we could apply a whole set of quality metrics across our tables as standard? Well that's exactly what Databricks Lakehouse Monitoring is!

In this video, Simon takes a quick look at Lakehouse Monitoring, enables it for a sample table and runs through the quality metrics that are captured. If you're not already monitoring the quality of data in your Lakehouse... why not start now?

For more details on Lakehouse Monitoring, check out: https://learn.microsoft.com/en-us/azu...

If you're after some deep-dive, hands-on Spark training for the festive period, why not check out: https://advancinganalytics.teachable.com

And if you're embarking on a Lakehouse journey, and want it to deliver serious value, why not give Advancing Analytics a call?