Advancing Spark - Lakehouse Observability with Unity Catalog System Tables

When Databricks SQL first came along, the governance nerds amongst us were very excited that we had a proper query history, but conversations with InfoSec still got very awkward when it came to table access, cluster costs, user behaviour and pretty much anything else we'd expect from modern platform telemetry. With Unity Catalog System Tables, we can now report on a huge amount of detail, as well as baking in various workflow triggers through Alerts!

In this video Simon walks through the setup of System Tables using dbdemos.ai, looks at the various artifacts created and shows some of the interesting insights we can now access!

For the official System Tables docs, take a look at: https://docs.databricks.com/administr...

For the demos - see: https://www.databricks.com/resources/...

If you're looking at overhauling your lake to give it the Lakehouse treatment, get in touch with Advancing Analytics