Monitoring Databricks with System Tables

In this video I focus on a different side of monitoring:
What do the Databricks system tables offer me for monitoring?
How much does this overlap with the application logs and Spark metrics?

Databricks System Tables are a public preview feature that can be enabled if you have Unity Catalog on your workspace. I introduce the concept in the first 3 minutes then summarize where this is most helpful in the last 3 minutes. In between are some example queries and table explanations.

All thoughts and opinions are my own *

Blog post with more detail:
https://dustinvannoy.com/2024/02/22/d...

Options to enable system tables:
Curl - https://learn.microsoft.com/en-us/azu...
dbdemos notebook - https://www.databricks.com/resources/...

Useful blogs:
  / databricks-system-tables-an-introduction  

https://www.databricks.com/blog/impro...

Useful docs:
https://learn.microsoft.com/en-us/azu...

https://learn.microsoft.com/en-us/azu...

More from Dustin:
Website: https://dustinvannoy.com
LinkedIn:   / dustinvannoy  
Github: https://github.com/datakickstart

CHAPTERS
0:00 Intro
1:15 Enable system tables
3:28 Billing usage query
6:55 Docs highlights
7:57 Lineage query
9:02 Audit query
11:17 Additional tables
13:20 Compare to Log4J/Metrics
14:08 Summary