TDPC February 2024: What's new in Azure Databricks by John miner

The Azure Databricks ecosystem has been around for about five years. During that time, the vendor has kept on improving both the interface and the design patterns. There are three new things that you should be using in your data engineering projects.
First, scheduling jobs has always been a part of the product. However, with workflows one can control the pattern in which notebooks are executed.
Second, delta live tables allow the developer to specify the source, the transformation and the destination of data using Python. This complete workflow can be scheduled as either a batch or real time streaming job.
Third, cloning technology has come to delta tables. One can use shallow clones to keep different environments in synch with the master. On the other hand, deep clones allow the developer to stamp out a table at a given point of time.
To recap, come to this presentation to learn about these new design patterns and get a demonstration on how to apply them in your data engineering projects.