Скачать или смотреть Quartile Data AI Summit 2022

Quartile is a leader in the Ads Management sector, with over $2B yearly Ad Spend under management and our core analytics tool is Databricks. On this presentation we will share our data architecture and our Common Data Model (CDM) strategy, which is responsible for presenting to our customers how well their sales are performing. The CDM architecture includes our integration with partners APIs (Amazon, Google, Instacart, Shopify), data storage in Azure Data Lake, transformation and moving data from Bronze to Silver to Gold tables, and further exposing the data using BI tools: Microsoft Power BI (for external customers) and DB SQL Dashboards (for internal teams).

As part of our processes for ensuring data quality, we have built an Anomaly Detection algorithm to help us detect when our ETL process is not working as expected quickly and before our customers notice or need to tell us. We will also go over the Databricks SQL Dashboards we have created for internal validation teams to add another layer of confidence to our data quality. When there are issues, we can quickly understand which level of table is generating the problem (Bronze, Silver, Gold).

During the development process of creating our CDM, we have often chosen strategies that were not so effective for our scenarios, and we noticed a lot of the information we needed was available in documentation and forums but not as easy to capture in a single place. AutoLoader strategies (streaming vs batching, schema evolution, source cleanup), optimization commands (Auto-Optimize, Z-Order, Vaccuum) and partitioning strategies (do you really need it?) are among those we will cover on this part o the lecture. You know those facts that everybody seem to understand but few talk about it? You will find them here!