Real-Time Ingestion And Transformation With Snowflake

Learn how Snowpipe Streaming and Dynamic Tables work together to address a very common use case: ingesting CDC events from source databases/applications and representing that data in Snowflake in multiple forms--current replica of the source tables, aggregation and summary tables, and full historical view of all table transactions using slowly changing dimensions.

These two new features bring highly scalable, near real-time streaming ingestion and single-row insert capabilities to Snowflake, plus a new level of simplicity to merge and transform all those events into consumable data sets with minimal latency, greatly reducing files and micro-batch approach. Over 50K events/second throughput from a laptop, at-source tokenization, AES encryption, and more.

Check out the resources linked in the video:

Snowpipe Streaming

API Docs: https://javadoc.io/doc/net.snowflake/...

Documentation:
https://docs.snowflake.com/user-guide...

GitHub Sample Client Project:
https://github.com/snowflakedb/snowfl...

Dynamic Tables

Article: https://www.snowflake.com/blog/dynami...

Documentation:
https://docs.snowflake.com/user-guide...

PII Quickstart:
https://quickstarts.snowflake.com/gui...

We also invite you to watch the webinar "Batch and Streaming Data Ingestion Best Practices with Snowflake" to learn how to leverage COPY, Snowpipe, and Snowpipe Streaming to build batch and streaming data pipelines at scale. Register now at: https://tinyurl.com/2cezbtp9