https://cnfl.io/podcast-episode-239 | Is it possible to build a real-time data platform without using stateful stream processing? Forecasty.ai is an artificial intelligence platform for forecasting commodity prices, imparting insights into the future valuations of raw materials for users. Nearly all AI models are batch-trained once, but precious commodities are linked to ever-fluctuating global financial markets, which require real-time insights. In this episode, Ralph Debusmann (CTO, Forecasty.ai) shares their journey of migrating from a batch machine learning platform to a real-time event streaming system with Apache Kafka® and delves into their approach to making the transition frictionless.
Ralph explains that Forecasty.ai was initially built on top of batch processing, however, updating the models with batch-data syncs was costly and environmentally taxing. There was also the question of scalability—progressing from 60 commodities on offer to their eventual plan of over 200 commodities. Ralph observed that most real-time systems are non-batch, streaming-based real-time data platforms with stateful stream processing, using Kafka Streams, Apache Flink®, or even Apache Samza. However, stateful stream processing involves resources, such as teams of stream processing specialists to solve the task.
With the existing team, Ralph decided to build a real-time data platform without using any sort of stateful stream processing. They strictly keep to the out-of-the-box components, such as Kafka topics, Kafka Producer API, Kafka Consumer API, and other Kafka connectors, along with a real-time database to process data streams and implement the necessary joins inside the database.
Additionally, Ralph shares the tool he built to handle historical data, kash.py—a Kafka shell based on Python; discusses issues the platform needed to overcome for success, and how they can make the migration from batch processing to stream processing painless for the data science team.
EPISODE LINKS
► Kafka Streams 101 course: https://cnfl.io/kafka-streams-101-epi...
► The Difference Engine for Unlocking the Kafka Black Box: https://cnfl.io/streampunk-the-differ...
► GitHub repo: kash.py: https://github.com/xdgrulez/kash.py
► Kris Jenkins’ Twitter: / krisajenkins
► Streaming Audio Playlist: • Streaming Audio Podcast | Apache Kafka®, C...
► Join the Confluent Community: https://cnfl.io/join-the-community-ep...
► Learn more with Kafka tutorials, resources, and guides at Confluent Developer: https://cnfl.io/confluent-developer-e...
► Live demo: Intro to Event-Driven Microservices with Confluent: https://cnfl.io/demo-intro-to-event-d...
► Use PODCAST100 to get an additional $100 of free Confluent Cloud usage: https://cnfl.io/try-confluent-cloud-e...
► Promo code details: https://cnfl.io/podcast100-details-ep...
TIMESTAMPS
0:00 - Intro
1:43 - What is Forecasty.ai?
3:20 - Using AI techniques for forecast modeling
9:51 - Migrating from batch to real-time stream processing
13:08 - Getting started with Apache Kafka
23:52 - Building kash.py—a Python-based Kafka shell
31:10 - Future plans for using Kafka
35:44 - It's a wrap!
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#streamprocessing #ai #apachekafka #microservices #kafka #confluent
Информация по комментариям в разработке