How to Build a Reliable Kafka Data Processing Pipeline, Focusing on Contention, Uptime and Latency

InfoQ Dev Summit Boston, a two-day conference of actionable advice from senior software developers hosted by InfoQ, will take place on June 24-25, 2024 Boston, Massachusetts.

Deep-dive into 20+ talks from senior software developers over 2 days with parallel breakout sessions. Clarify your immediate dev priorities and get practical advice to make development decisions easier and less risky.

Register now: https://bit.ly/47tNEWv
---------------------------------------------------------------------------------------------------
Video with transcript included on InfoQ: https://bit.ly/496fhXv

Lily Mara shares how OneSignal improved the performance and maintainability of its highest-throughput HTTP endpoints (backed by a Kafka consumer in Rust) by making it an asynchronous system. She shares how metrics changed when the system went from sync to async, and what unique sharding strategies were used to maximize concurrency and performance, while maintaining consistency for Kafka consumers.

#ApacheKafka #Performance #Reliability
------------------------------------------------------------------------------------------------------
Follow InfoQ:
- Mastodon: https://techhub.social/@infoq
- Twitter:   / infoq  
- LinkedIn:   / infoq  
- Facebook:   / infoqdotcom  
- Instagram: @infoqdotcom