Скачать или смотреть Security for Real-Time Data Stream Processing with Confluent Cloud

https://cnfl.io/podcast-episode-241 | Streaming real-time data at scale and processing it efficiently is critical to cybersecurity organizations like SecurityScorecard. Jared Smith, Senior Director of Threat Intelligence, and Brandon Brown, Senior Staff Software Engineer, Data Platform at SecurityScorecard, discuss their journey from using RabbitMQ to open-source Apache Kafka® for stream processing. As well as why turning to fully-managed Kafka on Confluent Cloud is the right choice for building real-time data pipelines at scale.

SecurityScorecard mines data from dozens of digital sources to discover security risks and flaws with the potential to expose their client’ data. This includes scanning and ingesting data from a large number of ports to identify suspicious IP addresses, exposed servers, out-of-date endpoints, malware-infected devices, and other potential cyber threats for more than 12 million companies worldwide.

To allow real-time stream processing for the organization, the team moved away from using RabbitMQ to open-source Kafka for processing a massive amount of data in a matter of milliseconds, instead of weeks or months. This makes the detection of a website’s security posture risk happen quickly for constantly evolving security threats. The team relied on batch pipelines to push data to and from Amazon S3 as well as expensive REST API based communication carrying data between systems. They also spent significant time and resources on open-source Kafka upgrades on Amazon MSK.

Self-maintaining the Kafka infrastructure increased operational overhead with escalating costs. In order to scale faster, govern data better, and ultimately lower the total cost of ownership (TOC), Brandon, lead of the organization’s Pipeline team, pivoted towards a fully-managed, cloud-native approach for more scalable streaming data pipelines, and for the development of a new Automatic Vendor Detection (AVD) product.

Jared and Brandon continue to leverage the Cloud for use cases including using PostgreSQL and pushing data to downstream systems using CSC connectors, increasing data governance and security for streaming scalability, and more.

EPISODE LINKS
► SecurityScorecard Case Study: https://cnfl.io/securityscorecard-cas...
► Building Data Pipelines with Apache Kafka and Confluent: https://cnfl.io/streaming-data-pipeli...
► Kris Jenkins’ Twitter:   / krisajenkins  
► Streaming Audio Playlist:    • Streaming Audio Podcast | Apache Kafka®, C...  
► Join the Confluent Community: https://cnfl.io/join-the-community-24...
► Learn more with Kafka tutorials, resources, and guides at Confluent Developer: https://cnfl.io/confluent-developer-2...
► Live demo: Intro to Event-Driven Microservices with Confluent: https://cnfl.io/demo-intro-to-event-d...
► Use PODCAST100 to get an additional $100 of free Confluent Cloud usage: https://cnfl.io/try-cloud-241-episode
► Promo code details: https://cnfl.io/podcast100-details-24...

TIMESTAMPS
0:00 - Intro
1:42 - What is SecurityScorecard?
6:36 - Scaling with Kafka clusters
16:36 - Moving from RabbitMQ to Kafka and Confluent Cloud
24:50 - Upsides of using Kafka for data processing
27:12 - Downsides of ingesting large amounts of data
37:08 - Leveraging the Cloud to increase data governance and security
41:25 - Helping teams create and consume new feeds of data
47:01 - It's a wrap!

ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.

#streamprocessing #apachekafka #kafka #confluent