Скачать или смотреть Systems that enable data agility

Talk given at Strata + Hadoop World, London, 6 May 2015.

Slides: https://speakerdeck.com/ept/systems-t...

Abstract:

Congratulations, you’ve got a lot of data! Now what? How do you enable your organisation to create value from that data? What tools do your data scientists need in order to create data-driven products? How do you empower your teams to experiment, to innovate, and to be agile in response to customer needs?

In this session we will discuss LinkedIn’s approach to solving these problems, and the open source tools that were created at LinkedIn to support data agility in a large organisation. The approach boils down to a few simple ideas:

1. Make all data available centrally, in real time. If it’s too difficult to access data across silos, you can’t derive value from it. For this purpose, LinkedIn created Apache Kafka, which can be the data exchange backbone of your organisation.

2. Make it easy to analyse and process that data. You’ve hired smart people, now empower them to easily try out new ideas for data-driven products, and rapidly get them into production if they are good. To support this, LinkedIn created Apache Samza, a stream processing framework that provides powerful, reliable tools for working with data in Kafka.

Since Kafka and Samza are open source, you can apply these lessons and start implementing your own agile data pipeline today. In this talk you’ll learn about:

How Kafka and Samza reliably scale to millions of messages per second
What kinds of real-time data problems you can solve with Samza
How Samza compares to other stream processing frameworks
How data streams support collaboration between different data science, product and engineering teams within an organisation
Lessons learnt on how to move fast without breaking things