Скачать или смотреть (Efficient) Data Exchange with "Foreign" Ecosystems - Uwe Korn [PyData Frankfurt]

Recorded at PyData Frankfurt Meetup.
Join us on meetup: https://www.meetup.com/PyData-Frankfurt
Twitter:   / pydataffm  

As a Data Scientist/Engineer in Python, we focus in our work to solve problems with large amounts of data but still stay in Python. This is where we are the most effective and feel comfortable. Libraries like Pandas and NumPy provide us with efficient interfaces to deal with this data while still getting optimal performance. The main problem appears when we have to deal with systems outside of our comfort ecosystem. We need to write cumbersome and mostly slow conversion code that ingests data from there into our pipeline until we can work efficiently. Using Apache Arrow and Parquet as base technologies, we get a set of tools that eases this interaction and also brings us a huge performance improvement. As part of the talk we will show a basic problem where we take data coming from a Java application through Python into using these tools.

Uwe is a Data & Machine Learning Engineer and open source developer. While he started his career building machine learning models, he quickly came into the business of developing the underlying platform that is needed to build and run successful data products. His main focus is on building software artifacts and a culture that supports a highly effective collaboration between data scientists and engineers.
As part of his work on providing an efficient data interchange between systems, he became a core committer to the Apache Parquet and Apache Arrow projects.


www.pydata.org

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVi...