Скачать или смотреть Distributed file-systems made easy with Python's fsspec

🔊 Recorded at PyCon DE & PyData 2025, April 24, 2025
https://2025.pycon.de/program/DEHZHK/

🎓 Python's fsspec library transforms complex distributed storage interactions into simple, consistent operations while enabling advanced data management capabilities.

Speakers:
Einat Orr, Barak Amar

Description:
This presentation explores how Python's fsspec library simplifies interactions with distributed file systems and cloud storage. Amar demonstrates how fsspec provides a unified interface for working with files across different storage backends, from local systems to cloud providers like AWS S3 and Google Cloud Storage. The talk examines fsspec's core functionality, including its consistent API for file operations, caching capabilities, and integration with popular data science libraries such as pandas. Through practical code examples, Amar illustrates how fsspec reduces complexity when handling distributed data storage by abstracting away provider-specific implementations. The presentation includes a detailed examination of fsspec's extensibility, demonstrated through a custom file system implementation and integration with lakeFS, an open-source versioning platform for data lakes. The discussion covers advanced features like transaction support and caching strategies, with particular attention to real-world applications in data engineering workflows. The session concludes with a live demonstration of fsspec's capabilities in conjunction with lakeFS, showcasing practical applications of atomic transactions and branching operations in data management scenarios.

⭐️ About PyCon DE & PyData:
The PyCon DE & PyData conference unite the Python, AI, and data science communities, offering a unique platform for collaboration and innovation. The PyCon DE & PyData 2025 conference, provided an exceptional experience, fostering deeper connections within the Python community while showcasing advancements in AI and data science. Attendees enjoyed a diverse and engaging program, solidifying the event as a highlight for Python and AI enthusiasts nationwide.

Follow us:
• LinkedIn:   / 28908640  
• X: https://www.x.com/pyconde

Links:
• Conference website: http://pycon.de
• Other sessions: https://2025.pycon.de/talks/

The conference is organized by
• Python Softwareverband e.V.: http://pysv.org
• NumFOCUS Inc.: http://numfocus.org
• Pioneers Hub gemeinnützige GmbH: http://pioneershub.org


If you enjoyed this session, please like, comment, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!

Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareDevelopment #LLMs #Community

Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.

About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.

NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.

Pioneers Hub gemeinnützige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
www.pydata.org

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.