Скачать или смотреть Scaling Pandas with Apache Spark + Koalas for ML at Virgin Hyperloop One

TL;DR - Virgin Hyperloop One engineers discuss the goals, implementation, and outcome of moving from Pandas code to Koalas code and using MLflow. Lots of code, notebooks, demos, etc.

Hyperloop is a new mode of transportation designed to eliminate the barriers of distance and time for both people and freight. It can travel at speeds approaching 700mph, connecting cities like metro stops — and it has zero direct emissions. By combining an ultra-efficient electric motor, magnetic levitation, and a low-drag environment, the Virgin Hyperloop One (VHO) system can reach airline speeds for up to 10X less energy.

Achieving such performance requires high fidelity data and complex analyses, skyrocketing data size and processing times. To reduce the time-to-insight from data to decisions, VHO has recently translated data processing scripts previously written in ‘pandas’ to ‘Koalas’ – a library open-sourced by Databricks and achieved significant scale in Databricks’ platform. In addition, VHO leverages MLflow, another open-source project by Databricks to easily track and manage model results.

Come hear Patryk Oleniuk, Software Engineer at Virgin Hyperloop (VHO) discuss how VHO has dramatically reduced processing time by 95%, while changing less than 1% of previously single-threaded, pandas-based python code. Attendees of this webinar will learn:

How VHO leverages public and private transportation data to optimize Hyperloop design
How to ‘Sparkify’ (scale) your pandas code by using ‘Koalas’ with minimal code changes
How to use ‘Koalas’ and MLflow for sweeping machine learning models and experiment results
Featured Speakers
Patryk Oleniuk, Lead Data Engineer, Virgin Hyperloop One
Yifan Cao, Senior Product Manager, Databricks

Hosted by Cyrielle Simeone, Senior Product Marketing Manager, Databricks


Slides: https://www.slideshare.net/databricks...

Koalas Notebook: https://pages.databricks.com/rs/094-Y... Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...