Скачать или смотреть Making PySpark Amazing—From Faster UDFs to Graphing! (Holden Karau and Bryan Cutler)

Holden is a transgender Canadian open source developer advocate @ Google with a focus on Apache Spark, BEAM, and related "big data" tools. Bryan Cutler is a software engineer at IBM's Spark Technology Center, where he works on big data analytics and machine learning systems.

PySpark is getting awesomer in Spark 2.3 with vectorized UDFs, and there is even more wonderful things on the horizon (and currently available as WIP packages). This talk will start by illustrating how to use PySpark’s new vectorized UDFs to make ML pipeline stages. Since most of us use Python in part because of its wonderful libraries, like pandas, numpy, and antigravity*, it’s important to be able to make sure that our dependencies are available on our cluster. Historically there’s been a few If there is time near the end we will talk about how to expose your Python code to Scala so everyone can use your fancy deep learning code (if you want them to). *Ok maybe not a real thing, but insert super specialized domain specific library you use instead 🙂

To learn more: https://docs.databricks.com/spark/lat...
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unifie...

Connect with us:
Website: https://databricks.com
Facebook:   / databricksinc  
Twitter:   / databricks  
LinkedIn:   / databricks  
Instagram:   / databricksinc   Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...