Скачать или смотреть Launching Python Applications on Peta scale Massively Parallel Systems | SciPy 2016 | Yu Feng

We introduce a method to launch python applications at near native speed on large high performance computing systems.
The python run-time and other dependencies are bundled and delivered to computing nodes via a broadcast operation. The interpreter is instructed to use the local version of the files on the computing node, removing the shared file system as a bottleneck during the application start-up.
Our method can be added as a preamble to the traditional job script, improving the performance of user applications in a non-invasive way. Further more, our method allows us to implement a three-tier system for the supporting components of an application, reducing the overhead of runs during the development phase of an application. The method is used for applications on Cray XC30 and Cray XT systems up to full machine capability with an overhead typically less than 2 minutes. We expect the method to be portable to similar applications in Julia or R. We also hope the three tier system for the supporting components provides some insight for the container based solutions for launching applications in a development environment. We provide the full source code of an implementation of the method at url{https://github.com/rainwoodman/python...}. Given that large scale Python applications can be launch extremely efficiently on state of art super-computing systems, it is the time for the high performance computing community to seriously consider building complicated computational applications at large scale with Python.