Скачать или смотреть Multi GPU Training with TensorFlow on Piz Daint - Day 1 - Morning

The Piz Daint supercomputer at CSCS provides an ideal platform for supporting intensive deep learning workloads as it comprises thousands of Tesla GPU compute nodes communicating through a high-speed interconnect. In this two-day course, we will look at how to run distributed deep learning workloads with TensorFlow on Piz Daint. We will use simple examples to demonstrate best practices for building efficient input pipelines to maximize the throughput of deep learning models with TensorFlow.

Supporting materials and instructions on how to run TF with JupyterLab at CSCS:
https://github.com/eth-cscs/tensorflo...

The course will focus on two main subjects: reading data through input pipelines and asynchronous distributed training.

More specifically, the following topics will be covered:

Running TensorFlow-2 on Piz Daint.
Creating efficient input pipelines with TensorFlow's Dataset API for optimizing throughput on Piz Daint.
-Reading and writing data as TFRecords files.
-Understanding the stochastic gradient descent and distributed synchronous stochastic gradient descent algorithms.
-Running distributed training with TensorFlow using the ring allreduce algorithm implemented in Horovod.
-TensorFlow's operations statistics with TensorBoard.

This course is an update from last years’s workshop Efficient and Distributed Training with TensorFlow on Piz Daint. In this year's edition we will be using TensorFlow-2, TensorBoard, and we will learn how to run multi GPU training from JupyterLab notebooks.

Instructors
Dr. Rafael Sarmiento (Computational Scientist, CSCS)
Henrique Mendonça (Computational Scientist, CSCS)