Workflows

Whenever one has to deal with multiple jobs on a HPC system, the idea of automating parts or all of the job management process involves describing and implementing so-called 'workflows'. Options for managing workflows are numerous and range from using basic scheduler features such as job arrays and job dependencies, up to using a complex system managed by a central, multi-user, database. This session aims at guiding participants towards the right tool for their use and help them reduce the time they spend managing their jobs by automating what can be automated and follow best practices.

0:00 Introduction
4:40 Workflow management systems
7:46 GNU Make
14:20 Types of workflows
15:24 atools
18:26 slurmdagman
19:58 Makeflow
21:33 Snakemake
22:53 Cycl
24:40 Full featured systems
26:53 Maestro
45:35 Maestro demo


More information: https://indico.cism.ucl.ac.be/event/1...