Скачать или смотреть Managing Artifacts at Scale for CI and Data Processing - Caelan Urquhart & Julie Vogelman

Managing Artifacts at Scale for CI and Data Processing - Caelan Urquhart, Pipekit & Julie Vogelman, Intuit

Almost all workflows require passing data along a string of steps in the workflow, using the output of one step as the input to a subsequent step. Managing these data artifacts along the workflow lifecycle gets increasingly cumbersome as workflows grow in size and complexity, and as the number of workflows and users accumulate on a given cluster. Because of these factors, maintaining a robust artifact management strategy is critical for scaling up any Argo Workflows instance. In this talk, we will show the best practices we’ve learned from using Argo Workflows at Intuit and Pipekit that address these challenges. We will detail how to configure artifact repos with Key-only Artifacts to simplify workflow management, and how to use Artifact Garbage Collection (GC) strategies to automate artifact cleanup after a successful run, plus explain the benefits of adopting each of these features. We'll then demonstrate how to implement these features in a couple example workflows, covering both data and CI use cases. We hope that listeners of this talk will learn how to level up their artifact management Argo Workflows, whether they are CI or data users, plus feel confident trying out more complicated artifact management approaches for their own workflows.