Scaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers - Jack Ong

Scaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers - Jack Min Ong, Jina AI

With the rise of AI and machine learning applications, GPU resources have become a critical bottleneck in scaling infrastructure to efficiently serve AI workloads. Kubernetes, an open-source container orchestration platform, provides a solution to this problem through the NVIDIA device plugin which allows multiple containers to share access to GPU devices. In this talk, we will explore how Kubernetes can be used to efficiently scale AI workloads by sharing GPU resources across multiple containers. We will discuss the challenges of GPU resource management, explore various techniques for optimizing GPU usage and set resource limits to ensure fair and efficient allocation of GPU resources among containers. By the end of this talk, attendees will have a solid understanding of how Kubernetes can be used to share GPU resources across multiple containers, allowing them to make the most of their GPU investments and achieve faster, more accurate results in their AI applications.