Скачать или смотреть Serving LLMs on Kubernetes: Common challenges

Members of Red Hat OpenShift AI’s engineering team describe some of the common challenges serving large language models (LLMs) on Kubernetes, such as:
Sharing of computational resources across multiple LoRA (Low-Rank Adaptation) adapters or use cases based on the same foundation model.
Shortening model loading time, particularly with massive models.
Providing more efficient ways to fetch models from an Open Container Initiative (OCI) image registry.
Hear how these challenges are being addressed in collaborative upstream open source Kubernetes, Kserve, and vLLM working groups thorough projects and features like Modelcars, llm-instance-gateway, and OCI Volume Source.

More about:
Building and deploying a ModelCar container (Red Hat Developer, Jan. 2025): https://developers.redhat.com/article...
vLLM (Red Hat topics, Jan. 2025): https://www.redhat.com/en/topics/ai/w...
Kserve (Red Hat Developer, Mar. 2024): https://developers.redhat.com/article...
Red Hat AI: https://red.ht/AI
https://cloud.redhat.com/learn/how-ru...