Скачать или смотреть vLLM Office Hours - SOTA Tool-Calling Implementation in vLLM - November 7, 2024

In this session, we dive deep into the implementation of state-of-the-art (SOTA) tool-calling in vLLM. We discuss the importance of tools and functions in open-source AI and provide insights into the challenges and solutions around OpenAI-style tools in vLLM.

During the Q&A, we explored questions around serving multiple models on a single vLLM server, the benefits of partial JSON decoding from a delta stream, and specific application examples where partial visibility into JSON arguments proves advantageous. Additional questions covered plans for supporting OpenAI’s "strict" field in tool definitions for structured output, best practices for tool-calling formats in model fine-tuning, and the choice of OpenAI's chat completions API as a standard over the assistant’s API for tool selection.

Session slides: https://docs.google.com/presentation/...

Stay connected and join our bi-weekly vLLM Office Hours to learn about the latest updates: https://hubs.li/Q02Y5Pbh0