Deep Papers Episode 3 - Toolformer: Training LLMs To Use Tools

Deep Papers is a podcast series featuring deep dives on today’s seminal AI papers and research. Hosted by AI Pub creator Brian Burns and Arize AI founders Jason Lopatecki and Aparna Dhinakaran, each episode profiles the people and techniques behind cutting-edge breakthroughs in machine learning.

In this episode, we interview Timo Schick and Thomas Scialom, the Research Scientists at Meta AI behind Toolformer. "Vanilla" language models cannot access information about the external world. But what if we gave language models access to calculators, question-answer search, and other APIs to generate more powerful and accurate output? Further, how do we train such a model? How can we automatically generate a dataset of API-call-annotated text at internet scale, without human labeling?

Timo and Thomas give a step-by-step walkthrough of building and training Toolformer, what motivated them to do it, and what we should expect in the next generation of tool-LLM powered products.

0:09: Intro to Deep Papers, Toolformer, podcast hosts and guests.
1:38: "Elevator Pitch" for the Toolformer paper: Current language models can't use tools, nor interact with the external world! What if we could teach them to read from and use APIs?
2:39: What are examples of non-tool LLM limitations? What motivated the authors to create Toolformer?
4:45: Aside from e.g. calculators, where do the most powerful LLMs struggle when compared to tool-based ones? And why? Time-based information, and short context windows as a result of n^2 attention.
7:25: Step by step, how does Toolformer actually use tools, given LLMs are only able to generate the next token? Special API tokens, etc.
9:45: How does this compare with mixture-of-experts models? One way to think about Toolformer is a mixture-of-experts model, where some of the experts are API calls.
10:45: Why start with the set of tools used in Toolformer? What other tools could tool-based LLMs use in the future?
13:08: Where does this research lead, long term? Not just querying APIs, but taking actions in the external world.
14:45: Question: The meat of the paper - how did the authors automatically generate a massive text dataset annotated with API calls, to train Toolformer?
16:25: Answer, Part I The meat of the paper - how did the authors automatically generate a massive text dataset annotated with API calls, to train Toolformer?
17:10: Step 1: use a language model to sample a massive number of API calls, to generate a poorly-annotated dataset.
18:15: Step 1: An example of using LLMs to auto-annotate text with API calls, including the prompt.
19:30: When would the model know when to call a tool rather than output more text? At this step, the model doesn't know! The model radically oversamples API calls - 90-99% of the calls at this point are useless.
21:00: Step 2: How to filter down the 90-99% of the oversampled API calls in the dataset? How to do so automatically, and at scale? Using perplexity and explainability of the following text as a filter.
23:15: How adding tools allows LLMs to avoid hallucination and give greater context to their "thinking" process.
24:35: Now that you've generated a dataset of API-call-annotated text, how do you train Toolformer?
26:35: What are the next frontiers in research for tool-augmented LLMs? Multiple tools, called recursively and iteratively. Also: using the language model itself as a tool!
28:30: What sort of products will emerge, using Toolformer?
29:40: In a world where people are using tool-based LLMs, should there be constraints to the kind of calls users can make? Toolformer becomes more dangerous when it can take actions rather than just access information.
31:20: Tool use is much more computationally and time efficient than querying the LLM itself! Adding that to the loss is part of future work.
32:30: Where can listeners follow your work for updates and learn more about Toolformer?

Additional Resources:
Read the paper: https://arxiv.org/abs/2302.04761
Listen & Subscribe: https://arize.com/blog/toolformer-lar...
Learn more about Arize and sign up for a free account: https://www.arize.com
Learn about how to monitor text-based LLMs: https://arize.com/blog-course/generat...