Скачать или смотреть Google PaLM: Scaling Language Modeling with Pathways

Google PaLM is a 540B parameter dense Transformer language model trained on 780B tokens of high-quality, diverse text. It has been trained for 3 different sizes: 8B, 62B, and 540B using 6144 TPU v4 chips using Pathways, a new ML system for highly efficient training across multiple TPU (Tensor Processing Units) Pods. When it was introduced it led to SOTA few-shot learning results on hundreds of NLU and NLG benchmarks. This includes steep performance increase for Big-Bench tasks, and also significant improvements for Multilingual NLG and source code generation capabilities. It has also been shown to generate great explanations using chain-of-thought prompting for explaining jokes or logical inference.

Here is the agenda for the video

00:00:00 What is PaLM?
00:02:50 How does PaLM’s model architecture look like?
00:04:45 What is the Pathways system?
00:07:49 How does PaLM perform on popular English NLP tasks?
00:09:29 How does PaLM perform on Massive Multitask Language Understanding?
00:10:45 How does PaLM perform on Big bench tasks?
00:13:35 How does PaLM perform on Reasoning tasks?
00:16:52 How does PaLM perform on Coding tasks?
00:26:07 How does PaLM perform on Translation tasks?
00:29:27 How does PaLM perform on Multilingual NLG Tasks?
00:32:27 Is PaLM memorizing data?
00:35:40 What kinds of explanations PaLM can generate?
00:37:35 How does PaLM perform from a bias and toxicity perspective?

For more details, please look at https://arxiv.org/pdf/2204.02311.pdf and https://ai.googleblog.com/2022/04/pat...

Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham et al. "Palm: Scaling language modeling with pathways." arXiv preprint arXiv:2204.02311 (2022).