NLP Seminar Series - Text Summarization and Evaluation in the Era of GPT-3 - Tanya Goyal

Titel: Text Summarization and Evaluation in the Era of GPT-3
Speaker: Tanya Goyal, PhD student at University of Texas at Austin
Abstract: The recent success of zero- and few-shot prompting with models like GPT-3 has led to a paradigm shift in NLP research. We study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how zero-shot GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics, e.g. recently proposed QA- or entailment-based factuality approaches, cannot reliably evaluate zero-shot summaries. Finally, we discuss future research challenges beyond generic summarization, specifically, keyword- and aspect-based summarization, showing how dominant fine-tuning approaches compare to zero-shot prompting.

AI Sweden is the Swedish national center for applied artificial intelligence. Our mission is to accelerate the use of AI for the benefit of our society, our competitiveness, and for everyone living in Sweden.

Website: https://ai.se
Contact: [email protected]

Follow us
Newsletter: https://www.ai.se/en/AI-Sweden-newsle...
LinkedIn:   / aisweden  
Medium:   / ai