Скачать или смотреть Nucleus Sampling: The Curious Case of Neural Text Degeneration (Research Paper Walkthrough)

#textgeneration #naturallanguageprocessing #decoding
This ICLR 2020 paper introduces a novel text decoding strategy called Nucleus Sampling (Top-p sampling). This strategy overcomes the limitation of generating bland, repetitive and incoherent text from other decoding strategies like Beam Search, Top-k sampling, Greedy Decoding and Softmax with Temperature. This text decoding strategy enhances diversity without sacrificing fluency and coherence.

⏩ Abstract: Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive.In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

Please feel free to share out the content and subscribe to my channel :)
⏩ Subscribe -    / @techvizthedatascienceguy  

⏩ OUTLINE:
0:00 - Background on text generation and decoding strategies
2:21 - Abstract
3:33 - Beam Search Limitation
5:21 - Maximization-based decoding
6:53 - Example generations from multiple text decoding methods (Beam Search, Pure Sampling, Sampling with Temperature, Top-k, Top-k with Temperature, Nucleus Sampling)
10:24 - Flat Distribution Vs Peaked Distribution

⏩ Paper Title: The Curious Case of Neural Text Degeneration
⏩ Paper: https://arxiv.org/abs/1904.09751
⏩ Author: Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi
⏩ Organisation: Allen School of Computer Science & Engineering, University of Washington, Allen Institute for Artificial Intelligence, University of Cape Town

⏩ IMPORTANT LINKS
Human-Centric Text Generation Evaluation Methods -    • Evaluation of Text Generation: A Survey | ...  
BLEURT: Learning Robust Metrics for Text Generation -    • BLEURT: Learning Robust Metrics for Text G...  

*********************************************
If you want to support me financially which totally optional and voluntary :) ❤️
You can consider buying me chai ( because i don't drink coffee :) ) at https://www.buymeacoffee.com/TechvizC...

*********************************************
⏩ Youtube -    / techvizthedatascienceguy  
⏩ Blog - https://prakhartechviz.blogspot.com
⏩ LinkedIn -   / prakhar21  
⏩ Medium -   / prakhar.mishra  
⏩ GitHub - https://github.com/prakhar21
*********************************************

Please feel free to share out the content and subscribe to my channel :)
⏩ Subscribe -    / @techvizthedatascienceguy  

Tools I use for making videos :)
⏩ iPad - https://tinyurl.com/y39p6pwc
⏩ Apple Pencil - https://tinyurl.com/y5rk8txn
⏩ GoodNotes - https://tinyurl.com/y627cfsa

#techviz #datascienceguy #nlproc #researchpaper #explained #languagemodel #sampling #nlg #nucleussampling