Скачать или смотреть Beyond Sentences & Paragraphs: Towards Document-level & Multi-doc Understanding/Arman Cohan (UW&AI2)

ABSTRACT:
In this talk, I will describe a few of our recent works on developing Transformer-based models that target document-level and multi-document natural language tasks. I will first introduce Specter, a method for producing document representations using a Transformer model that incorporates document-level relatedness signals. I will then discuss Longformer, an efficient transformer model that can process and contextualize information across inputs of several thousands of tokens. This is achieved by replacing the full self-attention mechanism in transformers with sparse local and global attention patterns. I will then discuss two of our efforts in developing general language models for multi-document tasks. CDLM is an encoder-only model for multi-document tasks that uses multiple related documents during pretraining and pretrains a dynamic global attention for multi document tasks. I will then briefly discuss our recent work on PRIMER, a general pre-trained model for multi-document summarization tasks. Finally, I will discuss some of our other efforts on creating challenging document level benchmarks.

BIO-DATA:
Arman Cohan is a Research Scientist at the Allen Institute for AI (AI2) and an Affiliate Assistant Professor at University of Washington. His research focused on developing natural language processing (NLP) models for document-level and multi-document understanding, natural language generation and summarization as well as information discovery and filtering.He is also interested in applications of NLP in science and health domains. His research has been recognized with multiple awards including a best paper award at EMNLP 2017, an honorable mention at COLING 2018, and Harold N. Glassman Distinguished Doctoral Dissertation award in 2019.

Slides link (via Speakerdeck): https://speakerdeck.com/wingnus/beyon...
Related Link: https://wing-nus.github.io/ir-seminar...