DINO: Emerging Properties in Self-Supervised Vision Transformers (paper illustrated)

➤Emerging Properties in Self-Supervised Vision Transformers - DINO

➤ Paper Abstract:
In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

➤ Paper Link: https://arxiv.org/pdf/2104.14294.pdf
➤ Official Code: https://github.com/facebookresearch/dino
➤ Official Blog: https://bit.ly/3hWN7Xh

➤ Video Outline:
0:00 - Introduction
0:42 - Self-training and Knowledge Distillation
2:02 - Self-Supervised Learning
2:49 - Self Distillation with No labels (DINO) main idea
5:52 - Evaluation and Results
7:02 - Interesting Properties

➤ AI Bites
YouTube:    / aibites​  
Twitter:   / ai_bites​  
Patreon:   / ai_bites​  
Github: https://github.com/ai-bites​

➤ Swin Transformers:    • Swin Transformer: Hierarchical Vision...  
➤ Vision Transformers (ViT):    • Vision Transformer (ViT) - An Image i...  
➤ Data Efficient Image Transformer (DeiT):    • DeiT - Data-efficient image transform...  

📚 📚 📚 BOOKS I HAVE READ, REFER AND RECOMMEND 📚 📚 📚
📖 Deep Learning by Ian Goodfellow - https://amzn.to/3Wnyixv
📙 Pattern Recognition and Machine Learning by Christopher M. Bishop - https://amzn.to/3ZVnQQA
📗 Machine Learning: A Probabilistic Perspective by Kevin Murphy - https://amzn.to/3kAqThb
📘 Multiple View Geometry in Computer Vision by R Hartley and A Zisserman - https://amzn.to/3XKVOWi

Music: https://www.bensound.com

#deeplearning #machinelearning #aibites