Fine-Tuning Vision Transformers for handling the Challenges of Small Datasets

In the realm of computer vision, Vision Transformers (ViT) have demonstrated exceptional performance on large datasets but face challenges when applied to smaller ones like the PH2 skin lesion dataset. This study proposes a novel approach to enhance ViT's performance on PH2 by leveraging transfer learning from the ISIC dataset. Through a series of experiments, ranging from training ViT on PH2 from scratch to fine-tuning specific layers, we demonstrate significant improvements in accuracy. Fine-tuning all layers yielded the highest accuracy. Our findings highlight the efficacy of transfer learning and tailored fine-tuning strategies for enhancing ViT's adaptability to smaller datasets. Furthermore, this research prompts contemplation on the broader landscape of adapting pretrained models to small datasets, inviting further exploration.