Скачать или смотреть ManGouard: Unlocking the Hidden Potential of CLIP for Deepfake Detection | FIT3183 Milestone 3

Monash University | FIT3183 – Malicious AI & Dark-Side Security
Team ManGouard: Lee Cheng Yan & Chao Zhen Hao

Project Title: Unlocking the Hidden Potential of CLIP in Generalizable Deepfake Detection

In this project, we explore how OpenAI’s CLIP (ViT-L/14) that was originally designed for image-text understanding which can be repurposed as a deepfake detector.

Instead of retraining a new model from scratch, we focused on making CLIP transparent and explainable by adding:
• Attention Heatmaps — visualize where the model focuses
• Occlusion Sensitivity Maps — reveal regions that influence fake detection
• Robustness and Consistency Tests — assess model stability under image perturbations

Our goal: to build a trustworthy and interpretable deepfake detection pipeline that explains why an AI model labels something as fake.

Technologies Used:
Python • PyTorch • Transformers (CLIP) • Google Colab • OpenAI ViT-L/14

Research Paper Reference:
“Unlocking the Hidden Potential of CLIP in Generalizable Deepfake Detection” (Yermakov, Cech & Matas, 2025)

Project Components:
• Deepfake detection using CLIP’s visual encoder
• Visualization via attention & occlusion maps
• Insights on model generalization and interpretability

Colab Notebook (Final Report): https://colab.research.google.com/dri...

Key Takeaway:
Even without retraining, CLIP demonstrates strong generalization for deepfake detection and with added interpretability tools, we can finally see how it makes those decisions.

#FIT3183 #DeepfakeDetection #CLIP #AIExplainability #TrustworthyAI #MonashUniversity #ManGouard