Скачать или смотреть FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models (ICCV 2025)

In this work, we introduce FedMVP: a multimodal visual prompt tuning framework for VLMs in Federated Learning. FedMVP conditions on visual features and rich textual attributes of LLMs, generating multimodal prompts to fine-tune the CLIP image encoder for classification. In addition, we propose a novel PromptFormer network, consisting of a multi-head cross attention (MHCA) module followed by a feed-forward network (FFN) to generate context-aware multimodal visual prompts from class-agnostic LLM attributes and patch-level visual embeddings. We only share the parameters of PromptFormer in between clients and server using FedAvg algorithm. Moreover, we enable a second stage training with LoRA fine-tuning of PromptFormer, switching off its own update, below a certain loss threshold of each specific clients, reducing the complexity by 267 times.