Скачать или смотреть MM24 ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding

The 32nd ACM International Conference on Multimedia (ACM MM)

ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding

In this paper, we propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ReSVG) model, to improve the model's understanding of relation and semantic in multiple instances. Firstly, we enhance the model's understanding of fine-grained semantics by injecting semantic prior information derived from text queries into the model. This is achieved by leveraging text-to-image generation models to produce images representing the semantic attributes of target objects described in queries. Secondly, we tackle the lack of training samples with multiple distractions by introducing a relation-sensitive data augmentation method. This method generates additional training data by synthesizing images containing multiple objects of the same category and pseudo queries based on their spatial relationships. The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions.

Project Page: https://minghangz.github.io/publication/re...
Paper Link: https://arxiv.org/pdf/2408.16314
Code: https://github.com/minghangz/ResVG

Authors: Minghang Zheng, Jiahua Zhang, Qingchao Chen, Yuxin Peng
, Yang Liu∗
Organisation: Wangxuan Institute of Computer
Technology, Peking University; National Institute of Health Data
Science, Peking University