Exploring Variational Information Bottleneck and Quality-Guided Modality Selection for Multi-modal Entity Linking

摘要

Multi-modal Entity Linking (MEL) aims to leverage multi-modal context to associate ambiguous mentions with their corresponding entities in a knowledge base. Existing MEL methods attempt to bridge the modality gap by introducing multi-modal interaction and fusion mechanisms, and perform entity linking through multi-level matching. However, these approaches still suffer from two key challenges: (i) modal noise—noisy signals in the multi-modal information of mentions or entities may lead to incorrect linking; and (ii) dynamic modality selection—how to adaptively determine the importance of different modalities for each sample. To address these issues, we propose VQMEL, a novel MEL framework based on Variational information bottleneck (VIB) and Quality-guided modality selection. Specifically, VQMEL consists of two modules: (1) a VIB-enhanced modality expert training module, which employs multi-modal variational encoders to probabilistically model modality representations, and introduces a multi-modal information bottleneck regularization loss to suppress modality-specific noise, while independently training three modality expert networks; and (2) a quality-guided modality gating network, which dynamically selects the optimal combination of expert networks for each sample based on the quality of multi-modal information from both mentions and entities. We evaluated VQMEL by integrating three state-of-the-art multi-modal matching methods, the results on three benchmark datasets demonstrate that our approach consistently outperforms existing state-of-the-art baselines and exhibits strong generalization capability.