Denoise Within, Fuse Across: Reliable Multi-Level Matching for Multi-modal Entity Linking

摘要

Multi-modal Entity Linking (MEL) utilizes multi-modal context to associate ambiguous mentions with corresponding entities in a knowledge base. Previous methods have primarily focused on multi-modal feature fusion to enhance mention and entity representations. Recently, multi-level matching methods have emerged as an effective alternative, which independently computes intra-modal and inter-modal matching scores and combines them at the decision level instead of learning a unified multi-modal representation. However, existing methods face two limitations. First, modality-specific noise reduces the reliability of individual matching decisions. Second, uniformly averaging decision scores ignores that the contribution of each modality varies across samples. To address these challenges, we propose \texttt{DeFu}, a plug-and-play \underline{De}noise-within and \underline{Fu}se-across framework for reliable MEL. Specifically, to enhance the reliability of individual decision, \texttt{DeFu} introduces a multi-modal variational encoder and information regularization objective, which suppresses modal noise while preserving discriminative information. To improve the reliability of decision fusion, \texttt{DeFu} further proposes a confidence-aware gating network. It dynamically assigns weights to intra-modal and inter-modal matching scores by prediction uncertainty derived from multiple variational samples. \texttt{DeFu} can be seamlessly integrated with existing multi-level matching methods. Extensive experiments on three benchmark datasets demonstrate that \texttt{DeFu} consistently achieves optimal performance and exhibits strong generalizability.