Generative Reasoning for Multi-Modal Entity Alignment with Open Knowledge-Enhanced Quantization

摘要

Multi-Modal Entity Alignment (MMEA) aims to identify entities referring to the same real-world object across heterogeneous multi-modal knowledge graphs. Existing methods primarily learn entity representations through cross-modal fusion and perform alignment based on embedding similarity. Despite their progress, these methods often struggle with highly heterogeneous entities due to insufficient prior knowledge, while their reliance on static embedding spaces limits their reasoning capability. Large language models (LLMs), with rich open-world knowledge and strong reasoning abilities, offer a promising solution; however, their token-oriented architecture is inherently incompatible with continuous multi-modal representations. To bridge this gap, we propose \texttt{VQMEA}, an LLM-enhanced MMEA framework based on open-world knowledge augmentation and residual vector quantization. \texttt{VQMEA} first employs LLM-generated entity descriptions as semantic anchors to guide multi-modal representation learning and uses an expert-gated mechanism to adaptively fuse information from different modalities. It then converts the fused continuous representations into discrete token sequences through residual vector quantization, representing entity semantics in a coarse-to-fine manner. A cross-KG contrastive objective further equips the learned tokens with alignment capability. Based on these discrete tokens, we extend the LLMs' tokenizer vocabulary and construct task-specific instruction data to fine-tune LLMs. During inference, candidate entities retrieved by conventional MMEA methods are provided to the LLMs for generative reasoning. Extensive experiments on five benchmark datasets demonstrate the effectiveness and generalizability of \texttt{VQMEA}, suggesting a new paradigm for integrating the knowledge and reasoning capabilities of LLMs into MMEA.