From Embeddings to Tokens: Generative Multi-Modal Entity Alignment with Open Knowledge-Enhanced Quantization

摘要

Multi-Modal Entity Alignment (MMEA) aims to identify equivalent entities across heterogeneous multi-modal knowledge graphs (MMKGs). Existing approaches have made significant progress by designing sophisticated cross-modal fusion strategies. However, these embedding-based methods suffer from two key limitations: insufficient prior knowledge to handle highly heterogeneous entities, and restricted reasoning and generalization ability due to their reliance on static vector spaces. Recently, Large Language Models (LLMs) have demonstrated strong capabilities in understanding and reasoning, offering a promising paradigm for addressing these challenges. Nevertheless, LLMs are inherently tokenoriented, which is incompatible with continuous multi-modal embeddings. In view of this, we propose VQMEA, which bridges this gap via open knowledge-enhanced vector quantization. Specifically, we first design an LLM-enhanced multi-modal encoding module, where open-world knowledge generated by LLMs is injected as semantic anchors to guide representation learning, and an expertgated mechanism is employed for adaptive multi-modal fusion. A residual quantization module is then introduced to transform continuous embeddings into hierarchical discrete code sequences, capturing both coarse- and fine-grained semantics. To further enhance alignment capability, we incorporate multi-modal contrastive objectives. Finally, based on the learned code sequences, we construct instruction-oriented data to fine-tune LLMs, enabling entity alignment to be performed in a generative manner. Extensive experiments on five benchmark datasets demonstrate the effectiveness and generalizability of VQMEA, highlighting a new direction for integrating LLMs with MMEA tasks.