DiffImaginE: Imagine to Verify Entity Types with Diffusion

摘要

Multimodal named entity recognition (MNER) determines, for each candidate span, whether an entity-type hypothesis is supported by the joint textual and visual evidence. Existing imagine-and-compare verifiers typically map each (span, type) pair to a single predicted visual feature and compare it with the observed image representation. Such deterministic imagination compresses the diverse visual realisations of an entity type into one prototype, making the verifier brittle when the same type appears through substantially different visual cues. Moreover, the resulting compatibility score lacks a probabilistic interpretation and provides only indirect supervision for rejecting confusable type hypotheses. We propose DiffImaginE, which formulates MNER type verification as conditional latent diffusion inference. Given span-localised visual evidence, a type-conditioned denoiser predicts the noise injected into its standardised latent representation. The resulting denoising error provides an ELBO-consistent surrogate for the type-conditional negative log-likelihood, enabling different type hypotheses to be compared according to how well they explain the observed evidence. DiffImaginE preserves a standard multimodal encoder stack and replaces only the deterministic verifier with a classifier-free-guided diffusion scorer trained using Min-SNR weighting. To bridge generative likelihood estimation and discriminative prediction, we directly supervise per-type diffusion scores as classification logits, learn to aggregate evidence across noise levels, and use antithetic sampling to reduce Monte-Carlo comparison variance. Our theoretical analysis shows that classifier-free guidance sharpens the induced type posterior and characterises when antithetic pairing yields lower variance at equal denoiser cost. Experiments on Twitter-2015 and Twitter-2017 show that DiffImaginE consistently improves over a matched deterministic ImaginE control under the same encoder, auxiliary objectives, and evaluation protocol, with further support from ablations and paired significance tests.