Vision Transfers, Text Remembers: Modality-Asymmetric Prompting for Few-Shot Class-Incremental Learning

摘要

Few-Shot Class-Incremental Learning (FSCIL) requires a model to acquire novel classes from a few examples while retaining previously learned ones. Prompt tuning on frozen vision-language models is an effective solution, yet existing methods adapt the visual and textual branches with the same machinery. The two sides of the stability-plasticity trade-off, however, do not live in the same branch. Transfer to novel classes is available in the visual branch: visual semantics are compositional, so a new category is largely built from primitives that base data already provide. Forgetting, once the encoders are frozen, arises only in the textual branch: class identity is nominal, an atomic label that must not be reconstructible from the labels already registered. Both requirements bear on one quantity — the residual of an item against the span of an existing basis — and pull it in opposite directions. We propose AsymPrompt, which minimizes this residual for visual composition and holds it above a margin for textual identity. Compositional Primitive Coding encodes each salient patch as a sparse, non-negative code over a shared primitive dictionary and assembles the prompt as an explicit composition. Atomic Identity Anchors constrain each identity to be irreducible with respect to the subspace spanned by its neighbours; we show that standard pairwise separation cannot guarantee this. Composition-to-Name Registration maps a class-level composition signature to an identity code in closed form and applies the minimal deformation restoring irreducibility, registering a novel class without gradients. Experiments on CIFAR-100, miniImageNet and CUB-200 show consistent improvements over recent FSCIL methods, and the gradient-free variant remains competitive. Code and data will be released upon acceptance.