Where Forgotten Samples Go: Visual Boundary Guard for CLIP-Based Class-Incremental Learning

摘要

Class-incremental learning aims to acquire new classes sequentially while preserving previously learned knowledge. Although existing methods have substantially improved continual adaptation of vision-language models, they mainly measure how much knowledge is forgotten, leaving the destination and underlying mechanism of forgotten samples underexplored. In this work, we systematically investigate where forgotten samples go in CLIP-based continual learning. We find that forgotten samples are not randomly reassigned but are predominantly absorbed by similar class neighborhoods, with visual similarity explaining more errors than textual similarity. Feature-trajectory analysis further shows that new-task optimization induces directional deformation of old feature regions. However, such drift becomes harmful only when vulnerable old features exhaust their local safety margins and cross nearby decision boundaries. Motivated by these findings, we propose a drift-aware visual boundary guarding framework. It first constructs a compact memory containing class prototypes and low-margin boundary anchors. During new-task learning, current samples are used as local probes to estimate the potential displacement of visually related old boundary regions. The predicted post-update anchors are then guarded against their strongest new or visually similar old competitors, while harmless drift within a safe class region remains unconstrained. Our method stores no raw images, is applicable to different parameter-efficient tuning strategies, and introduces no additional inference cost. Experiments on CIFAR-100, ImageNetR, ImageNet-100, and VTAB show consistent improvements across MoE-, adapter-, prompt-, and LoRA-based continual learners, improving average incremental and final accuracy.