CARAT: Do Materials LLMs Reason or Recite?

摘要

We introduce CARAT, a controlled evaluation framework for distinguishing structural evidence use from answer recitation in materials language models. Accuracy alone cannot resolve the distinction, because a structural description may already contain the answer it is scored against. CARAT holds each question and gold answer fixed across eight matched evidence views, uses GraphSpace to expose typed structural relations, and combines matched fine-tuning with answer masking, evidence injection, paired inference over material-clustered intervals, and a preregistered rule that can withhold a claim. Three results follow. First, the grounded representation improves benchmark accuracy by 17.3 percentage points over formula-only inputs on readback-hardened families. Second, turning the same scrutiny on our own comparison resizes it: of a 19.3-point advantage over a plain periodic graph, 1.96 points survive on families where both views carry the inputs their question requires, while 46.7 points come from three families whose plain rendering cannot answer at all, which is a gap in information content rather than in representation. Third, a parameter-free pointer-blind rule solves four of seven hardened families, so we rebuild them until eighteen measured shortcut routes sit near chance. The frozen model then transcribes the pointer yet answers identically for 95.6% of cases in which only the pointer moves, whereas matched supervision reaches 99.8% and, once the pointer is deleted, falls to 23.4%, below the 27.0% measured shortcut maximum: the composition is learnable, not absent.