Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning

摘要

Reinforcement learning has become a common approach for training large language model agents on long-horizon tasks, where many interdependent actions may precede a single sparse reward. Critic-free, group-relative methods such as GRPO are well suited to this setting, but they assign the same trajectory-level signal to every step and therefore cannot localize credit to individual decisions. GiGPO introduces a step-level signal by grouping steps that share an anchor state, yet combines step- and episode-level estimates using a fixed rule across steps. We propose GACA (Granularity-Adaptive Credit Assignment), a critic-free advantage estimator that adaptively mixes episode- and step-level credit using a per-step NLL-based uncertainty score. GACA computes the mean negative log-likelihood of each sampled action, normalizes it within the trajectory, and increases the weight on the fine-grained step-level advantage as this score grows, without additional rollouts or a learned critic. We derive an exact risk decomposition for the implemented mixture and show that sufficiently small adaptive modulation improves over fixed mixing under an alignment condition. A separate population result bounds local action-value variation using expected NLL, while an error-projection analysis characterizes when two-level mixing captures information unavailable to scalar uncertainty reweighting. On ALFWorld and WebShop, GACA achieves higher task success than GRPO and GiGPO at both 1.5B and 7B scales.