NestMem: Hierarchical Memory Tokens for Reusable Procedures in Large Language Models

摘要

LLM applications repeatedly apply the same task procedure, such as following a fixed reasoning pattern, producing a prescribed output format, or executing a recurring sequence of tool calls. These task procedures are commonly supplied through natural-language instructions, forcing the model to repeatedly process redundant procedural knowledge. To mitigate this redundancy, the prior approach is to construct a procedural memory bank composed of learnable tokens, where each token parameterizes a reusable procedure. During inference, generating a specific token essentially recalls the associated procedure from memory, whose embedding then supplies the necessary procedural knowledge for subsequent generation. However, without a structured organization, these procedure-level tokens frequently encode common procedural knowledge shared across different tasks. Furthermore, as the memory bank expands, predicting the target token from an unorganized space becomes increasingly difficult. To overcome these issues, we introduce NestMem, a framework that organizes procedure-level tokens into a hierarchical procedural memory. Specifically, NestMem decomposes each procedure into a three-level token hierarchy: coarse-category, fine-category, and procedure-level tokens. Similar tasks share category-level tokens to reuse common procedural knowledge. Furthermore, this coarse-to-fine hierarchy provides progressively more specific conditioning for accurate procedure selection, enabling accurate procedure-level token prediction during inference. We evaluate NestMem in two distinct settings: atomic procedure reuse on Super-Natural Instructions and compositional procedure reuse in multi-step function calling. Our results show that NestMem achieves scalable procedure recall and robust downstream performance using only lightweight memory-token updates.