In the rapidly evolving landscape of AI-powered development, the industry has often fixated on a single, seductive metric: token count. For many, the assumption has been that the path to lower costs and faster performance lies in minimizing the number of tokens processed by an AI agent in every interaction. However, new research and engineering refinements at GitHub suggest that this focus on "local" efficiency can be a trap. By optimizing for shorter individual responses rather than the overall task, developers may inadvertently increase the complexity of an agent’s workflow, leading to higher costs and slower results.
GitHub is now shifting its engineering philosophy toward a more holistic view of agentic performance. By prioritizing the final outcome over the immediate cost of a single tool call, the team behind GitHub Copilot has implemented a series of strategic changes designed to reduce wasted effort across the entire coding task. These improvements, which have been integrated into the Copilot CLI and are cascading through the broader suite of Copilot products, demonstrate that true efficiency is found not in brevity, but in context-aware utility.
The Local Metric Trap
The allure of "token minimization" is easy to understand. When an agent interacts with a system—such as a terminal or a code repository—the output it receives is often voluminous. Tools like RTK (Rust Token Killer) have gained popularity by automatically stripping away parts of the shell output to provide the AI with a cleaner, shorter stream of data. On the surface, this appears to be a clear win: the model receives fewer tokens, and the cost per interaction drops.
However, GitHub’s internal evaluation using agentic coding benchmarks revealed a more nuanced reality. While these tools successfully shortened responses, they often stripped away critical information. When the AI agent encountered a task that required the missing data, it was forced to compensate. This typically manifested as the agent rerunning the command or requesting the full, uncompressed output to "recover" the necessary context.
This cycle of omission and recovery is counterproductive. The initial tool response might have been shorter, but the agent was forced to undertake extra turns, carry more context forward, and spend significantly more resources to reach the same conclusion. In effect, the system saved tokens locally only to spend more globally. The lesson for the GitHub engineering team was definitive: tokens per tool call is a misleading objective. The only meaningful metric for efficiency is the end-to-end cost and duration of the task, measured from the moment the user makes a request until the final result is achieved.
Compressing Noise While Preserving Intent
The challenge, therefore, was to find a way to shorten repetitive output without forcing the agent to retrace its steps. Analysis of build, test, and lint logs showed that while these systems often generate massive amounts of predictable noise, source-code-like output and arbitrary command results are highly information-dense.
GitHub’s approach to solving this was the development of a selective output compressor. Early prototypes were overly aggressive, frequently stripping away information that the agents later deemed essential. For instance, an initial attempt to compress git diff output was quickly abandoned after benchmarks showed agents repeatedly opening the original files to recover missing context.
Through a process of rigorous evaluation, the team adopted a conservative, three-part policy for output management. First, it identifies and leaves source-code output untouched. Second, it reorganizes search results to maximize readability without discarding any matches. Finally, it applies selective compression to predictable, repetitive noise—such as installation logs—while always providing a "recovery path." This path serves as both a safety net for the agent and a diagnostic signal for engineers. If the agent frequently opts to open the original, full-length output, it serves as clear evidence that the compression was too aggressive. In production, this system has proven highly effective, showing no significant regressions in task success while reducing overall AI credit usage.
Prioritizing Information Over Formatting
Beyond compressing output, the team identified "low-hanging fruit" in the form of redundant formatting. A prime example was the view tool, which agents use to read file contents. Historically, this tool prefixed every line of code with a line number. While these numbers were useful for legacy editing tools, modern agents now rely on matching surrounding code snippets to apply changes, rendering the line-number prefixes entirely redundant.
Though each prefix might seem negligible, the cumulative effect of thousands of lines of code being read across a session resulted in significant, unnecessary token usage. By removing these prefixes, GitHub reduced model-inference costs by approximately 5% in offline benchmarks. Because the change was purely structural and did not remove actual code, it required no new instructions for the model and provided no risk of information loss. This represents the ideal type of optimization: it cleans the input stream without requiring the agent to make any new decisions or recovery attempts.
The Nuance of Prompt Compression
Prompts serve as the "operating system" for an AI agent, carrying the instructions that define its behavior. While shortening these prompts is a clear way to save tokens, it carries the risk of stripping away intended behaviors or guardrails.
GitHub encountered this risk firsthand when using a meta-prompting loop—where Copilot was tasked with refining its own instructions—to shrink its prompt by half. While the new prompt was significantly shorter, it introduced an unintended side effect: it turned cautious instructions about parallel task execution into a rigid, sequential scheduling policy. The agent, attempting to be efficient, had effectively neutered its own ability to perform multiple tasks simultaneously.
This highlighted a critical lesson: prompt behavior must be backed by regression testing. If a behavior isn’t explicitly tested, a shorter prompt is likely to inadvertently remove it. After identifying the regression, the team replaced a complex, restrictive set of instructions with a single, clear directive: "Independent agents can run in parallel; consider side effects." This change was both shorter and more permissive, allowing the model to make intelligent decisions rather than following a rigid rule. The result was a prompt that was 1,300 tokens shorter per turn, with no degradation in the agent’s ability to handle parallel tasks.
Orchestrating Background Work
Finally, GitHub addressed the "retrieval detour"—a phenomenon where agents spent extra turns waiting for background tasks to complete. Previously, if an agent launched a shell command and a sub-agent simultaneously, it would have to wait for each to finish, potentially triggering multiple model calls to request and process the results.
The new harness now batches these notifications, delivering all completed results in a single, consolidated event. Instead of the model "polling" for the status of a background task, the system proactively pushes the data once it is ready. This change eliminates the need for the agent to pause, request, and re-process, allowing it to continue with the full context of its work in one fluid motion. By streamlining these round trips, GitHub saw a further 2.3% reduction in AI credit usage.
Lessons for the Future of Agentic Development
These improvements—ranging from the elimination of redundant formatting to the batching of background tasks—underscore a fundamental shift in how developers should approach AI efficiency. None of these changes were about making the underlying models "smarter." Instead, they were about removing the friction and unnecessary work that models were never meant to perform in the first place.
As GitHub continues to refine these systems, the broader message to the developer community is clear: optimization should be viewed as an end-to-end endeavor. The most effective agents are not those that use the fewest tokens in isolation, but those that are given the most relevant context and the fewest distractions, allowing them to complete their objectives with minimal interference. By measuring success through the lens of the final outcome, GitHub is setting a standard for the next generation of efficient, agentic coding tools.
