MIT and Sakana AI introduce SIFT, a framework using an LLM judge to cut evaluation costs for self-improving coding agents, boosting accuracy on Polyglot.

The pursuit of autonomous coding agents that can write, test, and refine their own source code has long been hindered by a brutal economic reality: the cost of verification. For an AI agent to improve itself, it must propose a change, test that change against a benchmark, and determine if the result is an actual improvement. When scaling this process across thousands of candidates, the computational bill and the time required often become prohibitive for all but the largest labs.

To solve this, researchers from MIT and Sakana AI have developed a new framework called Recursive Self-Improvement via Fast Tree Search, or SIFT. The core innovation is the introduction of an LLM judge that acts as a filter, predicting which agent patches are likely to succeed before the system commits to the expensive process of running a full benchmark.

The high price of recursive self-improvement

Recursive self-improvement is the process where an agent examines its own failures and proposes a patch to its harness—the set of prompts, tools, and code that guide its behavior. Because a successful patch improves the system that will propose the next patch, these gains can compound over time. However, the search for the right patch is often a needle-in-a-haystack problem.

Traditional methods require running every candidate through a test suite to see if the accuracy score increases. This process can consume thousands of CPU hours and cost thousands of dollars in API credits. Previous architectures, such as the Darwin Gödel Machine (DGM), attempted to manage this by using increasingly larger task sets—starting with 10 tasks and moving to 50—to weed out poor candidates. The Huxley-Gödel Machine (HGM) took a different approach by varying the number of randomly sampled tasks. Despite these efforts, the sheer volume of evaluations remained a bottleneck for rapid iteration.

How SIFT utilizes a pairwise LLM judge

SIFT shifts the paradigm from absolute scoring to relative comparison. Instead of asking a model to provide a numerical score for a single agent, the framework uses a separate language model to compare two candidate agents side-by-side. The researchers argue that this approach mimics human intuition; it is similar to a hiring manager choosing between two finalists rather than assigning an arbitrary grade to one person in a vacuum.

This judge-mediated search allows SIFT to be far more selective about which agents undergo full evaluation. Furthermore, the framework does not operate in a strict linear sequence. SIFT can propose new changes while earlier candidates are still being evaluated, allowing the search to explore multiple branches of the improvement tree simultaneously. This prevents the system from idling while waiting for a full test suite to complete after every single patch.

Breaking down the economics of SIFT

The financial efficiency of SIFT is where the framework shows its most immediate value for business applications. By utilizing the LLM judge to trim the search space, the cost of discovering high-performing agents drops precipitously. According to data shared by VentureBeat, the costs are broken down into granular steps:

Each proposed patch costs approximately 12 cents to generate, and each pairwise judge call costs 4.4 cents. While a full agent evaluation on 50 Polyglot tasks still costs around $6 and requires 2.6 CPU hours, SIFT ensures these expensive runs are reserved only for the most promising candidates.

The results of this efficiency are stark. In one specific run on the Polyglot coding benchmark, SIFT reached 35.1% accuracy in under five hours of wall-clock time. This was achieved using only 42 CPU hours and roughly $150 in API credits. To put this in perspective, running the same framework without the LLM judge resulted in a lower accuracy of 29.8%, proving that the judge is not just a cost-saver but a driver of performance.

Performance benchmarks against industry baselines

When tested against established baselines, SIFT demonstrated superior efficiency and accuracy. Using the o3-mini model, SIFT outperformed the Darwin Gödel Machine, hitting 35.1% accuracy compared to DGM's 30.7%. The framework also showed versatility across different model architectures. When paired with the open-weight Qwen3-Coder-30B, SIFT slightly outscored the Huxley-Gödel Machine while utilizing approximately one-third fewer CPU hours.

The effectiveness of the judge was further validated on TerminalBench 2.1. In this environment, the judge's top-picked agent averaged 36.7% on full runs, whereas the agent with the highest score from the general search averaged only 28.1%. This suggests that the LLM judge is often better at identifying the potential of a patch than the raw benchmark scores are at reflecting it in the short term.

Scaling AI agents for the enterprise

For entrepreneurs and tech leaders, the SIFT framework represents a shift toward sustainable AI development. The ability to tune coding-agent harnesses against well-defined tasks without blowing through a cloud budget is a critical requirement for moving AI agents from research papers into production environments. When agents can self-improve cheaply, the cycle of deployment and optimization accelerates.

This development aligns with a broader trend in the industry to put AI agents on a budget. For instance, recent updates to enterprise agent platforms have focused on adding user quotas, rate limits, and organization-wide caps to prevent runaway costs. SIFT provides a technical solution to the cost problem at the architectural level, rather than simply capping usage at the administrative level.

Global implications for US and UK enterprises

For businesses in the USA and UK, the SIFT framework lowers the barrier to entry for creating proprietary, self-optimizing software tools. In these markets, where the competition for engineering talent is fierce and the cost of developer hours is high, automating the refinement of coding agents provides a significant competitive edge. Companies can now iterate on specialized agents for legacy codebases or niche proprietary languages without requiring a massive compute cluster.

From a regulatory perspective, the shift toward more efficient, judge-mediated AI systems is particularly relevant. While the US currently relies on a patchwork of executive orders and voluntary commitments, and the UK maintains a pro-innovation, non-statutory approach, the focus is increasingly on the transparency and reliability of AI outputs. SIFT's use of a judge to validate improvements creates a more structured audit trail of why an agent was modified, which may assist firms in meeting future safety and governance standards regarding autonomous code generation.

FAQ

---

Sources: Venturebeat, Newsbreak, Aiweekly ·