ALTK-Evolve Achieves Top Score on τ²-bench for Customer Service Tasks

ALTK-Evolve has demonstrated significant advancements in customer service tasks by utilizing GPT-4.1, achieving a score of 67.5% Pass^4 on the τ²-bench benchmark. This score indicates that the agent successfully solves tasks on all four independent attempts for 67.5% of the tasks, surpassing all retail entries in the latest leaderboard snapshot from October 12, 2026.

The system operates on the benchmark’s standard ReAct framework, enhanced by learned guidelines integrated into its system prompt. ALTK-Evolve constructs these guidelines by analyzing the agent’s training attempts, extracting successful and unsuccessful strategies into concise behavioral rules. These rules guide the agent on how to authenticate customers, confirm transactions, and manage returns, all without requiring weight updates or human annotation.

The τ²-bench, developed by Sierra, evaluates how effectively an agent manages customer service interactions under real-world constraints. Each task involves the agent interacting with a simulated customer who has specific goals and must adhere to a defined policy document. The agent’s performance is measured based on its ability to accurately extract details, comply with policies, and execute appropriate actions, with a task only counted as solved when the final database state aligns with the expected outcome.

In the testing process, a basic ReAct agent was run using GPT-4.1 as the model and GPT-5.5 as the customer simulator, with four trials conducted for each task. The only modification made was the addition of learned guidelines to the system prompt, while all other components remained unchanged.

ALTK-Evolve’s guidelines significantly enhance both the capability and reliability of the agent. The evaluation compared three configurations: a baseline with no guidelines, a sup3 configuration retaining only high-support guidelines, and a lossless configuration including all guidelines. The results showed that the lossless configuration led to a notable increase in both mean accuracy and Pass^4 scores across retail and airline domains.

In retail, the lossless configuration achieved a mean accuracy of 82.5% and a Pass^4 score of 67.5%, reflecting a substantial improvement over the baseline. In the airline domain, the lossless configuration also outperformed the baseline, although the sup3 configuration proved more reliable in terms of Pass^4.

While the guidelines increase input costs due to the larger system prompt, they do not lengthen conversations. In fact, the agent’s interactions became slightly shorter, leading to lower latency and faster completion times for customers. This trade-off results in higher accuracy and reliability without sacrificing efficiency.

Furthermore, the learned guidelines also proved beneficial when tested with a frontier model, GPT-5.5. In this scenario, the addition of guidelines improved the agent’s accuracy and reliability, demonstrating that the approach is effective across different model architectures.

ALTK-Evolve’s performance on the τ²-bench leaderboard highlights the effectiveness of learned guidelines in enhancing customer service agents. The agent achieved a Pass^4 score of 67.5% in retail, outperforming frontier models, and reached 82.0% in the airline domain, solidifying its position at the top of the leaderboard.

Source: Hugging Face Blog