Identifying Key Issues in Agent Trajectories for Improved Performance
To enhance agent performance, analyzing their trajectories is essential. These trajectories detail each step taken and the reasons behind them, revealing challenges that may not be reflected in evaluation scores. For instance, during the evaluation of Dev Proxy, five attempts to create a CI script using the Dev Proxy API revealed 15 distinct issues, with each run making 29 to 39 requests and consuming up to 1.2 million tokens.
What to Look for in a Trajectory
Agents often face challenges when interacting with technology. The following patterns can be observed in agent trajectories, illustrated by examples from the Dev Proxy evaluation.
Agents Guess Your URLs Before They Search
Agents may skip searching and directly fetch URLs they believe contain relevant pages, leading to incorrect guesses. For example, four out of five Dev Proxy runs attempted to access a URL that resulted in a 404 error due to a misidentified page name. This indicates that a logical URL structure can still mislead agents.
Check your product: Review your 404 logs for nearly existing documentation paths and include documentation URLs in your CLI output to prevent guessing.
Agents Probe When Your Interface Stops One Step Short
Help outputs and error messages designed for human users may not provide sufficient guidance for agents. For instance, when a run called an API endpoint without required parameters, it received a 400 error due to missing information. Agents then made further guesses, consuming requests unnecessarily.
Check your product: Ensure error messages provide valid input examples or lists of acceptable values.
Every Run Reinvents the Same Workaround
When a product lacks necessary features, agents create their own solutions. In the case of Dev Proxy, four runs polled for output completion without a clear indication of when it was ready, leading to inefficient workarounds.
Check your product: Ensure every operation that completes after a call returns a method to inform the agent of its completion.
Agents Fill Gaps from Memory
Documentation may not cover integration with other runtimes, leading agents to rely on memory. For example, all runs correctly identified a setting required for routing a Node.js app through Dev Proxy, but none referenced the documentation.
Check your product: Ensure documentation covers integrations with commonly used runtimes and frameworks.
Agents Reject Defaults Built for a First Run
Default configurations may hinder specific tasks. In the case of Dev Proxy, agents avoided a default configuration that injected random errors, leading to inconsistent setups.
Check your product: Provide presets for common tasks alongside default configurations.
Agents Trust Whatever Search Shows Them
Agents often access guidance through search engines, which may yield outdated information. For instance, a run encountered outdated API documentation that did not reflect a recent requirement for a token.
Check your product: Ensure version requirements are clearly stated on documentation pages.
Agents Assume the Defaults Your Help Text Doesn’t State
Help text should clearly state defaults. In Dev Proxy, the default setting routed all machine traffic through the proxy, which was not clearly indicated in the help text, leading to unintended side effects.
Check your product: Review help text to ensure it accurately reflects defaults and how to disable them.
Fix Your Product and Documentation Before Implementing a Skill
Identifying issues in agent trajectories can indicate necessary changes to your product or documentation. While deploying a skill may seem appealing, it merely shifts the problem to developers. Fixing product or documentation issues benefits all agents without requiring installation.
How to Start Reading Trajectories
Examine trajectories from successful runs to identify workarounds that may be mistaken for standard steps. Tools like ATIF Preview can help visualize trajectories. Each finding remains a hypothesis until tested against the same tasks with and without fixes to measure effectiveness.
Related link: aka.ms
Source: Microsoft Developer Blogs