Make the outcome inspectable
An order agent may need to clarify a request, obtain approval and reserve stock. A test should inspect those outcomes and the permitted changes to system state.
A single text similarity score is rarely a complete description of success for a workflow with tools and approvals.
Build a useful failure taxonomy
Separate the causes of failure so the results support an engineering decision. A missing tool capability requires a different fix from a misunderstood policy.
- Context: relevant evidence was missing or misunderstood.
- Planning: the proposed path could not satisfy the constraints.
- Authorization: an approval or policy requirement was bypassed.
- Execution: the action changed the wrong record or failed to complete.
- Environment or grader: the simulation or scoring rejected a legitimate solution.
Hold the setup constant
Record the model version, instructions, tool schema, agent harness, budget and retry policy. Repeated trials can expose instability that one successful attempt hides.
When comparing systems, report the same evaluation conditions and the limitations of a small task set. Do not turn a handpicked demonstration into a general reliability claim.
Protect the evaluation set
Use separate training and evaluation tasks. Closely related examples can leak answers even when their identifiers differ. Consider separating source businesses, time periods or task families where the design supports it.
Track what has been released, and describe contamination limits in the report. Keep public examples separate from private held-out material.
Validate the test as well as the agent
Have experts check solvability and acceptable alternatives. Investigate disagreements between the grader and human review. Treat scoring bugs as defects in the evaluation rather than evidence of an agent’s weakness.