What it does

Testing and evaluation move you past ad-hoc manual checks. You run defined test sets and simulated multi-turn conversations against an agent to see whether it answers correctly and behaves consistently as you change it.

Key facts

• The Test pane gives live single-turn checks while you author; evaluation adds repeatable, structured runs.
• Multi-turn simulations matter because generative agents behave differently across a conversation than in one shot.
• Test sets let you rerun the same cases after every change and catch regressions.
• Evaluation is essential precisely because orchestration is non-deterministic — one manual pass proves nothing.

When to use / skip

Use structured evaluation for any agent going near production, especially anything using generative orchestration. Skip formal test sets only for a throwaway proof of concept you'll never ship.

Configuration decisions

• The set of representative and edge-case scenarios worth codifying.
• What "correct" means — exact answer, cited source, or acceptable behaviour.
• When evaluation runs: on each change, before each release, or continuously.

Gotchas

• A generative agent can pass a demo and fail the tenth real conversation — single runs lull you into false confidence.
• Test data drifts from reality; refresh cases as the agent's knowledge and use change.

Consultant notes

• Build the test set alongside the agent, not after go-live — it's your regression net for every future change.
• Make evaluation a release gate in ALM, not an optional nicety.
• Involve the business in defining "correct"; engineers and users often disagree on acceptable answers.

---

*Confirm current evalua