Playground and testing
Test assistants with real prompts before users or systems depend on them.
The playground is where you find out whether the assistant is actually useful.
Do not test only with friendly demo prompts. Test with the questions users already ask.
Retest the same prompts after every source, brief, or training change. Stable repeated answers matter more than demo success.
Build a test set
Create 10 to 30 prompts that represent the job:
- common questions
- hard questions
- questions with missing information
- questions that require escalation
- questions that should not be answered
Read every weak answer
When an answer is bad, classify the failure:
| Failure | Likely fix |
|---|---|
| Missing or wrong fact | Add, update, or remove a knowledge source |
| Right facts, wrong tone | Add better training examples |
| Right facts, wrong format | Tighten the assistant brief or examples |
| Hallucinated details | Narrow the task, improve sources, or require citations |
| Inconsistent answers | Split the project or add clearer examples |
Retest the same prompts
Do not keep changing the test after each fix. Reuse the same prompts so you can see whether quality is improving.
Review each answer
For every answer, record:
- pass or fail
- failure reason
- expected behavior
- source that should have been used
- fix owner
Production gate
Before deploy, test:
| Prompt class | Example signal |
|---|---|
| Common question | Answer is useful without handholding |
| Hard question | Assistant uses relevant source or asks for missing info |
| Missing info | Assistant does not invent facts |
| Out-of-scope | Assistant refuses or redirects |
| Risky request | Assistant escalates when needed |
Ready to deploy
An assistant is ready when it gives useful, predictable answers for the job you chose. It does not need to answer everything.