Run both Voice AI agent and test simulator on the same model

Follow me on LinkedIn - AI, GA4, BigQuery

Your Voice AI Agent uses a specific LLM model:

Make sure your test simulator (feature used for simulation testing) uses the same model as your voice agent:

When you create a new test case in Retell AI, it uses the GPT 4.1 mini model by default:

So, what can we conclude?

#1 A voice agent test usually involves two language models:

The agent model - This is the model your live production agent uses. It reads the agent prompt and generates the agent’s responses.
The test caller model - This model acts as the caller. It reads the test prompt, including the persona, scenario, script, and behaviour instructions.

#2 If you are not using the GPT 4.1 Mini for your live production agent, then your models won’t match.

Different LLMs can interpret the same prompt differently.

For example, Claude, GPT, and Gemini all read the same words and can produce vastly different outputs.

Even different versions of the same model (like GPT 4.1 vs GPT 5) can behave differently, which is why your agent's behaviour can shift when a provider releases a new version.

For simple instructions, this may not matter much.

A prompt like “be friendly” will usually produce similar behaviour across models.

But voice agent prompts are rarely simple. They often include:

conditional logic.
exception handling.
escalation rules.
refusal rules.
customer-specific scenarios.
pressure-testing instructions.

The more complex the prompt, the more model differences matter.

A 3,000-word prompt interpreted by Claude, GPT, Gemini, or different versions of the same model can lead to meaningfully different behaviour.

What goes wrong when models don't match?

When the test simulator uses a different model from the production agent, your test results can become unreliable (false positives, false negatives).

The test caller may behave differently from what you intended.

You may have designed the caller to:

stack inputs at a specific point.
push back after a refusal.
mention they are a returning customer only when asked.
stay within a defined persona.

But a different model may interpret those instructions differently.

So the agent is no longer being tested against the scenario you designed.

If the test environment uses a different model, you may end up testing a version of the agent that does not match production.

Always use the same model for both Voice AI agent and test simulator to get accurate test results.

This makes the test environment closer to production and makes the results more meaningful.

Many platforms default to a cheaper model for the test caller. That may be fine for lightweight checks, but it is risky for serious testing.

If you cannot match the models because of cost or capability limits, treat the results as approximate.

Do not make confident production decisions from mismatched models.

Label the limitation clearly when interpreting results.

For example:

This test passed with a weaker simulator model, so production behaviour may still be different than the test suggests.