On the Bayes Platform, an agent never knows which model is in use. Gemma 4 31B, Qwen3.8-27B and Mistral Small 3.2 run on our own GPUs behind the same OpenAI-compatible API, served by vLLM. What differs between them (generation settings, which files they accept) lives in one class per model, about twenty lines each. The guards every model needs are common to all of them, in the shared code. The agents, their tools, their sources and their rules stay the same when the model changes.
This post details how we add a new model: what is specific to it, what it gets from the shared code, and how we check that nothing breaks.
Why is one API not enough?
vLLM gives every model the same chat endpoint. That removes the transport problem, not the behaviour problem. Each model was trained with its own conventions, and its authors publish the settings with which they get the best results. A wrong setting is a silent failure: the call succeeds, no technical error comes back, and the gap only shows in the content of the answers.
| What differs | Gemma 4 31B | Qwen3.8-27B | Mistral Small 3.2 |
|---|---|---|---|
| Default temperature (authors’ recommendation) | 1 | 0.7 | 0.15 |
| A PDF attachment | Page images | Page images | Not accepted for structured output |
The temperature follows each model’s authors. How we handle a PDF is our choice, model by model, based on the measured performance of each one.
What does a model declaration look like in the code?
Adding a model is a class that declares its options. For example, for Qwen, this is the whole provider:
@Injectable()
export class AISDKQwenProvider extends AISDKVllmProviderBase<AgentProvider.Qwen> {
protected readonly providerName = "Qwen Provider"
protected readonly modelOptions: Record<QwenAgentModel, ModelOptions> = {
// Qwen's recommended non-thinking sampling (model card): temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5.
[AgentModel.Qwen38_27B]: defineModelOptions(
z.object({ temperature: z.number().min(0).max(2) }),
{ temperature: 0.7 },
),
}
protected readonly modelInfos: Record<QwenAgentModel, ModelInfos> = {
[AgentModel.Qwen38_27B]: {},
}
getAgentProvider(): AgentProvider.Qwen {
return AgentProvider.Qwen
}
}The default temperature comes from the model card, and the comment says so. A team that changes it later knows what it moves away from.
How do we set a specific behaviour? A simple override
The shared base exposes a few methods a provider can override: for the system prompt, for the messages of a structured-output call, or for reading its result. For example, if we need to rewrite the default parsing of the structured output:
protected override readStructuredOutput(result: {
text: string
output: Record<string, unknown>
}): Record<string, unknown> {
return this.parseStructuredOutputText(result.text)
}The difference stays in the provider. The agent that asks for an extraction gets the same object, whatever the model.
What does every model get for free?
Some failures are not specific to one model, so the base handles them once, in the call every provider shares:
- Broken tool arguments are repaired. When a model emits a tool call whose arguments are not valid JSON, the AI SDK’s
experimental_repairToolCallhook regenerates the arguments through a structured-output call instead of dropping the call. - A tool call written as text is recovered. A model sometimes writes the call in its answer instead of emitting it. The stream is captured raw, the leaked call is taken out of the text and executed after the stream.
- Text repeated after a tool result is dropped. A model can write its message, call a tool in the same step, then write the same message again when the result comes back. The steps are stored as one reply, so the user would read it twice. A filter drops a step’s text when it repeats the previous one word for word, and logs it.
- A generation error is never silent. When streaming, the SDK does not throw when a generation fails: it puts the error inside the stream, among the text. Code that reads only the text then gets an empty answer and no reason. We read every part of the stream, so an error stops the call and reaches our logs.
- Strict tool schemas only where the API supports them.
supportsStrictTools()is off by default and on for the providers that enforce schemas server side.
An agent author writes none of this, and a new model inherits all of it.
What must stay out of the model layer?
The rule we hold to: the agent layer never asks which model is running. No if (model === "gemma") in an agent, a tool or a prompt template. A difference between models goes in the provider, or it does not go in.
An example. When a question needs two independent tools, such as showing a resource card and searching the knowledge base, a model can call them one after the other instead of both at once. Each extra call costs a round trip.
We tested a prompt line that asked for these calls to be grouped. On a first try, the result looked better. Measured then on several models, with real-shaped messages, the line also changed which tools got called, for better or worse depending on the model. We did not put it in production: a line added for one model acts on all the others. We make many decisions of this kind, always backed by tests: the measurement script stays in the repository, so the next wording gets the same test.
How do we know a change to the harness is safe?
Every provider runs the same live contract test suites, outside CI because they call real endpoints:
- a retrieval turn on a fixture handbook whose value is deliberately not the statutory one, so a correct answer proves the agent read the document;
- a turn that needs two independent tools;
- a no-retrieval agent with a long system prompt and strict guardrails, across a greeting, a service question, a refusal and an off-topic message.
Take a recent fix: we corrected the English of an instruction sent to Mistral on every turn with tools. It is one sentence, but we treated it as a change of the harness: unit test updated, then the live test suite run on Mistral before and after. Five of the six checks passed. The sixth one already failed before the fix: that is why we run the test suite before and after.
What’s next
The live test suites tell us whether a model keeps the contract. They do not tell us whether it serves a given agent’s users better. That is the job of per-agent evaluation datasets, built from real conversations, which an MCP client can now run against a candidate model. We will most likely write about that loop next: how a new model release becomes a measured decision instead of an announcement.
The code is in the Bayes Platform repository, under apps/api/src/external/llm/ai-sdk/.
