Multi-Model AI: Why We Use 11 Models

The Risk of Single-Provider Dependency

Building on a single AI provider is seductive. One SDK, one set of documentation, one billing relationship, one set of rate limits to understand. The simplicity is real, until it isn't. Providers deprecate models with weeks of notice. Pricing changes arrive as emails, not negotiations. Rate limits tighten as demand grows. And outages (complete, multi-hour outages) happen to every provider, including the largest ones.

For a recruitment platform where AI powers core workflows (screening candidates, searching talent pools, extracting CV data, generating communications), a provider outage means the product stops working. Not "works slower" or "has degraded quality." Stops. Recruiters can't screen. Searches return nothing. CVs sit unprocessed. The business impact isn't theoretical; it's immediate and measurable in lost productivity and delayed hiring decisions.

Beyond availability, single-provider dependency creates strategic vulnerability. If your only provider raises prices by 40% (as has happened in this market) you have no negotiating position and no alternative. If they deprecate the model your product depends on, you're rewriting integration code on their timeline, not yours. If they introduce usage policies that conflict with your product's needs, you comply or you scramble.

The Routing Layer

Our architecture places a model router between application code and AI providers. Application services don't know (and don't need to know) which provider or model serves their request. They describe what they need: a task type, a capability requirement, a quality threshold. The router handles provider selection, failover, and optimisation transparently.

This abstraction means that application code is provider-agnostic by design. A screening service that evaluates candidates against criteria doesn't import a specific provider's SDK. It calls the router with the task description and receives a response. Whether that response came from Gemini, Claude, GPT, or Nova is invisible to the calling service, and irrelevant to its logic.

The router maintains real-time health status for every provider endpoint. When a provider experiences degraded latency, elevated error rates, or complete unavailability, the router automatically redirects traffic to healthy alternatives. This happens in milliseconds, faster than a user would notice a delay, let alone an outage. The system self-heals without human intervention.

Task-Model Matching

Not every task requires the most powerful (and most expensive) model available. Extracting structured data from a CV (name, email, work history, qualifications) is a well-defined task with clear expected outputs. It doesn't need a 200-billion parameter reasoning model. A smaller, faster, cheaper model handles it with equivalent accuracy at a fraction of the cost and latency.

Conversely, evaluating a candidate against nuanced screening criteria ("demonstrates strategic thinking appropriate for a director-level role") requires genuine reasoning capability. The model needs to understand context, weigh evidence, and make qualitative judgements. This is where premium models deliver value that cheaper alternatives cannot replicate.

Our routing layer maps task types to capability tiers automatically. When a service requests CV extraction, the router selects from the pool of efficient models suited to structured data tasks. When a service requests candidate evaluation with complex criteria, the router selects from premium reasoning models. The application code doesn't make this decision. It doesn't even know the decision exists. It simply describes what it needs, and the infrastructure delivers the appropriate capability.

Provider Resilience in Practice

In the past six months, we've experienced at least one provider degradation per month, ranging from elevated latency to complete API unavailability. In every case, our users experienced zero downtime. The router detected the degradation within seconds, rerouted traffic to healthy providers, and continued serving requests without interruption. Most of our team didn't notice until reviewing the routing logs later.

This resilience is automatic and requires no human decision-making during an incident. The router doesn't page an engineer and wait for a failover decision. It monitors health continuously and acts immediately when thresholds are breached. By the time an engineer might notice a provider status page turning yellow, traffic has already been redirected and users are unaffected.

The resilience architecture also handles partial degradation, situations where a provider is technically "up" but responding slowly or producing lower-quality outputs. Latency monitoring triggers rerouting before timeout thresholds are reached. Quality monitoring (comparing outputs against expected patterns for known task types) can detect degradation that wouldn't show up as errors in traditional health checks.

Cost Optimisation Without Quality Sacrifice

The same task, handled by different providers, can have meaningfully different costs (sometimes 3x to 5x variation) with negligible quality difference for that specific task type. The routing layer exploits this variation. For tasks where multiple providers deliver equivalent quality, it routes to the most cost-effective option. Credits stretch further without any perceptible change in output quality.

This optimisation is continuous. As providers adjust pricing, as new models launch, as we accumulate performance data across task types, the routing configuration evolves. A model that was cost-optimal for CV extraction six months ago might not be today. A newer, cheaper alternative may have launched since. The routing layer adapts to the current landscape without requiring application code changes.

Importantly, cost optimisation never overrides quality requirements. The router maintains minimum quality thresholds per task type. If the cheapest available model doesn't meet the quality bar for candidate evaluation, it's not selected, regardless of cost savings. The system optimises within constraints, not at the expense of them.

Future-Proofing Through Abstraction

New AI models launch monthly. New providers enter the market quarterly. Existing providers release updated versions of their models with improved capabilities and different pricing. In this environment, coupling your application to any specific model or provider is a guarantee of technical debt.

Adding a new model to our platform is a configuration change, not a code change. Define the model's capability profile (what task types it handles, what quality tier it occupies), register its endpoint and authentication, set its cost parameters, and it's immediately available for routing. No application service needs to be updated. No deployment is required for the services that consume AI. The new model simply becomes another option in the router's selection pool.

This abstraction has already paid for itself multiple times. When major new models have launched (models that significantly outperform their predecessors at lower cost) we've had them serving production traffic within hours of API access. Our customers benefited from improved AI capabilities without knowing the underlying model changed. That's the point of the abstraction: continuous improvement without continuous disruption.

What We Tried First, and Why It Failed

The router isn't the first thing we built. The first version was a single adapter class with a hardcoded primary provider and a try/catch that fell back to a secondary provider on exception. It worked in the demo. It failed in production within the first month. The problem was that provider failures are rarely clean exceptions. A provider under load doesn't throw a 500; it returns a 200 with a truncated response, or it takes 40 seconds to answer a request that normally takes 2. Our try/catch never fired because there was no error to catch. The request "succeeded" and handed a garbage screening result back to a recruiter.

The second version added timeouts. Better, but still brittle. A fixed 10-second timeout is wrong for both a CV extraction (which should return in under a second) and a long reasoning evaluation (which legitimately takes 8 to 12 seconds). A single timeout number cannot serve both. We were either aborting slow-but-valid reasoning calls or waiting far too long on a stalled extraction. That failure pushed us toward per-task-type budgets rather than a global timeout, which is the model the router uses now.

The lesson we took from those two attempts: health is not a boolean, and failure is mostly gradual. A provider degrades along latency and quality axes long before it returns an error code. So the router tracks a rolling window of per-endpoint latency percentiles and error rates rather than a simple up/down flag, and it makes routing decisions on the trend, not on the last single response.

Latency Budgets and the Cost of Failover

Failover is not free. When the router redirects a request from a degraded provider to a healthy one, the caller pays for the time already spent on the degraded attempt. That is the number that matters, and it's why we don't wait for a hard timeout before switching. As an illustrative shape of the trade-off: a CV extraction carries a budget in the low hundreds of milliseconds, so the router treats a provider that drifts above roughly 500ms on that task as a candidate for demotion rather than waiting for a multi-second timeout to fire. A reasoning evaluation carries a budget of several seconds, so the same 500ms figure there is normal and triggers nothing.

We also cap failover attempts. An early design retried across every healthy provider in sequence before giving up. Under a broad multi-provider incident, that meant a single user request could spend 30-plus seconds walking down the list before failing anyway, which is worse than failing fast. The router now attempts at most two providers for a given request and then returns a clear, retryable error to the caller. Failing fast and honestly beats failing slow and silently.

• Extraction-class tasks: tight budget, aggressive demotion of slow endpoints, cheapest qualifying model preferred.
• Search and ranking tasks: moderate budget, balanced tier, failover to a same-tier alternative before dropping tiers.
• Reasoning-class evaluation: generous budget, premium tier only, quality floor enforced even during an incident.

The Trade-Offs We Accepted

An abstraction layer this deep has real costs, and pretending otherwise would be dishonest. Debugging is harder. When a screening result looks wrong, the first question ("which model produced this?") now requires a log lookup rather than being obvious from the calling code. We pay for that with mandatory per-request metadata: every response records the provider, model version, latency, and routing reason, so the answer is one query away rather than un-answerable.

Determinism is another cost. Because the same task can be served by different models depending on health and load, two identical requests a minute apart can produce slightly different phrasing. For extraction that's fine, the structured fields are the same. For anything a customer might compare side by side, we pin the model for that operation so the output is reproducible, accepting reduced failover flexibility in exchange for consistency. That is a deliberate, per-task choice rather than a global default.

The last trade-off is operational surface area. Eleven models across multiple providers means more credentials to rotate, more rate limits to track, and more provider-specific quirks to encode in the router. We decided that carrying that complexity centrally, in one well-tested layer, is far cheaper than letting it leak into every application service. The complexity doesn't disappear. It just lives in one place where we can reason about it.