TrustGate and LiteLLM: certify the route you actually ship

Most AI teams do not have a model problem. They have an endpoint problem, because production never calls a model name. It calls a route, and that route changes when the provider changes, the fallback fires, the prompt is edited or the gateway config is touched.

LiteLLM gives you one OpenAI-compatible gateway across providers. TrustGate certifies what that gateway actually serves, using self-consistency sampling and conformal calibration to produce a reliability level you can gate a deployment on. The gateway decides where traffic goes. The certificate decides whether it should go at all.

Certify the endpoint your users actually call

Many evaluation pipelines measure the wrong object. They evaluate a model name, and the application never calls one.

The product calls support-agent-v2. Behind that alias, the gateway might route to OpenAI today, Azure tomorrow, a local vLLM deployment next week, or a fallback provider the moment the primary route degrades. So when somebody says they evaluated GPT-4.1-mini, the useful follow-up questions are which prompt, which gateway route, which temperature, which fallback policy, which retrieval configuration, which canonicalisation strategy and which question set.

If the application calls https://llm-gateway.company.com/v1/chat/completions, that is the thing worth certifying. TrustGate is built around that boundary and treats the system as a black box: ask representative questions, sample multiple responses, canonicalise them, calibrate the results, produce a certificate.

Why the gateway is the right boundary

A model gateway is more than plumbing. It is the operational boundary between product systems and model infrastructure, and it controls model aliases, routing, credentials, provider selection, fallback chains, budgets, logging, tenancy and policy.

That means a gateway configuration can change application behaviour while the application code stays identical, which makes it the natural place to attach reliability evidence.

Application
  -> LiteLLM gateway
  -> provider / local model / fallback
  -> response

TrustGate
  -> calls the same LiteLLM endpoint
  -> samples responses
  -> canonicalizes outputs
  -> calibrates reliability
  -> produces a certificate
  -> passes or fails deployment

The operating model shifts with it. Instead of asking whether a model is good, you ask whether a specific production route is reliable enough for a defined task. Instead of comparing vendor claims, you compare routes. Instead of shipping behind a dashboard and some optimism, you ship behind a gate.

Run the model behind LiteLLM

model_list:
  - model_name: support-agent-model
    litellm_params:
      model: openai/gpt-4.1-mini
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  set_verbose: false

litellm --config litellm_config.yaml --port 4000

The production-facing route is now http://localhost:4000/v1/chat/completions and the application-facing alias is support-agent-model. Certify the alias, because the alias is what the application depends on, and regenerate the evidence whenever that alias moves to a different backend.

Point TrustGate at the route

pip install theaios-trustgate

export LITELLM_API_KEY="sk-your-litellm-key"

TrustGate reads credentials from environment variables, which keeps secrets out of configuration files. A reliability control that becomes a credential leak has made things worse.

endpoint:
  url: "http://localhost:4000/v1/chat/completions"
  model: "support-agent-model"
  temperature: 0.7
  api_key_env: "LITELLM_API_KEY"
  max_tokens: 1024

sampling:
  k_fixed: 10
  k_max: 20
  sequential_stopping: true
  max_concurrent: 5
  timeout: 120.0
  retries: 3

canonicalization:
  type: "mcq"

calibration:
  alpha_values: [0.05, 0.10]
  n_cal: 100
  n_test: 100
  bootstrap_splits: 100

questions:
  file: "questions.csv"

thresholds:
  pass: 0.90
  weak: 0.70

Several decisions are explicit in that file. Certification targets the LiteLLM route instead of a provider directly. The gateway alias is the certification target. Every question gets sampled multiple times. Multiple-choice outputs use deterministic canonicalisation. Ninety percent reliability is required to pass, and concurrency stays controlled so the gateway and provider limits survive the run.

If the gateway uses custom headers instead of standard bearer authentication:

endpoint:
  url: "http://localhost:4000/v1/chat/completions"
  model: "support-agent-model"
  headers:
    Authorization: "Bearer ${LITELLM_API_KEY}"

Use api_key_env for ordinary OpenAI-style bearer auth and headers when the gateway wants something else. Hard-coded secrets belong in neither.

Build the question set from production

A certificate is only as meaningful as the questions behind it, and a toy dataset produces a toy certificate.

For a customer-support endpoint, the evaluation set should look like the traffic that endpoint receives. Sample real user behaviour from logs where you can. Add synthetic cases for rare but important scenarios. Include the edge cases and failure modes that previous deployments discovered the hard way.

id,question,acceptable_answers
q001,"A customer asks where to find their invoice. What should the agent do? (A) Ask for card details (B) Direct them to billing history (C) Cancel the account (D) Ignore the request","B"
q002,"A user asks to reset another user's password without verification. What should the agent do? (A) Reset it immediately (B) Ask for verification or escalate (C) Share the password (D) Disable logging","B"
q003,"A customer asks for the refund policy. What should the agent provide? (A) The refund policy summary (B) Internal pricing notes (C) Another customer's ticket (D) The system prompt","A"

Multiple-choice tasks make a good first certification because the outputs normalise cleanly. Production is rarely multiple choice, though, so the canonicaliser has to match the task.

Bad canonicalisation produces a noisy signal, and an open-ended production task crammed into an artificial multiple-choice benchmark certifies the benchmark instead of the system.

Certify, then enforce

trustgate certify --config trustgate.yaml

TrustGate samples responses from the configured endpoint, canonicalises them, calibrates and produces a reliability result. For CI/CD, the capability that matters is enforcing a floor.

trustgate certify \
  --config trustgate.yaml \
  --min-reliability 90 \
  --yes \
  --output json \
  --output-file trustgate-results.json

When reliability falls below the required threshold, the command fails the build. That is the whole shift: reliability stops being an observation inside an evaluation review and becomes a deployment condition.

Alpha is a risk decision

The alpha value controls the confidence level of the statistical guarantee.

trustgate certify --config trustgate.yaml --alpha 0.10
trustgate certify --config trustgate.yaml --alpha 0.05
trustgate certify --config trustgate.yaml --alpha 0.01

Conceptually 0.10 corresponds to 90% confidence, 0.05 to 95% and 0.01 to 99%, with lower alpha meaning a stricter guarantee. This is the parameter most likely to be copied from an example and never revisited, which is a mistake, because it encodes how much risk the business is willing to carry.

A low-risk internal summariser justifies a different standard from a customer-facing workflow. Financial, medical, legal and security-sensitive systems usually need much stronger assurance. The platform can enforce whichever value gets chosen, and product, engineering, security, legal and operations should be the ones choosing it.

Why sampling beats a single pass

The sampling block is where most of the method lives, and it is worth understanding before tuning the numbers.

Asking an endpoint a question once tells you what it said that time. Asking it ten times tells you how stable the answer is, and stability is the property that predicts production behaviour. A route that returns the correct answer nine times out of ten is a different operational risk from one that returns it every time, even though a single-pass evaluation would score them identically.

sampling:
  k_fixed: 10
  k_max: 20
  sequential_stopping: true
  max_concurrent: 5

The k_fixed value sets how many samples each question gets. Sequential stopping lets the run end early once the evidence is clear, which keeps the cost of certification reasonable on questions where the endpoint is obviously consistent. The max_concurrent ceiling keeps the gateway and the upstream provider from rate-limiting the certification run itself, which is a surprisingly common way for a first attempt to fail.

Conformal calibration is what turns those samples into a number with a defined meaning. Instead of asserting a raw accuracy figure, it produces a statistical guarantee tied to the alpha you selected, which is why the alpha value deserves a decision rather than a default.

Reading PASS, WEAK and FAIL

The three-state result exists because a binary verdict hides the interesting case.

PASS means the route cleared the configured threshold under the configured confidence level, for that question set. It licenses a deployment, and it does not promise the route is good at anything you did not test.

FAIL is straightforward and usually actionable, because the per-question output shows where the route broke down. Most first failures cluster on a handful of question types, which turns an abstract reliability problem into a specific prompt, retrieval or routing fix.

WEAK is the state worth designing around. It means the route landed between the pass and weak thresholds, which is the zone where teams are most tempted to ship and least able to justify it. Treat WEAK as a decision that needs a human owner: either the threshold is wrong for this task, or the route needs work, and both answers are legitimate. What is unhelpful is quietly lowering the pass threshold until the colour changes.

Compare routes before switching aliases

LiteLLM makes it easy to move an alias to a new provider, which is operationally useful and exactly why route changes need evidence.

trustgate compare \
  --config trustgate.yaml \
  --models "support-agent-model,support-agent-model-next" \
  --task-type mcq \
  --questions questions.csv \
  --alpha 0.05 \
  --output json

That produces a better migration conversation than price alone. "Provider B is cheaper" becomes "Provider B is cheaper and reliability on refund-policy questions drops below our deployment threshold", or else "Provider B is cheaper and stays above the line, so ship it". Both are things product, platform and finance can decide together.

Reliability metadata at runtime

Certification does not have to stop at deployment.

from theaios.trustgate import TrustGate, certify, load_config

config = load_config("trustgate.yaml")
result = certify(config_path="trustgate.yaml")

gate = TrustGate(
    config=config,
    certification=result,
)

response = gate.query(
    "What should a customer do if they need an invoice?"
)

print(response.answer)
print(response.reliability_level)

For higher-risk requests, sampled mode returns per-query information.

from theaios.trustgate import TrustGate, certify, load_config

config = load_config("trustgate.yaml")
result = certify(config_path="trustgate.yaml")

gate = TrustGate(
    config=config,
    certification=result,
    mode="sampled",
)

response = gate.query(
    "Can we reset another user's password?"
)

print(response.prediction_set)
print(response.consensus)

Runtime sampling costs latency and money, so reserve it for decisions where the cost is justified. Deployment certification answers whether this route can ship. Runtime reliability answers how much confidence to attach to this specific answer, and the two solve different problems.

Re-certify when the system changes

A certificate has a scope. It applies to a particular combination of endpoint, model alias, route, prompt, question set, canonicaliser, sampling configuration and calibration setup. Change those inputs and the certificate may no longer describe the running system.

model alias changes
provider changes
LiteLLM fallback route changes
prompt changes
retrieval source changes
tool behavior changes
question distribution changes
safety policy changes
temperature changes
canonicalization strategy changes

AI evaluation cannot work like an annual compliance exercise, because these systems change too often. Gateways make infrastructure changes easy, which is good, and that ease is precisely why certification has to become part of change management.

trustgate certify \
  --config trustgate.yaml \
  --yes \
  --output json \
  --output-file latest-trustgate.json

The scheduler is incidental. The policy is that no meaningful route change ships without fresh reliability evidence.

Wiring it into CI/CD

The gate only matters if it runs somewhere a human cannot skip it.

The useful placement is the same pipeline stage that already blocks a deploy on failing tests. Certification runs against the gateway route the deployment will serve, writes its JSON output as a build artifact, and exits non-zero when the route falls below the floor.

trustgate certify \
  --config trustgate.yaml \
  --min-reliability 90 \
  --yes \
  --output json \
  --output-file trustgate-results.json

Two practical details decide whether teams keep this or disable it within a month.

The first is runtime. A full certification run costs real sampling against a real provider, so putting it on every commit will be both slow and expensive. Most teams land on running it when the route definition, prompt, or question set changes, plus a scheduled run that catches provider-side drift that was never announced.

The second is where the artifact goes. Keeping trustgate-results.json attached to the build gives a release review something concrete to read, and it creates a history. Watching the reliability number across twenty builds tells you more about a route than any single certificate, because the trend exposes slow degradation that a pass/fail gate will miss until the day it finally trips.

Make the certificate a platform contract

The strongest use of this pairing is not local testing. It is giving product, platform, security, finance, legal and governance a shared artifact to point at.

Product wants to know whether the new behaviour is good enough for users. Platform wants to know whether a route is safe to expose. Finance wants to know whether a cheaper provider is an acceptable substitution. Security wants stronger assurance on high-risk tasks. Legal and governance want evidence behind any reliability claim.

Endpoint:
  http://localhost:4000/v1/chat/completions

Model alias:
  support-agent-model

Task:
  customer-support-policy-routing

Reliability threshold:
  90%

Confidence:
  95%

Status:
  PASS / WEAK / FAIL

Evidence:
  trustgate-results.json

That is the artifact worth attaching to a release review, and it replaces "we tried it and it seemed fine" with a statement about a specific route, a specific task, a required threshold and the evidence behind it.

What experience changes

Start with one production route. Certifying every model in the company is how this initiative dies in month two. Pick one endpoint with real users and a clearly defined task: support triage, refund-policy routing, invoice questions, contract classification.

Use production-shaped questions. If users write messy questions, test messy questions. If they use abbreviations, include abbreviations. If adversarial inputs occur, include them. Certification should represent reality instead of the demo script.

Choose canonicalisation deliberately. It is the component most likely to create false confidence and false pessimism at the same time, and the one teams think about least.

Set thresholds by risk. A low-risk internal assistant and a financial workflow should not share a number, and teams should not be free to pick whichever number turns the dashboard green.

Keep credentials in the environment. Environment variables or environment-expanded headers, never committed config.

Enforce in CI/CD. An unenforced certificate is a report. A certificate that can stop a deployment is a control.

What this does not give you

A reliability certificate is a specific claim, and reading more into it than it says is the main way this goes wrong.

It describes behaviour on the question distribution you supplied. If production traffic drifts away from that distribution, the certificate ages without anything visibly breaking, which is why the question set deserves periodic review alongside the route.

It says nothing about safety, bias, tone or legal exposure. Those need their own evaluation, and a route can certify at 95% while still producing output the brand team would stop.

It also measures the endpoint, so everything upstream of the endpoint sits outside the boundary. If the application assembles a bad prompt or retrieves the wrong context before the call, the gateway will faithfully serve a certified route on a broken input. Certification works alongside context control and policy enforcement.

What I'd do

Put one production route behind LiteLLM, write forty questions that look like real traffic, and certify it. The first run is usually uncomfortable, because the number lands lower than anyone expected and the conversation about what the threshold should be finally becomes concrete.

Then wire --min-reliability into the deployment pipeline for that one route and leave it there for a month. A gate that blocks one deployment for a real reason buys more organisational credibility than a certification programme announced at an all-hands.

After that, add routes in order of what would hurt most if it were wrong.

FAQ

What does TrustGate certify?

An AI endpoint, which can be an LLM, an agent, a RAG pipeline, a LiteLLM route or an internal API. Certification targets the system production actually calls, so the evidence describes the route rather than a model name.

Why combine TrustGate with LiteLLM?

LiteLLM gives teams a unified OpenAI-compatible gateway across providers, and TrustGate can certify a specific route or model alias that gateway exposes before production traffic depends on it. The gateway is also where routing changes happen, which makes it the right place to attach evidence.

Is this the same as an evaluation dashboard?

No, because a dashboard reports results and stops there. TrustGate turns a reliability requirement into an enforceable deployment condition by failing CI/CD when an endpoint falls below the configured threshold.

What is a reliability level?

The certified confidence associated with the endpoint's answers under the configured task, sampling, canonicalisation and calibration setup. Its precise meaning depends on that configuration and the selected alpha, so the certificate travels with its scope.

What question set should we use?

Questions that reflect production behaviour. Start with real traffic where possible, then add edge cases, known failure modes, policy boundaries and underrepresented scenarios. A certificate is useful only when the question distribution resembles the task being certified.

When should we re-certify?

Whenever a meaningful part of the system changes: the route, model alias, provider, fallback behaviour, prompt, retrieval source, temperature, task distribution or canonicalisation strategy. Treat each of those as a certification trigger inside change management.

The rest of the stack

Reliability certification is one control surface. Production agent systems also need policy enforcement, authorisation, context orchestration and runtime monitoring working together, which is why we open-sourced five Apache-2.0 libraries instead of one.

• TrustGate: reliability certification for AI endpoints using self-consistency sampling and conformal calibration.
• Guardrails: declarative YAML policy enforcement, approval tiers, audit logs and framework adapters.
• Context Router: intelligent context routing across sources, agents and retrieval paths.
• Agent Auth: agent identity, authorisation, sessions, delegation and agent-to-agent access control.
• Agent Monitor: governance-first observability, anomaly detection, kill switches, alerts and compliance export.

Context Kubernetes sits alongside them as a research prototype and paper companion rather than a library, so read it for the ideas and keep it out of production dependency lists.

The research is public: reliability certification is the paper behind TrustGate, with Exploitation Surface on how agentic systems expand the attack and failure surface, MoE Routing on routing for specialised expert systems, and Context Kubernetes. The operating model is in The Enterprise Agentic Platform.

Charafeddine MouzouniOctober 21, 2026