Jev Agent Routing with LangChain & Vercel

In an AI agent, Jev handles the small decisions between LLM calls: which tool or skill to use, which model should take the next step, whether the agent should continue or hand over, and whether a tool call looks too risky to run. TypeSafe and its partners document patterns for intent routing, model choice, continue/stop decisions and tool-risk gating; a pattern for deciding retries is not documented. Before moving any of these decisions from an LLM to Jev, replay your own logged decisions through both and compare accuracy, latency and cost.

This guide collects what the official docs from TypeSafe, LangChain, Vercel and Cloudflare show, as of October 2026, and marks clearly where we are describing a design idea instead of a documented feature. We have not deployed Jev in a client system. For background on the model itself, see what Jev is.

Where do routing decisions sit in an agent?

Most agents run a loop: read the task and the latest tool result, decide what to do next, act, repeat. The “decide” part usually hides several separate questions that an LLM answers implicitly inside one long prompt. A decision model makes them explicit. The sketch below is vendor-neutral pseudocode, not Jev API code:

loop:
    state = compact summary of task, last tool result, step count
    answers = decision_model(state, [which_tool, needs_strong_model,
                                     tool_call_risky, should_continue])
    if any answer is below its confidence threshold:
        hand the step to an LLM or a person
    elif tool_call_risky is likely:
        refuse the call, or ask for approval
    else:
        run the chosen tool with the chosen model
    if not should_continue: stop

Two points from TypeSafe’s docs shape this design. Questions should be atomic, with broad judgements split into several questions and combined in code. And the state should be short: Jev “suffers from context rot” when the state is full of irrelevant detail (Jev 1.13 jaggedness). Passing the full transcript is a poor default; pass what the decision needs.

Intent and tool routing

TypeSafe’s intent routing pattern is the clearest official example. A Choice question classifies the request, a Score question rates complexity, and plain code decides where it goes: order-status queries to deterministic code, product and returns questions to specialist LLM prompts, complaints to an LLM or a human depending on complexity. Low confidence goes to a human. The routing function, as published:

def route_ticket(ticket_id, response):
    intent = response.answers["intent"]
    complexity = response.answers["complexity"]

    if intent.confidence < 0.5:
        # If we don't have enough confidence to classify, route to a human agent
        return route_to_human_agent(ticket_id)

    if intent.choice == "order_status":
        handle_order_status(ticket_id)
    elif intent.choice == "product_question":
        handle_with_llm(ticket_id, PRODUCT_SPECIALIST)
    elif intent.choice == "return_exchange":
        handle_with_llm(ticket_id, RETURNS_SPECIALIST)
    elif intent.choice == "complaint":
        low_confidence = complexity.confidence < 0.5
        # A higher complexity.score leans toward the "escalation needed" end of the scale.
        if complexity.score > 1 or low_confidence:
            # Too complex for safe automation, or we're not sure about the complexity; route to a human.
            route_to_human_agent(ticket_id)
        else:
            handle_with_llm(ticket_id, COMPLAINT_RESOLUTION)

For tool selection proper, TypeSafe has two cookbooks. The function calling cookbook maps a function name and its closed-set arguments (literals and booleans) to Choice and Noul questions, one request per command. Integer, free-text and date arguments keep their defaults, which tells you where the approach stops: Jev picks the function, but it does not fill in open values. The skill suggestion cookbook uses two Jev requests to pick at most one of 182 skills. In TypeSafe’s own test of 488 requests against Claude Haiku 4.5, the rate of loading the wrong skill fell from 16.8% to 7.3%, and loading a skill when none fit fell from 9.8% to 4.0%. That is a vendor result on a vendor-chosen task.

Two limits matter for tool menus. A Choice holds at most 255 options, and TypeSafe’s docs note that Jev “leans toward the option that comes first”. If your tool list has a fixed order, test whether the first tool wins more often than it should.

Model choice with LangChain

LangChain maintains a langchain-typesafe package (alpha, 0.0.1a3 at the time of writing) with a TypeSafeClassifier Runnable and two middleware classes marked experimental. ModelRouterMiddleware picks which model runs a step. This excerpt is from the LangChain provider page:

from langchain.agents import create_agent
from langchain_typesafe.experimental.middleware import (
    ModelChoice,
    ModelRouterMiddleware,
)

router = ModelRouterMiddleware(
    choices={
        "fast": ModelChoice(
            model="openai:gpt-5.6-terra",
            criteria="Direct lookups, extraction, and localized changes with explicit targets.",
        ),
        "powerful": ModelChoice(
            model="openai:gpt-6-astra",
            criteria="Architecture, novel root-cause reasoning, and high-stakes decisions.",
        ),
    },
    instructions="Choose the least costly model that can complete the task safely.",
)

agent = create_agent("openai:gpt-5.6-terra", middleware=[router])

The full example on that page continues with agent.invoke(...) and reads the chosen route from result["model_route"].choice. Model routing is also one of the uses Armin Ronacher, CTO of Earendil, suggested when TechCrunch asked him about Jev. The economics are simple to reason about: the router call is cheap, so it pays for itself if it sends enough steps to the cheaper model without hurting outcomes. Whether it does is exactly what you have to measure. Our LLM cost comparison gives the price gap between model tiers that such a router would exploit.

Retries, continuing and escalation

Here the documentation is thin, and it is worth being precise. TypeSafe’s official retry content is about transport: the SDKs retry 429 and 529 errors with exponential backoff and honour retry-after. There is no official pattern for using Jev to decide whether an agent should retry a failed step.

The closest documented example is Vercel’s continue/stop check, from its AI Gateway changelog:

import { TypeSafeClient } from '@typesafe-ai/sdk';

const client = new TypeSafeClient({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/typesafe',
});

const result = await client.systemOne({
  state: 'The agent fixed the checkout bug and all tests pass.',
  questions: {
    continueWorking: {
      type: 'noul',
      instructions: 'Should the agent take another step?',
    },
  },
});

const shouldContinue = result.answers.continueWorking.noul >= 0.8;

A retry decision could follow the same shape, and we present it only as a design idea. After a failed tool call, ask yes/no questions such as whether the error looks transient, and whether the same call with different arguments is likely to succeed. Keep the retry count and backoff in code; Jev’s docs say it does not count reliably. If the answers are uncertain, escalate instead of retrying.

For escalation thresholds, TypeSafe’s confidence routing pattern shows a voice-banking flow with a confidence floor of 0.6 and auto-approval of a transfer only above 0.85. Note that Noul answers have no confidence field, so for yes/no questions the threshold sits on the probability itself, as in Vercel’s >= 0.8. An independent test by AY Automate used a 0.80 confidence cut-off to send uncertain cases to GPT-5.6 Terra and reached 90.0% accuracy at about 26% of Terra’s cost. Those numbers are theirs, on their data; your thresholds have to come from yours.

Tool-risk gating

LangChain’s experimental AutoModeMiddleware(tools=[...]) “asks for the probability that a tool call is risky or insufficiently authorized”. Calls judged risky return an error ToolMessage instead of running. LangChain’s own warning is worth repeating: “This middleware refuses risky calls. It does not request approval.” If your agent needs a human approval step, you build that yourself.

The bigger caveat is prompt injection. Jev’s docs say state is data and the model “does not treat it as hostile by default”. VentureBeat reported an Octomind test in which a fake “pre-approved” field moved the block probability for rm -rf ~/.ssh from 0.76 to 0.48. Pydantic’s position, quoted in the same piece, is the sensible one: a Jev guard “belongs alongside deterministic checks, not instead of them”. In practice that means allow and deny lists, argument validation and permission checks in code first, with Jev scoring the grey zone that rules cannot cover.

Integration notes: LangChain, Vercel, Cloudflare and Pydantic

LangChain: langchain-typesafe is maintained in the langchain-ai repository but is alpha, and both middleware classes are experimental. Expect API changes.
Vercel AI Gateway: the model is typesafe-ai/jev, reachable through the TypeSafe client with a gateway base URL (as above), an HTTP /v1/evaluate endpoint, or the AI SDK via @ai-sdk/typesafe-ai. Vercel’s pages disagree on the AI SDK function name (experimental_evaluate in the changelog, experimental_decide in the newer integrations guide), so check the current docs before copying code. The gateway’s native API calls the yes/no type boolean with a .probability field, where TypeSafe uses noul. Vercel also says its eve agent framework uses Jev as the default evaluation model.
Cloudflare Workers AI: the call is env.AI.run('typesafe/jev', { state, questions }) with TypeSafe’s question schema (model page). Cloudflare lists zero data retention and a 32,000-token context window, against the 64k per request that TypeSafe states.
Pydantic AI: TypeSafeModel and TypeSafeProvider via the pydantic-ai-slim[typesafe] extra, with your own TypeSafe key and no streaming (Pydantic).

Whatever the route, pin the model version (jev-1.13.0) once you have tuned thresholds. The jev-latest alias will move.

How to evaluate Jev on your own traffic

Published benchmarks disagree, and TypeSafe’s own evaluations score against labels produced by two frontier LLMs, so they measure agreement, not correctness. The only numbers that should decide a switch are ones from your own traffic. A workable method:

Pull logged decisions from your agent traces: the state at decision time, the choice your current LLM made, and the outcome where you know it.
Label a sample by hand, including the hard and costly cases. Using another LLM as the reference repeats the problem with vendor benchmarks.
Write each decision as atomic Jev questions. Expect to iterate; in the phishing test summarised by Beri.net, splitting one broad question into five changed the result more than the choice of model did.
Replay the sample offline through Jev and through your current LLM baseline.
Compare per decision type: accuracy or macro-F1, the confusion matrix for the expensive mistakes, and calibration. Fit thresholds per question; Beri.net found Jev’s calibration unreliable out of the box.
Measure coverage at each threshold: what share Jev handles alone, and what share escalates to the LLM.
Measure latency p50 and p95 from the region you deploy in. Vendor figures were taken from the US West Coast.
Compute cost per 1,000 decisions, including the LLM calls for escalated cases.
Add an adversarial set with injected instructions in the state, especially for tool-risk questions.
Run in shadow mode on live traffic before the cut-over, and keep the LLM path as fallback.

If the result is close, the deciding factors tend to be the volume of decisions and how costly each type of error is. A guide to the wider cost picture of agents is in our article on AI agent development cost, and the broader trade-offs are in Jev vs LLMs.

FAQ

Does Jev work with LangChain?

Yes. LangChain maintains langchain-typesafe, which has a TypeSafeClassifier Runnable plus experimental ModelRouterMiddleware for model choice and AutoModeMiddleware for tool-risk gating. The package is in alpha.

Can Jev choose tools for an AI agent?

Yes, for closed choices. TypeSafe’s function calling cookbook maps function names and closed-set arguments to Choice and Noul questions, while integer, free-text and date arguments need another source. A single Choice holds up to 255 options.

Does Jev have a built-in retry policy?

Only for transport errors: the SDKs retry 429 and 529 responses with backoff. Deciding whether an agent should retry a failed step is not a documented Jev pattern, though Vercel shows a related continue/stop check.

Is Jev safe to use as the only guard on tool calls?

No. Injected content in the state can shift its answers, as an Octomind test reported by VentureBeat showed. Use it alongside deterministic permission checks, and note that LangChain’s middleware refuses risky calls but does not ask for approval.

How should I set confidence thresholds for routing?

Fit them per question on your own labelled data. TypeSafe’s patterns use 0.5, 0.6 and 0.85 in different examples, and Vercel’s example uses 0.8 on a Noul probability, but each was chosen for a specific example.

Building the routing layer

A decision model changes the shape of an agent: routing logic moves out of prompts and into typed questions, thresholds and code you can test. That can make the system cheaper to run and easier to audit, provided someone owns the evaluation and the fallbacks. AI Superior designs and builds AI agents and LLM applications for B2B clients, with model routing, offline evaluation and cost control as part of the architecture. See our LLM deployment services if you are planning this for production.

Jev Agent Routing: Tool Selection, Model Choice and Retries