When does Jev routing actually reduce LLM costs?

Adding Jev saves money only when the work it avoids costs more than the Jev call, the accepted route, fallback and any extra overhead. A very cheap decision call can still increase your bill if almost every request goes on to the same LLM, or if a wrong route triggers another full attempt.

This guide provides a break-even formula, three worked scenarios and a downloadable Python calculator. It is for developers comparing complete routing architectures. The calculations are illustrative and were tested locally; they are not live Jev measurements or Ofox price quotes. For Jev’s interface and limitations, see the Jev API introduction.

Decide which architecture you are comparing

Start with the system you would deploy if you did not add Jev. A strong-model-only system is one baseline, but a rules layer or a cheap generative model may be a better economical alternative.

A router that chooses a cheaper model is not eliminating generation: the chosen model still needs to produce the final answer. If the accepted branch is truly deterministic, its model cost may be zero, but its operational cost is not necessarily zero. Be explicit about the boundary of the calculation.

Published research also warns against choosing only a convenient baseline. REFLEX reports benefits in a controlled agent benchmark, while external evaluations show limited advantages over a cheap generative cascade. Treat that as a reason to include the cheap cascade, not as a universal verdict for or against Jev. REFLEX paper.

One concrete REFLEX result illustrates why the baseline matters. In its τ²-bench validation, the reported per-episode costs were $0.2111 for the strong-only system, $0.0572 for REFLEX, and $0.0411 for the cheap cascade. Observed success rates were 90.0%, 85.0% and 91.7%, respectively. The paired success differences were statistically unresolved, so these are not proof of equal quality or a universal winner. They do show why quoting the strong-only cost reduction while omitting the cheaper cascade would mislead a buyer.

The paper’s separate cross-family experiment distinguishes calls from dollars: Qwen strong-model calls fell 71.9%, while verified monetary cost fell 52.2%. Verified monetary savings were not supplied for the Kimi and DeepSeek rows. Your bill depends on token lengths and rates, not just the number of calls removed. These are the paper’s historical results, not the calculator’s assumptions or our measurements.

Use the correct Jev billing unit

As checked on October 2, 2026, TypeSafe’s direct model documentation lists jev-1.13.0 at USD 0.042 per million input tokens, with output tokens free. The page also distinguishes the total request budget from the state-plus-longest-question budget. This is TypeSafe’s direct published rate, not an Ofox offer or a promise about every gateway. Current model documentation.

Original English documentation screenshot captured October 2, 2026. Recheck the live source before budgeting; the calculations below preserve this dated rate.

Include all billable input in the request: state and questions, not just the short instruction you see in your application code. If you repeat the same state in separate requests, account for those inputs each time unless your provider’s documented billing explicitly says otherwise. Do not invent a cache discount that the price source does not provide.

For an illustrative 2,000-input-token request with one billed attempt:

Jev cost = 2,000 × 0.042 / 1,000,000
         = USD 0.000084 per request
100,000 such requests = USD 8.40

That USD 8.40 covers the Jev decision stage under these assumptions. It says nothing about downstream generation, repeated attempts, monitoring or the cost of a wrong action.

Calculate request cost, then check successful-task cost

Let J be expected Jev cost per incoming request, including billed attempts. Let a be the share of incoming requests accepted onto the cheaper branch, L its average downstream cost, H the fallback branch’s average downstream cost, and X additional expected overhead. All costs use the same currency and per-request boundary.

For a simplified two-branch system:

Direct cost per request = H
Routed cost per request = J + a × L + (1 − a) × H + X
Savings per request    = a × (H − L) − J − X
Break-even acceptance  = (J + X) / (H − L), when H > L

a must be measured over all eligible incoming requests, including those with routing failures. It is not the model’s confidence value. For example, a threshold of 0.9 does not imply 90% acceptance. Use the confidence evaluation workflow to estimate coverage on representative labeled data.

This simple formula assumes a fallback request costs H and an accepted route costs L. If a cheap handler runs and then escalates, that branch pays both costs; put the observed conditional average into the model or use a more detailed branch table. If fallback requests are much longer than ordinary requests, do not reuse a global average that hides the difference.

Three scenarios that change the decision

The following downstream rates are hypothetical, chosen to make the arithmetic inspectable. They do not describe a named provider or measured model. All scenarios use 100,000 incoming requests and the dated Jev decision cost above.

In the first scenario, break-even acceptance is about 0.933% because each accepted request avoids a large cost difference. In the third, it is 84% because the cost difference between branches is small. Neither number is a recommended threshold or a forecast of real acceptance.

The first scenario’s attractive savings still need a quality test. If its accepted branch gives unusable results, you have not saved the cost of delivering the same service. Compare cost per accepted, correctly completed task as well as cost per incoming request. Keep the success definition identical across systems.

For an illustrative quality check on the first scenario, suppose the direct system successfully completes 95,000 of the 100,000 requests, while routing completes only 40,000. Direct cost per success is $1,000 / 95,000 = $0.01053; routed cost per success is $468.40 / 40,000 = $0.01171. The lower total bill now buys a more expensive successful result and far more failures. Those success counts are invented solely to demonstrate the denominator, not measured outcomes. The supplied calculator does not model quality or compute this metric automatically.

Use a trace-level acceptance test to obtain real success counts. Report both unresolved and incorrect tasks, and include any paid rework already incurred. If business costs from failures or human review matter, keep them in the same accounting scope for both systems; API cost per success alone cannot price every consequence.

Run and change the calculator

Download the offline decision kit, unzip it and run from its directory:

python3 cost.py
python3 cost.py --accepted 0.005
python3 cost.py --cheap 0.0001 --strong 0.0002

The default run returns direct_total: 1000.0, routed_total: 468.4 and savings: 531.6. We executed this locally. The script uses Python’s standard library, performs no API calls and needs no key.

Replace the assumptions with observed inputs:

python3 cost.py --requests 100000 --tokens 2000 \
  --rate 0.042 --attempts 1.2 --accepted 0.6 \
  --cheap 0.001 --strong 0.01 --extra 0.0001

Here --attempts 1.2 is an assumed mean of billed Jev attempts, not a claim that every retry is charged. Verify billing from your provider’s usage records. --extra is expected added cost per incoming request, such as paid verification or the measured incremental cost of escalation beyond the simple branches. Do not include a cost twice if it is already in cheap or strong.

The calculator reports no ordinary break-even ratio when the strong branch is no more expensive than the cheap branch. That is a useful signal to inspect the architecture, not an arithmetic error to work around. It also rejects negative or nonfinite inputs and acceptance outside 0–1.

Account for retries, caches and latency

Retries: distinguish a transport attempt, a billed request and a completed task. A timeout does not tell you whether the upstream processed or billed the request. Use request IDs and usage records. Set bounded retries and avoid retrying downstream side effects without idempotency controls.

Cache behavior: changing, shortening or reordering a prompt can change cache reuse in a downstream system. A routing layer may reduce input length while also reducing cache hits. Measure the actual downstream bill with the new prompt pattern; do not subtract a theoretical token saving from an old cache-hit bill.

Shared state: multiple independent questions can sometimes fit in one Jev request. That may avoid resending state, but the context constraints and question semantics still apply. Questions that depend on earlier answers need a real dependency in code rather than an assumption that answers in one request feed each other. TypeSafe’s fan-out pattern.

Latency: on a fallback path, a sequential Jev decision adds work before the strong model. Compare complete request p50 and p95, including retries and queues. A lower average token bill does not prove a faster tail response. Parallel speculative execution can reduce waiting but pays for work you may discard, so its cost model differs from this two-branch calculator.

Validate with a trace table, then a small rollout

Log one row per incoming task with its ID, actual model version, routing status, selected branch, all attempt IDs, billed usage, final outcome and end-to-end duration. Keep prompt and question versions in the manifest. Without those fields, a later price or prompt change can look like a routing improvement.

Use shadow mode to observe proposed routes while your current system produces the authoritative result. Label enough cases to inspect accepted errors and subgroup behavior. Then replay or carefully test the alternative handlers under the same acceptance rule. A shadow label alone cannot tell you the quality or bill of a downstream call that never ran.

Move to a limited rollout only after quality, latency, fallback capacity and expected costs all pass your written criteria. Reconcile estimates with actual usage. If savings disappear, identify whether acceptance changed, branch costs rose, retries increased or cache behavior changed before changing the threshold.

For evidence about Jev’s strengths and limits, read the benchmark interpretation guide. It helps keep model quality claims separate from your application economics.

The decision to make

Use Jev when a validated bounded decision sends enough traffic to a genuinely cheaper successful path. Keep a rules layer or cheap model cascade in the comparison. If nearly everything still needs the same expensive model, adding a router is another stage to pay for and maintain.

The next useful action is to replace the calculator’s hypothetical L, H and a with your own branch costs and measured acceptance. That produces a budget you can inspect, rather than a savings claim borrowed from somebody else’s workflow.