Jev vs Laya Benchmarks: Accuracy, Calibration and Latency
Numbers below are quoted from public model cards, repository benchmark scripts and launch-week write-ups, with the caveat that both projects shipped multiple versions inside the first week. Re-run anything you intend to build on.
Accuracy on the shared typed-decisions benchmark
| Model | Routing accuracy | Notes |
|---|---|---|
| Laya (open, ~421M) | 0.766 | Self-published on the public decision benchmark |
| Hosted frontier model (Jev 1.13.0) | 0.727 | Reference in the same comparison |
| openJev-verdict-2.0 (151M, community) | 0.771 acc, Brier 0.0636, ECE 0.0144 | Community checkpoint claiming both accuracy and calibration |
| open-jev typed decision engine (150M, community) | 0.697 | Earlier community attempt, still short of both |
Calibration
Calibration is the property that makes a probability usable: a 0.8 should be right about eight times in ten. Laya's published checkpoint reports Brier ≈ 0.062 and ECE ≈ 0.213, with a score-answer MAE of about 0.242. The community verdict model advertises a much lower ECE (0.0144), which is the number to be most sceptical of — ECE is sensitive to binning choices, so ask for the script, not the summary.
Latency
| Runtime | Short decision latency |
|---|---|
| Core ML on the Apple Neural Engine | ~5 ms |
| MLX, M3 Max, FP16 | 7–14 ms |
| Reference Python, GPU | ~35 ms |
| Reference Python, CPU only | Tens of ms; runs in 1–2 GB RAM |
| Hosted API | Sub-second including network, from developer demos |
Cost
- Hosted: ~$0.042 per million input tokens, output unmetered. Developer-reported runs: ~5,000 mixed decision requests for about $2; a browser agent flight search at $0.0039.
- Local: zero marginal cost; the bill is hardware and your time. On a laptop that already exists, that arithmetic is hard to beat at volume.
Fair-comparison checklist
- Score on your own labelled sample, not a public set — a few hundred real cases minimum.
- Report accuracy and calibration; an accurate model with poor calibration destroys a threshold-based pipeline.
- Separate the three question types. A model can be strong at choice and weak at score.
- Measure p95 latency, not mean, and include model load in cold-start paths.
- Pin every model version next to the number, or the table is meaningless within a week.
Last updated: 2026-09-21 · sources & corrections