How to Run Laya Locally: pip, ONNX, MLX and Core ML

Laya is small enough that "local" is not a project — it is a dependency. Pick the runtime that matches your machine, because they differ by an order of magnitude in latency.

Python (reference path)

pip install laya

# then build a decision the same way you would for Jev
# state in, typed questions out, probabilities attached

The reference package is the one to start with: same question semantics as the hosted API, one process, no server. Expect a second or two of model load and tens of milliseconds per decision on CPU.

Node / JavaScript, via ONNX

npm install @receptron/laya

The ONNX export runs through onnxruntime-node, which makes it the pragmatic choice when the decision lives in a serverless function or a desktop app written in TypeScript. The trade is cold-start load time — keep the session warm in long-running processes.

Apple silicon: MLX

The MLX port is the general-purpose GPU path: 7–14 ms for short decisions on an M3 Max, with FP16 quantisation, and a playground for experimenting without writing glue code. Use it when your workload is latency-sensitive but not battery-bound.

Apple silicon: Core ML and the Neural Engine

The Core ML port runs on the Neural Engine and is the fastest path measured so far — on the order of 5 ms per short decision, with reproducible speed and energy benchmarks in the repository. That is the one to pick for always-on features in a desktop or iOS app: the power draw is low enough that the classic "call home to a GPU" design stops making sense.

Sizing and warm-up

RAM: budget 1–2 GB for the model plus your process; a 421M parameter model in FP16 is small, but tokenizer and runtime overhead are not zero.
Load once: initialise the model at process start. Per-request cold starts dominate every latency table above.
Batch when you can: decisions are independent, so throughput scales better with batch size than with prompt tuning.
Quantise deliberately: smaller weights cost accuracy on the score and noul types first. Re-run your labelled sample after any quantisation change.
Watch the tokenizer: multilingual checkpoints are a different artefact from the English one. Pick based on the actual text you will feed it.

Getting the weights

Weights and model cards are published on Hugging Face by the ConvAI Innovations team; the ports live under separate maintainers (MLX and Core ML repositories) with their own benchmark scripts, so you can reproduce the latency numbers on your own hardware instead of trusting a table. That reproduction habit is the point — every number on this page exists because somebody published a script next to it.

Next: Core ML walkthrough · MLX walkthrough · Laya vs Jev

Last updated: 2026-09-21 · sources & corrections