Run Laya on Apple's Neural Engine with Core ML (~5 ms)

The Core ML port of Laya is the fastest way to run a decision model on Apple hardware: validated weights, decisions in the region of 5 milliseconds, and the Neural Engine doing the work while the CPU stays asleep. It is the configuration that makes an on-device decision layer plausible in an app that must not drain a battery.

Why the Neural Engine changes the design

Once a decision costs milliseconds and almost no power, features that used to be "server-side only" become local: ranking, spam triage, content classification, per-keystroke routing between a small and a large on-device model. The published energy benchmarks matter more than the latency headline here — a fast model that spins up the GPU for every call will still eat a laptop battery, and ANE-targeted inference is the fix.

What the port ships with

• Converted Laya weights validated against the reference implementation, so answers match the Python package rather than drifting in conversion.
• Speed and energy benchmark scripts, so the ~5 ms and the power numbers are reproducible on your own machine.
• Short-decision benchmarks on M-series hardware — the workload the model was designed for, not long classification prompts.

Setup path

1. Take the Core ML artefacts from the port repository (link on the projects page) and add them to your Xcode target or Swift package.
2. Load the model once at launch and keep the handle alive; per-request model creation is the usual reason someone measures 40 ms instead of 5 ms.
3. Send the same typed question structure used everywhere else — choice, score, noul with criteria spelled out. The port does not change prompt design, only where inference happens.
4. Threshold on the returned probabilities. On-device means you can afford to escalate the uncertain cases to a bigger model or to a human, and you should.
5. Re-run your labelled sample after any conversion or OS update; ANE kernels change between releases.

Gotchas worth knowing before you ship

• Tokenisation is the hidden cost. On-device tokenisers in Swift are easy to mismatch against the Python reference; validate a few hundred strings byte-for-byte.
• Model size vs accuracy. Aggressive quantisation targeted at ANE-friendly shapes can move score answers more than choice answers. Measure per question type, not overall.
• Thermals, not averages. Sustained throughput on a phone in a pocket is a different number from a benchmark on a cool desk.

If your target is a Mac app rather than iOS, the MLX port is the more flexible sibling: slightly higher latency, easier experiments. For everything else — servers, CI, Linux boxes — the plain local install is enough.

Last updated: 2026-09-21 · sources & corrections