In-Browser LLM Inference: Why Your Next AI App Might Never Touch a Server
I spend most of my working life gluing things together: n8n pipelines, Claude API calls, HubSpot workflows, the odd Supabase table that should never have grown as large as it has. So when I kept seeing claims that a browser tab could now run a proper large language model, no server, no API key, no round trip to someone else's data centre, I was sceptical in the way you only get after enough production incidents.
I went and read three papers to settle it for myself: a Microsoft Research study on just-in-time kernel generation for in-browser inference, an Intel developer guide to the current JavaScript frameworks, and the technical paper behind WebLLM itself. What follows is what I took away from all three, filtered through the eyes of someone who actually has to ship this sort of thing.
Why Would You Even Want This?
The pitch is genuinely appealing. Keep the user's data on the user's device instead of sending it across the internet, which matters given how often data breaches get cited as a reason to avoid cloud round trips in the first place (Jia et al., 2024). Cut cloud compute costs to zero for that workload, a point both Google and Microsoft have made about their own on-device deployments (Jia et al., 2024). Get instant responses with no network latency. And ship one codebase that behaves the same on Windows, macOS, iOS and Android (Singh and Ravi, 2025).
There's a line in the WebLLM paper that stuck with me: the browser is a naturally agentic environment, already the place where people manage calendars, write emails and draft documents, so an LLM that lives there is sitting right where the work happens rather than bolted on from outside (Ruan et al., 2026). As someone who builds agentic workflows for a living, that framing makes a lot of sense to me.
What's Actually Happening Under the Bonnet
Running deep learning inside a sandboxed browser tab is not a small ask. Singh and Ravi (2025) break the stack down into four layers: the hardware itself (CPU, GPU, NPU), the OS-native APIs that manage it, the browser APIs that expose it safely, and the JavaScript frameworks that abstract all of that away so the rest of us don't have to think about it daily.
Four layers from top to bottom: JS Frameworks, Browser APIs, OS Native APIs, and Hardware. JS FRAMEWORKSTransformers.jsONNX Runtime WebWebLLM BROWSER APIsWebGL / WebGPUWebNNWebAssembly OS NATIVE APIsmacOS / iOSWindowsAndroidLinux HARDWARECPUGPUNPU
The in-browser AI stack: frameworks sit on top of browser APIs, which reach down through the OS to the hardware itself (adapted from Singh and Ravi, 2025).
Two browser technologies are doing almost all the heavy lifting:
WebAssembly (Wasm) is a compact, portable binary format that runs inside a virtual machine in the browser. It's language-agnostic, so C and C++ code compiles straight into it, and it takes advantage of modern CPU features like SIMD, giving it a real edge over plain JavaScript for this kind of work (Jia et al., 2024).
WebGPU, the successor to WebGL, gives direct, low-level access to the GPU for general computation rather than just rendering graphics. WebGPU code gets translated into native GPU APIs such as Vulkan, Metal or Direct3D depending on what you're running (Jia et al., 2024).
A newer addition, WebNN, is being built specifically to reach the dedicated NPUs now turning up in AI PCs, which is going to matter more and more as laptop chips ship with neural processing hardware baked in for efficient local inference (Singh and Ravi, 2025).
The Frameworks Worth Knowing About
Three frameworks currently lead the conversation, and each one takes a genuinely different approach.
WebLLM is built specifically for LLM inference and offers an OpenAI-style API, so code already written against the OpenAI client needs surprisingly little rework to point at it instead (Ruan et al., 2026). Behind the scenes, it uses machine learning compilers, MLC-LLM and Apache TVM, to compile models ahead of time into optimised WebGPU kernels, since WebGPU doesn't come with anything like CUDA's mature kernel libraries out of the box (Ruan et al., 2026). It also runs inference inside a web worker, a background JavaScript thread, so a model chewing through tokens doesn't freeze the page.
Bar chart showing decoding throughput of WebLLM as a percentage of native MLC-LLM for two models, tested on an Apple M3 Max.WebLLM vs Native MLC-LLM (Apple M3 Max) 0%50%100% 71.2%Llama-3.1-8B 79.6%Phi-3.5-mini (3.8B) Decoding throughput retained, WebLLM v0.2.75 vs MLC-LLM
WebLLM retains a large share of native decoding throughput on the same hardware (data from Ruan et al., 2026).
In testing on an Apple M3, WebLLM held onto roughly 80% of the decoding throughput of the equivalent native MLC-LLM deployment, which is a smaller gap than I expected to see (Ruan et al., 2026).
Transformers.js, built on ONNX Runtime Web, mirrors Hugging Face's Python library closely enough that the jump across is almost painless if you've used the original (Singh and Ravi, 2025). It supports WebGPU with WebNN and Wasm as fallbacks, and is really aimed more broadly at NLP, vision and multimodal work than LLMs specifically.
ONNX Runtime Web is the most flexible of the three, covering WebGPU, WebGL, WebNN and WebAssembly, but that flexibility comes with more manual wiring and a steeper setup than the other two (Singh and Ravi, 2025).
Older names like TensorFlow.js still have their place for smaller BERT-style models, but weren't designed with LLM-scale workloads in mind, and MediaPipe LLM Inference remains genuinely experimental for now (Singh and Ravi, 2025).
The Bit Nobody Talks About: One Kernel Doesn't Fit All Devices
Picking a framework only solves half the problem, and this is the part I found most interesting. Edge devices are a mess of different hardware, and a kernel tuned for one GPU can run several times slower on another. Yet most in-browser frameworks ship a single "one-for-all" kernel regardless of what it ends up running on (Jia et al., 2024). Testing by the nnJIT team found TensorFlow.js kernels running anywhere from 2% to 246% slower than device-customised versions depending on the hardware underneath (Jia et al., 2024). That's a huge range for something a user never sees or controls.
The obvious answer, generate a bespoke kernel for every device, turns out to be a dead end in practice. Jia et al. (2024) found that tuning a single matrix multiplication kernel with a conventional compiler took nearly two hours and 437 rounds to hit its best performance, and a real model needs several tens of kernels. Their fix, a system called nnJIT, cuts the compilation cost per candidate from minutes down to milliseconds by skipping redundant compiler passes, and shrinks the search space from millions of possible configurations down to a few dozen using web-specific heuristics about memory and thread usage.
Bar chart comparing average kernel speedup of nnJIT against TF.js, ORT-Web and pre-tuned AutoTVM, split by CPU (Wasm) and GPU (WebGPU).nnJIT Average Kernel Speedup vs Baselines 0x2x4x 4.59xCPU (Wasm) 2.77xGPU (WebGPU) Average speedup across tested devices and kernels vs TF.js, ORT-Web and pre-tuned AutoTVM
nnJIT's just-in-time kernel generation delivers a substantial average speedup on both CPU and GPU backends (data from Jia et al., 2024).
Tested across ARM, Intel, AMD and Nvidia hardware, it reached up to 8.2 times faster model inference within 30 seconds compared to existing baselines (Jia et al., 2024). It even beat WebLLM by 39.4% running Llama 2 7B at the time of testing, which tells you there's still plenty of headroom left in kernel-level optimisation (Jia et al., 2024).
What I'd Actually Tell a Developer
If you're weighing up whether to build an in-browser AI feature, here's what I'd pass on from all this reading:
Build for Chromium first. It has the strongest current support for both WebGPU and WebNN (Singh and Ravi, 2025).
You may need to flip a switch. In Chromium browsers, enable the "Unsafe WebGPU" flag at
chrome://flags. In Firefox, turn ondom.webgpu.enabledandgfx.webgpu.force-enabledinabout:config(Singh and Ravi, 2025).Pick the framework to fit the job, not the other way around. WebLLM gets you to a working chatbot-style feature fastest. Transformers.js suits broader NLP or vision work. ONNX Runtime Web suits teams who need maximum backend flexibility and don't mind doing more of the integration themselves (Singh and Ravi, 2025).
Mind the memory ceiling on phones. Larger models can blow straight through a mobile browser's available memory, a real constraint that laptops mostly don't share (Jia et al., 2024).
Keep an eye on kernel-tuning research. Just-in-time approaches like nnJIT aren't in mainstream frameworks yet, but they show where in-browser performance is heading next, and it's worth knowing before you assume today's numbers are the ceiling.
My Take
None of this replaces server-hosted frontier models for the heaviest workloads, and I'm not about to rip out my Claude API calls. But for the growing range of small, capable open-weight models, running entirely client-side isn't a research curiosity any more. It's a genuinely usable option today, already retaining a large share of native performance (Ruan et al., 2026), with active research pushing kernel efficiency further still (Jia et al., 2024). For anyone building AI-powered web tools, the browser itself is becoming a legitimate inference target in its own right, not just a client dialling out to somebody else's API. I suspect I'll be experimenting with this properly before the year is out.
References
Jia, F., Jiang, S., Cao, T., Cui, W., Xia, T., Cao, X., Li, Y., Wang, Q., Zhang, D., Ren, J., Liu, Y., Qiu, L. and Yang, M. (2024) 'Empowering in-browser deep learning inference on edge devices with just-in-time kernel optimizations', in Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services (MOBISYS '24), Minato-ku, Tokyo, 3–7 June. New York: ACM, pp. 438–450.
Ruan, C.F., Qin, Y., Parthasarathy, A.R., Zhou, X., Lai, R., Jin, H., Dong, Y., Hou, B., Yu, M-S., Zhai, Y., Agarwal, S., Cao, H., Feng, S. and Chen, T. (2026) 'WebLLM: a high-performance in-browser LLM inference engine', arXiv preprint, arXiv:2412.15803v2.
Singh, A. and Ravi, R. (2025) The web developer's guide to in-browser LLMs. Intel Corporation. Available at: Intel Developer Zone (Accessed: 9 July 2026).