Running an AI Image Upscaler Entirely in the Browser with onnxruntime-web
How ToolRunner runs Real-ESRGAN super-resolution client-side: ONNX session setup, WebGL/WASM fallback, 128px tiling with feathered stitching, no server.
by Dowon Oh
Every tool on ToolRunner follows the same rule: your data never leaves the browser. That is an easy promise to keep for a Base64 encoder. It gets more interesting when the tool is an AI image upscaler that runs a neural super-resolution network on megapixel images — a workload that most sites would ship off to a GPU server behind an upload form. This article is a walkthrough of how the upscaler actually works: which model it runs, how the ONNX session is created and which execution provider it lands on, why the image is cut into 128-pixel tiles before inference, and how the site stays fast for the 95% of visitors who never touch this tool. I am a backend engineer by trade, so parts of this were familiar (tensors, memory pressure) and parts were very much not — the browser is a hostile place to run a compute-bound loop.
Why run super-resolution client-side at all
The obvious architecture for an image upscaler is a server with a GPU: upload a photo, a queue worker runs the model, download the result. That design has three costs I did not want to pay: infrastructure (GPU instances are the most expensive thing you can rent), latency ceremony (upload, queue, poll, download), and — the one that matters most for this site — trust. People upscale family photos, ID scans, screenshots of private dashboards. "We delete your images after processing" is a policy; "your image never leaves your machine" is a property. You can verify it yourself in the network tab: after the model file is cached, an upscale run makes no request that carries your image — the only traffic you will see is the site's analytics and ad scripts doing their page-level business, none of it touching your pixels.
The enabling technology is ONNX Runtime Web, Microsoft's JavaScript build of the ONNX Runtime inference engine. It ships execution providers that run models on WebAssembly (CPU) and WebGL (GPU), so a model exported to ONNX format can run in any modern browser with no plugins and no server. The trade-off is that you inherit the browser's constraints: limited memory, a single main thread that must stay responsive, and wildly heterogeneous hardware. Most of the engineering below is about living inside those constraints.
The model: Real-ESRGAN x4v3 in under 5 MB
The upscaler runs Real-ESRGAN, specifically the compact x4v3 variant, stored as a 4.9 MB ONNX file served from the site's own origin at /models/realesrgan-x4v3.onnx. Real-ESRGAN comes from the paper Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data (Wang et al., ICCVW 2021), and its distinguishing feature is right there in the title: it is trained on synthetically degraded images that model real-world damage — JPEG artifacts, sensor noise, resampling blur — rather than clean bicubic downscales. That makes it noticeably better on the images people actually feed an upscaler: old photos, compressed screenshots, images that have been through three messaging apps.
The x4v3 build matters for the browser use case because it is the compact edition — a small VGG-style network designed for practical deployment rather than the full RRDB architecture of the flagship weights. At 4.9 MB it is small enough to download on demand (more on that later) and light enough to run tile-by-tile on WebGL or CPU WebAssembly without exhausting browser memory. The model's native scale factor is fixed at 4x; the tool's 2x mode is derived from it, which turns out to be a feature rather than a compromise. (The models directory also carries a tiny ESPCN x2 network from an earlier iteration, but the current pipeline runs Real-ESRGAN for both scale factors.)
Creating the session: WebGL first, WASM as the fallback
Session setup is deliberately boring: one cached InferenceSession, execution providers tried in order of expected speed.
import * as ort from 'onnxruntime-web';
const MODEL_PATH = '/models/realesrgan-x4v3.onnx';
let session: ort.InferenceSession | null = null;
export async function loadModel(): Promise<void> {
if (session) return;
ort.env.wasm.numThreads = 1;
try {
session = await ort.InferenceSession.create(MODEL_PATH, {
executionProviders: ['webgl'],
});
} catch {
session = await ort.InferenceSession.create(MODEL_PATH, {
executionProviders: ['wasm'],
});
}
}
Two decisions here deserve explanation.
WebGL, not WebGPU. ONNX Runtime Web also offers a WebGPU execution provider, and on paper it is the future. In practice, WebGL is the provider that exists everywhere today, while WebGPU support still varies by browser, OS, and driver. For a public tool where "it silently fails on your laptop" is the worst possible outcome, the pipeline tries WebGL and falls back to plain WebAssembly if session creation throws for any reason — an unsupported operator in the graph, a context-creation failure, a headless environment. The WASM path is slower, but it runs anywhere JavaScript runs.
Single-threaded WASM. The numThreads = 1 line pins the WebAssembly backend to one thread. Multi-threaded WASM requires SharedArrayBuffer, which browsers only enable when the page is cross-origin isolated via COOP/COEP response headers — headers that constrain how every other resource on the site loads. (If that sentence sounds like the kind of thing that breaks unrelated pages in production, it is; it is a close cousin of the deployment traps I wrote about in the CSP article.) Turning the whole site cross-origin isolated to speed up one tool's fallback path was a bad trade, so the fallback stays single-threaded and the tiling design absorbs the cost.
Tiling: 128-pixel squares with a 16-pixel overlap
You cannot feed a 4000×3000 photo into the network in one shot. A super-resolution model's memory footprint scales with input area across dozens of intermediate feature maps, and both the WebGL provider (texture size limits) and the WASM provider (heap limits) will fall over long before that. So the upscaler never runs the model on the full image: it slices the input into 128×128 tiles with a 16-pixel overlap between neighbours, runs inference on each tile independently, and stitches the 512×512 outputs back together.
The overlap is the part that makes or breaks output quality. Convolutional networks behave differently at the edge of their input — there is simply less context there — so if you tile with no overlap, every seam is visible in the result as a faint grid. With overlap, every seam region is predicted twice, once by each neighbouring tile, and the stitcher blends the two predictions with a feathering weight that ramps linearly across the overlap zone: a pixel deep inside a tile gets full weight, a pixel near the tile edge contributes almost nothing, and the accumulated result is divided by the total weight at each output pixel. The 16-pixel overlap becomes a 64-pixel blend zone at 4x scale, which is wide enough that I have not been able to find a seam by eye even on flat gradient backgrounds.
Tiling also solves a problem the browser adds and servers do not have: keeping the page alive. Inference is compute-bound, and a long synchronous stretch on the main thread freezes the tab — no progress bar, no cancel button, eventually the "page unresponsive" dialog. Because the work is already chopped into per-tile units, the loop yields back to the event loop between tiles. That single yield per tile is what makes the progress ring animate, keeps the cancel button clickable (cancellation runs through an AbortSignal checked at every tile boundary), and stops the browser from declaring the page dead. Tiling started as a memory constraint and ended up being the tool's entire concurrency model.
Tensors in, pixels out
The bridge between the browser's pixel world and the model's tensor world is the Canvas API. The input image is drawn to an off-screen canvas and read back with getImageData, which — per the MDN ImageData reference — hands you a flat Uint8ClampedArray of RGBA bytes in row-major order. The model wants something different on every axis: float32 instead of bytes, values normalized to [0, 1] instead of [0, 255], three channels instead of four (alpha is dropped), and NCHW layout — all the red values, then all the green, then all the blue — instead of interleaved RGBA. So preprocessing each tile is a nested loop that walks the tile's pixels, divides each channel by 255, and writes into the planar layout:
const inputTensor = new ort.Tensor('float32', tileRGB, [1, 3, tileH, tileW]);
const results = await session.run({ [session.inputNames[0]]: inputTensor });
const output = results[session.outputNames[0]].data as Float32Array;
The shape [1, 3, tileH, tileW] is batch size 1, three channels, then spatial dimensions — and note that the tile is fed at its native size with no padding or letterboxing, because the network is fully convolutional and accepts arbitrary spatial dimensions. The output tensor comes back in the same NCHW layout at 4x the spatial size.
Postprocessing runs the whole dance in reverse. After stitching, the three float channels are interleaved back into RGBA order, multiplied by 255, clamped to the valid byte range (the network can and does overshoot slightly — a value of 1.02 must become 255, not wrap around), and the alpha channel is set fully opaque. The result becomes an ImageData, goes onto a canvas with putImageData, and from there canvas.toBlob produces the downloadable file in PNG, JPEG, or WebP. Every step is a plain browser API; the only exotic dependency in the entire pipeline is the runtime itself.
One honest limitation falls out of this design: because alpha is dropped at the tensor boundary, transparency does not survive the trip. The tool also enforces input guardrails before any of this starts — 10 MB file size cap and a 4096-pixel limit per dimension — because a 4096×4096 input already means over a thousand tile inferences, and past that point "the browser can technically do it" stops being the same as "the browser should."
Getting 2x out of a 4x model
The tool offers 2x and 4x modes, but the network only knows how to do 4x. The 2x path runs the exact same pipeline — full 4x inference on every tile — and then downscales the stitched result to 2x using a canvas draw with imageSmoothingQuality: 'high'.
Running 4x inference to produce a 2x image sounds wasteful, and computationally it is: the 2x mode costs exactly as much as the 4x mode. But the alternative — shipping a second trained network for 2x — costs a second model download, a second code path, and a second set of quality quirks to debug. And there is a genuine quality argument for the supersampled route: generating detail at 4x and then downsampling averages away some of the high-frequency hallucination artifacts that GAN-trained super-resolution models are prone to, the same way rendering a 3D scene at double resolution and scaling down produces cleaner edges. The 2x output is, in a real sense, an antialiased 4x.
Keeping the other 30 tools fast
ToolRunner is 30+ tools sharing one bundle, and most visitors arrive for a JSON viewer or a QR code generator, not a neural network. The upscaler must not tax them. The discipline has three layers.
The model downloads on demand — and late. The 4.9 MB ONNX file is a static asset, not part of any JavaScript bundle, and loadModel() is not called when the page mounts. It is called inside the upscale function itself, on the first click of the Start button. Visiting /image-upscaler/ to read about the tool costs you nothing; the model transfer starts only when you actually commit to an upscale, and the cached session makes every subsequent run on the page start instantly. The runtime's own WASM binaries behave the same way — ONNX Runtime Web fetches its .wasm files when a session is first created, not at import time.
The dev server is told to leave onnxruntime-web alone. In vite.config.ts, the package sits in optimizeDeps.exclude. Vite's dev-time dependency optimizer pre-bundles imports with esbuild on startup, and onnxruntime-web is by far the heaviest dependency in the project — letting the optimizer crunch it slows every dev-server cold start for a package with its own loading strategy. Excluding it keeps the feedback loop fast for the 95% of development sessions that never open the upscaler.
Everything is same-origin. Model, runtime, WASM binaries — all served from the site's own domain, no CDN, no third-party fetch at inference time. That is partly a performance choice and partly the same self-contained posture the rest of the site takes, for reasons the HTTP security headers guide goes into: a page that loads nothing from anyone else's origin is a page with a very short list of things that can go wrong.
The result is a tool that would conventionally require a GPU backend, running as a static file on a static host. Qualitatively, small images finish in seconds on WebGL-capable hardware and larger ones take proportionally longer — the progress ring exists because the honest answer to "how long will this take" is "it depends on your machine," which is the deal you accept when the user's machine is the server. I think it is a good deal. The upscaler has no queue, no quota, no upload — and no server bill, which for a free tool is the difference between existing and not.