How we measure browser inference

Measurement scope

Inference runs locally in a dedicated Web Worker using WebLLM and WebGPU. Model files come from external hosts; prompts and outputs are not sent to a model API. Each report identifies the artifact, quantization, runtime, browser, environment, workload and cache observations.

Browser deployment also depends on memory use, page responsiveness, load time and resources consumed while the model is idle.

Run protocol

The published comparison uses the session below. Local runs can use other schedules and settings.

Only one request runs at a time. The next starts as soon as the previous request finishes; no deliberate delay is inserted inside a window. Slower models take longer to complete the same count. The model remains loaded between windows: these are three periods of use, not three independent experiments or browser states.

The current batch repeats one short autumn prompt 60 times. Temperature 0, top-p 1, seed 42, 4,096-token context, up to 64 output tokens per measured request and 16 for warmup. Chat resets before every request. Qwen3 thinking is configured off. Actual text, token counts and finish reasons remain in the report; early stopping and truncation are not normalized away.

Performance definitions and formulas

For request i, startᵢ is measured after chat reset, immediately before the streaming completion call; firstᵢ is the first non-empty text chunk; endᵢ is stream exhaustion. Durations below use seconds unless labeled otherwise.

First-output latency

Lᵢ = firstᵢ − startᵢ
Headline: median of valid measured Lᵢ, displayed in ms. A chunk may contain multiple tokens. This measures worker output availability, before screen presentation.

End-to-end response

Tᵢ = endᵢ − startᵢ
Headline: median of valid measured Tᵢ. Includes input processing and generation, excludes model loading and the preceding chat reset. Reported for the actual generated response length.

Output speed

D = median(runtime decode_tokens_per_sᵢ)
WebLLM supplies its internal decode throughput. It is not recomputed as output tokens ÷ total response time. Native tokenizer counts differ across models.

Loading

load = engine ready − initialization start
Includes required network/cache reads and runtime initialization. The first warmup duration is shown separately and includes generation, not just compilation.

Sequential throughput
(derived definition)

requests/s = completed requests ÷ Σ request-window seconds
output tokens/s = Σ output tokens ÷ Σ request-window seconds
Includes dispatch/reset overhead inside request windows; excludes loading, warmup and the 90-second idle gaps. This derived batch metric is distinct from runtime decode speed.

Missing values stay missing. Summary medians exclude warmup and invalid requests; failed runs and context sweeps do not enter aggregate performance ordering. For an even number of values, the median is the mean of the two middle sorted values. Sixty repeats of one prompt describe timing variability, not sixty distinct capability tests.

Resource scope and calculations

Resource measurements identify their process scope, time window and sample coverage. Browser-only reports expose less than reports paired with the native macOS collector. A missing native signal is unavailable, never zero.

Browser memory

footprint(t) = Σ accounted process footprint(t)
sampled peak = max footprint(t) within the selected window
Dedicated Chrome process tree, including helpers. This sum is not deduplicated physical RAM. Main-page JS heap, worker WASM memory and tracked GPU buffers are separate views of memory; do not add them to the footprint or call buffer allocations physical VRAM. MB = bytes ÷ 1,000,000. Sampling may miss brief peaks.

CPU capacity

CPU% = 100 × ΔCPU seconds ÷ (Δwall seconds × logical CPUs)
equivalent logical CPUs = CPU% × logical CPUs ÷ 100
ΔCPU is summed CPU time across the dedicated Chrome process tree. Equivalent logical CPUs describe average capacity used, not reserved cores. Incomplete process intervals are excluded.

Chrome GPU time
(experimental)

GPU ms/s = counter delta in ms ÷ interval seconds
Attributed Chrome GPU-client time, checked against Metal System Trace in a diagnostic capture. It is not shader occupancy, GPU cores used or a percentage of physical capacity; concurrent accounting can exceed 1,000 ms/s. Counter discontinuities remain gaps. Metal validation applies to the diagnostic capture, not every report.

Host activity & power

Driver GPU activity and CPU/GPU/ANE power are whole-machine signals, including other applications. They are neither model-only GPU utilization nor device wall-plug power.

Page responsiveness

timer delay = max(0, actual interval − scheduled interval)
Main-page scheduling delay indicates contention. It is not CPU utilization or a complete interaction-latency benchmark.

Planned browser suitability score

No suitability score is published yet. Answer accuracy has not been evaluated with a task suite. Basic output validity requires non-empty text, positive token counts, no Unicode replacement characters and no interruption. It does not establish correctness.

A versioned profile must define the device class, task suite, answer checks, observation windows and limits before evaluation. Its budget gate may require Lᵢ ≤ Bfirst, Tᵢ ≤ Bresponse, Mpeak,ᵢ ≤ Bmemory and Uᵢ ≤ Bdelay. Here, Mpeak,ᵢ is sampled peak browser footprint and Uᵢ is main-page timer delay over the task’s defined window. Each B is the corresponding limit in matching units. Thresholds remain unset until a profile is defined.

Model failures and timeouts count as unsuccessful scheduled tasks. Interrupted evaluations or missing required telemetry make the score incomplete, not a pass or a zero measurement. Task quality can also be reported separately as Accuracy = 100 × Σ correctᵢ ÷ N. Neither formula can be calculated from the existing repeated-prompt sessions.

Performance, memory and efficiency will remain visible alongside any future score. Experimental GPU counters and whole-host power are not model-only resource budgets. Cold and cached startup requirements need separate, controlled tests.

Comparison rules

Compare matching protocols, hardware, browser/runtime versions, context, cache, precision, thinking mode and visibility conditions. The leaderboard groups reports by recorded environment, settings and protocol, then shows one result per model package. Token rates use each model’s tokenizer, and actual output lengths differ. One session cannot establish a population failure rate or variability across independent runs.

Published JSON reports load automatically from the repository. Imported reports stay private in the user’s browser and do not change the public leaderboard. Charts use recorded run boundaries and aligned clocks; counter intervals crossing a boundary are excluded, while energy integration uses clipped overlap. Missing coverage, clock uncertainty and other host activity remain limitations.