Tesseract in the Zip: What "100% Local OCR" Costs in an MV3 Extension

AnnMax · November 5, 202611 min read

The Promise, and What It Forbids

This is the story behind VKT SnapOCR, a screenshot-to-text extension I ship for Chrome and Edge. The store line is "100% local OCR — screenshots never leave your machine". This post is about what that sentence forces you to write in the manifest, the build script, and the crop math.

"No upload" is mostly a sentence about what does not happen, which is why it is so easy to say and so easy to half-implement. Concretely, three things disappear from the design: there is no server to build or pay for, no API key to protect, and no queue where a user's screenshot sits in somebody else's storage. What replaces them is a pile of constraints that all land in four files.

The most legible one is the permission list, because it is what a reviewer and a curious user read first:

// manifest.json
"permissions": ["activeTab", "contextMenus", "storage", "unlimitedStorage", "scripting", "clipboardWrite"],
"host_permissions": ["https://api.annmax1983.com/*"],
"content_security_policy": { "extension_pages": "script-src 'self'; object-src 'self'" },
"offline_enabled": true

One host, and it is the licence endpoint — not an OCR endpoint. activeTab plus scripting instead of tabs plus <all_urls>, because the extension only ever needs the tab you are looking at, at the moment you asked for it. offline_enabled: true is a declaration, not a feature: it is the promise written down, and it is only honest if the engine is already inside the package.

What Actually Ships

So the engine has to be bytes in the zip. SnapOCR's build script does not bundle with a linker; it copies from node_modules into publish/ through two explicit lists:

// scripts/build.js
const COPY_LIST = ['manifest.json', 'background.js', 'content.js', 'content.css',
  'license.js', 'quota.js', 'popup', 'result', '_locales', 'icons', 'assets'];

const VENDOR = {
  'node_modules/tesseract.js/dist/tesseract.min.js':  'vendor/tesseract/tesseract.min.js',
  'node_modules/tesseract.js/dist/worker.min.js':     'vendor/tesseract/worker.min.js',
  'node_modules/tesseract.js-core/tesseract-core-simd-lstm.wasm.js': 'vendor/tesseract/core/...',
  'node_modules/tesseract.js-core/tesseract-core-simd-lstm.wasm':    'vendor/tesseract/core/...',
  'node_modules/tesseract.js-core/tesseract-core-lstm.wasm.js':      'vendor/tesseract/core/...',
  'node_modules/tesseract.js-core/tesseract-core-lstm.wasm':         'vendor/tesseract/core/...',
  'node_modules/@tesseract.js-data/eng/4.0.0_best_int/eng.traineddata.gz':     'vendor/tessdata/eng.traineddata.gz',
  'node_modules/@tesseract.js-data/chi_sim/4.0.0_best_int/chi_sim.traineddata.gz': 'vendor/tessdata/chi_sim.traineddata.gz'
};

Four things worth noticing in that map. It is pinned (tesseract.js: 5.1.1, @tesseract.js-data/eng: 1.0.0), so the published extension contains a specific engine, not "whatever npm gave me today". Both a SIMD and a non-SIMD core ship, because the SIMD build is faster and cannot load at all on a CPU without SIMD support — choosing between them is a runtime decision inside the library, and packaging only the fast one silently breaks the slow machines. The language data is shipped compressed (.traineddata.gz), because two packs are the difference between a reasonable download and a silly one. And the two packs that ship are English and Simplified Chinese — the rest are a different story, below.

COPY_LIST is a whitelist rather than "copy the folder" for a reason that has nothing to do with size: the package is reviewed by humans. Whatever is in the zip is auditable; a build that sweeps node_modules into the payload makes every permission question harder to answer. The four runtime scripts are minified with terser, and the manifest, locales, icons and vendor bytes are deliberately not touched.

CSP, and the Question I Had to Test

The CSP line above has no unsafe-eval and no remote script source, which matches how the engine is used: Tesseract.recognize() is handed three paths that all resolve inside the extension origin.

// result/result.js
function runOCR(image, lang) {
  return Tesseract.recognize(image, lang, {
    workerPath: chrome.runtime.getURL('vendor/tesseract/worker.min.js'),
    corePath: chrome.runtime.getURL('vendor/tesseract/core'),
    langPath: langPathFor(lang),
    logger: (m) => { if (m.status === 'recognizing text') setProgress(m.progress); }
  }).then(res => (res.data.text || '').trim());
}

Nothing is fetched from a CDN to run the engine, and no string of JavaScript is ever evaluated — the "code" is a wasm module loaded from the package. That is the whole argument for the strict policy, and it is also the reason the strict policy is affordable.

One caveat I want to record honestly, because it is the kind of thing you only find on a real install: MV3's default extension-page policy already includes 'wasm-unsafe-eval' inside script-src. Writing your own script-src replaces that default instead of adding to it, and a wasm engine is exactly the code that notices. So the CSP block is not a "nice to have" line to tweak and re-zip: after any change to it, load the unpacked extension and run one recognition before you call it done. If the core fails to compile, the fix is script-src 'self' 'wasm-unsafe-eval' — which is still nowhere near the flexibility people assume they need.

CSS Pixels Are Not the Image

The capture path is where "it works on my laptop" goes to die. The user drags a box on a page; the page lives in CSS pixels; the screenshot does not.

// content.js — mouseup
if (rect.width < 8 || rect.height < 8) { cleanup(); return; }
chrome.runtime.sendMessage({ type: 'SNAPOCR_SELECTION', rect, dpr: window.devicePixelRatio || 1 });

// background.js
const rect = msg.rect;
if (!rect || rect.width < 4 || rect.height < 4) return;
const dataUrl = await chrome.tabs.captureVisibleTab(windowId, { format: 'png' });
const cropped = await cropDataUrl(dataUrl, rect, msg.dpr || 1);

// background.js — cropDataUrl(dataUrl, rect, dpr)
const scale = dpr || 1;
const sx = Math.max(0, Math.round(rect.x * scale));
const sy = Math.max(0, Math.round(rect.y * scale));
const sw = Math.min(bitmap.width - sx, Math.round(rect.width * scale));
const sh = Math.min(bitmap.height - sy, Math.round(rect.height * scale));
if (sw <= 0 || sh <= 0) throw new Error('empty crop');
const canvas = new OffscreenCanvas(sw, sh);
canvas.getContext('2d').drawImage(bitmap, sx, sy, sw, sh, 0, 0, sw, sh);

Three details here, each of which came from a failure rather than from a document. First, the two thresholds are different numbers on purpose: 8 CSS pixels in the page kills a stray click before it becomes a request, and 4 in the worker is a backstop behind a message that could have come from anywhere. Second, the multiply-then-clamp order matters: Math.min(bitmap.width - sx, ...) is what stops a rect that ends below the fold from producing a negative height on a 150% display, where one CSS pixel is 1.5 device pixels and rounding drift accumulates across the whole box. Third — the reason the crop multiplies at all — captureVisibleTab returns the viewport at device resolution, so cropping in CSS pixels throws away exactly the pixels OCR needed. On a HiDPI screen a 1× crop of small text is the difference between readable output and mush.

The overlay also freezes scrolling while you drag:

// content.js — begin()
// Freeze the page so captureVisibleTab matches what the user sees
savedOverflow = document.documentElement.style.overflow;
document.documentElement.style.overflow = 'hidden';

That comment is the bug report. captureVisibleTab captures the viewport, so if the page can scroll between the mouse-up and the capture — and it can, because the handoff is asynchronous — you crop a rectangle out of a different image than the one the user selected. Scrolling is not inconvenient there; it is incorrect.

The last piece of the crop is a size limit you hit once and remember. The PNG has to come back out as a data URL, and the naive way to do that explodes:

// background.js — blobToDataURL(blob)
let bin = '';
const chunk = 0x8000;
for (let i = 0; i < bytes.length; i += chunk) {
  bin += String.fromCharCode.apply(null, bytes.subarray(i, i + chunk));
}

String.fromCharCode.apply(null, bytes) with a few megabytes of a screenshot is a RangeError — the argument list overflows. 32 KB per call is a number that works, is comfortably below the limit, and is not something I plan to revisit.

The Image Travels Through Storage, Not the URL

The cropped image has to reach the result page. The obvious route — put it in the query string — is a bad idea twice over: a multi-megabyte base64 string in a URL is a length-limit problem, and it is also the sort of thing that ends up in history entries and crash reports. So the handoff is a single storage key, and the receiving end deletes it as it reads:

// background.js
await chrome.storage.local.set({ [PENDING_KEY]: { image: cropped, lang, ts: Date.now() } });
try { await chrome.tabs.sendMessage(sender.tab.id, { type: 'SNAPOCR_DISMISS' }); } catch (e) { /* tab may be gone */ }
await chrome.windows.create({ url: chrome.runtime.getURL('result/result.html'), type: 'popup', width: 580, height: 660 });

// result/result.js — init()
const stored = await chrome.storage.local.get(PENDING_KEY);
await chrome.storage.local.remove(PENDING_KEY);
const pending = stored[PENDING_KEY];
if (!pending || !pending.image) { hideProgress(); showI18nError('errNoCapture'); return; }

Get-then-remove gives the page a one-shot handoff: reload the result window and it says "no capture" instead of re-running — and re-billing — the previous image. The DISMISS message before windows.create is another small ordering thing: the overlay lives in the page, and if the popup steals focus first, a semi-transparent full-page overlay stays on top of a website you did not intend to freeze.

A Quota That Bills Finished Runs, Not Failed Ones

The free tier is 20 recognitions a day, and where the counter lives is a product decision disguised as an implementation detail:

// quota.js
 * The counter lives in chrome.storage.local on purpose. A server-side counter
 * would lock people out whenever the network is unavailable, which contradicts
 * the "works fully offline" promise behind the bundled languages.
const FREE_DAILY_QUOTA = 20;

/** Local calendar day (not UTC) so the reset feels like "tomorrow" to the user. */
function _today() { /* YYYY-MM-DD from new Date(), padded */ }

/** Called after OCR succeeds, never before — a failed capture must not eat a Free user's quota. */
async function consumeQuota() { ... }

The ordering is enforced at the call site, not inside the quota module:

// result/result.js
const text = await runOCR(pending.image, lang);
if (!isPro) await consumeQuota(); // only bill a successful run
showResult(text, lang);

which means an engine crash or an unreadable crop costs the user nothing. That is the kind of fairness that is invisible when you do it and loudly resented when you don't — and it is one line and a comment. Read that line literally, though, and one edge is left open on purpose: runOCR resolving with nothing readable is still a resolve, so an empty result spends the run. The comment in the module says "a failed capture must not eat a Free user's quota", and an empty crop is not a failed capture — the engine ran, found nothing, and reported. Whether the text was any good is not something this call site can decide, so it bills the completion. If that ever reads as unfair to real users, the fix is one clause at the call site, not a change of heart in the quota module.

The local-day key is the same argument in a smaller register: _today() builds YYYY-MM-DD from the machine's own clock, so the counter rolls over at the user's midnight and the reset genuinely feels like "tomorrow" — a UTC key would flip it at 09:00 for someone in Tokyo, which is a time that suits my logs and nobody else's. And yes, a counter in chrome.storage.local is editable by anyone who opens DevTools on the extension page. I decided that trade deliberately: the alternative is a server that has to be reachable before a person can read text off their own screen, which is worse than being occasionally cheated by a tool that costs $14.99 once. The gate checks a cached licence first and the counter second, and the zero-quota state replaces the whole footer — copy button, export, the second upsell — so the wall page has exactly one call to action instead of three.

The One Network Path That Is Not Your Licence

Two languages ship in the zip. The other seven in the picker do not:

// result/result.js
const BUNDLED_LANGS = ['eng', 'chi_sim'];
const CDN_LANG_BASE = 'https://tessdata.projectnaptha.com/4.0.0';

function langPathFor(lang) {
  return BUNDLED_LANGS.includes(lang)
    ? chrome.runtime.getURL('vendor/tessdata')
    : CDN_LANG_BASE;
}

Two bundled packs are already a large extension; nine would be a different product, so the remaining languages are fetched from the public tessdata mirror once and cached by the engine afterwards. That is the only place in the code where a third-party URL appears outside the licence host, and I keep it visible on purpose: the honesty of "local OCR" rests on the fact that this request carries a language name, not an image. The recognition itself never leaves the machine.

Two consequences you have to design around. First, a non-bundled language is not offline-usable until it has been downloaded once online — which is why the result page's language badge marks bundled packs explicitly, and why the "works offline" claim in the store copy is scoped to English and Simplified Chinese. Second, a remote path is a dependency. The mirror publishes specific version segments, and the segment in the first build was copied from the npm package layout rather than from the mirror — the mirror does not publish that segment at all, so all seven downloadable languages returned 404 and the paid tier was quietly unusable, with no error a user could act on. I found it by curling the paths by hand, which is the point: if you ship something like this, put a HEAD request for every language path in CI. It is the cheapest test in the project and it catches the exact class of failure that makes a paid feature quietly unusable.

Cross-origin fetch from an extension page normally wants a host permission, but a policy you can test in one curl: the mirror answers Access-Control-Allow-Origin: *, so the download needs no extra grant in host_permissions. Worth knowing, because "add the host to the manifest" is the reflex answer that makes the permission list grow for no reason.

Injection, and the Places This Cannot Work

The overlay is not declared in the manifest as a content script. It is injected on demand, guarded twice:

// background.js — startSelection(tab)
const url = tab.url || '';
if (!/^(https?|file):/i.test(url)) return;   // chrome://, edge://, Web Store…

const [{ result }] = await chrome.scripting.executeScript({
  target: { tabId: tab.id }, func: () => !!window.__snapocrLoaded });
if (!injected) {
  await chrome.scripting.insertCSS({ target: { tabId: tab.id }, files: ['content.css'] });
  await chrome.scripting.executeScript({ target: { tabId: tab.id }, files: ['content.js'] });
}
await chrome.tabs.sendMessage(tab.id, { type: 'SNAPOCR_BEGIN' });

The probe reads a flag the IIFE sets for itself (if (window.__snapocrLoaded) return;), which is the difference between "inject once" and "two overlays, two listeners, one Escape key that closes neither". Dynamic injection also means a session where you never start a capture never runs the script — no permanent content script sitting on every page, which is both the permission story (activeTab) and the performance story.

And the list of places it genuinely does not work, which belongs in the product page rather than in a support ticket: chrome:// and Web Store pages are off-limits to injection, so a settings screen cannot be read; only the visible tab is captured, so text scrolled out of the viewport is out of reach; and one drag is one region, so a whole table means several boxes. A tool that says what it cannot do gets fewer angry reviews than one that implies it can do everything.

Checklist

If the promise is "no upload", write it where reviewers look: one host permission, no <all_urls>, offline_enabled: true, and a build that proves the engine is packaged.
Ship the fallback wasm core, not only the SIMD one — and pin the library version, so the zip contains a known engine.
Copy sources with a whitelist. Every byte in the package is a byte someone may ask you about.
Read MV3 CSP as a replacement, not an addition, and re-test the wasm path after touching it.
Capture is in device pixels, selection is in CSS pixels: multiply by devicePixelRatio, then clamp to the bitmap, then refuse empty crops.
Freeze page scroll during selection, or the crop rectangle lands on a different image than the one the user saw.
Never build a data URL with one giant String.fromCharCode.apply. Chunk it.
Hand big payloads through chrome.storage.local and delete on read; a reload should fail cleanly, not re-run the world.
Bill a quota after success, reset on the user's calendar, and pick the offline-first storage location for the counter on purpose — then write the reason down.
Mark which languages are packaged and which are fetched; treat every remote path as a dependency with a test.
Publish the list of things the extension cannot do. It is shorter than the list of tickets it prevents.