How OCR actually works: from image to text

The real engine behind the tool: Tesseract, via WebAssembly

OCR (Optical Character Recognition) is the process of turning the pixels of an image, a photo of a document, a screenshot, a scanned receipt, into actual text: editable, searchable, copyable. The name hides the fact that there is no single way to do this. Each OCR engine is a specific piece of software, with its own pipeline and its own history, and that difference is why the same photo can come out perfect in one app and unreadable in another.

This site’s Image to Text tool uses tesseract.js, a WebAssembly package of Tesseract, the original open-source OCR engine, not a reimplementation sharing the name. Per the project’s own documentation, tesseract.js "aims to bring the Tesseract OCR engine to the browser and Node.js" by "wrapping a WebAssembly port of Tesseract". In this tool’s own code that is directly checkable: the engine is dynamically imported (`await import('tesseract.js')`) only the moment you click extract, which keeps the WASM core out of the page’s initial load, and there is no server route for this tool at all. Recognition runs to completion inside your own browser tab, meaning the image and the extracted text never leave your device, a concrete, checkable difference, not a generic privacy promise, the same instinct worth applying to what a photo reveals on its own before you share it, the subject of the EXIF metadata guide.

OCR

Optical Character Recognition: turning an image’s pixels into real, editable, searchable text.

Tesseract

The original open-source OCR engine, maintained on the tesseract-ocr GitHub project, currently on major version 5.

tesseract.js

The package that compiles Tesseract to WebAssembly and exposes it as a JavaScript API, used by this tool.

PSM (Page Segmentation Mode)

The strategy that tells Tesseract whether the image is a paragraph, a single line, or sparse, disconnected text.

LSTM

Long Short-Term Memory, a type of recurrent neural network that processes a sequence (here, a line of text) as a whole.

From HP to open source: Tesseract’s real history

Tesseract did not start as a community project, it started as commercial software inside a hardware maker, decades before becoming the base of a browser tool. The timeline below follows the project’s own official documentation (README and tessdoc, on the tesseract-ocr GitHub).

1. 1985 to 1998Born as commercial HP software

   Developed at Hewlett-Packard Laboratories in Bristol, UK, and at HP in Greeley, Colorado (USA), between 1985 and 1994, with changes to port it to Windows in 1996 and a conversion of the codebase to C++ in 1998.

2. 1995Strong showing in UNLV’s annual test

   In the UNLV Annual Test of OCR Accuracy, an evaluation run by the University of Nevada, Las Vegas, Tesseract scored around 98% character accuracy on one of that era’s test sets, a standout result against the competing commercial engines of the time.

3. 2005HP open-sources the code

   Hewlett-Packard releases Tesseract as an open-source project, after years without active development as a commercial product.

4. 2006 to 2017Google takes over development

   From 2006 until August 2017, Tesseract’s development is led by Google, under Ray Smith, the same engineer behind the original engine.

5. 2018Tesseract 4: the LSTM neural engine

   Tesseract 4.0.0 ships on October 29, 2018, with a new recognition engine based on LSTM neural networks, alongside the previous classic engine.

6. 2021 to todayTesseract 5 and community maintenance

   Tesseract 5.0.0 ships on November 30, 2021 and becomes the main stable version. Since Google’s lead stepped back, the project has continued as open source maintained by a community of contributors, not an internal team at one company.

The real pipeline: from raw image to text

An OCR engine never jumps straight from image to letters. It goes through three phases: preprocessing (cleaning and normalizing the image), segmentation (finding where the blocks and lines of text are before trying to read them), and only then recognition (identifying the characters). The steps below are exactly the ones the Image to Text tool runs, in the order the code itself executes them.

1. Decode respecting EXIF orientationThe image is decoded in a way that respects the camera’s rotation metadata, so a photo taken sideways does not start upside down before it even reaches the rest of the pipeline.
2. Upscale the image when it is smallIf the image’s longer side is under 1,600 px, it is scaled up, by at most 3x, aiming for that 1,600 px edge. Already-large images are never shrunk. Thin strokes of small letters vanish in a small image before binarization even happens; upscaling gives those strokes enough pixels to survive the next stages.
3. Convert to grayscaleColor is discarded using the ITU-R BT.601 luma formula (0.299×red + 0.587×green + 0.114×blue). What matters for finding a letter’s outline is brightness, not color.
4. Invert if the background is darkIf the image’s mean brightness falls below 128 (on a 0 to 255 scale), it is automatically inverted. A dark-mode screenshot, light text on a dark background, becomes dark text on a light background, the pattern the next stages expect.
5. Stretch the contrastThe brightness range between the image’s 1st and 99th percentile is stretched to fill the full 0 to 255 range. A faded receipt, where the darkest text and the lightest paper sit only a few shades apart, gets back a contrast that binarization can actually separate.
6. Segment and recognize, across up to 3 passesIn auto mode, the first pass assumes an ordinary paragraph. If the result comes back empty or under 55% confidence, a second pass binarizes the image with Otsu’s method and turns off the word dictionary (useful for codes and plates, which are not dictionary words). If confidence is still low, a third pass reads it as a single line. The tool keeps whichever pass scored the highest confidence.

A real example with real numbers: an 800×600 px receipt photo has its longer side at 800 px, below the 1,600 px target. The scale factor is the smaller of 3 and 1,600/800, which is 2. The final canvas comes out at 1,600×1,200 px, double the original size, before it is even turned to grayscale.

computeUpscaleFactor(800, 600)
  lado maior = 800 px (abaixo da meta de 1600 px)
  fator = min(3, 1600 / 800) = min(3, 2) = 2
  canvas final = 1600 x 1200 px

A second example, now from the recognition stage: a photo of a license plate is sparse text, no sentences, the worst case for a language dictionary. In auto mode, the first (paragraph) pass likely comes back empty or low-confidence, because "ABC-1234" is not a word. The tool then tries the second pass, binarized and dictionary-off, which usually resolves this kind of content. Recognition here is not a single attempt, it is a search for which strategy works for that specific image. It is also worth remembering that the source file’s format already skews the first stage: a JPEG re-saved several times blurs exactly the thin strokes binarization needs to cleanly separate, the same reason, explained in the image formats for the web guide, that a QR code turns unreadable in that format. Prefer PNG, or the camera’s original file, when the end goal is text.

Classic vs. neural: Tesseract 4’s shift

Through Tesseract 3, and in most OCR engines through the early 2010s, recognition worked character by character: the software isolated each shape as a separate "blob" and matched that shape against learned patterns. Tesseract 4.0.0, released in 2018, changed that by adding a second engine, based on an LSTM neural network. Per the project’s own documentation, this new subsystem "is integrated into Tesseract as a line recognizer", meaning it reads a whole line as a sequence, not one isolated character at a time.

That explains a practical detail of this tool: the "paragraph" and "line" content modes use the engine’s default pass (which today is the LSTM one), while "sparse" mode adds binarization and turns the dictionary off, a combination aimed at text that looks the least like ordinary prose, codes, plates and loose symbols.

The real failure modes, and which stage each one breaks

An OCR engine does not fail randomly. Every common failure type maps to a specific stage of the pipeline described above, breaking in a specific way. Knowing which stage breaks is what makes a bad result fixable, instead of just frustrating.

A skewed or rotated photo breaks segmentation. The layout analysis that groups shapes into lines expects a roughly horizontal baseline. At a few degrees of tilt, the line recognizer usually copes, because it reads along a fitted baseline. As the tilt grows, though, the segmenter starts drawing boxes that cut through one line’s ascenders and the next line’s descenders, so words come out chopped mid-letter, before any classifier even runs. This tool does not auto-correct skew: it offers manual 90° rotation and a crop editor, so the practical fix for any angle outside that grid is straightening the photo before running OCR, not trusting the engine to compensate on its own, use the image resizer and cropper if you need more control than the built-in editor offers.

Low contrast or uneven lighting breaks binarization. Otsu’s method picks one single brightness cutoff for the whole image. If the light falls unevenly, a bright corner, a dark corner, a shadow crossing half the page, no single value separates ink from paper everywhere at once: parts of the text vanish into the background, other parts pick up a halo of noise. This tool’s contrast stretch helps when the whole image is evenly faint, but does not fix a problem that is the unevenness itself.

Decorative or unusual fonts break the classifier, classic or LSTM alike, because both were trained on the shapes of typical printed typefaces. A blackletter, stylized script or heavily ornamented font has letterforms outside what the model learned to recognize, so confidence drops and substitutions climb, a decorative "S" reading as a "5", for instance.

Busy or textured backgrounds break segmentation and binarization together. Texture behind or around the text creates edges and contrast that the segmentation stage can mistake for characters, or that blend with the real letters, so the boundary between "this is ink" and "this is not" turns blurry, producing noise the recognizer tries, and fails, to read as characters.

Handwriting is a different problem, not just a harder one. Printed OCR, including Tesseract’s LSTM engine, is trained on the comparatively narrow, consistent shape vocabulary of printed fonts: a printed "a" looks nearly the same every time. Handwriting varies enormously between people, and even within one person’s own hand, with no fixed "alphabet" of shapes, and in cursive the strokes connect without cleanly separating into single-character blobs. That is why handwriting recognition is treated as its own field, called HTR (Handwritten Text Recognition), with its own models and training data, rather than something the same printed-text engine reliably handles.