Benchmarks

Honest, reproducible accuracy benchmarks for pdfspine's text extraction, measured against objective ground truth and against three reference engines — PyMuPDF (fitz), pypdfium2 (Google PDFium), and pdfminer.six. Generated by the harness in conformance/ — regenerate any time; numbers below are a snapshot (text extraction 2026-06-16; pypdfium2 column + rendering added 2026-06-18; multi-column and PMC reading order re-verified 2026-06-20; OCR added 2026-06-19).

Verified state (as of 2026-06-21). Text extraction is now at fitz parity for multi-column too: the born-digital column corpus reads at order 0.996 / jaccard 0.965 vs fitz 0.965 (dead-even on word-set), and the PMC scientific corpus at order 0.965 mean / 0.995 median (fitz 0.975 / 0.997) — see §1. Rendering reaches SSIM 0.945 mean / 0.986 median vs fitz (§5), now that Indexed / Separation / DeviceN colorspaces (+ /Decode) and embedded Type1 (/FontFile, PFB/PFA) charstrings render. OCR recognize() is rayon-parallel (3.49× on a 42-box page; §6). API parity is 88.7% (682/769) — see PARITY.md. Numbers not re-measured here are date-noted as "as of <date>".

Clean-room note. PyMuPDF (AGPL), pypdfium2 (BSD/Apache PDFium), and pdfminer run locally as diff references ONLY; no reference output is ever committed — only similarity scores. pdfspine is Apache-2.0 and shares no code with MuPDF or PDFium.

1. Objective ground-truth benchmark (58 documents)

Each extractor is scored against the same true ground truth (born-digital source text, or the publisher's JATS-XML / official text rendition), not against another extractor. Metrics: lev (edit similarity), f1 (token F1), jaccard (word-set overlap), order (reading-order similarity). Mean over all 58 docs.

pdfspine is within ~1–2% of fitz on every metric (word-set Jaccard is dead-even: 0.834 vs 0.836), edges out pypdfium2 (Google's PDFium C library) on every metric, and beats pdfminer on edit-similarity and reading order. On reading order specifically, pdfspine matches or exceeds fitz on 13/58 documents. The three production-grade engines (pdfspine / fitz / pypdfium2) cluster within ~1–2pp; pdfminer trails on reading order (0.918). All four are scored against the SAME ground truth with the SAME score.py.

Per-corpus breakdown

born-digital, multi-column (n=6) — CSS-column PDFs from public-domain prose; source order is the known ground-truth reading order.

pdfspine is at parity with fitz and pypdfium2 here (jaccard and f1 identical); pdfminer reads these row-major (order 0.78), which pdfspine's column reconstruction avoids.

PMC scientific papers (n=12) — real CC-BY articles; ground truth = publisher JATS-XML body text. (Absolute scores are low for ALL extractors because XML ≠ PDF exactly — running heads/refs/tables differ; the pdfspine-vs-fitz gap is the meaningful signal.)

EUR-Lex, 8 languages (n=40) — real EU legal PDFs (CC-BY) in EN/FR/DE/ES/IT/EL(Greek)/BG(Cyrillic)/PL; ground truth = the official text rendition.

pdfspine handles accented Latin, Greek, and Cyrillic at near-parity with fitz.

2. Real-corpus differential vs fitz (30 documents) — before/after

The original 30-doc corpus (public-domain US federal: IRS forms/pubs, GovInfo bills, CDC MMWR, NASA, USGS, NIST), scored as text similarity vs PyMuPDF. This shows the impact of the 2026-06-16 fixes:

Open rate 30/30 (100%), no panics/aborts/timeouts, qpdf structural check 12/12 on re-saved output.

3. Robustness — never-panic on diverse real-world PDFs

GovDocs1 sample (23 heterogeneous US-government PDFs, producers spanning PDF 1.2–1.6), opened + text-extracted in isolated subprocesses so a panic cannot be hidden:

• Open rate: 23/23 (100%), 0 repaired, 0 failed.
• Robustness: 0 aborts, 0 timeouts — no panics on any input.
• Text similarity vs fitz: Levenshtein mean 0.827 (median 0.959), Jaccard 0.883.

(Scales to the full GovDocs1 / SafeDocs corpora via conformance/gt/fetch_robustness.py.)

Domain breadth (GovInfo, 30 docs)

Public-domain US federal documents across three more domains — court opinions (USCOURTS), GAO audit reports, and the Federal Register (dense multi-column regulatory). 30/30 open, 0 panics. Text similarity vs fitz:

Tables (find_tables vs fitz, 30 docs / 1581 pages)

pdfspine's find_tables is at near-parity with fitz after gating detection on real ruling-line evidence (borderless prose no longer produces spurious tables).

4. What changed (2026-06-16)

Five extraction fixes closed the gap to fitz:

1. Column-major reading order — occupancy-valley gutter detection (was row-major interleaving on multi-column pages).
2. Inter-word space synthesis — recover word spaces on TJ-kerned PDFs that omit space glyphs (LaTeX/scientific typesetting).
3. Device-space gap threshold — scale the word-gap threshold by the rendered glyph size, not the raw Tf operand (fixes word-shatter on PDFs that bake scale into the CTM).
4. Baseline-merged column split — separate columns whose lines share an exact baseline (was character-interleaving a few tight-column lines).
5. find_tables ruling-line gating — detect tables from real vector rulings, not prose whitespace (was over-detecting tables 9× on borderless multi-column text).

5. Rendering (get_pixmap) — at/near-parity (SSIM re-measured 2026-06-21)

Page rendering went from SSIM ~0.58 → 0.984 mean / 0.989 median vs fitz (MAE-sim 0.992) over the same 46-doc sample (corpus-born 6, eurlex 10/40, robustness 10/23, pmc 10/12, fixtures/govinfo 10/30; DPI 150, page 1, seed 1234). This is a fresh aggregate that includes every landed render-fidelity fix — up from the stale 0.945 / 0.986 measured 2026-06-17 (mean +0.039), with the verdict crossing from "CLOSE" to "AT/NEAR PARITY". Per-corpus SSIM: corpus-born 0.995, pmc 0.991, eurlex 0.988, robustness 0.977, fixtures/govinfo 0.974. Full machine report: conformance/gt/RENDER-REPORT.md (regenerate with conformance/gt/render_diff.py; numbers above trace to that report — not hand-edited).

The fixes that closed the gap, in two waves. Wave 1 (the 0.58 → 0.945 jump): full per-glyph Trm into the render path; bare-CFF FontFile3 parsing; CCITT/JBIG2 1-bpc polarity; CID-keyed CFF charset CID→GID. Wave 2 (the 0.945 → 0.984 jump re-measured here) eliminated the long tail of blank / near-blank pages: non-embedded standard-14 body text now falls back to the OFL Liberation fonts; embedded Type1 (/FontFile, PFB/PFA) charstrings rasterize; Indexed / Separation / DeviceN colorspaces + /Decode arrays render (pixel-exact vs fitz on synthetic cases); CMYK black-point and Symbol/ZapfDingbats fallback handled.

The effect is starkest on the prior bottom of the distribution. The old worst page — eurlex 32006L0112_ES, a non-embedded Type1 page that rendered nearly blank — went 0.527 → 0.993 (ink-gap now 0.0, no longer near-blank). The two prior near-blank robustness scans govdocs1-00000 and govdocs1-00019 rose 0.541 → 0.954 and 0.558 → 0.969. No page in the sample now falls below 0.92; the new worst-10 are all flagged "good parity" (residual AA / hinting / sub-pixel differences only): the lowest are the IRS AcroForm pages irs-f8843 (0.922) and irs-fw4 (0.952) and the text-dense scans govdocs1-00000 (0.954) / govdocs1-00012 (0.955). Remaining residuals (pre-existing naive CMYK→RGB on photographic CMYK, fine AcroForm widget AA) are tracked in docs/PRD-NEXT.md.

6. OCR accuracy — PaddleOCR vs Tesseract (= fitz's OCR) (2026-06-19)

pdfspine ships two OCR backends behind one API (page.get_textpage_ocr(engine=...) / doc.pdfocr_*): the default "tesseract" (the system Tesseract CLI — exactly what PyMuPDF/fitz uses, since fitz's OCR is Tesseract-only) and "paddle" (pdfspine's pure-Rust PP-OCRv4 engine, embedded in the wheel, no external binary). This benchmark quantifies the CJK win.

Corpus. 16 deterministic synthetic SCANS (conformance/ocr/), each a mixed Chinese + Latin + digit page rendered with a real CJK font (STHeiti), varying font size (28–44 px), line count (3–6), and scan-like degradation (Gaussian blur ≤1.0 px and per-pixel noise on a subset). Each image is fed through the same pdfspine pipeline (Page.insert_image → image-only PDF → get_textpage_ocr at 150 dpi) for both engines. Ground truth is known per-image and split into a CJK stream and a Latin stream so each script is scored independently.

Metrics. CJK = character accuracy (1 − normalized Levenshtein) over the pure-CJK character streams. Latin = per-token best-match accuracy (for each ground-truth Latin token, the closest token the engine emitted) — deliberately the metric most favorable to Tesseract, so a CJK-blind engine's ASCII garbage for Chinese glyphs is not charged against its Latin score. Tesseract runs with its default eng language — the same default fitz uses — i.e. no Chinese model is loaded.

The win: PaddleOCR scores 1.000 on Chinese vs Tesseract's 0.000. With only the default English model — which is all fitz's OCR offers out of the box — Tesseract cannot read a single Chinese character (it emits ASCII noise like RUSTSCEM, RZEMETASARATTZ), so its CJK accuracy is a flat zero across all 16 scans. PaddleOCR recovers the Chinese perfectly (16/16 images at 1.000) with no external binary and no model download. On Latin the two are close and Tesseract edges ahead (0.988 vs 0.867 — PaddleOCR occasionally mis-segments a Latin token); for a CJK or mixed-script document the PaddleOCR engine is the one that actually works.

Per-image numbers and the raw recognized text live in conformance/ocr/results.json. The corpus is regenerable and the scoring is deterministic.

OCR speed (PaddleOCR, CPU, single-thread) (2026-06-19)

Measured on a representative 1100×2372 page (~42 text boxes) via the paddle_prof harness on Apple Silicon (release build, single-threaded):

Per-stage breakdown of a page (WARM): detection ≈ 0.8 s, classification ≈ 0.2 s, recognition ≈ 3.5 s (recognition dominates — one CRNN+CTC inference per box). The COLD↔WARM gap (~0.7 s) is the one-time tract into_optimized() cost paid per distinct input shape: detection (1 shape) + classifier (1 fixed shape) + recognition (one shape per width bucket). The runnable cache lives on the PaddleOcr instance and persists across pages, and the whole-document path (pdfocr_save / pdfocr_tobytes) builds one engine and reuses it for every page — so that optimize cost is amortized to ~zero after the first page.

Tuning applied: the recognition width bucket was coarsened from 32 px to 64 px (crates/pdf-ocr/src/paddle/recognize.rs). On this page that cuts distinct recognition shapes from 9 to 6, shrinking the COLD penalty (and improving cross-page cache hits) with no accuracy change (the extra ≤63 px right-pad is CTC blank): CJK 1.000 and Latin 0.867 are byte-for-byte unchanged from the 32 px baseline. Reproduce the timing with cargo test -p pdf-ocr --release --test paddle_prof -- --nocapture --ignored.

OCR speed — rayon parallelism (2026-06-21)

Recognition dominates per-page time (≈3.5 s of a ~4.5 s WARM page; one CRNN+CTC inference per box), and the per-box loop is independent, so PaddleOcr::recognize now runs it as a rayon par_iter. On the same 42-box page this is a 3.49× speedup (16 cores: 2858 ms → 819 ms), and the output is byte-identical to the sequential version — the par_iter().map(..).collect::<Vec<_>>() is an indexed collect, so box order is preserved (verified against a captured 1-thread baseline). rayon is a feature-gated (paddle-ocr) optional dependency and is not pulled into the lean base wheel.

7. Reproduce

# OCR accuracy — PaddleOCR vs Tesseract (= fitz's OCR), one line:
bash conformance/ocr/run_ocr_bench.sh

# objective ground-truth (build corpora first, then score):
conformance/gt/born_digital.py  --out conformance/gt/corpus-born
conformance/gt/pmc_fetch.py     --out conformance/gt/corpus-pmc   --n 12
conformance/gt/fetch_eurlex.py  --out conformance/gt/corpus-eurlex
conformance/gt/run_gt.py --manifest <each>/manifest.json --report conformance/gt/GT-REPORT.md
# pypdfium2 column: same manifests + same score.py, extraction via pypdfium2 (in .venv-oracle).

# real-corpus differential vs fitz:
conformance/fetch_corpus.py
conformance/run_validation.py --corpus fixtures/corpus --report conformance/REPORT.md

Oracles (PyMuPDF/pdfminer) live in a separate local venv (.venv-oracle) and are never imported into the Apache-2.0 build. See conformance/REPORT.md and conformance/gt/GT-REPORT.md for the latest machine-generated numbers.