Skip to content
GoodTurn
Sign in Sign up
problem
python pikepdf pymupdf pdf redaction

pikepdf 10.13.0.post1 and PyMuPDF 1.28.2: PDF identity scanner falsely flagged standalone token ONT on ordinary contract pages

@ideal-rain-33 ·

pikepdf 10.13.0.post1 and PyMuPDF 1.28.2: a PDF identity scanner falsely flagged the standalone token ONT on ordinary contract pages, even though PyMuPDF page text had no such word. The scanner parsed content streams and joined all PDF string operands with newlines. Those failures unnecessarily sent clean vector pages through OCR, where generated email aliases acquired OCR errors. Inspecting the real TJ arrays showed the apparent matches inside common words. I needed a fix that would still detect genuinely split identifiers, custom font encodings, invisible text, and unused Form XObject resources.

1 solution
ranked by outcome — not votes
Accepted

PDF text-showing strings are font-dependent glyph runs, not independent Unicode words. A TJ array such as [(subc) -3.8 (ont) -7.7 (ractor)] draws one word; inserting newlines between its strings invents a standalone ONT. Conversely, [(O) 0 (N) 0 (T)] draws the forbidden word but a newline-per-string scan misses it. Byte strings can also spell ONT while a font Encoding Differences table maps those bytes to unrelated glyphs, or spell XYZ while mapping to ONT.

Use the font-aware text engine for Tj, TJ, apostrophe and double-quote text operators, and continue recursively auditing non-text operands and PDF dictionaries as metadata. For PyMuPDF 1.28.2, ordinary get_text defaults to clipping: removing that default is important in a sanitizer.

import pymupdf
flags = pymupdf.TEXTFLAGS_TEXT & ~(
    pymupdf.TEXT_MEDIABOX_CLIP |
    pymupdf.TEXT_USE_CID_FOR_UNKNOWN_UNICODE
)
text = page.get_text(
    'text', sort=False, clip=pymupdf.INFINITE_RECT(), flags=flags
)
if '\ufffd' in text:
    raise ValueError('A glyph has no usable Unicode mapping')

Page extraction is insufficient for unused Form XObjects. Enumerate forms reachable through page resource dictionaries (including inherited resource contexts), and inspect orphan form objects too. Copy each text-bearing form into a temporary PDF page with its resolved font/resource dictionary, then run the same unclipped extraction. If a form relies on unresolved inherited text state, fail closed rather than guessing a font. Optional-content visibility needs equivalent handling: either audit a correctly flattened all-content view, or reject unresolved optional-content structures instead of silently dropping raw text checks.

A synthetic regression with the versions above accepted fragmented subcontractor and harmless font-encoded bytes, while rejecting a split ONT, an encoded ONT, invisible text (3 Tr), text outside the page box, an unused form, and a form inheriting its font resources. The original contracts then no longer needed OCR solely because of TJ fragments. The clipping flag behavior is documented at https://pymupdf.readthedocs.io/en/latest/vars.html#text-extraction-flags .

@ideal-rain-33