Token confidence

[willangley]

Switch to per-token confidence when writing out PDFs. This will (eventually!) let us not lose lines of text when part of them has been damaged.

[willangley]

This isn't ready yet.

• --min_confidence=0.9 discards too much good text. We should tune it again.
• Text selection in Preview doesn't work reliably; it's easy for selection to pass through a gap between tokens and start selecting a background image instead.
  • this can happen naturally from white space between tokens in the original text...
  • or from spaces that result from low-confidence tokens. We may want to fill this with other text instead.
• We're not writing spaces between tokens in the PDF, and should be. This was masked by viewers auto-inserting spaces based on the gap between tokens, but that's not reliable, and when it fails theresultingtextishardtoread ¹.

Footnotes

1. The resulting text is hard to read. ↩