library for unicode text segmentation

[mmozeiko]

Unicode text can contain codepoints that must be grouped together with other codepoints next to it when processing text as characters for display. When you're processing text for rendering or otherwise you must know where the boundaries graphemes are.

It would be nice to have nice API in C for incremental processing text and determining these boundaries.
Unicode spec for algorithm doing this is here: https://unicode.org/reports/tr29/

API should offer state machine that accepts individual unicode codepoints on input and returns when boundary starts/ends. Don't worry about encodings, leave that to user's code.

Existing open source libraries offering this kind of functionality:

• harfbuzz - https://harfbuzz.github.io/clusters.html
• icu - https://unicode-org.github.io/icu/userguide/boundaryanalysis/

Good reference on how to compactly store properties for codepoints here: https://www.strchr.com/multi-stage_tables

As a bonus for more complete unicode text processing solution you can think of offering these features too:

• bidirectional algorithm: https://unicode.org/reports/tr9/
• line breaking algorithm: https://www.unicode.org/reports/tr14/
• normalization: https://www.unicode.org/reports/tr15/

[calebarg]

I'm interested in working on this.

[lukekasz]

C99 library that provides this functionality: https://libs.suckless.org/libgrapheme/