fix: fix hybrid chunker token constraint

[vagenas]

Resolves DS4SD/docling#723

• Root cause was that delimiters (both (1) between headings/captions & text, and (2) within headings/captions) were not properly accounted for when calculating the available number of tokens, which would manifest itself with different tokenizers (e.g. granite) or even with the default tokenizer (all-MiniLM-L6-v2) when used with more complex delimiters like ####.
• Refactored its inner workings to reuse common API elements (.serialize(), .delim) instead of unnecessarily replicating serialization logic.
• Added test to cover addressed case.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

Wonderful, this rule succeeded.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

[dolfim-ibm]

python3.13 should be fixed with f79791f