Ignore invalid UTF-8 input in the BPE tokenizer#11729

Merged

ggerganov merged 1 commit intoggerganov:masterfrom cfillion:bpe-tokenize-mojibake-noexcept

Feb 7, 2025

+8-1

Commits on Feb 7, 2025

ignore invalid UTF-8 input in the BPE tokenizer
cfillion
committed