Implement max character check

[samlhuillier]

Returns unknown token for strings greater than param maxInputCharsPerWord (defaulting to 100). Fixes #397

[xenova]

Thanks so much! Just note that the config is instantiated with the tokenizer.json, and as a result, relies on the exact same casing... which in this case, means it has to be max_input_chars_per_word

See https://huggingface.co/Xenova/bert-base-uncased/raw/main/tokenizer.json:

I also think it will be useful to add a test case for this here, something like:
https://github.com/xenova/transformers.js/blob/4e4148cb5ce7f4a9265f58b4eeb660c64bed0386/tests/tokenizers.test.js#L42-L50.

[xenova]

Thanks so much! 🚀 The same output as the python library is obtained now:

It took  1.5605939999222755 ms to tokenize
Encoded:  [101,  100, 1009, 100, 1027, 1027, 102]
Decoded:  [CLS] [UNK] + [UNK] = = [SEP]

(and the time taken is now in the same range as the python library: 1.67ms for python, 1.56ms for JS... not extensive tests, just ran a few times and took the average)

[samlhuillier]

Amazing! I shall update my package :)

[xenova]

A new release (2.8.1) will be out shortly, but in the meantime, you can install from source (GitHub) if this fix is urgent.