Update tokenizer.py

[ghost]

Added proper tokenizer support for Hindi Language which would prevent crash while fine tuning Hindi language.

[eginhard]

Thanks a lot for the PR, this is great!

It includes some changes that are not related to Hindi because there are some small differences in the code between this fork and the original repo and it looks like you started from the original one. Could you revert these unrelated changes?

[ghost]

Changes fixed, pls check.

[manash997]

Hi @akshatrocky i am getting this error, while trying to run tokenizer script from your branch:
tokenizer.py", line 806, in test_expand_numbers_multilingual
assert out == b, f"'{out}' vs '{b}'"
AssertionError: 'Через двенадцать целых пять десятых секунды.' vs 'Через двенадцать запятая пять секунды.'

[ghost]

Which language are you trying to generate @manash997 ?

(Edit) - How are you running the tokenizer script, from fine tuning XTTS?

[manash997]

Which language are you trying to generate @manash997 ?

(Edit) - How are you running the tokenizer script, from fine tuning XTTS?

i just tried running it as a standalone python script. The changes for Hindi, look good to me. However wanted to test the entire script once.

[ghost]

Running this script(standalone), even without changes made by me also produces an error. The main use of tokenizer.py is for fine tuning XTTS. When we fine tune any language, the program gets the tokenizer from which it gets fine tuned. Which means this script is only used in fine tuning, as in my knowledge.

[eginhard]

Yes, the checks in this file are not actually run during any tests and were probably already broken for a while. Otherwise everything looks good. The author deleted their account, so I had to recreate the PR in #64.