Microsoft.DeepDev.TokenizerLib 1.3.0 open source release.
- You can find the published NuGet package here.
- You can find the Tokenizer.exe in the attached Tokenizer_1.3.0.zip.
- To run performance benchmark, you can download the attached PerfBenchmark_1.3.0.zip.