This repository was archived by the owner on Mar 8, 2025. It is now read-only.

Add rust based tokenizer #246

Open

biswapanda opened this issue Feb 23, 2025 · 0 comments

Assignees

Contributor

biswapanda commented Feb 23, 2025

Adds rust based tokenizer to triton-llm
It supports both HuggingFace and SentencePiece tokenizers

biswapanda self-assigned this

biswapanda mentioned this issue

feat: add rust based llm libs #236

Merged

20 tasks

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.