ultravox 0.4.1 doesn't work with vllm

[Madoshakalaka]

with the dependency:

[tool.poetry.dependencies]
vllm = {extras = ["audio"], version = "^0.6.6.post1"}
transformers = "^4.47.0" # same as ultravox main branch

following this example here:
(but with "fixie-ai/ultravox-v0_3" changed to "fixie-ai/ultravox-v0_4_1-llama-3_1-8b")

https://github.com/vllm-project/vllm/blob/0794e7446efca1fd7b8ea1cde96777897660cdea/examples/offline_inference/audio_language.py#L27-L45

the error I got:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ME/Projects/my-project/./main.py", line 28, in <module>
[rank0]:     llm = LLM(model="fixie-ai/ultravox-v0_4_1-llama-3_1-8b", task="generate", trust_remote_code=True)
...
[rank0]:   File "/home/ME/.cache/pypoetry/virtualenvs/my-project-7y5QMYrF-py3.10/lib/python3.10/site-packages/vllm/inputs/registry.py", line 108, in get_hf_processor
[rank0]:     raise TypeError("Invalid type of HuggingFace processor. "
[rank0]: TypeError: Invalid type of HuggingFace processor. Expected type: <class 'transformers.processing_utils.ProcessorMixin'>, but found type: <class 'transformers.tokenization_utils_fast.PreTrainedTokenizerFast'>

[zqhuang211]

@petersalas I am not sure if the vllm example needs to be updated to work with the latest ultravox model. Please take a look.

[petersalas]

I think the issue is that for some reason the UltravoxProcessor isn't registered as an AutoProcessor in our recent HF models. (vLLM recently changed to depend on UltravoxProcessor)

Will take a look this week.

[Madoshakalaka]

any update on this?

[petersalas]

Still working through the issue, it seems like there were some changes to UltravoxProcessor relative to 0.3 that are incompatible with latest vLLM. (A workaround in the short-term is to use an older version of vLLM -- I think 0.6.4 should work.)

[Madoshakalaka]

Thanks! I can confirm 0.6.4 works, except that <|audio|> token needs to be changed to <|reserved_special_token_0|>