modelscope installed will lead some problems

[jijivski]

ran eval code on alibaba cloud

lm_eval --model vllm \
    --model_args "pretrained=Qwen2.5-3B,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=1" \
    --tasks gsm8k \
    --batch_size auto \
    --device cuda:0 \

pip uninstall modelscope -y # it will lead to modelscope to download the dataset, but since they have different name in modelscope and huggingface, this will fail

or: modelscope.hub.errors.RequestError: Url = https://www.modelscope.cn/api/v1/datasets/EleutherAI/hendrycks_math, Request id=a81d68cd-35f2-4c2d-ab08-92aa27e913ac Code = 10020101002 Message = 不存在的数据集(means dataset not exist), Please specify correct dataset_name and namespace.

export VLLM_USE_MODELSCOPE=0 # if set to True vllm will load modelscope and fail, this will lead to vllm mot loaded and no exception reported
mey

or: NameError: name 'LLM' is not defined when using vllm in lm_eval

[baberabb]

Hi! you can change the dataset name in the task configs. for example for gsm8k here