AttributeError: 'LlamaForCausalLM' object has no attribute 'update'

[scigeek72]

Hi, I am trying to run one of your codes for fine tuning code llama 34B model. Instead of running it on Colab, I have downloaded the file and running it on Azure VM (A100 with 80GB GPU RAM). Here are the particulars:

Unsloth 2024.11.7: Fast Llama patching. Transformers = 4.46.2.
GPU: NVIDIA A100 80GB PCIe. Max memory: 79.151 GB. Platform = Linux. Pytorch: 2.5.1+cu124. CUDA = 8.0. CUDA Toolkit = 12.4.
Bfloat16 = TRUE. FA [Xformers = 0.0.28.post3. FA2 = False]

While running this line:

 model, tokenizer = FastLanguageModel.from_pretrained(model_name = "unsloth/codellama-34b-bnb-4bit",max_seq_length = max_seq_length,dtype = dtype, load_in_4bit = load_in_4bit)

I am getting the following error:

Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00,  1.05s/it]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda/envs/llm/lib/python3.12/site-packages/unsloth/models/loader.py", line 350, in from_pretrained
    model, tokenizer = dispatch_model.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/llm/lib/python3.12/site-packages/unsloth/models/llama.py", line 1632, in from_pretrained
    model, tokenizer = model_patcher.post_patch(model, tokenizer)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/llm/lib/python3.12/site-packages/unsloth/models/llama.py", line 1815, in post_patch
    model, tokenizer = patch_model_and_tokenizer(model, tokenizer, downcast_rope = True)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/llm/lib/python3.12/site-packages/unsloth_zoo/patching_utils.py", line 161, in patch_model_and_tokenizer
    current_model.update({"unsloth_optimized" : True})
    ^^^^^^^^^^^^^^^^^^^^
  File "/anaconda/envs/llm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1931, in __getattr__
    raise AttributeError(
AttributeError: 'LlamaForCausalLM' object has no attribute 'update'

I would appreciate it if you could help me fix this issue.
The script I am running as is can be found here:
Thanks.

[danielhanchen]

@scigeek72 Apologies on the issue just fixed it in the nightly branch! Please update Unsloth-Zoo via pip uninstall unsloth-zoo -y && pip install git+https://github.com/unslothai/unsloth-zoo.git@nightly