getting error while finetuning llama2 using qlora

[himanshushukla12]

While executing the file in folder Olive/examples/llama2 i got the error of
TypeError: LlamaForCausalLM.forward() got an unexpected keyword argument 'past_key_values.0.key'
while executing :
python llama2.py --model_name meta-llama/Llama-2-7b-hf

[2024-08-14 11:29:21,087] [INFO] [run.py:138:run_engine] Running workflow default_workflow
[2024-08-14 11:29:21,089] [INFO] [cache.py:51:__init__] Using cache directory: /home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/cache/default_workflow
[2024-08-14 11:29:21,091] [INFO] [engine.py:1015:save_olive_config] Saved Olive config to /home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/cache/default_workflow/olive_config.json
[2024-08-14 11:29:21,092] [INFO] [accelerator_creator.py:224:create_accelerators] Running workflow on accelerator specs: cpu-cpu
[2024-08-14 11:29:21,104] [INFO] [engine.py:276:run] Running Olive on accelerator: cpu-cpu
[2024-08-14 11:29:21,104] [INFO] [engine.py:1112:_create_system] Creating target system ...
[2024-08-14 11:29:21,104] [INFO] [engine.py:1115:_create_system] Target system created in 0.000047 seconds
[2024-08-14 11:29:21,104] [INFO] [engine.py:1124:_create_system] Creating host system ...
[2024-08-14 11:29:21,104] [INFO] [engine.py:1127:_create_system] Host system created in 0.000033 seconds
/home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/venvLlama3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
/home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/venvLlama3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.05s/it]
Traceback (most recent call last):
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/llama2.py", line 198, in <module>
    main()
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/llama2.py", line 134, in main
    olive_run(template_json, tempdir=args.tempdir)  # pylint: disable=not-callable
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/workflows/run/run.py", line 321, in run
    return run_engine(package_config, run_config)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/workflows/run/run.py", line 261, in run_engine
    engine.run(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/engine/engine.py", line 278, in run
    run_result = self.run_accelerator(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/engine/engine.py", line 337, in run_accelerator
    results = self._evaluate_model(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/engine/engine.py", line 1079, in _evaluate_model
    signal = self.target.evaluate_model(model_config, evaluator_config, accelerator_spec)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/systems/local.py", line 45, in evaluate_model
    return evaluator.evaluate(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/evaluator/olive_evaluator.py", line 314, in evaluate
    metrics_res[metric.name] = self._evaluate_latency(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/evaluator/olive_evaluator.py", line 247, in _evaluate_latency
    latencies = self._evaluate_raw_latency(model, metric, dataloader, post_func, device, execution_providers)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/venvLlama3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/evaluator/olive_evaluator.py", line 804, in _evaluate_raw_latency
    model.run_session(session, input_data, **run_kwargs)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/model/handler/pytorch.py", line 45, in run_session
    results = session.generate(**inputs, **kwargs) if self.generative else session(**inputs, **kwargs)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/venvLlama3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/venvLlama3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: LlamaForCausalLM.forward() got an unexpected keyword argument 'past_key_values.0.key'

Olive config
Add Olive configurations here.

Olive logs
Add logs here.

Other information

• OS: Ubuntu 22.04
• Olive version: 0.7.0
• onnxruntime-genai-cuda 0.3.0
• onnxruntime-gpu 1.18.1
• Python: 3.10.11

Additional context
I build olive from source using pip install -e .

Please help

[xiaoyu-work]

Are you trying to use qlora? Can you try python llama2.py --model_name meta-llama/Llama-2-7b-hf --qlora?

[himanshushukla12]

python llama2.py --model_name meta-llama/Llama-2-7b-hf --qlora

Here is the complete output

/Olive/examples/llama2$ python llama2.py --model_name meta-llama/Llama-2-7b-hf --qlora
[2024-08-21 10:14:32,863] [INFO] [config.py:252:validate_evaluate_input_model] No evaluator is specified, skip to evaluate model
[2024-08-21 10:14:32,864] [INFO] [run.py:138:run_engine] Running workflow default_workflow
[2024-08-21 10:14:32,864] [INFO] [cache.py:51:__init__] Using cache directory: /home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/.olive-cache/default_workflow
[2024-08-21 10:14:32,865] [INFO] [engine.py:1012:save_olive_config] Saved Olive config to /home/z004x2xz/WorkAssignedByMatt/Olive/examples/llama2/.olive-cache/default_workflow/olive_config.json
[2024-08-21 10:14:33,733] [INFO] [accelerator_creator.py:224:create_accelerators] Running workflow on accelerator specs: gpu-cuda
[2024-08-21 10:14:33,741] [INFO] [engine.py:275:run] Running Olive on accelerator: gpu-cuda
[2024-08-21 10:14:33,741] [INFO] [engine.py:1109:_create_system] Creating target system ...
[2024-08-21 10:14:33,741] [INFO] [engine.py:1112:_create_system] Target system created in 0.000035 seconds
[2024-08-21 10:14:33,741] [INFO] [engine.py:1121:_create_system] Creating host system ...
[2024-08-21 10:14:33,741] [INFO] [engine.py:1124:_create_system] Host system created in 0.000026 seconds
[2024-08-21 10:14:33,767] [INFO] [engine.py:877:_run_pass] Running pass f:QLoRA
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.55s/it]
You are calling `save_pretrained` to a 4-bit converted model, but your `bitsandbytes` version doesn't support it. If you want to save 4-bit models, make sure to have `bitsandbytes>=0.41.3` installed.
Generating train split: 0 examples [00:01, ? examples/s]
[2024-08-21 10:14:43,729] [ERROR] [engine.py:974:_run_pass] Pass run failed.
Traceback (most recent call last):
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1750, in _prepare_split_single
    for key, record in generator:
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/packaged_modules/generator/generator.py", line 32, in _generate_examples
    for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/pytorch/lora.py", line 257, in data_generator
    dataset = data_container.pre_process(data_container.load_dataset())
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/data/container/data_container.py", line 26, in load_dataset
    return self.config.load_dataset(**self.config.load_dataset_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/data/component/load_dataset.py", line 39, in huggingface_dataset
    dataset = load_dataset(path=data_name, name=subset, data_dir=data_dir, split=split, data_files=data_files, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 2594, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 2266, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 1914, in dataset_module_factory
    raise e1 from None
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 1896, in dataset_module_factory
    ).get_module()
      ^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 1225, in get_module
    standalone_yaml_path = cached_path(
                           ^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 201, in cached_path
    output_path = get_from_cache(
                  ^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 635, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})")
ConnectionError: Couldn't reach https://huggingface.co/datasets/nampdn-ai/tiny-codes/resolve/9aebe5ee8b406356d5f5f2d603bc0a1684ee8ce7/.huggingface.yaml (error 403)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/engine/engine.py", line 962, in _run_pass
    output_model_config = host.run_pass(p, input_model_config, output_model_path, pass_search_point)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/systems/local.py", line 30, in run_pass
    output_model = the_pass.run(model, output_model_path, point)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/olive_pass.py", line 206, in run
    output_model = self._run_for_config(model, config, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/pytorch/lora.py", line 688, in _run_for_config
    output_model = self.train_and_save_new_model(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/pytorch/lora.py", line 483, in train_and_save_new_model
    train_dataset, eval_dataset = self.get_datasets(config)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/pytorch/lora.py", line 269, in get_datasets
    train_dataset = Dataset.from_generator(data_generator, gen_kwargs={"data_config": train_data_config})
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 1130, in from_generator
    ).read()
      ^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/io/generator.py", line 47, in read
    self.builder.download_and_prepare(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1029, in download_and_prepare
    self._download_and_prepare(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1791, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1124, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1629, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1786, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
[2024-08-21 10:14:43,763] [WARNING] [engine.py:370:run_accelerator] Failed to run Olive on gpu-cuda.
Traceback (most recent call last):
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1750, in _prepare_split_single
    for key, record in generator:
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/packaged_modules/generator/generator.py", line 32, in _generate_examples
    for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/pytorch/lora.py", line 257, in data_generator
    dataset = data_container.pre_process(data_container.load_dataset())
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/data/container/data_container.py", line 26, in load_dataset
    return self.config.load_dataset(**self.config.load_dataset_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/data/component/load_dataset.py", line 39, in huggingface_dataset
    dataset = load_dataset(path=data_name, name=subset, data_dir=data_dir, split=split, data_files=data_files, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 2594, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 2266, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 1914, in dataset_module_factory
    raise e1 from None
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 1896, in dataset_module_factory
    ).get_module()
      ^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/load.py", line 1225, in get_module
    standalone_yaml_path = cached_path(
                           ^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 201, in cached_path
    output_path = get_from_cache(
                  ^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 635, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})")
ConnectionError: Couldn't reach https://huggingface.co/datasets/nampdn-ai/tiny-codes/resolve/9aebe5ee8b406356d5f5f2d603bc0a1684ee8ce7/.huggingface.yaml (error 403)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/engine/engine.py", line 349, in run_accelerator
    output_footprint = self.run_no_search(
                       ^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/engine/engine.py", line 441, in run_no_search
    should_prune, signal, model_ids = self._run_passes(
                                      ^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/engine/engine.py", line 814, in _run_passes
    model_config, model_id, output_model_hash = self._run_pass(
                                                ^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/engine/engine.py", line 962, in _run_pass
    output_model_config = host.run_pass(p, input_model_config, output_model_path, pass_search_point)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/systems/local.py", line 30, in run_pass
    output_model = the_pass.run(model, output_model_path, point)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/olive_pass.py", line 206, in run
    output_model = self._run_for_config(model, config, output_model_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/pytorch/lora.py", line 688, in _run_for_config
    output_model = self.train_and_save_new_model(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/pytorch/lora.py", line 483, in train_and_save_new_model
    train_dataset, eval_dataset = self.get_datasets(config)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/olive/passes/pytorch/lora.py", line 269, in get_datasets
    train_dataset = Dataset.from_generator(data_generator, gen_kwargs={"data_config": train_data_config})
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 1130, in from_generator
    ).read()
      ^^^^^^
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/io/generator.py", line 47, in read
    self.builder.download_and_prepare(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1029, in download_and_prepare
    self._download_and_prepare(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1791, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1124, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1629, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/z004x2xz/WorkAssignedByMatt/Olive/venv3.11/lib/python3.11/site-packages/datasets/builder.py", line 1786, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
[2024-08-21 10:14:43,768] [INFO] [engine.py:292:run] Run history for gpu-cuda:
[2024-08-21 10:14:43,771] [INFO] [engine.py:587:dump_run_history] run history:
+------------+-------------------+-------------+----------------+-----------+
| model_id   | parent_model_id   | from_pass   | duration_sec   | metrics   |
+============+===================+=============+================+===========+
| 1ea91e35   |                   |             |                |           |
+------------+-------------------+-------------+----------------+-----------+
[2024-08-21 10:14:43,772] [INFO] [engine.py:307:run] No packaging config provided, skip packaging artifacts

[xiaoyu-work]

Can you login huggingface first? https://huggingface.co/docs/huggingface_hub/en/guides/cli#huggingface-cli-login

[devang-ml]

You need to gain access to tiny code data set. https://huggingface.co/datasets/nampdn-ai/tiny-codes