Code

added checkpointing to support LLMs#114

Code

Merged

zhenghh04 merged 27 commits intomainargonne-lcf/dlio_benchmark:mainfrom feature/checkpointargonne-lcf/dlio_benchmark:feature/checkpointCopy head branch name to clipboardJan 9, 2024

+647-74

Commits

Commits on Nov 21, 2023

• added checkpointing to support LLMs

  hariharan-devarajan

  committed
  557afed

  Copy full SHA for 557afed

• added indexed binary data support for LLMs.

  hariharan-devarajan

  committed
  8dd835e

  Copy full SHA for 8dd835e

• added configuration for megatron deepspeed.

  hariharan-devarajan

  committed
  0905c36

  Copy full SHA for 0905c36

• fixes for out of core data generation

  hariharan-devarajan

  committed
  c50f6fd

  Copy full SHA for c50f6fd

• fixes for out of core data generation

  hariharan-devarajan

  committed
  146ce85

  Copy full SHA for 146ce85

Commits on Nov 22, 2023

• fixes for out of core data generation

  hariharan-devarajan

  committed
  381880c

  Copy full SHA for 381880c

• added dlrm configuration

  hariharan-devarajan

  committed
  db73681

  Copy full SHA for db73681

• added changes to support mmapped file.

  hariharan-devarajan

  committed
  5cb6d0e

  Copy full SHA for 5cb6d0e

• added changes to support mmapped file.

  hariharan-devarajan

  committed
  0564dfe

  Copy full SHA for 0564dfe

• added changes to support mmapped file.

  hariharan-devarajan

  committed
  1b0d42e

  Copy full SHA for 1b0d42e

Commits on Nov 28, 2023

• added changes to support mmapped file.

  hariharan-devarajan

  committed
  52011b4

  Copy full SHA for 52011b4

• added changes to support mmapped file.

  hariharan-devarajan

  committed
  6221b33

  Copy full SHA for 6221b33

Commits on Nov 29, 2023

• fixed checkpointing for tensors

  hariharan-devarajan

  committed
  03796ad

  Copy full SHA for 03796ad

Commits on Dec 16, 2023

• Merge branch 'main' into feature/checkpoint

  hariharan-devarajan

  authored
  900cb53

  Copy full SHA for 900cb53

Commits on Jan 4, 2024

• Merge branch 'main' into feature/checkpoint

  hariharan-devarajan

  authored
  7ade86b

  Copy full SHA for 7ade86b

• Update torch_framework.py

  Show description for 96eefa2
  hariharan-devarajan

  authored
  96eefa2

  Copy full SHA for 96eefa2

Commits on Jan 5, 2024

• Update indexed_binary_generator.py

  Show description for b73cea5
  hariharan-devarajan

  authored
  b73cea5

  Copy full SHA for b73cea5

Commits on Jan 8, 2024

• Update megatron_deepspeed.yaml

  hariharan-devarajan

  authored
  fc42c86

  Copy full SHA for fc42c86

• refactor enum for better naming

  hariharan-devarajan

  committed
  0c058ce

  Copy full SHA for 0c058ce

• documentation for the checkpointing.

  hariharan-devarajan

  committed
  3f28662

  Copy full SHA for 3f28662

• make data generation buffer_size configurable.

  hariharan-devarajan

  committed
  7fdead2

  Copy full SHA for 7fdead2

• Update tf_framework.py

  Show description for 189b2e4
  hariharan-devarajan

  authored
  189b2e4

  Copy full SHA for 189b2e4

• Update tf_framework.py

  hariharan-devarajan

  authored
  8209980

  Copy full SHA for 8209980

• Update megatron_deepspeed.yaml

  hariharan-devarajan

  authored
  baf023f

  Copy full SHA for baf023f

• Update megatron_deepspeed.yaml

  hariharan-devarajan

  authored
  63ff8c4

  Copy full SHA for 63ff8c4

• make data generation buffer_size configurable.

  hariharan-devarajan

  committed
  a397d11

  Copy full SHA for a397d11

Commits on Jan 9, 2024

• Merge remote-tracking branch 'origin/feature/checkpoint' into feature/checkpoint

  hariharan-devarajan

  committed
  3727e5a

  Copy full SHA for 3727e5a