diff --git a/.github/workflows/build_and_deploy_documentation.yml b/.github/workflows/build_and_deploy_documentation.yml index 70694a0c..a8e154f3 100644 --- a/.github/workflows/build_and_deploy_documentation.yml +++ b/.github/workflows/build_and_deploy_documentation.yml @@ -21,8 +21,8 @@ jobs: pip install sphinx-rtd-theme sphinx-autodoc-typehints sphinx-click sphinx-automodapi texext - name: "Parse into HTML" run: | - sphinx-build -M html docs/source/ docs/build/ sphinx-apidoc -o docs/source/api src/modalities + sphinx-build -M html docs/source/ docs/build/ - name: Deploy to GitHub Pages uses: peaceiris/actions-gh-pages@v3 with: diff --git a/config_files/config_lorem_ipsum.yaml b/config_files/config_lorem_ipsum.yaml index 2fdf305e..989a4de3 100644 --- a/config_files/config_lorem_ipsum.yaml +++ b/config_files/config_lorem_ipsum.yaml @@ -5,7 +5,7 @@ modalities_setup: wandb: project_name: modalities - mode: ONLINE + mode: OFFLINE data: sample_key: "input_ids" diff --git a/docs/logo.jpg b/docs/logo.jpg deleted file mode 100644 index 7d16cb8d..00000000 Binary files a/docs/logo.jpg and /dev/null differ diff --git a/docs/source/conf.py b/docs/source/conf.py index 98dc8c66..f7860c65 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -100,7 +100,7 @@ # The name of an image file (relative to this directory) to place at the top # of the sidebar. # -html_logo = "../logo.jpg" +html_logo = "logo.jpg" # -- Options for HTMLHelp output --------------------------------------------- diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst index bc0d0480..e51335fa 100644 --- a/docs/source/configuration.rst +++ b/docs/source/configuration.rst @@ -4,9 +4,7 @@ Configuration ======================================================================== -**EDIT "docs/source/configuration.rst" IN ORDER TO MAKE CHANGES HERE** - -Training config is defined in yaml formatted files. See :file:`data/config_lorem_ipsum.yaml`. These configs are very explicit specifying all training parameters to keep model trainings as transparent and reproducible as possible. Each config setting is reflected in pydantic classes in :file:`src/llm_gym/config/*.py`. In the config you need to define which config classes to load in field type_hint. This specifies the concrete class. A second parameter, config, then takes all the constructor arguments for that config class. This way it is easy to change i.e. DataLoaders while still having input validation in place. +Training config is defined in yaml formatted files. See :file:`data/config_lorem_ipsum.yaml`. These configs are very explicit specifying all training parameters to keep model trainings as transparent and reproducible as possible. Each config setting is reflected in pydantic classes in :file:`src/modalities/config/*.py`. In the config you need to define which config classes to load in field type_hint. This specifies the concrete class. A second parameter, config, then takes all the constructor arguments for that config class. This way it is easy to change i.e. DataLoaders while still having input validation in place. Pydantic and ClassResolver ------------------------------------------------------------------------ @@ -39,7 +37,7 @@ With this we utilize Pydantics feature to auto-select a fitting type based on th In our implmentation we go a step further, as both, -* a :python:`type_hint` in a :python:`BaseModel` config must be of type :python:`llm_gym.config.lookup_types.LookupEnum` and +* a :python:`type_hint` in a :python:`BaseModel` config must be of type :python:`modalities.config.lookup_types.LookupEnum` and * :python:`config` is a union of allowed concrete configs of base type :python:`BaseModel`. :python:`config` hereby replaces :python:`activation_kwargs` in the example above, and replaces it with pydantic-validated :python:`BaseModel` configs. diff --git a/docs/source/entrypoints.rst b/docs/source/entrypoints.rst index b370026e..134bbaff 100644 --- a/docs/source/entrypoints.rst +++ b/docs/source/entrypoints.rst @@ -8,13 +8,10 @@ Entrypoints ======================================================= - -**EDIT "docs/source/entrypoints.rst" IN ORDER TO MAKE CHANGES HERE** - We use `click `_ as a tool to add new entry points and their CLI arguments. For this we have a main entry point from which all other entry points are started. -The main entry point is :file:`src/llm_gym/__main__.py:main()`. +The main entry point is :file:`src/modalities/__main__.py:main()`. We register other sub-entrypoints by using our main :python:`click.group`, called :python:`main`, as follows: .. code-block:: python @@ -64,8 +61,8 @@ With .. code-block:: python [project.scripts] - llm_gym = "llm_gym.__main__:main" + modalities = "modalities.__main__:main" -in our :file:`pyproject.toml`, we can start only main with :python:`llm_gym` (which does nothing), or a specific sub-entrypoint e.g. :bash:`llm_gym do_stuff --config_file_path config_files/config.yaml --my_cli_argument 3537`. +in our :file:`pyproject.toml`, we can start only main with :python:`modalities` (which does nothing), or a specific sub-entrypoint e.g. :bash:`modalities do_stuff --config_file_path config_files/config.yaml --my_cli_argument 3537`. -Alternatively, directly use :bash:`src/llm_gym/__main__.py do_stuff --config_file_path config_files/config.yaml --my_cli_argument 3537`. +Alternatively, directly use :bash:`src/modalities/__main__.py do_stuff --config_file_path config_files/config.yaml --my_cli_argument 3537`. diff --git a/docs/source/index.rst b/docs/source/index.rst index 9fbc05e7..24283df1 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,12 +1,17 @@ Welcome to Modalities's documentation! ====================================================================== -**EDIT "docs/source/index.rst" IN ORDER TO MAKE CHANGES HERE** +We propose a novel training framework for Multimodal Large Language Models (LLMs) that prioritizes code readability and efficiency. +The codebase adheres to the principles of "clean code," minimizing Lines of Code (LoC) while maintaining extensibility. +A single, comprehensive configuration file enables easy customization of various model and training parameters. - still needed: USPs, key features; include FSDP here;> - - +A key innovation is the adoption of a PyTorch-native training loop integrated with the Fully Sharded Data Parallelism (FSDP) technique. +FSDP optimizes memory usage and training speed, enhancing scalability for large-scale multimodal models. +By leveraging PyTorch's native capabilities, our framework simplifies the development process and promotes ease of maintenance. +The framework's modular design facilitates experimentation with different multimodal architectures and training strategies. +Users can seamlessly integrate diverse datasets and model components, allowing for comprehensive exploration of multimodal learning tasks. +The combination of clean code, minimal configuration, and PyTorch-native training with FSDP contributes to a user-friendly and efficient platform for developing state-of-the-art multimodal language models. .. note:: diff --git a/docs/source/known_issues.rst b/docs/source/known_issues.rst index af2a5a14..6ac9a579 100644 --- a/docs/source/known_issues.rst +++ b/docs/source/known_issues.rst @@ -1,7 +1,7 @@ Known Issues ================================================================== -**EDIT "docs/source/known_issues.rst" IN ORDER TO MAKE CHANGES HERE** +`GitHub Issues `_ 1. hardcoded dataset path :file:`/raid/s3/opengptx/mehdi/temp/temp_data/train_text_document.bin` in :file:`config/config.yaml` 2. Dependency on weights&biases diff --git a/docs/source/logo.jpg b/docs/source/logo.jpg new file mode 100644 index 00000000..b298fdd5 Binary files /dev/null and b/docs/source/logo.jpg differ diff --git a/docs/source/memmap.rst b/docs/source/memmap.rst index 6af91015..22793c08 100644 --- a/docs/source/memmap.rst +++ b/docs/source/memmap.rst @@ -7,8 +7,6 @@ MemMap Datasets ==================================================== -**EDIT "docs/source/memmap.rst" IN ORDER TO MAKE CHANGES HERE** - MemMapDataset Index Generator ------------------------------------------------------------------------------ @@ -18,7 +16,7 @@ The :python:`MemMapDataset` requires an index file providing the necessary point modalities create_memmap_index -The index will be created in the same directory as the raw data file. For further options you may look into the usage documentation via :bash:`llm_gym create_memmap_index --help`. +The index will be created in the same directory as the raw data file. For further options you may look into the usage documentation via :bash:`modalities create_memmap_index --help`. Packed Dataset Generator -------------------------------------------------------------------------------- @@ -29,7 +27,7 @@ The :python:`PackedMemMapDatasetContinuous` and :python:`PackedMemMapDatasetMega modalities create_packed_data -The packed data file will be created in the same directory as the raw data file. For further options you may look into the usage documentation via :bash:`llm_gym create_packed_data --help`. +The packed data file will be created in the same directory as the raw data file. For further options you may look into the usage documentation via :bash:`modalities create_packed_data --help`. Packed Data Format ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/model_cards.rst b/docs/source/model_cards.rst index 2dbf99d6..dd7814fe 100644 --- a/docs/source/model_cards.rst +++ b/docs/source/model_cards.rst @@ -1,4 +1,3 @@ Model Cards ==================================================== -**EDIT "docs/source/model_cards.rst" IN ORDER TO MAKE CHANGES HERE** diff --git a/docs/source/quickstart.rst b/docs/source/quickstart.rst index 9ce55bba..d39a78a8 100644 --- a/docs/source/quickstart.rst +++ b/docs/source/quickstart.rst @@ -1,8 +1,6 @@ Quickstart ==================================================== -**EDIT "docs/source/quickstart.rst" IN ORDER TO MAKE CHANGES HERE** - Installation ----------------------------------------------------- Setup a conda environment `conda create -n modalities python=3.10 & conda activate modalities` and install the requirements `pip install -e .`. diff --git a/docs/source/vs_code_setup.rst b/docs/source/vs_code_setup.rst index 96272915..22dfc74f 100644 --- a/docs/source/vs_code_setup.rst +++ b/docs/source/vs_code_setup.rst @@ -1,33 +1,71 @@ VSCode Setup ==================================================== -**EDIT "docs/source/vs_code_setup.rst" IN ORDER TO MAKE CHANGES HERE** + + +We recommend a docker environment based on the most recent pytorch e.g.: + +.. code-block:: bash + + FROM pytorch/pytorch:2.1.2-cuda12.1-cudnn8-devel + RUN apt-get update && apt-get install -y wget openssh-client git-core bash-completion + RUN wget -O /tmp/git-lfs.deb https://packagecloud.io/github/git-lfs/packages/ubuntu/focal/git-lfs_2.13.3_amd64.deb/download.deb && \ + dpkg -i /tmp/git-lfs.deb && \ + rm /tmp/git-lfs.deb + RUN echo 'source /usr/share/bash-completion/completions/git' >> ~/.bashrc + CMD ["/bin/bash"] + +This works seamlessly in combination with the VSCode DevContainer extention: + +.. code-block:: json + + { + "name": "Dev Container", + "dockerFile": "Dockerfile", + "runArgs": [ + "--network", + "host", + "--gpus", + "all" + ], + "customizations": { + "vscode": { + "settings": { + "terminal.integrated.shell.linux": "/bin/bash" + }, + "extensions": [ + "ms-python.python" + ] + } + } + } In VSCode, add this to your :file:`launch.json`: .. code-block:: json - { - "name": "Torchrun Main", - "type": "python", - "request": "launch", - "module": "torch.distributed.run", - "env": { - "CUDA_VISIBLE_DEVICES": "0" - }, - "args": [ - "--nnodes", - "1", - "--nproc_per_node", - "2", - "--rdzv-endpoint=0.0.0.0:29503", - "src/modalities/__main__.py", - "run", - "--config_file_path", - "config_files/config.yaml", - ], - "console": "integratedTerminal", - "justMyCode": true, - "envFile": "${workspaceFolder}/.env" - } + { + "name": "Torchrun Train and Eval", + "type": "python", + "request": "launch", + "module": "torch.distributed.run", + "env": { + "CUDA_VISIBLE_DEVICES": "4,5" + }, + "args": [ + "--nnodes", + "1", + "--nproc_per_node", + "2", + "--rdzv-endpoint=0.0.0.0:29503", + "src/modalities/__main__.py", + "run", + "--config_file_path", + "config_files/config_lorem_ipsum.yaml", + ], + "console": "integratedTerminal", + "justMyCode": true, + "envFile": "${workspaceFolder}/.env", + "cwd": "${workspaceFolder}/modalities" + }