Skip to content

Commit

Permalink
README 8 (#45)
Browse files Browse the repository at this point in the history
  • Loading branch information
tchaton authored Feb 28, 2024
1 parent cd36ac3 commit 407db1e
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Once optimized, efficient distributed training becomes practical regardless of w

Lightning Data supports **images, text, video, audio, geo-spatial, and multimodal data** types, is already adopted by frameworks such as [Lit-GPT](https://github.com/Lightning-AI/lit-gpt/blob/main/pretrain/tinyllama.py) to pretrain LLMs and integrates smoothly with [PyTorch Lightning](https://lightning.ai/docs/pytorch/stable/), [Lightning Fabric](https://lightning.ai/docs/fabric/stable/), and [PyTorch](https://pytorch.org/docs/stable/index.html).

[Runnable templates](#runnable-templates) published on the [Lightning.AI Platform](https://lightning.ai) are available at the end, they are fully reproducible with 1-click.
[Runnable templates](#runnable-templates) published on the [Lightning.AI Platform](https://lightning.ai) are available at the end, **reproducible in 1-click**.

### Table of Contents

Expand All @@ -37,7 +37,7 @@ Lightning Data supports **images, text, video, audio, geo-spatial, and multimoda

## Installation

Lightning Data can be installed with `pip`:
Install **Lightning Data** with `pip`

```bash
pip install litdata
Expand All @@ -47,7 +47,7 @@ pip install litdata

### 1. Prepare Your Data

Convert your raw dataset into Lightning Streaming Format using the `optimize` operator.
Convert your raw dataset into **Lightning Data Optimized Streaming Format** using the `optimize` operator.

Here is an example with some random images.

Expand Down Expand Up @@ -82,7 +82,7 @@ The `optimize` operator supports any data structures and types. Serialize whatev

### 2. Upload Your Data to Cloud Storage

Cloud providers such as [AWS](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [Google Cloud](https://cloud.google.com/storage/docs/uploading-objects?hl=en#upload-object-cli), [Azure](https://learn.microsoft.com/en-us/azure/import-export/storage-import-export-data-to-files?tabs=azure-portal-preview), etc.. provide command line client to upload your data to their storage.
Cloud providers such as [AWS](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), [Google Cloud](https://cloud.google.com/storage/docs/uploading-objects?hl=en#upload-object-cli), [Azure](https://learn.microsoft.com/en-us/azure/import-export/storage-import-export-data-to-files?tabs=azure-portal-preview) provide command line client to upload your data to their storage.

Here is how to upload the optimized dataset using the [AWS CLI](https://aws.amazon.com/s3) to [AWS S3](https://aws.amazon.com/s3/).

Expand Down

0 comments on commit 407db1e

Please sign in to comment.