Skip to content
This repository has been archived by the owner on Feb 15, 2025. It is now read-only.

Commit

Permalink
more clarity in the README
Browse files Browse the repository at this point in the history
  • Loading branch information
justinthelaw authored Oct 3, 2024
1 parent 99c27c9 commit c106e10
Showing 1 changed file with 15 additions and 8 deletions.
23 changes: 15 additions & 8 deletions packages/vllm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,17 @@ See the LeapfrogAI documentation website for [system requirements](https://docs.

The default model that comes with this backend in this repository's officially released images is a [4-bit quantization of the Synthia-7b model](https://huggingface.co/TheBloke/SynthIA-7B-v2.0-GPTQ).

All of the commands in this sub-section are executed with the `packages/vllm` sub-directory.
All of the commands in this sub-section are executed within this `packages/vllm` sub-directory.

Optionally, you can specify a different model during Zarf create:
Optionally, you can specify a different model during Zarf creation:

```bash
uds zarf package create --confirm --set MODEL_REPO_ID=defenseunicorns/Hermes-2-Pro-Mistral-7B-4bit-32g --set MODEL_REVISION=main
```

If you decide to use a different model, there will likely be a need to change generation and engine runtime configurations, please see the [Zarf Package Config](./zarf-config.yaml) and the [values override file](./values/upstream-values.yaml) for details on what runtime parameters can be modified. These parameters are model-specific, and can be found in the HuggingFace model cards and/or configuration files (e.g., prompt templates).

For example, during deployment, you can override the Zarf Package Config defaults by doing the following:
For example, during Zarf deployment, you can override the Zarf Package Config defaults by doing the following:

```bash
uds zarf package deploy zarf-package-vllm-amd64-dev.tar.zst --confirm --set ENFORCE_EAGER=True
Expand All @@ -47,18 +47,25 @@ uds zarf package deploy packages/vllm/zarf-package-vllm-*-dev.tar.zst --confirm

### Local Development

The [config.yaml](./config.yaml) and [.env.example](./.env.example) must be modified if the model has changed away from the default.
In local development the [config.yaml](./config.yaml) and [.env.example](./.env.example) must be modified if the model has changed away from the default. The LeapfrogAI SDK picks up the `config.yaml` automatically, and the `.env` must be sourced into the Python environment.

Create a `.env` file based on the [.env.example](./.env.example):
> [!IMPORTANT]
> Execute the following commands from this sub-directory
Create a `.env` file based on the [`.env.example`](./.env.example):

```bash
cp .env.example .env
source .env
```

To run the vllm backend locally:
As necessary, modify the existing [`config.yaml`](./config.yaml):

> [!IMPORTANT]
> Execute the following commands from this sub-directory
```bash
vim config.yaml
```

To run the vllm backend locally:

```bash
# Install dev and runtime dependencies
Expand Down

0 comments on commit c106e10

Please sign in to comment.