Skip to content

Commit

Permalink
docs: air-gapped deployments (huggingface#326)
Browse files Browse the repository at this point in the history
  • Loading branch information
OlivierDehaene authored and pi314ever committed Aug 20, 2024
1 parent 9c52d34 commit 7ad1a19
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 2 deletions.
23 changes: 23 additions & 0 deletions docs/source/en/quick_tour.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,26 @@ curl 127.0.0.1:8080/predict \
-d '{"inputs":[["I like you."], ["I hate pineapples"]]}' \
-H 'Content-Type: application/json'
```

## Air gapped deployment

To deploy Text Embeddings Inference in an air-gapped environment, first download the weights and then mount them inside
the container using a volume.

For example:

```shell
# (Optional) create a `models` directory
mkdir models
cd models

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5

# Set the models directory as the volume path
volume=$PWD

# Mount the models directory inside the container with a volume and set the model ID
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:1.4 --model-id /data/gte-base-en-v1.5
```
4 changes: 2 additions & 2 deletions router/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
struct Args {
/// The name of the model to load.
/// Can be a MODEL_ID as listed on <https://hf.co/models> like
/// `thenlper/gte-base`.
/// `Alibaba-NLP/gte-base-en-v1.5`.
/// Or it can be a local directory containing the necessary files
/// as saved by `save_pretrained(...)` methods of transformers
#[clap(default_value = "thenlper/gte-base", long, env)]
#[clap(default_value = "Alibaba-NLP/gte-base-en-v1.5", long, env)]
#[redact(partial)]
model_id: String,

Expand Down

0 comments on commit 7ad1a19

Please sign in to comment.