feat(vllm)!: upgrade vllm backend and refactor deployment

[justinthelaw]

OVERVIEW

See #835 for more details on rationale and findings. Also related to #623.

IMPORTANT NOTE: there are still ongoing AsyncLLMEngineDead and RoPE scaling + Ray issues upstream that may prevent us from upgrading past 0.4.x.

BREAKING CHANGES:

• moves all ENV specific to LeapfrogAI SDK to a ConfigMap using volumeMount for runtime injection and modification
  • in local dev, this is defined via config.yaml
• moves all ENV specific to vLLM to a ConfigMap, using envFrom for runtime injection and modification
  • in local dev, this is defined via .env
• ZARF_CONFIG is used to define create-time and deploy-time variables for (e.g., MODEL_REPO_ID, ENFORCE_EAGER)
  • updates Make targets and workflows with new ZARF_CONFIG variable
  • updates UDS bundles with new Zarf deployment variable overrides
  • allows delivery engineer's declarative definition of the backend configs and model
• re-introduces LFAI SDK config.yaml configuration method for local development and testing
• MUST upgrade API and backends together due to FinishReason proto change

CHANGES:

• updates docs for running vLLM locally, in docker, and in-cluster
• update to Python 3.11.9 to align with Registry1 base image
• upgrades vLLM from 0.4.2 to 0.4.3 for:
  • adds BFloat16 quantized model support (the "why is this important" here)
• exposes more backend configurations for Zarf build (see packages/vllm/README.md)
  • adds full set of QUANTIZATION options to existing configuration field
  • exposes everything via a Zarf variable and the values files
• removes default vLLM engine configurations from the Dockerfile (no duplicates, hardcoding)
• fixes issue where request params are not used in generation of the response (e.g., temperature)
  • uses LFAI SDK-received request object for inferencing params
  • gracefully handles vLLM versus SDK param names/differences for generation

ADDITIONAL CONTEXT

The default model is still Synthia-7b, until #976 is resolved. The below description is only being kept for future context:

The new model option, defenseunicorns/Hermes-2-Pro-Mistral-7B-4bit-32g, takes ~4.16Gb to load on to RAM or vRAM. At 6Gb of vRAM, the max_context_length variable has to be reduced to ~400 tokens; at 8Gb vRAM, ~10K tokens; and at 12Gb of vRAM, ~15K tokens. All cases require the max_gpu_utilization variable to be 0.99 in order to max out the KV cache size for the context length that is to be reserved at inference.

To achieve the maximum context length (~32K tokens) of the model, 16Gb of vRAM is required.

Reaching the defined max_context_length during a completion or chatting will result in the activation of vLLM's automatic sliding window handler, which drops quality of the final responses significantly.

[netlify]

✅ Deploy Preview for leapfrogai-docs canceled.

Name	Link
🔨 Latest commit	c9c2408
🔍 Latest deploy log	https://app.netlify.com/sites/leapfrogai-docs/deploys/66a90e7290146c0008f391e7

[netlify]

✅ Deploy Preview for leapfrogai-docs canceled.

Name	Link
🔨 Latest commit	d92b572
🔍 Latest deploy log	https://app.netlify.com/sites/leapfrogai-docs/deploys/66fea1d7f34a34000865ec5b

[justinthelaw]

For PR review, I recommend deploying / testing out different build arguments (see the Dockerfile), to include swapping out the model and default backend configurations, to make sure things work with the new vLLM engine.

[justinthelaw]

See this issue on CPU offloading: vllm-project/vllm#6952

The offloading works... but the actual inferencing runs into issues when certain parts of the code are looking for a model weight that isn't where it expects - specifically for Phi-3. That is why I haven't added this feature/parameter to this PR.

[justinthelaw]

Converting back to draft - some odd engine behavior / interaction with quantized Phi-3 RoPE scaling and engine crashing. Investigating...

[justinthelaw]

The LLM responds correctly; however, the engine runs into some blocking process that causes a timeout on subsequent responses when the conversation reaches 2-5 messages long.

See this issue upstream: vllm-project/vllm#5901

[justinthelaw]

https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard - for checking Hermes-2-Pro-Mistral-7b's current standings against other open source LLMs.

[jalling97]

Glad to see the process for updating models is simplified (both locally and in deployments) and the docs are clear as to how it works! Can confirm that vllm builds and deploys to a local cluster without issue. It's a shame about the issues with newer version of vLLM, there are some features that would be nice to utilize. Great stuff!