Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

auto-inject add openai compatible tags #1435

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

michaelfeil
Copy link
Contributor

@michaelfeil michaelfeil commented Mar 7, 2025

🚀 What

This will happen on every push

  • client side
truss-py3.9root@michaelfeil-dev-pod-h100-0:~/truss_upstream/tmp_email/spec-dec-qwen-lade# truss push --publish
? 🎮 Which remote do you want to connect to? baseten

TRT-LLM models should have model_meta_data/tags section with either ['openai-compatible'] or ['force-legacy-api-non-openai-compatible']. 
``yaml
model_metadata: 
tags:
- openai-compatible
``
Your current tags are `[]`.
As temporary measure, we are injecting the tags=['force-legacy-api-non-openai-compatible']
We strongly recommend migrating to model_metadata['tags']=['openai-compatible'] for openai compatibility."

✨ Model Qwen2.5-0.5B-lade fp8 2 was successfully pushed ✨
🪵  View logs for your deployment at https://app.baseten.co/models/4w59kgpq/logs/q81v68w
  • requires also a briton side change (if no tag set, use default)

💻 How

🔬 Testing

@michaelfeil michaelfeil changed the title add openai compatible tags auto-inject add openai compatible tags Mar 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant