Release Candidate v0.0.1 #88
Conversation
* Default handling of MESH_DEVICE for Llama 3.x models
* Modified setup script improvements:
* Improved environment variable handling and persistence storage integration
* Added IMPL_ID field (set to "tt-metal" for all current models)
* Introduced MODEL_VERSION and MODEL_ID variables for better versioning
* Add image input support for image-text-to-text models in client scripts and tools
* Added support for image input in trace capturing
* Added new parameters for image width and height
* Implemented handling of both text-only and image+text trace captures
* Rename client side scripts batch_size options to max_concurrent to indicate client side concurrent request limits
* Fixed the vLLM model registration logic. Added missing ModelRegistry.register_model call for TTLlamaForCausalLM for legacy implementation models
* Updated benchmark path handling to use $HOME environment variable instead of hardcoded /home/user path
* Add benchmark summary support handling for vllm benchmark script, add documentation example
* Added support for a new model "DeepSeek-R1-Distill-Llama-70B" in the model setup configurations
tstescoTT
requested review from
ppetrovicTT,
milank94,
rpavlovicTT and
mvanniasingheTT
| --shm-size 32G \ | ||
| --publish 7000:7000 \ | ||
| ghcr.io/tenstorrent/tt-inference-server/tt-metal-llama3-70b-src-base-vllm:v0.0.1-tt-metal-v0.54.0-rc2-953161188c50 | ||
| ghcr.io/tenstorrent/tt-inference-server/vllm-llama3-src-dev-ubuntu-20.04-amd64:v0.0.1-47fb1a2fb6e0-2f33504bad49 |
There was a problem hiding this comment.
Is this commit of tt-metal (47fb1a2fb6e0) the latest supported, but missing a release tag? i.e. from the LLM support table, v0.55.0-rc12 is listed as the latest release.
There was a problem hiding this comment.
I'd recommend additional testing to use the tt-metal RC release tags. If RC tags are available when we want to start testing for the next drop we can use them from the jump. I believe the RC tags are nightly cuts from main so likelihood of breaking changes introduced is low. I agree we should transition to using tt-metal RCs and releases.
| "WH_ARCH_YAML": "wormhole_b0_80_arch_eth_dispatch.yaml", | ||
| } | ||
| env_var_map = { | ||
| "meta-llama/Llama-3.1-70B-Instruct": { |
There was a problem hiding this comment.
How come only the 70B models require this env_var_map?
There was a problem hiding this comment.
Only the legacy implementations need those additional environment variables currently.
change log