This repository was archived by the owner on Mar 8, 2025. It is now read-only.

Deployment blueprints #294

Open

grahamking opened this issue Feb 27, 2025 · 0 comments

Labels

Contributor

grahamking commented Feb 27, 2025

(brainstorm / notes) We should include advice on how to setup an inference service with triton distributed.

Questions that would inform the advice:

Is the HTTP -> backend traffic encrypted within the data center (HTTP vs HTTPS) ?
If they have a DPU should it be in DPU mode or NIC mode?
Is there a load balancer in front of the HTTP node, and is that a terminating proxy?
What is the HTTP -> backend network - Infiniband, RoCE, TCP, etc.

Lots more thinking here, but a placeholder for later.

grahamking added the documentation label

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.