Electriclizard triton solution

[electriclizard]

Hi @rsolovev, i've made a new solution, based on nvidia-triton, metrics raise x2 on local tests, can you please explain me how can i correctly deploy an application using docker-compose? Now it starts with docker-compose up

[electriclizard]

Hi @rsolovev, finally i've decided to avoid from docker-compose back to a single dockerfile, now it builds in ci and metrics raises again, hope to see a test a results on your environment
now service starts with docker run --gpus 0 --shm-size 16G -p 8080:8080 image-name --shm-size is recuired for tritonserver

local test results

[rsolovev]

hey @electriclizard, thank you for this iteration, unfortunately this didnt work out:

shm-size is not configurable in k8s [1]. There are some workarounds with mounting an empty volume, but in k8s 1.24 they removed dockershim support completely [2] so this workarounds kinda go against our infrastructure "standards" (not that it is impossible to configure, just "feels unstable" in this form). Also image size is quite large (25GB) -- it failed to build on github runner and I had to build it locally.

I'm sure that with more time we could make this approach feasible and stable, but right now it is inconvenient to follow

[electriclizard]

but right now it is inconvenient to follow

hey @rsolovev, thank you for a try anyway, yes image is quite big, thats true. I just wanted to test it, because i am sure and local tests confirm my theory, that it might be impossible to run models on python with the same productivity

[electriclizard]

Hi @rsolovev, i can decrease image size to fix bulding on a github runner, but i cannot change infrastructure, may be can i try to add these lines to helm?

volumes:
        - name: dshm
          emptyDir:
            medium: Memory

Because i've checked it on 1.24 api k8s version and it works

[rsolovev]

Because i've checked it on 1.24 api k8s version and it works

that sounds great, lets try reducing the image size first, and then I'll tweak the volume confs backstage

[electriclizard]

Because i've checked it on 1.24 api k8s version and it works

that sounds great, lets try reducing the image size first, and then I'll tweak the volume confs backstage

great, i'll push the commit later

[electriclizard]

hey @rsolovev i decreased the image size to 20.1gb, by pulling models before the app start, not while building the image, it takes a few minutes before start now

[rsolovev]

@electriclizard impressive! I take my words back, no issues with dshm volume from the first try. An absolute peak rps and total requests record (121 rps, 26850 total) -- full stats

p.s added shm mount as a reference

[electriclizard]

@electriclizard impressive! I take my words back, no issues with dshm volume from the first try. An absolute peak rps and total requests record (121 rps, 26850 total) -- full stats

That's great!!! Thank you for a run, it was my latest commit to a challenge))