The speed of inference

[Baiyajing]

I wonder What's the approximate speed of the inference. I use a H100 gpu and inferencing a video costs about half an hour. Is there any way to increase the speed?

[kirikorneev]

Hello, we tried to solve the issue.

This is what we did:

To improve inference speed on H100 GPU, we'll make the following changes:

1. Use torch.compile() to optimize the model
2. Implement mixed precision training
3. Increase batch size for parallel processing
4. Optimize data loading and preprocessing
5. Add an option for lower precision inference

You can review changes in this commit: clarin-ebtio800090@5cc759c.

Caution

Disclaimer: The concept of solution was created by AI and you should never copy paste this code before you check the correctness of generated code. Solution might not be complete, you should use this code as an inspiration only.

Latta AI seeks to solve problems in open source projects as part of its mission to support developers around the world. Learn more about our mission at https://latta.ai/ourmission . If you no longer want Latta AI to attempt solving issues on your repository, you can block this account.

Is this code going to be merged into the main repository?