Question: How to set up stream_pipeline_online for inference?

[nguyenphuvinhtoan]

Description

I'm trying to implement real-time inference using stream_pipeline_online, but I'm unclear about the proper setup process. I've successfully run the offline pipeline using stream_pipeline_offline, but need guidance on the online version.

Current Setup

• I have successfully implemented stream_pipeline_offline for inference
• Looking to transition to real-time processing with stream_pipeline_online

Questions

1. Are there any additional dependencies or configurations needed specifically for online streaming?
2. Is there a basic example showing how to initialize and run stream_pipeline_online?

Additional Context

If there's any documentation or examples specifically for stream_pipeline_online, please point me in the right direction. Thank you!

[digital-avatar]

@nguyenphuvinhtoan
You can simply modify this line to from stream_pipeline_online import StreamSDK and then inference with --cfg_pkl "./checkpoints/ditto_cfg/v0.4_hubert_cfg_trt_online.pkl".

[nguyenphuvinhtoan]

@digital-avatar
Oh, I see! When I changed it to from stream_pipeline_online import StreamSDK and used the argument --cfg_pkl "./checkpoints/ditto_cfg/v0.4_hubert_cfg_trt_online.pkl" for inference, the processing seems to finish faster. However, the resulting video doesn't have proper lip synchronization.

I also tried implementing a server to handle the streaming response, but I couldn't get it to work successfully.

Could you suggest some solutions to manage the streaming response results effectively?

Thank you very much!

[Neonisu]

I also curious about this. I think need to modify audio2motion_worker in stream_pipeline_online.py for streaming silent and real audio frames immediately and then setup api to push audio to SDK?

[digital-avatar]

@nguyenphuvinhtoan
Can you show a example of incorrect lip sync? Also, do you test all online results with incorrect lip sync?