Video pre-processing pipeline?

[hkunzhe]

Hi, great work! I can't find a specific data collection and pre-processing pipeline. Could you elaborate on video sources and text annotations?

[nankepan]

Thank you for your attention. Please refer to the data collection and preprocessing procedures in our paper.

[hkunzhe]

Hi @nankepan, Could you mind sharing the data-processing scripts?

[hkunzhe]

@nankepan In the third section of the paper, why is computing the semantic similarity between adjacent frames to differentiate between static and flicker videos, instead of directly compute the motion score?

Consider the following two scenarios: 1) A person speaking to the camera with slight bodily movements, and 2) camera movements such as dolly or pan shots. In the former case, since the background remains stationary, the similarity of CLIP features should also be relatively high. Conversely, in the latter scenario, due to the camera's motion, the semantic similarity between sampled adjacent frames might actually be quite lower. However, neither of these belong to the static or flicker videos.

Could you provide a few examples to illustrate the necessity of this operation?

[moonbow721]

Hi @nankepan, I also have a question about data preprocessing. How do you annotate the camera motion classes such as static, pan right, zoom in, zoom in + tilt down, etc.? It would be more helpful if you could release the code or detailed tutorial.