The saved reference frames are not the original frames

[Yumin-Sun-00]

** The generated video's reference frames are not the same to the original one's.

As I noticed, the original reference frames have, in my case, size 1280, 720. The saved reference frames are first resized to 1280,704, and than later resize back to 1280, 720. This makes the saved reference frames not identical to the original frames anymore.

And this will have artifacts on the interpolated frames.

The easiest solution, I considered, is to first patch the video width and height to some multiple of 32, to avoid the resize, and cut the patch after interpolation.

** The reverse normalization does not give the same result.
Another issue is that, the reverse normalization TP, do not give the same image.

[avinashpaliwal]

Thanks for pointing this out. I will add the padding logic shortly.

The reverse normalization does not give the same result.

Can you elaborate more on this issue? And also specify your setup.

[Yumin-Sun-00]

Thanks for pointing this out. I will add the padding logic shortly.

The reverse normalization does not give the same result.

Can you elaborate more on this issue? And also specify your setup.

So, if you remove the resize and pre-process the frames to proper size no matter you crop them or pad them, the normalization and reverse normalization will leave some artifacts on the reference frames. I suggest that when it is the reference frame you just “copy“ them into the temp output folder with the same size as interpolated frames

[avinashpaliwal]

I don't understand why the normalization would cause artifacts. In the implementation, it's just subtraction and addition of a constant from the image. Well anyway I can change the implementation to copy the original image, but if you have artifacts on the interpolated frame due to normalization then it would still be very noticeable.

[Yumin-Sun-00]

I don't understand why the normalization would cause artifacts. In the implementation, it's just subtraction and addition of a constant from the image. Well anyway I can change the implementation to copy the original image, but if you have artifacts on the interpolated frame due to normalization then it would still be very noticeable.

you know the color is stored as 0-255 integers, and when you do normalization and reverse, it may cause some small difference. Maybe you can test more about this.

[avinashpaliwal]

you know the color is stored as 0-255 integers, and when you do normalization and reverse, it may cause some small difference. Maybe you can test more about this.

The dataloader in this code stores pixel intensities as float in range 0-1. I'll try to make some modifications. Personally, I have not seen any artifacts on my system (CPU and GPU).

[Thomasedv]

I'm not entirely sure if it's the case in this given situation, but i've had some experience with ffmpeg, when i extract frames to png for example, they also have some other metadata which sets something gamma i think, which changes the look of the image. PIL however doesn't care and while it loads that metadata, it doesn't resave it. Which causes those images opened and resaved to be slightly different due to lack of gamma, despite the RGB values being the exact same for each image. I initially noticed this myself making something similar as superslomo, where instead of loading reference images, i just copied them to the output folder with the correct name/index. Opening the images in a image viewer and stepping through the folder the change between interpolated images and original ones really shows.

I do however note, ffmpeg doesn't care about it when creating an output video, so no flickering happens, it either keep or just strips the gamma values from the images, so it's not a big deal in practice, assuming it's the same bug.

I also think padding images is smarter than cropping, and can be done with PIL quite easily, and added to the Transforms for normalization.