Smooth Transitions from Original -> X iterations (video)

[gateway]

@rkhamilton I'm really trying to wrap my head around creating a nice smooth transition of video frames from the original image to my image thats generated by x iterations.

The variables I think that we have to play with and adjust are the following.

• video frame rate
• number of video frames
• iterations per frame
• iterations for first frame
• change prompt every x
• output frame rate encode ffmpeg
• output frame rate encode RIFE

My wish and goal is to create a smooth or as smooth as you can get video from the original source to my desired iterations see image below.

Ideally it would be great if I could slowly transform from the original image to the final over 5 or 10 seconds..

Do you see a way to do this, every test I try it seems like it goes pretty quick from the original to the final.

[rkhamilton]

Cool image! And I think the video you are trying for will look great.

Something to understand is that changing the init_image to the final image depends only on the optimization algorithm and the number of iterations (not video frames, but iterations). More iterations means more change in the output image. You have two ways to slow down the video of this process:

• Slow the optimization itself: change the config.learning_rate to something lower than the default of 0.1. That should make it take more iterations to get to the final solution. You could also try a different optimizer than Adam (RMSProp may be slower) but it will also likely change the look of the final output. I have only messed with lower learning_rate values once, and it did slow the rate of change like I’m saying, but I can’t promise it won’t change the look of the result as well.
• Make more video frames from the same number of iterations. You may be able to make the transition take more iterations by messing with learning_rate, but you will still end up with a limited number of frames for the transition. Let’s say it’s 50 iterations for the transition. You can stretch those iterations into more video frames by doing iterations_per_frame = 1 or 2. Then 50 iterations becomes 50 frames (so, number of frames = 50). Then encode that as a 10 fps video with ffmpeg and it will be (50 frames / 10 fps) 5 seconds long. Then use RIFE to convert 10 fps to 40 fps video (default) or 160 fps video (16 options).

[gateway]

Ill try both approaches and report back. Im not sure I 100% understand bullet point (behind the scenes magic) one outside of messing with the value of config.learning_rate and maybe a different optimizer.

I did test this..

eng_config = VQGAN_CLIP_Config()
num_video_frames = 200
iterations_per_frame = 5
eng_config.output_image_size = [300, 300]
text_prompts = 'Spirit Hawk with a third eye sacred geometry Zbrush Central'
image_prompts = []
noise_prompts = []
change_prompts_on_frame = [200],
init_image = 'animals/hawk2.jpg'
video_frames_path = f'example media{os.sep}custom video frames'
z_smoother = True
z_smoother_buffer_len = 3
z_smoother_alpha = 0.7
verbose = True
eng_config.adaptiveLR = True

So from what I gather, it will save a frame every 5 iterations over 200 frames or? So if I did iterations 1, ideally would that go up to frame 200 = iteration 200 which is close to my original image..

From the tests w/o the current tries on what you mentioned on the bullet points it looked wonky :)

[rkhamilton]

Heads up I was just trying this out myself, and I found a bug in my code that hits hard when iterations_per_frame is lower than about 5. I've just pushed v2.1.1 with the fix. It's not 100% what I want, but it works perfectly for non-zoom videos. As long as zoom_scale==1.0, and shift_x, and shift_y, are 0, you can freely set iterations_per_frame to 1 and get expected results. If you do use those parameters, iterations_per_frame needs to be higher than about 5 or you won't see the images change like you expect. I need to think about how to fix that... Point being, update your package before you try your video again and you should get very different, better results for low iterations_per_frame.

you got it right, that code will save a frame every 5 iterations for each of 200 frames, for 1000 total iterations.

I'll try again to explain my bullet points:
It takes 100 iterations to fully change hawk2.jpg into your result. That's just the way VQGAN works with the default setup that everybody uses, regardless of anything I coded in my repo. Above, your code is going to run 1000 total iterations, but the action is all over in the first 100 iterations. If you want VQGAN to take more iterations to change hawk2 into your final.jpg, there's not a lot of knobs I can offer. You can lower learning_rate, or you can try another optimizer. Another optimizer will definitely change the look of the result, and a lower learning rate might, I don't know. You maybe can't change it so that it takes more than 100 iterations to fully generate your result.

So, if you've only got 100 iterations available to you, you can try to make those into more frames of video. Lower iterations_per_frame is the easiest way to do that.