Video Interpolation using Optical Flow

Intermediate frame interpolation using optical flow with FlowNet2
Explore the repository»

View Problem Statement View Report

tags : frame interpolation, optical flow, lucas kande, multiscale, horn schunk, digital video, deep learning, pytorch

About The Project

This project deals with the task of video frame interpolation with estimated optical flow. In particular, we estimate the forward optical flow (flow from Frame N to Frame N + 2) and the backward flow (flow from Frame N + 2 to Frame N) and use both of them to estimate the intermediate Frame N. To estimate the optical flow we three popular methods — Lucas-Kanade algorithm, Multiscale Lucas-Kanade algorithm (with iterative tuning) and Discrete Horn-Schunk algorithm. We explore the interpolation performance on Sphere dataset and Corridor dataset. We observe that the quality of interpolated frames is comparable to original with Sphere datasets and is poor for Corridor dataset. A detailed description of interpolation algorithms and analysis of the results are available in the Report.

Built With

This project was built with

• python v3.8.5
• The environment used for developing this project is available at environment.yml.

Getting Started

Clone the repository into a local machine using

git clone https://github.com/vineeths96/Video-Interpolation-using-Optical-Flow

Prerequisites

Create a new conda environment and install all the libraries by running the following command

conda env create -f environment.yml

The dataset used in this project is already available in this repository. To test on other datasets, download them and put them in the input/ folder.

Instructions to run

We explore three popular methods — Lucas-Kanade algorithm, Multiscale Lucas-Kanade algorithm (with iterative tuning) and Discrete Horn-Schunk algorithm.

Lucas-Kanade Optical Flow Algorithm

To interpolate the frame with the Lucas-Kanade algorithm, run the following command. This will interpolate the intermediate frames and store it in this folder.

python lucas_kanade_interpolation.py

Multiscale Lucas-Kanade Optical Flow Algorithm

To interpolate the frame with the Multiscale Lucas-Kanade algorithm, run the following command. This will interpolate the intermediate frames and store it in this folder.

python multiscale_lucas_kanade_interpolation.py

Discrete Horn-Schunck Optical Flow Algorithm

To interpolate the frame with the Discrete Horn-Schunck algorithm, run the following command. This will interpolate the intermediate frames and store it in this folder.

python horn_schunck_interpolation.py

Results

Note that the GIFs below might not be in sync depending on the network quality. Clone the repository to your local machine and open them locally to see them in sync.

A detailed description of algorithms and analysis of the results are available in the Report.

The plots below shows the estimated optical flow for the datasets with the Lucas-Kanade method and the Horn-Schunck method. We can see that there are significant change in the estimated optical flow between the two methods. This is because Lucas-Kanade method imposes local smoothness constraint and Horn-Schunck method imposes global smoothness constraints.

Corridor Dataset	Lucas-Kanade Optical Flow	Horn-Schunck Optical Flow

Sphere Dataset	Lucas-Kanade Optical Flow	Horn-Schunck Optical Flow

The plots below shows the interpolated frames for the datasets with the Lucas-Kanade method, Multiscale Lucas-Kanade method, and Horn-Schunck method. We can see that there is the quality of interpolated frames for the Sphere dataset is comparable to the original whereas that of Corridor dataset is quite poor.

Corridor Dataset Ground Truth	Lucas-Kanade Interpolated Frame	Multi-Scale Lucas-Kanade Interpolated Frame	Horn-Schunck Interpolated Frame

Sphere Dataset Ground Truth	Lucas-Kanade Interpolated Frame	Multi-Scale Lucas-Kanade Interpolated Frame	Horn-Schunck Interpolated Frame

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Vineeth S - [email protected]

Project Link: https://github.com/vineeths96/Video-Interpolation-using-Optical-Flow

Acknowledgements

• NVIDIA FlowNet2

  Fitsum Reda et al. flownet2-pytorch: Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. https://github.com/NVIDIA/flownet2-pytorch . 2017.