Into_Film--Pix2Pix-and-SinGAN

Using pix2pix and SinGAN to get into the movie

Pix2Pix

• This project is inspired by the paper, which aims to investigate how to build and train a conditional generative adversarial network (cGAN) called pix2pix, which learns image translation training from input images to output images.

• pix2pix can be used for a wide range of tasks, including compositing photos from tagged maps, generating color photos from black and white images, converting Google Maps photos to satellite aerial images, or converting sketches to photos.

SinGAN

• The second part of this project uses SinGAN for image fusion. The authors of this paper are from Technion and Google.

• In general, training GANs to generate images requires at least thousands of training data to have good results. SinGAN can train a generative model with only a single natural image.

Purpose

Situational application :

• Get into the movies of the past

  Recreate black and white classic movie footage into color training with Pix2Pix.

• Get into the movies of the future

  Incorporating my pre-taken photos into the movie training with SinGAN.

Training Process

Step 1: Pix2Pix training process

1. Model Training

   For the model trained by Pix2Pix, the image is mainly brown at the beginning. That is, the mixture of the three primary colors (red, yellow, and blue) in the painting becomes brown.

   After 45 epochs, it can be seen that the image does not change much. During the training and learning process, some pictures are obviously blurred.

   

2. Model Testing

   

Step 2: SinGAN training process

1. Model Training

   For the model trained by SinGAN, you need to put a picture of Training, Input and Mask before use.

   To generate from any starting random samples first run RandomSamples :

     python random_samples.py --input_name [training_image] --mode random_samples --gen_start_scale [start scale]
   

   It will output several RandomSamples images, then run Harmonization and fill in the scale value

     python harmonization.py --input_name [training_image] --ref_name [pasted_reference_image] --harmonization_start_scale [scale]
   

   

2. Output

   

Conclusion

At present, the experimental result is to make an image, and there are also attempts to make the training process into a gif file. In the future, it may be possible to capture every frame of the movie clip to form an animation file.

I think that in the development and imagination of AI, adding real portraits to movies through AI model training can generate more imagination and interesting reactions for discussion.

Reference

Pix2Pix: Image-to-image translation with a conditional GAN--Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros

SinGAN: Learning a Generative Model from a Single Natural Image--Tamar Rott Shaham, Tali Dekel, Tomer Michaeli