Multi-Spectral Image Stitching via Global-Aware Quadrature Pyramid Regression

Zhiying Jiang, Zengxi Zhang, Jinyuan Liu, Xin Fan, Risheng Liu* ,“Multi-Spectral Image Stitching via Global-Aware Quadrature Pyramid Regression”, IEEE Transactions on Image Processing (TIP), 2024. [paper]

Overview

Updates

[2024-08-01] The testing code is available
[2024-10-26] The training code in pytorch version is available

Download

Download the pre-trained model and put it in 1_ImageAlignment_Pytorch/snapshot

• Google Drive
• Baidu Yun

Download the pre-trained model and put it in 2_ImageReconstruction_Pytorch/snapshot

• Google Drive
• Baidu Yun

(optional) Download the pre-trained model and put it in 2_Reconstruction_Tensorflow/snapshot

• Google Drive
• Baidu Yun

Download the pre-trained model and put it in 3_ImageFusion_Pytorch/snapshot

• Google Drive
• Baidu Yun

Requirements

Pytorch==1.12.1 Tensorflow==2.10.1

Test

Step 1: Generate aligned images

cd 1_ImageAlignment_Pytorch
python test.py

The generated images are used to the subsequent reconstruction.

Step 2: Generate stitched images

Tips: The original model was trained on tensorflow. For better use, we reproduced the whole model in Pytorch, but the image reconstruction effect deviated from the source code. Therefore, in addition to the pytorch version, we also open-sourced the source code for image reconstruction in Tensorflow. If you don't have tensorflow installed, you can also employ the Pytorch version in the experimental comparison.

 Option 1: Generate stitched images on Pytorch

cd ../2_ImageReconstruction_Pytorch
python test.py

 Option 2: Generate stitched images on Tensorflow

cd ../2_ImageReconstruction_Tensorflow
python test.py

Step 3: Fuse stitched images

cd ../3_ImageFusion_Pytorch
python test.py

Train

Step 1: Generate aligned images

cd 1_ImageAlignment_Pytorch
python train_H.py

Step 2: Generate stitched images

cd ../2_ImageReconstruction_Pytorch
python train.py

Step 3: Fuse stitched images

cd ../3_ImageFusion_Pytorch
python train.py

Dataset (IVSD dataset)

• Google Drive
• Baidu Yun

Citation

@ARTICLE{2024jianggqpr,
  author={Jiang, Zhiying and Zhang, Zengxi and Liu, Jinyuan and Fan, Xin and Liu, Risheng},
  journal={IEEE Transactions on Image Processing}, 
  title={Multispectral Image Stitching via Global-Aware Quadrature Pyramid Regression}, 
  year={2024},
  volume={},
  number={},
  pages={1-1},
  keywords={Image stitching;Estimation;Thermal sensors;Deformation;Optical sensors;Image reconstruction;Accuracy;Multi-Spectral image stitching;image fusion;infrared and visible images;pyramid structure;deep learning},
  doi={10.1109/TIP.2024.3430532}}

Contact

If you have any question or suggestion, please email to Zengxi Zhang or Zhiying Jiang.