Spatial information guided Convolution for Real-Time RGBD Semantic Segmentation

Lin-Zhuo Chen, Zheng Lin, Ziqin Wang, Yong-Liang Yang and Ming-Ming Cheng
⭐ Project Home »

---

The official repo of the TIP 2021 paper `` Spatial information guided Convolution for Real-Time RGBD Semantic Segmentation.

Results on NYUDv2 Dataset

Speed is related to the hardware spec (e.g. CPU, GPU, RAM, etc), so it is hard to make an equal comparison.

I get the following results under NVIDIA 1080TI GPU, Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz:

Model	mIoU(480x640)	mIoU(MS)	FPS(480x640)	FPS(425x560)
SGNet(Res50)	47.7%	48.6%	35	39
SGNet	49.8%	51.1%	26	28
SGNet_ASPP	50.2%	51.1%	24	26

If you want to measure speed on more advanced graphics card (such as 2080ti), you can use the environment of pytorch 0.4.1 CUDA 9.2 to measure inference speed.

Prerequisites

Environments

• PyTorch == 0.4.1
• tqdm
• CUDA==8.0
• CUDNN=7.1.4
• pillow
• numpy
• tensorboardX
• tqdm

Trained model and dataset

Download NYUDv2 dataset and trained model:

	Dataset	model	model	model
BaiduDrive(passwd: scon)	NYUDv2	SGNet_Res50	SGNet	SGNet_ASPP

Usage

1. Put the pretrained model into pretrained_weights folder and unzip the dataset into dataset folder.

2. To compile the InPlace-ABN and S-Conv operation, please run:

   ## compile InPlace-ABN 
   cd graphs/ops/libs
   sh build.sh
   python build.py
   ## compile S-Conv
   cd ..
   sh make.sh

3. Modify the config in configs/sgnet_nyud_test.json (mainly check "trained_model_path"). To test the model with imput size $480 \times 640$, please run:

   ## SGNet
   python main.py ./configs/sgnet_nyud_test.json
   
   ## SGNet_ASPP
   python main.py ./configs/sgnet_aspp_nyud_test.json
   
   ## SGNet_Res50
   python main.py ./configs/sgnet_res50_nyud_test.json

4. You can run the follow command to test the model inference speed, input the image size such as 480 x 640:

   ## SGNet
   python main.py ./configs/sgnet_nyud_fps.json
    
   ## SGNet_ASPP
   python main.py ./configs/sgnet_aspp_nyud_fps.json
   
   ## SGNet_Res50
   python main.py ./configs/sgnet_res50_nyud_fps.json

Citation

If you find this work is useful for your research, please cite our paper:

@article{21TIP-SGNet,   
  author={Lin-Zhuo Chen and Zheng Lin and Ziqin Wang and Yong-Liang Yang and Ming-Ming Cheng},   
  journal={IEEE Transactions on Image Processing},    
  title={Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation}, 
  year={2021},   
  volume={30},
  pages={2313-2324},  
  doi={10.1109/TIP.2021.3049332} 
}

Thanks to the Third Party Libs

inplace_abn

Pytorch-Deeplab

PyTorch-Encoding

PyTorch-segmentation-toolbox

Depth2HHA-python

Pytorch-Project-Template

Deformable-Convolution-V2-PyTorch

If you have any questions, feel free to contact me via linzhuochen🥳foxmail😲com