Dense Continuous-Time Optical Flow from Event Cameras

This is the official Pytorch implementation of the TPAMI 2024 paper Dense Continuous-Time Optical Flow from Event Cameras.

If you find this code useful, please cite us:

@Article{Gehrig2024pami,
  author        = {Mathias Gehrig and Manasi Muglikar and Davide Scaramuzza},
  title         = {Dense Continuous-Time Optical Flow from Event Cameras},
  journal       = {{IEEE} Trans. Pattern Anal. Mach. Intell. (T-PAMI)},
  year          = 2024
}

Conda Installation

We highly recommend to use Mambaforge to reduce the installation time.

conda create -y -n bflow python=3.11 pip
conda activate bflow
conda config --set channel_priority flexible

CUDA_VERSION=12.1

conda install -y h5py=3.10.0 blosc-hdf5-plugin=1.0.0 llvm-openmp=15.0.7 \
hydra-core=1.3.2 einops=0.7 tqdm numba \
pytorch=2.1.2 torchvision pytorch-cuda=$CUDA_VERSION \
-c pytorch -c nvidia -c conda-forge

python -m pip install pytorch-lightning==2.1.3 wandb==0.16.1 \
opencv-python==4.8.1.78 imageio==2.33.1 lpips==0.1.4 \
pandas==2.1.4 plotly==5.18.0 moviepy==1.0.3 tabulate==0.9.0 \
loguru==0.7.2 matplotlib==3.8.2 scikit-image==0.22.0 kaleido

Data

MultiFlow

	Train	Val
pre-processed dataset	download	download

DSEC

	Train	Test (input)
pre-processed dataset	download	download
crc32	c1b618fc	ffbacb7e

Checkpoints

MultiFlow

	Events only	Events + Images
pre-trained checkpoint	download	download
md5	61e102	2ce3aa

DSEC

	Events only	Events + Images
pre-trained checkpoint	download	download
md5	d17002	05770b

Training

MultiFlow

• Set DATA_DIR as the path to the MultiFlow dataset (parent of train and val dir)
• Set
  • MDL_CFG=E_I_LU5_BD10_lowpyramid to use both events and frames, or
  • MDL_CFG=E_LU5_BD10_lowpyramid to use only events
• Set LOG_ONLY_NUMBERS=true to avoid logging images (can require a lot of space). Set to false by default.

GPU_ID=0
python train.py model=raft-spline dataset=multiflow_regen dataset.path=${DATA_DIR} wandb.group_name=multiflow \
hardware.gpus=${GPU_ID} hardware.num_workers=6 +experiment/multiflow/raft_spline=${MLD_CFG} \
logging.only_numbers=${LOG_ONLY_NUMBERS}

DSEC

• Set DATA_DIR as the path to the DSEC dataset (parent of train and test dir)
• Set
  • MDL_CFG=E_I_LU4_BD2_lowpyramid to use both events and frames, or
  • MDL_CFG=E_LU4_BD2_lowpyramid to use only events
• Set LOG_ONLY_NUMBERS=true to avoid logging images (can require a lot of space). Set to false by default.

GPU_ID=0
python train.py model=raft-spline dataset=dsec dataset.path=${DATA_DIR} wandb.group_name=dsec \
hardware.gpus=${GPU_ID} hardware.num_workers=6 +experiment/dsec/raft_spline=${MLD_CFG} \
logging.only_numbers=${LOG_ONLY_NUMBERS}

Evaluation

MultiFlow

• Set DATA_DIR as the path to the MultiFlow dataset (parent of train and val dir)
• Set
  • MDL_CFG=E_I_LU5_BD10_lowpyramid to use both events and frames, or
  • MDL_CFG=E_LU5_BD10_lowpyramid to use only events
• Set CKPT to the path of the correct checkpoint

GPU_ID=0
python val.py model=raft-spline dataset=multiflow_regen dataset.path=${DATA_DIR} hardware.gpus=${GPU_ID} \
+experiment/multiflow/raft_spline=${MLD_CFG} checkpoint=${CKPT}

DSEC

work in progress

Code Acknowledgments

This project has used code from RAFT for parts of the model architecture.