Patch size during training vs. during inference

[dianemarquette]

Hi!

I understand that using patches of 3D volumes instead of the entire scans is required to fit the data and model in the GPU memory. However, I noticed that in the spleen 3D segmentation tutorial, the neural network was trained with (96 x 96 x 96) cuboids, but, during inference, the roi_size of the sliding window inference used (160 x 160 x 160) cuboids.

My questions

• Does using different sizes of patches during training and inference have an impact on the model performance? (Intuitively, I was thinking it might be confusing for the model to be given more context during inference than what it was used to during training)
• If I increase the resolution of my rescaling transform (i.e., lower the target pixel size), should I increase the size of the patch of the RandCropByPosNegLabeld transform so that the model always have the same amount of context of the original scans during training?

Thanks in advance for your clarifications.

[Nic-Ma]

Hi @dongyang0122 @wyli ,

Could you please help share some explanation in theory?

Thanks in advance.

[dongyang0122]

hi @dianemarquette

1. Yes, it has impacts over model performance (at least for 3D U-Net). Larger sliding-window size generally has more robust segmentation performance, and it is able to remove more artifacts (at window boundary) introduced by sliding-window inference;
2. Yes, in order to let model see the similar global contexts of images, you may need to further increase cropping size when increasing re-scaling spacing/resolution.

[dianemarquette]

Hi @dongyang0122

Thanks for the quick reply! Regarding point 1), I understand that larger windows make sliding window inference more robust. However, my question is whether using smaller patches for training (with RandCropByPosNegLabeld) compared to the window size used during inference penalize the network performance. Indeed, we train the model with small patches (96 x 96 x 96), but we evaluate with larger windows (160 x 160 x 160). What's your take on this point?

[dongyang0122]

Larger window size normally would improve model performance by a small margin (especially for U-Net). And smaller patch size would reduce GPU memory consumption and speed-up training in some way. It is task-dependent. Some tasks may require more global context (when boundary area is vague), and then it would be necessary to keep a large cropping size.

[dianemarquette]

@dongyang0122 Ok, thanks for your answer. Is there any rule of thumb regarding the ratio to have between the patches used for training and inference? In the spleen 3D segmentation tutorial, the patches used during inference (160x160x160) are 1,67 times larger than the ones used during training (96x96x96).

[dongyang0122]

It is task-dependent. One helpful guideline would be to use the same patch size or slightly larger patch size during validation/inference, and make sure the patch size is multiple of 16/32 for U-Net. Normally slightly larger patch size would further improve model performance, but the size cannot be too large. The performance would be degraded when the patch size increasing reaches certain points.

[perlmutter]

Hi,

I was also confused by the same thing as @dianemarquette and want to ask a few followup questions on this topic. I can't tell if @dongyang0122 recommendations are referring to training or inference. I understand that when choosing a U-Net patch size during training there is a tradeoff between contextual information (which influences accuracy) and memory consumption. However after that choice has been made and the model has been trained, during inference,

1. How is it even possible to pass a (160x160x160) tensor through a network trained on tensors of size (96x96x96)? Is this a general property of fully convolutional networks that they can accept images of variable size? Or is there some hidden cropping going on in sliding_window_inference?
2. How specifically is the output of sliding_window_inference affected by the choice of roi_size? Why did the tutorial choose to make roi_size different from the spatial_size in RandCropByPosNegLabeld?
3. Can a model trained with data at one resolution (eg. 1mm^3 / voxel) be expected to perform inference well on data at different resolution (eg 0.5mm^3 / voxel), even if the extent is doubled to match?

Thank you

[Nic-Ma]

Hi @dongyang0122 ,

Could you please help share more details?

Thanks in advance.

[dongyang0122]

1. You are right. Fully convolutional networks naturally can accept input with different size (or length as m * 2^n);
2. The overlapping ratio in sliding window inference is fixed. When roi size become larger, and number of sliding windows would be fewer; and overlapped regions of windows become larger;
3. It would work fine for image with close spacing. But it would probably not work for images with larger gaps in imaging spacing (unless you add a very aggressive random zooming as data augmentation).

[perlmutter]

Thanks @dongyang0122. So can you help me understand why the effects of roi_size you mentioned in point 2 -- fewer but bigger sliding windows with larger overlapped regions -- could "improve model performance by a small margin" during inference, as you explained to @dianemarquette above? By performance I assume you mean segmentation accuracy, not computation speed, correct? Why is the inference accuracy not fully determined by how the network is trained?