Impact of batch size on training performance

[XingtongGe]

Hi, I would like to know how much the batch size affects training performance. For example, is there a significant performance gap between training DMD2-SDXL with 8 GPUs (total batch size of 16) and training with 64 GPUs (total batch size of 128)? Thanks!

[tzhu01]

I followed his ReadMe and trained using an 8 GPUs (A100, 80G), and the results were almost identical to the data in the paper.