Download of Zip File Parts Is Slow

[chancharikmitra]

Hello! Thanks for contributing such a high-quality dataset! Very excited to use it. I have noticed that downloading the .zip files via wget is very fast, but for some reason the partaa type files don't download as quickly using wget. I don't know why this happens exactly. But I was wondering if those few files could be combined and included in the HF repo as .zip files for the sake of convenience.

Thanks!

[nankepan]

Sorry about that. We split some large files (> 50GB) into multiple smaller files, as the limit for a single HF file is 50GB.