[BUG] fsspec version 2024.10.0 results in truncated S3 writes

[mhagel-theorem]

Describe the bug

Versions:
flytekit: 1.13.11
flyteidl: 1.13.5
flytecore chart: 1.12.0

When using inter-Flyte task file IO for both pickles and LightGBM binary datasets, fsspec version 2024.10.0 results in incomplete/truncated writes. It is expected that this would be an issue with all file-based IO. Downstream tasks fail as a result with a respective incomplete read error.

Reverting to fsspec version 2024.9.0 resolves all issues we have had with incomplete writes and Flyte inter-task IO. If this can be reproduced, recommend pinning Flyte's version to <=2024.9.0.

Expected behavior

File-based writing and IO does not fail to complete writing files.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

Edit: Added versions for reproducibility

[welcome]

Thank you for opening your first issue here! 🛠

[mhagel-theorem]

I am happy to investigate this a bit further/deeper when I have the time @eapolinario

[eapolinario]

@mhagel-theorem , thank you. If you can share more details about your setup (other package versions, the shape of your dataframe, or how you're actually using FlyteFile to read/write, which cloud provider, etc).

More than happy to collaborate on this one as I tried to repro, but couldn't yet.