load_dataset from s3 file system through streaming can't not iterate data

[janineguo]

Describe the bug

I have a JSON file in my s3 file system(minio), I can use load_dataset to get the file link, but I can't iterate it

we can change 4 lines to fix this bug, you can check whether it is ok for us.

Steps to reproduce the bug

1. storage a file in you s3 file system
2. use load_dataset to read it through streaming
3. iterate it

Expected behavior

can iterate it successfully

Environment info

• datasets version: 2.12.0
• Platform: macOS-10.16-x86_64-i386-64bit
• Python version: 3.8.16
• Huggingface_hub version: 0.14.1
• PyArrow version: 12.0.0
• Pandas version: 2.0.1

[lhoestq]

This sounds related to #5281.

Can you try passing storage_options=s3_client.storage_options instead passing it to use_auth_token= ?

[janineguo]

I tried storage_options before, but it doesn't work, I checked our source code and I found that we even didn't pass this parameter to the following process. if I use storage_options instead of use_auth_token, then I also need to change another place of the code. the last line of streaming_download_manager.py. our code only passes the use_auth_token to the following handler, but does nothing to the storage_options

[lhoestq]

Cloud storage support is still experimental indeed and you can expect some bugs.

I think we need to pass the storage options anywhere use_auth_token is passed in indeed. Let me know if you'd be interested in contributing a fix !

[janineguo]

Oh, that's great, I really like to fix it. because datasets is really useful and most of our projects need to use it, but we can store our data on the internet due to security reasons. fix it not only make our own work more efficient but also can benefit others who use it.