[Feature Request] Allow tuning of read size requests and Direct I/O

[rexbron]

Trying out fclones on a ZFS data set that has large record sizes (16M) and large video files (100GB+ MXFs 40MB DPX Image sequences), fclones makes 2M read requests.

Ideally, it would request larger amounts of data at a time and direct_IO as the data will not be used again after scanning, so no need to cache it in ZFS's ARC.

The array is capable of 2.7GB/s read during a ZFS scrub, but around 1.6GB/s in usage with fclones.

Any thought or suggestions?

zpool iostat -r 2

mach2tank     sync_read    sync_write    async_read    async_write      scrub         trim         rebuild  
req_size      ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
512             0      0      0      0      0      0      0      0      0      0      0      0      0      0
1K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16K             0      0      0      0      0      0      0      0      0      0      0      0      0      0
32K             0      0      0      0      0      0      0      0      0      0      0      0      0      0
64K             0      0      0      0      0      0      0      0      0      0      0      0      0      0
128K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
256K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
512K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
1M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2M              0      0      0      0    408      0      0      0      0      0      0      0      0      0
4M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16M             0      0      0      0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------

mach2tank     sync_read    sync_write    async_read    async_write      scrub         trim         rebuild  
req_size      ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
512             0      0      0      0      0      0      0      0      0      0      0      0      0      0
1K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16K             0      0      0      0      0      0      0      0      0      0      0      0      0      0
32K             0      0      0      0      0      0      0      0      0      0      0      0      0      0
64K             0      0      0      0      0      0      0      0      0      0      0      0      0      0
128K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
256K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
512K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
1M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2M              0      0      0      0    416      0      0      0      0      0      0      0      0      0
4M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16M             0      0      0      0      0      0      0      0      0      0      0      0      0      0
------------------------------------------------------------------------------------------------------------