WIP kvs-watch: do not load all KVS blobs at once

[chu11]

Problem: With the FLUX_KVS_WATCH_APPEND and FLUX_KVS_STREAM flags, all content blobs in a valref treeobj will be retrieved from the content store at once. If the valref array is gigantic, this may be a very costly initial transaction. Not to mention, we probably wouldn't want to send an extremely large number (like millions) of content requests all at once.

Solution: Send content requests in more reasonable 32K chunks.

Fixes #6456

---

So this is what I initially started working on that lead more to #6531 and #6518.

This issue is mostly theoretical, as I don't have a use case where this is a problem at the moment. (i.e. perhaps it's a solution looking for a problem?) I want to try and manufacture a use case and test it, which is why I'm posting this as WIP. But I thought I'd post and see if folks think it's atleast a good idea.

I should note, I did run the test suite when having the "max loads" set to 2 instead of 32K, so it generally speaking appears to work safely. Although the question of how big the "maximum number of loads at a time" number could possibly be tweaked up or down some.

[chu11]

I tried an experiment where I created an eventlog with about 400K entries (all individually appended onto the log) to see if this helps or not.

before

>time flux kvs get test.bigdata > /dev/null                             
2.775u 0.699s 0:20.04 17.2%     0+0k 816+0io 4pf+0w                                                                                         
>time flux kvs get --stream test.bigdata > /dev/null                    
4.825u 2.510s 0:15.82 46.3%     0+0k 0+0io 0pf+0w          

after

>time flux kvs get test.bigdata > /dev/null                             
2.768u 0.757s 0:19.51 17.9%     0+0k 0+0io 0pf+0w                                                                                           
>time flux kvs get --stream test.bigdata > /dev/null                    
5.408u 3.985s 0:17.70 52.9%     0+0k 0+0io 0pf+0w                                                                                           

It doesn't seem like it really matters much.

If it doesn't seem to matter much at 400K entries, it probably just isn't worthwhile. Perhaps it would theoretically matter at like milllions of entries, but that just isn't going to be likely or common.

So I'll close and just keep a mental note that this work was done at one point in time.

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 8 lines in your changes missing coverage. Please review.

Project coverage is 83.65%. Comparing base (9278fa8) to head (8604a3d).
Report is 40 commits behind head on master.

Files with missing lines	Patch %	Lines
src/modules/kvs-watch/kvs-watch.c	83.33%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6532      +/-   ##
==========================================
+ Coverage   83.62%   83.65%   +0.02%     
==========================================
  Files         522      522              
  Lines       87809    87829      +20     
==========================================
+ Hits        73433    73473      +40     
+ Misses      14376    14356      -20     

Files with missing lines	Coverage Δ
src/modules/kvs-watch/kvs-watch.c	77.93% <83.33%> (-0.38%)	⬇️

... and 8 files with indirect coverage changes