Behavior of SendRef can surprise the user

[sgasse]

Hi! In #70 , I commented our workaround to avoiding empty sends after already taking out a SendRef. We used to store the SendRef in an Option for the next loop iteration. This workaround came back to bite us. The behavior of SendRef is correct for all I understand but it can still be surprising.

Let's look at the following situation on a high-level:

• Thread 0 creates a SendRef and holds on to it, because its current loop iteration did not yield any data to send.
• Thread 1 sends something with a regular Sender::send.
• Thread 0 finally finds some data to send 10 seconds later and uses the SendRef (dropping it) that it had stored.

Naively, we might expect that the following is observable on the receiver side:

• The item from thread 1 is received immediately.
• 10 seconds later, the item from thread 0 is received.

Unfortunately, that is not what is happening. Nothing will be received until 10 seconds pass, and then both items are received in succession, but the item from thread 0 first (although it was "sent by dropping the slot" second). With the inner workings of thingbuf, this makes sense since the pointer to the next element to receive cannot advance while the first SendRef is still held.

Though I wonder if it might make sense to mention this in the documentation, so that less people for it. Especially with several tasks / threads, this can lead to ugly situations if someone holds on to a SendRef and thereby inadvertently stalls the receiver.

Thanks to @flxo for helping me in debugging 🙏

[sgasse]

An example can be found in the draft PR #89

[Dr-Emann]

I personally did expect exactly the actual behavior: I think of creating a sendref as "reserving my spot" in the queue, but I do think it's definitely a good idea to make it clearer in the docs.