Large quantities of stderr from exec() not always fully captured

[craigwalton-dsit]

The following test is a bit savage, but reliably reproduces the issue.

async def test_exec_reliability_stderr(
    sandbox_non_root: K8sSandboxEnvironment,
) -> None:
    head_limit = 1024 * 100  # 100 KiB
    cmd = ["sh", "-c", f"yes | head -c {head_limit} 1>&2"]

    for _ in range(10):
        awaitables = [sandbox_non_root.exec(cmd) for _ in range(100)]

        for result in await asyncio.gather(*awaitables):
            assert result.success
            assert len(result.stderr) == head_limit

Note the use of sandbox_non_root which is ubuntu without gVisor. This seems to repro the issue much more readily than the default sandbox with gVisor.

Hypothesis: WSClient is closed by us when we read the sentinel value, but not all of stderr has been sent over the websocket. It seemed that adding the sync command to the sh script improved this, but didn't make it 100% reliable.

When we don't manually close the websocket upon reading the sentinel value, this test passes.