Flooding GRPC errors when downloading models

[austinbv]

When trying to download models the downloads will start but I am flooded with

et (en0))'], ai-mac-5: ['ai-mac-4(Ethernet (en0))', 'ai-mac-3(Ethernet (en0))'], ai-mac-4: ['ai-mac-5(Ethernet (en0))', 'ai-mac-3(Ethernet (en0))']})
Error sending opaque status to ai-mac-4: <AioRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Received RST_STREAM with error code 7"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Received RST_STREAM with error code 7", grpc_status:14, created_time:"2025-01-29T15:55:34.05777-07:00"}"
>
Traceback (most recent call last):
  File "/Users/exo/workspace/exo/.venv/lib/python3.12/site-packages/exo/orchestration/node.py", line 606, in send_status_to_peer
    await asyncio.wait_for(peer.send_opaque_status(request_id, status), timeout=15.0)
  File "/Users/exo/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/Users/exo/workspace/exo/.venv/lib/python3.12/site-packages/exo/networking/grpc/grpc_peer_handle.py", line 197, in send_opaque_status
    await self.stub.SendOpaqueStatus(request)
  File "/Users/exo/workspace/exo/.venv/lib/python3.12/site-packages/grpc/aio/_call.py", line 327, in __await__
    raise _create_rpc_error(
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Received RST_STREAM with error code 7"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Received RST_STREAM with error code 7", grpc_status:14, created_time:"2025-01-29T15:55:34.05777-07:00"}"

[AlexCheema]

Looks like one node lost connection. Did it go away after a few seconds?

[austinbv]

It shouldn't have they are all thunderbolt and going though a 1gb switch.

[austinbv]

Following up. I don't think that's it. it floods thousands of these errors pretty consistently when downloading models. Tiny chat also freezes and downloads stall. Restarting exo solves for a bit but happens again on any large download

[cnsren]

I have exact same problems with larger model download, waiting for the resolution.

[austinbv]

I have exact same problems with larger model download, waiting for the resolution.

How are you connecting?

[AlexCheema]

Does the download complete successfully?
It looks like something related to high network load

[austinbv]

It does not. You need to restart the Exo process and restart the downloads

[dakecrazy]

me too

[dakecrazy]

I use thunderbolt 5