Skip to content

Commit

Permalink
[fix](spill) MultiCastDataStreamer returns data with partial loss
Browse files Browse the repository at this point in the history
  • Loading branch information
mrhhsg committed Jan 18, 2025
1 parent 34414ee commit 59a657b
Showing 1 changed file with 9 additions and 1 deletion.
10 changes: 9 additions & 1 deletion be/src/pipeline/exec/multi_cast_data_streamer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,15 @@ Status MultiCastDataStreamer::pull(RuntimeState* state, int sender_idx, vectoriz
*block = std::move(_cached_blocks[sender_idx].front());
_cached_blocks[sender_idx].erase(_cached_blocks[sender_idx].begin());

*eos = _cached_blocks[sender_idx].empty() && _spill_readers[sender_idx].empty() && _eos;
/** Eos:
* 1. `_eos` is true means no more data will be added into queue.
* 2. `_cached_blocks[sender_idx]` blocks recovered from spill.
* 3. `_spill_readers[sender_idx].empty()` means there are no blocks on disk.
* 4. `_sender_pos_to_read[sender_idx] == _multi_cast_blocks.end()` means no more blocks in queue.
*/
*eos = _eos && _cached_blocks[sender_idx].empty() &&
_spill_readers[sender_idx].empty() &&
_sender_pos_to_read[sender_idx] == _multi_cast_blocks.end();
return Status::OK();
}

Expand Down

0 comments on commit 59a657b

Please sign in to comment.