[Zulip] Missing ingest jobs

[junhaoliao]

Bug

Reporter - Nov 21, 2024

Hello,

Do you know why ingest jobs would be missing from the history? I can confirm they ran as I logged their outputs when the compression happened. And I think they're factoring into the after_compression size:
image.png

Ex: {"filename": "xyz_file", "compression_time": 1731985202, "job_id": 900, "pre_compression_size": 4975406174, "post_compression_size": 154088243, "compression_ratio": 32.29, "success_type": "Complete"}

Reporter - Dec 26, 2024

I wanted to raise this issue again as I've continued to see the issue in 0.2.1. Here are stats I've automatically tracked from the job logs:

{"filename": "example-filebeat_2024-12-26_09-55.log", "compression_start_time": 1735206963, "compression_end_time": 1735207732, "job_id": 191, "pre_compression_size": 32429607578, "post_compression_size": 881873387, "compression_ratio": 36.77, "success_type": "Complete"}
{"filename": "example-metricbeat_2024-12-26_10-10.log", "compression_start_time": 1735207861, "compression_end_time": 1735208652, "job_id": 192, "pre_compression_size": 32407121476, "post_compression_size": 984518492, "compression_ratio": 32.92, "success_type": "Complete"}
{"filename": "example-filebeat_2024-12-26_12-00.log", "compression_start_time": 1735214462, "compression_end_time": 1735215297, "job_id": 193, "pre_compression_size": 32254810578, "post_compression_size": 875193958, "compression_ratio": 36.85, "success_type": "Complete"}
{"filename": "example-auditbeat_2024-12-26_12-15.log", "compression_start_time": 1735215362, "compression_end_time": 1735216224, "job_id": 194, "pre_compression_size": 32404324547, "post_compression_size": 980491960, "compression_ratio": 33.05, "success_type": "Complete"}
{"filename": "example-filebeat_2024-12-26_14-05.log", "compression_start_time": 1735221962, "compression_end_time": 1735222755, "job_id": 195, "pre_compression_size": 32572757561, "post_compression_size": 885690204, "compression_ratio": 36.78, "success_type": "Complete"}
{"filename": "example-metricbeat_2024-12-26_15-10.log", "compression_start_time": 1735225862, "compression_end_time": 1735226657, "job_id": 196, "pre_compression_size": 32508491291, "post_compression_size": 987412561, "compression_ratio": 32.92, "success_type": "Complete"}
{"filename": "example-auditbeat_2024-12-26_15-35.log", "compression_start_time": 1735227362, "compression_end_time": 1735228208, "job_id": 197, "pre_compression_size": 32992141928, "post_compression_size": 995423682, "compression_ratio": 33.14, "success_type": "Complete"}
{"filename": "example-filebeat_2024-12-26_16-10.log", "compression_start_time": 1735229462, "compression_end_time": 1735230257, "job_id": 198, "pre_compression_size": 33338242901, "post_compression_size": 916728053, "compression_ratio": 36.37, "success_type": "Complete"}
{"filename": "example-filebeat_2024-12-26_18-15.log", "compression_start_time": 1735236962, "compression_end_time": 1735237848, "job_id": 199, "pre_compression_size": 33210669134, "post_compression_size": 913917870, "compression_ratio": 36.34, "success_type": "Complete"}
{"filename": "example-auditbeat_2024-12-26_18-50.log", "compression_start_time": 1735239063, "compression_end_time": 1735239936, "job_id": 200, "pre_compression_size": 32825229098, "post_compression_size": 985388810, "compression_ratio": 33.31, "success_type": "Complete"}
However the ingest tab of CLP does not show the jobs before 194:
image.png

Response

Thanks for bringing this to our attention and providing the detailed example - it’s very helpful.

From our investigation, the issue you’re seeing is likely related to how the ingest job history is displayed in the WebUI. Currently, the WebUI fetches the most recent jobs by polling the last 5 job IDs in descending order every 5 seconds. This approach can sometimes cause certain jobs to appear "missing," especially if multiple jobs are submitted in rapid succession.

We’re actively brainstorming solutions to address this and ensure a more reliable job history display. We will keep you updated.

YScope Investigation

Junhao was able to replicate this by submitting a batch of jobs. See "Reproduction steps" for details.

CLP version

8f00463

Environment

Junhao's

Ubuntu 22.04 WSL
Microsoft Edge: Version 132.0.2957.55 (Official build) stable app, beta channel (64-bit)

Reproduction steps

1. Start CLP package: https://docs.yscope.com/clp/main/user-guide/quick-start-cluster-setup/single-node.html
2. cd clp-package/sbin
3. Load the WebUI http://localhost:4000 in a browser and stay on the "Ingest" page.
4. Submit a batch of jobs at the same time (Note the & at the end of each command, which detaches the stdin from the commands and allows concurrent runs of the commands):

   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   ./compress.sh ~/samples/hive-24hr/i-00c90a0f/ &
   

5. Observe the jobs being added to the ingestion job table in the Web UI, but some jobs IDs (between 1 and 20) are missing.