docs: Validate the LogPipeline OTel Setup and test the performance of the log agent

docs/contributor/benchmarks/otlp-logs-validation.md

@@ -1,11 +1,17 @@
 # OTel LogPipeline Setup Validation
 
-This file documents the process of validating the whole LogPipeline with OTLP output flow. It defines the setup, that consists of the manually deployed log agent, the already-implemented log gateway, and log generators using flog.
+- [Setup Configuration Steps](#setup-configuration-steps)
+- [Resources Under Investigation](#resources-under-investigation)
+- [Benchmarking Setup](#benchmarking-setup)
+- [Performance Tests Results](#performance-tests-results)
+  - [📊 Benchmarking Session #1](#-benchmarking-session-1)
+  - [📊 Benchmarking Session #2](#-benchmarking-session-2)
+- [Conclusions](#conclusions)
 
-The scope is to performance test the agent, observing the resulting values (such as throughput, resource consumption, reaction to backpressure), and to compare the agent to the previous FluentBit-based setup.
 
+## Configuring the Log Agent
 
-## Setup Configuration Steps
+To configure the log agent, deploy the [OTLP Logs Validation YAML](./otlp-logs-validation.yaml) either with Helm or manually:
 
 - To set up the log agent with Helm, run:
 
@@ -30,26 +36,25 @@ The scope is to performance test the agent, observing the resulting values (such
     ```
 
 
-## Relevant/Configurable Resources
+## Resources Under Investigation
+We investigate the following resources (for details, see the [OTLP Logs Validation YAML](./otlp-logs-validation.yaml)):
 
 - Log Agent ConfigMap (OTel Config)
 - Log Agent DaemonSet
 
-See [OTLP Logs Validation YAML](./otlp-logs-validation.yaml)
 
-#### Things to take into consideration (at implementation)
+**Things to take into consideration, when implementing the Log Agent into Telemetry Manager:**
 - Dynamically include/exclude of namespaces, based on LogPipeline spec attributes.
 - Exclude FluentBit container in OTel configuration, and OTel container in FluentBit configuration.
 - `receivers/filelog/operators`: The copy body to `attributes.original` must be avoided if `dropLogRawBody` flag is enabled.
 
-### How does checkpointing work
+**How does checkpointing work?**
+By enabling the storeCheckpoint preset (Helm), the `file_storage` extension is activated in the filelog receiver.
+- The `file_storage` has the path `/var/lib/otelcol`.
+- Later, this path is mounted as a `hostPath` volume in the DaemonSet spec.
+- The extension is also set in the `storage` property of the filelog receiver.
 
-> By enabling the storeCheckpoint preset (Helm) the `file_storage` extension is activated in the receiver
-> - The `file_storage` has the path `/var/lib/otelcol`
-> - This path is later mounted as a `hostPath` volume in the DaemonSet spec
-> - The extension is also set in the `storage` property of the filelog receiver
-
-> `storage` = The ID of a storage extension to be used to store file offsets. File offsets enable the receiver to pick up where it left off in the case of a collector restart. If no storage extension is used, the receiver manages offsets only in memory.
+> **NOTE:** `storage` = The ID of a storage extension to be used to store file offsets. File offsets enable the filelog receiver to pick up where it left off in the case of a collector restart. If no storage extension is used, the receiver manages offsets only in memory.
 
 
 ## Benchmarking Setup
@@ -79,7 +84,7 @@ See [OTLP Logs Validation YAML](./otlp-logs-validation.yaml)
     k apply -f telemetry-manager/hack/load-tests/log-backpressure-config.yaml
     ```
 
-4. PromQL Queries used for measuring the results:
+4. You can use the following PromQL Queries for measuring the results (same/similar queries were used in measuring the results of the performance tests executed below):
     ``` sql
     -- RECEIVED
     round(sum(rate(otelcol_receiver_accepted_log_records{service="telemetry-log-agent-metrics"}[20m])))
@@ -100,15 +105,15 @@ See [OTLP Logs Validation YAML](./otlp-logs-validation.yaml)
 
 ## Performance Tests Results
 
-### 📊 Benchmarking Session #1
-
 | Icon | Meaning                                              |
 | ---- | ---------------------------------------------------- |
 | ⏳    | Full-test, involving the whole setup, usually 20 min |
 | 🪲    | Debugging session, usually shorter, not so reliable  |
 | 🏋️‍♀️    | Backpressure scenario                                |
 | ⭐️    | Best results observed (in a given scenario)          |
 
+### 📊 Benchmarking Session #1
+
 #### ⏳ 18 Dec 2024, 13:45 - 14:05 (20 min)
 - **Generator:** 10 replicas x 10 MB
 - **Agent:** no CPU limit, no queue
@@ -386,22 +391,22 @@ See [OTLP Logs Validation YAML](./otlp-logs-validation.yaml)
 
 
 ## Comparison with FluentBit Setup
-In the FluentBit setup, for the very same (initial) scenario (i.e. 10 generator replicas [old set-up] / 2 agents), the [load test](https://github.com/kyma-project/telemetry-manager/actions/runs/12691802471) outputs the following values for the agent:
+In the FluentBit setup, for the very same (initial) scenario (that is, 10 generator replicas [old set-up] or 2 agents), the [load test](https://github.com/kyma-project/telemetry-manager/actions/runs/12691802471) outputs the following values for the agent:
 - Exported Log Records/second: 27.8K
 
-
 ## Conclusions
-- Before 15 Jan. (first session):
+### Benchmarking Session #1 (before 15 Jan)
   - Compared to the FluentBit counterpart setup, a lower performance can be expected.
-  - Backpressure is currently not backpropagated from the gateway to the agent, resulting in logs being queued/lost on the gateway end. That's because the agent has no way of knowing when to stop, thus exports data continuously (this is a known issue, which is expected be solved by the OTel community in the next half year).
+  - Backpressure is currently not backpropagated from the gateway to the agent, resulting in logs being queued or lost on the gateway end. That's because the agent has no way of knowing when to stop, thus exports data continuously (this is a known issue, which is expected be solved by the OTel community in the next half year).
   - If the load is increased (that is, more generators, more logs, or more data), the log agent slows down.
   - The network communication between the agent and the gateway or/and the gateway represent a bottleneck in this setup. That's concluded because higher throughput was observed when using just a debug endpoint as an exporter.
   - CPU and memory consumption are surprisingly low, and this was not improved by removing the limits (quite the opposite was observed, with the CPU throttling more often and the throughput decreasing).
   - If the batch processor is enabled, throughput increased. But this comes at the cost of losing logs in some scenarios.
   - Further methods of improving the throughput might still be worth investigating.
-- After 15 jan. (second session):
-  - Removing the gateway improves throughput
-  - We now better understand the performance impact of each OTEL processor and of enabling/disabling compression
-  - Generators configuration greatly influence the setup => more generators exporting less data and taking less CPU leads to higher throughput than fewer generators taking more CPU and exporting more data
-  - There is a hard limit (see debug endpoint scenario) that we still not fully understand, since strictly based on the benchmarking numbers of OTEL, we should be getting higher throughput (i.e. something related to the infrastructure could be influencing this).
-  - We have now a more performant setup configuration, being more comparable with the numbers from the FluentBit setup
+
+### Benchmarking Session #2 (after 15 Jan)
+  - Removing the gateway improves throughput.
+  - We now better understand the performance impact of each OTel processor and of enabling or disabling compression.
+  - The generators' configuration greatly influences the setup: More generators exporting less data and taking less CPU leads to higher throughput than fewer generators taking more CPU and exporting more data.
+  - There is a hard limit (see debug endpoint scenario) that we still don't fully understand, because strictly based on the benchmarking numbers of OTel, we should be getting higher throughput. It's possible that something related to the infrastructure could be influencing this.
+  - We  now have a more performant setup configuration, being more comparable with the numbers from the FluentBit setup