docs: Describe reserved pipepline logs feature

KubeRocketCI · Feb 7, 2025 · 11dcdff · 11dcdff
1 parent 58f47fd
commit 11dcdff
Show file tree

Hide file tree

Showing 23 changed files with 280 additions and 41 deletions.
diff --git a/blog/2024-12-03/aws-eks-microsoft-entra-oidc-integration.md b/blog/2024-12-03/aws-eks-microsoft-entra-oidc-integration.md
@@ -333,7 +333,7 @@ To authenticate to the AWS EKS cluster using Microsoft Entra, you can use the `k
 
 ## Configuring KubeRocketCI Portal with Microsoft Entra OIDC Authentication
 
-1. Starting from version 3.11, KubeRocketCI platform supports Microsoft Entra as an Identity Provider for OIDC authentication in the Portal UI. To configure Microsoft Entra OIDC authentication, navigate to the [edp-install](https://github.com/epam/edp-install) Helm chart repository and set the following values in the `values.yaml` file:
+1. Starting from version 3.10, KubeRocketCI platform supports Microsoft Entra as an Identity Provider for OIDC authentication in the Portal UI. To configure Microsoft Entra OIDC authentication, navigate to the [edp-install](https://github.com/epam/edp-install) Helm chart repository and set the following values in the `values.yaml` file:
 
     ```yaml
     ...

diff --git a/docs/assets/operator-guide/ci/component-git-repository.png b/docs/assets/operator-guide/ci/component-git-repository.png
diff --git a/docs/assets/operator-guide/ci/create-pull-request.png b/docs/assets/operator-guide/ci/create-pull-request.png
diff --git a/docs/assets/operator-guide/ci/default-logs.png b/docs/assets/operator-guide/ci/default-logs.png
diff --git a/docs/assets/operator-guide/ci/delete-pipeline.png b/docs/assets/operator-guide/ci/delete-pipeline.png
diff --git a/docs/assets/operator-guide/ci/kuberocketci-portal.png b/docs/assets/operator-guide/ci/kuberocketci-portal.png
diff --git a/docs/assets/operator-guide/ci/long-term-logs.png b/docs/assets/operator-guide/ci/long-term-logs.png
diff --git a/docs/assets/operator-guide/ci/pipeline-logs.png b/docs/assets/operator-guide/ci/pipeline-logs.png
diff --git a/docs/assets/operator-guide/ci/portal-components.png b/docs/assets/operator-guide/ci/portal-components.png
diff --git a/docs/assets/operator-guide/ci/pr-checks.png b/docs/assets/operator-guide/ci/pr-checks.png
diff --git a/docs/assets/operator-guide/ci/reserved-logs.png b/docs/assets/operator-guide/ci/reserved-logs.png
diff --git a/docs/operator-guide/ci/tekton-long-term-storage.md b/docs/operator-guide/ci/tekton-long-term-storage.md
@@ -7,36 +7,155 @@ import TabItem from '@theme/TabItem';
   <link rel="canonical" href="https://docs.kuberocketci.io/docs/operator-guide/ci/tekton-long-term-storage/" />
 </head>
 
+KubeRocketCI Portal provides the capability to view long-term logs for pipelines. The long-term logs are stored in the OpenSearch cluster and can be viewed in the KubeRocketCI Portal after pipeline cleanup or deletion. This guide describes how to configure and view long-term logs in the KubeRocketCI Portal.
 
-This guide provides detailed instructions for setting up long-term log storage in Tekton. Without this configuration, Tekton pipeline logs are lost when associated resources are deleted. By following the steps in this guide, you can ensure that logs are preserved and remain accessible via the KubeRocketCI User Interface (UI), even after Tekton pipeline resources are removed.
+## Long-Term Log Access Workflow
+
+The following diagram illustrates the workflow for accessing long-term logs for pipelines in the KubeRocketCI Portal:
+
+```mermaid
+sequenceDiagram
+    actor U as User
+    participant KR as KubeRocketCI Portal
+    participant PC as Pipeline Controller
+    participant RW as Fluent Bit Agent
+    participant KD as KrakenD API Gateway
+    participant RA as OpenSearch
+
+    U->>KR: Trigger or create a pipeline
+    KR->>PC: Start pipeline execution
+    RW-->>PC: Collect pipeline logs
+    RW->>RA: Send logs to OpenSearch
+    Note over KR,RA: Pipeline is deleted or cleaned up
+    U->>KR: View pipeline logs
+    KR-->>KD: Request long-term logs
+    KD-->>RA: Fetch long-term logs
+    KR->>U: Return long-term logs
+```
 
 ## Configuration
 
-To configure long-term log storage, follow these steps:
+To configure long-term log storage for pipelines in the KubeRocketCI Portal, follow the steps below:
+
+1. Install and configure OpenSearch cluster:
+
+    :::note
+    To install OpenSearch cluster, we recommend to use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/opensearch) repository, which contains already prepared configurations for OpenSearch installation.
+    :::
+
+    To install OpenSearch cluster using the add-ons repository, follow the steps below:
+
+    1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/) repository.
+
+    2. Navigate to the `clusters/core/addons/opensearch` directory and configure the `values.yaml` file with the necessary values for OpenSearch cluster installation.
+
+        :::note
+        Ensure that [Fluent Bit](https://github.com/epam/edp-cluster-add-ons/blob/44ca88c079d464c826fcae38f3f03fe983d1f984/clusters/core/addons/opensearch/values.yaml#L391) is configured to send container logs to the `logstash-edp` index in the OpenSearch cluster, as the KubeRocketCI Portal uses this index to retrieve long-term logs.
+        :::
+
+    3. Install the OpenSearch cluster using Helm or [Argo CD](../add-ons-overview.md) methods.
+
+2. Install and configure KrakenD API Gateway:
+
+    :::note
+    To install KrakenD API Gateway with OpenSearch connection, we recommend to use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/krakend) repository, which contains already prepared configurations for KrakenD installation.
+    For more details, refer to the [KrakenD Integration](../extensions/krakend.md) guide.
+    :::
+
+    To install KrakenD API Gateway with OpenSearch connection, follow the steps below:
+
+    1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/) repository.
+
+    2. Navigate to the `clusters/core/addons/krakend` directory and configure the `values.yaml` file with the necessary values for KrakenD installation.
+
+        :::note
+        Ensure that the KrakenD endpoint object for OpenSearch has the `/search/logs` endpoint name, as the KubeRocketCI Portal uses this endpoint to fetch long-term logs.
+        :::
+
+    3. Ensure that the KrakenD [configuration secret](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/krakend/templates/external-secrets/krakend.yaml) contains the OpenSearch connection variables.
+
+        <Tabs
+          defaultValue="externalsecret"
+          values={[
+            {label: 'Manifests', value: 'manifests'},
+            {label: 'External Secrets Operator', value: 'externalsecret'},
+          ]}>
+
+          <TabItem value="manifests">
+
+          ```yaml
+          apiVersion: v1
+          kind: Secret
+          metadata:
+            name: krakend
+            namespace: krakend
+          type: Opaque
+          stringData:
+            OPENSEARCH_URL: https://opensearch-cluster-master.logging:9200
+            OPENSEARCH_CREDS: <base64-encoded-credentials>
+          ```
+
+          </TabItem>
+
+          <TabItem value="externalsecret">
+
+          ```json
+          {
+            "OPENSEARCH_URL": "https://opensearch-cluster-master.logging:9200",
+            "OPENSEARCH_CREDS": "<base64-encoded-credentials>"
+          }
+          ```
+          </TabItem>
+
+        </Tabs>
+
+    4. Install the KrakenD API Gateway using Helm or [Argo CD](../add-ons-overview.md) methods.
+
+## Viewing Long-Term Logs
+
+After configuring long-term log storage for Pipelines in the KubeRocketCI Portal, follow the steps below to view long-term logs:
+
+1. Navigate to the KubeRocketCI Portal and sign in with appropriate credentials.
+
+    ![KubeRocketCI Portal](../../assets/operator-guide/ci/kuberocketci-portal.png "KubeRocketCI Portal")
+
+2. In the **Components** section, navigate to the appropriate component (if component is not created yet, refer to the [Add Application](../../user-guide/add-application.md) guide to create a new one).
+
+    ![KubeRocketCI Portal Components](../../assets/operator-guide/ci/portal-components.png "KubeRocketCI Portal Components")
+
+3. In the component window, in the right upper corner, click the **GIT** button to navigate to the component repository (e.g., GitHub repository).
+
+    ![Component Git Repository](../../assets/operator-guide/ci/component-git-repository.png "Component Git Repository")
+
+4. In the component repository, create a Pull Request to trigger the review pipeline execution.
+
+    ![Create Pull Request](../../assets/operator-guide/ci/create-pull-request.png "Create Pull Request")
+
+5. In the Pull Request checks section, click the **Show all checks** button and navigate to the review pipeline in the KubeRocketCI Portal by clicking the **Details** button.
+
+    ![PR Checks](../../assets/operator-guide/ci/pr-checks.png "PR Checks")
+
+6. In the pipeline window, check the pipeline status. Ensure that the pipeline is successfully executed and the logs are available.
+
+    ![Pipeline Logs](../../assets/operator-guide/ci/pipeline-logs.png "Pipeline Logs")
+
+7. Navigate to the **Pipelines** section and find the executed review pipeline. Delete the pipeline by clicking the **Delete** button.
 
-1. Install and configure [OpenSearch](https://OpenSearch.org/) using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/OpenSearch) or manually.
+    ![Delete Pipeline](../../assets/operator-guide/ci/delete-pipeline.png "Delete Pipeline")
 
-2. Install KrakenD using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/krakend.md). Once installed, configure Krakend to ensure proper log retrieval and API handling. For detailed setup and configuration guidelines, refer to the [KrakenD Integration](../extensions/krakend.md).
+8. Return to the Pull Request and click the **Show all checks** button. Navigate to the review pipeline again by clicking the **Details** button.
 
-With this setup, even after the deletion of pods or PipelineRuns, information about all executed tasks and their logs will remain accessible within the KubeRocketCI platform.
+    ![PR Checks](../../assets/operator-guide/ci/pr-checks.png "PR Checks")
 
-    <Tabs
-      defaultValue="default"
-      values={[
-        {label: 'Default Logs', value: 'default'},
-        {label: 'Long-Term Logs', value: 'long-term'}
-      ]}>
+9. After the pipeline deletion, the long-term logs are displayed in the pipeline window.
 
-      <TabItem value="default">
-        ![Default Logs](../../assets/operator-guide/ci/default-logs.png "Default Logs")
-      </TabItem>
+    ![Long Term Logs](../../assets/operator-guide/ci/reserved-logs.png "Long Term Logs")
 
-      <TabItem value="long-term">
-        ![Long Term Logs](../../assets/operator-guide/ci/long-term-logs.png "Long Term Logs")
-      </TabItem>
-    </Tabs>
+This example demonstrates how to view long-term logs for review pipeline, but the same approach can be applied to other pipeline types.
 
 ## Related Articles
 
-* [Install Tekton](../install-tekton.md)
-* [Install KubeRocketCI](../install-kuberocketci.md)
+- [Add Application](../../user-guide/add-application.md)
+- [Install Tekton](../install-tekton.md)
+- [Install KubeRocketCI](../install-kuberocketci.md)
+- [KrakenD API Gateway](../extensions/krakend.md)
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/component-git-repository.png b/versioned_docs/version-3.10/assets/operator-guide/ci/component-git-repository.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/create-pull-request.png b/versioned_docs/version-3.10/assets/operator-guide/ci/create-pull-request.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/default-logs.png b/versioned_docs/version-3.10/assets/operator-guide/ci/default-logs.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/delete-pipeline.png b/versioned_docs/version-3.10/assets/operator-guide/ci/delete-pipeline.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/kuberocketci-portal.png b/versioned_docs/version-3.10/assets/operator-guide/ci/kuberocketci-portal.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/long-term-logs.png b/versioned_docs/version-3.10/assets/operator-guide/ci/long-term-logs.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/pipeline-logs.png b/versioned_docs/version-3.10/assets/operator-guide/ci/pipeline-logs.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/portal-components.png b/versioned_docs/version-3.10/assets/operator-guide/ci/portal-components.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/pr-checks.png b/versioned_docs/version-3.10/assets/operator-guide/ci/pr-checks.png
diff --git a/versioned_docs/version-3.10/assets/operator-guide/ci/reserved-logs.png b/versioned_docs/version-3.10/assets/operator-guide/ci/reserved-logs.png
diff --git a/versioned_docs/version-3.10/operator-guide/ci/tekton-long-term-storage.md b/versioned_docs/version-3.10/operator-guide/ci/tekton-long-term-storage.md
@@ -7,35 +7,155 @@ import TabItem from '@theme/TabItem';
   <link rel="canonical" href="https://docs.kuberocketci.io/docs/operator-guide/ci/tekton-long-term-storage/" />
 </head>
 
-This guide provides detailed instructions for setting up long-term log storage in Tekton. Without this configuration, Tekton pipeline logs are lost when associated resources are deleted. By following the steps in this guide, you can ensure that logs are preserved and remain accessible via the KubeRocketCI User Interface (UI), even after Tekton pipeline resources are removed.
+KubeRocketCI Portal provides the capability to view long-term logs for pipelines. The long-term logs are stored in the OpenSearch cluster and can be viewed in the KubeRocketCI Portal after pipeline cleanup or deletion. This guide describes how to configure and view long-term logs in the KubeRocketCI Portal.
+
+## Long-Term Log Access Workflow
+
+The following diagram illustrates the workflow for accessing long-term logs for pipelines in the KubeRocketCI Portal:
+
+```mermaid
+sequenceDiagram
+    actor U as User
+    participant KR as KubeRocketCI Portal
+    participant PC as Pipeline Controller
+    participant RW as Fluent Bit Agent
+    participant KD as KrakenD API Gateway
+    participant RA as OpenSearch
+
+    U->>KR: Trigger or create a pipeline
+    KR->>PC: Start pipeline execution
+    RW-->>PC: Collect pipeline logs
+    RW->>RA: Send logs to OpenSearch
+    Note over KR,RA: Pipeline is deleted or cleaned up
+    U->>KR: View pipeline logs
+    KR-->>KD: Request long-term logs
+    KD-->>RA: Fetch long-term logs
+    KR->>U: Return long-term logs
+```
 
 ## Configuration
 
-To configure long-term log storage, follow these steps:
+To configure long-term log storage for pipelines in the KubeRocketCI Portal, follow the steps below:
+
+1. Install and configure OpenSearch cluster:
+
+    :::note
+    To install OpenSearch cluster, we recommend to use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/opensearch) repository, which contains already prepared configurations for OpenSearch installation.
+    :::
+
+    To install OpenSearch cluster using the add-ons repository, follow the steps below:
+
+    1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/) repository.
+
+    2. Navigate to the `clusters/core/addons/opensearch` directory and configure the `values.yaml` file with the necessary values for OpenSearch cluster installation.
+
+        :::note
+        Ensure that [Fluent Bit](https://github.com/epam/edp-cluster-add-ons/blob/44ca88c079d464c826fcae38f3f03fe983d1f984/clusters/core/addons/opensearch/values.yaml#L391) is configured to send container logs to the `logstash-edp` index in the OpenSearch cluster, as the KubeRocketCI Portal uses this index to retrieve long-term logs.
+        :::
+
+    3. Install the OpenSearch cluster using Helm or [Argo CD](../add-ons-overview.md) methods.
+
+2. Install and configure KrakenD API Gateway:
+
+    :::note
+    To install KrakenD API Gateway with OpenSearch connection, we recommend to use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/krakend) repository, which contains already prepared configurations for KrakenD installation.
+    For more details, refer to the [KrakenD Integration](../extensions/krakend.md) guide.
+    :::
+
+    To install KrakenD API Gateway with OpenSearch connection, follow the steps below:
+
+    1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/) repository.
+
+    2. Navigate to the `clusters/core/addons/krakend` directory and configure the `values.yaml` file with the necessary values for KrakenD installation.
+
+        :::note
+        Ensure that the KrakenD endpoint object for OpenSearch has the `/search/logs` endpoint name, as the KubeRocketCI Portal uses this endpoint to fetch long-term logs.
+        :::
+
+    3. Ensure that the KrakenD [configuration secret](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/krakend/templates/external-secrets/krakend.yaml) contains the OpenSearch connection variables.
+
+        <Tabs
+          defaultValue="externalsecret"
+          values={[
+            {label: 'Manifests', value: 'manifests'},
+            {label: 'External Secrets Operator', value: 'externalsecret'},
+          ]}>
+
+          <TabItem value="manifests">
+
+          ```yaml
+          apiVersion: v1
+          kind: Secret
+          metadata:
+            name: krakend
+            namespace: krakend
+          type: Opaque
+          stringData:
+            OPENSEARCH_URL: https://opensearch-cluster-master.logging:9200
+            OPENSEARCH_CREDS: <base64-encoded-credentials>
+          ```
+
+          </TabItem>
+
+          <TabItem value="externalsecret">
+
+          ```json
+          {
+            "OPENSEARCH_URL": "https://opensearch-cluster-master.logging:9200",
+            "OPENSEARCH_CREDS": "<base64-encoded-credentials>"
+          }
+          ```
+          </TabItem>
+
+        </Tabs>
+
+    4. Install the KrakenD API Gateway using Helm or [Argo CD](../add-ons-overview.md) methods.
+
+## Viewing Long-Term Logs
+
+After configuring long-term log storage for Pipelines in the KubeRocketCI Portal, follow the steps below to view long-term logs:
+
+1. Navigate to the KubeRocketCI Portal and sign in with appropriate credentials.
+
+    ![KubeRocketCI Portal](../../assets/operator-guide/ci/kuberocketci-portal.png "KubeRocketCI Portal")
+
+2. In the **Components** section, navigate to the appropriate component (if component is not created yet, refer to the [Add Application](../../user-guide/add-application.md) guide to create a new one).
+
+    ![KubeRocketCI Portal Components](../../assets/operator-guide/ci/portal-components.png "KubeRocketCI Portal Components")
+
+3. In the component window, in the right upper corner, click the **GIT** button to navigate to the component repository (e.g., GitHub repository).
+
+    ![Component Git Repository](../../assets/operator-guide/ci/component-git-repository.png "Component Git Repository")
+
+4. In the component repository, create a Pull Request to trigger the review pipeline execution.
+
+    ![Create Pull Request](../../assets/operator-guide/ci/create-pull-request.png "Create Pull Request")
+
+5. In the Pull Request checks section, click the **Show all checks** button and navigate to the review pipeline in the KubeRocketCI Portal by clicking the **Details** button.
+
+    ![PR Checks](../../assets/operator-guide/ci/pr-checks.png "PR Checks")
+
+6. In the pipeline window, check the pipeline status. Ensure that the pipeline is successfully executed and the logs are available.
+
+    ![Pipeline Logs](../../assets/operator-guide/ci/pipeline-logs.png "Pipeline Logs")
+
+7. Navigate to the **Pipelines** section and find the executed review pipeline. Delete the pipeline by clicking the **Delete** button.
 
-1. Install and configure [OpenSearch](https://OpenSearch.org/) using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/OpenSearch) or manually.
+    ![Delete Pipeline](../../assets/operator-guide/ci/delete-pipeline.png "Delete Pipeline")
 
-2. Install KrakenD using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/krakend.md). Once installed, configure Krakend to ensure proper log retrieval and API handling. For detailed setup and configuration guidelines, refer to the [KrakenD Integration](../extensions/krakend.md).
+8. Return to the Pull Request and click the **Show all checks** button. Navigate to the review pipeline again by clicking the **Details** button.
 
-With this setup, even after the deletion of pods or PipelineRuns, information about all executed tasks and their logs will remain accessible within the KubeRocketCI platform.
+    ![PR Checks](../../assets/operator-guide/ci/pr-checks.png "PR Checks")
 
-    <Tabs
-      defaultValue="default"
-      values={[
-        {label: 'Default Logs', value: 'default'},
-        {label: 'Long-Term Logs', value: 'long-term'}
-      ]}>
+9. After the pipeline deletion, the long-term logs are displayed in the pipeline window.
 
-      <TabItem value="default">
-        ![Default Logs](../../assets/operator-guide/ci/default-logs.png "Default Logs")
-      </TabItem>
+    ![Long Term Logs](../../assets/operator-guide/ci/reserved-logs.png "Long Term Logs")
 
-      <TabItem value="long-term">
-        ![Long Term Logs](../../assets/operator-guide/ci/long-term-logs.png "Long Term Logs")
-      </TabItem>
-    </Tabs>
+This example demonstrates how to view long-term logs for review pipeline, but the same approach can be applied to other pipeline types.
 
 ## Related Articles
 
-* [Install Tekton](../install-tekton.md)
-* [Install KubeRocketCI](../install-kuberocketci.md)
+- [Add Application](../../user-guide/add-application.md)
+- [Install Tekton](../install-tekton.md)
+- [Install KubeRocketCI](../install-kuberocketci.md)
+- [KrakenD API Gateway](../extensions/krakend.md)
-Original file line number
+Diff line change
@@ Expand Up @@
     ## Configuring KubeRocketCI Portal with Microsoft Entra OIDC Authentication
-. Starting from version 3.11, KubeRocketCI platform supports Microsoft Entra as an Identity Provider for OIDC authentication in the Portal UI. To configure Microsoft Entra OIDC authentication, navigate to the [edp-install](https://github.com/epam/edp-install) Helm chart repository and set the following values in the `values.yaml` file:
+. Starting from version 3.10, KubeRocketCI platform supports Microsoft Entra as an Identity Provider for OIDC authentication in the Portal UI. To configure Microsoft Entra OIDC authentication, navigate to the [edp-install](https://github.com/epam/edp-install) Helm chart repository and set the following values in the `values.yaml` file:
         ```yaml
         ...
@@ Expand Down @@