From c30284fbd062790cb35393f4878d8d5eda7b289c Mon Sep 17 00:00:00 2001 From: Paulo Janotti Date: Tue, 22 Oct 2024 20:19:55 -0700 Subject: [PATCH 1/2] Replace ecs-metadata with awsecscontainermetrics Update the docs to deprecate ecs-metadata and replace it with the `awsecscontainermetrics`. Corresponding change on the Splunk Collector: https://github.com/signalfx/splunk-otel-collector/pull/5541 --- .../deployments/deployments-ecs-ec2.rst | 69 ++++++++----------- 1 file changed, 27 insertions(+), 42 deletions(-) diff --git a/gdi/opentelemetry/deployments/deployments-ecs-ec2.rst b/gdi/opentelemetry/deployments/deployments-ecs-ec2.rst index dcdd283bf..cbf6eb60a 100644 --- a/gdi/opentelemetry/deployments/deployments-ecs-ec2.rst +++ b/gdi/opentelemetry/deployments/deployments-ecs-ec2.rst @@ -5,9 +5,9 @@ Deploy the Collector with Amazon ECS EC2 ******************************************************** .. meta:: - :description: Deploy the Splunk Observability Cloud OpenTelemetry Collector as a Daemon service in an Amazon ECS EC2 cluster. + :description: Deploy the Splunk Observability Cloud OpenTelemetry Collector as a Sidecar in an Amazon ECS EC2 cluster. -Use the guided setup to deploy the Collector as a Daemon service in an Amazon ECS EC2 cluster. The guided setup provides a JSON task definition for the Collector. +Use the guided setup to deploy the Collector as a Sidecar in an Amazon ECS EC2 cluster. The guided setup provides a JSON task definition for the Collector. Choose one of the following Collector configuration options: @@ -28,60 +28,26 @@ Getting started The following sections describe how to create a task definition and launch the Collector. A task definition is required to run Docker containers in Amazon ECS. After creating the task definition, you need to launch the Collector. -Create a task definition +Add the Collector as a Sidecar --------------------------------- .. note:: Knowledge of Amazon ECS using launch type EC2 is assumed. See :new-page:`Getting started with the classic console using Amazon EC2 ` for further reading. -Creating the task definition requires using release v0.34.1 or newer (which corresponds to image tag 0.34.1 and newer) of the Collector. See the :new-page:`image repository ` to download the latest image. - -To create the task definition: +Open the ECS task definition to which the Collector Sidecar is going to be added: 1. Locate the task definition for the Collector from the :new-page:`repository `. -2. Replace ``MY_SPLUNK_ACCESS_TOKEN`` and ``MY_SPLUNK_REALM`` with valid values. You should pin the image version to a specific version instead of ``latest`` to avoid upgrade issues. -3. Create a task definition of EC2 launch type. See :new-page:`Creating a task definition using the new console ` for the instructions. The supplied task definition is a minimal definition. See :new-page:`Task definition parameters ` for additional configuration options. +2. Merge the definitions of the Collector with the existing ECS task definition. +3. Replace ``MY_SPLUNK_ACCESS_TOKEN`` and ``MY_SPLUNK_REALM`` with valid values. You can pin the image version to a specific version instead of ``latest`` if you want to avoid automatic upgrades. The Collector is configured to use the default configuration file ``/etc/otel/collector/ecs_ec2_config.yaml``. The Collector image Dockerfile is available at :new-page:`Dockerfile ` and the contents of the default configuration file can be seen at :new-page:`ECS EC2 configuration `. .. note:: - You do not need the ``smartagent/ecs-metadata`` metrics receiver in the default configuration file if all you want is tracing. You can take the default configuration, remove the receiver, then use the configuration in a custom configuration following the directions in :ref:`ecs-ec2-custom-config`. - -The configured network mode for the task is ``host``. This means that task metadata endpoint version 2 used by the ``smartagent/ecs-metadata`` receiver is not activated by default. See :new-page:`task metadata endpoint ` to determine if task metadata endpoint version 3 is activated by default for your task. If this version is activated, then add the following to the environment list in the task definition: - -.. code-block:: none - - { - "name": "ECS_TASK_METADATA_ENDPOINT", - "value": "${ECS_CONTAINER_METADATA_URI}/task" - }, - { - "name": "ECS_TASK_STATS_ENDPOINT", - "value": "${ECS_CONTAINER_METADATA_URI}/task/stats" - } + You do not need the ``awsecscontainermetrics`` receiver in the default configuration file if all you want is tracing. You can take the default configuration, remove the receiver, then use the configuration in a custom configuration following the directions in :ref:`ecs-ec2-custom-config`. Assign a stringified array of metrics you want excluded to environment variable ``METRICS_TO_EXCLUDE``. You can set the memory limit for the ``memory_limiter`` processor using environment variable ``SPLUNK_MEMORY_LIMIT_MIB``. The default memory limit is 512 MiB. -Launch the Collector -============================= -The Collector is designed to be run as a Daemon service in an EC2 ECS cluster. To create a Collector service from the Amazon ECS console: - -#. Go to your cluster in the console. -#. Select :guilabel:`Services`. -#. Select :guilabel:`Create`. -#. Select the following options: - #. Launch Type: EC2 - #. Task Definition (Family): splunk-otel-collector - #. Task Definition (Revision): 1 (or whatever the latest is in your case) - #. Service Name: splunk-otel-collector - #. Service type: DAEMON - #. Leave everything else at default. -#. Select :guilabel:`Next step`. -#. Leave everything on this next page at their defaults and select :guilabel:`Next step`. -#. Leave everything on this next page at their defaults and select :guilabel:`Next step`. -#. Select :guilabel:`Create Service` to deploy the Collector onto each node in the ECS cluster. You should see infrastructure and docker metrics flowing soon. - .. _ecs-ec2-custom-config: Use a custom configuration @@ -94,7 +60,7 @@ Alternatively, you can specify the custom configuration YAML directly using the ``ecs_observer`` configuration -------------------------------- -Use extension Amazon Elastic Container Service Observer (``ecs_observer``) in your custom configuration to discover metrics targets in running tasks, filtered by service names, task definitions, and container labels. ``ecs_observer`` is currently limited to Prometheus targets and requires the read-only permissions below. You can add the permissions to the task role by adding them to a customer-managed policy that is attached to the task role. +Use extension Amazon Elastic Container Service Observer (``ecs_observer``) in your custom configuration to discover metrics targets in running tasks, filtered by service names, task definitions, and container labels. ``ecs_observer`` is currently limited to Prometheus targets and requires the read-only permissions below. The Collector should be configured to run as an ECS Daemon. You can add the permissions to the task role by adding them to a customer-managed policy that is attached to the task role. .. code-block:: yaml @@ -147,6 +113,25 @@ The results are written to ``/etc/ecs_sd_targets.yaml``. The ``prometheus`` rece .. _aws-parameter-store: +Launch the Collector as a Daemon +------------------------------- +To launch the Collector from the Amazon ECS console: + +#. Go to your cluster in the console. +#. Select :guilabel:`Services`. +#. Select :guilabel:`Create`. +#. Select the following options: + #. Launch Type: EC2 + #. Task Definition (Family): splunk-otel-collector + #. Task Definition (Revision): 1 (or whatever the latest is in your case) + #. Service Name: splunk-otel-collector + #. Service type: DAEMON + #. Leave everything else at default. +#. Select :guilabel:`Next step`. +#. Leave everything on this next page at their defaults and select :guilabel:`Next step`. +#. Leave everything on this next page at their defaults and select :guilabel:`Next step`. +#. Select :guilabel:`Create Service` to deploy the Collector onto each node in the ECS cluster. You should see infrastructure and docker metrics flowing soon. + Use the AWS Parameter Store ---------------------------- From 1ce55f6e43946794a022050de49f4425752e923c Mon Sep 17 00:00:00 2001 From: Anna U <104845867+aurbiztondo-splunk@users.noreply.github.com> Date: Wed, 23 Oct 2024 06:45:06 +0200 Subject: [PATCH 2/2] Update deployments-ecs-ec2.rst Minor edits --- .../deployments/deployments-ecs-ec2.rst | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/gdi/opentelemetry/deployments/deployments-ecs-ec2.rst b/gdi/opentelemetry/deployments/deployments-ecs-ec2.rst index cbf6eb60a..6b6decab7 100644 --- a/gdi/opentelemetry/deployments/deployments-ecs-ec2.rst +++ b/gdi/opentelemetry/deployments/deployments-ecs-ec2.rst @@ -30,9 +30,7 @@ The following sections describe how to create a task definition and launch the C Add the Collector as a Sidecar --------------------------------- -.. note:: - - Knowledge of Amazon ECS using launch type EC2 is assumed. See :new-page:`Getting started with the classic console using Amazon EC2 ` for further reading. +.. note:: To use this option you need to be familiar with Amazon ECS EC2 launch type. See :new-page:`Getting started with the classic console using Amazon EC2 ` for further reading. Open the ECS task definition to which the Collector Sidecar is going to be added: @@ -42,29 +40,32 @@ Open the ECS task definition to which the Collector Sidecar is going to be added The Collector is configured to use the default configuration file ``/etc/otel/collector/ecs_ec2_config.yaml``. The Collector image Dockerfile is available at :new-page:`Dockerfile ` and the contents of the default configuration file can be seen at :new-page:`ECS EC2 configuration `. -.. note:: - - You do not need the ``awsecscontainermetrics`` receiver in the default configuration file if all you want is tracing. You can take the default configuration, remove the receiver, then use the configuration in a custom configuration following the directions in :ref:`ecs-ec2-custom-config`. +Notes: + +* You do not need the ``awsecscontainermetrics`` receiver in the default configuration file if all you want is tracing. You can take the default configuration, remove the receiver, then use the configuration in a custom configuration following the directions in :ref:`ecs-ec2-custom-config`. -Assign a stringified array of metrics you want excluded to environment variable ``METRICS_TO_EXCLUDE``. You can set the memory limit for the ``memory_limiter`` processor using environment variable ``SPLUNK_MEMORY_LIMIT_MIB``. The default memory limit is 512 MiB. +* To exclude metrics assign them as a stringified array to environment variable ``METRICS_TO_EXCLUDE``. + +* You can set the memory limit for the ``memory_limiter`` processor using environment variable ``SPLUNK_MEMORY_LIMIT_MIB``. The default memory limit is 512 MiB. .. _ecs-ec2-custom-config: Use a custom configuration ============================== + To use a custom configuration file, replace the value of the ``SPLUNK_CONFIG`` environment variable with the file path of the custom configuration file in the Collector task definition. Alternatively, you can specify the custom configuration YAML directly using the ``SPLUNK_CONFIG_YAML`` environment variable, as described in :ref:`ecs-observer-config`. .. _ecs-observer-config: -``ecs_observer`` configuration +Configure ``ecs_observer`` -------------------------------- + Use extension Amazon Elastic Container Service Observer (``ecs_observer``) in your custom configuration to discover metrics targets in running tasks, filtered by service names, task definitions, and container labels. ``ecs_observer`` is currently limited to Prometheus targets and requires the read-only permissions below. The Collector should be configured to run as an ECS Daemon. You can add the permissions to the task role by adding them to a customer-managed policy that is attached to the task role. .. code-block:: yaml - ecs:List* ecs:Describe* @@ -74,7 +75,6 @@ The results are written to ``/etc/ecs_sd_targets.yaml``. The ``prometheus`` rece .. code-block:: yaml - extensions: ecs_observer: refresh_interval: 10s @@ -114,7 +114,8 @@ The results are written to ``/etc/ecs_sd_targets.yaml``. The ``prometheus`` rece .. _aws-parameter-store: Launch the Collector as a Daemon -------------------------------- +-------------------------------------------- + To launch the Collector from the Amazon ECS console: #. Go to your cluster in the console.