diff --git a/apm/profiling/get-data-in-profiling.rst b/apm/profiling/get-data-in-profiling.rst index 70fc5663a..23a6b42d6 100644 --- a/apm/profiling/get-data-in-profiling.rst +++ b/apm/profiling/get-data-in-profiling.rst @@ -86,6 +86,10 @@ AlwaysOn Profiling requires APM tracing data to correlate stack traces to your a * - .NET - SignalFx Instrumentation for .NET version 1.0.0 or higher - :ref:`instrument-dotnet-applications` + * - Python (in beta) + - Splunk Distribution of OpenTelemetry Python version 1.15 or higher + - * :ref:`instrument-python-applications` + * :ref:`profiling-configuration-python` .. note:: See :ref:`apm-data-retention` for information on profiling data retention. @@ -199,6 +203,52 @@ To activate AlwaysOn Profiling, follow the steps for the appropriate programming For more configuration options, including setting a separate endpoint for profiling data, see :ref:`profiling-configuration-dotnet`. + .. group-tab:: Python + + .. note:: + AlwaysOn Profiling for Python is in beta development. This feature is provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Use of this feature is subject to the :new-page:`Splunk General Terms `. + + :strong:`Requirements` + + AlwaysOn Profiling requires Python 3.7.2 or higher. + + :strong:`Instrumentation` + + Activate the profiler by setting the ``SPLUNK_PROFILER_ENABLED`` environment variable to ``true`` or call the ``start_profiling`` function in your application code. + + Check the OTLP endpoint in the ``SPLUNK_PROFILER_LOGS_ENDPOINT`` environment variable: + + - For non-Kubernetes environments, make sure that the ``SPLUNK_PROFILER_LOGS_ENDPOINT`` environment variable points to \http://localhost:4317. + - For Kubernetes deployments, the OTLP endpoint has to point to \http://$(K8S_NODE_IP):4317 where the ``K8S_NODE_IP`` is fetched from the Kubernetes downstream API by setting the environment configuration on the Kubernetes pod running the application. For example: + + .. code-block:: yaml + + env: + - name: K8S_NODE_IP + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: status.hostIP + + The following example shows how to activate the profiler from your application's code: + + .. code-block:: python + + from splunk_otel.profiling import start_profiling + + # Activates CPU profiling + # All arguments are optional + start_profiling( + service_name='my-python-service', + resource_attributes={ + 'service.version': '3.1' + 'deployment.environment': 'production', + } + endpoint='http://localhost:4317' + ) + + For more configuration options, see :ref:`profiling-configuration-python`. + .. _profiling-check-data-coming-in: Check that Splunk Observability Cloud is receiving profiling data diff --git a/apm/profiling/intro-profiling.rst b/apm/profiling/intro-profiling.rst index 5681a47e3..dbbfa6e1b 100644 --- a/apm/profiling/intro-profiling.rst +++ b/apm/profiling/intro-profiling.rst @@ -85,6 +85,10 @@ The following programming languages have instrumentation available: * - .NET - SignalFx Instrumentation for .NET version 1.0.0 or higher - :ref:`instrument-dotnet-applications` + * - Python (in beta) + - Splunk Distribution of OpenTelemetry Python version 1.15 or higher + - * :ref:`instrument-python-applications` + * :ref:`profiling-configuration-python` .. raw:: html diff --git a/gdi/get-data-in/application/java/instrumentation/instrument-java-application.rst b/gdi/get-data-in/application/java/instrumentation/instrument-java-application.rst index fc9d65875..cad75195a 100644 --- a/gdi/get-data-in/application/java/instrumentation/instrument-java-application.rst +++ b/gdi/get-data-in/application/java/instrumentation/instrument-java-application.rst @@ -112,7 +112,7 @@ If you need to add custom attributes to spans or want to manually generate spans .. _enable_profiling_java: Activate AlwaysOn Profiling -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------------- To activate AlwaysOn Profiling, use the following system property argument. You can also use the ``SPLUNK_PROFILER_ENABLED`` environment variable. For more information, see :ref:`profiling-intro`. @@ -135,7 +135,7 @@ See :ref:`get-data-in-profiling` for more information. For more settings, see :r .. _enable_automatic_metric_collection: Activate metrics collection -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +--------------------------------------- To activate automatic metric collection, activate the metrics feature using a system property argument. You can also use the ``SPLUNK_METRICS_ENABLED`` environment variable. @@ -153,7 +153,7 @@ If your metrics endpoint is different than the default value, set the ``SPLUNK_M .. _ignore_endpoints_java: Ignore specific endpoints -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------------------ By default, the Java agent collects traces from all the endpoints of your application. To ignore specific endpoints, use the ``rules`` sampler and define ``drop`` rules. diff --git a/gdi/get-data-in/application/nodejs/instrumentation/instrument-nodejs-application.rst b/gdi/get-data-in/application/nodejs/instrumentation/instrument-nodejs-application.rst index 3c8402c60..70f174b30 100644 --- a/gdi/get-data-in/application/nodejs/instrumentation/instrument-nodejs-application.rst +++ b/gdi/get-data-in/application/nodejs/instrumentation/instrument-nodejs-application.rst @@ -107,7 +107,7 @@ If no data appears in APM, see :ref:`common-nodejs-troubleshooting`. .. _enable_profiling_nodejs: Activate AlwaysOn Profiling -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------------------- To activate AlwaysOn Profiling, set the ``SPLUNK_PROFILER_ENABLED`` environment variable to ``true``. @@ -130,7 +130,7 @@ See :ref:`get-data-in-profiling` for more information. For more settings, see :r .. _enable_automatic_metric_collection_nodejs: Activate metrics collection -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------------------------ To activate automatic runtime metric collection, activate the metrics feature using the ``SPLUNK_METRICS_ENABLED`` environment variable. See :ref:`metrics-configuration-nodejs` for more information. @@ -216,7 +216,7 @@ For a list of supported instrumentations, see :new-page:`https://github.com/open .. _kubernetes_nodejs_agent: Deploy the Node.js distribution in Kubernetes ------------------------------------------------------ +======================================================= To deploy the Splunk Distribution of OpenTelemetry JS in Kubernetes, configure the Kubernetes Downward API to expose environment variables to Kubernetes resources. @@ -250,7 +250,7 @@ The following example shows how to update a deployment to expose environment var .. _export-directly-to-olly-cloud-nodejs: Send data directly to Splunk Observability Cloud ------------------------------------------------------ +============================================================= By default, all telemetry is sent to the local instance of the Splunk Distribution of OpenTelemetry Collector. diff --git a/gdi/get-data-in/application/python/configuration/advanced-python-otel-configuration.rst b/gdi/get-data-in/application/python/configuration/advanced-python-otel-configuration.rst index 824d4ccfe..dd8e8e459 100644 --- a/gdi/get-data-in/application/python/configuration/advanced-python-otel-configuration.rst +++ b/gdi/get-data-in/application/python/configuration/advanced-python-otel-configuration.rst @@ -115,6 +115,28 @@ For backward compatibility with the SignalFx Python Tracing Library, use the b3m $env:OTEL_PROPAGATORS=b3multi +.. _profiling-configuration-python: + +Python settings for AlwaysOn Profiling +==================================================== + +The following settings control the AlwaysOn Profiling feature for the Python agent: + +.. list-table:: + :header-rows: 1 + :widths: 40, 60 + + * - Environment variable + - Description + * - ``SPLUNK_PROFILER_ENABLED`` + - Activates AlwaysOn Profiling. The default value is ``false``. + * - ``SPLUNK_PROFILER_LOGS_ENDPOINT`` + - The collector endpoint for profiler logs. By default, it takes the value of ``http://localhost:4317``. + * - ``SPLUNK_PROFILER_CALL_STACK_INTERVAL`` + - The frequency of call stack sampling, in milliseconds. The default value is ``1000``. + * - ``SPLUNK_PROFILER_INCLUDE_INTERNAL_STACKS`` + - Determines whether to include stack traces from internal profiler threads. The default value is ``false``. + .. _server-trace-information-python: Server trace information diff --git a/gdi/get-data-in/application/python/instrumentation/instrument-python-application.rst b/gdi/get-data-in/application/python/instrumentation/instrument-python-application.rst index 745327fb4..4e4cf9d68 100644 --- a/gdi/get-data-in/application/python/instrumentation/instrument-python-application.rst +++ b/gdi/get-data-in/application/python/instrumentation/instrument-python-application.rst @@ -123,6 +123,35 @@ Application metrics are collected by default. See :ref:`python-otel-metrics` for If no data appears in APM, see :ref:`common-python-troubleshooting`. +.. _enable-profiling-python: + +Activate AlwaysOn Profiling +------------------------------------------------ + +.. note:: + AlwaysOn Profiling for Python is in beta development. This feature is provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Use of this feature is subject to the :new-page:`Splunk General Terms `. + +To activate AlwaysOn Profiling, set the ``SPLUNK_PROFILER_ENABLED`` environment variable to ``true`` or call the ``start_profiling`` function in your application code. + +The following example shows how to activate the profiler from your application code: + +.. code-block:: python + + from splunk_otel.profiling import start_profiling + + # Activates CPU profiling + # All arguments are optional + start_profiling( + service_name='my-python-service', + resource_attributes={ + 'service.version': '3.1' + 'deployment.environment': 'production', + } + endpoint='http://localhost:4317' + ) + +See :ref:`get-data-in-profiling` for more information. For additional settings, see :ref:`profiling-configuration-python`. + .. _configure-python-instrumentation: Configure the Python agent @@ -135,7 +164,7 @@ For advanced configuration of the Python agent, like changing trace propagation .. _kubernetes_python_agent: Deploy the Python agent in Kubernetes ----------------------------------------------------- +=================================================== To deploy the Python agent in Kubernetes, configure the Kubernetes Downward API to expose environment variables to Kubernetes resources. @@ -168,7 +197,7 @@ The following example shows how to update a deployment to expose environment var .. _export-directly-to-olly-cloud-python: Send data directly to Splunk Observability Cloud ----------------------------------------------------- +========================================================== By default, the agent sends all telemetry to the local instance of the Splunk Distribution of OpenTelemetry Collector. @@ -203,6 +232,6 @@ The realm name appears in the :guilabel:`Organizations` section. .. _instrument_aws_python_functions: Instrument Lambda functions ----------------------------------------------------- +============================================= You can instrument AWS Lambda functions using the Splunk OpenTelemetry Lambda Layer. See :ref:`instrument-aws-lambda-functions` for more information. diff --git a/gdi/opentelemetry/kubernetes-config-advanced.rst b/gdi/opentelemetry/kubernetes-config-advanced.rst index 4bcb80e21..20632fccb 100644 --- a/gdi/opentelemetry/kubernetes-config-advanced.rst +++ b/gdi/opentelemetry/kubernetes-config-advanced.rst @@ -45,6 +45,8 @@ This custom configuration is merged into the default agent configuration. .. caution:: After merging the files you need to fully redefine parts of the configuration, for example ``service``, ``pipelines``, ``logs``, and ``processors``. +.. _otel-kubernetes-config-advanced-control-plane: + Override a control plane configuration ============================================================== diff --git a/gdi/opentelemetry/metrics-ootb-k8s.rst b/gdi/opentelemetry/metrics-ootb-k8s.rst index 84d59c2da..46b706268 100644 --- a/gdi/opentelemetry/metrics-ootb-k8s.rst +++ b/gdi/opentelemetry/metrics-ootb-k8s.rst @@ -12,8 +12,6 @@ The following sections list the collected metrics when you use the default confi The Collector processes collected data as configured in your pipelines. Therefore, metrics that are imported by receivers might be excluded further into the pipeline by other components. For instance, the :ref:`signalfx-exporter`, included in the default Collector configuration, drops certain metrics and applies :ref:`translation rules ` that impact the metrics the Collector sends to Splunk Observability Cloud. -.. caution:: Check the :strong:`Kept?` column in the tables below to see if the metric is finally sent to Splunk Observability Cloud after going through the Collector pipelines. - Learn more about the Collector's configuration and data processing at: * :ref:`otel-configuration-ootb` @@ -26,114 +24,120 @@ Learn more about the Collector's configuration and data processing at: Container level metrics and dimensions ============================================================================ +.. caution:: + + The :strong:`Translated?` column indicates whether the metric is translated by the SignalFx exporter. + + The :strong:`Exported?` column indicates if the metric is finally sent to Splunk Observability Cloud after going through the Collector pipelines. + .. list-table:: - :widths: 30 30 30 10 + :widths: 25 25 30 10 10 :width: 100% :header-rows: 1 * - Metric name - Metric description - Attributes - - Kept? + - Translated? + - Exported? * - ``container.cpu.utilization`` - Container CPU utilization - - - - No + - :ref:`ootb-metrics-k8s-attributes` + - Yes + - Yes * - ``container.cpu.time`` - Container CPU time - - No - + - No + * - ``container.memory.available`` - Container memory available - - No + - No * - ``container.memory.usage`` - Container memory usage - - No + - No * - ``container.memory.rss`` - Container memory rss - - No + - No * - ``container.memory.working_set`` - Container memory ``working_set`` - - No + - No * - ``container.memory.page_faults`` - Container memory ``page_faults`` - - No + - No * - ``container.memory.major_page_faults`` - Container memory ``major_page_faults`` - - No + - No * - ``container.filesystem.available`` - Container filesystem available - :ref:`ootb-metrics-k8s-attributes` - No + - Yes * - ``container.filesystem.capacity`` - Container filesystem capacity - :ref:`ootb-metrics-k8s-attributes` - No + - Yes * - ``container.filesystem.usage`` - Container filesystem usage - :ref:`ootb-metrics-k8s-attributes` - No + - Yes * - ``k8s.container.cpu_limit`` - Container ``cpu_limit`` - :ref:`ootb-metrics-k8s-attributes` - No + - Yes * - ``k8s.container.cpu_request`` - Container ``cpu_request`` - :ref:`ootb-metrics-k8s-attributes` - No + - Yes * - ``k8s.container.memory_limit`` - Container ``memory_limit`` - :ref:`ootb-metrics-k8s-attributes` - No + - Yes * - ``k8s.container.memory_request`` - Container ``memory_request`` - :ref:`ootb-metrics-k8s-attributes` - No - -Container level metrics and dimensions after translation -============================================================================ - -.. note:: These metrics are compatible with the :ref:`signalfx-exporter`. - -.. list-table:: - :widths: 30 30 30 10 - :width: 100% - :header-rows: 1 - - * - Metric name - - Metric description - - Attributes - - Kept? - - * - ``container.cpu.utilization`` - - Container CPU utilization in centicores - - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes Pod level metrics and dimensions ============================================================================ +.. caution:: + + The :strong:`Exported?` column indicates if the metric is finally sent to Splunk Observability Cloud after going through the Collector pipelines. + .. list-table:: :widths: 30 30 30 10 :width: 100% @@ -142,7 +146,7 @@ Pod level metrics and dimensions * - Metric name - Metric description - Attributes - - Kept? + - Exported? * - ``k8s.pod.cpu.time`` - Pod CPU time @@ -204,18 +208,24 @@ Pod level metrics and dimensions - * ``interface``. See :ref:`kubelet-stats-receiver` * ``direction``. See :ref:`kubelet-stats-receiver` * :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``k8s.pod.network.io`` - Pod network IO - * ``interface``. See :ref:`kubelet-stats-receiver` * ``direction``. See :ref:`kubelet-stats-receiver` * :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes Node level metrics and dimensions ============================================================================ +.. caution:: + + The :strong:`Translated?` column indicates whether the metric is translated by the SignalFx exporter. + + The :strong:`Exported?` column indicates if the metric is finally sent to Splunk Observability Cloud after going through the Collector pipelines. + .. list-table:: :widths: 30 30 30 10 :width: 100% @@ -224,7 +234,7 @@ Node level metrics and dimensions * - Metric name - Metric description - Attributes - - Kept? + - Exported? * - ``k8s.node.cpu.time`` - Node CPU time @@ -250,7 +260,7 @@ Node level metrics and dimensions - Node filesystem usage - - No - + * - ``k8s.node.memory.available`` - Node memory available - @@ -305,7 +315,7 @@ Node level metrics and dimensions - * ``cpu``. See :ref:`host-metrics-receiver` * ``state``. See :ref:`host-metrics-receiver` * :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``system.filesystem.usage`` - Filesystem bytes used @@ -315,7 +325,7 @@ Node level metrics and dimensions * ``type``. See :ref:`host-metrics-receiver` * ``state``. See :ref:`host-metrics-receiver` * :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``system.filesystem.utilization`` - Fraction of filesystem bytes used @@ -324,41 +334,45 @@ Node level metrics and dimensions * ``mountpoint``. See :ref:`host-metrics-receiver` * ``type``. See :ref:`host-metrics-receiver` * :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``system.cpu.load_average.1m`` - Average CPU Load over 1 minute - - - **Yes** + - Yes * - ``system.cpu.load_average.5m`` - Average CPU Load over 5 minutes - - - **Yes** + - Yes * - ``system.cpu.load_average.15m`` - Average CPU Load over 15 minutes - - - **Yes** + - Yes * - ``system.memory.usage`` - Bytes of memory in use - ``state``. See :ref:`host-metrics-receiver` - - **Yes** + - Yes * - ``system.memory.utilization`` - Percentage of memory bytes in use - ``state``. See :ref:`host-metrics-receiver` - - **Yes** + - Yes * - ``system.paging.utilization`` - Swap (Unix) or pagefile (Windows) utilization - - - **Yes** + - Yes Node level metrics and dimensions after translation ============================================================================ +.. caution:: + + The :strong:`Exported?` column indicates if the metric is finally sent to Splunk Observability Cloud after going through the Collector pipelines. + .. note:: These metrics are compatible with the :ref:`signalfx-exporter`. .. list-table:: @@ -369,83 +383,83 @@ Node level metrics and dimensions after translation * - Metric name - Metric description - Attributes - - Kept? + - Exported? * - ``cpu.idle`` - CPU time in centicores spent in any state other than those in the table - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.interrupt`` - CPU time in centicores spent while servicing hardware interrupts - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.nice`` - CPU time in centicores spent in userspace running ``'nice'-ed processes`` - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.softirq`` - CPU time in centicores spent while servicing software interrupts - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.steal`` - CPU time in centicores spent waiting for a hypervisor to service requests from other virtual machines - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.system`` - CPU time in centicores spent running in the kernel - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.user`` - CPU time in centicores spent running in userspace - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.wait`` - CPU time in centicores spent idle while waiting for an I/O operation to complete - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.num_processors`` - The number of logical processors on the host - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``cpu.utilization`` - Percent of CPU used on this host - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``disk.summary_utilization`` - Percent of disk space utilized on all volumes on this host - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``disk.utilization`` - Percent of disk used on this volume - * ``device`` * :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``memory.total`` - Total bytes of system memory on the system - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``memory.utilization`` - Percent of memory in use on this host - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``network.total`` - Total amount of inbound and outbound network traffic on this host, in bytes - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``process.cpu_time_seconds`` - Total CPU usage of the process in seconds @@ -457,11 +471,16 @@ Node level metrics and dimensions after translation * ``process.command_line`` * ``process.owner`` * :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes Volume level metrics and dimensions ============================================================================ +.. caution:: + + The :strong:`Exported?` column indicates if the metric is finally sent to Splunk Observability Cloud after going through the Collector pipelines. + + .. list-table:: :widths: 30 30 30 10 :width: 100% @@ -470,17 +489,38 @@ Volume level metrics and dimensions * - Metric name - Metric description - Attributes - - Kept? + - Exported? * - ``k8s.volume.available`` - The number of available bytes in the volume - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes * - ``k8s.volume.capacity`` - The number of capacity bytes in the volume - :ref:`ootb-metrics-k8s-attributes` - - **Yes** + - Yes + +.. _ootb-metrics-k8s-control-plane: + +Control plane metrics +============================================================================ + +To see the control plane metrics the Collector provides, see: + +* :ref:`CoreDNS ` +* :ref:`etcd` +* :ref:`Kubernetes controller manager ` +* :ref:`Kubernetes API server ` +* :ref:`Kubernetes proxy ` +* :ref:`Kubernetes scheduler ` + +The following distributions support control plane metrics configuration: + +* Kubernetes 1.22 (kops created) +* OpenShift version 4.9 + +For information about control plane metrics, see :ref:`otel-kubernetes-config-advanced-control-plane`. .. _ootb-metrics-k8s-attributes: