Update docs for operatorcrds migration (#1619)

.chloggen/move-operator-crd-install-method.yaml

@@ -1,12 +1,16 @@
 # One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
-change_type: bug_fix
+change_type: breaking
 # The name of the component, or a single word describing the area of concern, (e.g. agent, clusterReceiver, gateway, operator, chart, other)
 component: operator
 # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
 note: Move operator CRD installation to the crds/ folder via a subchart to resolve Helm install ordering issues
 # One or more tracking issues related to the change
-issues: [1561]
+issues: [1561,1619]
 # (Optional) One or more lines of additional information to render under the primary note.
 # These lines will be padded with 2 spaces and then inserted directly into the document.
 # Use pipe (|) for multiline entries.
-subtext:
+subtext: |
+  - Users enabling the operator (`.Values.operator.enabled=true`) must now set `operatorcrds.install=true` in Helm values or [manually manage CRD installation](https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/auto-instrumentation-install.md#crd-management).
+  - Previously, CRDs were installed using templates (`operator.crds.create=true`), which could cause race conditions and installation failures.
+  - CRD installation is now handled via Helm's native `crds/` directory for better stability, using a [localized subchart](https://github.com/signalfx/splunk-otel-collector-chart/tree/main/helm-charts/splunk-otel-collector/charts/opentelemetry-operator-crds).
+  - If you use `operator.enabled=true` you may have to follow some migration steps, please see the [Upgrade guidelines](https://github.com/signalfx/splunk-otel-collector-chart/blob/main/UPGRADING.md#0113-to-0116).

UPGRADING.md

@@ -1,5 +1,86 @@
 # Upgrade guidelines
 
+## 0.113.0 to 0.116.0
+
+This guide provides steps for new users, transitioning users, and those maintaining previous operator CRD configurations:
+- New users: No migration for CRDs is required.
+- Previous users: Migration may be needed if using `operator.enabled=true`.
+
+CRD deployment has evolved over chart versions:
+- Before 0.110.0: CRDs were deployed via a crds/ directory (upstream default).
+- 0.110.0 to 1.113.0: CRDs were deployed using Helm templates  (upstream default), which had reported issues.
+- 0.116.0 and later:  Users must now explicitly configure their preferred CRD deployment method or deploy the
+  [CRDs manually](https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/auto-instrumentation-install.md#crd-management)
+  to avoid potential issues. Users can deploy CRDs via a crds/ directory again by enabling a newly added value.
+
+### **New Users**
+
+New users are advised to deploy CRDs via the `crds/` directory. For a fresh installation, use the following Helm values:
+
+```yaml
+operatorcrds:
+  install: true
+operator:
+  enabled: true
+```
+
+To install the chart:
+
+```bash
+helm install <release-name> splunk-otel-collector-chart/splunk-otel-collector --set operatorcrds.install=true,operator.enabled=true <extra_args>
+```
+
+### **Current Users (Recommended Migration to `crds/` Directory)**
+
+If you're using chart versions 0.110.0 to 1.113.0, CRDs are likely deployed via Helm templates. To migrate to the recommended `crds/` directory deployment:
+
+#### Step 1: Delete the Existing Chart
+
+Remove the chart to prepare for a fresh installation:
+
+```bash
+helm delete <release-name>
+```
+
+#### Step 2: Verify or Remove Existing CRDs
+
+Check if the following CRDs are present and delete them if necessary:
+
+```bash
+kubectl get crds | grep opentelemetry
+```
+
+```bash
+kubectl delete crd opentelemetrycollectors.opentelemetry.io
+kubectl delete crd opampbridges.opentelemetry.io
+kubectl delete crd instrumentations.opentelemetry.io
+```
+
+#### Step 3: Reinstall with Recommended Values
+
+Reinstall the chart with the updated configuration:
+```bash
+helm install <release-name> splunk-otel-collector --set operatorcrds.install=true,operator.enabled=true <extra_args>
+```
+
+### **Previous Users (Maintaining Legacy Helm Templates)**
+
+If you're using chart versions 0.110.0 to 1.113.0 and prefer to continue deploying CRDs via Helm templates (not recommended), you can do so with the following values:
+
+```yaml
+operator:
+  enabled: true
+operator:
+  crds:
+    create: true
+```
+
+**Warning**: This method may cause race conditions during installation or upgrades, leading to errors like:
+```plaintext
+ERROR: INSTALLATION FAILED: failed post-install: warning: Hook post-install splunk-otel-collector/templates/operator/instrumentation.yaml failed: 1 error occurred:
+* Internal error occurred: failed calling webhook "minstrumentation.kb.io": failed to call webhook: Post "https://splunk-otel-collector-operator-webhook.default.svc:443/mutate-opentelemetry-io-v1alpha1-instrumentation?timeout=10s": dial tcp X.X.X.X:443: connect: connection refused
+```
+
 ## 0.105.5 to 0.108.0
 
 We've simplified the Helm chart configuration for `operator` auto-instrumentation.

docs/auto-instrumentation-install.md

@@ -26,6 +26,8 @@ these frameworks often have pre-built instrumentation capabilities already avail
 ### 1. Deploy the Helm Chart with the Operator enabled
 
 - **Operator Deployment (Required)**
+  - `operatorcrds.install`: Set to `true` to install the CRDs required by the operator.
+    - **Required**: Must be set unless CRDs are pre-installed manually.
   - `operator.enabled`: Set to `true` to enable deploying the operator.
     - **Required**: This configuration is necessary for the operator's deployment within your cluster.
 
@@ -70,7 +72,7 @@ these frameworks often have pre-built instrumentation capabilities already avail
 kubectl get pods -l app=cert-manager --all-namespaces
 
 # If cert-manager is not deployed, make sure to add certmanager.enabled=true to the list of values to set
-helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
+helm install splunk-otel-collector -f ./my_values.yaml --set operatorcrds.install=true,operator.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
 ```
 
 ### 2. Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully
@@ -284,6 +286,11 @@ in a Kubernetes environment. An operator is a method of packaging, deploying, an
 In the context of setting up observability in a Kubernetes environment, an operator simplifies the management of
 application auto-instrumentation, making it easier to gain valuable insights into application performance.
 
+The OpenTelemetry operator relies on
+[Custom Resource Definitions (CRDs)](https://github.com/signalfx/splunk-otel-collector-chart/tree/main/helm-charts/splunk-otel-collector/charts/opentelemetry-operator-crds)
+to manage auto-instrumentation configurations in Kubernetes.
+Ensure the required CRDs are deployed before the operator by configuring `operatorcrds.install=true`.
+
 With this Splunk OTel Collector chart, the
 [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection)
 can be deployed (by configuring `operator.enabled=true`) to your cluster and start auto-instrumenting your applications.
@@ -403,6 +410,58 @@ provides best effort support with issues related to native OpenTelemetry instrum
 | apache-httpd            | OpenTelemetry | Available   | Needs Validation |                              | [Link](https://github.com/open-telemetry/opentelemetry-apache-httpd-instrumentation) | ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd |
 | nginx                   | OpenTelemetry | Available   | Needs Validation |                              | [Link](https://github.com/open-telemetry/opentelemetry-apache-httpd-instrumentation) | ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd |
 
+### CRD Management
+
+When deploying the operator, the required Custom Resource Definitions (CRDs) must be deployed beforehand.
+
+#### Recommended Approach: Automated CRD Deployment
+
+Set the Helm chart value `operatorcrds.install=true` to allow the chart to handle CRD installation automatically.
+_This option deploys the CRDs using a local subchart, available at [opentelemetry-operator-crds](https://github.com/signalfx/splunk-otel-collector-chart/tree/main/helm-charts/splunk-otel-collector/charts/opentelemetry-operator-crds)._
+_Please note, helm will not update or delete these CRDs after initial install as noted in their [documentation](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations)._
+
+#### Alternative Approach: Manual CRD Deployment
+
+If you prefer to manage CRD deployment manually, apply the CRDs using the commands below before installing the Helm chart:
+
+```bash
+curl -sL https://raw.githubusercontent.com/splunk-otel-collector/charts/opentelemetry-operator-crds/crds/opentelemetry.io_opentelemetrycollectors.yaml | kubectl apply -f -
+curl -sL https://raw.githubusercontent.com/splunk-otel-collector/charts/opentelemetry-operator-crds/crds/opentelemetry.io_opampbridges.yaml | kubectl apply -f -
+curl -sL https://raw.githubusercontent.com/splunk-otel-collector/charts/opentelemetry-operator-crds/crds/opentelemetry.io_instrumentations.yaml | kubectl apply -f -
+```
+
+You can also use below helm template command to get the CRD yamls from the helm chart. This method can be helpful in keeping CRDs in-sync with the version bundled with our helm chart.
+
+```bash
+helm template splunk-otel-collector-chart/splunk-otel-collector --include-crds \
+--set="splunkObservability.realm=us0,splunkObservability.accessToken=xxxxxx,clusterName=my-cluster,operatorcrds.install=true" \
+| yq e '. | select(.kind == "CustomResourceDefinition")' \
+| kubectl apply -f -
+```
+
+#### CRD Updates
+
+With Helm v3.0 and later, CRDs created by this chart are not updated automatically. To update CRDs, you must apply the updated CRD definitions manually.
+Refer to the [Helm Documentation on CRDs](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) for more details.
+
+#### CRD Cleanup
+
+When uninstalling this chart, the OpenTelemetry CRDs are not removed automatically. To delete them manually, use the following commands:
+
+```bash
+kubectl delete crd opentelemetrycollectors.opentelemetry.io
+kubectl delete crd opampbridges.opentelemetry.io
+kubectl delete crd instrumentations.opentelemetry.io
+```
+You can use below combination of helm and kubectl command to delete CRDs.
+
+```bash
+helm template splunk-otel-collector-chart/splunk-otel-collector --include-crds \
+--set="splunkObservability.realm=us0,splunkObservability.accessToken=xxxxxx,clusterName=my-cluster,operatorcrds.install=true" \
+| yq e '. | select(.kind == "CustomResourceDefinition")' \
+| kubectl delete --dry-run=client -f -
+```
+
 ### Documentation Resources
 
 - https://developers.redhat.com/devnation/tech-talks/using-opentelemetry-on-kubernetes

examples/enable-operator-and-auto-instrumentation/enable-operator-and-auto-instrumentation-values.yaml

@@ -7,6 +7,8 @@ splunkObservability:
 clusterName: CHANGEME
 environment: CHANGEME
 
+operatorcrds:
+  install: true
 operator:
   enabled: true
 certmanager:

examples/enable-operator-and-auto-instrumentation/otel-demo-dotnet.md

@@ -24,15 +24,15 @@ curl https://raw.githubusercontent.com/signalfx/splunk-otel-collector-chart/main
 #### 2.1 Deploy the Helm Chart with the Operator enabled
 
 To install the chart with operator in an existing cluster, make sure you have cert-manager installed and available.
-Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set certmanager.enabled=true,operator.enabled=true`.
+Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true`.
 These helm install commands will deploy the chart to the current namespace for this example.
 
 ```bash
 # Check if a cert-manager is already installed by looking for cert-manager pods.
 kubectl get pods -l app=cert-manager --all-namespaces
 
 # If cert-manager is deployed, make sure to remove certmanager.enabled=true to the list of values to set
-helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
+helm install splunk-otel-collector -f ./my_values.yaml --set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
 ```
 
 #### 2.2 Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully

examples/enable-operator-and-auto-instrumentation/otel-demo-nodejs.md

@@ -20,15 +20,15 @@ curl https://raw.githubusercontent.com/signalfx/splunk-otel-collector-chart/main
 #### 2.1 Deploy the Helm Chart with the Operator enabled
 
 To install the chart with operator in an existing cluster, make sure you have cert-manager installed and available.
-Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set certmanager.enabled=true,operator.enabled=true`.
+Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true`.
 These helm install commands will deploy the chart to the current namespace for this example.
 
 ```bash
 # Check if a cert-manager is already installed by looking for cert-manager pods.
 kubectl get pods -l app=cert-manager --all-namespaces
 
 # If cert-manager is deployed, make sure to remove certmanager.enabled=true to the list of values to set
-helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
+helm install splunk-otel-collector -f ./my_values.yaml --set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
 ```
 
 #### 2.2 Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully

examples/enable-operator-and-auto-instrumentation/spring-petclinic-java.md

@@ -20,15 +20,15 @@ curl https://raw.githubusercontent.com/signalfx/splunk-otel-collector-chart/main
 #### 2.1 Deploy the Helm Chart with the Operator enabled
 
 To install the chart with operator in an existing cluster, make sure you have cert-manager installed and available.
-Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set certmanager.enabled=true,operator.enabled=true`.
+Both the cert-manager and operator are subcharts of this chart and can be enabled with `--set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true`.
 These helm install commands will deploy the chart to the current namespace for this example.
 
 ```bash
 # Check if a cert-manager is already installed by looking for cert-manager pods.
 kubectl get pods -l app=cert-manager --all-namespaces
 
 # If cert-manager is deployed, make sure to remove certmanager.enabled=true to the list of values to set
-helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
+helm install splunk-otel-collector -f ./my_values.yaml --set operatorcrds.install=true,operator.enabled=true,certmanager.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
 ```
 
 #### 2.2 Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully

helm-charts/splunk-otel-collector/Chart.yaml

@@ -28,7 +28,7 @@ dependencies:
     repository: https://charts.jetstack.io
     condition: certmanager.enabled
   - name: opentelemetry-operator-crds
-    version: "0.0.1"
+    version: 0.0.1
     alias: operatorcrds
     condition: operatorcrds.install
   - name: opentelemetry-operator

helm-charts/splunk-otel-collector/values.schema.json

@@ -1424,6 +1424,24 @@
       "type": "boolean",
       "deprecated": true
     },
+    "operatorcrds": {
+      "description": "Deploys Operator related CRDs from https://github.com/open-telemetry/opentelemetry-operator/tree/main/config/crd via a Helm subchart and using the chart crds/ directory.",
+      "type": "object",
+      "properties": {
+        "install": {
+          "type": "boolean"
+        }
+      }
+    },
+    "operator-crds": {
+      "description": "Configuration for operator-crds (legacy). Prefer using 'operatorcrds' for new installations. Deploys Operator related CRDs from https://github.com/open-telemetry/opentelemetry-operator/tree/main/config/crd via a Helm subchart and using the chart crds/ directory.",
+      "type": "object",
+      "properties": {
+        "install": {
+          "type": "boolean"
+        }
+      }
+    },
     "operator": {
       "description": "OpenTelemetry Operator configuration. A subchart that is used to install the operator, see https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-operator/values.schema.json for more info.",
       "type": "object",
@@ -1684,24 +1702,6 @@
         }
       }
     },
-    "operatorcrds": {
-      "description": "Deploys Operator related CRDs from https://github.com/open-telemetry/opentelemetry-operator/tree/main/config/crd via a Helm subchart and using the chart crds/ directory.",
-      "type": "object",
-      "properties": {
-        "install": {
-          "type": "boolean"
-        }
-      }
-    },
-    "operator-crds": {
-      "description": "Configuration for operator-crds (legacy). Prefer using 'operatorcrds' for new installations. Deploys Operator related CRDs from https://github.com/open-telemetry/opentelemetry-operator/tree/main/config/crd via a Helm subchart and using the chart crds/ directory.",
-      "type": "object",
-      "properties": {
-        "install": {
-          "type": "boolean"
-        }
-      }
-    },
     "certmanager": {
       "description": "cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates.",
       "type": "object",

helm-charts/splunk-otel-collector/values.yaml

@@ -1174,9 +1174,10 @@ service:
 # Full list of Helm value configurations: https://artifacthub.io/packages/helm/opentelemetry-helm/opentelemetry-operator?modal=values
 ################################################################################
 
-# Should the CRDs be installed by a chart's crd/ directory or by using templates.
+# Specify whether the chart should install CRDs automatically.
+# Related Documentation: https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/auto-instrumentation-install.md#crd-management
 operatorcrds:
-  # Should the CRDs be installed
+  # Set to true to install CRDs automatically, or false to manage them manually.
   install: false
 
 operator: