Nested variable rendering when using file-type parameters

[reinhard-brandstaedter]

Hello,
I'm trying to make my config better readable as having DQL queries directly in JSON templates is really badly human readable.
So I have moved from having DQL in json like this (long complex newline/escaped string):

    {
      "name": "Check for Ingest Indices",
      "description": "When Ingest Indices are present instead of aliases, this means that something went wrong during the data ingestion process",
      "objectiveType": "DQL",
      "dqlQuery": "fetch bizevents\n| filter event.type == \"SuggestSearchService-Report\"\n| filter rail == \"{{ .rail }}\"\n| limit 1\n| parse report, \"JSON:json\"\n| fieldsKeep event, json, timestamp\n| fieldsFlatten json\n| fieldsAdd indicesWithIngest = json.indicesWithIngest\n| expand indicesWithIngest\n| filter isNotNull(indicesWithIngest)\n| summarize count = count()\n| fieldsAdd count=if(isNull(count),0,else: count)",
      "comparisonOperator": "LESS_THAN_OR_EQUAL",
      "target": 0.0
    }

to externalizing it via a single file, referenced in the config.yaml:

configs:
- id: srg_suggest_search
    parameters:
      rail: webshop
      locales: 
        type: value
        value: 
          - cs_cz
          - de_ch
          - hr_hr
      check_for_ingest_indices:
        type: file
        path: dql/check_for_ingest_indices.dql
        references:
          - rail

This works fine. First I even did with a compound type and have the DQL inline in the config.yaml. I can then reference the DQL in the json template like this, which is much cleaner:

{
      "name": "Check for Ingest Indices",
      "description": "When Ingest Indices are present instead of aliases, this means that something went wrong during the data ingestion process",
      "objectiveType": "DQL",
      "dqlQuery": "{{ .check_for_ingest_indices }}",
      "comparisonOperator": "LESS_THAN_OR_EQUAL",
      "target": 0.0
    }

However now comes the tricky part. When I want to use a template iteration in the json template that should replace a placeholder in the DQL file content (which wasn't replaced by the initial parameter rendering, this doesn't work:

{{- range .locales }}
    {
      "name": "Check for indices - locale: {{ . }}",
      "description": "This objective checks if the number of documents indexed has significantly fallen",
      "objectiveType": "DQL",
      "dqlQuery": "{{ $.check_for_ingest_indices }}",
      "comparisonOperator": "GREATER_THAN_OR_EQUAL",
      "autoAdaptiveThresholdEnabled": true
    }
{{- end }}

This is rendered into an objective for every locale in the list, but the content of the file (which also contains placeholders like "{{ . }}") is not rendered anymore.
I know go template doesn't support nested placeholders, but has anyone found a workaround for this usecase?

[Laubi]

Hi @reinhard-brandstaedter, I was not able to reproduce your issue, but I made a small change to the loop you're using, adding commas to the JSON array using {{if $index}},{{end}}.

I am using the following setup:

template.yaml

{
  "name" : "Test",
  "description" : "My description",
  "tags" : [ ],
  "variables" : [ ],
  "objectives" : [
    {{- range $index, $element :=  .locales }}
    {{if $index}},{{end}}
    {
      "name": "Check for indices - locale: {{ . }}",
      "description": "This objective checks if the number of documents indexed has significantly fallen",
      "objectiveType": "DQL",
      "dqlQuery": "{{ $.check_for_ingest_indices }}",
      "comparisonOperator": "GREATER_THAN_OR_EQUAL",
      "autoAdaptiveThresholdEnabled": true
    }
    {{- end }}
  ]
}

config.yaml

configs:
- id: guardian
  type:
    settings:
      schema: app:dynatrace.site.reliability.guardian:guardians
      scope: tenant
  config:
    template: template.json
    parameters:
      rail: webshop
      locales:
        type: value
        value:
        - en
        - de
      check_for_ingest_indices:
        type: file
        path: dql.txt
        references:
        - rail

dql.txt

fetch bizevents
| filter event.type == "SuggestSearchService-Report"
| filter rail == "{{ .rail }}"
| limit 1
| parse report, "JSON:json"
| fieldsKeep event, json, timestamp
| fieldsFlatten json
| fieldsAdd indicesWithIngest = json.indicesWithIngest
| expand indicesWithIngest
| filter isNotNull(indicesWithIngest)
| summarize count = count()
| fieldsAdd count=if(isNull(count),0,else: count)

This results in the following SRGs:

Could you please provide a reproducible sample of the issue you are facing so that we can investigate further?

[reinhard-brandstaedter]

Hi @Laubi ,
thanks for your example!
I was able to debug that a little bit further and found the potential issue. My dql file also has another placeholder ( {{ . }} ), which

query.dql

fetch bizevents
| filter event.type == "SuggestSearchService-Report" and rail == "{{ .rail }}"
...
| filter indexName == "products-{{ .rail }}-{{ . }}-search"
...

template.json

    {{- range $index, $element :=  .locales }}
    {{if $index}},{{end}}
    {
      "name": "Check for minimum search documents - categories - {{ . }}",
      "description": "This objective checks if the number of documents indexed has significantly fallen",
      "objectiveType": "DQL",
      "dqlQuery": "{{ $.check_for_minimum_search_documents_categories }}",
      "comparisonOperator": "GREATER_THAN_OR_EQUAL",
      "autoAdaptiveThresholdEnabled": true
    },
    {
      "name": "Check for minimum search documents - products - {{ . }}",
      "description": "This objective checks if the number of documents indexed has significantly fallen",
      "objectiveType": "DQL",
      "dqlQuery": "{{ $.check_for_minimum_search_documents_products }}",
      "comparisonOperator": "GREATER_THAN_OR_EQUAL",
      "autoAdaptiveThresholdEnabled": true
    },
    {
      "name": "Check for minimum search documents - keywords - {{ . }}",
      "description": "This objective checks if the number of documents indexed has significantly fallen",
      "objectiveType": "DQL",
      "dqlQuery": "{{ $.check_for_minimum_search_documents_keywords }}",
      "comparisonOperator": "GREATER_THAN_OR_EQUAL",
      "autoAdaptiveThresholdEnabled": true
    }
    {{- end }}

This creates a recursive (?) invalide payload inside the rendered JSON template which fails when posting to the API. Instead of the locale variable the placeholder is replaced with the whole map during the file reference rendering:

So what I was wondering is if it would be possible to achieve this by:

• first render the dql file via the file type parameter and only its reference variables (only rail in my case)
• then when the json template is rendered, with the content of the file, render this (json escaped string) again with the values of the range statement

[Laubi]

Why are you using {{ . }}? What do you try to achieve?

[reinhard-brandstaedter]

My intention:

• replace the {{ .rail }} in the file DQL with the reference variable from the config.yaml (this is rendered upon the "import" via the file type parameter)
• for every locale/rail combination create one objective, where the locale is part of the DQL query (this is/should be rendered via the template.json range iteration)

So in the end I should have in the json:

{
      "name": "Title - de",
      ...
      "dqlQuery": "fetch bizevents | filter rail == "webshop" | filter locale == "de",
},
{
      "name": "Title - at",
      ...
      "dqlQuery": "fetch bizevents | filter rail == "webshop" | filter locale == "at",
}

[Laubi]

Ok I understand now, thanks for the clarification.
Sadly, this is currently not possible, but thanks for the feedback. I am not sure if we can solve this issue short term, but I've noted it down for us to have a look at.

[reinhard-brandstaedter]

@Laubi just a thought:
Rendering everything in one place (the file type reference) and then including the rendered content as is in the json template will also not work, right?
Because the file type parameter would be all JSON string escaped, I guess?