[9.0](backport #6602) Add support for variables in outputs and default provider

[mergify]

What does this PR do?

Adds support for context variables (ONLY) in outputs. Adds support for a default provide prefix to be defined for variables (default is env). This provides support for the original ${ES_PASSWORD} when used in outputs to still work as it will automatically map that to ${env.ES_PASSWORD} and it will get resolved for the context variables the same as it was being done by go-ucfg.

This includes an improvement to how variables are observed in the composable controller and how it is used by the coordinator. Now when a set of observable's are passed to the composable controller it will return the current set of variables after the debounce time, this ensures that before the variables are substituted that it is using the latest set of variables. Without this change running would always show an error at first with ${env.ES_PASSWORD} is an unknown variable and then less than a few milliseconds it would find it. This change removes that behavior and is able to find the variable on initial render.

Why is it important?

This allows the ability for all context provides to be able used in the outputs configuration. This is useful for using say the kubernetes_secrets provider in the policy for credentials or the new filesource provider to get a value from a file into the outputs. This is done in a way to not break compatibility with ${ES_PASSWORD} that could be used in outputs when it was rendered by go-ucfg, by the addition of a default provider setting (which defaults to env).

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test (great coverage in unit tests)

Disruptive User Impact

None.

How to test this PR locally

Replace a field in the elastic-agent.yml with a variable like from the environment provider.

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    username: "${ES_USER}"
    password: "${ES_PASSWORD}"
    preset: balanced

Run with the variables.

ES_USER=elastic ES_PASSWORD=password ./elastic-agent run -e

Observe that the ES_USER and ES_PASSWORD are substituted.

Related issues

• Relates Support context variables in the Elastic Agent policy output section #6376

---

This is an automatic backport of pull request #6602 done by [Mergify](https://mergify.com).

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pierrehilbert]

buildkite test this

[blakerouse]

@pierrehilbert I think I might have broken buildkite because the commit message for the change includes ${env.ES_PASSWORD}. I don't think it likes that in the step.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @blakerouse? 🙏

[blakerouse]

buildkite test this

[elastic-sonarqube]

Quality Gate passed

Issues
6 New issues
2 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
82.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube