[Change Proposal] Allow deployment modes to specify allowed inputs

[jen-huang]

Related to deployment modes: #738
Related to deployment modes input configuration: #805

Following discussion in #805, it was decided that Fleet will maintain a blocklist of inputs that are not allowed for agentless deployment modes. This was implemented in elastic/kibana#202091.

In elastic/kibana#211092, we saw that additional inputs needed to be blocked as new agentless integrations became available. This made us consider moving to an allowlist instead of a blocklist, but per investigation in elastic/kibana#211275, the initial allowlist was too restrictive.

With this new information, it seems unsustainable for Fleet to define what is allowed for agentless integrations. We should move this to the package spec instead so that allowed inputs can be controlled by integrations. The existing deployment_modes property seems to be a good place to put it:

...
policy_templates:
  - name: billing
    title: AWS Billing
    description: Collect billing metrics with Elastic Agent
    deployment_modes:
      default:
        enabled: false
      agentless:
        enabled: true
        inputs: # <--- new field
          - aws/metrics
          - aws/s3
...

When inputs is specified under a deployment mode, Fleet should only show configuration for those inputs. If this field is not present, then the default behavior is to show all inputs specified by this policy template.

[jen-huang]

cc @qcorporation @jamiehynds @jsoriano @seanrathier WDYT?

[jsoriano]

On a first look this seems a bit redundant to me. Policy templates define their own inputs, so in principle they are already restricted.

I guess this can be an issue only if a policy template supports multiple deployment modes? In that case, if the policy template is only partially compatible with a deployment mode, wouldn't it be better to split it in two policy templates, so each one of them is fully compatible with the deployment modes they support?

I mean, instead of:

policy_templates:
  - name: billing
    title: AWS Billing
    description: Collect billing metrics with Elastic Agent
    deployment_modes:
      default:
        enabled: false
      agentless:
        enabled: true
        inputs: # <--- new field
          - aws/metrics
          - aws/s3
    inputs:
      - type: aws/metrics
        ...
      - type: aws/s3
        ...
      - type: httpjson
        ...

Do something like this, that is more explicit and already supported, even if may lead to some duplication:

policy_templates:
  - name: billing
    title: AWS Billing
    description: Collect billing metrics with Elastic Agent
    inputs:
      - type: aws/metrics
        ...
      - type: aws/s3
        ...
      - type: httpjson
        ...
  - name: billing_agentless
    title: AWS Billing
    description: Collect billing metrics with Elastic Agent
    deployment_modes:
      default:
        enabled: false
      agentless:
        enabled: true
    inputs:
      - type: aws/metrics
        ...
      - type: aws/s3
        ...

If we don't want this duplication, and we definitely want to add something to explicitly support this, I think it would be better to make this restriction part of the input:

policy_templates:
  - name: billing
    title: AWS Billing
    description: Collect billing metrics with Elastic Agent
    deployment_modes:
      default:
        enabled: false
      agentless:
        enabled: true
    inputs:
      - type: aws/metrics
        ...
      - type: aws/s3
        ...
      - type: httpjson
        deployment_modes: ['default'] <-- New field.
        ...

[jen-huang]

If we don't want this duplication, and we definitely want to add something to explicitly support this, I think it would be better to make this restriction part of the input

That works for me

I guess this can be an issue only if a policy template supports multiple deployment modes? In that case, if the policy template is only partially compatible with a deployment mode, wouldn't it be better to split it in two policy templates, so each one of them is fully compatible with the deployment modes they support?

Technically multiple policy templates can work for this, but we are trying to avoid this kind of duplication. Per feedback from @qcorporation:

My personal opinion would be that it would fit well as a package spec configuration if we can specify the input types at the deployment level, i.e. For Agentless and Agent based inputs we can specify different inputs while continuing to have one policy template.
The reason why we went with the input filtering logic before was because with the current capability we'd have to duplicate policy_templates for the same integration for Agent and Agentless based deployment. That translates to 2 cards within the integration onboarding page which I believe could have confused people.
If we can have one policy_template that would be great!

[jsoriano]

That translates to 2 cards within the integration onboarding page which I believe could have confused people.

Good point 👍