Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[system tests] [transforms] Support commas in the source indices definition #2388

Merged
merged 4 commits into from
Feb 5, 2025

Conversation

mrodm
Copy link
Contributor

@mrodm mrodm commented Feb 4, 2025

Relates #2341

Support to check transforms which source indices definition contains a list of comma-separated strings.

For instance (link)

source:
  index: "metrics-elasticsearch.stack_monitoring.index*,.monitoring-es-8*,metricbeat-*"

elastic-package needs to check each source index defined in the transform independently to validate whether or not it matches with the data stream tested.

Notes:

  • Discovered that Fleet creates the transforms with logs- prefix
  • Added a configuration to skip the validation of transforms (required for elasticsearch package).
    • In 8.10.1, the destination index is not created:
      • Even, if the destination index is created beforehand, it fails with the same error.
      • Probably related to the aggregations ? link
      • Error: "Source indices have been deleted or closed."
    • It looks like that the filters are too restrictive for the documents ingested in system test.
      Error: "must specify at least one document in [docs]"

@mrodm
Copy link
Contributor Author

mrodm commented Feb 4, 2025

test integrations

@elastic-vault-github-plugin-prod

Created or updated PR in integrations repository to test this version. Check elastic/integrations#12604

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

cc @mrodm

Comment on lines +2060 to +2063
if config.SkipTransformValidation {
return nil
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the errors found in the elasticsearch package from integrations (see PR description) when testing to check each index independently, it looks like it would be needed to add this kind of setting in the system tests.

@mrodm mrodm marked this pull request as ready for review February 5, 2025 10:27
@mrodm mrodm requested a review from a team February 5, 2025 10:27
@mrodm mrodm merged commit b7445a2 into elastic:main Feb 5, 2025
3 checks passed
@mrodm mrodm deleted the support_commas_indices_transform branch February 5, 2025 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants