Skip to content

Commit

Permalink
Update to Package Spec v3 and corpora generator 0.7.0 (#1508)
Browse files Browse the repository at this point in the history 
Co-authored-by: Andrea Spacca <[email protected]>
  • Loading branch information
@jsoriano
jsoriano and Andrea Spacca authored Oct 18, 2023
1 parent 97303ff commit e2863c9
Show file tree
Hide file tree
Showing 11 changed files with 414 additions and 91 deletions.
4 changes: 2 additions & 2 deletions docs/howto/sample_metric.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -630,7 +630,7 @@
},
"parameter": {
"package": "system_benchmarks",
"description": "Benchmark 20MiB of data ingested",
"description": "Benchmark 20000 events ingested",
"version": "999.999.999",
"policy_template": "testpo",
"input": "filestream",
Expand All @@ -648,7 +648,7 @@
"wait_for_data_timeout": 60000000000,
"corpora": {
"generator": {
"size": "20MiB",
"total_events": "20000",
"template": {
"raw": "",
"path": "./logs-benchmark/template.log",
Expand Down
54 changes: 27 additions & 27 deletions docs/howto/system_benchmarking.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -227,43 +227,43 @@ want to benchmark. There can be multiple scenarios defined for the same package.
The `<scenario>.yml` files allow you to define various settings for the benchmark scenario
along with values for package and data stream-level variables. These are the available configuration options for system benchmarks.
| Option | Type | Required | Description |
|---|---|---|---|
| package | string | | The name of the package. If omitted will pick the current package, this is to allow for future definition of benchmarks outside of the packages folders. |
| description | string | | A description for the scenario. |
| version | string | | The version of the package to benchmark. If omitted will pick the current version of the package. |
| policy_template | string | | The policy template to test. If omitted will pick the first one. |
| input | string | yes | Input type to test (e.g. logfile, httpjson, etc). Defaults to the input used by the first stream in the data stream manifest. |
| vars | dictionary | | Package level variables to set (i.e. declared in `$package_root/manifest.yml`). If not specified the defaults from the manifest are used. |
| data_stream.name | string | yes | The data stream to benchmark. |
| data_stream.vars | dictionary | | Data stream level variables to set (i.e. declared in `package_root/data_stream/$data_stream/manifest.yml`). If not specified the defaults from the manifest are used. |
| warmup_time_period | duration | | Warmup time period. All data prior to this period will be ignored in the benchmark results. |
| benchmark_time_period | duration | | Amount of time the benchmark needs to run for. If set the benchmark will stop after this period even though more data is still pending to be ingested. |
| wait_for_data_timeout | duration | | Amount of time to wait for data to be present in Elasticsearch. Defaults to 10m. |
| corpora.generator.size | string | | String describing the amount of data to generate. Example: `20MiB` |
| corpora.generator.template.raw | string | | Raw template for the corpus generator. |
| corpora.generator.template.path | string | | Path to the template for the corpus generator. If a `path` is defined, it will override any `raw` template definition. |
| corpora.generator.template.type | string | | Type of the template for the corpus generator. Default `placeholder`. |
| corpora.generator.config.raw | dictionary | | Raw config for the corpus generator. |
| corpora.generator.config.path | string | | Path to the config for the corpus generator. If a `path` is defined, it will override any `raw` config definition. |
| corpora.generator.fields.raw | dictionary | | Raw fields for the corpus generator. |
| corpora.generator.fields.path | string | | Path to the fields for the corpus generator. If a `path` is defined, it will override any `raw` fields definition. |
| corpora.input_service.name | string | | Name of the input service to use (defined in the `deploy` folder). |
| corpora.input_service.signal | string | | Signal to send to the input service once the benchmark is ready to start. |
| Option | Type | Required | Description |
|---------------------------------|------------|---|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| package | string | | The name of the package. If omitted will pick the current package, this is to allow for future definition of benchmarks outside of the packages folders. |
| description | string | | A description for the scenario. |
| version | string | | The version of the package to benchmark. If omitted will pick the current version of the package. |
| policy_template | string | | The policy template to test. If omitted will pick the first one. |
| input | string | yes | Input type to test (e.g. logfile, httpjson, etc). Defaults to the input used by the first stream in the data stream manifest. |
| vars | dictionary | | Package level variables to set (i.e. declared in `$package_root/manifest.yml`). If not specified the defaults from the manifest are used. |
| data_stream.name | string | yes | The data stream to benchmark. |
| data_stream.vars | dictionary | | Data stream level variables to set (i.e. declared in `package_root/data_stream/$data_stream/manifest.yml`). If not specified the defaults from the manifest are used. |
| warmup_time_period | duration | | Warmup time period. All data prior to this period will be ignored in the benchmark results. |
| benchmark_time_period | duration | | Amount of time the benchmark needs to run for. If set the benchmark will stop after this period even though more data is still pending to be ingested. |
| wait_for_data_timeout | duration | | Amount of time to wait for data to be present in Elasticsearch. Defaults to 10m. |
| corpora.generator.total_events | uint64 | | Number of total events to generate. Example: `20000` |
| corpora.generator.template.raw | string | | Raw template for the corpus generator. |
| corpora.generator.template.path | string | | Path to the template for the corpus generator. If a `path` is defined, it will override any `raw` template definition. |
| corpora.generator.template.type | string | | Type of the template for the corpus generator. Default `placeholder`. |
| corpora.generator.config.raw | dictionary | | Raw config for the corpus generator. |
| corpora.generator.config.path | string | | Path to the config for the corpus generator. If a `path` is defined, it will override any `raw` config definition. |
| corpora.generator.fields.raw | dictionary | | Raw fields for the corpus generator. |
| corpora.generator.fields.path | string | | Path to the fields for the corpus generator. If a `path` is defined, it will override any `raw` fields definition. |
| corpora.input_service.name | string | | Name of the input service to use (defined in the `deploy` folder). |
| corpora.input_service.signal | string | | Signal to send to the input service once the benchmark is ready to start. |
Example:
`logs-benchmark.yml`
```yaml
---
description: Benchmark 100MiB of data ingested
description: Benchmark 100000 events ingested
input: filestream
vars: ~
data_stream.name: test
data_stream.vars.paths:
- "{{SERVICE_LOGS_DIR}}/corpus-*"
warmup_time_period: 10s
corpora.generator.size: 100MiB
corpora.generator.total_events: 100000
corpora.generator.template.path: ./logs-benchmark/template.log
corpora.generator.config.path: ./logs-benchmark/config.yml
corpora.generator.fields.path: ./logs-benchmark/fields.yml
Expand Down Expand Up @@ -322,7 +322,7 @@ elastic-package benchmark system --benchmark logs-benchmark -v
│ info │
├──────────────┬──────────────────────────────────────┤
│ benchmark │ logs-benchmark │
│ description │ Benchmark 100MiB of data ingested │
│ description │ Benchmark 100000 events ingested │
│ run ID │ d2960c04-0028-42c9-bafc-35e599563cb1 │
│ package │ system_benchmarks │
│ start ts (s) │ 1682320355 │
Expand All @@ -339,7 +339,7 @@ elastic-package benchmark system --benchmark logs-benchmark -v
│ warmup time period │ 10s │
│ benchmark time period │ 0s │
│ wait for data timeout │ 0s │
│ corpora.generator.size │ 100MiB │
│ corpora.generator.total_events │ 100000 │
│ corpora.generator.template.path │ ./logs-benchmark/template.log │
│ corpora.generator.template.raw │ │
│ corpora.generator.template.type │ │
Expand Down
19 changes: 10 additions & 9 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@ require (
github.com/cbroglie/mustache v1.4.0
github.com/cespare/xxhash/v2 v2.2.0
github.com/dustin/go-humanize v1.0.1
github.com/elastic/elastic-integration-corpus-generator-tool v0.5.0
github.com/elastic/elastic-integration-corpus-generator-tool v0.7.0
github.com/elastic/go-elasticsearch/v7 v7.17.10
github.com/elastic/go-licenser v0.4.1
github.com/elastic/go-resource v0.1.1
github.com/elastic/go-ucfg v0.8.6
github.com/elastic/package-spec/v3 v3.0.0-rc1
github.com/elastic/package-spec/v3 v3.0.0
github.com/fatih/color v1.15.0
github.com/go-git/go-billy/v5 v5.5.0
github.com/go-git/go-git/v5 v5.9.0
Expand All @@ -28,7 +28,7 @@ require (
github.com/magefile/mage v1.15.0
github.com/mholt/archiver/v3 v3.5.1
github.com/olekukonko/tablewriter v0.0.5
github.com/pmezard/go-difflib v1.0.0
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
github.com/shirou/gopsutil/v3 v3.23.9
github.com/spf13/cobra v1.7.0
github.com/stretchr/testify v1.8.4
Expand Down Expand Up @@ -63,7 +63,7 @@ require (
github.com/cloudflare/circl v1.3.3 // indirect
github.com/creasty/defaults v1.7.0 // indirect
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dnephin/pflag v1.0.7 // indirect
github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5 // indirect
github.com/elastic/gojsonschema v1.2.1 // indirect
Expand Down Expand Up @@ -95,15 +95,15 @@ require (
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/huandu/xstrings v1.4.0 // indirect
github.com/imdario/mergo v0.3.15 // indirect
github.com/imdario/mergo v0.3.16 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/klauspost/compress v1.16.0 // indirect
github.com/klauspost/compress v1.17.0 // indirect
github.com/klauspost/pgzip v1.2.5 // indirect
github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
Expand Down Expand Up @@ -136,7 +136,8 @@ require (
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/shopspring/decimal v1.3.1 // indirect
github.com/skeema/knownhosts v1.2.0 // indirect
github.com/spf13/cast v1.5.0 // indirect
github.com/spf13/afero v1.10.0 // indirect
github.com/spf13/cast v1.5.1 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stretchr/objx v0.5.1 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
Expand All @@ -154,14 +155,14 @@ require (
golang.org/x/exp/typeparams v0.0.0-20221208152030-732eee02a75a // indirect
golang.org/x/mod v0.13.0 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/oauth2 v0.8.0 // indirect
golang.org/x/oauth2 v0.12.0 // indirect
golang.org/x/sync v0.4.0 // indirect
golang.org/x/sys v0.13.0 // indirect
golang.org/x/term v0.13.0 // indirect
golang.org/x/text v0.13.0 // indirect
golang.org/x/time v0.3.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/protobuf v1.30.0 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
Expand Down
Loading

0 comments on commit e2863c9

Please sign in to comment.