100.00 Error Count and 1.00 Error Rate for Simple Experiment

[shubhamgoenka]

Describe the bug
I'm trying to add a simple experiment as a GitHub action to load test an HTTP service, but any basic URL that I try gives me an error-count of 100.00 and an error-rate of 1.00.
Apart from the actual URL I am trying to test, I also tried with https://httpbin.org/get, http://127.0.0.1/get, and https://www.google.com, but I got the same error values.

What is interesting is that the other values (e.g.: latency) seem to look okay to me.

To Reproduce
Monitor runs at https://github.com/ordrsmart/ordrsmart-public/actions

Expected behavior
The GitHub action job should pass with an error-count of 0 and an error-rate of 0.

Screenshots

Desktop (please complete the following information):

• OS: MacOS (but running on Ubuntu as part of the GitHub action)
• Output of the iter8 version command

  Run iter8-tools/iter8-action@v1
    with:
      chart: load-test-http
      valuesFile: loadtest-config.yaml
      validateSLOs: true
      logLevel: info
  /usr/bin/docker run --name a6c8ca57471[2](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:2)8b61e4cafbcd613c55dca680f_06a47d --label 2a6c8c --workdir /github/workspace --rm -e "INPUT_CHART" -e "INPUT_VALUESFILE" -e "INPUT_VALIDATESLOS" -e "INPUT_LOGLEVEL" -e "INPUT_CHARTREPO" -e "INPUT_CHARTVERSION" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/ordrsmart-public/ordrsmart-public":"/github/workspace" 2a6c8c:a5747128b61e4cafbcd613c55dca680f
  Verify version of Iter8
  cmd.BuildInfo{Version:"v0.9+unreleased", GitCommit:"", GoVersion:"go1.17.8"}
  Identify loglevel if set
  Identify chart repository
  Identify any chartVersionConstraint file
  Identify values file
  Calling: /bin/iter8 launch -c load-test-http  -f loadtest-config.yaml  -l info
  time=2023-06-23 10:52:37 level=info msg=pulling load-test-http
  time=2023-06-23 10:52:38 level=info msg=created experiment.yaml file
  time=2023-06-23 10:52:38 level=info msg=starting local experiment
  time=2023-06-23 10:52:38 level=info msg=task 1: gen-load-and-collect-metrics-http : started
  time=2023-06-23 10:52:50 level=info msg=task 1: gen-load-and-collect-metrics-http : completed
  time=2023-06-23 10:52:50 level=info msg=task 2: assess-app-versions : started
  time=2023-06-23 10:52:50 level=info msg=task 2: assess-app-versions : completed
  # task 1: generate HTTP requests for application URL
  # collect Iter8's built-in HTTP latency and error-related metrics
  - task: gen-load-and-collect-metrics-http
    with:
      versionInfo:
      - url: https://httpbin.org/get
  # task 2: validate service level objectives for app using
  # the metrics collected in the above task
  - task: assess-app-versions
    with:
      SLOs:
      - metric: http/error-rate
        upperLimit: 0
  Log benchmarks
  
  Experiment summary:
  *******************
  
    Experiment completed: true
    No task failures: true
    Total number of tasks: 2
    Number of completed tasks: 2
  
  Whether or not service level objectives (SLOs) are satisfied:
  *************************************************************
  
    SLO Conditions       |Satisfied
    --------------       |---------
    http/error-rate <= 0 |false
  
  
  Latest observed values for metrics:
  ***********************************
  
    Metric                     |value
    -------                    |-----
    http/error-count           |100.00
    http/error-rate            |1.00
    http/latency-max (msec)    |274.09
    http/latency-mean (msec)   |151.22
    http/latency-min (msec)    |1[3](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:3)7.6[4](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:4)
    http/latency-p[5](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:5)0 (msec)    |148.12
    http/latency-p75 (msec)    |155.94
    http/latency-p90 (msec)    |1[6](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:6)5.00
    http/latency-p95 (msec)    |1[7](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:8)7.50
    http/latency-p99 (msec)    |26[8](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:9).06
    http/latency-p[9](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:10)9.9 (msec)  |273.48
    http/latency-stddev (msec) |25.41
    http/request-count         |[10](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:11)0.00
  
  Experiment completed
  Asserting SLOs satisfied
  time=[20](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:21)[23](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:24)-06-23 10:52:[51](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:52) level=info msg=experiment completed
  time=2023-06-23 10:[52](https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355593840/jobs/9714056350#step:4:53):51 level=info msg=experiment has no failure
  time=2023-06-23 10:52:51 level=info msg=SLOs are not satisfied
  time=2023-06-23 10:52:51 level=info msg=not all conditions were satisfied
  time=2023-06-23 10:52:51 level=error msg=assert conditions failed
  time=2023-06-23 10:52:51 level=error msg=assert conditions failed
  

Additional context
Example of failed log: https://github.com/ordrsmart/ordrsmart-public/actions/runs/5355447968/jobs/9713740255
YAML file containing GitHub action configuration that runs iter8: https://github.com/ordrsmart/ordrsmart-public/blob/main/.github/workflows/http-load-testing-staging.yaml

[kalantar]

@shubhamgoenka it looks like you were using a very old version of Iter8 (v0.9). Since that time, the github install action has been deprecated. Instead you should be able to use the recommended helm commands directly in your workflow. See, for example, https://iter8.tools/0.18/getting-started/first-performance/.

Is this an option for your use case?