Add more details about metrics, pod startup testing

keps/sig-storage/20190204-scale-testing.md

@@ -14,11 +14,10 @@ approvers:
   - "@wojtek-t"
 editor: TBD
 creation-date: 2019-02-04
-last-updated: 2019-02-04
+last-updated: 2019-02-14
 status: provisional
 see-also:
 - https://github.com/kubernetes/community/blob/master/sig-scalability/slos/pod_startup_latency.md
-- TODO link to stateful pod SLI
 - https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/design.md
 replaces:
 superseded-by:
@@ -33,16 +32,20 @@ superseded-by:
    * [Goals](#goals)
    * [Non-Goals](#non-goals)
 * [Proposal](#proposal)
-   * [WIP Test Environment](#wip-test-environment)
+   * [Test Framework](#test-framework)
+   * [Test Rollout](#test-rollout)
+   * [Test Portability](#test-portability)
    * [Test Cases](#test-cases)
       * [Pod Startup](#pod-startup)
+   * [WIP Future Test Cases](#wip-future-test-cases)
       * [Pod Teardown](#pod-teardown)
-      * [WIP: PV Binding Tests](#wip-pv-binding-tests)
-      * [WIP: PV Provisioning Tests](#wip-pv-provisioning-tests)
-      * [WIP: PV Deletion Tests](#wip-pv-deletion-tests)
+      * [PV Binding Tests](#pv-binding-tests)
+      * [PV Provisioning Tests](#pv-provisioning-tests)
+      * [PV Deletion Tests](#pv-deletion-tests)
 * [Graduation Criteria](#graduation-criteria)
    * [Phase 1](#phase-1)
    * [Phase 2](#phase-2)
+   * [Phase 3](#phase-3)
 * [Implementation History](#implementation-history)
 
 
@@ -89,25 +92,38 @@ The tests should be developed and run in sig-scalability’s test
 [Cluster Loader framework](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/README.md)
 and infrastructure.
 
-The framework already supports measurements for Pod startup latency and
-scheduling metrics. More measurements can be added to get finer-grained details
-for volume operations:
+The framework already supports measurements for Pod startup latency and can be
+used to determine a [Pod startup SLO with
+volumes](https://github.com/kubernetes/community/pull/3242).
 
-* Gather [volume operation
+Additional measurements can be added to the framework to get informational details for
+for volume operations. These are not strictly required but are nice to have:
+
+* Gather existing [volume operation
   metrics](https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/metrics.go)
-  from kube-controller-manager
+  from kube-controller-manager and kubelet.
   * Caveat: These metrics measure the time of a single operation attempt and not
     the e2e time across retries. These metrics may not be suitable to measure
     latency from a pod's perspective.
-* Measure time starting from pod scheduling to volume operation completion times
-  for provision, attach and mount. This requires changes in Kubernetes to add
-  timestamps in PersistentVolumeClaim, VolumeAttachment, and Pod statuses,
-  respectively.
-  * TODO: how to measure time reliably for deletion events, where the API object
-    deletion is the trigger?
 * Enhance the scheduler metrics collection to include
   [volume scheduling
   metrics](https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/persistentvolume/scheduler_bind_cache_metrics.go)
+* Timing of volume events. However, events should not be considered a reliable
+  measurement because they can be dropped or garbage collected. Also there are
+  only events for provision/delete, attach/detach. Mount events were removed due
+  to the high frequency of remount operations for API volume types (secrets,
+  etc).
+
+Enhancements to Kubernetes itself can also be considered not just for scale
+testing but for general improvements to supportability and debuggability of the
+system. These are not strictly required and can be considered as a future
+enhancement:
+
+* Add e2e metrics for volume operations that can track the time across operation
+  retries.
+* Add timestamps to the appropriate API object status whenever an operation is
+  completed.
+
 
 ### Test Rollout
 
@@ -142,14 +158,34 @@ Therefore these tests have the following prerequisites:
 
 #### Pod Startup
 
-These tests should measure how long it takes to start up a pod with volumes, assuming that the volume has already been provisioned.
+These tests should measure how long it takes to start up a pod with unique volumes,
+assuming that the volumes have already been provisioned and volumes are not
+shared between pods.
 
-For each volume type:
+The requirement for non-shared pods is so that we can produce an initial
+consistent baseline measurement. The performance of pod startup latency when using
+shared volumes has a lot of variable factors such as volume type (RWO vs RWX),
+and whether or not the scheduler decided to schedule a replacement pod on the
+same or different node. We can consider adding more test cases in the future
+that can handle this scenario once we establish an initial baseline.
+
+For the initial baseline, the test case can be run for each volume type by changing
+various dimensions (X):
+
+* Create 1 pod with X volumes and 1 node. This can help determine a limit for
+  max number of volumes in a single Pod.
+* Create X pods with 1 volume each on 1 node in parallel. This can help determine a limit
+  for max number of volumes on a single node.
+* Create X pods with 1 volume each in parallel.
 
-* Create many pods with 1 volume each in parallel.
-* Create 1 pod with many volumes.
-  * For PVC types, test with and without PV.NodeAffinity as one more dimension.
-  * Measure pod startup time, with a breakdown of time spent in volume scheduling, attach and mount operations.
+All the test cases should measure pod startup time. A breakdown of time spent in volume scheduling,
+attach and mount operations is not strictly required but nice to have.
+
+
+### WIP Future Test Cases
+
+These test cases are still under development and need to be refined before they
+can be implemented in the future.
 
 #### Pod Teardown
 
@@ -162,11 +198,11 @@ For each volume type:
   * Measure pod deletion time, with a breakdown of time spent in unmount and detach operations.
     * Note: Detach can only be measured for CSI volumes by the removal of the VolumeAttachment object.
 
-#### WIP: PV Binding Tests
+#### PV Binding Tests
 
 These tests should measure the time it takes to bind a PVC to a preprovisioned available PV.
 
-#### WIP: PV Provisioning Tests
+#### PV Provisioning Tests
 
 These tests should measure the time it takes to bind a PVC to a dynamically provisioned PV.
 
@@ -176,7 +212,7 @@ For each volume type that supports provisioning:
 * Create many PVCs with delayed binding in parallel.
   * Measure volume provisioning time.
 
-#### WIP: PV Deletion Tests
+#### PV Deletion Tests
 
 These tests should measure the time it takes to delete a PVC and PV.
 
@@ -197,18 +233,12 @@ For each volume type that supports deletion:
 
 * Improve pod startup latency measurements to get finer-grained volume operation
   latencies.
+* Establish SLO for pod startup latency with volumes.
+* Tests fail if results exceeds SLO.
 
 ### Phase 3
 
-* Pod teardown tests with results published.
-* PV provisioning and deletion tests with results published.
-* PV binding tests with results published.
-
-### Phase 4
-
-* Establish SLOs for these operations. This can happen in parallel as each new
-  test becomes stable.
-* Tests fail if results exceeds SLO.
+* Add additional tests for Pod deletion and volume binding/provisioning.
 
 ## Implementation History