Provide ability to generate and evaluate coverage details for experiments, and which variants have been run.

[jejacks0n]

Summary: In tests, we should collect and render the structure of experiments that have been run, how they might be nested, and which variants have been run (and not run) for a given experiment in each of the paths it's been run within.

Background:

One of the challenges I've seen is identifying the complex interrelationship that forms when using a lot of feature flags, and/or when they get nested. If you have a way to toggle feature flags, and aren't evaluating what other feature flags might be effected by toggling one on or off, you can start to have inadvertent impacts without understanding them.

Posed as a question: Is there a way to see if all of the branches of feature flag(s) are evaluated? And when a feature flag is toggled, do you understand which new feature flags are going to be checked, and which old feature flags are not going to be checked?

Coverage reports can generate some of this information, but involve inspecting them to understand which lines of code, and which branches of a feature flag aren't evaluated well. Sometimes they're deeply nested somewhere that's not obviously related, and so how can we help surface information about this challenge?

Before we try to illustrate this, let's clarify the words control, and treatment. A control is nothing, or no change. In a feature flag, this would be the false branch (not active), and the treatment is the new thing or change (when the feature flag is true).

Now that we have names for these branches let's take this relatively simple example of feature flag branching:

if Feature.enabled?(:my_feature_flag) # treatment
  if Feature.enabled?(:my_nested_feature_flag) # nested_treatment
    "treatment:nested_treatment"
  else # nested_control
    "treatment:nested_control"
  end
else # control
  "control"
end

It's a contrived example, for sure, but is useful to illustrate the tests to cover this relatively simple scenario, which can be even more complex if the nested feature flag is duplicated in the treatment and control branches.

context "when my_feature_flag is active" do
  context "when my_nested_feature_flag is active" do
  end
 
  context "when my_nested_feature_flag is inactive" do
  end
end

context "when my_feature_flag is inactive" do
end

When you start to consider what it looks like to have 5 or 6 feature flags used in rendering a page you start to understand the complex interrelationships that can form and how difficult it can be to identify and understand the interrelationship between them.

It's hard to know if any of the branches aren't being hit, and it gets even worse when these lines aren't in the same file or even section of the application -- for instance, if :my_feature_flag is in a controller and :my_nested_feature_flag is in a partial somewhere. Turning on :my_feature_flag will all of a sudden have an interrelationship with :my_nested_feature_flag that wasn't identified or understood.

With Active Experiments we actually have the opportunity to collect and provide these insights, and we already get a warning in our logs when nesting experiments:

class MyExperiment < ActiveExperiment::Base
  control { "control" }
  variant(:treatment) { "treatment:" + MyNestedExperiment.run }
end

class MyNestedExperiment < ActiveExperiment::Base
  control { "nested_control" }
  variant(:treatment) { "nested_treatment" } 
end

MyExperiment.run

MyNestedExperiment[2788c76e]  Nesting experiment in MyExperiment[230c4214]

Let's collect these details (and more) and allow them to be displayed or inspected at the end of a test run somehow.