Active Experiment is a framework for defining and running experiments. It supports using a variety of rollout and reporting strategies and/or services.
Experiments can be everything from determining which query has the best performance, to which feature gets the most engagement, to rolling out a canary version of a new api service.
Experimentation is complex. There are a lot of different ways to run experiments, and even more ways to report on them. Active Experiment is designed to be flexible enough to support a variety of use cases, but also to be consistent and easy to use.
Define your experiments using easily testable classes:
class MyExperiment < ActiveExperiment::Base
variant(:red) { "red" }
variant(:blue) { "blue" }
endThis experiment can be generated using the Rails generator:
rails generate experiment my_experiment red blueRun the experiment with a context, like the current user, or the post being rendered:
MyExperiment.run(current_user) # => "red" or "blue"Optionally override the defaults using local scope and helpers:
MyExperiment.run(current_user) do |experiment|
experiment.on(:red) { redirect_to red_path }
experiment.on(:blue) { redirect_to blue_path }
endThat's it! When this experiment is encountered by different users, half†will get the red variant, half will get the blue variant, and each will always get the same.
†roughly half, for the statistically pedantic.
Add this line to your Gemfile:
gem "activeexperiment"Or install the latest version with RubyGems:
gem install activeexperimentSource code can be downloaded as part of the project on GitHub:
Adapters can be added to integrate with various services:
This area provides a high level overview of the tools that more complex experiments can benefit from.
For example, some experiments need to define a default variant (also known as a control) that will be assigned if the experiment is skipped:
class MyExperiment < ActiveExperiment::Base
variant(:red) { "red" }
variant(:blue) { "blue" }
# The term control is simply a convention that means the default variant, and
# any variant can be set as the default with +use_default_variant(:red)+
control { "default" }
endCallbacks can be used to hook into the lifecycle when experiments are run, and can be targeted to when a specific variant has been assigned:
class MyExperiment < ActiveExperiment::Base
control { "default" }
variant(:red) { "red" }
variant(:blue) { "blue" }
# Skipping an experiment will always assign the default variant, which could
# be nothing, but since there's a control defined, it will be used.
before_run { skip if context.admin? }
# Only invoked when the red variant has been assigned.
before_variant(:red) { puts "running the red variant" }
# Maybe there's cleanup or logging to do afterwards?
after_run { puts "run complete with the #{variant} variant" unless skipped? }
endSegment rules can be used to assign specific variants for certain cases:
class MyExperiment < ActiveExperiment::Base
control { "default" }
variant(:red) { "red" }
variant(:blue) { "blue" }
segment :admins, into: :red
segment :old_accounts, into: :control
private
def admins
context.admin?
end
def old_accounts
context.created_at < 1.year.ago
end
endRollouts are a core concept in Active Experiment. They allow specifying how an experiment should be rolled out, and even if it should be skipped or not. For example, the default rollout in Active Experiment is percentage based and accepts distribution rules -- if no rules are provided, even distribution is used.
A rollout can implement any number of different strategies, interact with services, and can be used on a per-experiment basis.
Here's an example of using the default percent rollout with custom distribution rules:
class MyExperiment < ActiveExperiment::Base
variant(:red) { "red" }
variant(:blue) { "blue" }
variant(:green) { "green" }
# Will assign the green variant 80% of the time, red and blue 10% each.
use_rollout :percent, rules: { red: 10, blue: 10, green: 80 }
endProject specific rollouts can be defined and registered too. To illustrate, here's a custom rollout that inherits from the base rollout, uses a fictional feature flag library, and assigns a random variant.
class FeatureFlagRollout < ActiveExperiment::Rollouts::BaseRollout
register_as :feature_flag
def skipped_for(experiment)
!Feature.enabled?(@rollout_options[:flag_name] || experiment.name)
end
def variant_for(experiment)
experiment.variant_names.sample
end
endThis rollout can now be used the same way the built-in rollouts are:
class MyExperiment < ActiveExperiment::Base
variant(:red) { "red" }
variant(:blue) { "blue" }
# Using a custom rollout with options.
use_rollout :feature_flag, flag_name: "my_feature_flag"
endCustom rollouts can be registered to autoload as well, so they're only loaded when needed:
ActiveExperiment::Rollouts.register(
:feature_flag,
"lib/feature_flag_rollout.rb"
)There's a world of flexibility with custom rollouts. One creative and simple rollout concept is to use the experiment itself:
class MyExperiment < ActiveExperiment::Base
variant(:red) { "red" }
variant(:blue) { "blue" }
def self.skipped_for(*)
false
end
def self.variant_for(*)
variant_names.sample
end
use_rollout self
endReporting is a core concept in Active Experiment. It allows for collecting data about experiments and variants, and can be used to track performance metrics, analyze results, and more.
Some simple reporting strategies might simply be added to after_run callbacks, but more complex reporting strategies can be implemented using a subscriber.
A subscriber can be used to listen for experiment events and report them to a service. For example, here's a subscriber that reports to a fictional analytics service:
class MyAnalyticsSubscriber < ActiveSupport::Subscriber
attach_to :active_experiment
def process_run(event)
experiment = event.payload[:experiment]
return if experiment.skipped?
Analytics.report(
experiment.serialize,
error: event.payload[:exception_object]
)
end
endThe following Active Experiment events are available for subscribers:
start_experiment- The experiment has begun.process_segment_callbacks- The experiment has processed all segment rules. A variant may have been resolved through this step.process_variant_steps- An experiment variant has been run.process_variant_callbacks- The experiment has processed variant callbacks.process_run_callbacks- The experiment has processed run callbacks.process_run- The experiment has completed and can be reported on.
In each of these events, the experiment instance is available in the event.payload hash.
Experiments can be used in views, just like in any other part of your application. Sometimes though, you might want to render markup inside your run block too, and to do this, you'll need to "capture" the experiment.
To accomplish this, you can ask the experiment to capture itself by providing the view scope. The following examples (HAML or ERB) help illustrate how to avoid duplicating markup within each variant block by putting it (the container div for instance) in the run block.
Remember to include the ActiveExperiment::Capturable module in your experiment class:
class MyExperiment < ActiveExperiment::Base
include ActiveExperiment::Capturable
variant(:red) { "red" }
variant(:blue) { "blue" }
endExpand HAML example
!= MyExperiment.set(capture: self).run(current_user) do |experiment|
%div.container
= experiment.on(:red) do
%button.red-pill Red
= experiment.on(:blue) do
%button.blue-pill BlueExpand ERB example
<%== MyExperiment.set(capture: self).run(current_user) do |experiment| %>
<div class="container">
<%= experiment.on(:red) do %>
<button class="red-pill">Red</button>
<% end %>
<%= experiment.on(:blue) do %>
<button class="blue-pill">Blue</button>
<% end %>
</div>
<% end %>If you don't need to capture the experiment, simply run like you would anywhere else:
<% MyExperiment.run(current_user) do |experiment| %>
<% experiment.on(:red) do %>
<button class="red-pill">Red</button>
<% end %>
<% experiment.on(:blue) do %>
<button class="blue-pill">Blue</button>
<% end %>
<% end %>While Active Experiment doesn't include any specific tooling for client side experimentation at this time, it does provide the ability to surface experiments in the client layer.
Whenever an experiment is run in the request lifecycle, it's stored so it can be provided to the client. This means that if an experiment is run in controller, a view, a helper, etc. it will be available to the client.
In the layout, the experiment data can be rendered as JSON for instance:
<title>My App</title>
<script>
window.experiments = <%== ActiveExperiment::Executed.to_json %>
</script>Or each experiment can be iterated over and rendered individually:
<% ActiveExperiment::Executed.as_array.each do |experiment| %>
<meta name="<%= experiment.name %>" content="<%== experiment.serialize.to_json %>">
<% end %>Active Experiment provides a test helper that can be used to stub experiments and assert that the expected experiments have been run.
To use the test helper, include it in your test case:
class MyTestCase < ActiveSupport::TestCase
include ActiveExperiment::TestHelper
endNow you can stub experiments in your tests:
test "stubbing experiments" do
stub_experiment(MyExperiment, :red) do
# Now all MyExperiment experiments will assign the :red variant.
end
stub_experiment(MyExperiment, skip: true) do
# Now all MyExperiment experiments will be skipped.
end
endAssertion helpers are also available:
test "asserting experiments" do
# Assert that no experiments have been run.
assert_no_experiments
MyExperiment.run(id: 1)
# Assert that 1 experiment has been run.
assert_experiments 1
# Assert that within the block, 2 experiments will be run.
assert_experiments 2 do
MyExperiment.run(id: 2)
MyExperiment.run(id: 3)
end
# Assert an experiment has been run with a given context.
assert_experiment_with(MyExperiment, context: { id: 1 })
# Assert that within the block, a matching experiment will be run.
assert_experiment_with(MyExperiment, variant: :red, context: { id: 4 }) do
MyExperiment.set(variant: :red).run(id: 4)
end
endRSpec support can be added by requiring active_experiment/rspec in the appropriate spec helper.
Active Experiment supports GlobalID serialization for experiment contexts. This is part of what makes it possible to utilize Active Record objects as context to consistently assign the same variant across multiple runs.
- Vanity - Experiment Driven Development framework for Rails.
- Scientist - A Ruby library for carefully refactoring critical paths.
- Gitlab::Experiment - A framework for running experiments, by GitLab.
- Split - The Rack Based A/B testing framework.
Active Experiment is released under the MIT license:
Copyright 2022 jejacks0n