Should control code also be eval'd?

[jkeroes]

There are two implementations I'd like to consider: the original code (dat-science) and the reference code (Scientist). The former is simpler. Let's start there, but first, a caveat: I'm not a Ruby expert, so please check my conclusions.

This comment says that the errors become part of the observation:
https://github.com/github-archive/dat-science/blob/master/lib/dat/science/experiment.rb#L40

Also, it looks like the control's errors are propagated:
https://github.com/github-archive/dat-science/blob/master/lib/dat/science/experiment.rb#L107

I don't see the candidate's errors getting propagated. Let's consider a few cases demonstrating how this behavior would work in practice:

Scientist->new( use => { die "good failure" }, try => { 1 } )->run; # should fail
Scientist->new( use => { die "good failure" }, try => { die "good failure" } )->run; # should pass

Keep in mind that we want these two controls to die. That means we better compare them! (After that, we can die and let some other code catch the control's exception).

Let's also consider:

Scientist->new( use => { 1 }, try => { die "bad failure" } )->run; # should fail

In this case, we want the code to keep running and we also want to know that there was a discrepancy.

Scientist->new( use => { die "old failure" }, try => { die "new failure" } )->run; # should fail

This this final case, the code should die (because of the die in use()) and we should also be alerted about the discrepancy.

Let's look at the reference implementation next: https://github.com/github/scientist/blob/master/lib/scientist/experiment.rb#L41

It's stringifying the exceptions and prepended to both observations. (Again, please take my Ruby reading with a grain of salt).

The tests may better explain desired behavior:

hide die in try: https://github.com/github/scientist/blob/master/test/scientist/experiment_test.rb#L90

propagate die in use: https://github.com/github/scientist/blob/master/test/scientist/experiment_test.rb#L97

die in use: https://github.com/github/scientist/blob/master/test/scientist/experiment_test.rb#L398

die in try: https://github.com/github/scientist/blob/master/test/scientist/experiment_test.rb#L405

MismatchError: https://github.com/github/scientist/blob/master/test/scientist/experiment_test.rb#L444

MismatchError with stack trace: https://github.com/github/scientist/blob/master/test/scientist/experiment_test.rb#L475

This implementation also has finer-grained control over how to handle exceptions. See: https://github.com/github/scientist/blob/master/lib/scientist/experiment.rb#L219 . We could add that support.

Conclusions:

1. we want to propagate control exceptions; this is, it blows up. (works as expected)
2. we want to continue running in spite of candidate exceptions (fixed in fix #16 #22 )
3. we should ensure that run() compares exceptions.
4. we could add finer control to when and how exceptions are raised.