[WIP] Code coverage analysis

[dentarg]

Locally with SimpleCov and on Travis CI with Coveralls.

Close #81.

[coveralls]

Changes Unknown when pulling fedb255 on code-coverage into * on master*.

[dentarg]

@twingly/dev WDYT?

[walro]

I don't really understand what this is/why this is needed – do you have more info?

[dentarg]

Sure, I should have mentioned it. I has to do that we have split our tests over several different rake tasks and need to merge the results. I got it from https://github.com/colszowka/simplecov#merging-results. I tried doing SimpleCov.command_name = "PID#{Process.pid}" in spec_helper.rb, so we shouldn't have to add something to every spec file, but it didn't work well. It saved the results from every run, so it didn't discover regressions in coverage, and the "hits/line" went up on every run.

The command names can be seen in the footer in report:

Generated by simplecov v0.12.0 and simplecov-html v0.10.0
using hasher_spec, null_url_spec, url_spec, utilities_spec

[dentarg]

The command names can be seen in the footer in report:

Also in the spec run output: https://travis-ci.org/twingly/twingly-url/jobs/156254738

[walro]

I commented out all tests in hasher_spec.rb and I still got 100% coverage on hasher.rb.

[dentarg]

I commented out all tests in hasher_spec.rb and I still got 100% coverage on hasher.rb.

Found a similar issue, simplecov-ruby/simplecov#388, which is also using simplecov and coveralls, for a gem project, so I will take a look on what they've done

[dentarg]

Hmm, the comment at simplecov-ruby/simplecov#314 (comment) was somewhat insightful:

So, it's important understand how SimpleCov, and the stdlib Coverage lib it depends on works.

It tracks when code is evaluated for the first time (i.e. when loaded or run )

Thus, if you run SimpleCov.start, load a file, then run SimpleCov.result, you will see a certain amount of 'code coverage'. That is, running tests is a way to increase the coverage of a file above the coverage you get for just loading it.

In your test helper you have Dir[File.join(File.dirname(__FILE__), '../lib/**/*.rb')].sort.each { |f| require f } which loads all your ruby folders under lib. Thus, you see files 'covered' that weren't tested.

I was thinking it might the combination of lib/twingly/url/hasher.rb, lib/twingly/url/null_url.rb and lib/twingly/url/utilities.rb all doing require_relative "../url"... and the merging of tests results. But I have to check some more.

Example: bundle exec rspec spec/lib/twingly/url/hasher_spec.rb, with only these lines in hasher_spec.rb:

SimpleCov.command_name(File.basename(__FILE__, ".rb"))

require "twingly/url/hasher"

produces:

[dentarg]

I ran the specs one by one (no modifications) (bundle exec rspec <spec file>) and removed tmp/coverage before each run.

[dentarg]

@walro what do you think of just ignoring hasher and utilities for now?

diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
index f84a87a..a4b1734 100644
--- a/spec/spec_helper.rb
+++ b/spec/spec_helper.rb
@@ -7,6 +7,9 @@ SimpleCov.formatters = [
 SimpleCov.start do
   add_filter "spec/"
   add_filter "vendor/"
+
+  add_filter "hasher.rb"
+  add_filter "utilities.rb"
 end

 RSpec.configure do |config|

[walro]

@walro what do you think of just ignoring hasher and utilities for now?

Hehe, nah. We could add new .rb files in the future that will exhibit the same weirdness so I think we'll need to find another route.

[dentarg]

Interesting? https://www.relishapp.com/rspec/rspec-core/v/3-5/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process

[walro]

I feel this is quite outdated now, I suspect things have moved forward since, close?

[dentarg]

I feel this is quite outdated now, I suspect things have moved forward since, close?

Outdated how?

Sure, there has been a decent amount of changes made since this was started, but our tests are still structured in the same way.

[walro]

Well, that's one part. The other part is that the tools have moved forward. Ruby has gained more things natively and so on. I feel that either we give this branch some love or just close it. It's approaching 18 months.

[dentarg]

Sure, it is probably best to start over.

I will note here what I think is an issue. Our setup is a bit special in that we run rspec multiple times when we run our spec suite (we started doing this to catch require errors):

$ bundle exec rake -T spec
rake spec:hasher_spec     # Run RSpec code examples
rake spec:null_url_spec   # Run RSpec code examples
rake spec:url_spec        # Run RSpec code examples
rake spec:utilities_spec  # Run RSpec code examples

twingly-url/Rakefile

Lines 19 to 32 in fedb255

	spec_files = Dir.glob(File.join("spec/**", "*_spec.rb"))
	spec_tasks = []
	namespace(:spec) do
	spec_files.each do |spec_file|
	task_name = File.basename(spec_file, ".rb").to_sym
	spec_tasks << "spec:#{task_name}"
	RSpec::Core::RakeTask.new(task_name) do |task|
	task.pattern = spec_file
	end
	end
	end