Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage experiment #300

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Coverage experiment #300

wants to merge 5 commits into from

Conversation

jmid
Copy link
Collaborator

@jmid jmid commented Feb 24, 2023

I've played with computing coverage to have an idea of how much ground we cover.
For now, the approach takes a copy of each tested Stdlib module and instruments that with bisect_ppx.

I do not consider this to be particularly pretty... 😅
For example, this will not report any coverage on extern-defined functions
Nevertheless, I consider it good enough to obtain some basic numbers.

$ make coverage-summary 
bisect-ppx-report summary --per-file
 28.84 %   109/378    src/array/array.ml
100.00 %     6/6      src/atomic/atomic.ml
 17.28 %    42/243    src/buffer/buffer.ml
  4.29 %    34/792    src/bytes/bytescp.ml
 24.75 %    25/101    src/domain/domain.ml
 27.78 %   115/414    src/ephemeron/ephemeron.ml
 36.94 %   140/379    src/floatarray/float.ml
 28.57 %   104/364    src/hashtbl/hashtbl.ml
 12.50 %     8/64     src/io/in_channel.ml
  0.00 %     0/8      src/io/out_channel.ml
 81.48 %    22/27     src/lazy/lazy.ml
 43.40 %    23/53     src/queue/queue.ml
 41.03 %    16/39     src/semaphore/semaphore.ml
 65.38 %    17/26     src/stack/stack.ml
 43.48 %    10/23     src/sys/sys.ml
 68.57 %   168/245    src/weak/weak.ml
 26.53 %   839/3162   Project coverage

These were obtained by running the below command for the instrumented modules and computing a summary and report at the end:

dune runtest -f src/array --instrument-with bisect_ppx
....
dune runtest -f src/weak --instrument-with bisect_ppx

I'm also including the HTML in a discard-able commit (don't know if it is viewable through the GitHub UI).

Note:

  • For Bigarray no coverage is reported! (4/5 tested functions defined as extern it seems)
  • For Bytes the simple approach needed adjusting to avoid conflicts with the Bytes required by qcheck-core
  • For Domain, instrumenting a copy crashes on my machine:
    $ dune runtest -f src/domain --instrument-with bisect_ppx
    File "src/domain/dune", line 15, characters 7-23:
    15 |  (name domain_spawntree)
                ^^^^^^^^^^^^^^^^
    Command got signal SEGV.
    File "src/domain/dune", line 6, characters 7-23:
    6 |  (name domain_joingraph)
               ^^^^^^^^^^^^^^^^
    Command got signal SEGV.
    
    Some coverage is reported though - probably up to the crash
  • For Dynlink - no immediate dynlink.ml or dynlink/dynlink.ml to copy
  • For Out_channel coverage is for some reason reported as 0%. I'm unsure why.
  • For Thread and Threadomain - no immediate thread.ml or threads/thread.ml to copy

I'm not sure we would want this to be merged.
It gives us some numbers nevertheless that we can compare to, e.g., with the multiple-ts of #112

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant