Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test explorer stability: Use test results as main truth source for explorer #1874

Merged
merged 40 commits into from
Aug 19, 2023

Commits on Jul 8, 2023

  1. Configuration menu
    Copy the full SHA
    425764d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fcff548 View commit details
    Browse the repository at this point in the history
  3. Run tests from trx

    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    607d328 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2adb235 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    0077521 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d43615c View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    3614b5d View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    8fcc22d View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    1926480 View commit details
    Browse the repository at this point in the history
  10. Update test tree based on test results

    This is especially helpful for tests that don't have code locations.
    The user can run a subset of tests
    and changes to that portion of the
    test tree will be reflected
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    fb8eea1 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    5330dc4 View commit details
    Browse the repository at this point in the history
  12. Simplify test handler

    Refactored state updates around individual projects.
    This reduces mapping since tests execution is actualized by project
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    3fac404 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    4b8587b View commit details
    Browse the repository at this point in the history
  14. Generalize test hierarchy builder

    This is a preperatory refactor in anticipation of changing
    test discovery from trx to --list-tests
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    d0f2627 View commit details
    Browse the repository at this point in the history
  15. Support nested class hierarchies

    I.e. when there are mixed + and . separators in the test name
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    f0a0f98 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    c3996ad View commit details
    Browse the repository at this point in the history
  17. Change filters to balance speed and discovery

    The previous filter found all known test items to run from the explorer
    and matched them exactly.
    But this won't allow the explorer to discover tests from test results.
    If the user selects a group of tests, then it should run all tests in
    that group even if they aren't in the explorer
    then updated the explorer if there are differences.
    This is especially critical for C# tests which can't match code locations
    and update that way.
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    13cde47 View commit details
    Browse the repository at this point in the history
  18. Write test console output to Test Result log.

    Improves discoverability of error and failure messages,
    especially for tests without code locations.
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    3c1cf86 View commit details
    Browse the repository at this point in the history
  19. Stablize initial test explorer discovery

    Building projects in parallel was leading to build failures
    and tests being discovered incorrectly.
    Also effected explorer refresh.
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    4261a47 View commit details
    Browse the repository at this point in the history
  20. Fix resource starvation for solutions with many projects

    Restoring in parallel was spawning too many promises (i.e. 87 in FAKE solution)
    and causing computer lag
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    72428a2 View commit details
    Browse the repository at this point in the history
  21. Add constrained promise parallelism

    Added a new method that runs at most a certain number of Promises at a time.
    Allows a balance of resource usage safety and speed.
    
    It's faster than sequential, but know we won't end up queuing so many
    promises we end up with resource starvation.
    I.e. restoring against 87 projects in the FAKE repo
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    d4e8ba7 View commit details
    Browse the repository at this point in the history
  22. Fix small collection errors in executeWithMaxParallel

    take and skip are unsafe operations, causing executeWithMaxParallel
    to error if the input collection is smaller than the max parallel count
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    b7d0d25 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    e55775c View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    765b710 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    5714bf4 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    e4b0026 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    01a70cc View commit details
    Browse the repository at this point in the history
  28. Run tests and test discovery against a single target framework

    This fixes indeterminism if a project has multiple target frameworks.
    Especially if it runs some tests only on certain targets.
    
    Always using the first target framework aligns with other Ionide behavior.
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    80580fb View commit details
    Browse the repository at this point in the history
  29. Partition tests by project

    The top level of the test explorer is organized by project.
    This will simplify some efforts like progressive loading
    and supporting multiple Target Frameworks
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    8e9e4f0 View commit details
    Browse the repository at this point in the history
  30. Add target framework to project test items

    Paves the way for multiple Target frameworks
    and provides context so users can figure out why the test count is
    different than the console or other test explorers
    
    See this discussion: ionide#1874 (comment)
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    64d0ab7 View commit details
    Browse the repository at this point in the history
  31. Preserve trailing space in test names using list discovery

    Trailing spaces were causing test run results to not match up with
    the expected test items.
    
    The same issue still exists for leading spaces in a test name.
    However, It's unclear how consistent the indentation of test names
    would be in the console output.
    Rather than increase the risk of failure,
    I opted to retain the potential error for leading spaces.
    Leading spaces in a test name should not be common use case and is most likely an mistype
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    7b45b7b View commit details
    Browse the repository at this point in the history
  32. Ask user to restore before test discovery

    Moving responsibility for initial restore to the user.
    It was proving difficult to consistently report restore errors,
    and, since we don't have access to optimal build orders,
    restore was adding a lot of time to test discovery
    when we nominally expect it to already be done.
    farlee2121 committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    a411c12 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2023

  1. Improve error reporting during test discovery

    Detect build failures and when a test run likely failed to complete
    farlee2121 committed Jul 9, 2023
    Configuration menu
    Copy the full SHA
    d279a97 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2023

  1. Limit parallel test project execution

    Prevents performance issues experienced with a large number of test projects.
    farlee2121 committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    95a2b78 View commit details
    Browse the repository at this point in the history
  2. Linear build when running tests & centralize enqueue

    Parallel project builds can conflict if they have mutual dependencies.
    This was more prominent during discovery and already fixed there,
    but it can also happen when running tests.
    
    Linearizing the build and limiting parallelism also changed the test status UX.
    Explicitly showing all enqueued right away makes it more intuitive that
    that the explorer responded correctly to the user's run request
    farlee2121 committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    6bbc1a4 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. Respect cancel tokens in explorer

    Pass cancel tokens through the test, build, and refresh call chains.
    Attempt to kill any long-running processes spawned
    (i.e. dotnet test, msbuild)
    
    Unfortunately, secondary processes (like testhost)
    aren't cancelling when their parent does.
    farlee2121 committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    6567b12 View commit details
    Browse the repository at this point in the history
  2. Clarify implicit test framework property

    Used to for workarounds specific to NUnit.
    farlee2121 committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    88c642d View commit details
    Browse the repository at this point in the history
  3. Fix NUnit filters with spaces when code loads before discovery

    NUnit can't handle spaces in test filter expressions.
    The mitigation for it depends on knowing the test is an NUnit test.
    However, that information was only available if
    code-sourced test info is merged after code discovery.
    This makes that info available from trx discovery too.
    farlee2121 committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    e882c87 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2023

  1. Configuration menu
    Copy the full SHA
    61e938c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    538f8c6 View commit details
    Browse the repository at this point in the history