Universum 0.19.13

CHANGELOG.rst

@@ -1,6 +1,26 @@
 Change log
 ==========
 
+0.19.13 (2022-10-10)
+--------------------
+
+New features
+~~~~~~~~~~~~
+
+* **automation:** add full support of GitHub Actions CI server (as alternative to Jenkins and TeamCity)
+* **config:** add path to configuration file in :doc:`configuration_support <configuration_support>`;
+  this might be very useful in non-CI mode when launched from directory other than project root
+* **report:** status reports now include more useful info; previous short form can be enabled by
+  ``-rofs`` `command-line option <args.html#Result\ reporting>`__
+
+Bug fixes
+~~~~~~~~~
+
+* **config:** update error message if config is empty after step filtering
+* **doc:** add explanation of filter usage in :ref:`non-CI mode description <additional_commands#run>`
+* **analyzer:** update error message on missing input file
+
+
 0.19.12 (2022-01-26)
 --------------------
 

doc/additional_commands.rst

@@ -45,6 +45,9 @@ Universum in non-CI mode has the following differences from default mode:
     :prog: {python} -m universum
     :path: run
 
+    --filter -f : @replace
+        .. include:: filter_description.rst
+
 
 .. _additional_commands#poll:
 

doc/args.rst

@@ -18,21 +18,7 @@ add comments on found issues right to the selected code review system.
         Display product name & version instead of launching.
 
     --filter -f : @replace
-        | Allows to filter which steps to execute during launch.
-         String value representing single filter or a set of filters separated by '**:**'.
-         To define exclude pattern use '**!**' symbol at the beginning of the pattern.
-        |
-        | A Universum step match specified pattern when 'filter' is a substring of step 'name'.
-         This functionality is similar to 'boosttest' and 'gtest' filtering, except special characters
-         (like '*', '?', etc.) are ignored.
-        |
-        | Examples:
-        | * -f='run test'               - run only steps that contain 'run test' substring in their names
-        | * -f='!run test'              - run all steps except those containing 'run test' substring in their
-         names
-        | * -f='test 1:test 2'          - run all steps with 'test 1' OR 'test 2' substring in their names
-        | * -f='test 1:!unit test 1'    - run all steps with 'test 1' substring in their names except those
-         containing 'unit test 1'
+        .. include:: filter_description.rst
 
     --html-log -hl : @after
         To make sure all the interactive features of such a page work right in Jenkins artifacts,

doc/filter_description.rst

@@ -0,0 +1,16 @@
+
+| Allows to filter which steps to execute during launch.
+ String value representing single filter or a set of filters separated by '**:**'.
+ To define exclude pattern use '**!**' symbol at the beginning of the pattern.
+|
+| A Universum step match specified pattern when 'filter' is a substring of step 'name'.
+ This functionality is similar to 'boosttest' and 'gtest' filtering, except special characters
+ (like '*', '?', etc.) are ignored.
+|
+| Examples:
+| * -f='run test'               - run only steps that contain 'run test' substring in their names
+| * -f='!run test'              - run all steps except those containing 'run test' substring in their
+ names
+| * -f='test 1:test 2'          - run all steps with 'test 1' OR 'test 2' substring in their names
+| * -f='test 1:!unit test 1'    - run all steps with 'test 1' substring in their names except those
+ containing 'unit test 1'

doc/github_actions.rst

@@ -0,0 +1,46 @@
+Integration with GitHub Actions
+===============================
+
+`Universum` requires no special integration with `GitHub Actions <https://docs.github.com/en/actions>`_. It is usually
+launched as one long step in a single build stage.
+
+.. warning::
+
+    'GitHub Actions' CI system is not compatible with ``report_artifacts`` :doc:`configuration Step key <configuration_support>`.
+    If this key is set nevertheless, links to artifacts will be posted, but won't work.
+
+
+Command line
+------------
+
+Here's an example of a command line to be used for running Universum in GitHub Actions::
+
+    python -m universum --vcs-type=git --git-repo "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}" --git-refspec "${GITHUB_REF_NAME}"
+
+All environment variables mentioned in the example are `GitHub Environment variables
+<https://docs.github.com/en/actions/learn-github-actions/environment-variables>`_.
+
+`Universum` also supports reporting to code review systems ('--report-to-review' option) from 'GitHub Actions'. Link to
+build is created using environment variables and do not require any additional command line parameters.
+
+Logs
+----
+
+GitHub Actions web interface currently supports single-level grouping of log lines
+(`without nesting <https://github.com/actions/runner/issues/802>`_). Because of that, Universum logs are printed 
+as follows: an already opened group is closed if any other (including nested) group is opened.
+
+
+Artifacts
+---------
+
+Artifacts can be stored in GitHub Actions with explicitly provided name via separate workflow step.
+It is possible to store multiple files in a single `artifact
+<https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts>`_, but it is not possible to
+retrieve only one file from such artifact. This leads to the following limitations:
+
+    - links to GitHub Actions artifacts can only be retrieved after actual artifact creation
+      (which happens after `Universum` run)
+    - links to single artifact files cannot be provided at all
+
+This is the reason the ``report_artifacts`` key can not be processed correctly and shouldn't be set in configuration.

doc/github_handler.rst

@@ -7,8 +7,8 @@ and trigger an already set up automation server to perform these checks. GitHub
 and passes them to the triggered builds.
 
 
-How to set up GitHub auto check using Unversum
-----------------------------------------------
+How to set up GitHub auto check using Universum
+-----------------------------------------------
 
 Default Universum ('main' mode) can post `check run` statuses to GitHub to be depicted both on 'Checks' page
 in pull requests and as a simple icon near any checked commit.

doc/index.rst

@@ -15,7 +15,9 @@ Project 'Universum'
     universum_docs.rst
     teamcity.rst
     jenkins.rst
+    github_actions.rst
     examples.rst
+    internal.rst
     changelog_ref.rst
 
 .. image:: _static/logo.svg
@@ -27,7 +29,24 @@ Project `Universum` is a continuous integration framework, containing
 a collection of functions that simplify implementation of the
 automatic build, testing, static analysis and other steps.
 The goal of this project is to provide unified approach for adding continuous integration
-to any project. It currently supports Perforce, Git, Gerrit, Swarm, Jenkins and TeamCity.
+to any project.
+
+Supported VCS:
+
+- Git
+- Perforce
+
+Supported review systems:
+
+- Gerrit
+- Swarm
+- GitHub
+
+Supported CI:
+
+- Jenkins
+- TeamCity
+- Github Actions
 
 Sometimes `Universum` system can be referred to as the framework or just CI.
 

doc/internal.rst

@@ -0,0 +1,9 @@
+Internal documentation
+----------------------
+
+.. automodule:: universum.modules.output.base_output
+    :members:
+    :special-members:
+
+    :mod:`universum.modules.output.base_output`
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

noxfile.py

@@ -17,15 +17,16 @@ def send_report():
     token = os.getenv("TELEGRAM_BOT_TOKEN")
     chat = os.getenv("TELEGRAM_CHAT_ID")
     requests.post(url=f"https://api.telegram.org/bot{token}/sendMessage",
-                  data={"chat_id": chat, "text": report})
+                  data={"chat_id": chat, "text": report},
+                  timeout=30)
 
 
 @nox.session(python=["3.6", "3.7", "3.8", "3.9"])
 def test(session):
     try:
         session.run("make", "rebuild", silent=True, external=True)
         session.install(".[test]")
-        session.run("make", "test", external=True)
+        session.run("make", "test", external=True, env={"UNIVERSUM_NOX_REGRESSION": "True"})
         add_report_line(f"\U00002600 testing for Python {session.python} succeeded")
     except nox.command.CommandFailed:
         add_report_line(f"\U00002601 testing for Python {session.python} failed")

tests/conftest.py

@@ -47,14 +47,14 @@ def assert_absent_calls_with_param(self, pattern_to_search, is_regexp=False):
 
 @pytest.fixture()
 def stdout_checker(request):
-    with mock.patch('universum.modules.output.terminal_based_output.stdout') as logging_mock:
+    with mock.patch('universum.modules.output.terminal_based_output.TerminalBasedOutput._stdout') as logging_mock:
         result = FuzzyCallChecker(logging_mock)
         yield result
 
 
 @pytest.fixture()
 def log_exception_checker(request):
-    with mock.patch('universum.modules.output.terminal_based_output.TerminalBasedOutput.log_exception') as logging_mock:
+    with mock.patch('universum.modules.output.terminal_based_output.TerminalBasedOutput.log_error') as logging_mock:
         result = FuzzyCallChecker(logging_mock)
         yield result
 

tests/test_code_report.py

@@ -276,8 +276,8 @@ def test_uncrustify_file_diff(runner_with_analyzers: UniversumRunner,
 
     expected_log = log_success if expected_success else log_fail
     assert re.findall(expected_log, log), f"'{expected_log}' is not found in '{log}'"
-    expected_log = r"Collecting 'source_file.html' - [^\n]*Success" if expected_artifact \
-        else r"Collecting 'source_file.html' - [^\n]*Failed"
+    expected_artifacts_state = "Success" if expected_artifact else "Failed"
+    expected_log = f"Collecting artifacts for the 'Run uncrustify' step - [^\n]*{expected_artifacts_state}"
     assert re.findall(expected_log, log), f"'{expected_log}' is not found in '{log}'"
 
 

tests/test_html_output.py

@@ -175,7 +175,7 @@ def check_body_coloring(body_element):
 def check_title_and_status_coloring(steps_body):
     check_section_coloring(steps_body.get_section_by_name("Success links step"))
     check_section_coloring(steps_body.get_section_by_name("Failed step"), is_failed=True)
-    check_section_coloring(steps_body.get_section_by_name("Partially success step"), has_inner_fail=True)
+    check_section_coloring(steps_body.get_section_by_name("Partially success step"), is_failed=True)
 
     composite_step_body = steps_body.get_section_body_by_name("Partially success step")
     check_section_coloring(composite_step_body.get_section_by_name("Success links step"))
@@ -212,14 +212,13 @@ def check_errors_tags_coloring(steps_body):
     assert stderr_tag.color == Color.YELLOW
 
 
-def check_section_coloring(step, is_failed=False, has_inner_fail=False):
-    is_section_failed = is_failed or has_inner_fail
-    check_text_item_style(step.get_section_title(), is_section_failed, normal_color=Color.BLUE)
-    if not is_section_failed:
-        step.click() # open section body
+def check_section_coloring(step, is_failed=False):
+    check_text_item_style(step.get_section_title(), is_failed, normal_color=Color.BLUE)
+    if not is_failed:
+        step.click()  # open section body
     check_text_item_style(step.get_section_status(), is_failed, normal_color=Color.GREEN)
-    if not is_section_failed:
-        step.click() # close section body
+    if not is_failed:
+        step.click()  # close section body
 
 
 def check_text_item_style(item, is_failed, normal_color):

tests/test_integration.py

@@ -39,13 +39,14 @@ def test_artifacts(docker_main: UniversumRunner):
 files2 = Configuration([dict(name=" one/three/file.sh", command=["one/three/file.sh"])])
 
 artifacts = Configuration([dict(name="Existing artifacts", artifacts="one/**/file*", report_artifacts="one/*"),
-                        dict(name="Missing artifacts", artifacts="something", report_artifacts="something_else")])
+                           dict(name="Missing report artifacts", report_artifacts="non_existing_file"),
+                           dict(name="Missing all artifacts", artifacts="something", report_artifacts="something_else")])
 
 configs = mkdir * dirs1 + mkdir * dirs2 + mkfile * files1 + mkfile * files2 + artifacts
     """
     log = docker_main.run(config)
-    assert 'Failed' in get_line_with_text("Collecting 'something' - ", log)
-    assert 'Success' in get_line_with_text("Collecting 'something_else' for report - ", log)
+    assert 'Failed' in get_line_with_text("Collecting artifacts for the 'Missing all artifacts' step - ", log)
+    assert 'Success' in get_line_with_text("Collecting artifacts for the 'Missing report artifacts' step - ", log)
 
     assert os.path.exists(os.path.join(docker_main.artifact_dir, "three.zip"))
     assert os.path.exists(os.path.join(docker_main.artifact_dir, "two2.zip"))
@@ -111,7 +112,7 @@ def test_critical_steps(docker_main_and_nonci: UniversumRunner):
                          dict(name="Bad step", command=["ls", "not_a_file"], critical=True),
                          dict(name="Extra step", command=["echo", "This shouldn't be in log."])])
 """)
-    assert "Extra step skipped because of critical step failure" in log
+    assert "'Extra step' skipped because of critical step failure" in log
     assert "This shouldn't be in log." not in log
 
     # Test embedded: critical step, critical substep
@@ -129,8 +130,8 @@ def test_critical_steps(docker_main_and_nonci: UniversumRunner):
 
 configs = upper * lower
 """)
-    assert "Group 3, step 1 skipped because of critical step failure" in log
-    assert "Group 2, step 1 skipped because of critical step failure" not in log
+    assert "'Group 3, step 1' skipped because of critical step failure" in log
+    assert "'Group 2, step 1' skipped because of critical step failure" not in log
 
     # Test embedded: critical step, non-critical substep
     docker_main_and_nonci.clean_artifacts()
@@ -146,7 +147,7 @@ def test_critical_steps(docker_main_and_nonci: UniversumRunner):
 
 configs = upper * lower
 """)
-    assert "Group 2, step 1 skipped because of critical step failure" in log
+    assert "'Group 2, step 1' skipped because of critical step failure" in log
     assert "This should be in log." in log
 
     # Test critical non-commands
@@ -165,6 +166,21 @@ def test_critical_steps(docker_main_and_nonci: UniversumRunner):
 """)
     assert "This shouldn't be in log." not in log
 
+    # Test successful critical step after failing non-critical step
+    docker_main_and_nonci.clean_artifacts()
+    log = docker_main_and_nonci.run("""
+from universum.configuration_support import Configuration
+
+configs = Configuration([dict(name="Group 1")])
+
+configs *= Configuration([dict(name=", step 1", command=["echo", "step succeeded"]),
+                          dict(name=", step 2", command=["this-is-not-a-command"]),
+                          dict(name=", step 3", command=["echo", "This should be in log 1."], critical=True),
+                          dict(name=", step 4", command=["echo", "This should be in log 2."])])
+    """)
+    assert "This should be in log 1." in log
+    assert "This should be in log 2." in log
+
     # Test background
     docker_main_and_nonci.clean_artifacts()
     log = docker_main_and_nonci.run("""

tests/test_regression.py

@@ -115,11 +115,13 @@ def mocking_function(*args, **kwargs):
     monkeypatch.setattr(P4.P4, 'run_opened', mocking_function, raising=False)
 
 
+@utils.nox_only
 def test_p4_failed_opened(perforce_environment: P4TestEnvironment, mock_opened: None):
     perforce_environment.run()
 
 
 # TODO: move this test to 'test_api.py' after test refactoring and Docker use reduction
+@utils.nox_only
 def test_p4_api_failed_opened(perforce_environment: P4TestEnvironment, mock_opened: None):
     step_name = "API"
     config = f"""