Add pending state for results #3485
Conversation
thrix
added
ci | full test
Resolves #3304 Signed-off-by: Miroslav Vadkerti <[email protected]>
| self._results.clear() | ||
| for index, result in enumerate(self._results): | ||
| if result.result != ResultOutcome.PENDING: | ||
| self._results[index] = Result(name=result.name, result=ResultOutcome.PENDING) |
There was a problem hiding this comment.
I'd say this should recreate the list from scratch, i.e. not overwriting the existing results. Let's say there was a test with custom results, creating 100 of results for that single test. Before running the test for the first time, there would be a single PENDING placeholder for the test; after the operation above, results.yaml may contain 100 results for that test, each with PENDING results. And since tmt does not control creation of these results, the test may easily decide to not produce several of them this time, and therefore that particular result should never appear in our results.yaml. Dropping all results except the single PENDING one per test seems like better option to me.
| @@ -415,6 +417,15 @@ def go(self, force: bool = False) -> None: | |||
| test.serial_number == result.serial_number): | |||
| self._failed_tests[test_phase].append(test) | |||
|
|
|||
| # Initialize results with pending outcome if no results for it available | |||
| results = self.plan.execute.results() | |||
| for index, (test, result) in enumerate(zip_longest(self.tests(), results)): | |||
There was a problem hiding this comment.
I believe it won't be as easy as zip(). Results may be listed in an arbitrary order - yes, results are added once their tests finish, but I wouldn't rely on it, and there is certainly no code enforcing the order.
Check the Execute.results_for_tests() which lists results and test pairs, you should be interested in creating new results for pairs where the test is set and the result is None, that's a test that has not run yet.
| # Go through all results and replace those which were pending | ||
| for index, result in enumerate(execute_phases[0].results()): | ||
| if self._results[index].result == ResultOutcome.PENDING: | ||
| self._results[index] = result |
There was a problem hiding this comment.
Hmmm, this is weird. I think it will hit custom results again: for a single test, there may be 100 results in execute_phases[0].results() but not in `self._results. They would then land all in the very same index.
I believe we need a more robust approach here, based on matching the result's test serial numbers, constructing the list from two sources, execute_phases[0] and self._results. I'd try walking self._results, and find out whether there are any results in execute_phases[0].results()for the same test: if there are, add them to futureself._results, if there are not, copy the pending one. Eventually, I'd expect to be left with consumed execute_phase[0].results(), because all of those are valid results, and possibly not consumed self._resultsbecause some were moved to the new list, and some didn't run and remainPENDING`. Those I would add to the new list as well.
|
This pull request covers #3238 as well, right? |
thrix
added
the
priority | must
Resolves #3304
Pull Request Checklist