Improve reporting for evaluation methods

[chrisvanrun]

Closes: #58

With the same error introduced, it is now clear how many failed. Since this is an example error, all processes finished before the error handling could come in and cancel all of them. Hence, 3 errors are shown.

Current:

Stdout

Input Files:
['/input/predictions.json',
 '/input/f2b13839-1e36-4e44-9cab-2397cbd47410/output/output-value.json',
 '/input/3104132e-8f98-48b9-8d92-68c2f2a0d72a/output/output-value.json',
 '/input/bf54b0be-a0f1-4083-ac9a-522b9a28775e/output/output-value.json']

Stderr

KeyError: 'a'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/app/evaluate.py", line 162, in <module>
    raise SystemExit(main())
                     ^^^^^^
  File "/opt/app/evaluate.py", line 96, in main
    metrics["results"] = run_prediction_processing(fn=process, predictions=predictions)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/helpers.py", line 77, in run_prediction_processing
    raise PredictionProcessingError(
helpers.PredictionProcessingError: Error for prediction {'pk': 'bf54b0be-a0f1-4083-ac9a-522b9a28775e', 'inputs': [{'file': None, 'image': {'name': 'the_original_filename_of_the_file_that_was_uploaded.suffix'}, 'value': None, 'interface': {'slug': 'input-ci-slug', 'kind': 'Image', 'super_kind': 'Image', 'relative_path': 'images/input-value', 'example_value': None}}], 'outputs': [{'file': 'https://grand-challenge.org/media/some-link/output-value.json', 'image': None, 'value': None, 'interface': {'slug': 'another-output-ci-slug', 'kind': 'Anything', 'super_kind': 'File', 'relative_path': 'output-value.json', 'example_value': {'key': 'value'}}}], 'status': 'Succeeded', 'started_at': '2024-11-29T10:31:25.691799Z', 'completed_at': '2024-11-29T10:31:50.691799Z'}: 'a'

Now:

Input Files:
├── predictions.json
├── 3367e4e8-509f-4bef-95cf-220cacf6ea99
│   └── output
│       └── output-value.json
├── d9a93e9b-839b-4323-ba7c-c24ff7df0e07
│   └── output
│       └── output-value.json
└── 36cda6fd-9d30-45e0-8fd2-69a6bded3da9
    └── output
        └── output-value.json

PROCESSING REPORT
Succeeded (0/3):
        -
Canceled (0/3):
        -
Failed (3/3):
        d9a93e9b-839b-4323-ba7c-c24ff7df0e07
        3367e4e8-509f-4bef-95cf-220cacf6ea99
        36cda6fd-9d30-45e0-8fd2-69a6bded3da9

Stderr

Error in prediction: 36cda6fd-9d30-45e0-8fd2-69a6bded3da9
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/evaluate.py", line 47, in process
    {"k":1}['a']
    ~~~~~~~^^^^^
KeyError: 'a'
"""

The above exception was the direct cause of the following exception:

KeyError: 'a'

Error in prediction: d9a93e9b-839b-4323-ba7c-c24ff7df0e07
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/evaluate.py", line 47, in process
    {"k":1}['a']
    ~~~~~~~^^^^^
KeyError: 'a'
"""

The above exception was the direct cause of the following exception:

KeyError: 'a'

Error in prediction: 3367e4e8-509f-4bef-95cf-220cacf6ea99
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/evaluate.py", line 47, in process
    {"k":1}['a']
    ~~~~~~~^^^^^
KeyError: 'a'
"""

The above exception was the direct cause of the following exception:

KeyError: 'a'

Traceback (most recent call last):
  File "/opt/app/evaluate.py", line 161, in <module>
    raise SystemExit(main())
                     ^^^^^^
  File "/opt/app/evaluate.py", line 96, in main
    metrics["results"] = run_prediction_processing(fn=process, predictions=predictions)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app/helpers.py", line 103, in run_prediction_processing
    raise PredictionProcessingError()
helpers.PredictionProcessingError

Stdout

[amickan]

Looks good to me!