Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding Coco Metric for inference #1056

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Adding Coco Metric for inference #1056

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8411958
added metric() method
ai-fast-track Feb 2, 2022
6222e4a
remove private method
ai-fast-track Feb 2, 2022
4ce00b9
add False to getattr otherwise mask_head is added
ai-fast-track Feb 2, 2022
cc9ebc7
added metric_from_preds
ai-fast-track Feb 2, 2022
689e6b6
added COCO metric calculation at inference time
ai-fast-track Feb 3, 2022
b40b13c
added coco metric at inference time and clean-up
ai-fast-track Feb 3, 2022
edaafe0
add path to record
ai-fast-track Feb 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions icevision/data/dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ def __repr__(self):
@classmethod
def from_images(
cls,
images: Sequence[np.array],
images: Union[Sequence[np.array], Sequence[str]],
tfm: Transform = None,
class_map: Optional[ClassMap] = None,
):
Expand All @@ -56,11 +56,17 @@ def from_images(
images: `Sequence` of images in memory (numpy arrays).
tfm: Transforms to be applied to each item.
"""

records = []
for i, image in enumerate(images):
record = BaseRecord((ImageRecordComponent(),))
record.set_record_id(i)
record.set_img(image)
# If it's a path
if isinstance(image, str):
record.set_img(PIL.Image.open(Path(image)))
record.filepath = image
else:
record.set_img(image)
records.append(record)

# TODO, HACK: adding class map because of `convert_raw_prediction`
Expand Down
41 changes: 36 additions & 5 deletions icevision/metrics/coco_metric/coco_metric.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
__all__ = ["COCOMetric", "COCOMetricType"]
__all__ = [
"COCOMetric",
"COCOMetricType",
]

from icevision.imports import *
from icevision.utils import *
Expand Down Expand Up @@ -48,18 +51,47 @@ def accumulate(self, preds):
self._preds.append(pred.pred)

def finalize(self) -> Dict[str, float]:
logs = self.metric(
records=self._records, preds=self._preds, print_summary=self.print_summary
)

self._reset()
return logs

def metric_from_preds(
self,
preds: List[
Prediction
], # Prediction holds both the ground truth and the prediction records
print_summary: bool = False,
) -> Dict[str, float]:

gt_list = [pred.ground_truth for pred in preds]
preds_list = [pred.pred for pred in preds]

logs = self.metric(
records=gt_list, preds=preds_list, print_summary=print_summary
)
return logs

def metric(
self,
records, # Ground Truth records
preds, # Prediction records
print_summary: bool = False,
) -> Dict[str, float]:
with CaptureStdout():
coco_eval = create_coco_eval(
records=self._records,
preds=self._preds,
records=records,
preds=preds,
metric_type=self.metric_type.value,
iou_thresholds=self.iou_thresholds,
show_pbar=self.show_pbar,
)
coco_eval.evaluate()
coco_eval.accumulate()

with CaptureStdout(propagate_stdout=self.print_summary):
with CaptureStdout(propagate_stdout=print_summary):
coco_eval.summarize()

stats = coco_eval.stats
Expand All @@ -78,5 +110,4 @@ def finalize(self) -> Dict[str, float]:
"AR (IoU=0.50:0.95) area=large maxDets=100": stats[11],
}

self._reset()
return logs
4 changes: 2 additions & 2 deletions icevision/models/mmdet/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ def param_groups(model):
layers += [model.bbox_head]

# YOLACT has mask_head and segm_head
if getattr(model, "mask_head"):
if getattr(model, "mask_head", False):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did we set this one to False?
same thing for line 86

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit confusing. I had a discussion with Dickson in his PR. I had the reaction as you.

Right now, there is a bug because False is missing

If False is missing it will add mask_head even for retinanet model for example.

The way to read is if model has a mask_head the statement is True otherwise set it to False.

Copy link
Contributor

@FraPochetti FraPochetti Feb 3, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry man.
I still don't get it.

If False is missing it will add mask_head

But that's what we want. No?

Here how I understand the following statement:

if getattr(model, "mask_head"): layers += [model.mask_head]

if the model object has a "mask_head" attribute then add model.mask_head to the layers list.

What am I missing?

What is False needed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The False is the default value of getattr if the object does not have the attribute, otherwise, an AttributeError will be raise if no default is provided. We should change getattr for hasattr as it is the function intended for the usecase.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, alright! Got it. Thanks Frederik.

layers += [model.mask_head]
if getattr(model, "segm_head"):
if getattr(model, "segm_head", False):
layers += [model.segm_head]

elif isinstance(model, TwoStageDetector):
Expand Down
Loading