Best model architecture + Backbone for accuracy ?

[sudo-vinnie]

What is the best combination model architecture - backbone for accuracy?

I trained two networks
1.- Lightweight OpenPose + ResNet18
2.- OpenPose + ResNet50

After performing the evaluation here are the results:

I got better performance with resnet18 and with fewer iterations.
I added a couple of example images.
I also noticed the images used for evaluation are inconsistent. but regardless of the variation, 7-11 images is a very small sample for evaluation.

ResNet 18 ( 100 000 iterations)

Evluating on total 11 images...
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.188
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.587
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.104  
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.182  
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.281  
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.265  
Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.650  
Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.200  
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.180  
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.520

ResNet 50 ( 1 000 000 iterations)

Evluating on total 7 images
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.007  
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.059  
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.027  
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.033  
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.029  
Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.176  
Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.000  
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.025  
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.100

[Gyx-One]

Hello @sudo-vinnie

1. All the model evaluation metric reported in this repo is over the whole MSCOCO or MPII dataset, so maybe you forget to give the function evaluate the parameter called total_eval_num which gives instruction on how many images are going to be evaluated. I'll have a check on the default setting.
   2.The Openpose + Vgg19(default backbone) is the largest model with the best accuracy.
   3.Tipycally to train a model needs 400000+ iterations, so you may need to wait more iterations for model to converge to a better optim. (The larger the model is, often the more iterations it needs to converge well)
   Thanks for using our library! :)

[sudo-vinnie]

@Gyx-One Thaks for the answer I tried training the network with Vgg19 as a backbone, but I get the following error when evaluating the network (​This does not happen with the ResNet18):

Config:

# >>> set model name is necessary to distinguish models (neccesarry)
Config.set_model_name("Human_OP_VGG")
# >>> set model architecture using Config.MODEL enum class (neccesarry)
Config.set_model_type(Config.MODEL.Openpose)
# >>> set model backbone using Config.BACKBONE enum class (not neccessary, each model has its default backbone)
Config.set_model_backbone(Config.BACKBONE.Vgg19)
# >>> set dataset to use (neccesarry)
Config.set_dataset_type(Config.DATA.MSCOCO)
# >>> set training type (not neccesary, default is single training, can use parallel training)
Config.set_train_type(Config.TRAIN.Single_train)
# >>> congratulations!, the simplest configuration is done, it's time to assemble the model and training pipeline

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-9-d69ae292f048> in <module>
----> 1 evalu(model,dataset)

~/Kyutech/hyperpose-2.1.0_Human/hyperpose/Model/openpose/eval.py in evaluate(model, dataset, config, vis_num, total_eval_num)
    135 
    136     result_dic={"annotations":pd_anns}
--> 137     dataset.official_eval(result_dic,vis_dir)

~/Kyutech/hyperpose-2.1.0_Human/hyperpose/Dataset/mscoco_dataset/dataset.py in official_eval(self, pd_json, eval_dir)
    231         print(f"evluating on total {len(image_ids)} images...")
    232         gt_coco=COCO(gt_json_path)
--> 233         pd_coco=gt_coco.loadRes(pd_json_path)
    234 
    235         '''

~/anaconda3/envs/hyperpose/lib/python3.7/site-packages/pycocotools/coco.py in loadRes(self, resFile)
    326         assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \
    327                'Results do not correspond to current coco set'
--> 328         if 'caption' in anns[0]:
    329             imgIds = set([img['id'] for img in res.dataset['images']]) & set([ann['image_id'] for ann in anns])
    330             res.dataset['images'] = [img for img in res.dataset['images'] if img['id'] in imgIds]

IndexError: list index out of range