Use EgoNet on custom data

[ratnam18]

Thank you for sharing your work.
I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?

[Nicholasli1995]

Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?

Hi, to use EgoNet on your custom data:

1. You need to prepare 2D or 3D proposals for your images before using EgoNet.
2. Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.

[ratnam18]

Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?

Hi, to use EgoNet on your custom data:

1. You need to prepare 2D or 3D proposals for your images before using EgoNet.
2. Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.

Thank you for the response. Also, to visualize the orientation arrow it requires 'kpts_3d_gt', are these the 3D dimensions that need to there before doing predictions for orientation estimation, as I am not able to visualize the arrow for KITTI test data.

[Nicholasli1995]

Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?

Hi, to use EgoNet on your custom data:

1. You need to prepare 2D or 3D proposals for your images before using EgoNet.
2. Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.

Thank you for the response. Also, to visualize the orientation arrow it requires 'kpts_3d_gt', are these the 3D dimensions that need to there before doing predictions for orientation estimation, as I am not able to visualize the arrow for KITTI test data.

You can use the predicted cuboid ("kpts_3d_pred") to visualize the orientation arrow instead. In this way, you don't need a 3D box input.

[ratnam18]

Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?

Hi, to use EgoNet on your custom data:

1. You need to prepare 2D or 3D proposals for your images before using EgoNet.
2. Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.

Thank you for the response. Also, to visualize the orientation arrow it requires 'kpts_3d_gt', are these the 3D dimensions that need to there before doing predictions for orientation estimation, as I am not able to visualize the arrow for KITTI test data.

You can use the predicted cuboid ("kpts_3d_pred") to visualize the orientation arrow instead. In this way, you don't need a 3D box input.

But the below code for adding an orientation arrow can be possible only by using both 'kpts_3d_gt' and 'kpts_3d_pred'. I understand that Keypoints are different from 3D bounding boxes, but if I only use 'resources/text_boxes' prediction files, that just have the correct 2D bounding box location then the 'annot_dict' contains 'kpts_3d_before' and not 'kpts_3d_gt'. Thus, not able to get the orientation arrow. So I will have to use the txt files that also have the correct 6 fields (other than 2D bboxes) related to the 3D coordinate system for the code to have 'kpts_3d_gt'. As when I use the output files from D4LCN, the 'annot_dict' has 'kpts_3d_gt', which gives me an orientation arrow (Adding the difference in image visualizations below).
Please let me know if I understood this correctly or not. Thank you for the help and sorry for so many questions!

def add_orientation_arrow(self, record):
        """
        Generate an arrow for each predicted orientation for visualization.
        """      
        pred_kpts = record['kpts_3d_pred']
        gt_kpts = record['kpts_3d_gt']
        K = record['K']
        arrow_2d = np.zeros((len(pred_kpts), 2, 2))
        for idx in range(len(pred_kpts)):
            vector_3d = (pred_kpts[idx][1] - pred_kpts[idx][5])
            arrow_3d = np.concatenate([gt_kpts[idx][0].reshape(3, 1), 
                                      (gt_kpts[idx][0] + vector_3d).reshape(3, 1)],
                                      axis=1)
            projected = K @ arrow_3d
            arrow_2d[idx][0] = projected[0, :] / projected[2, :]
            arrow_2d[idx][1] = projected[1, :] / projected[2, :]
            # fix the arrow length if not fore-shortened
            vector_2d = arrow_2d[idx][:,1] - arrow_2d[idx][:,0]
            length = np.linalg.norm(vector_2d)
            if length > 50:
                vector_2d = vector_2d/length * 60
            arrow_2d[idx][:,1] = arrow_2d[idx][:,0] + vector_2d
        return arrow_2d

[Nicholasli1995]

Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?

Hi, to use EgoNet on your custom data:

1. You need to prepare 2D or 3D proposals for your images before using EgoNet.
2. Re-train/fine-tune EgoNet on your data if your domain shift is significant enough. The easiest way is to prepare your images, annotations, and camera parameters in the KITTI style and the existing code can directly be used.

Thank you for the response. Also, to visualize the orientation arrow it requires 'kpts_3d_gt', are these the 3D dimensions that need to there before doing predictions for orientation estimation, as I am not able to visualize the arrow for KITTI test data.

You can use the predicted cuboid ("kpts_3d_pred") to visualize the orientation arrow instead. In this way, you don't need a 3D box input.

But the below code for adding an orientation arrow can be possible only by using both 'kpts_3d_gt' and 'kpts_3d_pred'. I understand that Keypoints are different from 3D bounding boxes, but if I only use 'resources/text_boxes' prediction files, that just have the correct 2D bounding box location then the 'annot_dict' contains 'kpts_3d_before' and not 'kpts_3d_gt'. Thus, not able to get the orientation arrow. So I will have to use the txt files that also have the correct 6 fields (other than 2D bboxes) related to the 3D coordinate system for the code to have 'kpts_3d_gt'. As when I use the output files from D4LCN, the 'annot_dict' has 'kpts_3d_gt', which gives me an orientation arrow (Adding the difference in image visualizations below). Please let me know if I understood this correctly or not. Thank you for the help and sorry for so many questions!

def add_orientation_arrow(self, record):
        """
        Generate an arrow for each predicted orientation for visualization.
        """      
        pred_kpts = record['kpts_3d_pred']
        gt_kpts = record['kpts_3d_gt']
        K = record['K']
        arrow_2d = np.zeros((len(pred_kpts), 2, 2))
        for idx in range(len(pred_kpts)):
            vector_3d = (pred_kpts[idx][1] - pred_kpts[idx][5])
            arrow_3d = np.concatenate([gt_kpts[idx][0].reshape(3, 1), 
                                      (gt_kpts[idx][0] + vector_3d).reshape(3, 1)],
                                      axis=1)
            projected = K @ arrow_3d
            arrow_2d[idx][0] = projected[0, :] / projected[2, :]
            arrow_2d[idx][1] = projected[1, :] / projected[2, :]
            # fix the arrow length if not fore-shortened
            vector_2d = arrow_2d[idx][:,1] - arrow_2d[idx][:,0]
            length = np.linalg.norm(vector_2d)
            if length > 50:
                vector_2d = vector_2d/length * 60
            arrow_2d[idx][:,1] = arrow_2d[idx][:,0] + vector_2d
        return arrow_2d

Hi, the gt_kpts were used only to get the translation so that the arrow can be computed in the camera coordinate system.

1. If you have a 3D proposal, you can simply modify the code to use the proposed translation as well.
2. If you only have a 2D proposal, you can draw the arrow in the Bird's Eye View. Otherwise, you can approximate the 2D arrow with the predicted orientation.