"unlabelled" door in the instance segmentation map

[MhdMohammadi]

I was using semantic segmentation and instance segmentation maps, and I realized object door_lvncak_0 in Beechwood_0_garden is marked as unlabelled in instance segmentation map, however, it is correctly detected as door category in semantic segmentation map. I wonder why this issue exists, and how we can access the id of this door when it's observed.

[hang-yin]

Hi @MhdMohammadi , I just tried importing a Fetch robot into Beechwood_0_garden and had it point its camera to door_lvncak_0. It seems to me that the returned instance segmentation map has door_lvncak_0 correctly marked with an id instead of unlabelled. Would you mind sending me a script to reproduce this? It could be possible that the order of seeing these objects in the scene affect how their IDs are shown.

[MhdMohammadi]

Hi @hang-yin ! thank you so much for your quick response. I checked it in a simple setting, and you are right. However, the issue still exists, but in a different setting.

First I should explain that because we could not rescale the robot, we decided to take non-physical actions, and also we take magic actions to open doors. We used the existing functions in OmniGibson codebase, and improved them to work without an error.

To reproduce the problem, I would like to kindly ask you to run a code as below. dummy_step is a function to perform a non-physical update, and open_object is used to open doors. If you know any alternative implementation for dummy_step, please let us know!

import omnigibson as og
from omnigibson import object_states 
from omnigibson.object_states.open_state import _compute_joint_threshold
from omnigibson.utils.usd_utils import PoseAPI

def dummy_step():
    PoseAPI._refresh()
    for i in range(1):
        og.sim._non_physics_step()
    for i in range(3):
        og.sim.render()

def open_object(object):
    _, relevant_joints, joint_directions = object.states[object_states.Open].relevant_joints_info

    # Go through the relevant joints & set random positions.
    for joint, joint_direction in zip(relevant_joints, joint_directions):
        threshold, open_end, closed_end = _compute_joint_threshold(joint, joint_direction * 1)

        joint_pos = open_end

        # Save sampled position.
        joint.set_pos(joint_pos)

    for joint, joint_direction in zip(relevant_joints, joint_directions):
        joint.keep_still()

    dummy_step()

    object.states[object_states.Open].clear_cache()

def main():
    env = TODO # Initialize the environment

    doors = env.scene.object_registry('category', 'door')
    for door in doors:
        if door.name in ['door_lvgliq_0', 'door_lvncak_0']:
            open_object(door)
     
    ## TODO: navigate to `door_lvncak_0`
    

if __name__ == "__main__":
    main()

In my experiments, I realized door_lvncak_0 is unlabelled in instance segmentation map. I attached the visualization for rgb sensors, and another visualization that marks the unlabelled part in white color.

I realized if I open both of door_lvgliq_0 and door_lvncak_0, this issue appears, and if I only open door_lvncak_0, this issue doesn't exist anymore.

I truly appreciate if you can test the described setting, and if you have any questions, please let me know!

[ChengshuLi]

@MhdMohammadi Does this problem still persist?

[hang-yin]

Hi @MhdMohammadi , here's a proposed fix for your problem: #962

I was able to reproduce your issue with this script:

import torch as th

import omnigibson as og
import omnigibson.lazy as lazy
from omnigibson.macros import gm
from omnigibson import object_states
from omnigibson.object_states.open_state import _compute_joint_threshold
from omnigibson.utils.usd_utils import PoseAPI


def dummy_step():
    PoseAPI._refresh()
    for i in range(1):
        og.sim._non_physics_step()
    for i in range(3):
        og.sim.render()


def open_object(object):
    _, relevant_joints, joint_directions = object.states[object_states.Open].relevant_joints_info

    # Go through the relevant joints & set random positions.
    for joint, joint_direction in zip(relevant_joints, joint_directions):
        threshold, open_end, closed_end = _compute_joint_threshold(joint, joint_direction * 1)

        joint_pos = open_end

        # Save sampled position.
        joint.set_pos(joint_pos)

    for joint, joint_direction in zip(relevant_joints, joint_directions):
        joint.keep_still()

    dummy_step()

    object.states[object_states.Open].clear_cache()


def main():

    cfg = {
        "scene": {
            "type": "InteractiveTraversableScene",
            "scene_model": "Beechwood_0_garden",
            "load_object_categories": ["floors", "walls", "ceilings", "door"],
        },
    }

    # Load the environment
    env = og.Environment(configs=cfg)

    # Allow user to teleoperate the camera
    cam_mover = og.sim.enable_viewer_camera_teleoperation()

    og.sim.viewer_camera.add_modality("seg_semantic")
    og.sim.viewer_camera.add_modality("seg_instance")

    # og.sim.viewer_camera.set_position_orientation(
    #     position=[0.6210, 13.0696, 1.2124], orientation=[0.6892, -0.0175, -0.0184, 0.7241]
    # )

    doors = env.scene.object_registry("category", "door")
    for door in doors:
        if door.name in ["door_lvgliq_0", "door_lvncak_0"]:
            open_object(door)

    import matplotlib.pyplot as plt

    step = 0
    while True:
        env.step([])
        obs, info = og.sim.viewer_camera.get_obs()
        if step % 2000 == 0:
            instance_seg_map = obs["seg_instance"].cpu()
            # for all pixels in this map, if they are 1, change to 256, if not, set to 0
            instance_seg_map[instance_seg_map == 1] = 256
            instance_seg_map[instance_seg_map != 256] = 0
            plt.imshow(instance_seg_map)
            plt.show()
            breakpoint()
        step += 1


if __name__ == "__main__":
    main()

Please let me know if this works for you!! : )

[MhdMohammadi]

hi @hang-yin,

thank you for your response! I tried the fix in a few settings, and it seems to work. I'll get back to you later if I face another issue.