Permit Debug Access to AI2-THOR's Third-Party Cameras

[ThomasSchellenbergNextCentury]

From: Falk https://machinecommonsense.slack.com/archives/CTR31V12M/p1586813963013500

Priority: Backlog

Components: Python API, Unity Env

Blockers: None

Goal: Can we add a function to the MCS_Controller_AI2THOR class in our Python API that uses AI2-THOR to generate a “third-party camera” in our MCS scene and return a series of images from it following each action step?

Possible issues: In MCSPerformerManager, we’re overriding FinalizeMultiAgentMetadata as a “hack” to send multiple images from Unity to our Python API that have been made over each of our X action “substeps”. This may or may not affect how the images from the third-party cameras are returned to the Python API.

Tasks:

• In the Python API, create a new function in the MCS_Controller and MCS_Controller_AI2THOR to add a third-person camera in a specific location (do something simple for now, and we can change it during the code review if needed).
• Ensure that AI2-THOR will properly generate the third-person camera and return its images.
• In the Python API, add a new property to MCS_Step_Output for the third-person camera images. Correctly populate that property in the output of the “start_scene” and “step” functions in MCS_Controller_AI2THOR.

Docs: https://ai2thor.allenai.org/ithor/documentation/actions/miscellaneous-actions/#add-camera

TA2 internal JIRA ticket: https://nextcentury.atlassian.net/browse/MCS-165