This repo is the official code release for MAGIC:
|
One-Shot Manipulation Strategy Learning by Making Contact Analogies
|
We present a novel approach, MAGIC (manipulation analogies for generalizable intelligent contacts), for one-shot learning of manipulation strategies with fast and extensive generalization to novel objects. By leveraging a reference action trajectory, MAGIC effectively identifies similar contact points and sequences of actions on novel objects to replicate a demonstrated strategy, such as using different hooks to retrieve distant objects of different shapes and sizes.
Our method is based on a two-stage contact-point matching process that combines global shape matching using pretrained neural features with local curvature analysis to ensure precise and physically plausible contact points. We experiment with three tasks including scooping, hanging, and hooking objects. Our pipeline combines data-driven (pretrained DINOv2 visual features) and analytic (curvature) approaches from one single demonstration, without any additional training or task specific datasets.
MAGIC demonstrates superior performance over existing methods, achieving significant improvements in runtime speed and generalization to different object categories.
Start by creating a new conda environment.
conda create -n magic python=3.10
conda activate magic
pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121Then install Jacinle.
git clone https://github.com/vacancy/Jacinle --recursive
export PATH=<path_to_jacinle>/bin:$PATH
cd Jacinle
pip install -e .
cd ..The commands in the following block is optional, only required if you want to try out image features from SD-DINO and DIFT.
cd image_features/third_party/Mask2Former
pip intall -e .
cd ../ODISE
pip install -e .
cd ../../..If it raises error The detected CUDA version (xx.x) mismatches the version that was used to compile PyTorch (xx.x). Please make sure to use the same CUDA versions. while building detectron2, please install the corresponding CUDA toolkit with sudo apt-get -y install cuda-toolkit-xx-x, and then add export PATH=/usr/local/cuda-xx.x/bin/:$PATH to ~/.bashrc.
Finally, install the other dependencies.
pip install -r requirements.txtDownload the meshes and urdf files from this link, and extract it as the ./assets folder. The assets folder should look like this:
assets
├── custom_objects
├── ig
├── panda
├── shapenet_mug
└── shapenet_spoon
To reproduce our simulation results with one click, run the following commands. You can see the rendered videos of the trajectory in ./videos, the visualization of global contact points found by DINOv2 and the local alignment established by curvature ./results. Before running the experiments, make sure that Vulkan is configured correctly following the instructions here, otherwise Segmentation Fault might be raised.
# scooping
# spoon_id can be 1, 2, ..., 6
jac-run scripts/scooping.py --spoon_id 1 --target 0.035 --save_video 1
# hanging mug_id can be 0, 1, ..., 200
jac-run scripts/hanging.py --mug_id 1 --save_video 1
# hooking
jac-run scripts/hooking.py --tool scissors --target 0.015 --save_video 1
jac-run scripts/hooking.py --tool hanger --target 0.02 --save_video 1
jac-run scripts/hooking.py --tool caliper --target 0.025 --save_video 1
jac-run scripts/hooking.py --tool watch --target 0.035 --save_video 1Moreover, you can visualize the similarity heatmap generated by DINOv2, SD-DINO, and DIFT with in the jupyter notebook visualize_heatmap.ipynb.
MAGIC is based on many open-source projects, including SAPIEN2, ManiSkill2, MPLib, SD-DINO, and DIFT. We thank all these authors for their nicely open sourced code and their great contributions to the community.
MAGIC is licensed under the MIT license. See the LICENSE file for details.