Pengyuan Wang

I am a PhD student at TUM CAMP computer vision group supervised by Prof. Dr. Nassir Navab. My research focuses on object pose estimations and robotic manipulations.

Email  /  Scholar  /  Linkedin  /  Github

Research

My research interest is in the pose estimation of category-level objects and novel objects, which is important for robotic applications. Also I am interested in vision-language-action models for generic robotic manipulations.

GS-Pose enables training category-level object pose estimations from only a few synthetic CAD models , and achieves comparable results with baselines trained on real pose dataset.

A large-scale category-level pose dataset with accurate ground truth pose annotations. The dataset features photometrically-challenging categories including transparent and metallic objects, and more diverse object poses in the real scenes. Feel free to test your pose estimation methods on our benchmark.

We propose novel self-supervision losses to self-supervise the category-level pose estimations for transparent objects. Only RGB-D with polarization images and no real pose annotations are needed in the self-supervision stage.

CCD-3DR works on generative networks imagining object point clouds from a single RGB image. Accurate point clouds are generated by diffusion networks with consistent object center conditioning.

How to estimate the object pose of metallic objects from RGB-D inputs in presence of depth artifacts? CroCPS proposes a multi-modal approach to improve pose estimation accuracy in real scenes for metallic category.

PPP-Net proposes a hybrid model that utilizes polarizaiotn information with physical priors in a data-driven learning strategy to improve the accuracy of pose predictions for photometric challenging objects.

We propose a novel method to annotate object poses accurately with a robot arm, especially we design a calibration pipeline for hand-eye calibration of the cameras. Our method annotates the object poses accurately, even for photometrically-challenging objects such as glasses and cutlery.

DemoGrasp proposes to learn robotic grasping from human demonstrations. In the method we learn the grasping pose and object shape jointly in a demonstration sequence. Afterwards we propose a novel object pose estimator to get object poses in the real scenes as well as the grasping point.

Side Projects

Visual prompting recognizes the keypoint positions in the images by specifying the keypoint names with CLIP. The method is achieved by drawing circles around the keypoints. We unofficially implement the method with CLIP models and optimal transport functions, and provide a testing example in github.