Mehmet Baha Dursun · University of Southern Denmark · supervised by Iñigo Iturrate
Student Research Assistant, October 2025 - May 2026
Project Report• Starting Paper• Short Result Video• Supervisor
I built and integrated the complete learning pipeline for a bimanual suturing task on the physical da Vinci Research Kit: robot and camera acquisition, synchronized demonstrations, dataset conversion, policy architecture, training, ROS 1 deployment, closed-loop execution, diagnostics, and real-robot evaluation.
I first came across Diffusion Policy while watching LeRobot project videos. From there, I followed the references to Cheng Chi's original work and studied related robot-learning research from Toyota Research Institute in parallel. That was how I learned the action-diffusion idea before adapting it to the SDU dVRK setup.
The robot interface was grounded in Johns Hopkins University's open da Vinci Research Kit software. I built the project-specific recording, policy, control, and evaluation layers around the physical SDU setup.
The local starting point was an SDU study of uncertainty-aware diffusion policies and control-barrier safety in simulation. My work addressed the missing physical pipeline and the train-to-robot gap on the real dVRK.
I adapted feature-decoding ideas and utilities from LeWorldModel, whose authors include Yann LeCun, then integrated them with my own physical pose probes, modality attribution, and needle-localization tests.
My first policies used the full proprioceptive state together with the endoscope image. At the beginning I treated the failure as a data or control representation problem. I rebuilt the pipeline around absolute Cartesian targets, step-relative motion, and finally anchored-relative targets. I also changed the rotation representation to rot6d, reworked normalization, and checked that the dataset labels and the ROS execution path decoded the same 20D command for PSM1 and PSM2.
Those changes fixed real implementation problems, but not the main behavior. If I moved the needle or raised the suturing pad, the ResNet policy still tended to replay a grasp associated with the current arm pose. By then it was clear to me that collecting more of the same demonstrations would not explain the failure. I first needed to know what the image encoder was actually preserving.
Inspired by the state-input dropout used in NVIDIA Isaac GR00T, I trained ResNet policies while randomly or completely removing the robot state. The physical results became much worse instead of recovering from the image. That was an important clue: removing the shortcut did not help if the visual features still failed to represent the needle.
To inspect the encoder directly, I started from the decoder feature-inversion approach in LeWorldModel, developed by Yann LeCun and collaborators, and adapted it to the dVRK checkpoints. I then expanded it into a diagnostic pipeline with decoder reconstructions, linear and MLP tool-pose probes, image-versus-state counterfactual ablations, and a patch-similarity test that queried the thin suturing needle across different frames.
The decoder was the first useful visualization. ResNet retained enough of the broad scene and robot geometry to support low offline loss, but its reconstruction blurred away the thin needle. The DINOv2 reconstruction kept both the needle and the suture much more clearly. A decoder image alone is not proof, so I used the other probes to check the same conclusion from different directions.
After the ResNet diagnostics, I replaced the visual front-end with a fine-tuned DINOv2 ViT-S/14. I kept the dataset, anchored-relative action representation, diffusion model, and physical deployment path fixed so that the encoder was the main change. The DINO patch features followed the needle across frames, whereas the ResNet similarity remained spread over the tissue and tool bodies.
DINOv2 without dropout still used some robot state, so I trained a controlled sweep with state-dropout probabilities 0, 0.2, and 1.0. The measured vision share rose from 0.76 to 0.975 and then 0.995. In the final state-free run, the policy had to form its action from the endoscope images and no longer showed the same state-linked failure in the tested physical trials.
The encoder comparison is documented as an individual project report. The physical observations are qualitative, and the report states the remaining resolution and probe-sample confounds explicitly.
