Abstract
Efficient and robust tracking of surgical robotic instruments is important for robot-assisted minimally invasive surgery, yet remains challenging due to the complexity of surgical scenes and the unconventional geometry of surgical instruments. Keypoint-based approaches are efficient, but their performance depends on reliable feature detection. Improving these detectors with real-world supervision is difficult because accurate real-world annotations are costly to obtain at scale. To address this limitation, we introduce a tracker-guided self-training framework that adapts a model pretrained on synthetic images to unlabeled real-world videos. Given measured robot joint states, an uncertainty-aware EKF recursively corrects the instrument pose and the observable joint angles by comparing projected model features with detected keypoints, shaft boundaries, and mask-derived cues. An RTS smoother subsequently refines the resulting trajectory, which is projected into pseudo-labels for fine-tuning the feature detector without laborious pose annotations. Experiments on real-world videos demonstrate consistent improvements from self-training across all evaluated keypoint metrics, and the resulting model outperforms prior approaches in both accuracy and runtime. The code and data will be released upon publication.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Hu, H., Liang, Z., Richter, F., & Yip, M. C. (2026). Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training. https://omanscience.com/en/articles/robust-surgical-robotic-instrument-tracking-via-sequential-multi-cue-fusion-and-sim-to-real-self-training
MLA 9
Hu, Hanyang, et al. "Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training." https://omanscience.com/en/articles/robust-surgical-robotic-instrument-tracking-via-sequential-multi-cue-fusion-and-sim-to-real-self-training.
Chicago (author–date)
Hu, Hanyang, Zekai Liang, Florian Richter, and Michael C. Yip. 2026. "Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training." https://omanscience.com/en/articles/robust-surgical-robotic-instrument-tracking-via-sequential-multi-cue-fusion-and-sim-to-real-self-training.
Harvard
Hu, H., Liang, Z., Richter, F. and Yip, M. C. (2026) 'Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training', Available at: https://omanscience.com/en/articles/robust-surgical-robotic-instrument-tracking-via-sequential-multi-cue-fusion-and-sim-to-real-self-training.
Vancouver
Hu H, Liang Z, Richter F, Yip MC. Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training. https://omanscience.com/en/articles/robust-surgical-robotic-instrument-tracking-via-sequential-multi-cue-fusion-and-sim-to-real-self-training
IEEE
H. Hu, Z. Liang, F. Richter, and M. C. Yip, "Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training," https://omanscience.com/en/articles/robust-surgical-robotic-instrument-tracking-via-sequential-multi-cue-fusion-and-sim-to-real-self-training.