Surgical AR Review
Recent advancements in AR for robotic surgery
- venue
- MDPI Actuators
- year
- 2023
Abstract
Robots are expanding from industrial applications to daily life, and augmented reality (AR) has been integrated into many robotic applications across medical and industrial domains. This survey reviews AR for medical robots in preoperative and surgical task planning, image-guided robotic surgery, surgical training and simulation, and telesurgery; and for industrial robots in human–robot interaction and collaboration, path planning and task allocation, training and simulation, and teleoperation control/assistance. Limitations, challenges, and future perspectives are discussed for researchers working at the intersection of AR and robotics.
Introduction
Robotic systems have become prevalent in medical and industrial applications, offering high positioning accuracy, improved dexterity, and reduced trauma in surgery, and productivity and consistency in manufacturing. AR can transcend physical interaction limits by integrating holographic information onto real scenarios, improving situational awareness before and during task execution. This review synthesizes how AR has been applied to medical and industrial robotic systems, organizing the literature by application theme and identifying cross-cutting challenges.
Methodology
The survey follows a structured literature review across two domains. Medical robotics is organized into four themes: (1) preoperative and surgical task planning with CT/MRI registration and port placement optimization; (2) image-guided robotic surgery with rigid and deformable registration on da Vinci and custom platforms; (3) surgical training and simulation including AR-guided curricula and haptic augmentation; and (4) telesurgery covering remote visualization, teleoperation, and latency mitigation via predictive displays and digital twins. Industrial robotics is reviewed across human–robot collaboration, path planning, training/simulation, and teleoperation, spanning HMD, projector-, and tablet-based AR on platforms from HoloLens to KUKA and Universal Robots.
Results
Across medical AR robotics, preoperative planning applications improve target localization and robot configuration; image-guided surgery overlays span rigid orthopedics to deformable abdominal procedures; training systems demonstrate skill acquisition benefits with AR feedback; and telesurgery work addresses latency through predictive displays. In industrial settings, AR assists gesture-, gaze-, and BCI-driven collaboration, intuitive path planning and programming, mixed-reality training, and teleoperation with virtual fixtures. Common findings include trade-offs between registration accuracy, field of view, hardware ergonomics, and cognitive load across both domains.
Conclusion
AR holds promise for improving operator awareness in robotic applications, but immersion, hardware comfort, registration accuracy, safety, communication latency, and simulation-to-real transfer remain open challenges. Future directions include more ergonomic displays, multimodal interaction (haptic, audio, gaze), reduced hardware cost as consumer AR matures, and tighter integration of digital twins for predictive assistance in both surgical and industrial teleoperation.
Collaborators
- Junling Fu
- Shufei Li
- Jianzhuang Zhao
- Qingsheng Liu
- Elisa Iovene
- Giancarlo Ferrigno
- Elena De Momi