J Thorac Cardiovasc Surg 2008;135:196-202
© 2008 The American Association for Thoracic Surgery
Effects of visual force feedback on robot-assisted surgical task performance
Carol E. Reiley, MSa,
Takintope Akinbiyi, MSa,
Darius Burschka, PhDb,
David C. Chang, PhD, MPH, MBAc,
Allison M. Okamura, PhDa,
David D. Yuh, MDa,d,*
a Engineering Research Center and Computer Integrated Surgery Systems and Technology, Johns Hopkins University, Baltimore, Md
b Department of Computer Science, Technical University of Munich, Munich, Germany
c Program for Outcomes Research, Department of Surgery, Johns Hopkins School of Medicine, Baltimore, Md
d Division of Cardiac Surgery, Johns Hopkins Medical Institutions, Baltimore, Md.
Received for publication April 16, 2007; revisions received August 9, 2007; accepted for publication August 30, 2007.
* Address for reprints: David D. Yuh, MD, Division of Cardiac Surgery, Johns Hopkins Hospital, 600 North Wolfe Street, Blalock 618, Baltimore, MD 21287-4618. (Email: dyuh{at}csurg.jhmi.jhu.edu).
 |
Abstract
|
|---|
Objective: Direct haptic (force or tactile) feedback is negligible in current surgical robotic systems. The relevance of haptic feedback in robot-assisted performances of surgical tasks is controversial. We studied the effects of visual force feedback, a haptic feedback surrogate, on tying surgical knots with fine sutures similar to those used in cardiovascular surgery.
Methods: By using a modified da Vinci robotic system (Intuitive Surgical, Inc, Sunnyvale, Calif) equipped with force-sensing instrument tips and real-time visual force feedback overlays in the console image, 10 surgeons each tied 10 knots with and 10 knots without visual force feedback. Four surgeons had significant prior da Vinci experience, and the remaining 6 surgeons did not. Performance parameters, including suture breakage and secure knots, peak and standard deviation of applied forces, and completion times using 5-0 silk sutures, were recorded. Chi-square and Student t test analyses determined the differences between groups.
Results: Among surgeon subjects with robotic experience, no differences in measured performance parameters were found between robot-assisted knot ties executed with and without visual force feedback. Among surgeons without robotic experience, however, visual force feedback was associated with lower suture breakage rates, peak applied forces, and standard deviations of applied forces. Visual force feedback did not impart differences in knot completion times or loose knots for either surgeon group.
Conclusions: Visual force feedback resulted in reduced suture breakage, lower forces, and decreased force inconsistencies among novice robotic surgeons, although elapsed time and knot quality were unaffected. In contrast, visual force feedback did not affect these metrics among surgeons experienced with the da Vinci system. These results suggest that visual force feedback primarily benefits novice robot-assisted surgeons, with diminishing benefits among experienced surgeons.
Abbreviations and Acronyms VFF = visual force feedback
|
Introduction
|
|---|
| |
Dr Yuh
|
|
Robot-assisted surgical systems, such as the teleoperated da Vinci surgical system1
(Intuitive Surgical Inc., Sunnyvale, Calif), have facilitated minimally invasive approaches to selected cardiothoracic operations, including mitral valve repair,2 coronary artery bypass grafting,3 and thoracic tumor resections.4 The enhanced degrees of freedom afforded by such computer-augmented systems over conventional manually actuated thoracoscopic instruments permit the dextrous manipulations required to perform these operations in the relatively confined and rigid thoracic space. However, currently available robotic surgical systems do not yet provide noticeable force or tactile feedback, otherwise known as haptics, to the operating surgeon. There exist several practical and technical challenges that need to be overcome to implement direct haptic capabilities to the hands of the surgeon in complex surgical systems.
Although it would seem intuitive that haptic feedback would be critically important in the manipulation of the delicate tissues and suture materials inherent in cardiothoracic operations, its clinical relevance in the performance of robot-assisted surgical tasks is controversial.5,6 The effects of haptic feedback on the performance of robot-assisted cardiothoracic surgery have not been well characterized. Many successful robot-assisted surgeons suggest that the lack of sensory force feedback can be adequately compensated for with the use of visual cues, such as local tissue deformation incurred with tension, retraction, or needle insertion. Conversely, we previously reported that sensory substitution, a provisional construct to approximate true haptic feedback, resulted in objective advantages during the performance of robot-assisted surgical tasks, including improved applied force accuracy and consistency.5
We hypothesize that haptic feedback confers quantifiable improvements in applied force and technical error metrics during robot-assisted surgical task performances. We use a modified da Vinci robotic system equipped with force-sensing instrument tips and real-time graphic overlays to study the effects of visual force feedback (VFF) on tying surgical knots with fine suture materials. We also sought to determine whether the effects of VFF differed between surgeons with and without prior experience operating with the da Vinci system.
|
Materials and Methods
|
|---|
da Vinci Surgical Robotic System
We used the da Vinci surgical system as our teleoperated robotic platform. The da Vinci surgical system consists of 3 major components: an input device (master-side console), a computer interface, and an output device (patient-side console).
da Vinci Instrument Modifications
Force sensors were added to detect the instruments’ bending forces. Two sets of strain gauges in full Wheatstone bridge arrangements were orthogonally placed along each of the instrument shafts to measure the forces being applied at each instrument’s tip (Figure 1). Customized "clamshell" cannulae were used to accommodate the modified instruments and prevent damage to the strain gauges. Simulations were performed to determine ideal strain gauge positions in relation to the instrument tip. The position and orientation of the da Vinci instrument tips and stereoendoscope were tracked using robot kinematics. Because of the low-resolution encoders on the robot "setup" joints (ie, the joints closest to the robot base that remain stationary during instrument manipulations) and flexure in the kinematic chain, we noted a discrepancy of up to 3 cm between the true position of the instrument tips and where the da Vinci Application Program Interface software reports them to be. To correct this offset, a calibration was performed between each instrument and the stereoendoscope.
View larger version (80K):
[in this window]
[in a new window]
|
Figure 1. Two sets of strain gauges orthogonally mounted on the da Vinci (Intuitive Surgical, Inc, Sunnyvale, Calif) instrument shaft detect bending forces in 2 degrees of freedom. The applied forces at each instrument tip were calculated as the vector sum for F1 and F2, where F1 and F2 are the forces detected by the 2 orthogonally placed strain gauge bridges. A clamshell cannula was designed to secure the instrument shaft while preventing damage to the force sensors and wires that ran along the shaft.
|
|
Real-time Visual Force Feedback Graphic Overlays
We developed a VFF system schematically represented in Figure 2. Frame grabbers are image acquisition computer boards that capture and store image data. In this system, frame grabbers capture real-time stereoscopic images from the da Vinci InSite video system and convert the video signals from analog to digital to be processed by the computer. A semitransparent graphic overlay is superimposed on the instrument image displayed on the surgeon’s master console. This overlay is derived from force magnitude data obtained from the corresponding strain gauges and computed kinematics data from the da Vinci software (ie, Application Program Interface).
View larger version (16K):
[in this window]
[in a new window]
|
Figure 2. System schematic of VFF. Frame grabbers obtain stereoscopic images from the da Vinci InSite stereoscopic video system, and a semitransparent overlay is placed over the instrument image based on force magnitude data obtained from the instrument strain gauges and computed kinematics data from the da Vinci application program interface. VGA, Video graphics array.
|
|
The applied forces at each instrument tip were calculated as the vector sum for F1 and F2, where F1 and F2 are the forces detected by the strain gauge bridges (Figure 1). A weighted average over a time interval of these measurements was used to smooth out high-frequency readings. The force magnitudes for each of the 2 force-sensing da Vinci instruments were graphically overlaid on the surgeon’s stereoscopic display console at 30 frames per second. The force display consists of 2 semitransparent circles superimposed over the corresponding instrument tips as they move freely in space (Figure 3). These VFF overlays changed color across 3 discrete force magnitude intervals dictated by bending forces measured by the strain gauges on the instrument shafts. The force thresholds were set by an experienced da Vinci surgeon on the basis of an average of the forces the surgeon exerted while performing this task.
View larger version (53K):
[in this window]
[in a new window]
|
Figure 3. Video sequence of knot-tying trial showing the semitransparent colored circle VFF overlays. Minimal to no applied force (green circle), ideal applied force (yellow circle), and excessive applied force (red circle).
|
|
Force Range
The force ranges were as follows. Minimal force range (green indicator): minimal or no force applied by the surgeon. Ideal force range (yellow indicator): ideal applied force range for tying a secure knot. Excessive force range (red indicator): excessive force with significant potential to break suture or damage tissue.
Human Subjects
Ten surgeon subjects (3 attending cardiac surgeons and 7 postgraduate year VIII cardiothoracic surgical residents; Division of Cardiac Surgery, The Johns Hopkins Hospital, Baltimore, Md) participated in this study. Four of the surgeon subjects had completed a formal da Vinci training course and had substantial prior experience using the da Vinci system. These studies were performed under a protocol titled "Augmented Reality Displays for Robot-Assisted Surgical Systems," which was approved by the Johns Hopkins University Institutional Review Board on January 27, 2005.
Knot-tying Task Trials
The surgical task consisted of securing the second throw of a 5-0 silk surgical knot on a standard knot-tying training board (SmithKline Beecham, SKB-10; London, England) with the modified da Vinci surgical system. To standardize the task, all surgeons were instructed to tighten the knot by (1) grasping the suture with the 2 forceps close to the knot and (2) pulling the suture laterally in the plane of the task board. We selected 5-0 silk sutures for this experiment to render our findings relevant to a broad spectrum of surgical specialties and (2) to avoid the potentially confounding effects of suture breakage caused by fracture (rather than excessive tension) in grasping the suture with the da Vinci forceps.
Each surgeon performed 20 knot-tying trials, 10 with VFF and 10 without VFF. The first trials with and without VFF were used as practice trials and were not included in the analysis. Thus, for the purposes of analysis, each surgeon subject performed a total of 18 trials. A total of 180 trials were performed by these surgeons, 72 from surgeons with da Vinci experience and 108 from surgeons without prior experience. The order of trials was randomized. For each trial, the recorded parameters included instrument/environment bending forces, instrument tip position, suture breakage, knot security (ie, secure vs loose knots), and task completion time.
Survey
A written qualitative survey was completed by each subject at the conclusion of the trials to assess whether or not VFF was preferred and the reason(s) for the preference.
Statistical Analysis
By using a significance level of P less than .05, we conducted chi-square analyses to determine whether significant differences in the number of broken sutures or loose knots existed between trials performed with and without VFF. Student t test analyses were used for peak force, force standard deviation, and task completion time comparisons. We also compared these performance variables between surgeons with and without prior operative experience with the da Vinci system. Statistical computations were performed with Stata 9.2 statistical software (StataCorp LP, College Station, Tex).
|
Results
|
|---|
Effect of Visual Force Feedback on Task Performance
In the knot-tying trials, there was only 1 suture breakage incident in 36 trials among da Vinci-trained surgeons compared with 11 incidents in 54 trials among surgeons without prior da Vinci experience (2.8% vs 20.4%, P < .01, 
2) (Figure 4). The average suture breaking force was 4.40 Newtons with a standard deviation of 1.62 compared with 2.71 ± 1.14 for the trials without suture breakage. Notably, VFF completely eliminated suture breakage among surgeons without prior da Vinci training.
View larger version (19K):
[in this window]
[in a new window]
|
Figure 4. Comparisons of broken sutures (A), peak applied force (B), standard deviations (C), knot-tying completion times (D), and loose knots (E) between tasks performed with and without VFF. *P < .05, **P < .01. VFF, Visual force feedback.
|
|
Among surgeons without da Vinci experience, significantly reduced peak applied forces (2.42 vs 3.12 Newtons, P < .01, t test) and lower applied force standard deviations (0.64 vs 1.20 Newtons, P < .03, t test) in knot-tying trials were associated with VFF. VFF did not affect any of these performance measures among surgeons with prior da Vinci experience.
Finally, VFF did not significantly reduce task completion times or the number of loose knots regardless of prior da Vinci experience. Task completion times among da Vinci-trained surgeons with and without VFF were 6.04 versus 6.31 seconds, respectively (P > .05, t test). Among surgeons without robotic experience, completion times were 6.46 versus 6.62 seconds (P > .05, t test), respectively. Among da Vinci-trained surgeons, no loose knots were noted in 36 trials with VFF versus 2 of 36 trials without VFF (0% vs 5.6%, P > .05, 2). Among surgeons without prior da Vinci experience, 3 loose knots were noted among 54 trials (5.6%) performed with and without VFF.
Qualitative Survey Results
VFF was preferred by all subjects during the knot-tying trials largely because of the confidence it conferred in avoiding excessive applied forces while ensuring a secure knot.
|
Discussion
|
|---|
The computerized interface between the surgeon and the robotic surgical instrumentation provides several advantages over manually actuated thoracoscopic approaches in performing minimally invasive operations. Increased dexterity, tremor filtering, and motion scaling afforded by these systems have greatly facilitated the minimally invasive performance of highly dextrous surgical tasks inherent in many cardiothoracic operations. Nevertheless, the physical separation between the operating surgeon and instrumentation in current teleoperated robot-assisted surgical systems, including the da Vinci, presently deprive the surgeon of noticeable force and tactile feedback during performance of conventional suture and tissue manipulation. The commercially available da Vinci system does have direct haptic feedback capabilities, but the magnitude of force fed back to the surgeon during typical surgical maneuvers is so low as to have a negligible effect on perception and performance. The low-quality feedback is due to inaccuracies in the estimation of tool/tissue interaction forces.
The clinical importance of haptic feedback in the conduct of robot-assisted operations is controversial and has prompted several recent studies.7,8 During a blunt dissection task, Wagner and colleagues9 found that the presence of force feedback in a teleoperated system reduced the peak force magnitude by a factor of 2 and the number of errors by a factor of 3 versus no force feedback. Wagner and coworkers also concluded that these gains resulted in better user performance. Other studies have suggested that inadvertent tissue trauma is, at least in part, related to the lack of haptic feedback.6,10,11 Nevertheless, most published series of robot-assisted cardiac operations do not report heightened complications related to the lack of haptic feedback.2,12-14
Because direct force feedback has heretofore not been integrated into current commercial surgical robotic systems, recent experimental work has relied on a provisional construct to approximate true haptic feedback. Sensory substitution is the replacement of one sense, in this case direct tactile or force feedback, with another sensory modality that, here, relays to the surgeon a representation of the forces applied by the robotic telemanipulators. Our group previously compared applied force parameters during a standardized surgical knot-tying task under 4 different sensory-substitution scenarios: no feedback, auditory feedback, visual feedback, and combined auditory-visual feedback.5 We found that either sensory substitution mode, alone or in combination, resulted in quantifiable improvements of applied force accuracy and consistency during traditional or robot-assisted task executions. Similarly, Prasad and colleagues,15 Sabatini and colleagues,16 and Tavakoli and colleagues17 reported functional advantages in performing telemanipulations with sensory substitution. We developed a more sophisticated 3-dimensional stereo VFF overlay that tracks and moves with the robotic instrumentation in real time, unlike the early experimental sensory substitution systems that used static VFF overlay information. We believe that this is the first report of a VFF augmented reality system that is fully integrated into the da Vinci system. The tool tracking overlay allows surgeons to more accurately estimate the force magnitudes they are applying without having to divert their eyes away from the instruments. On the basis of our qualitative survey, all surgeons derived confidence or reassurance of appropriate force application from our VFF system and preferred performing the task with it on because it helped reduce the mental workload required to perform the task.
No significant differences in the measured performance parameters were noted between task execution with and without VFF among surgeon subjects with prior da Vinci experience. However, significantly fewer broken sutures and reduced peak and standard deviations of applied forces were observed among surgeons with little to no da Vinci experience. This observation suggests that VFF provides novice da Vinci surgeons the means to avoid excessive forces and to apply more consistent forces in this particular task performance. It is probable that experienced da Vinci surgeons were more adept at using natural visual cues to gauge suture tension than their less-experienced colleagues.
Despite the differential results between da Vinci-trained and untrained surgeons, our qualitative subjective survey indicates that all surgeons preferred having VFF because it conferred confidence as an objective indicator of applied force magnitudes. The quantal green-yellow-red color scheme used to segment the different force ranges quickly and effectively conveyed force information to the surgeon. Although we considered more graded VFF schemes in the design process, we thought they would be too distracting to the surgeon and too subtle to quickly alert the surgeon of excessive applied forces.
VFF did not significantly reduce the number of loose knots or task completion times. The lack of effect of VFF on knot security might be due to the fact that knot security can be readily determined visually without haptic feedback. Similar task completion times between trials with and without VFF might be explained by our observation that there was substantial variation between how each subject used the overlay. For example, in trials with VFF, some subjects released the suture exactly when the overlay indicated excessive tension, and others varied tension between the ideal and the excessive ranges to ensure a secure knot.
The technical challenges of this experiment led to several study shortcomings. First, we recognize that 5-0 silk sutures are not widely used in cardiac operations. We selected this material to avoid confounding our results with suture breakage caused by monofilament fracture from excessive instrument grasping forces that we could not measure. We selected a fine braided suture to minimize this effect and to make our results translatable to other noncardiac specialties. Second, we were only able to measure applied instrument bending forces rather than grasping forces or forces parallel to the instrument axis. We attempted to mitigate this shortcoming by orienting the instruments in the workspace such that the knot-tying forces would be primarily in the directions that we could measure. Third, as noted above, we did not standardize the manner in which VFF was used by individual surgeon subjects, leading to some methodologic variability. We thought that it would be difficult and artificial to enforce a "correct" way of using this force-feedback modality and, in fact, gained insight on how VFF was used differently to achieve the same goals.
Future studies will include the incorporation of phantom tissues models to more accurately simulate surgical task performance on deformable materials. Furthermore, we are exploring whether haptic feedback or surrogate (eg, VFF) could be integrated into robotic simulators to facilitate novice training and the transition to the actual haptic-deficient operative environment presented by current robotic systems.
|
Conclusions
|
|---|
We describe an instrument-tracking VFF system that is fully integrated into the da Vinci robotic system. Our results suggest that haptic feedback specifically moderates excessive force application and consistency during the performance of delicate, repetitious robotic manipulation of fine sutures among surgeons who are new to the da Vinci system. Measurable benefits, however, were lacking among experienced users of the da Vinci system.
|
Acknowledgments
|
|---|
The authors acknowledge the encouragement and assistance of Dr Christopher J. Hasser of Intuitive Surgical Inc, and Phillip Randall Brown, DVM, MSc, Sue Eller, RVT, and Chrissy Sheckells of the Johns Hopkins Minimally Invasive Surgical Training Center for their assistance and access to the da Vinci Surgical System. We also thank Joyce Koons and the Koons family for their generous philanthropic support of this research.
|
Footnotes
|
|---|
Supported by grants from the National Science Foundation (EEC-9731478), Whitaker Foundation (RG-01-911), Koons Family Fund, and National Institutes of Health (R01-EB002004). Allison Okamura reports ownership of equity in Immersion Corp.
|
References
|
|---|
- Guthart GS, Salisbury JK. The intuitive TM telesurgery system: overview and application. From proceedings of the IEEE International Conference on Robotics and Automation 2000;1:618-621.
- Chitwood WR. Current status of endoscopic and robotic mitral valve surgery. Ann Thorac Surg 2005;79:S2248-S2253.[Abstract/Free Full Text]
- Fleck T, Tschernko E, Hutschala D, et al. Total endoscopic CABG using robotics on beating heart. Heart Surg Forum 2005;8:E266-E268.[Medline]
- Yoshino I, Hashizume M, Shimada M, et al. Thoracoscopic thymectomy with the da Vinci computer-enhanced surgical system. J Thorac Cardiovasc Surg 2001;122:783-785.[Free Full Text]
- Kitagawa M, Dokko D, Okamura AM, Yuh DD. Effect of sensory substitution on suture manipulation forces for robotic surgical systems. J Thorac Cardiovasc Surg 2005;129:151-158.[Abstract/Free Full Text]
- Shennib H, Bastawisy A, Mack M, et al. Computer-assisted telemanipulation: an enabling technology for endoscopic coronary artery bypass. Ann Thorac Surg 1998;66:1060-1063.[Abstract/Free Full Text]
- Tholey G, Desai GP, Castellanos AE. Force feedback plays a significant role in minimally invasive surgery—results and analysis. Ann Surg 2005;241:102-109.[Medline]
- Okamura AM. Methods for haptic feedback in teleoperated robot-assisted surgery. Ind Rob 2004;31:499-508.[Medline]
- Wagner CR, Stylopoulos N, Jackson PG, Howe RD. The benefit of force feedback in surgery: examination of blunt dissection. Presence: teleoperators and virtual environments 2007;16:252-262.
- Hashizume M, Shimada M, Tomikawa Y, et al. Early experiences of endoscopic procedures in general surgery assisted by a computer-enhanced surgical system. Surg Endosc 2002;16:1187-1191.[Medline]
- Sung GT, Gill IS. Robotic laparoscopic surgery: a comparison of the da Vinci and Zeus systems. Urology 2001;58:893-898.[Medline]
- Mohr FW, Falk V, Diegeler A, et al. Computer-enhanced "robotic" cardiac surgery: Experience in 149 patients. J Thorac Cardiovasc Surg 2001;121:842-853.[Abstract/Free Full Text]
- Nifong LW, Chitwood WR, Pappas PS, et al. Robotic mitral valve surgery: a United States multicenter trial. J Thorac Cardiovasc Surg 2005;129:1395-1404.[Abstract/Free Full Text]
- Autschbach R, Onnasch JF, Falk V, et al. The Leipzig experience with robotic valve surgery. J Card Surg 2000;15:82-87.[Medline]
- Prasad S, Fischer GS, Zand J, et al. A modular 2 of force sensing instrument for laparoscopic surgery. 2003Proceedings of the 6th International Conference on Medical Image Computing and Computer Assisted Intervention, Montreal, Canada, November 15-18.
- Sabatini AM, Bergamasco M, Dario P. Force feedback based telemicromanipulation for robot surgery on soft tissues. 1989Proceedings of the IEEE Engineering in Medicine and Biology Society 11th Annual International Conference, Seattle, Washington, November 9-12.
- Tavakoli M, Patel RV, Moallem M. Haptic feedback and sensory substitution during telemanipulated suturing. 2005. pp. 543-544From proceedings of the Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.
This article has been cited by other articles:
|
|
|
|
 
P. Modi, E. Rodriguez, and W. R. Chitwood Jr.
Robot-assisted cardiac surgery
Interact CardioVasc Thorac Surg,
September 1, 2009;
9(3):
500 - 505.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Introduction
