Tele-robotic pinching for intra-operative palpation
Transcripción
Tele-robotic pinching for intra-operative palpation
06/04/2011 TTelerobotic pinching for l b ti i hi f intraoperative palpation Departamento de Inge eniería de Sistemas y Automáticaa (1‐10‐2010 – 31‐3‐2011) Jesús Manuel Gómez de Gabriel ISRG School of Systems Engineering University of Reading Contents • Short CV & Previous works • Introduction • Haptic feedback in surgical robotics Departamento de Inge eniería de Sistemas y Automáticaa – Touch feedback – Force feedback • • • • • • • • Goal Master System Interaction with virtual environments Virtual prototyping Slave System Control System Experiments Conclusions 1 06/04/2011 Departamento de Inge eniería de Sistemas y Automáticaa Short CV • Jesús M. Gómez de Gabriel, is an Associate Professor at the Department of Systems Engineering and Automation, University of Málaga (Spain) University of Málaga (Spain) • Engineering and Ph.D. degrees in Computer Science from the UMA in 1990 and 1999, respectively. • Has led a project on telerobotic surgery and participated in different medical robotics projects. • Current research interests include mechatronics education, medical robotics, and indoor mobile robotics. • Currently in a Sabbatical Year in the SSE y (UoR) during course 2010‐11. Engineering School at UMA Previous works on medical robotics • ERM cameraman robot. Departamento de Inge eniería de Sistemas y Automáticaa – Now transfered to SENER – Clinical trials – Adaptive control and surgeon/patient interaction/compliance • Telerobotic system y for minimally invasive surgery. – DPI 2003‐08263 project – Interfaces for delayed teleoperation 2 06/04/2011 Introduction Departamento de Inge eniería de Sistemas y Automáticaa • Minimally Invasive Surgery techniques (MIS) – Small incisions in the patient using specialized instruments, – Reduction on clinical complications and hospitalization time – Abdominal MIS Techniques known as laparoscopic. – Constraints and motion inversion • Robotics & MIS: – Thanks to the special constraints (instrument and motion restriction), today, commercial robotic systems for surgery can be found. – Can enhance the performance of these tools by means of scaling, filtering and other aids. Introduction Departamento de Inge eniería de Sistemas y Automáticaa • Lack of tactile feedback in MIS. – The surgeon loses tactile feedback, but can feel forces at the instrument handle – Standard MIS training provides the surgeons the skill to do manual tasks with video feedback only. – Force or tactile feedback can be necessary/important for many manipulation tasks, – Growing interest of the surgeons in recovering the sense of touch, for grasping and touching the patient tissues. • Haptic feedback in MIS robotics – With robots we can loose force feedback too – No tactile feedback is commonly used 3 06/04/2011 Haptic feedback in surgical robotics Departamento de Inge eniería de Sistemas y Automáticaa • Haptics generally describes touch feedback (Okamura, 2009), which may include kinesthetic (force) and cutaneous (tactile) feedback. feedback • Graphical display can be used as a sensorial substitution system • Special master system: Impedance type device is the most popular because of the lower costs (there are no force sensors) and high responsiveness to human inputs. Departamento de Inge eniería de Sistemas y Automáticaa Tactile feedback in surgical robotics • Useful for exploratory tasks such as palpation, in which distributed pressure or deformation information can be distributed pressure or deformation information can be used to identify hard lumps in surrounding soft tissue. • Also experiments show how tactile feedback induces reduced grasping force in robot‐assisted surgery (King, 2009) • It remains difficult to design both tactile sensors and displays that are compatible with the surgical displays that are compatible with the surgical environment. 4 06/04/2011 Departamento de Inge eniería de Sistemas y Automáticaa Force feedback in surgical robotics • Easy to implement and process. • Haptics and special manipulators for needle insertion p p p control (Zarrad, 2007) considering the changes in environment stiffness. • Force Feedback in dentistry Learning (San Diego, 2008) • Teleoperated palpation for calcified vessel detection (Gwilliam, 2009). • Force controlled telerobotic grasper (Rosen, 1999) can identify automatically different kinds of animal tissues identify automatically different kinds of animal tissues. • Special wheeled haptic probe for the identification of soft tissues abnormalities (Liu, 2009) Soft tissue mechanics Departamento de Inge eniería de Sistemas y Automáticaa • Experimental measures (Rosen, 1999) • Dual Maxwell Model with nonlinear functions (Liu, 2009) 5 06/04/2011 Surgeon Grasping Mechanics • Thirty‐one surgeons of varying skill were recorded performing three different surgical tasks (Brown, 2004) – Force Departamento de Inge eniería de Sistemas y Automáticaa • The mean force applied to the tool handles during tissue grasps was 8.52 N ± 2.77 N; • maximum force was 68.17 N. – Frequency • Ninety‐five percent of the handle angle frequency content was below 1.98 Hz ± 0.98 Hz. • Mean percentage of Fg that lies below 5 Hz, during tissue grasps: 99.35% ± 99 35% 1.35% 1 35% – Time • Average grasp time was 2.29 s ± 1.65 s, and 95% of all grasps were held for 8.86 s ± 7.06 s or less. • The average maximum grasp time was 13.37 s ± 11.42 s. Goal of this work Departamento de Inge eniería de Sistemas y Automáticaa • Goal: Design (Build) the robotic instruments for the task of intraoperative lump detection in laparoscopic surgery. • Moveable and overlapped organs (Two or more fingers). • No grasping needed. 6 06/04/2011 Departamento de Inge eniería de Sistemas y Automáticaa First Experiments • Some preliminary experiments have been done in order to check if the proposed task can be performed proposed task can be performed without tactile feedback. • They showed the feasibility of a force feedback control system by means of a set of haptic devices without touch sensors. • Rigid fixed inclusions (bones) convert grasping force into displacement force. Master System • Master system already developed at the THRILL Lab: Departamento de Inge eniería de Sistemas y Automáticaa – 2 to 3 Fingers (3 active DOF each) – 2 or 3D immersive display – C++ environment with Haptic, Math and graphics libraries 7 06/04/2011 Interaction with virtual environments (I) Departamento de Inge eniería de Sistemas y Automáticaa • Simulators available used for surgery and medical applications (Halvorsen, 2005). • Laparoscopic simulators used for training L i i l df i i on the use of laparoscopic instruments rarely use force feedback. • Difficult environment modelling. – Mechanical properties of the tissues – Flexible tissues, dynamic (cut, suture,…), • Preoperative planning. Models with data from p g g( , , the real patient. Medical imaging (CT, MRI, etc.) • 3D/2D graphical simulation • Force feedback commonly used in orthopaedic surgery training. Interaction with virtual environments (II) Departamento de Inge eniería de Sistemas y Automáticaa • Same question: Is it possible to successfully achieve the proposed task without tactile achieve the proposed task without tactile feedback ? – Simulation for the validation of the idea. Not for training – Kinesthetic feedback only (force feedback) and visual • Experiments (4 Videos): – Moving/fixed and – visible/hidden inclusions 8 06/04/2011 Interaction with virtual environments (III) Hidden Fixed Departamento de Inge eniería de Sistemas y Automáticaa Mobile Visible Virtual prototyping • It is necessary to reduce the number of joints and sensors to a minimum. • The virtual prototyping of the different instruments allows testing The virtual prototyping of the different instruments allows testing the physical and sensorial constraints of the proposed instruments. Master Slave Departamento de Inge eniería de Sistemas y Automáticaa Video display Video camera Virtual instrument Position Multi finger h i d i haptic device Force Intrument Position Mechanical constraints Constraints Forces Feedback Forces + Meassured Forces Sensorial constraints Interaction Forces Fingers with position control Force sensors • Different virtual prototypes for the robotic instrument have been tested, and full information about the task execution (positions, and measured forces) has been recorded for analysis. 9 06/04/2011 Departamento de Inge eniería de Sistemas y Automáticaa Virtual prototypes (I) • Virtual prototypes are composed of master position and slave force sensing constraints (Virtual fixtures). • Positions are constrained before sending to the slave. P ii i db f di h l • Force readings are filtered before feeding back to the haptics. • The forces imposed by the virtual constraints are implemented using virtual high stiffness springs. • Instrument dynamics has not been considered. FY0 FX0 Y0 • General prototype: G l t t FY1 – Unconstrained. FX1 Y1 • 3 DOF each finger • 3 axis force sensing • Figure shows horizontal XY plane X0 X1 Virtual Prototypes (II) Departamento de Inge eniería de Sistemas y Automáticaa – Parallel horizontal grasper (3 Dofs) • • • • SSame Y coordinate. Y di t Horizontal plane (Z = 0). Grasping only force sensing Figure shows horizontal XY plane Y0 Master Position Slave Position K F0 F1 (Y0+Y1)/2 K Y1 10 06/04/2011 Virtual Prototypes (III) – Parallel horizontal symmetric grasper (2 Dofs) Departamento de Inge eniería de Sistemas y Automáticaa • • • • • SSame Y coordinate. Y di t Horizontal plane (Z = 0) Symmetry along X axis X axis only force sensing Figure shows horizontal XY plane Master Position Y0 F1 Slave Position (Y0+Y1)/2 F0 Y1 ‐X1 X0 X=0 ‐X0 X1 Slave system Version 1.0 Two slave fingers design 1.0 with external force sensors 3 DOFs per finger. L l Local proportional position control servos 1 KHz update rate. i l ii l 1 KH d 10 bits position resolution (≈ 0.3 degrees) Low cost but high unmeasured compliance Departamento de Inge eniería de Sistemas y Automáticaa • • • • • 11 06/04/2011 Slave System Version 1.1 Camera Camera boom Monitor Departamento de Inge eniería de Sistemas y Automáticaa Mirror Master Haptic Supporting frame Base Hanging Fingers Tissues Experimental Bilateral Teleoperation Setup Slave System Force sensing • • • • Departamento de Inge eniería de Sistemas y Automáticaa • Thanks to previous development from TRHILL Lab on its force sensor. 750g Micro‐load cells g On board signal conditioning and data acquisition 10 bits 1KHz Rearranged for uncoupled serial configuration. Last finger link body. FZ Fingertip FY FX 3 DAQ 2 1 Sensor base 12 06/04/2011 Slave System Version 1.2 Departamento de Inge eniería de Sistemas y Automáticaa • Same components as 1.0 • Shortened axis distances by changing the servo links. – Less compliance – Better cartesian space resolution. • Smaller Smaller workspace due to workspace due to shorter links and angle limits. Bilateral Control System • Classical implementation of a bilateral Force‐Position control system Xm Surgeon Fh ‐ Haptic Master manipulator (Impedance) Force Sensor Patient Xs ‐ Departamento de Inge eniería de Sistemas y Automáticaa Position control Kf Fs Slave device (Admitance) • Master Master workstation is based on a set of haptic devices with workstation is based on a set of haptic devices with impedance control • Slave setup as a impedance control (position control and force sensors). • A video feedback channel has also been added to the system. • Better than standard position‐position control scheme in non‐ contact motion. • Problem of transparency and stability. 13 06/04/2011 Bilateral control system (II) Departamento de Inge eniería de Sistemas y Automáticaa • Bilateral control system model • Force feedback gain depends on the overall dynamics. • In medical robotics teleoperation experiments, the changes in the environment stiffness can be abrupt and large in the environment stiffness can be abrupt and large in magnitude (e.g. touching a muscle, an organ, a bone, another instrument or a rib for cardiothoracic surgery) • Stiffness estimation can be difficult due to organ motion (breathing or displacements) Bilateral control system (II) • Possible Improvements to be implemented: Adaptive Position‐ Position control system (Poignet, 2009; Zarrad, 2007) Departamento de Inge eniería de Sistemas y Automáticaa – Environment Stiffness estimator (Needs force sensing at the slave) Environment Stiffness estimator (Needs force sensing at the slave) – Active observer and Kalman filter Dynamic gain and p p p haptic palpation ??? 14 06/04/2011 Departamento de Inge eniería de Sistemas y Automáticaa Virtual prototype experiments • Task goal/scoring: • To find a rigid object (crystal T fi d i id bj t ( t l ball 1.6 mm dia.) inside a three‐sections foam “organ”. • Guess the approximate size/shape of the object. • Reposition the organ for better handling better handling. Virtual prototype experiments (I) • Free motion Departamento de Inge eniería de Sistemas y Automáticaa – 6 Dofs – Full force feedback Full force feedback • Results: – Easy to find inclusion – Easy to guess the size – Fair organ manipulation http://www.youtube.com/watch?v=ETZayk2op3I 15 06/04/2011 Virtual prototype experiments (II) • Parallel grasper Departamento de Inge eniería de Sistemas y Automáticaa – 3 Dofs – Grasping force Grasping force feedback • Results: – Fair to find inclusion F i fi d i l i – No shape guessing – Limited organ manipulation http://www.youtube.com/watch?v=1Q0c1juUY3E Virtual prototype experiments (III) • Symmetrical parallel grasper Departamento de Inge eniería de Sistemas y Automáticaa – 2 2 Dofs Dofs – Grasping force feedback • Results: – Hard to find inclusion Hard to find inclusion – No guessing of size/shape – Bad organ manipulation http://www.youtube.com/watch?v=v‐wBtc9hCYA 16 06/04/2011 Data analysis • To be done: – Second force sensor reading and filtering Second force sensor reading and filtering – Tissue dentification assistance? Departamento de Inge eniería de Sistemas y Automáticaa Left Finger force Finger distance Open Open closed Slave system Version 2.0 Lower compliance 4x spatial resolution Higher torque Different kinematic Same control system and protocol • More expensive than 1.x • To be tested Departamento de Inge eniería de Sistemas y Automáticaa • • • • • 17 06/04/2011 Departamento de Inge eniería de Sistemas y Automáticaa Conclusions • This is an ongoing work. No final conclusions yet. • The foam phantom used is much more elastic than real animal tissues (sausage) Needs improvement tissues (sausage). Needs improvement. • Processing of the sensorial information for some way of dynamic gain and tissue identification has to be implemented. • Although it may not be necessary, we miss a touch sensor/display system. • This system allows the study of the performance of different instruments for different telemanipulation tasks without a physical implementation. • Also, the analysis of the obtained information during the trials Al h l i f h b i di f i d i h i l provides a better understanding of the use of the instruments, so further optimizations of the prototype can be made. References • • • Departamento de Inge eniería de Sistemas y Automáticaa • • • • • • • Bogado Torres, J.M., 2007, “Control bilateral de robots treleoperados por convergencia de estados”, PhD Thesis, Universidad Politécnica de Madrid. Brown, J.D. et al, “Quantifying Surgeon Grasping Mechanics in Laparoscopy Using the Blue DRAGON y , gy Medicine Meets Virtual Reality, Newport Beach, y, p , System”, Studies in Health Technology and Informatics ‐ CA, January 2004. J.C. Gwilliam, M. Mahvash, B. Vagvolgyi, A. Vacharat, D D. Yuh, and A. M. Okamura, “Effects of Haptic and Graphical Force Feedback on Teleoperated Palpation”, 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan, May 12‐17, 2009 Halvorsen, F.H. et al. 2005, “Simulators in Surgery”, Minimally Invasive Therapy 14:4‐5; pp. 214‐223. King, C.‐H.; Culjat, M.O.; Franco, M.L.; Lewis, C.E.; Dutson, E.P.; Grundfest, W.S.; Bisley, J.W.; , "Tactile Feedback Induces Reduced Grasping Force in Robot‐Assisted Surgery," Haptics, IEEE Transactions on , vol.2, no.2, pp.103‐110, April‐June 2009 Hongbin Liu Elhage, O. Dasgupta, P. Challacombe, B. Murphy, D. Seneviratne, L. Althoefer, K., “A haptic probe for soft tissue abnormality identification during minimally invasive surgery”, Reconfigurable Mechanisms and Robots, 2009. ReMAR 2009. ASME/IFToMM International Conference on, 22‐24 June 2009 Okamura, A. M., 2009, “Haptic Feedback in Robot‐Assited Minimally Invasive Surgery”, Curr. Opin. Urol, January 2009. Poignet, P, et al. 2009. “Some control related issues in mini‐robotics for endoluminal surgery”, 31st Annual Intl. Conf. Of the IEEE EMB, pp. 6850‐6855 Rosen, J. Hannaford, B. MacFarlane, M.P. Sinanan, M.N., “Force controlled and teleoperated endoscopic grasper for minimally invasive surgery‐experimental performance evaluation”, Biomedical Engineering, IEEE Transactions on, 46:10; 1999 Zarrad, W. et al 2007, “Stability and Transparency Analysis of a Haptic Feedback Controller for Medical Applications”, 46th IEEE Conference on Decision and Control New Orleans. 18 06/04/2011 Departamento de Inge eniería de Sistemas y Automáticaa Thanks and acknowledgements • Special Thanks to Prof. William Harwin, Víctor Becerra, Rui Loureiro and the University of Reading Becerra, Rui and the University of Reading • Thrill Lab: Alastair Barrow, Brian Tse and Balazs Janko • Ministerio de Educación y Ciencia of Spain • SSE’s Workshop staff • University of Málaga Contact Departamento de Inge eniería de Sistemas y Automáticaa Jesús M. Gomez de Gabriel Dt I Dto. Ingeniería de Sistemas y Automática i í d Si t A t áti Universidad de Málaga E‐mail: [email protected] Web: www.isa.uma.es/c10/degabriel / / g Blog: www.hombremecatronico.es YouTube: www.youtube.com/roboticario 19