Int J CARS (2007) 2 (Suppl 1):S236–S245 DOI 10.1007/s11548-007-0097-1

CARDIOVASCULAR SURGERY

Interventional guidance for cardiac resynchronization therapies: merging anatomic X-ray imaging with functional ultrasound imaging based on mutually-shared landmarks R. Manzkea, F. Tournouxb, B. v. d. Brinkc, O. Gerardd, R. v. d. Boomenc, G. Shechtera, L. Gutierreza, J. Singhb, M. Picardb, R. C. Chana a Philips Research North America, Briarcliff Manor, NY, USA b Department of Cardiology, Massachusetts General Hospital, Harvard Medical School c Philips Medical System, Best, Netherlands d Philips Medical Systems, Paris, France Abstract Detailed knowledge of cardiac anatomy and function is required for complex cardiac electrophysiology interventions. Cardiac resynchronization therapies (CRT), for example, requires information about coronary venous anatomy for left ventricular lead placement. In CRT, heart failure patients are equipped with dual-chamber pacemakers in order to improve cardiac output and heart failure symptoms. Cardiac function is mainly assessed with Ultrasound imaging. Fusion of complementary information from X-ray and Ultrasound is an essential step towards fully utilizing all available information for CRT guidance. We present an approach for fusion of anatomical information (coronary vein structure) from X-ray with functional information (left ventricular deformation and dynamics) from Ultrasound. We propose an image-based fusion approach based on mutually-shared landmarks which enable registration of both imaging spaces without the need for external tracking. Keywords Cardiac electrophysiology Æ Interventional guidance Æ Image guided therapies Æ Cardiac resynchronization therapies Æ Multi-modality imaging 1. Introduction Cardiac electrophysiology (EP) applications such as cardiac resynchronization therapies (CRT) require detailed knowledge of cardiac anatomy [1] (vein tree, ventricle) and functional parameters (ventricular synchronicity) [1–3]. In CRT procedures, a biventricular pacemaker is implanted, which, in contrast to old single chamber pacemakers, paces both the right and the left heart. The pacemaker paces the right atrium (lead screwed in right atrial tissue), the right ventricle (lead screwed into septal wall close to right ventricular apex) and the left ventricle (lead implanted in the coronary venous tree). With optimal placement of the left ventricular lead and optimal programming of the pacing sequence, cardiac output can be optimized to mitigate heart failure symptoms [4]. Currently, functional and anatomical measurements from the heart are generated on separate imaging systems. X-ray is the mainstay imaging modality for interventional EP guidance whereas 3D cardiac ultrasound is the modality of choice for characterization of ventricular function. Fusion of complementary information from these modalities is a necessary first step towards full utilization of all available information during interventional guidance. Multi-modality fusion of X-ray and Ultrasound data has been difficult due to lack of shared soft-tissue landmarks in the two datasets (X-ray visualizes venous anatomy based on contrast agent delivery whereas ultrasound images myocardial tissue characteristics without venous definition) [2, 3]. Several attempts have been made to use external registration

123

based on optical or electromagnetic tracking [2, 3, 5]. These approaches are, however, difficult to realize in clinical settings due to space and time constraints. We present an image-fusion approach based on mutually-shared image landmarks which enable registration of both imaging spaces without the need for external tracking. 2. Methods Rotational X-ray vein imaging was performed using a Philips Allura X-Per FD10 flat detector system during left ventricular lead implantation in a clinical CRT case. One hundred and twenty projection images were obtained over 4 s using a 110 degree rotation. The coronary venous tree was opacified in a retrograde fashion using manual iodinated contrast agent injection (Schering Ultravist, 20 ml) through the catheter at the time of image acquisition. Subsequently, coronary vein models were generated using a 3D centerline approach. Pre-procedural and post-procedural 3D ultrasound images of the heart were acquired (Philips iE33 cardiac 3D Ultrasound, X3-1 transducer). Post-procedural imaging was performed with the bi-ventricular pacemaker switched on and off. A 3D parametric mesh describing both left ventricular shape and synchronicity was calculated using QLab (Philips Medical Systems, Andover, MA, USA). 3D point positions were calculated from rotational X-ray space at the right ventricular lead tip, the right atrial lead tip and the coronary sinus using a 3D line intersection approach (see Fig. 1). The same landmarks were approximately determined in ultrasound space by manual selection within the volume (see Fig. 2). Registration was subsequently performed using Procrustes methods followed by manual refinement by the physician for additional fine-tuning, if necessary. Custom multi-modality visualization software was used for combined image representation of coronary vein tree and left ventricular shape colour-coded with functional dyssynchrony information (see Fig. 3). 3. Results The coronary venous tree from X-ray and the functional/anatomical mesh from 3D ultrasound can be reasonably aligned when using manual registration of extracted mutually-extracted reference points. When Procrustes method was applied for landmark point registration, an RMS registration error of 3–5 mm was achieved. Both the left ventricular shape and the coronary venous tree were aligned reasonably. However, the physician still desired manual interaction to fine-tune the registration (see Fig. 3).

Fig. 1 3D point determination of right atrial lead tip from rotational X-ray

Int J CARS (2007) 2 (Suppl 1):S236–S245

Fig. 2 Right atrial lead tip position derived in 3D Ultrasound space

Fig. 3 This figure shows the registered vein tree with the functional Ultrasound mesh after landmark registration (biventricular pacing on and off) Visualization of bi-ventricular pacing turned off and on nicely describes the functional improvement of the ventricle (see Fig. 3) due to optimal lead placement. This information is important for assessing CRT procedure outcome. 4. Conclusions Combined X-ray and 3D Ultrasound imaging can be used as a tool for interventional guidance in CRT procedures. Gross registration of both image modalities can be performed using mutually-shared landmarks and Procrustes method. Intrinsic anatomical landmarks such as the coronary sinus or points on the right ventricular and right atrial leads can be determined from both modalities and be used for gross registration. Intra-procedural 3D image acquisition and registration, in particular intra-procedural ultrasound, remains a significant challenge due to the sterile operating field (3D ultrasound) and procedure time constraints (3D coronary vein modelling). Integration of pre-procedural US imaging with intraprocedural X-ray imaging can use intrinsic anatomic landmarks (coronary sinus, apex of right ventricle, septal wall). For patients who are being upgraded from single to bi-ventricular pacemakers can be imaged with ultrasound pre-procedurally in order to obtain right-heart lead positions for intra-procedural integration as described above. Given the pre/intra-procedural availability of mutually-present landmarks, multi-modality interventional guidance of CRT to optimize left ventricular lead placement and cardiac output is a clinically-realistic goal. References 1. Singh J, Houser S, Heist EK, Ruskin J (2005) The coronary venous anatomy: a segmental approach to aid cardiac resynchronization therapy. J Am College Cardiol 46(1):68–74 2. Mansour M, Rasche V, Picard MH, Ruskin J (2006) Integration of three-dimensional coronary venous angiography with

S237 three-dimensional echocardiography for biventricular device implantation. Heart Rhythm 3(11):1391–1392. Case report 3. Mansour M, Reddy VY, Singh J, Mela T, Rasche V, Ruskin J (2005) Three-dimensional reconstruction of the coronary sinus using rotational angiography. J Cardiovascular Electrophysiol 16(6):675–676 4. Singh J, Heist KE, Ruskin J, Harthorne JW (2006) ‘‘Dialingin’’ cardiac resynchronization therapy: overcoming constraints of the coronary venous anatomy. J Cardiovascular Electrophysiol 17(1):51–58 5. Gutierrez L, Shechter G, Stanton D, Dalal S, Elgort D, Manzke D, Chan R, Zagorchev L (2007) Multimodality image guidance system integrating X-ray fluoroscopy and ultrasound image streams with electromagnetic tracking. SPIE Medical Imaging Conference Records 2007 (in press)

4D Ultrasound and 3D MRI registration of beating heart N. Herlambanga, H. Liaob, K. Matsumiyaa, K. Masamunea H. Tsukiharac, S. Takamotoc, T. Dohia a Graduate School of Information Science and Technology, the University of Tokyo, Japan b Graduate School of Engineering, the University of Tokyo, Japan c Graduate School of Medicine, the University of Tokyo, Japan Abstract To realize intra-cardiac surgery without cardio-pulmonary bypass, a medical imaging technique with both high image quality and data acquisition rate that is fast enough to follow heart beat movements is required. In this research, we proposed a method that utilized the image quality of MRI and the speed of ultrasound. We developed a 4D image reconstruction method using image registration of 3D MRI and 4D ultrasound images. The registration method consists of rigid registration between 3D MRI and 3D ultrasound with the same heart beat phase, and nonrigid registration between 3D ultrasound images from different heart beat phases. Non-rigid registration was performed with B-spline based registration using variable spring model. In phantom experiment using balloon phantom, registration accuracy was less than 2 mm for total heart volume variation range of 10%. We applied our registration method on 3D MRI and 4D ultrasound images of a volunteer’s beating heart data and confirmed through visual observation that heart beat pattern was well reproduced. Keywords 4D ultrasound Æ 3D MRI Æ Rigid registration Æ Non-rigid registration Æ Cardiac surgery 1. Introduction In minimally invasive cardiac surgery (MICS), surgery procedure without stopping the heart beat is important to prevent complications such as blood cell destruction or coagulation. Beating heart surgery has been applied in surgeries on heart surface such as coronary artery bypass surgery, but is not yet possible to be applied to intra-cardiac surgery with current technology. In order to apply it to intra-cardiac surgery such as mitral valve surgery, visualization of intra-cardiac organs such as mitral valve that is fast enough to follow heart beat movement is required. Especially, visualization on 3D display system with the sensation of direct vision will provide surgeons with instinctive perception of organ positional information. For that purpose, we have developed a real-time Integral Videography (IV) stereoscopic display system [1]. IV is a stereoscopic imaging system that is spatially accurate, can be observed with naked eye, and allows multiple spectators at the same time. Using 3D ultrasound (US) as data acquisition method, the speed required to follow heart movement is fulfilled, but image quality better than the noisy US image is required, especially because intra-cardiac organs are completely not visible

123

S238

Int J CARS (2007) 2 (Suppl 1):S236–S245

Fig. 1 3D MRI and 4D ultrasound registration scenario

Fig. 2 a Phantom experiment using balloon phantom and b registration accuracy

to surgeons. On the other hand, while medical imaging method with superior image quality such as MRI exists, it lacks speed. In this study, we proposed a method to utilize the advantages of both imaging modalities, the image quality of MRI and the speed of ultrasound, for future MICS navigation. The objective of this paper is to develop a 4D image reconstruction method using image registration of 3D MRI and 4D ultrasound images of beating heart for future MICS, including registration method development, registration accuracy assessment, and clinical feasibility study on human heart. 2. Methods We proposed a 4D ultrasound and 3D MRI registration scenario for a high quality 4D image creation (Fig. 1). Our method consists of rigid registration between 3D MRI and 3D ultrasound with the same heart beat phase, and non-rigid registration between 3D ultrasound images from different heart beat phases. Rigid registration aligned the position and orientation of 3D MRI and corresponding 3D ultrasound images. And from non-rigid registration, deformation field of each 3D ultrasound image relative to the one used in rigid registration is obtained. Applying the deformation fields of the 4D ultrasound image to 3D MRI image, we can produce a set of high-quality 4D image. Rigid registration is performed using mutual information as similarity measure and Powell optimization method. In non-rigid registration, we used B-spline based registration [2]. In B-spline based registration, images and deformation field are represented in smooth B-spline function that leads to fast calculation. B-spline based non-rigid registration is widely used because of its relatively low calculation cost and its ability to cover a wide range of deformation. However, as B-spline deformation model is an image based deformation model, the resulted registration is often

123

Fig. 3 Registration results. a Captured MRI image, b, c created images of different heart beat phases (marked area is left ventricle)

inaccurate in representing real-world deformation. We proposed a variable spring model to add organ elastic characteristic as physical constraint. In contrast with the original B-spline based registration, we added spring energy to sum of squared difference (SSD) for optimization criterion. Optimization is performed using gradient descent method as the use of B-spline function allows fast calculation of derivatives of both SSD and spring energy. Compared to the original algorithm, our method is superior in representing organ elastic deformation and preventing deformation field folding. 3. Results To evaluate registration accuracy, we performed a phantom experiment (Fig. 2a). We used a balloon with controllable volume as phantom. We used a syringe to control balloon volume. First, we acquired MRI and ultrasound images of balloon with volume from 100 ml increased by 4 ml each up to 140 ml. We used gadolinium and graphite powder as contrast agent during each MRI and ultrasound images acquisition. For the purpose of accuracy measurement, we attached 6 donut markers on the surface of the balloon during MRI images acquisition. Then, MRI image of balloon with volume of 100 ml is registered to ultrasound images. As the registration result, MRI images of balloon with volume from 104 to 140 ml are created. Then we assessed registration accuracy by comparing the resulted images to the acquired MRI images. Each created image is compared to corresponding MRI image. Distance between markers position in both images is measured as registration error. The registration accuracy evaluation result can be seen in Fig. 2b.

Int J CARS (2007) 2 (Suppl 1):S236–S245

S239

In clinical feasibility study, we applied our method to register 3D MRI and 4D ultrasound images of human heart. 4D ultrasound images is acquired using trans-vaginal ultrasound probe with frame rate of 5 vol/s. MRI image was acquired with 0.4 T MRI with heart beat gating. The resulted high quality 4D image of beating heart and its surface model is presented in Fig. 3. 4. Discussions The main cause of registration error is thought to be the resolution of ultrasound images that reached 5 mm at the marker position farthest from the convex ultrasound probe. As it has been reported before that total heart volume change less than 10% during a heart beat [3], registration error in that range was suppressed to less than 2 mm. However, as heart movement possesses a more complicated deformation which also includes local deformation, registration assessment using a more real model is required. Because of the narrow range of ultrasound data, movement of heart parts beyond ultrasound data acquisition range could not be reproduced. In our experiment, only left ventricle movement was reproduced. It is possible to combine deformation fields acquired from multiple ultrasound data acquisitions and apply them to MRI image so that a wider range of movement can be reproduced. 5. Conclusion We have developed a 4D ultrasound and 3D MRI registration method that is able to combine the data acquisition speed of ultrasound and the image quality of MRI for MICS navigation. In phantom experiment, registration error was less than 2 mm for volume change less than 10%. Clinical feasibility study showed that our method was able to be applied to human heart. Our future plan is to combine our registration method with Integral Videograpy (IV) stereoscopic imaging system for future MICS. Acknowledgments This study was supported in part by the Grant-in-Aid (17680037) of the Ministry of Education, Culture, Sport, Science and Technology in Japan (to H. L.), and the Grant-in-Aid (17100008) of the Japan Society for the Promotion of Science (to T. D.). References 1. Herlambang N, Liao H, Matsumiya K, Dohi T (2005) Design and evaluation of realtime integral videography using threedimensional ultrasound, Medical Image Computing and Computer-Assisted Intervention—MICCAI 2005, Palm Springs 2. Rueckert D, Sonoda LI, Hayes C, Hill DLG, Leach MO, Hawkes DJ (1999) Nonrigid registration using free-form deformations: application to breast mr images. IEEE Trans Med Imag 18(8):712–721 3. Carlsson M, Cain P, Holmqvist C, Stahlberg F, Lundback S, Arheden H (2004) Total heart volume variation throughout the cardiac cycle in humans. Am J Physiol-Heart Circulat Physiol 287(1):H243–H250

except for X-ray and ultrasound images from a TEE (transesophageal echocardiography) probe. However, exact placement of the stent-graft is crucial for the success of the procedure, thus there is a need for methods and tools aiding in the implantation of the stent-graft. Our system aims at supporting the surgeons during the intervention by visualizing registered pre-operative CTA images and intra-operative X-ray images. Thereby a roadmap for the catheter navigation can be displayed and the physician has access to all spatial 3D information necessary for the exact graft placement. By this method, we hope to enhance the accuracy of the surgeon’s actions and reduce the amount of contrast agent. Keywords Minimally invasive surgery Æ Aortic stent-graft implantation Æ Computer aided surgery 1. Introduction The implantation of an endovascular prosthesis, also referred to as stent-graft, inside the aorta is a minimally-invasive procedure for the treatment of aortic aneurysms and aortic dissections. The stent-graft remodels the structure of the aorta and excludes the aneurysm or dissection from the blood flow, removing the pressure from the inner aortic wall, to prevent further dilation or rupture of the aorta. In the current clinical workflow there is no technical guidance during the implantation of the stent-graft, except for X-Ray images provided by a mobile C-Arm and ultrasound images. However, exact placement of the stent-graft is crucial for the success of the procedure and avoidance of occluding vessels, branching from the aorta. In [1] the authors explored the possible prospects of a system that aids the implantation of stent-grafts by pre-operatively segmenting the aorta in the CTA (Computer Tomography Angiography) data set and registering the pre-operative CTA data of the patient with the intra-operative X-ray images. Thereby, the position of the stentgraft can be visualized in the 3D CTA data set, likewise the contour of the segmented aorta can be displayed in the 2D X-ray images without having to administer contrast agent to the patient. The proposed system aims on improving the accuracy of the placement of the stent-graft and helps in reducing the amount of radiation and contrast agent. In contrast to the work in [1] we focus on developing and implementing methods for the real interventional environment and not only a proof-of-concept-prototype. Therefore we have evaluated the practical usability of different radiopaque markers and developed an improved 2D/3D registration algorithm. 2. Methods 2.1 Pre-operative segmentation

Towards an integrated planning and navigation system for aortic stent-graft placement O. Kuttera, S. Kettnerb, E. Brauna, N. Navabb, R. Langea, R. Bauernschmitta a Department Cardiothoracic Surgery, Deutsches Herzzentrum Mu¨nchen, Germany b Chair for Computer Aided Medical Procedures, Technische Universita¨t Mu¨nchen, Germany Abstract The implantation of an endovascular stent-graft, inside the aorta is a minimally-invasive procedure for the treatment of aortic aneurysms and aortic dissections. In the current clinical workflow there is no technical guidance during the intervention

For the intra-operative visualization of the stent in the CTA data set and the display of the contour lines of the segmented aorta in the intra-operative images a 2D/3D registration of the CTA image and the X-ray image is required. In [1] the authors propose a distance-weighted graph-matching approach to find the marker correspondences. However, this approach only works well if the intrinsic parameters of the modelled camera are known. Otherwise this method can not be applied. For 2D/3D registration of X-ray and CT images several approaches exist in the literature. However the specific registration problem in this paper cannot be solved directly by most of these methods. Image-based registration [2–5] methods require the

Pre-operatively we apply the one-click segmentation method presented in [1] to segment CTA data sets of aortic aneurysms. For CTA data sets of aortic dissections we use an interactive graph-cuts approach, and employ the physicians’ expert knowledge for the detection of the false and true lumen inside the aorta. The result of the pre-operative segmentation is then used to generate a path inside the aorta by skeletonization of the segmented aorta, which is then used for the intra-operative visualization of the stent in the CTA data set. 2.2 Intra-operative registration and visualization

123

S240 intrinsic parameters of the modelled C-Arm camera to be known, and distortion corrected X-ray images. Camera calibration is tedious and time-consuming, additionally the X-ray image quality is degraded by the presence of a distortion correction pattern. Another problem in the realization of an image-based 2D/3D registration is the presence of surgical instruments, e.g. the catheter, guide-wires and the stent within the X-ray images and their absence in the pre-operative CTA data, when computing the similarity metric of the DRR and X-ray image. A landmark-based approach [6, 9] also turned out to be inapplicable. The interventional X-ray images show only few and dimly visible bony structures. Thus, anatomical landmarks on bony structure cannot be used. Feature-based methods [6–8] are mostly applied for registering X-ray images, but are also not applicable. The contrasted aorta is in general well suited for a vascular feature based 2D/3D registration, however for this specific problem we are also interested in registering images without the presence of contrast agent, as one of the main goals is to reduce the amount of contrast agent during the intervention. Therefore we also opted for a point based 2D/3D registration using radiopaque markers affixed to the patient’s skin as presented in [1]. However, the spherical markers used in [1] turned out to be impractical to use with real patients. Therefore, we chose to use flat markers (Fig. 1a), which proved to stay affixed to their position. Additional problems for the 2D/3D registration are the patient’s breathing motion, and the distortion of the X-ray image, which in general need to be compensated. The first problem can be solved satisfactorily, as the patient is connected to a breathing machine during the complete intervention. The patient’s breathing can be controlled and even be stopped for a short time to obtain a respiratory motion free X-ray image. The second problem can be solved by using special undistortion patterns. The main challenge of the 2D/3D registration problem presented in this paper is that the intrinsic parameters of the CArm are unknown and therefore the whole projection matrix P has to be determined from a set of 2D/3D markers with unknown correspondences. By the DLT (direct linear transformation) algorithm one can determine P from a set of at least six corresponding points. A brute force computation of P by exhaustively testing all possible marker correspondences is computationally too costly. In our approach we cluster the markers into groups and determine P by an exhaustive search by minimizing the registration error of the DLT for the reduced number of possibilities in the clusters. The radiopaque markers are arranged in two sets, one set of markers on the chest of the patient, the other on the back. Additionally two arrow shaped markers in the center of the markers on the patient’s chest are used. The 3D position of the radiopaque markers is determined pre-operatively by an interactive region growing algorithm from the CTA data set. The centroids of all segmented markers are computed by a first moment analysis. Intra-operatively a shape based matching algorithm, which is able to extract the cross-markers with high speed and

Fig. 1 Marker positions on the patient’s chest (a) and their visibility in the CTA (b) and X-ray image (c)

123

Int J CARS (2007) 2 (Suppl 1):S236–S245 accuracy, even for overlaid markers is applied. Using Ransac [10] a plane is fitted to all chest markers, which is divided into four clusters by the lines through the arrow markers. The same clustering strategy is applied to the 2D markers detected in the X-ray image, which gives us the set of 2D clusters. In the final step randomly four points out of the four 2D chest marker clusters and two points from set of 2D back markers. P is computed for all remaining possibilities of correspondences within the clusters. For each matrix P the mean squared error of the projected 3D markers and the closest 2D markers for a given distance threshold is determined. The projection matrix with the smallest mean squared error is then stored as the final projection matrix. 3. Results The radiopaque marker evaluation showed that the flat markers stay affixed to their positions and thus are more appropriate for the intervention than spherical markers, as used in [1]. For the experiments two types of markers were used; crosshair markers and arrow markers (Beekley Corporation, Bristol, CT, USA). The markers were placed on the patient’s chest (Fig. 1a) and back. The currently used crosshair markers are well visible in the CTA image (Fig. 1B). However they are only dimly visible in the intra-operative images (Fig. 7c). The presented 2D/3D—registration algorithm was tested twice on datasets acquired with a thorax phantom. Due to the marker visibility problem, the 2D/3D registration algorithm could not be evaluated with real patient data. For the phantom experiments, a generic tube was firmly fixed in the interior of a thorax phantom to represent the thoracic aorta and the aortic arc. Several markers were adhered to the surface and to the back of the phantom. The quality of the determined projection matrix P was assessed by the backprojection error. The back projection error was determined for P estimated from the original (distorted) X-ray image as well as P based on an image corrected for radial and sigmoidal distortions. The distortion correction can be omitted, as the computed projection geometry based on distorted images leads to similar mean back projection errors as the one for undistorted images. Moreover, our medical partners’ confirmed that the error of about one millimetre is acceptable for aortic stenting. Therefore, a minimum number of ten markers can be used for the 2D/3D registration. The maximum accuracy of less than 0.5 mm can be reached by using 16 or more markers. 4. Conclusion In this paper we present yet another method for marker based 2D/ 3D registration of intra-operative X-ray images and pre-operative CTA images. By using two different kinds of markers, the number of possible combinations of markers correspondences is reduced and the projection matrix efficiently determined from two sets of noncorresponding points. The 2D/3D registration was tested with synthetic ground-truth data sets and real CT and X-ray images of a thorax phantom. The usability of two different radiopaque markers for 2D/3D registration was evaluated, showing that flat markers are better suited than small spherical markers. In future work, we aim at solving the marker visibility problem in the intra-operative image, determining the best marker configuration by evaluating phantom and patient data sets. References 1. Feuerstein M, Filippatos K, Kutter O, Schirmbeck EU, Bauernschmitt R, Navab N (2006) A novel segmentation and navigation tool for endovascular stenting of aortic aneurysms. Intl J CARS 1(1):280–282 2. van de Kraats EB, Penney GP, Tomazˇevicˇ D, van Walsum T, Niessen WJ (2005)Standardized evaluation methodology for 2D-3D registration. IEEE TMI 24(9):1177–1189 3. Russakoff DB, Rohlfing T, Maurer C (2003) Fast intensitybased 2D–3D image-registration of clinical data using light fields. In: Proc IEEE ICCV2:416

Int J CARS (2007) 2 (Suppl 1):S236–S245 4. Rohlfing T, Russakoff DB, Denzler J, Maurer CR Jr (2004) Progressive attenuation fields:fast 2D–3D image registration without precomputation. In: Proc MICCAI, pp 631–638 5. Birkfellner W, Seemann R, Figl M, Hummel J, Ede C, Homolka P, Yang X, Niederer P, Bergmann H (2005) Fast DRR generation for 2D/3D registration. In: Proc MICCAI, pp 960–967 6. Hajnal JV, Hill DLG, Hawkes DJ (eds) (2001) Medical image registration; the Biomedical Engineering Series. CRC Press, Boca Raton 7. Chan H, Chung A, Yu S, Wells W (2004) 2d–3d vascular registration between digital subtraction X-ray (dsa) and magnetic resonance X-ray (mra) images. In: Proc IEEE ISBI, pp 708–711 8. Florin C, Williams J, Khamene A, Paragios N (2005) Registration of 3d X-ray and x-ray images using sequential monte carlo sampling. In: Proc Intl Workshop CVBIA, pp 427–436 9. Likar B, Pernus F (1999) Automatic extraction of corresponding points for the registration of medical images. Med Phys 26(8):1678–1686 10. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24:381–395

Development of surgical robot system with heartbeat canceller for endoscopic off-pump coronary artery bypass K. Toyodaa, T. Umedaa, M. Ouraa, Y.Iwamoria, K. Kawamuraa, Y. Kobayashia, H. Okayasua, J. Okamotob, M. Fujiec a Graduate school of Science and Engineering, Waseda University, Tokyo, Japan b Consolidated Research Institute for Advanced Science and Medical Care, Waseda University, Tokyo, Japan c Faculty of Science and Engineering, Waseda University, Tokyo, Japan Abstract The purpose of this study is to develop a surgical robot system with heartbeat canceller for endoscopic off-pump coronary artery bypass. The system is composed of the following two items. (i) heartbeat canceller (ii) master-slave surgical robot. In this paper, regarding (ii), we report the features of surgical robot and some experiments using it. The experiments are (a) insert a surgical needle into tissue (thymus) as in vivo experiment using a hog, (b) ligation of a rayon thread. As a result, we confirmed that the surgical robot accomplished two procedures. Keywords Cardiac surgery Æ Heartbeat canceller Æ Robotic surgery Æ Master-slave surgical robot 1. Introduction In recent years, minimally invasive surgery, aiming at reduction of the damage to patients during surgery and early recovery from the surgical trauma, has been getting popular. Endoscopic surgery is a typical surgery of the minimally invasive surgery. Since the endoscopic surgery is performed through small holes in the abdominal wall, damages to the patient is greatly reduced. Among coronary artery bypass surgery, the off-pump coronary artery bypass (OPCAB) is especially expected as a minimally invasive surgery. Therefore, many researchers and medical doctors have studied to perform OPCAB surgery using surgical robot such as da VinciÒ to reduce its invasiveness [1, 2]. However, in the OPCAB surgery, beating motion of coronary arteries caused by the heartbeat makes it very difficult to anastomose coronary arteries and ligate suture threads. Therefore, the purpose of this study is the development of surgical robot system with heartbeat canceller for supporting surgeons to perform the interarterial coronary anastomosis and ligation of suture thread.

S241 Main targets of this system are (1) Cancel heartbeat motion and (2) Perform the interarterial coronary anastomosis and ligation of suture threads with the surgical robot. The developmental system is composed of the following two items. (i) Mechanism of the heartbeat canceller. (ii) Master-slave surgical robot for interarterial coronary anastomosis and ligation of suture thread. The detail of the heartbeat canceller of (i) was reported in [3]. In this paper, we report the features of surgical robot and some experiments using it. 2. Surgical robot We have developed the master-slave surgical robot, as shown in Fig. 1. The degrees of freedom of the master manipulator [4] and the slave manipulator are seven for each. One slave manipulator consists of two manipulators; a positioning manipulator [5] and a surgical instrument manipulator. The tip of surgical instrument manipulator is formed of a forceps. We developed two slave manipulators. Therefore the operator can control the surgical robot by using both hands. The communication method between the master manipulator and the slave manipulator is using the network specification based on a TCP/IP protocol stack. The advantages of this robotic system are the following two: (i) Small size of the slave manipulator. (ii) Intuitive operation system. Each advantage is explained in detail. (i) Small size of the slave manipulator. There are many types of equipment placed around the surgical bed when OPCAB surgery is performed. Therefore, it’s preferred that the surgical robot is small and does not occupy the surgical work space above the patient. Furthermore, the surgical robot is required to be lightweight and easily movable resulting in the easy switching from the robotic surgery to the conventional surgery in case of any accidents. The size of the slave manipulator is width: 600[mm], depth: 800[mm], height: 1500[mm] (Fig. 2). The weight of this slave manipulator is 30 kg. (ii) Intuitive operation system. For the achievement of complex movement such as interarterial coronary anastomosis and ligation of suture thread, this surgical robot was designed as an intuitive operation system. In particular, each coordinate system of the manipulators, which is measured by 3D positioning sensor (OptotrakÒ), is adjusted to the endoscopic image’s coordinate system which is equivalent to the operator’s coordinate system. This provides intuitive operation for the surgeon because the slave manipulator showed on the monitor moves to the same direction as the surgeon moves the master manipulator. Because the tip of an endoscopic instrument moves to the

Fig. 1 The master-slave surgical robot system

123

S242

Int J CARS (2007) 2 (Suppl 1):S236–S245

Fig. 2 The size of slave manipulator

Fig. 4 In vivo experiment using a hog

opposite direction against the handheld gripper movement in conventional endoscopic surgery, this intuitive operation system of the surgical robots is very useful. 3. Experiment We performed some experiments with the surgical robot. The operation of the surgical robot is performed by a cardiac surgeon (Fig. 3). The experimental task is the following two. (a) We tested that operator could anastomose hog tissue by using the surgical robot. We chose thymus which is located above heart as a target of anastomosis. (b) We tested that it was possible to ligate a thread with the surgical robot. We used a thread which is made of rayon and 2 [mm] in diameter for ligation. A three-dimensional endoscope (SHINKO OPTICAL CO., LTD.) was used in both of the experiments (Figs. 4, 5). 4. Result and discussion (a) Anastomosis of tissue (thymus) in vivo experiment using a hog. The experiment of anastomosing the thymus was performed. In this experiment, it was possible to insert the surgical needle to the thymus and pull the needle with the surgical robot. But it was one time. So we need to carry out more experiments to anastomose tissue continuously. Furthermore, in the future, in order to accomplish OPCAB with the surgical robot, it is necessary to perform anastomosis experiments using real coronary arteries. (b) Ligation of the suture thread in a simple model. In the experiment of the ligation using the rayon thread, it was confirmed that the ligation was possible. Since three-dimensional endoscope was used, depth vision was good. However, in the experiment, it took a long time to ligate the thread because the thread was hanged up on forceps (a surgical instrument of the surgical robot). In the next experiment, it is supposed to be necessary to perform the ligation experiment using the suture thread which is used in OPCAB.

Fig. 3 The operation of the surgical robot in vivo

123

Fig. 5 The result of the ligated rayon thread

5. Conclusion We have developed the master-slave surgical robot for OPCAB surgery. Furthermore, we did in vivo experiments and we confirmed the feasibility of (a) inserting the surgical needle to the hog’s thymus and pulling the needle and (b) ligation of the suture thread using a simple model; a rayon thread was ligated with the surgical robot. In the future, the experiments of interarterial coronary anastomosis and ligation of the suture thread used in OPCAB will be performed. We will also perform the feasibility test using the integrated surgical robot system with the heartbeat canceller. Acknowledgments This work was supported in part by ‘‘the robotic medical technology cluster in Gifu prefecture,’’ Knowledge Cluster Initiative, Ministry of Education, Culture, Sports, Science and Technology, Japan ,the twentyfirst Century Center of Excellence (COE) Program ‘‘The innovative research on symbiosis technologies for human and robots in the elderly dominated society’’, Waseda University, Tokyo, Japan and ‘‘Establishment of Consolidated Research Institute for Advanced Science and Medical Care’’, Encouraging Development Strategic Research Centers Program, the Special Coordination Funds for Promoting Science and Technology, Ministry of Education, Culture, Sports, Science and Technology, Japan.

Int J CARS (2007) 2 (Suppl 1):S236–S245 References 1. Loulmet D et al (1999) Endoscopic coronary artery bypass grafting with the aid of robot assisted instruments. J Thoracic Cardiovasc Surg 118(1):4–10 2. Guthart G, Salisbury K (2000) The IntuitiveTM telesurgery system: overview and application. In: Proceedings of the IEEE international conference on robotics and automation, pp 618– 21 3. Umeda T, Okamoto J, Fujie MG (2005) Development of heartbeat sensing system for robotic surgery. The 1st international conference on manufacturing, machine design and tribology, ppDDI-402 4. Toyoda K et al (2006) Dexterous master-slave surgical robot for minimally invasive surgery-intuitive interface and interchangeable surgical instruments. In: Proceedings of of the 20th international congress and exhibition, Computer Assisted Radiology and Surgery, pp 503–504 5. Oura M, Kobayashi Y, Okamoto J, Fujie MG (2006) Development of MRI compatible versatile manipulator for minimally invasive surgery, BioRob 2006: the 1st IEEE/RASEMBS international conference on biomedical robotics and biomechatronics, pp 176–181

Motion estimation and reconstruction of a heart surface by means of 2D-/3D-membrane models K. Roberts, U. D. Hanebeck Intelligent Sensor-Actuator-Systems Laboratory, Institute of Computer Science and Engineering, University of Karlsruhe (TH), Germany. Abstract In order to assist surgeons during minimally invasive interventions on the beating heart, it would be helpful to develop a robotic surgery system, which synchronizes the instruments with the heart surface, so that their positions do not change relative to the point of interest (POI). The synchronization of the robotic manipulators requires an estimation of the heart surface motion. In this paper, a model-based motion estimation of the heart surface is presented. The motion of a partition of the heart surface is modelled by means of a thin or thick vibrating membrane in order to represent the epicardial surface or the connected epicard and myocard. The membrane motion is described by means of a system of coupled linear partial differential equations (PDEs), whose 3D-input function is assumed to be known. After spatial discretization of the PDE solution space by the Finite Spectral Element Method, a bank of lumped systems is obtained. A Kalman filter is used to estimate the state of the lumped systems by incorporating noisy measurements of the heart surface. Measurements can be the position or velocity of sonomicrometrybased sensors or of certain landmarks, which are tracked by optical sensors. With the model-based estimation it is possible to reconstruct the entire partition of the heart surface even at nonmeasurement points and thus at each POI. Keywords Motion synchronization Æ Heart surface model Æ Modelbased motion estimation 1. Introduction In order to assist surgeons during minimally invasive interventions on the beating heart, it would be helpful to develop a robotic surgery system, which synchronizes the instruments with the beating heart, so that their positions do not change relative to the point of interest (POI). The new alternative could facilitate interventions such as the placing of epimycardial left ventricular electrodes within the cardiac resynchronization therapy or the coronary artery bypass grafting (CABG) for patient with

S243 atherosclerosis. Instead of using the traditional heart-lung machine (on pump) during CABG, which leads to several risks like temporal cognitive loss, stroke and kidney complications, in many cases, this procedure can be replaced by an off pump CABG surgery. Until now, the most used master-slave robotic surgery systems daVinci and ZEUS did not offer an autonomous motion synchronization of the manipulators with the beating heart. The system architecture of a robotic surgery system in order to operate on the beating heart, first introduced in [1], should have two new features. First, in order to give the surgeon the impression that he operates on a motionless heart the image from the intervention area has to be stabilized. During the intervention the manipulators are synchronized with the heart surface based on the estimation of the heart surface motion. Second, for the synchronization, a method for motion prediction of the beating heart at each POI on the organ’s surface, based on measurements of heart motion, is required. Several studies for motion compensation algorithms with respect to the beating heart exist. In order to predict the heart’s motion at a POI in one direction, a displacement model with weighted Fourier series is used in [4]. In [1], 2D-motion of a POI is estimated by an autoregressive model based on 2D position measurements from a single camera. In [5] a method for predicting motion of surface points based on their x- and y-trajectories, respiration pressure signal, and ECG signal observed in the past is presented. The features are tracked robustly even with short-time occlusions. A predictive-control approach for filtering the heart motion is presented in [6, 7]. In [6], a 6 DOF robot is set up, which is synchronized with an oscillating target using visual servoing techniques and a modified generalized predictive controller to learn and predict the organ’s motion. The model-based predictive controller described in [7] is based on a motion model containing sonomicrometry measurements of the POI-trajectory out of the past heart beat cycle. In [3] a dense 3D structure recovery for temporal motion tracking of deformable surfaces from stereo image pairs is presented. In this approach a constrained disparity registration is used after an image rectification. In order to generate a still image of a moving surface in [2] a real-time binocular eye tracking of the surgeon is used. In the above mentioned methods, with exception of [5], the possibility of reconstruction of the heart motion at non-measurement points or occluded points is not discussed. In [8], we have presented a model-based estimator in order to predict and reconstruct a partition of the heart surface even at non-measurement points. The heart surface is modelled as a round vibrating 2D-membrane, whose motion in z-direction is described by a partial differential equation (PDE). The input function of the PDE is assumed periodic and is reconstructed. In this article, our approach from [8] is extended to a rectangular membrane which can move in all three dimensions. The external function is assumed to be known. In particular, the membrane can be either thin to represent the epicardial surface or thick to represent the connected epicard and myocard. In the following, in Sect. 2 the derivation of the membranes state models and the used state estimator is shortly discussed. In Sect. 3 simulation results are presented. 2. Method The elasticity of the heart surface is assumed to be linear. Hence, the motion of the membranes modulating the moving epicardial surface or the connected epicard and myocard are described by the Lame´ equations of the form:

ðk þ lÞrdiv u þ lr2 u þ du_ þ ðf
q€ uÞ ¼ 0

ð1Þ

123

S244

Int J CARS (2007) 2 (Suppl 1):S236–S245

Fig. 1 Reconstruction at a non-measurement point for the thick membrane in direction x (a), and in direction y (b)

Fig. 2 Reconstruction at a non-measurement point for the thick membrane in direction y (a), and in direction z (b) The Lame´ equations are a system of coupled PDEs, where the solution function u ¼ u1 ; u2 ; u3 describes the displacement of a membrane point (x,y,z) in the three dimensions x, y, and z. The function f ¼ f1 ; f2 ; f3 is the external force, which deforms the membrane in all three dimensions. In (1), the parameters k, l, and q are material parameters and d is the damping term. The thick membrane is described as a cuboid with six Neumann boundary conditions. The Neumann boundary conditions permit free membrane motion in all three directions. In the case of the thin membrane, the variables and derivations in z-dimension in (1) are neglected. As a result, the thin membrane is defined as a rectangle with four Neumann boundary conditions. For spatial discretization of the solution space of each linear PDE equation of the system (1), we use the Spectral Finite Element Method as discussed in [9]. In this paper, we consider only one finite element. Hence, for each direction j the solution uj is approximated by a finite sum according to

123

uj ðx; y; z; tÞ ¼

N X

ui ðx; y; zÞai ðtÞ

i¼1



N X

ui ðx; y; zÞai ðtÞ

for j = 1,2,3

ð2Þ

i¼1

In (2), ui(x,y,z) are three-dimensional polynomials and ai(t) are their time dependent weights. As described in [9], the system of PDEs can be transformed to a bank of lumped systems of order two, where the state consists of the ai and a_ i for each dimension j. The measurement equation, which maps the state to the position and velocity of a measurement point, is defined similar as in [9]. Then a Kalman Filter is used in order to predict the state, based on the system equation, and to update the state by incorporating noisy measurements.

Int J CARS (2007) 2 (Suppl 1):S236–S245

S245

Fig. 3 Reconstruction over time of the thin membrane (Fig. 2a) with 36 measurement points and of the thick membrane (Fig. 2b) with 48 measurement points 3. Results With the model-based estimation described in Section 2, it is possible to reconstruct the entire rectangular partition of the heart surface even at non-measurement points. The membrane models are evaluated, while the moving heart surface is represented by a Finite Element solution. The noisy position and velocity measurements are generated, while on the solution of local points Gaussian noise is added. Figure 1 shows the estimate of the solution u for the thin membrane and Fig. 2 for the thick membrane at a non-measurement point. In both cases the external force describes a rotation around the z-axis and a sinus bulge in z-direction. In Fig. 3 reconstruction results for the two membranes over time are presented. 4. Conclusions The reconstruction results of the simulated heart surface in Sect. 3 are promising and seem to be applicable. Future work is concerned with adaption to the individual patient heart motion by estimating the model parameter, e.g. the material parameters and to estimate the external force in a training model to predict the distributed heart motion. Acknowledgments This work was supported by the German Research Foundation (DFG) within the Research Training Group GRK 1126 ‘‘Development of New Computer-based Methods for the Future Working Environment in Visceral Surgery’’. References 1. Nakamura Y, Kishi K, Kawakami H (2001) Heartbeat synchronization for robotic cardiac surgery. In: Proceedings of the 2001 IEEE international conference on robotics and automation (ICRA 2001), pp 2014–2019 2. Mylonas GP, Stoynaov D, Deligianni F, Darzi A, Yang G-Z (2005) Gaze-contingent soft tissue deformation tracking for

3.

4.

5.

6.

7.

8.

9.

minimally invasive robotic surgery. Medical Image Computing and Computer Assisted Intervention (MICCAI 2005), vol 3749 of LNCS. Springer, Heidelberg, pp 843–850 Stoyanov D, Darzi A, Yang GZ (2004) A practical approach towards accurate dense 3D depth recovery for robotic laparoscopic surgery. Comput Aided Surg 10(4):199–208 Thakral A, Wallace J, Tomlin D, Seth N, Thakor NV, Surgical Motion Adaptive Robotic Technoloy (S.M.A.R.T): Taking the Motion out of Physiological Motion (2001) Medical image computing and computer assisted intervention (MICCAI 2001), vol 2208 of LNCS. Springer, Heidelberg, pp 317–325 Ortmaier T, Groeger M, Boehm DH, Falk V, Hirzinger G (2005) Motion estimation in beating heart surgery. IEEE Trans Biomed Eng 52:1729–1740 Ginhoux R, Gangloff J, de Mathelin M, Soler L, Sanchez MMA, Marescaux J (2005) Active filtering of physiological motion in robotized surgery using predictive control. IEEE Trans Robot 21:67–79 Bebek O, Cavusoglu MC (2006) Model based control algorithms for robotic assisted beating heart surgery. IEEE Eng Med Biol Soc EMBC 2006), pp 823–828 Sawo F, Roberts K, Hanebeck UD (2006) Bayesian estimation of distributed phenomena using discretized representations of partial differential equations. In: Proceedings of the 3rd international conference on informatics in control, automation and robotics (ICINCO 2006), pp 16–23 Bader T, Wiedemann A, Roberts V, Hanebeck UD (2007) Model-based motion estimation of elastic surfaces for minimally invasive cardiac surgery. 2007 IEEE international conference on robotics and automation (ICRA 2007), Italy

123