Toward High-Fidelity Telepresence in Space and Surgery Robotics

Home , Six degrees of freedom

Detlef Reintsema Toward High-Fidelity [email protected] Telepresence in Space and Carsten Preusche Surgery Robotics Tobias Ortmaier Gerd Hirzinger German Aerospace Center (DLR) Institute of Robotics and Mechatronics 82230 Wessling, Germany Abstract

High-ﬁdelity telepresence is considered to be a key subject for the development of advanced space and surgery robotic systems. The emphasis of this paper are the key challenges like multimodal data servicing, bilateral and shared control concepts, and kinesthetic feedback devices. These technologies are the basic principles in the development of advanced space and surgery applications. Beside these technologies, advanced mechatronic systems are required as shown within this paper. The appli- cability of the high-ﬁdelity telepresence concept is explored by selected space and surgery scenarios.

1 Introduction

The realm of telerobotics is the manipulation of a remote physical environment. Today’s telerobotic systems are used in many situations to overcome several kinds of barriers that block the human operator from task fulfillment, such as distance, scale, material, or hazardous matter. For space missions, distance and inhuman and hazardous areas are perceived as the most characteris- tic barriers; whereas, for minimally invasive surgery (MIS), cramped spaces and restricted freedom of movement are more likely to be encountered. Therefore, the mechatronic manipulator used within a telerobotic system differs between the realm of space and MIS because of the constraints of the specific environment. But, as we will show, the basic telepresence-enabling system technology is quite the same, and independent of the realm. The telerobotic paradigm is typified by sensing the physical environment, measuring positions, forces, and accelerations, and responding with movements and forces to directly manipulate the physical environment (Conway, Volz, & Walker, 1987). Telepresence can be characterized as an advanced concept of telerobotics: the remote robot is directly operated by a human within a closed-loop control mode using a telerobotic system (the so-called teleoperator) to perform remote manipulations. Three kinds of telepresence definitions are commonly used (Draper, 1995):

● Simple telepresence summarizes the ability to function in a remote environment, such as controlling a remote machine. ● Cybernetic telepresence is deﬁned by an index of the quality of a man- machine interface, such as examining the performance capabilities and lim-

Presence, Vol. 13, No. 1, February 2004, 77–98 itations of the human operator. © 2004 by the Massachusetts Institute of Technology ● The experiential deﬁnition is deﬁned by the degree of perception in which

Reintsema et al. 77

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 78 PRESENCE: VOLUME 13, NUMBER 1

a user feels physically present within the remote the-art robotic systems are still not able to cope environment. For example, Sheridan (1992) pro- autonomously with these complex requirements. vides an ideal measure of telepresence quality in Telepresence as an extension to the supervisory or terms of immersiveness perceived by the operator: teleoperated control in the way of direct manipula- “With sufﬁciently good technology a person would tion of the robot’s behavior by the human operator not be able to discriminate among actual presence, is the only way to overcome the lack of sophisti- telepresence and virtual presence” (p. 6). cated autonomy.

Thus, experiential telepresence systems enable a hu- A high-fidelity telepresence concept may overcome man operator to manipulate tasks in an inaccessible en- these drawbacks and barriers within the field of space vironment while feeling present at the event while phys- and surgery robotics. But telepresence requests high ically being at some other place (space shifting) or time demands to the technologies lying beneath the applica- (time shifting). For high-fidelity telepresence, the hu- tion. In particular the robot needs sensors compared man operator must feel as if he/she is present at a dis- with the human senses to gather the remote environ- tant location and interpret the mechatronic manipulator ment, which has to be displayed to the human operator. as a natural extension of his/her own body. This sug- The exploration and manipulation capabilities need to gests that the human operator receives input to (almost) be similar to the human capabilities, and the communi- all the human senses (vision, hearing, haptic, sense of cation has to be a broadband communication with low smell, and degustation) and commands the teleoperator delay to transport the sensorial input and the operator’s in a nearly natural way by demonstration. The last two reactions (commands) almost instantaneously, otherwise senses (smell and degustation) have no practical evi- the human’s feeling of being present at the distant loca- dence at present because of the lack of sensors and ac- tion is disturbed. tuators to measure and display them. This article focuses on the current telepresence activi- The fact that there are at present few robots in space ties of our institute in the field of space and surgical ro- and surgery is explained mainly by the lack of broadly botics. As we will show, both applications are essentially available sophisticated autonomy and tactile feedback driven by the same technology to overcome a combina- within current systems. tion of the abovementioned barriers.

● The technique of MIS was established in the 1980s. Surgeons use long instruments through small inci- 2 General Telepresence System sions. The advantages of MIS compared to open surgery are, among others, reduced pain and Within the scope of designing a general telepres- trauma, shorter hospital stays, and cosmetic advan- ence system, several classifications may be considered; tages. But the surgeon now loses direct access to each one examines a different aspect. We are interested the operation field, which in essence yields to re- in two major points of view. duced sight and tactile feedback due to the long instruments. Robotic surgery in combination with ● The basic human supervisory control paradigm as high-fidelity telepresence technologies like sen- defined by Sheridan (1992) separates computers by sorized scalpels and forceps may overcome the a barrier of distance, inconvenience, or time—one drawbacks and physical barriers, such as the pa- that loops directly to the human and one that loops tient’s chest. remotely to the human. ● The absence of robots in space is explained by the ● The service layer architecture is a set of service com- lack of sophisticated autonomy. Tasks in inhuman, ponents for setting up connections, connection unknown environments require a high level of plan- management and multisensory information, con- ning and reaction to unforeseen events. State-of- trol, and transmission between separate computers.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 79

Figure 1. A Multimodal Telepresence and Teleaction System (TPTA) by Buss and Schmidt (1999).

2.1 Design by Control (OSI) reference model—is to enable partitioning of the problem domain into different areas of concern. Figure Based on the human supervisory control concept, 2 associates a layered abstraction for a TPTA system. Buss and Schmidt (1999) generalize a system structure Each layer offers services to the level above and provides for multimodal telepresence and teleaction (TPTA), its own services on the services of the layer below. A which is split up into the human operator and the tele- single layer may contain several entities providing the operator environment (see Figure 1). Toward telepres- same service. ence, multimodal control commands (messages) of the As long as the services are defined in terms of their human operator (e.g., voice or motion) have to be as- similated by adequate man-machine interfaces and functionality and not as a specific implementation (e.g., transmitted to the teleoperator, which interprets and program code), each layer’s services can be extended performs the command input. Thus, to provide human without destroying the overall structure. The layered senses with proprietary information, sophisticated MMIs approach also simplifies the protocol specification. In- are required, like kinesthetic displays in the case of the stead of a single all-embracing protocol, specific inde- human haptic sense. pendent protocols for each layer can be specified that One important aspect of multimodal supervisory con- are less complex than a single protocol. trol is the kinesthetic coupling of the human operator The network service layer provides transport network and the teleoperator based on the concept of bilateral communication and offers interfaces to common stan- control. Hence, sensory feedback of the teleoperator dardized protocol stacks (e.g., Internet Protocol (IP) raised during the interaction between the teleoperator Suite, Integrated Services Digital Network (ISDN), and its environment is needed as an input to the bilat- Asynchronous Transfer Mode (ATM), or Space Com- eral closed-loop control process and to the remote sen- munications Protocol Standards (SCPS)). Direct service sory feedback loop in the case of shared or autonomous access to the transport network requires the assembling control. (marshaling) and reassembling (unmarshaling) of application data into a form suitable for transmission in messages as requested by the specific transport network. 2.2 Design by Separation Thus, (un)marshaling should be performed by a special The purpose of layering and separation princi- mediator layer, the middleware. ples—as with the ISO Open System Interconnection A middleware layer acts as a mediator that separates a

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 80 PRESENCE: VOLUME 13, NUMBER 1

Figure 2. The service layer architecture of a TPTA system.

transport network from the application domain. The Whereas transactional and message-oriented ap- usage of a middleware layer enables transport network- proaches are normally used in distributed databases, independent communication without time-consuming procedural and even more object-oriented middleware adaptations and redevelopment of domain applications is in common use within telerobotic systems to resolve (Lin, Unterschu¨tz, Vogel, Reintsema, & Koeppe, scalability, interoperability, and transparency issues in 2001). A categorization of middleware technologies are heterogenous environments (Preusche, Hoogen, Reint- given by Emmerich (2000): sema, Schmidt, & Hirzinger, 2002; Bottazzi, Caselli, Reggiani, & Amoretti, 2002; Jia, Hada, Ye, & Takase, ● Transaction-oriented middleware, which ensures that an atomic operation either occurs completely 2002; Ghodoussi, Butner, & Wang, 2002). or not at all; At the end of the layered stack, the application layer ● Message-oriented middleware enables message trans- solves domain-speciﬁc real problems of interest. Based mission, receipt, and queuing services to provide on the control aspects, we are still working on a sub- asynchronous communication between components separation of the application layer into a telepresence (e.g., messages queues). stream control layer, a bilateral control layer, and two ● Procedural-based middleware enables components proprietary sublayers: one for local feedback control to to invoke a procedure or method call on a remote the human and another for remote shared control of the component, such as a Remote Procedure Call teleoperator. The sublayers on the top of the stack en- (RPC). capsulate domain-speciﬁc issues and facilities in space or ● Object-oriented middleware extends the functional- surgery robotics. The telepresence layer is introduced to ity of procedural middleware by the object-oriented offer a service interface to establish, maintain, and re- programming paradigm (e.g., DCOM, J2EE, or lease multisensorial data streams within the telepresence CORBA). system.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 81

Telerobotic systems are usually built on the top of upcoming complexity, we prefer a distributed modu- the transport network (e.g., Transport Control Pro- lar component design of the TPTA system built up tocol (TCP), User Datagram Protocol (UDP)). At on CORBA (Common Object Request Broker Archi- this point of abstraction, application engineers need tecture), an object-oriented middleware approach by to implement session and presentation layer function- the Object Management Group (OMG) (1998) be- ality as classified by OSI. In case of cooperative assis- cause CORBA is network and language independent tance as within multioperator and/or multiteleopera- and offers bindings to several programming languages tor environments, a multipoint configuration is such as JAVA, C, or Cϩϩ. necessary instead of a simple point-to-point commu- In the layered view presented before, both the nication model. Switched requests of multisensory human-system interface and the teleoperator are repre- information from various teleoperators is as complex sented by structural components within the application as implementing session or presentation issues, which layer, and information between them are typically trans- should be handled if possible by improved middle- ferred through the underlying system layers. Although ware technology. many researchers are still working on a modular component framework (Preusche et al., 2002; Bottazzi et al., 2002; Jia et al., 2002; Ghodoussi et al., 2002), a gen- 3 Key Challenges of Telepresence, eral or even standardized component architecture does Teleoperation, and Robotics not yet exist. But common to most of the modulariza- tion approaches is the client-server paradigm: remote Multimodal telepresence systems extend the pow- physical resources like the manipulator or several sensors erful sensory and manipulative skills of a human opera- provide services that can be requested by the local oper- tor to a remote environment. For this purpose, the tele- ator station for task programming and/or monitoring operator (TOP) at the remote site is connected through issues. In the case of bilateral control, we need sufficient a transport network to the human-system interface servants on both sites. (HSI) on the operator site. Visual, auditory, kinesthetic, The telepresence layer we have in mind separates the and tactile information has to be measured, displayed, application layer components from multisensorial data served, and controlled within the process of telepresence transmission because control mechanisms for different control. data and connection types are reusable in different tele- We suggest that network servicing and bilateral and presence applications. Thus, the telepresence layer offers kinesthetic feedback control in combination with differ- service interfaces to establish, maintain, and release mul- ent types of control autonomy are the most important tisensorial data streams within a TPTA system, similar to fields that influence the stages of immersion achieved by OMG (2000). telepresence technology. 3.1.1 Control and Data Streams. The deploy- ment of telepresence technology in complex telerobotic 3.1 Telepresence Service Architecture environments presumes timely synchronized transporta- Telerobotic systems are traditionally built upon tion of multimodal media data if a good immersion of a dedicated system architecture providing a sufficient the human operator should be achieved. Multimodal master-slave-system connection (Sheridan, 1992). In media data are minimally a combination of audio, case of a single-operator–single-teleoperator telepres- (stereo)video, and haptic/tactile information. Besides ence scenario, the system architecture and intercon- synchronization by time, each modality causes very spe- nection is less complex than in single-operator– cific and different demands on required quality of ser- multiple-teleoperator or even multiple-operator– vice (QoS), such as against the transmission bandwidth multiple-teleoperator environments. To manage the and maximally tolerable time delay. It seems to be a

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 82 PRESENCE: VOLUME 13, NUMBER 1

Figure 3. The telepresence stream control sublayer.

good approach to establish and reserve a separate virtual stream data (such as images) and is directly built upon data connection for each modality within a telepresence the transport network protocols (such as TCP or UDP). system to enable modality-specific network QoS bind- In the typed case, communication is via operations de- ing. fined in OMG IDL. Events such as control data are Multisensorial data such as positions, forces, or passed by means of the parameters, which can be de- torques have to be streamed between distributed service fined in any manner desired. As illustrated in Figure 3, ports (e.g., shared memory or message queues) of the two kinds of network connectors are suggested to uti- bilateral control layer. Thereby, a single data stream rep- lize these communications: a network connector for resents a set of continuous sequenced multisensory data stream connections and a middleware connector for frames. Multisensorial data can be transfered in one di- control events. rection only (unidirectional transfer, e.g., stereovideo images) or in both directions (bidirectional transfer, 3.1.2 Stream Interconnection. The control e.g., kinesthetic information). and management of multimodal data streams addresses High-level system control as well as stream control many issues: stream topologies, multiple streams, stream messages are requests that require a synchronized pro- description, stream setup and release, stream modifica- cessing within the TPTA system. Therefore, they will be tion and termination, multiple protocols, stream syn- transferred by separated nonstream data connections. chronization, and security. Instead of using a dedicated network connection, such The several multimodal data channels of a TPTA system event messages will be transfered upon dedicated opera- need to be synchronized. Beneath embedding synchroni- tions defined in OMG IDL. zation typically used within MPEG streams, external sys- The communication itself can be either generic or temwide synchronization of all configured data channels is typed. In the generic case, all communication is by essential for the telepresence stream control layer. The syn- means of a generic character buffer that packages all the chronization process is defined by specific protocols.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 83

We suggest a hierarchical protocol delegation: a con- extremes of automatic control and manual control nect control protocol manages and monitors the stream (Sheridan, 1992): “Human operators are intermittently connection process, while each particular data stream programming and continually receiving information has to provide an adapted transmit control protocol, from a computer that itself closes an autonomous con- which is related to the connect control protocol. A par- trol loop through artificial effectors and sensors.” In the ticular transmit control protocol may serve further case of telepresence, the received information about the stream-specific protocols like carrier protocols such as teleoperator has to be displayed in a sufficiently natural HTTP, XML, TCP, UDP, or AAL-5, as well as non- way (hype the human senses). standardized self-defined protocols. A first approach of a To enable telepresence in space and surgery, we pre- generic interconnection framework is given by Preusche fer a sufficient shared control concept (Conway et al., et al. (2002). 1987; Hayati & Venkataraman, 1989) for the control of The stream controller offers the service interface to the teleoperator. Herein, shared control is based on lo- establish, maintain, and release stream connections. cal sensory feedback loops (Hirzinger, Brunner, Stream control services like configure, connect and Dietrich, & Heindl, 1994) at the teleoperator site (see disconnect streams, or start and stop data trans- Figure 1), by which gross commands were refined au- mission applied on a stream are simultaneously dele- tonomously providing the teleoperator with a modest gated to all configured flows or just a subset of them kind of sensory intelligence. The human operator origi- according to the stream configuration. nates gross path commands by using a kinesthetic feed- The system configurator is responsible for stream ne- back device, and these commands are “fine tuned” by gotiation and configuration as requested by the specific the teleoperator. application. The several control channels serve the syn- In space applications, we favor a shared autonomy chronization of the bilateral control process, the loca- concept that distributes intelligence between the operation of different teleoperators, or the shared and secure tor and the teleoperator in the sense of a task-directed access to one or more teleoperators. The system con- approach (tele-sensor-programming), see Figure 4 figurator uses standardized CORBA services (OMG, (Brunner, Arbter, & Hirzinger, 1994). As a part of this 1997), such as event, property, or security services to concept, the shared control approach offers the exten- provide network transparency within the TPTA system. sion and addition of telepresence control of servicing Thereby, network transparency is accomplished by ac- robots in space. cess transparency, which is achieved by CORBA’s Ob- In the field of surgery robot assistance, a shared con- ject Request Broker, and location transparency, which is trol approach as shown in Figure 5 can be used to com- achieved by CORBA’s naming service. pensate for organ movements. This compensation is a Human-system interfaces for telepresence control highly desired functionality if the surgeon has to work have to combine one or more items of multimodal dis- on a moving organ. The robot compensates for this displays. Using the telepresence service layer, it should be turbing motion, such that the relative pose between the able to bind sufficient streams to provide a virtual target area and the surgical instrument remains con- human-system interface configured according to the stant. The surgeon can then work on a virtually stabi- problem domain such as space or surgery robotics. lized organ. This is especially the case in beating-heart bypass grafts. Mechanical stabilizers (e.g., Octopus by Medtronic) are used in these operations to reduce the 3.2 Telerobotic Concepts Based on motion of the beating heart. Shared Control The remaining motion of the operating field, which is The fundamental control concept of telepresence located between the two stabilizer arms (see Figure 6), is human supervisory control. Sheridan characterizes has an amplitude of approximately 1.5 mm (Jansen, human supervisory control as related between the two 1998). However, this motion complicates vascular

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 84 PRESENCE: VOLUME 13, NUMBER 1

over, the laparoscope can capture the motion of the whole operating field and not only of a few selected points. Using artificial landmarks is problematic, too, due to the limited space. Therefore, prominent image structures on the heart surface are used as natural landmarks. The motion of the landmark is approximated by an affine motion model. In most cases, a reduced model that takes only translation into account is sufficient. The parameters of the motion model are found at the mini- mum of a dissimilarity measure (e.g., sum of squared differences). Thus, the past motion of the landmark is found and can be used to predict the motion in the near future. A promising approach for prediction that provides stable results is time series analysis. The obtained near-future positions of the landmarks are used to command the robot such that both heart and instrument move synchronously (see Figure 6).

3.3 Bilateral Control

In the control of a TPTA system, different degrees of kinesthetic coupling can be distinguished between the master and the slave system. A pure remotely controlled slave system provides no kinesthetic feedback to Figure 4. The concept of tele-sensor-programming as demonstrated the operator. As within simple telepresence systems, during the ROTEX mission. only visual (and/or auditory) feedback of the remote scene is given. This limits the feeling of being present at the remote site, but it reduces the problem of stability within the TPTA system. anastomosis and undesirably extends the operating time If the master and the slave are coupled by kinesthetic (Kodera, Kiaii, Rayman, Novick, & Boyd, 2001). feedback, the overall control scheme represents a bilateral control scheme. By transmitting data with physical The reliable measurement and prediction of the mo- meaning (such as force, position, and velocity) between tion is a prerequisite for the compensation of the re- the slave and the master system, these two systems are maining heart motion (Ortmaier, Gro¨ger, & Hirzinger, coupled energetically. Under the presence of time delay, 2002). In the case of contact between a surgical instru- a stability problem arises that needs to be handled by an ment and the heart surface, the motion of the heart at appropriate control scheme. this contact point can be estimated indirectly via force Basically, there are four different possibilities to cou- sensors integrated into the instrument. If there is no ple a master-salve system (Yokokohji & Yoshikawa, contact between instrument and heart surface, contact- 1992), that can be extended by combining two or more less sensors are applied. of these options: The laparoscope, which is present anyway, seems at- tractive because introducing additional sensors into the ● position–position strongly crowded operating ﬁeld is problematic. More- ● position–force

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 85

Figure 5. Overview of the shared control concept in minimally invasive surgery.

measurement of the overall system position change. Damping this position deviation will also stabilize the system in the presence of time delay. Figure 7 illustrates the control structure of the teleoperation scenario using position commands for the slave robot and force feedback for the operator. The “Comm” block signiﬁes the communication between the two systems including the corresponding time delay. The oper- Figure 6. Natural landmarks and tracking area on a beating heart. ator’s dynamic is merged with the dynamic of the master device as considered with the “Master Device”

block. Herein, fm is the force that is commanded to the

● force–position master device, and fOP is the force due to the human ● force–force operator’s intentions. The force feedback control law is then given by 3.3.1 Force–Position Coupling. First, one can ϭ ͑ Ϫ ͒ ϩ use a straightforward control scheme: putting the slave fm KP xs xm Kf fs, (1) robot in position control mode and displaying the mea-

sured forces to the human through a force-controlled where xs and xm are the positions of the slave robot and

master device. But this concept is very sensitive because the master device, respectively, and fs is the measured

of the presence of time delay within the communication force at the slave’s end effector. KP describes the posi-

channel; thus, the system becomes unstable in its pres- tional coupling between the two devices. KP can be ence. considered as a mechanical spring between the Cartesian To stabilize the master-slave system, additional damp- positions of the master and the slave robot. High values ing needs to be introduced to the system. We used the indicate a very stiff coupling, so the human operator will system’s position deviation with respect to the time as a directly feel the movement of the slave robot with its

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 86 PRESENCE: VOLUME 13, NUMBER 1

Figure 7. The control structure for a force–position teleoperation.

Figure 8. The control structure for a force–force teleoperation.

dynamic properties plus the additional dynamic due to vides better results. This control scheme has its beneﬁts, the communication time delay (intervening dynamic; especially in stiff contact situations with large time de- Yokokohji & Yoshikawa, 1990). lays when the force–position scheme produces high Such a coupling introduces a virtual force that does feedback forces. not exist in the real world to the operator. When the Figure 8 shows the complete teleoperation system for slave manipulator reaches the desired position, this force the force–force coupling with additional damping disappears, and, in the static situation, unbiased force through position deviation feedback (virtual coupling). feedback from the force/torque sensor occurs. Analo- From the slave robot, the measured Cartesian position gous to the terminology in virtual reality (VR) applica- and force is fed back to the operator side. The force tions, we call this a “virtual coupling” between the mas- feedback control law is the same as in the position com-

ter and the slave system. manded case. (See Equation 1.) Again, KP is the posi- This control scheme focuses on the position following tion coupling between the two robots. The slave robot

of the slave robot. This means that the position devia- is force controlled to establish fdesired to the environ- tion between the master and the slave is small, but the ment. force displayed to the operator can vary from the force present at the slave robot. 3.3.3 Adapting the Virtual Coupling. In both cases presented before, we have a force associated with 3.3.2 Force–Force Coupling. If the slave robot the position error between the two robots in the force needs to follow a speciﬁc force trajectory given by the feedback control law. This force of the virtual coupling human operator, then a force–force control scheme pro- is needed to reduce the master device/human dynamic

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 87

to handle the position overshooting and to overcome ● low impedance for free motion and high impedance the stability problems induced by the communication for the simulation of hard impact and contact with

time delay. The suitable values for KP change depend- rigid surfaces, and ing on the contact situation at the slave side. ● applicable forces/torques must be a tradeoff be-

In free motion, KP can be small because the felt force tween security aspects and ﬁdelity during the task. at the master side should be small and the slave robot acts with its best dynamic. When near and in contact, The last two points are especially incompatible be-

KP has to be higher to reduce overshooting and to cause high motor torques are required to display low avoid instability (Yoshikawa, 1993). This leads directly impedance. So the control structure itself has to cope

to adapting KP during the operation task. with the security problem, including high precision and

If KP is exactly the environment stiffness, the result- redundant sensor information. ing virtual force disappears for the user because the Hence, a robot with kinematics and sensory feedback force signal from the force sensor is interpolated in the capabilities similar to the human arm is required. DLR’s sample time of the master device. To achieve this, the new lightweight robot incorporates many advanced environment stiffness is calculated as technologies needed by such a force feedback device. The most important features are high load-to-own- ⌬ fs K ϭ ϭ K . (2) weight ratio, joint torque control (allowing programma- Env ⌬x P s ble impedance, stiffness, and damping), increased flexi- Of course, Equation 2 represents only the static case, bility and manipulability due to seven-DOF kinematics, but, in the two applications in this paper, the operation and fully integrated electronics and sensors. Before the speed is limited. In space and surgery, safety consider- underlying mechatronics concept is given, a description ations lead to a limitation of the maximum execution of the desired control strategies to achieve high-quality speed that is beyond the contact dynamic. In the case of force feedback follows. general purposes, the exact physical environment dynamic has to be considered. 3.4.2 Force Control on a Torque-Controlled Device. Figure 9 shows a general force control scheme for a redundant, compliant robot. In the case of a robot 3.4 Kinesthetic Feedback Devices with torque-controlled joints, ␶ is used to command and Control d the robot. An easy control law for this control approach To provide high-fidelity multimodal feedback to is given by the human operator, it is very important in the context of manipulation to give a kinesthetic feedback such that ␶ ϭ ͑ ϩ ͑ Ϫ ͒͒ ϩ ␶ d JR fd Kf fd fm dyn, (3) the operator can feel what he/she is doing. This is an important step towards experiential and immersive tele- ␶ where d is the torque commanded to the joints, JR is presence. So kinesthetic feedback devices and appropri- the Jacobian matrix of the robot, and fd is the desired ate control is definitely an enabling key technology to and fm is the measured force, respectively. The parame- telepresence applications. ter Kf modifies the dynamic of the force controller and is the significant parameter for the performance of the 3.4.1 The Ideal Force Feedback Device. The ␶ system; Kf is a force gain, and dyn are the torques to requirements for a ideal force feedback device with re- compensate for the robot’s dynamics. The torques ␶ spect to a general application are dyn are computed from the robot dynamics as follows: ● six-DOF workspace comparable to the human arm ␶ ϭ ͑ ͒ ϩ ͑ ͒ ϩ ͑ ͒ and/or according to the desired task, dyn M q q¨ C q,q˙ G q , (4)

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 88 PRESENCE: VOLUME 13, NUMBER 1

Figure 9. Force control based on torque controlled joints.

where q are the joint angles, M(q) is the mass matrix, Table 1. Different Control Parameter Combinations Subject

C(q,q˙) the Coriolis and centrifugal forces, and G(q) is to Different Robot Stiffnesses KC the gravity term. K P P P The force control based on the torque interface of the C R fc total robot meets the requirement of low impedance of the case 1 500 N/m 500 40.03 20,015 device very well, but it has a disadvantage when Carte- case 2 4000 N/m 4000 5.03 20,120 sian accuracy is required. Depending on the Jacobian

matrix, JR, the ﬁdelity in the Cartesian space is de- formed or even degraded in the case of a singularity. which can be controlled by the joint state feedback con- 3.4.3 Force Control on an Impedance Con- troller. trolled Robot. Generally a cascaded controller with a Then the total open loop gain is assigned to position or velocity controller in the inner loop leads to P ϭ P ͑1/K ͒P ͑K ͒, (6) better Cartesian accuracy and control of the manipula- total fc C R C tion task by the human operator. Here, the disadvan- where Pfc, the force control gain, is inverse and PR is tage is the higher impedance of the feedback device, directly proportional to the Cartesian stiffness (KC)of which can be felt by the operator. For improvement, an the robot, which is controlled by the joint impedance impedance controller is used at joint space as described controller. This means that the gain of the force con- next. troller can be chosen either in favor of the force control- For the design of the Cartesian force controller, a PT1 ler itself (case 1) or in favor of the joint impedance con- behavior with additional dead time of the robot in each troller (case 2). The overall bandwidth of the system degree of freedom is assumed: remains the same. (See Table 1.) The gain can now be chosen to assign desired properties to the robot in free 1 Ϫ ͑ ͒ ϭ TDRS GR,i s PR ϩ e , (5) motion or during interaction with the remote environ- 1 TRS ment.

where i denotes the corresponding DOF, TDR the dead

time, TR the robot rise time, and PR the resulting pro- 3.4.4 Passivity Oberserver and Passivity Con-

portional gain in the Cartesian space. PR and TR are troller of Data Input. Another source of instability of dependent on the internal impedance of the robot, the kinesthetic feedback device can be the input of ac-

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 89

tive energy from the teleoperator system. The human space as possible. We use motors without housings (see arm is capable of handling a certain amount of active also the following motor explanation), special short and energy when dissipating it through its muscles (damp- lightweight harmonic drive gears and modiﬁed electro- ing), but if this capability is exceeded the system be- magnetic brakes with reduced power consumption and comes unstable. Hannaford and Ryu (2002) introduced decreased weight. the concept of Time Domain Passivity Control, in The gears are provided with aluminum wave genera- which a passivity observer (PO) integrates the amount tors and circular splines. All housings are made of alumi- of energy passing through a communication port in real num (saving 40% weight) and are designed to transfer time and a passivity controller (PC) adds damping if the thermal energy from the motor to the surrounding air. communication port becomes active. This concept was The joint is also equipped with hollow shafts for the extended to multi-DOF kinesthetic feedback devices internal cabling, a key feature for serial kinematic struc- (Preusche, Hirzinger, Ryu, & Hannaford, 2003) prefer- tures later on. ing a geometrically induced allocation of the damping among the degrees of freedom of the device. 4.1.1 Electronic Components of the Intelli- In pure exploration tasks, the communication port is gent Joint. Because the digital, the analog, and the never allowed to become active, assuming that no active power converter electronics have to be closely inte- elements are in the remote environment, but, in the grated, it is inevitable that the required voltages have to case of manipulation, some active energy may be as- be generated galvanically isolated. The lightweight ro- signed by the remote system and has to be transferred bot has a two-step conversion system. In the base of the to the master system. So this intended amount of en- arm, a 20 kHz, 100 V peak to peak, square-wave volt- ergy has to be transmitted as well, and the concept of age is generated. This AC voltage can be transformed reference energy behavior (Ryu, Hannaford, Preusche, on the supply boards via tiny ring core transformers. & Hirzinger, 2003) is implemented to let the human Besides the galvanic isolation, the grounding scheme operator feel this behavior of the remote environment. has to be properly conﬁgured to avoid ground loops and interference among the individual electronic components. 4 Telepresence-Enabling Mechatronic The analog interface (AI) controls the motor currents Concepts at DLR and performs analog/digital conversion of the DC motor supply voltage, motor currents, analog hall sensor The telepresence technologies presented previ- signals, the joint torque sensor, and the status signals. ously are based on a sophisticated mechatronic robot The digital signal processor board (DSP) performs and sensor hardware also developed at the DLR, which the joint control and handles the communication to the is shortly presented in the following subsections. host computer via SERCOS interface, a standardized real-time communication system. Also integrated into the joint are the power electron- 4.1 Intelligent Joint ics driving a three-phase brushless motor. This electron- The intelligent joint is designed in a pure mecha- ics include a miniaturized, temperature-compensated tronic approach to optimize the interaction between current sensor. mechanical, electrical, and control (informatic) parts for best performance. The joint contains a torque sensor as 4.1.2 Brushless DC Motor with Position Sen- well as link-side and motor-side position sensors. Fur- sor. As a matter of fact, robot manufacturers have in thermore, the joint is equipped with electromagnetic the past taken the best available motors “off the shelf” brakes. All these components, including motor and for their robots, without being optimized for robotic gear, are placed inside the housing to save as much applications (comparably slow rotational speed but high

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 90 PRESENCE: VOLUME 13, NUMBER 1

Figure 11. DLR’s state feedback controller with gravity compensation. Figure 10. Components of DLR’s intelligent joint.

dynamics, permanently reversing operation around zero Using joint parameter identification, an accurate joint speed) and aiming at minimal weight and power losses. model was obtained, which was subsequently used for Thus, we have gone through two years of a concurrent the design and simulation of model-based controllers engineering and optimization process that took into (Albu-Scha¨ffer & Hirzinger, 2001). account all the electromagnetic and other physical ef- On the joint level, we use a state feedback control- fects, short copper paths, optimal coil winding, and coil ler with compensation of the gravity torque vector filling aspects between the magnetic iron poles. As a (Figure 11). In our opinion, this structure represents conclusion, the stator poles had to be subdivided and the minimal configuration that is able to provide both wound separately. positioning accuracy and effective oscillation damp- The result is DLR’s high-energy ROBOdrive with ing. The controller constitutes a direct extension of only half of the weight and half of the power loss of the the PD controller still used in most industrial robots (to our knowledge) best available motors (see Figure in the case of manipulators with joint elasticity. It is 10). Simulative and experimental results differed by only proven to be passive and globally asymptotically sta- 1–2%. ble for a wide range of parameters (Albu-Scha¨ffer & Hirzinger, 2000, 2001). An important feature of this 4.1.3 State Feedback Joint Controller. A chal- controller structure is, by a suitable parameterization, lenging control problem results from the lightweight that it can be used to provide a position, a torque, as design of the joint, which inherently leads to increased well as an impedance controller. elasticity. So, a flexible joint robot model must be assumed (Spong, 1987), which implies a fourth-order 4.2 Mechatronic Manipulators model for the joint. By measuring the motor position as well as the joint torque and by computing the numerical The experience with the highly integrated mecha- derivatives, the complete joint state can be obtained. tronic joint is the basis for the mechatronic robot devel- Our control strategy is to use the available sensors to opments at DLR. A task-specific lightweight robot for implement the desired task behavior and to compensate space applications and a special medical robot for mini- for the effects of joint elasticity. mal invasive surgery are developed.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 91

botic structure with a high payload-to-weight ratio, like our first two LWR developments (Hirzinger et al., 2000; Hirzinger, Albu-Scha¨ffer, Ha¨hnle, Schaefer, & Sporer, 2001). For the LWR III, the idea of a completely modular assembly system with only a few basic components concerning joint mechanics, electronics (i.e., mechatronics), and links is realized, so that completely different configurations can be composed in a short time. Indeed, the new concept is based on a fully modular joint-link-assembly system, with only a few basic components, namely three one-DOF robot joint link types and one two-DOF wrist joint. These modularity concepts are supported by a powerful kinematic-dynamic analysis and design software based on concepts of concurrent engineering. The kinematic-dynamic real-time simulation shows the aris- ing forces/torques in the joints, and—after passing FEM calculations—provides clues to necessary material strengths (e.g., in bearings and carbon fiber structures). Another result of the kinematic-dynamic simulation is imitating the human wrist joint: a ball-shaped, two-axis wrist joint shows a much higher mobility. Indeed, the most important joints of a robot are the wrist joints because manipulability of the robot is significantly depen- Figure 12. Third generation of the DLR lightweight robot (LWR III). dent on the underlying kinematic configuration.

4.2.2 Surgical Robot. An advanced robotic sur- 4.2.1 Lightweight Robot for Space Missions. gery arm should exploit a kinematically redundant struc- For space (as a technology driver) but also for the wide ture (i.e., at least seven joints, see Figure 13). This al- variety of future terrestrial service robot applications, lows for null-space motion enabling the robot’s pose to definitely soft and sensor-controlled lightweight arms be reconfigured while the position and orientation of (in contrast to the stiff and heavy industrial solutions) the instrument remains unchanged (Craig, 1989). If in are needed in addition to articulated, multifingered addition to the redundant structure joint torque sensors hands, which come closer and closer to the delicacy of are also implemented, pose reconfiguration of the arm human performance. This requirement is evident espe- can be achieved in an intuitive way by touching and cially in the scope of high-fidelity multimodal telepres- pushing the robot into the desired direction. Further- ence because the teleoperator as “natural extension” of more, the redundancy can be used to implement the human arm/hand needs to have human-like abili- collision-avoiding arm control, thus leading to a more ties. The lightweight robots also cope with the require- flexible operating room setup. This represents an advan- ments of low-impedance robots for kinesthetic feedback tage in comparison to nonredundant systems in which on the operator side of the telepresence system. surgeons often have to cope with collisions between the Actually, the third generation of DLR’s lightweight robotic arms during surgical interventions. robot (LWR III) is now operable (see Figure 12). Based The link lengths of the robotic arm have a big effect on the intelligent joint, it is a seven-joint redundant ro- on the performance of the robot. It is therefore impor-

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 92 PRESENCE: VOLUME 13, NUMBER 1

Ortmaier, & Hirzinger, 2001), usually requiring high sample rates (approximately 1 ms). Therefore, to com- municate with sensors, actors, and computational resources at this high level, a high-speed serial bus is necessary (Haidacher et al., 2003). The application of high- speed communication protocols allows us to place as many computational resources as possible outside of the robotic arm, which leads to further weight reduction of the robotic arm itself. To meet the requirements for advanced surgical robots (such as lightweight structure allowing handling by one person and redundant kinematics introducing more flexibility into the operating room) as defined before, it is necessary to exploit today’s possibilities of mechatronics. Therefore, surgical robots developed at DLR will have similar characteristics as the lightweight robots previously described. Components and technologies used in the lightweight robots form the base for the develop- Figure 13. The redundant link configuration of DLR’s kinemedic concept. ment of an advanced and powerful surgical robotic arm.

4.3 Sensors and End Effectors

tant to determine the link lengths such that the robotic For telepresence applications, one important as- arm performs optimally with respect to dexterity and pect is also the “sensibility” of the teleoperated robot. accuracy in all considered MIS application scenarios. An As the extended arm of the human, the teleoperator analysis of this type was conducted by Konietschke needs an equal sensory input to provide sufficient feed- (2001), where especially the situation of thoracic sur- back for the human operator and to fulfill the shared gery, totally endoscopic bypass grafts, mitral/aortic autonomous tasks. Therefore, the DLR is developing valve repairs/replacements, and tricuspid valve repairs sophisticated end effector sensors. are considered. To be easily mounted or removed by a nurse during 4.3.1 ROTEX Gripper. The ROTEX end effec- an operation, the robotic arm should be lightweight. tor is a complex, multisensory, two-finger gripper. The Additionally, the robotic arm has to be stiff enough to gripper sensors belong to the new generation of DLR ensure high-precision operation. As this usually contra- robot sensors based on a real multisensory concept with dicts the lightweight design goal, the choice of appro- all analog preprocessing and digital computations per- priate materials (e.g., carbon fiber, aluminum) as well as formed inside each sensor or at least in the wrist in a a detailed analysis of highly stressed parts with regard to completely modular way. In the gripper, 15 sensors are stiffness, fatigue, and durability is important to meet provided, in particular these goals. To achieve a high level of immersion of the surgeon ● an array of nine laser range finders, into the remotely performed operation, advanced con- ● a4ϫ 8 tactile sensing array, trol algorithms (such as impedance control in combina- ● a stiff and compliant six-DOF force-torque sensor, tion with kinesthetic feedback) are necessary (Preusche, ● a pair of CCD cameras, and

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 93

● an electrical gripper drive (approximately 1 N grip- ping force resolution).

4.3.2 Compliant Force/Torque Sensor. In- spired by the problems of industrial robot assembly, where stiff heavy robots are working in stiff environments, the DLR developed a compliant force-torque sensor. This new force-torque sensor technology is based on a position sensing detector (PSD). With a set of these sensors, the deviation between the sensor’s basis and the sensor head, which are coupled by a set of springs, is measured. From the knowledge of the spring stiffnesses and geometry, the resulting force can be calculated from the position signals delivered by the PSDs. This sensor is now offered by an industrial partner of the DLR.

4.3.3 DLR Laser Scanner. The DLR laser range scanner is an active, optical sensor for the exploration of Figure 14. Second generation of DLR’s robotic hand. 3D environments. The miniaturized sensor includes a scan head rotating at a constant speed. In the scan head, a patented laser triangulation distance sensor is inte- fingers with three DOF each, plus one DOF for recon- grated. The distance signal of every scanned point is figuration of the palm. Each finger can carry approxi- computed by the integrated electronics of the scan head mately 3 kg and has a six-DOF force-torque sensor in and transmitted to the laser range scanner’s chassis in a its fingertip. contact-less way by an optical link. The power supply of the scan head is also implemented contact-less using 4.3.5 Sensorized Scalpel. A sensorized scalpel coils. Driven by a stepper motor, the scan head gets its for MIS with a diameter of only 10 mm (see Figure 15) reference by an optical index mark. The distance signal was developed to offer the measurement of contact of every scanned point is correlated with the rotation forces inside the patient’s chest and to provide the sur- angle of the scan head and sent to a CAN bus interface. geon direct kinesthetic information. A small Steward With each revolution of the scan head, up to 400 points platform-based structure was chosen as the basis for a lying in the scan plane can be sensorized. The scan head newly designed force-torque sensor (see Figure 16). rotates with 25 revolutions per second, which guaran- This concept allows for the integration into surgical in- tees compatibility with external video-based sensor sys- struments near the instrument’s tip. Manipulation forces tems. With an additional referenced movement of the can be measured in all three dimensions and so can scanner, objects can be digitized and 3D world models torques. The sample rate of the instrument is approxi- computed. mately 800 Hz, and the resolution is approximately 9.5 bits, which is sufficient for high-quality kinesthetic feed- 4.3.4 DLR Robotic Hand. The second genera- back and force control in abdominal and cardiac sur- tion of the DLR robotic hand is a consistent and further gery. development, beginning with the primitive multisensory, two-finger gripper developed for ROTEX. The 4.3.6 Sensorized Forceps. A rigid instrument DLR Robotic Hand 2, as shown in Figure 14, has four limits the work space behind the trocar. Full dexterity

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 94 PRESENCE: VOLUME 13, NUMBER 1

diameter of 10 mm, and because the instrument is equipped with sensors close to the tip, real manipulation forces can be measured, allowing for a variety of additional functions.

5 Overview of Experimental Telepresence Applications at DLR

The following subsections present an overview of our telepresence activities in advanced space and surgery robotics. Although the constraints of both applications require distinguished mechatronical manipulators, the basic telepresence control concepts described before are Figure 15. The DLR sensorized scalpel concept. used within both of them.

5.1 Space Robotics

High-fidelity telepresence seems to be the only way to fulfill complex tasks in space because currently available autonomy and artificial intelligence is not able to perform such tasks in an unknown environment. The creation of complex models describing this unknown environment is too delicate to rely on, and the employ- ment of humans working in space conditions is costly, risky, and stressful for the humans involved. Telepres- ence is the solution to extend sophisticated human intelligence to hazardous space missions. In practice, there are different stages of immersion, or how perfectly the feeling of being present is achieved. Figure 16. The DLR sensorized forceps concept. The first approach is the stereovideo image display. Here, the control loop is closed via the eyes and the human brain. For more delicate tasks the feedback of cannot be achieved because additional DOF similar to tactile and kinesthetic information is needed to perform the human hand are missing. A pair of forceps with two the task. additional actuated degrees of freedom near the tool tip With ROTEX (RObot Technology EXperiment), the is currently being developed. This would enable the sur- first teleoperated robot in space, and ETS VII (Engi- geon to move the tool tip of the instrument in six DOF neering Test Satellite No. VII), we introduced and veri- inside the human body. Thus, the surgeon regains the fied an applicable combination of shared autonomy, full dexterity of open surgery and can therefore work shared control, and a powerful delay-compensating, 3D intuitively in a manner similar to open surgery. graphics simulation concept enabling simple telepres- The drives for the joints and the forceps themselves ence by providing a stereovideo image display and per- are realized as electromechanical actuators and are lo- forming teleoperated control of the space robot (Brun- cated outside the body. As stated, the forceps have a ner et al., 1994).

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 95

torque are fed back to the human operator on the ground. Using DLR’s newest force-feedback joystick, the operator supervises the robot in space by generating control commands depending on the selected control law. High-rate up- and downlink channels are used to involve the human operator directly in the control loop. In close cooperation with the Russian Babakin Space Center, the DLR is investigating the possibility of satellite-based robotic applications in space. The TECSAS (TEChnology SAtellite for demonstration and verification of Space systems) project investigates the feasibility of experimental satellite-based missions, with emphasis on robotics. Its aim is the on-orbit qualification of the Figure 17. The mockup of the ROKVISS experiment. key robotics elements, based upon telepresence-enabling technologies as qualified within ROKVISS, for advanced space servicing systems, especially docking and robot- Of special interest for future on-orbit servicing prob- based capturing procedures. It is planned to perform lems is the exploration of high fidelity or experiential sensitive operations, such as rendezvous and close ap- telepresence for direct human operator interaction in proach maneuvers (see Figure 18), which will be neces- space from the ground. With ROKVISS—Robotics sary for further servicing activities. The technology- Component Verification on the International Space Sta- based experiments are considered a step toward on-orbit tion (ISS)—an exploration mission for telepresence- servicing missions (for example, for orbits crowded with enabling technologies in space is under preparation (see satellites out of operation). Figure 17). ROKVISS, Germany’s new space robot project, is 5.2 Surgical Robotic Scenario scheduled to fly to the Russian part of the ISS in 2004. Its aim is the verification and qualification of the DLR’s In the realm of minimally invasive surgery robot- newest lightweight robot joint technologies. This new ics, we are still working on a scenario as shown in Figure generation of a ultralight, impedance-controllable robot 19. Using telepresence technology in surgery allows the is essential to enable high fidelity telepresence in ad- surgeon to operate as relaxed as possible through a vanced space robotics. Of special interest for future on- near-distance console. It should be possible to assign orbit servicing problems is the verification of the tele- the control of the surgical robot assistance between two presence control mode. An adaptation of the tele- surgeons in the case of a time-consuming intervention. presence service layer will be integrated into DLR’s Thereby, bimanual manipulation is achieved by two co- MARCO system to support and evaluate bilateral con- operating robot-based surgery assistants, both guided trol concepts. To verify the operability of the light- under shared control (as described previously), although weight robot, different kinds of telemanipulation tasks, different kinds of bilateral control modes are possible to such as peg-in-hole or contour following, are planned command the robots. (Preusche, Reintsema, Landzettel, & Hirzinger, 2003). A robot-based, autonomous laparoscopic guidance The ROKVISS mockup is built up by a small 2-joint system for manual MIS was developed at the DLR to robot, mounted on a Universal Workplate (UWP), and provide an optimal field of view for the surgeon (Wei, a mechanical contour device. A stereo camera, mounted Arbter, & Hirzinger, 1997). This robot holds the lapa- on the second joint, completes the mockup. Video im- roscope and aligns itself with the surgical instruments ages and measurements of the current robot joint and autonomously and is guided by a color marker, which is

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 96 PRESENCE: VOLUME 13, NUMBER 1

Figure 18. TEChnology SAtellite for demonstration and veriﬁcation of Space systems.

MIS TPTA system is based on the generic interconnection framework presented by Preusche and colleagues (2002). It represents an object framework divided into several servants that handle the haptic data stream. The integration of the video stream is under preparation.

6 Conclusion

Artificial intelligence for use in space and surgery is too delicate to rely upon. Humans working in space or surgery conditions is costly, risky, and stressful for the humans involved. High-fidelity telepresence is the solution to extend sophisticated human intelligence. A high-fidelity telepresence concept is able to overcome these drawbacks and barriers within the field of space and surgery robotics. But telepresence makes high demands to the technologies lying beneath the applica- Figure 19. The robotic surgery scenario at DLR. tion. In particular, the robot needs sensors comparable to the human senses to gather the remote environment, which has to be displayed to the human operator. The exploration and manipulation capabilities need to be placed near the tip of the instrument. The autonomous similar to human capabilities, and the communication laparoscopic guidance system completes our scenario of has to be a broadband communication with low delay to robot assistance in MIS and provides stereovideo images transport the sensor input and the operators (re)actions within our MIS-TPTA system. almost instantaneously; otherwise, the user’s sense of The system interconnectivity and control of our current immersion is degraded.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 Reintsema et al. 97

Acknowledgements national Conference on Intelligent Robots and Automation, 2, 1882–1888. This work was supported in part by the German Research Haidacher, S., Butterfaß, J., Fischer, M., Grebenstein, M., Foundation (DFG) within the Collaborative Research Center- Joehl, K., Kunze, K. et al. (2003). DLR Hand II: Hard- Sonderforschungsbereich SFB 453—on “High-Fidelity Tele- and software architecture for information processing. Pro- presence and Teleaction.” ceedings of the IEEE International Conference on Robotics and Automation, 684–689. Hannaford, B., & Ryu, J. (2002). Time domain passivity control of haptic interfaces. IEEE Transactions on Robotics and References Automation, 18 (1), 1–10. Hayati, S., & Venkataraman, S. (1989). Design and imple- Albu-Scha¨ffer, A., & Hirzinger, G. (2000). State feedback mentation of a robot control system with traded and shared controller for flexible joint robots: A globally stable ap- control capability. Proceedings of the IEEE International proach implemented on DLR’s light-weight robots. Pro- Conference on Robotics and Automation, 3, 1310–1315. ceedings of the IEEE/RSJ International Conference on Intel- Hirzinger, G., Albu-Scha¨ffer, A., Ha¨hnle, M., Schaefer, I., & ligent Robots and Systems, 4, 1087–1093. Sporer, N. (2001). On a new generation of torque con- ———. (2001). Parameter identification and passivity based trolled light-weight robots. Proceedings of the IEEE Interna- joint control for a 7 DOF torque controlled light-weight tional Conference on Robotics and Automation, 4, 3356– robot. Proceedings of the IEEE International Conference on 3363. Robotics and Automation, 3, 2852–2858. Hirzinger, G., Brunner, B., Dietrich, J., & Heindl, J. (1994, Bottazzi, S., Caselli, S., Reggiani, M., & Amoretti, M. (2002). May). ROTEX—The first remotely controlled robot in A software framework based on real-time CORBA for tele- space. Proceedings of the IEEE International Conference on robotic systems. Proceedings of the IEEE International Con- Robotics and Automation, 3, 2604–2611. ference on Intelligent Robots and Systems, 3, 3011–3017. Hirzinger, G., Butterfaß, J., Fischer, M., Grebenstein, M., Brunner, B., Arbter, K., & Hirzinger, G. (1994). Task di- Ha¨hnle, M., Liu, H., et al. (2000). A mechatronic approach rected programming of sensor based robots. Proceedings of to the design of light-weight arms and multifingered hands. the IEEE/RSJ International Conference on intelligent robots Proceedings of the IEEE International Conference on Robotics and systems, 2, 1080–1087. and Automation, 1, 46–54. Buss, M., & Schmidt, G. (1999). Control problems in multi- Jansen, E. W. (1998). Towards minimally invasive coronary modal telepresence systems. Advances in Control: Highlights artery bypass grafting. Utrecht: Brouwer Uithof. of European Control Conference (ECC ’99), 65–101, Lon- Jia, S., Hada, Y., Ye, G., & Takase, K. (2002). Distributed don: Springer-Verlag. telecare robotic systems using CORBA as a communication Conway, L., Volz, R., & Walker, M. (1987). Tele- architecture. Proceedings of the IEEE International Confer- autonomous systems: Methods and architectures for inter- ence on Intelligent Robots and Automation, 2, 2202–2208. mingling autonomous and telerobotic technology. Proceed- Kodera, K., Kiaii, B., Rayman, R., Novick, R., & Boyd, W. ings of the IEEE International Conference on Robotics and (2001). Closed chest CABG on the beating heart with a Automation, 2, 1121–1130. computer-enhanced articulating system: Case report. Heart Craig, J. (1989). Introduction to robotics: Mechanics and con- Surgery Forum, 4 (4), 305–306. trol. Reading, MA: Addison-Wesley. Konietschke, R. (2001). Aufbauoptimierung fu¨r Roboter in Draper, J. (1995). Teleoperators for advanced manufacturing: medizinischen Anwendungen. Munich University of Technol- Applications and human factors challenges. International ogy (Tech. Rep.). Munich University of Technology. Journal of Human Factors in Manufactoring, 5(1), 53–85. Lin, G., Unterschu¨tz, T., Vogel, V., Reintsema, D., & Emmerich, W. (2000). Software engineering and middleware: Koeppe, R. (2001). Telekommunikationsnetze in CORBA- A roadmap. In A. Finkelstein (Ed.), The future of software basierten Echtzeit-Systemen am Beispiel wriklichkeitsnaher engineering (pp. 117–129). Limerick, Ireland: ACM Press. Telepra¨senzanwendungen. In U. Killat & W. Lamersdorf Ghodoussi, M., Butner, S., & Wang, Y. (2002). Robotic sur- (Eds.), Kommunikation in verteilten systemen (kivs) (pp. gery—The transatlantic case. Proceedings of the IEEE Inter- 241–244). Hamburg: Springer Verlag.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021 98 PRESENCE: VOLUME 13, NUMBER 1

Object Management Group (OMG). (1997). CORBAservices: vanced space missions. Humanoids 2003—The Third IEEE Common object services specification. OMG Document 97- International Conference on Humanoid Robots, available on 12-02. CD-ROM. ———. (1998). The common object request broker: Architecture Ryu, J., Hannaford, B., Preusche, C., & Hirzinger, G. (2003). and specification. OMG Document 98-02-33. Time domain passivity control with reference energy behav- ———. (2000). Audio/video stream specification. OMG Doc- ior. Proceedings of the IEEE/RSJ International Conference ument 00-01-03. on Intelligent Robots and Systems, 3, 2932–2937. Ortmaier, T., Gro¨ger, M., & Hirzinger, G. (2002). Robust Sheridan, T. (1992). Telerobotics, automation, and human su- motion estimation in robotic surgery on the beating heart. pervisory control. Cambridge: The MIT Press. Proceedings of Computer Assisted Radiology and Surgery— Spong, M. (1987). Modeling and control of elastic joint ro- Cars, 206–211. bots. IEEE Transaction on Robotics and Automation, RA-3 Preusche, C., Hirzinger, G., Ryu, J., & Hannaford, B. (2003). (4), 291–300. Time domain passivity control for six degrees of freedom Wei, G.-Q., Arbter, K., & Hirzinger, G. (1997). Real-time haptic displays. Proceedings of the IEEE/RSJ International visual servoing for laparoscopic surgery. IEEE Engineering Conference on Intelligent Robots and Systems, 3, 2944–2949. in Medicine and Biology, 16(1), 40–45. Preusche, C., Hoogen, J., Reintsema, D., Schmidt, G., & Hir- Yokokohji, Y., & Yoshikawa, T. (1990). Bilateral control of zinger, G. (2002). Flexible multimodal telepresent assembly master-slave manipulators for ideal kinesthetic coupling. using a generic interconnection framework. Proceedings of Proceedings of the IEEE/RSJ International Conference on the IEEE International Conference on Robotics and Automa- Intelligent Robots and Systems, 355–362. tion, 4, 3712–3718. ———. (1992). Bilateral control of master-slave manipulators Preusche, C., Ortmaier, T., & Hirzinger, G. (2001). Teleop- for ideal kinesthetic coupling. Proceedings of the IEEE Inter- eration concepts in minimal invasive surgery. Proceedings of national Conference on Robotics and Automation, 1, 849– 1. IFAC Conference on Telematics Application in Automa- 858. tion and Robotics, 409–414. Yoshikawa, T. (1993). Dynamics shaping in robot force con- Preusche, C., Reintsema, D., Landzettel, K., & Hirzinger, G. trol and artificial reality. Proceedings of the 6th International (2003). ROKVISS—Towards telepresence control in ad- Conference on Advanced Robotics, 3–8.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/105474604774048243 by guest on 26 September 2021