Robotic Middlewares to the Physical Agent System

Layered Implementation of Robotic Middlewares to the Physical Agent System

IROS’06 Workshop WS4 Robotics Standardization Makoto MIZUKAWA, Ph.D. [email protected] Shibaura Institute of Technology, Tokyo Chair ORiN Forum/JARA, Project Leader ISO TC184/SC2 RAPI Contact of OMG Robotics-DTF to ISO TC184/SC2 Oct. 10, 2006, International Conference Beijing, China on Iintelligent Robots and Systems 1 Outline

Needs for standard Environment

Physical Agent System

Layered Implementation of Middlewares

ORiN for Robot Collaboration

LwRTC for Embedded controller

Future work

2 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Needs for standard Environment

3 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Robotics in future

Hazardous task Unmanned Autonomy substitute in fields systems

Manufacturing High speed Human Industrial robots work /precision Human-Robot-Interaction

Embedded/ Service robots Ubiquitous robots symbiosis Entertainment robots HouseholdHousehold Pet/Pet/ healinghealing WorkWork assistassist DailyDaily assistassist New media Communication media RescueRescue MedicalMedical 4 Oct.Daily 10, 2006, lifeBeijing, support China International Conference on Intelligent Robots and Systems I’REX2005 DAEJEON

5 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Related Activities (examples) forfor TelecomTelecom forfor MfgMfg FIPAFIPA:The:TheFoundations Foundationsfor for IntellIntelligentigent ISOISO TC-1TC-18484,, JARAJARA PhysicalPhysical AgentsAgents RobRoboots,ts, DeviceDevice proproffiling.iling. middleware for application http://www.fipa.org/ middleware for application http://http://www.www.oorrin.jp/in.jp/ forfor IT+RTIT+RT servicesservices forfor controllers/OScontrollers/OS MPHPT:Ministry of Public Management, Home Affairs, Posts OROCOSOROCOS and Telecommunications, Japan SoftwSoftwaaresresfforor Robot controller http://Robwww.ot controrocoosller.org/ http://www.soumu.go.jp/english/index.html

MIC: Ministry of Information and OMGOMG Communication, KOREA SDOSDO DAIS www.vcl.uh.edu/~rcv03/materials/slides/SangRok.ppt DAIS 6 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Simple Idea

Robot is the object that has physical entity which interacts with the real world Collaboration with others through networking From service-applications to devices Tasks: Computers in the logical world vs. Robots in the real world Sensing and processing Interact with the real world Hard real-time issues in controllers Various structure and configuration Various architecture Physical & logical interoperability with other devices and robots 7 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Issues

Interoperability

Connectivity DomainDomain specificspecific DomainDomain independentindependent Modularity

Reusability

RT-middleware for robot services Keys:Keys: RT-component-service middleware for controllers

RT-component standard interface 8 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems RT middleware-layers PART1 PART2 PART3

controller application application service level common API mediator Model, data-framework RTMW1 Robot RTMW2 Internal task level profiles NW Model, data-framework Vision Ctrl. Sensor Ctrl. Common controller Dev.Profile Dev.Profile motion level I/F Model, data-framework controller Body Ctrl. Arm Ctrl.

Robot A Robot B Dev.Profile Dev.Profile

motors arm

camera devices sensors

9 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Physical Agent System

10 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Motivation From control issues to service issues

Daily activity support and collaboration using remotely controlled robot •Operation based on information mainly through video cameras mounted on the remotely controlled robot •Gesture transmission to assist collaboration between remote sites (laser pointer/arm) Media to support Master-slave system equivalent/close

relationship 11 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems PAS: Physical Agent System

A remotely controlled autonomous robot that acts as an AGENT to support human collaboration over distant places PAS as media for

Sharing spatial information between remote sites

Transmitting intension/attention of operator via motion

direction/orderdirection/order

spatial/field info. spatial/field info. 12 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems PPhysicalhysical AAgentgent SSystemystem InputInput devices devices RemotRemotee site Voice control Daily life site PC support

Joystick

Agent robot Agent with Agent robot Agent with The Internet Info. Sharing Pad physicalphysical camera interactioninteraction Laser pointer •Distributed•Distributed control control PC arm •System•System description description •Image•Image processing processing Activity support Sensory Glove in hazardous environments autonomy WEB Simulator ・・mobilitymobility ・・obstacleobstacle avoi avoidancedance ・・postureposture stabili stabilizationzation camera,camera, LP LP 13 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Applications of PAS

Disaster/Rescue Daily life support

Elderly/Handicapped

Communication within Family in distance

Home-electronics・personal robots

Distributed/Embedded PAS in Environments Reflects human intension Device profiling using XML to allow interoperability and flexible configuration

14 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Prototype of Laser Pointer the PAR04R CCD Camera Gimbal Mechanism

5DOF Arm

Crawler

15 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems PAS System Configuration

< device > < device > < device > < device > < device > Display Keyboard Sensory Glove Joystick Mouse

< processor > < on board> < processor > 「TCP/IP」･wireless LAN < on board> ･ Agent-Server ･PCI Client-Server input device (windows2000) ･IP7000 (windows2000)

< management > < management > ･ Preservation of task and re-execution command ･ Simulator system Agent Robot ･ Communication with Client-Server(TCP/IP) Operation ･ Input control ･ Communication with Data-Server(TCP/IP) ･ Display the interface to an operator ･ Communication control of CAN bus CAN ･ Process of camera image (Controller Area Network)「CAN」

< device > < device > < device > < device > H8S for Power management H8S for Arm control H8S for Crawler control H8S for Camera/LP control Real-time OS Real-time OS Real-time OS Real-time OS

< on board> < on board> < on board> < on board> ･ H8S ･ Arm ･ Machine ･ PSD ･ Camera ･ H8S ･ Ultrasonic Sensor ･ LP ･ H8S ･ H8S

< management > < management > < management > < management > ･Task control ･Arm control ･Machine control ･Camera/LP control ･Get Current’s value ･Command in another device ･Get Sensor’s value ･Get Angle’s value ･Store Current/Voltage logs ･Command in another device ･Command in another device ･Command in another device

16 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Physical Agent Robot (PAS) PAR04R

17 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Aichi Expo 2005 Robot week, June 9-19

Oct. 10, 2006, International Conference Beijing, China on Iintelligent Robots and Systems 18 Robot Week (Jun 9-19) Aichi Expo

Intelligent Wheel Echono-vehicle Chair, “Komawari-kun” Meijo Univ Apri-Alpha & the helper robot Apri-Attenda UEC Toshiba & SUT PAR04R Shibaura Inst. Tech.

Collaboration using RT middleware-ORiN Component based on RT Middleware from AIST19 Oct. 10, 2006, Beijing, China InternationalApriAlpha Conference on using Intelligent ORCA Robots and Systems Layered Implementation of Middlewares

ORiN for Robot Collaboration

LwRTC for Embedded controller

20 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems RT middleware-layers PART1 PART2 PART3

Robot A Robot B Dev.Profile Dev.Profile

motors arm

camera devices sensors

21 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Toshiba robot

Voice operation operator

projector visitors Robot profiles XML(CRD)

robot control & management system Robot status management robot status storage & display system monitor and refer robot status and location robot control & status acquisition using RAC

cameras

UEC robots Meijo Univ robots PAR04R 22 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems System integration using ORiN

control data

robot control & management system GUI Robot control application operation VB.NET command operator table

CAO Engine

CRD provider RAC provider

Robot profiles RAC requests XML(CRD)

Voice operation operator 23 Oct. 10, 2006, Beijing, China International Conference on IntelligentRobots Robots and Systems ORiN: Open Resource/Robot Interface for the Network

Application

ORiN Provider

Robot Controller

24 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems ORiN structure

Application App. App. App. OPC UPnP DDE Non-CAO Application (ex. OPC Client, UPnP Control Point) X Y Z

Application IF Abstract Dev. Engine Device B CAO Engine Device IF

Provider A Co. B Co. CAP CRD Other Standards (ex. OPC Server)

SOAP Proprietary Protocols

Internet Device CRD File (XML) Non-ORiN Device

25 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Key Technologies of ORiN （Open Robot Interface Model） ¾ Controller Access Object, CAO 9 A middleware providing a standard program interface for the applications and the devices. 9 It’s based on the distributed object technology, DCOM or CORBA.

¾ Controller Access Protocol, CAP 9 A standard communication protocol for the Internet to allow data-exchange over firewall. 9 It’s based on the SOAP technology.

¾ Controller Resource Definition language, CRD 9 A standard data schema for representing the data in a device. 9 It’s based on the eXtensible Markup Language, XML-Schema. 26 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems RAPIRAPI

TheThe RobotRobot InformationInformation FrameworkFramework andand ApplicationApplication ProgramProgram InterfaceInterface

ISO TC184/SC2 New Work Item

27 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Objective of RAPI

to provide standard methods for accessing robot/device system information when developing a multi-vendor Mfg. system •a unified access method •a common application platform •a common API •a unified robot/device profile description •networking capability using distributed object

28 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Relationship between ORiN and RAPI

ORiNORiN: :PC PC technologytechnology ofof thethe statesstates ofof artart

•Implementation•Implementation dependingdepending onon MicrosoftMicrosoft TechnologyTechnology butbut •• Available Available andand Ready-to-applyReady-to-apply technologytechnology

RAPI:RAPI:TheThe R Robotobot Information Information Framework Framework and and A Applicationpplication P Programrogram I nterfaceInterface IndependentIndependent fromfrom softwaresoftware productsproducts fromfrom thethe specificspecific vendorsvendors •• AbstractionAbstraction ofof ORiNORiN •• SubsetSubset ofof ORiNORiN inin functionalityfunctionality LimitLimit toto functionsfunctions thethe leastleast requirementrequirement ExcludeExclude RAPRAP 29 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Merits of RAPI

End Users Software Vendors Cost effective operation Common Packages for Industries • Multi-vendor Mfg. system to meet • Cost reduction by standardizing user needs. software package. • Mfg. Monitoring and Remote • New robot software market is Maintenance is easier. generated.

RAPIRAPI StandardsStandards CommonCommon applicationapplication platformplatform andand APIAPI IndependentIndependent fromfrom thethe ApplicationsApplications andand thethe RobotsRobots System Integrators Robot Makers Easy customizing of FA services Expand the Robot Application Field • Reduce development cost by • Reduce development cost for using a common platform. Robot communication Interface. • Independent from the robot • New robot market for various makers. types of application is generated. 30 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems ISO TC184/SC2 Schedule 2006 2007 2008 2009 3rd Q 4th Q 1st Q 2nd Q 3rd Q 4th Q 1st Q 2nd Q 3rd Q 4th Q 1st Q 2nd Q Creating the 1st draft Updating Finalizing 6/15 ISO meeting in Paris RAPI 1st revision Commit the NWIP RAPI draft by the ORiN forum Start the discussion by the PT

Objective: To create a standard framework for Robot communications. Term of Activity: June 2006 to June 2009, 3 Years

Project Leader: Dr. Makoto MIZUKAWA, Shibaura Institute of Technology, Chair of ORiN Forum. Project Sub-leader: Satoshi SAKAKIBARA, DENSO WAVE INC. Project Secretariat: Japan Robot Association

31 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems ORiN Forum

Open Resource Interface for the Network / Open Robot Interface for the Network

ORiN forum The Japan Robot Association http://www.orin.jp/

32 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems LwRTC for Embedded controller

33 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems RT middleware-layers PART1 PART2 PART3

Robot A Robot B Dev.Profile Dev.Profile

motors arm

camera devices sensors

34 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems RTC-CAN System Overview Construction of Module based Robot with Embedded MPU, CAN, and RT-Middleware Æ RTC-CAN System RTC-CAN Implementation of LwRTC MPU (enough to work CORBA) Ethernet (CORBA) OS Supporting Function Proxy Proxy Proxy Proxy RTC RTC RTC RTC

Intercommunication with Ethernet Adapting CAN standard RTC RTC-CAN Gateway

Direct communication RTC-CAN Proxy software between CAN devices CAN

CAN device CAN device CAN device CAN device CAN-RTC CAN-RTC CAN-RTC CAN-RTC

35 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems CAN message identifier（11bit） at RTC-CAN system

Proxy-RTC CAN Æ Proxy Proxy-RTC Proxy-RTC Command Send Proxy-RTC ID: 010-0001-1111 InPort RTC RTC OutPort Proxy Æ CAN Command Send ID: 010-1111-0000

Proxy Æ CAN CAN Æ Proxy Proxy Æ CAN CAN Æ Proxy Data Communication Data Reply Command Reply Data Communication ID: 001-0000-0001 ID: 011-0000-1111 ID: 011-1111-0001 ID: 001-0000-0001

CAN Æ CAN CAN-RTC ID:0000 Command Send CAN-RTC ID:0001 CAN-RTC ID:0000 ID: 010-0000-0001 CAN-RTC ID:0001

0000-0001 InPort RTC RTC OutPort 0001-0001 ID:0001 ID:0000 ID:0000 ID:0001 CAN Æ CAN Subscribing Table Command Reply ID: 011-0001-0000

CAN Æ CAN Data Communication ID: 001-0001-0001 36 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Experiment of RTC-CAN System (Crawler Control)

Confirming the operation of CAN-RTC

Confirming the operation of Crawler

MPU (enough to work CORBA) Client PC SH4 OS Crawler Dummy Proxy-RTC Proxy-RTC

Ethernet RTC-CAN Gateway H8s RTC-CAN Proxy software

CAN

Crawler device Dummy device H8s Crawler Dummy CAN-RTC CAN-RTC

37 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Experiment of RTC-CAN System (Crawler Control)

The operation of CAN-RTC Working crawler-subsystem as RTC The operation of Crawler Confirming Gateway Function Direct communication between CAN devices

MPU (enough to work CORBA) Client PC SH4 OS Crawler Dummy Proxy-RTC Proxy-RTC

Ethernet RTC-CAN Gateway H8s RTC-CAN Proxy software

CAN

Crawler device Dummy device H8s

Crawler Dummy CAN-RTC CAN-RTC 38 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems MPU (enough to work CORBA) Ethernet (CORBA) OS Results Proxy Proxy Proxy Proxy RTC RTC RTC RTC

Ethernet RTC-CAN Gateway Communication Time Dilley RTC-CAN Proxy software 20msec (one way) It’s too late to control robot. CAN It is a cause that the processing of Gateway MPU reached the limit. CAN device CAN device CAN device CAN device CAN-RTC CAN-RTC CAN-RTC CAN-RTC

Specification

H8S/2638F（HD64F2638WF20) MPU ROM：256KB RAM：16KB

clock 20MHz 512KB（256K×16bit ） SRAM R1RW0416DSB-2LR（Renesas Technology）

39 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems Future work: Middleware services for RT Structured Ubiquitous services ORB services and intelligent Sensor network relations, functions, environment RFID tags, data mining processing Physical and interfaces, profiles informational repository, naming consistency services

software distributed objects, encapsulation services service, task, internal services motion embedded services Distributed processes, objects, processors , tag middleware devices, components, models, PIM, PSM controllers description, languages, ORiN Control software, wiring, system tools CAMUS PnP 40 Oct. 10, 2006, Beijing, China… International Conference on Intelligent Robots and Systems summary

Propose of PAS for daily-life services Demonstration in Aichi Expo. 3-layer model for the RT middleware design and implementation for seamless RT system integration Coordination with environment with embedded RT components/devices MDA for Robotics

41 Oct. 10, 2006, Beijing, China International Conference on Intelligent Robots and Systems