Building Ubiquitous Robot Systems
M. Broxvall, A. Saffiotti Oktober 29th, San Diego
Supported by: Electronics and Telecommunications Research Institute, Korea The Swedish Research Council (Vetenskapsrådet), Sweden Knowledge Foundation, Sweden
Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology
– Demonstrator environments – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues Ubiquitous Robot Systems
● Classical robotics: robot and environment as two distinct entities
– Distribute computational capability and intelligence in the environment
● Ubiquitous Robotics and the Ecologies of PEIS1 Project
– Combine ambient intelligence with AI and classical robotics.
Artificial Intelligence Autonomous Ambient Robotics Intelligence
Ubiquitous Robotics Computing
Sensor Networks 1PEIS is pronounced [pace]. Related Fields
Ambient Autonomous Artificial Intelligence Intelligence Robotics
Ubiquitous Robotics Computing
Sensor Networks
Related Fields
Ambient Autonomous Artificial Intelligence Intelligence Robotics
Ubiquitous Robotics Computing
Aims at the creation of a universal robot, but this is beyond our current capabilities
A PEIS Ecology exploits the interaction of many connected specialized devices to perSfoernms oar t aNsektworks
Related Fields
Ambient ConAnuetcto nwiodemo aurras ys of Artificial Intelligence Intelligence small devicesRobot forics remote sensing, but they do not deal with cognition and (complex) actuation
A PEIS Ecology can perform tasks that require Ubiquitous Robotics cognition and physical Computing action
Sensor Networks
Related Fields
Ambient Autonomous Artificial Intelligence Intelligence Robotics
Ubiquitous Robotics Computing
It is mainly an information technology, but it will not help us to perform full physical tasks
A PEIS Ecology will bring physical action and interaction into the user’s environment Sensor Networks
Scenario
Johanna is 72 years old. She lives in a small house.
Before she wakes up, her fridge realizes that something is smelling bad. Pippi, a robot carrying a capable artificial nose, goes to the fridge and performs a careful inspection. The milk is found to be bad, and the autonomous thrashcan is called to fetch it.
Soon after, the coffee machine turns on. Johanna’s personal robot, Emil, brings her a cup of coffee. It also tells her about the milk, and asks if she would like the trolley to go out and buy a new bottle.
Keypoints in scenario
● Many specialized robotic devices – Autonomous trolley, cleaning robot, monitoring cameras – Highly heterogeneous
● Devices communicate and cooperate – Functionality and intelligence emerge from cooperation
● Assistive task performed by the whole house – As opposed to building one single “super-robot”
What is ubiquitous robot systems?
● Sensing and Acting (robotic) components distributed ubiquitously in an intelligent living environment
– Provide users the right services at anytime and anywhere
● Illustrated by the Ecologies of PEIS project
– A ubiquitous robot system consisting of a number of Physically Embedded Intelligent Systems – OpenSource middleware and components aiding the development of ubiquitous robot systems
● Additional requirements such as distributed, decentralized intelligence and scalability of devices. Not required for all ubiquitous robotic systems
Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology
– Demonstrator environments – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues The PEIS Ecology approach
● Uniform notion of “robot”
– PEIS = Physically Embedded Intelligent System
● Uniform communication model PEIS – Distributed tuple space
● Uniform cooperation model
– Lending functionalities to one another
● PEIS Ecology
– A system consisting of a set of cooperating PEIS
PEIS: Physically Embedded Intelligent System
● PEIS Component Input Output – Software module
– Implements a functionality Sensor Actuator – Has input/output ports Environment – Can have sensors/actuators
● PEIS – A set of PEIS-Components M D – Includes sensing and/or actuation – Resides in one physical entity P C
Environment PEIS Ecology
● A collection of PEIS – that are embedded in the same environment – that can borrow capabilities from each-other
Eg. A camera tracking system Eg. An autonomous vacuum cleaner Eg. A camera tracking system localizing an autonomous vacuum cleaner
EnvironmentEnvironment Environment
A Simple PEIS Ecology
● An autonomous vacuum cleaner ● An overhead monitoring system ● An unknown box
Where am I?
Can I push you away?
A Simple Ecology
● An autonomous vacuum cleaner ● An overhead monitoring system
● An unknown box weight The parcel The vacuum cleaner The tracking system M D M D M D P C
P C P C location
Environment
The PEIS home
● Testbed environment built-up to test concepts and implementations. ● Implemented hardware and software components – Refrigerator – Cameras – Mobile robots – RFID-tagged floor – Electronic noses – Actuated window blinds, lamps
– plants ... PEIS Ecology Example
● Example run in an implemented PEIS Ecology
Benefits of a PEIS Ecology
● Technical – by-pass hard problems of todays robots – perception & action replaced by direct communication – robustness from redundancy
The parcel shape = {...... }
M D The vacuum cleaner The tracking system M D M D P C
P C P C
Environment
Benefits of a PEIS Ecology
● Technical – by-pass hard problems of today robots – perception & action replaced by direct communication – robustness from redundancy
● Pragmatical – can be easily customized – can be built and upgraded incrementally – easier to accept and afford than a full powerful robot
⇒ Smooth path to introduce robotic technologies into everyday life
Requirements on a PEIS Ecology
● Hardware and software
– Sensors and actuators distributed in environment – Processors running software components
● Mechanism implementing PEIS-components
– PEIS to PEIS communication – Component to component communication
● Collaborations
– Sharing data, requesting functionalities
– Creating configurations – Providing semantic information Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology 1. Heterogeneity – Demonstrator environments 2. Link physical and digital world 3. Self-configuration – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues Heterogeneity
Where am I?
Can I push you away?
● Very different devices and functionalities PEIS
● 3G Different platforms n a Z L i W g b – IR e processing hardware / software e R IR F ID – communication protocol / media ● Dynamic world PEIS can join and leave the ecology at any time
⇒ need middleware for integration in a common ecology The PEIS Middleware
● Provides uniform communication layer – Hides platform and network heterogeneity
● Manages all interactions – All PEIS components communicate via the middleware
● Implemented in the “PEIS Kernel” – All PEIS components must use it Component processes
PEIS Kernel
OS
CPU
PEIS Middleware: Example
Where am I?
Can I push you away?
PEIS Middleware: Example
The Parcel M D The Monitoring System The Vacuum Cleaner P C M D M D
P C P C
Environment
PEIS Middleware: Example
Distributed Tuple-Space
The Parcel M D The Monitoring System The Vacuum Cleaner P C M D M D
P C P C
Environm ent PEIS Middleware
● The PEIS Kernel – Communication model: shared tuple-space + events – Distributed over ad-hoc P2P network – Run-time C library (open source) nts e
n PTL o
Meta layer P mp Vision TC EI o Semantic discovery C S
Nose Player
S Configurator M P EI i d EI P Tuplespace layer
Distribution d S l
Topology e
K Subscriptions w e a r r
Communication layer n e e
Network communication l P2P Discovery
Hardware & OS
PEIS Middleware: Communication model
● Through the exchange of tuples < PEIS-ID, Component-ID, Key, Value(s) >
● Tuples live in a distributed tuple-space – Shared by all PEIS components – Physically distributed across PEIS –
● Communication triggered by events – PEIS subscribe to tuples by content – Subscribers notified when a matching tuple is inserted/updated
PEIS Middleware: Communication model
Subscribe: *.*.roomba-at = *
Insert: cam1.track.roomba-at = (3,5)
Notify: cam1.track.roomba-at = (3,5) ......
Insert: cam2.track.roomba-at = (1,-2)
Notify: cam2.track.roomba-at = (1,-2)
Time PEIS Middleware
Meta-level components ● EIS The P• PEIS initKernelialization – Co•m semanmunictica advertion tismemenodelts: sandha rdisedco tvuepryle-space + events Hardware• c/ onOfSigur poarttoarsbility Linu–x (rDobotis• .ts..r andibu tPCed) over ad-hoc P2P network Mac–OS (home PC) OpenRRun-time C library (open source) nts
TinyOSe
n PTL
... o
Meta layer P mp Vision TC EI o Peis-Init Semantic discovery C S
Nose Player
S Configurator M P EI i d EI P Tuplespace layer
Distribution d S l
Topology e
K Subscriptions w e a r r
Communication layer n e e
Network communication l P2P Discovery
Hardware & OS Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology 1. Heterogeneity – Demonstrator environment 2. Link physical and digital world 3. Self-configuration – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues Link the digital and physical world
● A “classical” robot – can read / modify the properties of a physical object by perception by manipulation
weight = 30
● A PEIS in a PEIS Ecology – can access another PEIS object by physical perception / manipulation by digital communication
⇒ how can we combine these two ?
Link the digital and physical world
● Did you ever experience this problem ?
I am a blond guy with a digital information Yes, I see you! pink shirt.
on ati form l in ica ys ph
Anchoring
● Establish the association between – Symbolic information ... – and perceptual data .... Symbolic system – through a common space
?
if lightweight(box-22) then PUSH(box-22) Perceptual system shape(box-22, box) color(box-22, green)
region-31.color = {0.0, 0.8, 0.2) region-31.pos = (24, 159)
Anchoring
myshape(box) mycolor(green) • Establish the association between myweight(600g) – Symbolic information ...
– and perceptual data .... Symbolic system Symbolic system – through a common space – across different PEIS !
if lightweight(box-22) then PUSH(box-22) Perceptual system shape(box-22, box) color(box-22, green)
region-31.color = {0.0, 0.8, 0.2) region-31.pos = (24, 159)
Anchoring
● An experimental tool – limited to position information – discretized (grid) representation of fusion space – fusion done by fuzzy logic
A simple experiment
2. Emil receives the task to take a photo of Malin 3. Emil decides to use Pippi to execute the task Emil 4. Emil generates a plan and sends the first action to Pippi: “GoTo Bed” 5. Pippi uses the overhead cameras to localize Malin 6. Pippi finds an unexpected box blocking the door 7. Pippi asks the box about its weight Pippi 8. The box replies: 500 gr 9. Pippi decides to push the box out of the way 10. Pippi signals completion of “goTo Bed” The ceiling cameras 11. Emil sends the next action: “Take Photo” 12. ...
The box A simple experiment: Configuration
Camera system position M D 2. Emil receives the task to take a photo of Malin P C 3. Emil decides to use Pippi to execute the task 4. Emil generates a plan and sends the first action to Pippi: “GoTo Bed” Emil Pippi 5. Pippi uses the overhead cameras to goals localize M D M D Box 6. Pippi finds an unexpected box blocking weight the door P C M D P C 7. Pippi asks the box about its weight 8. The box replies: 500 gr P C 9. Pippi decides to push the box out of the way 10. Pippi signals completion of “GoTo Bed” 11. Emil sends the next action: “Take Photo” 12. ... Environment
A simple experiment: Messages
2. Emil receives the task to take a photo of Malin The ceiling cameras 3. Emil decides to use Pippi to execute the Photo(malin) task 4. Emil generates a plan and sends the first action to Pippi: “GoTo Bed”
6. Pippi finds an unexpected box blocking
The box A simple experiment: Movie
2. Emil receives the task to take a photo of Malin 3. Emil decides to use Pippi to execute the task 4. Emil generates a plan and sends the first action to Pippi: “GoTo Bed” 5. Pippi uses the overhead cameras to localize 6. Pippi finds an unexpected box blocking the door 7. Pippi asks the box about its weight 8. The box replies: 500 gr 9. Pippi decides to push the box out of the way 10. Pippi signals completion of “GoTo Bed” 11. Emil sends the next action: “Take Photo” 12. ...
Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology 1. Heterogeneity – Demonstrator environment 2. Link physical and digital world 3. Self-configuration – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues Collaboration
● Which exchange of functionalities/services to perform?
– The configuration problem – Similar problem faced in eg. the semantic web community. – Major differences in problem
● Functionalities – Context and side effects – Not just “one shot” function calls ● Environment – Stochastic outcomes of actions – Weak world model (how can we model eg. humans?) ● Other – Computational and bandwidth con straints – Reuse existing robotic technologies Configurations - outline
Configuration of PEIS-Ecologies
● The configuration problem
● Plan-based generation of configurations
– Stand-alone – Integrated with PEIS-ecology
● Reactive generation of configurations
– Integrated with PEIS-ecology
Configuration – the problem
• The self-configuration problem – given the current context = { ecology, environment, goal } – generate a configuration for it – change in context gives change in configuration
Pippi tracker Go al cam-1
automatic door cam-2
Configuration – the problem
● The self-configuration problem
– given the current context = { ecology, environment, goal } – generate a configuration for it – change in context gives change in configuration Pippi
Go al
automatic door RFID-floor
Configurations - example
● Task
– Move to entrance – Pick up news paper – Move to bedroom
● Using
– Humanoid ceiling tracker in living room/hallway – No cameras in bedroom -> tracker not usable
Configurations
● Look at single-robot task execution – Plan-based approach – Reactive approach – Hybrid approach
● Derive approaches to PEIS Ecology task execution – Plan-based approach – Reactive approach – Hybrid approach
Configurations
Single robot execution: the “sense-plan-act” approach
Task
World model Planner
Plan
Controller
Environment
Configurations
Single robot execution: the reactive approach
Arbitration strategy
World status Arbiter
Behaviors
Environment
Configurations
Single robot execution: the hybrid approach
Task
World model Planner
Behavioral plan
World status Arbiter
Behaviors
Environment
Configurations
PEIS Ecology: the “sense-plan-act” approach
Task
Configuration Ecology model Configuration planner
Ontology DB Configuration DB Peis Middleware
Network / CPU’s / OS’s
Environment
Configurations
PEIS Ecology: the reactive approach
Configuration constraints
Configuration Ecology status maintainance
Ontology DB Configuration DB Peis Middleware
Network / CPU’s / OS’s
Environment
Configurations
PEIS Ecology: the reactive approach
Task
Configuration Configuration Ecology model planner constraints
Configuration Ecology status maintainance
Ontology DB Configuration DB Peis Middleware
Network / CPU’s / OS’s
Environ ment Configuration plan based approach
● Plan-based generation of configurations
– Generate ecology description
– Generate a suitable configuration
– Deploy the configuration Search strategy
component Configuration descriptions planner
deployer Go al
Plan-based configuration
● Plannning along two dimensions – Sequence of actions to achieve goal (causal flow) – Configuration to implement each action (data flow)
Ecology Task description introspection
fail plan action Action Configuration configuration Sequencer PEIS- planner planner Ecology
success/fail
Plan-based configuration
● Plannning along two dimensions – Sequence of actions to achieve goal (causal flow) – Configuration to implement each action (data flow)
(modified) introspection SHOP Planner Ecology Task description
fail
plan action configuration Action Configuration Sequencer PEIS- planner planner Ecology
success/fail PTL Plan
Self-configuration: Experiment
1. The milk has gone bad in the fridge. 2. Gas sensors in the fridge notices that something smells bad. 3. The Home Security Monitor decides to The fridge perform a closer inspection. The home security monitor 4. Pippi is sent to the fridge, carrying a sophisticated electronic nose. 5. Pippi is helped in its navigation by the ceiling cameras. 6. The Fridge opens its door. 7. Pippi docks to the fridge to start the Pippi smell operation. The ceiling cameras 8. Pippi is told the type of objects by the RFID tag reader in the fridge. 9. ...
RFID tag Self-configuration: Experiment
1. The milk has gone bad in the fridge. 2. Gas sensors in the fridge notices that something smells bad. 3. The Home Security Monitor decides to perform a closer inspection. 4. Pippi is sent to the fridge, carrying a sophisticated electronic nose. 5. Pippi is helped in its navigation by the ceiling cameras. HSM
6. The Fridge opens its door. deliberator tracker 7. Pippi docks to the fridge to start the smell operation. nose Pippi 8. Pippi is told the type of objects by the door nose vision TC RFID tag reader in the fridge. 9. ... RFID camera player
Fridge Self-configuration: Experiment
Move to kitchen Open fridge door
Smell fridge Move near fridge
Self-configuration: Experiment
Home monitoring system Pippi Refrigerator Global pos 4Ceiling Images Localization of Pippi ThinkingCap web system Move to kitchen Open Door cameras
Pippi
e-nose Odor classifier Food status Deliberator Pippi Odor Pos + orient Image of Pippi wrt Fridge ThinkingCap CCaammeerara Object tracker Home monitoring system Move near fridge Refrigerator
RFID- reader RFID- Reading Self-configuration: Experiment
Home monitoring system Pippi Refrigerator Global pos 4Ceiling Images Localization of Pippi ThinkingCap web system Move to kitchen Open Door cameras
Pippi
Food status e-nose Odor classifier Deliberator Pippi Odor Pos + orient Image of Pippi wrt Fridge ThinkingCap CCaammeerara Object tracker Home monitoring system Move near fridge Refrigerator
RFID- reader RFID- Reading Self-configuration: Experiment
Home monitoring system Pippi Refrigerator Global pos 4Ceiling Images Localization of Pippi ThinkingCap web system Move to kitchen Open Door cameras
Pippi
e-nose Odor classifier Food status Deliberator Pippi Odor Pos + orient Image of Pippi wrt Fridge ThinkingCap CCaammeerara Object tracker Home monitoring system Move near fridge Refrigerator
RFID- reader RFID- Reading Self-configuration: Experiment
Home monitoring system Pippi Refrigerator Global pos 4Ceiling Images Localization of Pippi ThinkingCap web system Move to kitchen Open Door cameras
Pippi
Food status e-nose Odor classifier Deliberator Pippi Odor Pos + orient Image of Pippi wrt Fridge ThinkingCap CCaammeerara Object tracker Home monitoring system Move near fridge Refrigerator RFID- reader RFID- Reading Self-configuration: Experiment
Home monitoring system Pippi Refrigerator Global pos 4Ceiling Images Localization of Pippi ThinkingCap web system Move to kitchen Open Door cameras
Pippi
Food status e-nose Odor classifier Deliberator Pippi Odor Pos + orient Image of Pippi wrt Fridge ThinkingCap CCaammeerara Object tracker Home monitoring system Move near fridge Refrigerator RFID- reader RFID- Reading Self-configuration: Experiment
Home monitoring system Pippi Refrigerator Global pos 4Ceiling Images Localization of Pippi ThinkingCap web system Move to kitchen Open Door cameras
Pippi
e-nose Odor classifier Food status Deliberator Pippi Odor Pos + orient Image of Pippi wrt Fridge ThinkingCap CCaammeerara Object tracker Home monitoring system Move near fridge Refrigerator RFID- reader RFID- Reading Plan-based configurations summary
1. Plan-based generation of configurations – Previously: component descriptions are hand-coded – New step: acquire these descriptions by introspection
component Configuration descriptions planner
deployer Go al
Configurations - outline
Configuration of PEIS-Ecologies
● The configuration problem
● Plan-based generation of configurations
– Stand-alone – Integrated with PEIS-ecology
● Reactive generation of configurations
– Integrated with PEIS-ecology
Configuration: reactive approach
Reactive generation of configurations – Use reactive agent to instantiate a local configuration
Templates
Relevant Configuration advertisements agent
deployer
Go al
Configuration: reactive approach
Reactive generation of configurations – Use reactive agent to instantiate a local configuration
Relevant Configuration Relevant Configuration advertisements agent advertisements agent
deployer deployer Go al
Configuration
Reactive generation of configurations – Use reactive agent to instantiate a local configuration
• Issues in reactive generation – How to identify the “relevant advertisements”? 1) broadcast a query 2) semantic matching 3) needs a shared ontology
Reactive configurations
● Has template configuration for 1. Find semantic description tuples specific tasks. Instantiate with for all available components: available components. *.component.*.description 2. Select a component for each ● Uses the introspection variable in template. mechanisms to find matching 2a. Match component functionality components. with functionality of template variable using the functionality ● Sends launch and configuration ontology. requests through tuplespace 2b. Match all inputs/outputs using the parameter ontology. ● Monitors components, if failure 3. Start all selected components and select another instantiation of establish connections. template. 4. Monitor the status of instantiated components.
Reactive configurator
● Configurators can themselves be used as if a component by other configurators. Both hierarchically and “serially”
– Example: Camera example. Some configurator has a template for Color camera + conversion system. Found and used by consumer of monochrome images. – Example: Template “babysitter” using “mobile camera” to photograph Malin, this uses another template “assemble robot” and component “camera”.
Take picture Monochrome cam
MCam Album CCam IP MCam
Reactive configurations - example
● Navigating a mobile robot requires a “navigator” configurator and “localizer” configurator (used by the navigator)
– To the left, the used configuration – To the right, the used templates Navigator L Pippi Workstation 1 Workstation 2 NC D M D M D M D IP S NC P C P C P C Cam O S D Localizer #1
Environment Cam IP L
Reactive configurations - example
● If the first localizer template fails, use second
● Starts a new component (Int) performing dead-reckoning on odometry data.
Navigator L Pippi Workstation 1 Workstation 2 NC D M D M D M D IP S Int NC P C P C P C Cam O S D Localizer #2
Environment O Int L
Reactive generation of configurations
● Use reactive agent to instantiate local configurations
● Issues in reactive generation of configurations
– How to identify the “relevant advertisements”? – How to monitor the recursive local configurations? 1) subscribe to “fail” signal 2) if “fail” received, try next local configuration
Reactive configuration: Issues
• Use reactive configuration agent – Decide and instantiate a local configuration
• Issues in reactive generation – How to identfy the “relevant advertisements”? 1) broadcast a query 2) semantic matching 3) needs a shared ontology
Reactive configuration: Issues
• Use reactive configuration agent – Decide and instantiate a local configuration
• Issues in reactive generation – How to identify the “relevant advertisements”? – How to monitor the recursive local configurations? 1) subscribe to “fail” signal 2) if “fail” received, try next local configuration
Reactive configuration: Issues
• Use reactive configuration agent – Decide and instantiate a local configuration
Relevant Configuration Relevant Configuration Fail ! advertisements agent advertisements agent
deployer
Go al
Reactive configuration: Issues
• Use reactive configuration agent – Decide and instantiate a local configuration
Relevant Configuration advertisements agent
deployer
Go al
Reactive configuration: Issues
• Use reactive configuration agent – Decide and instantiate a local configuration
• Issues in reactive generation – How to identfy the “relevant advertisements”? – How to monitor the recursive local configurations? – How to react to changes in the ecology? Use the same “fail” signals !
Reactive configuration: Issues
• Use reactive configuration agent – Decide and instantiate a local configuration
Relevant Configuration Relevant Configuration advertisements agent advertisements agent
deployer deployer Go al
Fail !
Reactive configuration: Issues
• Use reactive configuration agent – Decide and instantiate a local configuration
Relevant Configuration advertisements agent
deployer
Go al
Plan-based or reactive configuration ?
• Complementary pros and cons
Plan-based Reactive Good • Complete • Only requires local knowledge • Optimal • Can adapt to changes Bad • Requires global knowledge • Not complete •Computational resources • Sub-optimal
• Next: develop hybrid approach
Configurations
Summary
• Plan-based configuration – Extensively investigated off-line from PEIS Ecology – First experiments inside the PEIS Ecology
• Reactive configuration – First version of semantic matching of PEIS Components – First (very simple) version of reactive configuration agent
• Future research – Continue investigation on reactive configuration (semantics)
– Investigate hybrid approach Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology 1. Heterogeneity – Demonstrator environments 2. Link physical and digital world 3. Self-configuration – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues Building a Ubiquitous Robotics System
● Using the PEIS Ecology framework
– OpenSource middleware and components
● www.aass.oru.se/~peis – Interfaces
● Player interface for many robotic components
● TinyOS for sensor networks + actuation
● Interfaces for custom built hardware – Simulator environment
● Player/Gazebo. Custom 3D models and physics
The PEIS home
● Testbed environment built-up to test concepts and implementations. ● Implemented hardware and software components – Refrigerator – Cameras – Mobile robots – RFID-tagged floor – Electronic noses – Actuated window blinds, lamps
– plants, ... The PEIS Home
● Small (25 m2) Swedish bachelor’s apartment
The living room The kitchen
The bedroom The PEIS Home: PEIS
Several software components • Thinking Cap • PTL Plan • PEIS Vision System • PEIS Tracker • PEIS Player • Olfaction classifier • Tuple viewer • ... The localization system
The PEIS Fridge A few robots and embedded devices Hardware
● The kitchen – Electronic gas-sensors – Actuated refrigerator – Mobile robot(s) – Electronic nose – RFID-tagged floor – Static cameras – TV + Multi-media center – Loudspeakers – Sensor network – ...
The PEIS Home: Experiments
● Interacting with a box – Goal: study “anchoring” of perception of digital communication ● An olfaction ecology – Goal: solve a task beyond reach of today mobile robotics Blinds controller ● Self-configuring ecologies – Goal: study self-configuration and dynamic re-configuration ● A tiny ecology – Goal: integrate robots and embedded devices
Light/temperature sensor The PEIS Home: Experiments
● Interacting with a box – Goal: study “anchoring” of perception of digital communication
Where am I?
Can I push you away?
The PEIS Home: Experiments
● Interacting with a box – Goal: study “anchoring” of perception of digital communication
● An olfaction ecology – Goal: solve a task beyond reach of today mobile robotics
The PEIS Home: Experiments
● Interacting with a box – Goal: study “anchoring” of perception of digital communication
● An olfaction ecology – Goal: solve a task beyond reach of today mobile robotics
● Self-configuring ecologies – Goal: study self-configuration and dynamic re-configuration
The PEIS Home: Experiments
● Interacting with a box – Goal: study “anchoring” of perception of digital communication
● An olfaction ecology – Goal: solve a task beyond reach of today mobile robotics
● Self-configuring ecologies Blinds controller – Goal: study self-configuration and dynamic re-configuration
● A tiny ecology – Goal: integrate robots and embedded devices
Light/temperature sensor Behind the scenes
● The PEIS Floor – Regular grid of RFID tags – Each one RW, 64 x 32bit fields – Physical shared memory
● Possible uses – Self localization – Gradient following – Stigmergetic communication
Behind the scenes
● The PEIS Fridge – Opening mechanism – Simple e-noses (gas sensors) – Embedded computer
Behind the scenes
● Motorized window blinds
– Controlled by Pmote
● Light/Humidity sensors on plants
– Controlled by Tmote
● Lamps
– Controlled by Tmote
● ActiveMedia PeopleBot
– Embedded PC
● Humanoid tracker Pmote
– Stereo vision cameras in ceiling / walls Tmote – Tracking, pose estimation, face direction ETRI Testbed
● Testbed environment for URC (Ubiquitous Robotic Companion) based project
● Full scale appartment: robots, sensors, home automation devices...
● Retro-fitted as PEIS components
Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology
– Demonstrator environment – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
Player/Stage
● Reminder of Player/Stage
– Open Source – Player robot device interface
● Common robot systems – Stage robot simulator
● Many robots
● Low fidelity 2D simulator – Gazebo robot simulator
● Few robots
● High fidelity 3D simulator
Open Source Components
● Policy to release all components under an OpenSource licence
– Send email if interested in components
● (new announcements, help with problems etc.) – Download from http://www.aass.oru.se/~peis
● PEIS-kernel generation G4 (peis-0.4.0.tar.gz), peisplayer-0.4.0.tar.gz, peisinit-0.4.0.tar.gz, tupleview-0.4.0.tar.gz, peiscam-0.4.0.tar.gz, peisrfid- 0.4.0.tar.gz
Components
● Robot software ● System
– Vision system (Color segmentation) – PeisInit
– Humanoid tracking – Tupleviewer
– Thinking Cap – SimpleDB
– PTLplanner – Configurators
● Device access ● ...
– PEIS-player bridge
– TinyOS integration
Compone Co mponent Co – mponent Olfaction component Compon ent com ponent – Cameras (V4L/V4L2/Player
● Special devices
– Tracker, refrigerator
– RFID readers, 802.15.4 communication, ... Player/Stage and the PEIS-Ecology
● PeisPlayer
– Bridges Player <-> Peis Ecology devices
● Exports each devices as tuples
● Physical configuration
● Sensors as writes
● Actuators as reads sonar.0.pose = (0.0 0.0 0.0) sonar.0.range = 2.4m position.1.setvel = 0.5 – Real robots (player) or simulated (stage/gazebo)
PeisPlayer
● PeisPlayer
– Start player with ordinary configuration file and/or with stage/gazebo – Launch peisplayer on any host, commandline argument for player connection – One PEIS-component per player instance
● Exports each devices as tuples
● Physical configuration
● Sensors as writes
● Actuators as reads sonar.0.pose = (0.0 0.0 0.0) sonar.0.range = 2.4m position.1.setvel = 0.5 Compone Com ponent Comp onent co Co mponent mponent PeisInit component
● Install and create bootup script to launch
– Eg. In /etc/rc.local add su peis /usr/local/peisinit --peis-id 7400
● Create component descriptions
– One component description per PEIS-program – Place descriptions in /usr/local/share/peisinit – Example:
# /usr/local/share/peisinit/peiscam0.cmp Name=peiscam0 Id=+10 Exec='peiscam $PEISINIT -c /dev/video0 –width 320 –height 240 InitState=off SpawnDelay=3.0 Semantics='...' Tupleview component
● Tool for inspecting/accessing distributed tuplespace
– View all PEIS components and available tuples. – Inspect individual tuples as text or images. – Insert or manipulate text valued tuples – Save tuples to disk
The peiscam component
● Deliver images from V4L (Video For Linux) sources
– Handles RGB and YUV image sources
– Frame grabber cards, Web cameras
– JPG compressed images (lossy)
– PNG compressed images
● Image output as tuple “image”
– Mime-type header. Eg. image/jpg or image/png
● Simple built-in image operations
– Flip image, Rotate image
– Integrate multiple image (trading fps for color resolution) RFID driver
● Interface to Texas Instruments RFID reader
– Used onboard robot, in refrigerator (scan contents) – Publish list of nearby (< 10cm) RFID tags as tuple – Writing (temporarily) disabled from tuplespace
● (avoid consistency issues by accidentally writing on the floor) ● Tags
– Creditcard size, <1mm thick, flexible – 8 x 8bits unique identification number – 64 x 32bits read-write memory (2kbit)
Simulating the PEIS Home
● Simulation for
– Debugging tools – Setting up repeatable experiments quickly – Testing components before actual (physical) implementation – Off-load limited resources – Variable time scale
● Simulator
– Physical simulation of environment
– Interface to actuators and sens ors – Setting up hardware/software Creating a simulator
● OpenSource simulator Gazebo
– Rigid body physics – Player interface to actuators/sensors
● Customize
– 3D models for environment
● Lamps, fridge, walls, furniture, ... – Interfaces
● Player interface directly when also used for real hardware
● Player->Custom bridge when other direct sensors used in real world
– eg. cameras, customized e-noses... Customizing Gazebo
● Modules as loadable shared libraries
– Drawing object – Physical geometry – Input/Outputs
● Interfaces
– Standard player interfaces
● eg. position2d interface 65 degree webcamera view used for localization system – Specialized interfaces
● Use actArray interface
● Customized client interpreting data – Cameras as rendered images
– CPU intensive! Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology
– Demonstrator environment – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues Practical implementation of a PEIS Ecology experiment run
● How do all components, configurations and interactions work in practice?
● Show by dissecting an example run of anchoring experiment.
– Individual components – Configurations – Complete execution
Example experiment
Anchoring a seen box [ICRA-06] • The scenario
1. Pippi is executing the task “Go To Bedroom” 3. It finds an unexpected box blocking the door 4. Pippi asks the box about its “pushable” property 5. The box replies: “pushable = yes” 6. Pippi decides that it can push the box out of the way
Example experiment
Anchoring a seen box • The configuration pushable Box Pippi 1. Pippi is executing the task “Go To Bedroom” M D M D 3. It finds an unexpected box blocking the door P C P C 4. Pippi asks the box about its “pushable” property 5. The box replies: “pushable = yes” 6. Pippi decides that it can push the box out of the way Environment
Example experiment
Anchoring a seen box y → Peis-3 • The anchoring negotiation box-4 x
shape = (0.3, 0.2, ...) 1. Pippi is executing the task color = (0.1, 0.6, 0.2) “Go To Bedroom” 3. It finds an unexpected box blocking the door 4. Pippi asks the box about its “pushable” property Emil <7, App<,P ucsyhlaibnldee,r (*r>ed)> 5. The box replies: “pushable = yes” <3,
Tuple-space
Box
Example experiment
Anchoring a seen box • In practice ...
1. Pippi is executing the task “Go To Bedroom” 3. It finds an unexpected box blocking the door 4. Pippi asks the box about its “pushable” property 5. The box replies: “pushable = yes” 6. Pippi decides that it can push the box out of the way
Dissecting the experiment
● What are the participating PEIS?
– Pippi the robot (id's 3200 - 3299) – Emil the robot (id's 3300 - 3399) – Home security system (id's 6800 - 6899) – Ceiling cameras + PC – RFID tagged floor – The box (id's 6200 - 6299)
● Additionally (no effect on experiment)
– Refrigerator (id's 6600 - 6699) – E-nose
– Multimedia center (id's 6900 - 6999) Pippi the PEIS
● Hardware
– Magellan Pro, Camera, e-Nose (Cyranose Inc.)
● Software
– Player (with rflex interface) – PeisPlayer: Translating between player and tuplespace.
● Connected to local Player instance. – PeisCam: Grabbing V4L images from camera – ThinkingCap: Fuzzy behaviour based navigation controller
● Sometimes run onboard another PEIS instead – PeisNose: Exports raw sensor data from nose to tuplespace – PeisCSVision: Color segmentation based object recognition
● Sometimes run onboard another PEIS instead Configuration on Pippi (subset)
● Components borrowing functionalities
– PeisCSVision connected to PeisCam Vision
● 3240.use-camera-id = 3230 where 3240 is id of PeisCSVision, 3230 id of PeisCam TC
● Computationally expensive, can optionally run on another PEIS (eg.Homesecurity system PC) for faster framerate. 6840.use-camera-id = 3230
– ThinkingCap connected to PeisPlayer. Cam PeisPlayer
● 3220.use-robot-id = 3210 where 3220 is the id of the ThinkingCap instance, 3210 id of PeisPlayer 3220.use-vision-system = 3240 (or 6840 when running peiscsvision offboard)
● Can run offboard on developers PC for debugging purposes. Again, just change id's.
● Optionally use an odor classifier connected to PeisNose
● Optionally use localization information from external source Emil the PEIS
● Hardware
– Magellan Pro, SICK Laser scanner, Camera, RFID reader
● Software
– Player (with rflex interface) + PeisPlayer – peisCam – peisCSVision (sometimes run offboard) – Thinking Cap – PeisPTL: Conditional planner system with uncertainty – PeisRFID: driver for the RFID reader
● Configuration
– Thinking Cap and cameras simila r to configuration of Pippi – PeisPTL: Actions to move (use) ThinkingCap onboard Pippi or Emil Other PEIS
● Cameras and Home security system
– Standard webcameras – Standard PC
● RFID tagged floor
– Not a PEIS as such. Can be proxied by other components.
● Software
– PeisCam components for each camera – PeisCSVision for object detection – ...
Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology
– Demonstrator environment – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues Running PEIS components
● Compile and install library, the PEIS-kernel
– provides API access to PEIS Ecology
● PEIS components are normal OS “programs” linked against kernel.
● Each PEIS (H/W entity) runs meta component PEIS-init
– Started at boot time – Container for PEIS specific information
● Eg. Appearance tuples – Source of semantic information / monitors components
● Available components are normal installed “programs”
Using the PEIS-kernel
● Download peis-0.4.0.tar.gz, unpack etc.
– Install with normal procedures
● peis/ – peis/peiskernel # The kernel library – peis/samples # Sample code – peis/docs # Documentation (doxygen)
● Doxygen generated documentation
● Samples and old components
– simpleactuator / simplesensor: Examples of creating/consuming tuples – peismaster: A null component useful for network testing – Ignore all other samples (deprecated code)
PEIS kernel API
● General housekeeping (initialization, shutdown etc.) TC,PTL, Player, Cam, CS Vision, Fridgedoor, Nose, HUI-server, ... ● Single threaded version PEIS-Init, – Call peisk_step() as often as PE Semantic discovery,
IS Configurator Tupleview
possible (> 10 hz, preferably -mi 100hz) Tuplespace
dd Topology, PEI
– lewa Debugging
Handles all networking, calling S-
... kerne periodic functions, callbacks etc. re
● Network communication Multithreaded version l Discovery, P2P, ... – Separate thread handles I/O, periodics etc. Hardware & Operating system
– Callbacks executed in alternative thread -> avoid blocking! PEIS kernel API
● Tuplespace
– Housekeeping
● initTuple(Tuple*), initAbstractTuple(Tuple*) – Creating/Writing tuples (Write)
● insertTuple(Tuple*) – Subscribing to tuples from other PEIS
● subscribeAbstract(Tuple*) – Finding local copies of tuples (Read, value based)
● findTuples(Tuple*,ResultSet*) – Callbacks when new tuples received (Read, event based)
● RegisterAbstractTupleCallback( Tuple*, function-pointer )
PEIS kernel API
● Abstract tuples
– Use as prototypes for requesting tuples – Fill in the PeisTuple structure
● initAbstractTuple(Tuple*) – Leave some fields blank, eg. Timestamp, data, subkey etc. – Subscribe to tuples before accessing them
● subscribeAbstract(Tuple*) – Can use callbacks to receive notification on new tuples
● registerAbstractTupleCallback(&tuple,function-pointer)
PEIS kernel API
● Simplified interface
– Use simple wrappers for common cases
● Eg. fully qualified tuples (no wildcards)
● setStringTuple(char *name,char *data)
● getTuple(char *name,int *len,void **return-data)
● ... – No real notion of abstract tuples – Backwards compatible with previous generation of PEIS kernel
Simple PEIS components (1)
● Pseudocode for camera components
1. peisk_initialize() 2. setTuple(self,“name”,”peiscam”) 3. while not (getTuple(self,“do-quit”) = “yes”) 4. im=grab video image 5. setTuple(self,“image”,im) 6. elihw
Simple PEIS components (2) The old fashioned way
● Pseudocode for vision system
1. peisk_initialize() 2. setTuple(self,“name”,”peiscsvision”) 3. while not (getTuple(self,“do-quit”) = “yes”) 4. cam=getTuple(self,“use-camera-ID”) 5. if cam then subscribe(cam,”image”) 6. im=getTuple(cam,”image”) 7. if im then 8. objs=doObjectRecognition(im) 9. setTuple(self,”objects”,objs) USING im.timestamp 10. elihw
Simple PEIS components (2) Using meta tuples
● Pseudocode for vision system
1. peisk_initialize() 2. setTuple(self,“name”,”peiscsvision”) 3. while not (getTuple(self,“do-quit”) = “yes”) 4. im=getMetaTuple(cam,”mi-image”) 5. if im then 6. objs=doObjectRecognition(im) 7. setTuple(self,”objects”,objs) USING im.timestamp 8. elihw
Alternativly Simple PEIS 1.c prootomptypoe.nenowner =ts ANY (3) 2. prototype.key = “name” 3. prototype.value = “peiscam” 4. tuple = findTuple(prototype) ● Pseudocode for connecting ca5.mera cam =tandupl vie.siidon ...
1. for tuple IN getTuples(*,”name”) 2. if tuple.value = “peiscam” then cam=tuple.owner 3. if tuple.value = “peisvision” then vision=tuple.owner 4. sprintf(metaTuple,”(META %d image)”,cam); 5. setTuple(peiscsvision,”mi-image”,metaTuple) 6. ... 7. objects = getTuple(peiscsvision,”objects”)
Outline
● Theoretical
– Concepts and definitions – A case study: The PEIS Ecology
– Challenges in Ubiquitous Robotics
● Practical using the PEIS Ecology
– Demonstrator environments – Existing components and their use – Dissecting an experiment – Using the reference middleware (OpenSource)
– Other issues Other issues
● Human user interfaces
– Standard interfaces: TV based, Cellphone interface – Expressive face
● Sensor networks
● Comparing to other UbiRob middlewares
● The olfaction ecology
Sensor networks and Ubiquitous Robot Systems
● Motes
– Small networked sensing and computing devices
● 802.15.4 Network stack / Zigbee – 2 – 40KB of memory – Typically running TinyOS
● In Ubiquitous Robotics
– Integrate sensing with other robot systems – Actuation driven by motes – Computational power limits intelligence
● Centralize control from “real” devices, or
● Simple low-level intelligence PEIS Ecologies and Sensor Networks
● Tmote Sky
– Low-cost, low-energy, sensor networks – 10kB RAM,802.15.4 radio,8Mhz microcontroller
● Pmote
– Low-cost, sensing and actuation – 2kB RAM, 802.15.4 radio, 8Mhz microcontroller
● TinyOS is a small operating system written in NesC commonly used for Sensor Networks.
– Event based, mesh networking support – Maximum packet size: ~100 bytes
● How include these devices in the PEIS-Ecology? PEIS-kernel and Sensor networks
● Approach 1 – full port of PEIS-kernel
– Suitable for very small computers, eg. Gumstix
● 80 x 20mm, 802.11 networking, 64MB ram, 400Mhz – Motes lack memory, CPU and bandwidth to participate in general P2P network (routing, storing intermittent messages, etc.). Cannot store and transmit large tuples
● Approach 2 – port of tuplespace
– Needs only to implement same tuple mechanisms and some of the P2P protocolls, eg. distributed time.
– Gateway bridging ethernet network with 802.15.4 network. Also translating and filtering messages.
– Memory and bandwidth problems with large tuples
● Compressed and simplified protocoll The next step: Human in the Ecology
● “Symbiotic Robotic Systems” D Workspace i Tracker World g i
Model t
H E a l
w Perception
Task o
Execution H r Milk l Robot d E P h y s i c a l
w o rl d
Human User Interface
● Information and options ● Other human interaction
– cell-phone based interface – vocal, indirect, intention-based – TV interface – Not dealt with yet – Expressive Face – Future work – Under construction!
Expressive face interface
● Problems of human interface to a PEIS Ecology: – The user needs to interact with a large number of devices but still perceive the PEIS Ecology as one whole entity ⇒ Common Interface Point
– These devices are highly etherogeneous but the user should have one uniform way to interact ⇒ Expression-based semantics
Expressive-face interface
● Common interface point – integrates all the information from the different PEIS, – dispatches all the user’s commands to the different PEIS
Common PEIS 1 Interface Point Modality A PEIS 2 ⊕
Modality B PEIS n
Expressive-face interface
● Expression-based semantics – satisfaction of each PEIS measured by number in [0,1] – visualized by a human-understandable expression
dustbag s Battery low rather full c very sad rather worried quite happy ti n a m se 0.2 0.6 1.0 e lu va Too much sunlight Getting dry n o ti a liz a su vi
Expressive-face interface
● Combined satisfaction of a PEIS Ecology satisfaction values combined by fuzzy logic
cleaner-happy <− clean-floor ^(full-battery v empty-dustbag)
dustbag 1.0 0.1 battery min max 0.1 0.8
dirt-sensor 0.8 0.2 min
humidity 0.2 min 0.2 temperature 0.9
User Interface
Cell phone based interface
● Any cellphone capable of running Java (J2ME) applications
● Cannot run general PEIS-kernel on cellphones
– Expensive bandwidth, latencies, restricted CPU capability
● HUI-component running on any other PEIS
– Use introspection to get list of available options and information – Talk to cellphone/TV application with simple XML based protocoll, similar to simplified XHTML
XML Tuples
TV based interface
● Uses the same HUI-server to perform introspection and create user information
● New HUI-renderer talking XML with HUI-server, displays information, options and general status to the user.
● Typically, HUI-renderer and -server runs onboard a MediaCenter connected to TV, also allows other mediacenter abilities (eg. PVR) and other PEIS services.
HUI-renderer HUI-server HDMI Tuples
Introspection in user interfaces
● Use tuples to see which components exists, which capabilities can be performed. Present as option to users.
● Order information and options according to ontology
● Template based approach to present options.
● Eg. if there exists a successful configurator for mobile platform, then add a menu for operating on it
● Look for any status tuples with a failure, present status message and options for explaining failure. Compute expressive face.
● Eg. A send-robot configurator exists which accepts as inputs places, a map exists which contains a list of places. Therefore present option “send a robot to the bedroom” appears as a menu
option for the user. PEIS Ecology: Main message
“ Complex abilities are achieved through the cooperation of many simple robotic devices pervasively embedded in the environment ”
PEIS Ecology: Assesment
● Simplify some problems – by-pass many technological gaps
– easier to accept
● Introduce new challenges – heterogeneity
– link the physical and the digital world – self-configuration
● Next major step – include the human as part of the ecology
Thank you!
And thanks to: www.aass.oru.se/~peis Mirko Bordignon Marco Gritti Donatella Guarino Lars Karlsson Kevin LeBlanc Amy Loutfi Robert Lundh Jayedur Rashid
... and many others !
Do we really want intelligent devices ?
PEIS-ecologies / Super Distributed Objects
● Super Distributed Objects White Paper 1.00, Super Distributed Objects DSIG by Seiichi Shin, Katsumi Kawano and WG members.
● Organisation
– Both SDO's and PEIS distributes over a large scale ad-hoc environment.
– Both are completely decentralised (scalability) with dynamic relations.
– Individual units (PEIS or SDO) can be fully autonomous
– Units (PEIS or SDO) can be highly hetereogeneous
● Entities
– Operations of SDO dictated by a Service Logic (SL), one or more SDO's and one SL forms an Application Service (AS). User interfaces (UI) are separate entities.
– In PE each hardware object is a PEIS and contains one or more components.
Objects which are too dumb (eg. a co ffee cup)) or too smart (eg. a human) to participate directly in ecology can be represented by another component, a proxy. Comparing PEIS-ecologies
● Points of difference
– Semantics of SDO's and formations of AS is not specified. Logics of performing configurations is not explicitly expressed.
● Eg. How can an SL expecting a monochrome camera automatically use a colour camera from another vendor even if there exists a colour conversion SDO somewhere else in the network? – Communication, tuplespace vs. corba.
● In PE consistency enforced on semantic level (allows inference and conversions) instead of on a syntactic (parameter) level. – Introspection
● Semantic descriptions and associative tuplespace vs. Profiles and CORBA IDL – “Everything is a PEIS”. Hierarchical configurations possible. Proxied PEIS.
Example experiments in PEIS Home: The Olfaction Ecology
● Problems with artificial olfaction
– Electronic nose must be close to source – Navigation by smell is an unsolved problem – Odor classification only reliable for few classes
● The PEIS Ecology answer – Network of (simple) gas sensors provide alarms and location – Mobile robot carries sophisticated electronic nose – RFID tag provide olfactory context
The Olfaction Ecology
The Olfaction Ecology