US0089.09370B2 (12) United States Patent (10) Patent No.: US 8,909,370 B2 Stiehl et al. (45) Date of Patent: Dec. 9, 2014 (54) INTERACTIVE SYSTEMS EMPLOYING (56) References Cited ROBOTC COMPANONS U.S. PATENT DOCUMENTS (75) Inventors: Walter Dan Stiehl, Somerville, MA (US); Cynthia Breazeal, Cambridge, 4,951,510 A ck 8/1990 Holm-Kennedy et al. ........................ 73,862.041 MA (US); Jun Ki Lee, Cambridge, MA 5,083,466 A * 1/1992 Holm-Kennedy (US); Allan Z. Maymin, Cos Cob, CT et al. ........................ 73,862.041 (US); Heather Knight, Lexington, MA (US); Robert L. Toscano, Chula Vista, (Continued) CA (US); Iris M. Cheung, Rochester, OTHER PUBLICATIONS MN (US) Lumelsky et al., Sensitive skin, Jun. 2001, IEEE sensors Journal, vol. (73) Assignee: Massachusetts Institute of Technology, 1. No. 1. Cambridge,ambridge MA (US(US) (Continued) (*) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 Primary Examiner — Bhavesh V Amin U.S.C. 154(b) by 1354 days. (74) Attorney, Agent, or Firm — Stephen R. Otis (21)21) Appl. NoNo.: 12/117,3899 (57) ABSTRACT (22) Filed: May 8, 2008 An interactive system for interacting with a sentient being. The system includes a robotic companion of which the sen (65) Prior Publication Data tient being may be a user and an entity which employs the US 2009/OO55O19 A1 Feb. 26, 2009 robot as a participant in an activity involving the user. The robotic companion responds to inputs from an environment Related U.S. Application Data that includes the user during the activity. The robotic com (60) Provisional application No. 60/928,487, filed on May panion is capable of social and affective behavior either under 8, 2007. control of the entity or in response to the environment. The entity may provide an interface by which an operator may (51) Int. Cl. control the robotic companion. Example applications for the G06F 9/00 (2011.01) interactive system include as a system for communicating B25 3/00 (2006.01) with patients that have difficulties communicating verbally, a G06N3/00 (2006.01) system for teaching remotely-located Students or students B25 3/08 (2006.01) with communication difficulties, a system for facilitating B25.9/6 (2006.01) Social interaction between a remotely-located relative and a (52) U.S. Cl. child, and systems in which the user and the robot interact CPC ............. B25J 13/00 (2013.01); A63H 2200/00 with an entity Such as a Smartbook. Also disclosed are control (2013.01); G06N3/008 (2013.01); B25J 13/081 interfaces for the robotic companion, techniques for render (2013.01); B25J 9/1671 (2013.01) ing the robotic companion sensitive to touch and responding USPC ........................................... 700/245 700,264 to those touches, and techniques for providing quiet, back (58) Field of Classification Search drivable motion to components of the robotic companion. USPC .................................................. 7OOf245. 264 See application file for complete search history. 20 Claims, 37 Drawing Sheets 2383 8 we 8 we was 8 axxx w w w w 8 & rest & we was wer sks ...e.9-...sessessex sks e.g...eassesses w US 8,909,370 B2 Page 2 (56) References Cited 2005/0216121 A1* 9, 2005 Sawada et al. ................ TOO.245 2006, O184273 A1 8, 2006 Sawada et al. U.S. PATENT DOCUMENTS 2007/0150.107 A1* 6/2007 Ogawa et al. ................. TOO.245 OTHER PUBLICATIONS 5,095,762 A * 3, 1992 Holm-Kennedy et al. ........................ 73,862,041 5,101,669 A * 4, 1992 Holm-Kennedy Miller et al., Motion capture from inertial sensing for untethered et al. ........................ T3,862.626 humanoid teleoperation, 2004, IEEE Humanoids 2004.* 5,544,649 8, 1996 David et al. ................... 600/301 Koichi Koganezawa, Back drivable and inherently safe mechanisms 5,831,408 11, 1998 Jacobus et al. 318.568.11 for artificial finger.* 5,987,726 11, 1999 Akeel ........................ 29,407.08 http://www.ieee-ccnc.org/2006/conf program? demos index. 6,175,772 1, 2001 Kamiya et al. htmlisting (webpage as of Nov. 12, 2012) Webpage lists demonstra 6,535,793 3, 2003 Allard ........................... 7OO 259 tions in Jan. 2006 at Consumer Communications & Networking 6,629,242 9, 2003 Kamiya et al. 713,100 Conference 2006 (CCNC2006), Las Vegas, Nevada, USA, (including 6,684, 127 1, 2004 Fujita et al. 700,245 a demonstration of W. Stiehl, J. Lieberman, C. Breazeal, L. Basel, H. 6,845,297 1/2005 Allard ......... TOOf 259 Knight, L. Lalla, “The Huggable: A Therapeutic Robotic Companion 7,348,746 3, 2008 Ogawa et al. 318.568.12 for Relational, Affective Touch”). 7,720,572 5, 2010 Ziegler et al. ................. TOO.245 J. Lieberman, C. Breazeal, L. Basel, R. Cooper, H. Knight, L. Lalla, 7,778,731 8, 2010 Ogawa et al. ................. TOO.245 A. Maymin. The Huggable: A Therapeutic Robotic Companion for 7,813,836 10, 2010 Wang et al. 700,245 Relational, Affective Touch, 3rd IEEE Consumer Communications & 7,835,926 11, 2010 Naidoo et al. .................... 705/3 Networking Conference, 2006 (CCNC2006), Jan. 8-10, 2006, Las 7,912,733 3, 2011 Clements et al. ................. 705/2 Vegas, NV, USA. vol. 2, pp. 1290-1291. 2001 0013764 8, 2001 Blumenkranz et al. ... 318/.568. 11 Stiehl, W. D., Lieberman, J., Breazeal, C., Basel, L., Lalla, L., & 2002/0103576 8, 2002 Takamura et al. ............ 700/245 Wolf, M. (Aug. 2005). Design of atherapeutic robotic companion for 2002/O120362 8, 2002 Lathan et al. .. 700,245 relational, affective touch. Proceedings of Fourteenth IEEE Work 2002fO198626 12, 2002 Imai et al. ...... 700,245 shop on Robot and Human Interactive Communication (Ro-Man 2003, OOO4611 1, 2003 McKinney et al. 700,258 05), Nashville, TN, 2005, pp. 408-415. 2004.0034449 2, 2004 Yokono et al. ................ TOO.245 W. Stiehl, C. Breazeal, Affective Touch for Robotic Companions, 2004/O186623 9, 2004 Dooley et al. ................. TOO.245 ACII’05 Proceedings of the First international conference on Affec 2005, 0104964 5/2005 Bovyrin et al. 348,155 tive Computing and Intelligent Interaction, pp. 747-754, Springer 2005/O154265 7/2005 Miro et al. .................... 600/300 Verlag Berlin, Heidelberg 2005. 2005/O192676 9, 2005 Sears et al. ...................... 623/24 2005/O197739 9, 2005 Noda et al. .................... TOO.245 * cited by examiner U.S. Patent Dec. 9, 2014 Sheet 1 of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 2 of 37 US 8,909,370 B2 wyz × U.S. Patent Dec. 9, 2014 Sheet 3 Of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 4 of 37 US 8,909,370 B2 32 U.S. Patent Dec. 9, 2014 Sheet 5 Of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 6 of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 7 Of 37 US 8,909,370 B2 U.S. Patent US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 9 Of 37 US 8,909,370 B2 s U.S. Patent Dec. 9, 2014 Sheet 10 Of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 11 of 37 US 8,909,370 B2 Fig. 9 U.S. Patent Dec. 9, 2014 Sheet 12 Of 37 US 8,909,370 B2 3 s U.S. Patent Dec. 9, 2014 Sheet 13 Of 37 US 8,909,370 B2 O 1113 Q O 1101 1103 U.S. Patent Dec. 9, 2014 Sheet 14 of 37 US 8,909,370 B2 kiss Stasier 1214 123 Nitises Sistier - a 3 Kaixi &isia 8.33isites &gai 33 live visix 8&s i is a sikia 888i & is 888s 8e S F. 2 U.S. Patent Dec. 9, 2014 Sheet 15 Of 37 US 8,909,370 B2 $$$$ ------------------ : s's &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& U.S. Patent Dec. 9, 2014 Sheet 16 of 37 US 8,909,370 B2 pupIOUÐAp|ng spunos U.S. Patent Dec. 9, 2014 Sheet 17 Of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 18 Of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 19 Of 37 US 8,909,370 B2 F.G. 63 U.S. Patent Dec. 9, 2014 Sheet 20 of 37 US 8,909,370 B2 --------------------------ay-as-you-ay-a-M C. A U.S. Patent Dec. 9, 2014 Sheet 21 Of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 22 Of 37 US 8,909,370 B2 s U.S. Patent Dec. 9, 2014 Sheet 23 Of 37 US 8,909,370 B2 1909 Š7 1911 ------- U.S. Patent Dec. 9, 2014 Sheet 24 of 37 US 8,909,370 B2 Fig. 19B U.S. Patent Dec. 9, 2014 Sheet 25 Of 37 US 8,909,370 B2 2007 2009 Fig.20 U.S. Patent Dec. 9, 2014 Sheet 26 of 37 US 8,909,370 B2 U.S. Patent Dec. 9, 2014 Sheet 27 Of 37 US 8,909,370 B2 s X - M - M - M -8888 88 fi, U.S. Patent Sheet 28 Of 37 US 8,909,370 B2 ?| | } ~~~~(f)): ~~~~,quae/sanae+ ?ael,mae ***. U.S. Patent Dec. 9, 2014 Sheet 29 Of 37 US 8,909,370 B2 } } } { Wax Aaaa aaa. Aa Aaw W.W. **** so-8-aw & s 8 Y Saux &gazz •??**----******~~~~~~~ U.S. Patent Dec. 9, 2014 Sheet 30 Of 37 US 8,909,370 B2 ******************** .*,**** U.S. Patent Dec. 9, 2014 Sheet 31 Of 37 US 8,909,370 B2 a saw are a-sas a sex is a so sa as as a a sax e as a sa-awa as a s.