Creative Robotics Studio
An Interactive Qualifying Project Report submitted to the faculty of the Worcester Polytechnic Institute in partial fulfillment of the requirements for the degree of Bachelor of Science. By: Graham Held Walter Ho Paul Raynes Harrison Vaporciyan
Project Advisers: Scott Barton Craig Putnam Joshua Rosenstock
Creative Robotics Studio 2
Contents
Contents ...... 2 Abstract...... 4 Acknowledgements ...... 5 Introduction ...... 6 1 Background ...... 8 1.1 Relationship between Robots and Humans ...... 8 1.1.1 “Human Replacement” ...... 8 1.1.2 Humanizing Robots - Anthropomorphism ...... 9 1.1.3 Uncanny Valley ...... 11 1.1.4 Computers and Digital Technologies in Art ...... 12 1.1.5 Robots in Art ...... 13 1.2 Gesture ...... 18 1.2.1 Using Props ...... 18 1.2.2 Choreography ...... 19 1.2.3 Movement ...... 19 1.2.4 Music and Emotion ...... 20 1.2.5 Audio-visuals ...... 21 2 Methodology ...... 22 2.1 Surveys ...... 23 2.1.1 Questionnaire Design ...... 23 2.1.2 Background Survey Design ...... 24 2.1.3 Survey Distribution ...... 26 2.1.4 Performance Group Discussion and Surveys ...... 26 2.2 Performance Design ...... 27 2.2.1 Performance Hardware ...... 27 2.2.2 Storyboarding and Narrative ...... 29 2.2.3 Animatics and Animation ...... 29 2.2.4 Exporting the Performance ...... 31 2.2.5 Music ...... 31 2.2.6 Playing Back the Live Performance ...... 31 3 Data and Discussion ...... 32 3.1 Background Survey Data ...... 32 3.1.1 Comfort Scenarios ...... 37
Creative Robotics Studio 3
3.1.2 Correlations with Literacy Scores ...... 38 3.1.3 Changes in Opinion ...... 41 3.1.4 Open Ended Responses ...... 48 3.2 Developing an artist’s toolset ...... 49 3.2.1 Function before Aesthetics...... 50 3.2.2 Future Endeavors with Our Toolset ...... 50 3.3 The Live Performance ...... 52 3.3.1 The Animation Process ...... 53 3.3.2 Simulation vs. Reality ...... 53 3.3.3 The Synchronization Software ...... 55 Conclusion ...... 55 Works Cited ...... 57 Appendix 1 – Sync Software Code ...... 60 Appendix 2 – RAPID Code ...... 70
Creative Robotics Studio 4
Abstract
As robotic systems become more integrated with society, it is important to understand how humans perceive and accept robots. The Creative Robotics Studio (CRS) aims to accomplish four objectives throughout the course of the project: 1) Better understand how robots can be perceived as agents that have emotional and creative capacity. 2) Design a performance consisting of a narrative sequence of robotic gestures that evoke human empathy and emotion. 3) Draft a toolset which allows artists to realize their creative vision using an industrial arm and animation software. 4) Execute a performance using our toolset to display how the robotic arm gives the artist new creative opportunities.
Creative Robotics Studio 5
Acknowledgements
This project would not have existed in its current form if not for Andy Flessas’ (andyRobot) insights and perspectives on performance art with industrial arms. Our interview with Andy was pivotal for us and resulted in a new direction for the project. Andy gave us good advice going forward and valuable information regarding how his work functions in a real-world, social context. WPI’s Computer Operations Manager Randolph Robinson was a last-minute hero for our project, working on short notice to resolve sync software and IRC5 system network issues. If not for Randy we would not have been able to display our live performance. We would like to thank our three advisors on this project: Scott Barton, Craig Putnam, and Joshua Rosenstock, each of whom contributed insight and offered valuable time to make sure this project kept moving. Professor Barton was instrumental in keeping us considering the higher-level ideas and providing extensive feedback on our writing, surveys, and performance previews. Professor Putnam was a huge help with the robot hardware and design, taking time out of his schedule to supervise our live performance developments, and giving us pointers and resources to take our performance from animation previews to real-world execution. Professor Rosenstock was constantly pushing us to consider new and creative ways to incorporate and modify performance elements given our software, hardware, and time constraints. This project was possible because of their guidance and we are grateful to have worked with them.
Creative Robotics Studio 6
Introduction
At first glance, the fields of digital animation and industrial robotics appear to have little in common. Yet, as soon as one asks an animator about inverse kinematics or a roboticist about natural motion, the intersection becomes more evident. Both of these high level concepts heavily influence the designs and considerations of both fields. However, the abstract high level concepts need to be rationalized into real, physical examples in order to show off the intertwined nature of the two fields and how it might open a new realm of opportunities for artists and engineers alike. In general, the presence of robotics in everyday life grows a little more every year. For industrial applications robots are nearly ubiquitous, performing many tasks faster and with more precision than any human worker. The annual supply of industrial robots has nearly tripled in the last fifteen years1 – higher supply and higher demand means that availability is increasing and cost is decreasing. As a result of this, more artists obtain the opportunity to work with industrial robots. The versatility of industrial robots makes it possible to include them in a variety of unexplored artistic and performance concepts which would previously have been difficult or impossible. The music scene has soaked up a great deal of technology and novel ideas, so much so that there is a major conference dedicated to novel instruments called NIME: New Interfaces for Musical Expression. The outlandish, unique and inventive musical interfaces coalesce here to create something great. Innovators and creators have always pushed the envelope of their mediums so it was only a matter of time before robots were recruited to act as part of someone’s artistic vision. Soon enough someone had the idea to use industrial robots as the headlining performers in a fully choreographed audio-visual experience called Box.2 In context, simply the existence of an industrial arm on a performance set is impressive. For the arm alone, costs can run up to $80,000 and a full system may cost up to $150,000.3 In addition, hours upon hours of training are required in order to operate the robot and its systems safely. To pull off a performance like ours, in which a robotic arm must be animated to move in time with visuals and music, a team needs to draw from versatile individuals who can effectively practice and articulate multiple principles of design and techniques. Animation, robotics, mathematics, music, and composition hang in a balancing act – a weakness in one area will be evident in the final production. Access to equipment and resources like this are severely limited, which is why performances using industrial robots are few in number. Access to sufficient skill to manage all of these disciplines is even more limited, but not impossible. Box is a good benchmark of what is possible using unconventional hardware. Its creators, Bot and Dolly, developed a proprietary software called “BDMove”, a plugin for Maya which bridges the gap between animation and industrial robotics.4 Andy Flessas, alias andyRobot, shook the grounds of performing arts when he used his knowledge of robotics, music, effects design, animation, and programming to back musicians like Deadmau5 and Bon Jovi, and create a unique performance experience for the Oasis of the Seas cruise ship. Much
1 “Executive Summary.” 2 The Creators Project, Box. < https://www.youtube.com/watch?v=lX6JcybgDFo > 3 https://www.robots.com/faq/show/how-much-do-industrial-robots-cost 4 The Creators Project, Box by Bot & Dolly | Behind the Scenes. < https://www.youtube.com/watch?v=y4ajXJ3nj1Q >
Creative Robotics Studio 7 like Box, andyRobot presents visuals on large, robot-mounted screens that move and dance alongside human performers and music. Andy uses his own proprietary software called “RobotAnimator,” a plugin for Maya.5 Software exists to build practical bridges between disciplines, but it is rare and proprietary, often designed to solve a specific problem in a specific piece rather than provide a general implementation of an adaptable toolset. While bridges exist, their accessibility and publicity are not exactly in a healthy state. Previous performances and experiments have created amazing works to close the gaps between composition, performance, and engineering, but they stay inside of a bubble. The tools these artists create are proprietary and the knowledge they cultivate largely stays within their team. It seems as though someone who seeks to engage with these fields would have to attain years of experience with multiple technical and artistic fields and join a very select few in order to realize their vision. The problem is that these resources are difficult to learn and are still rare – arguably the time investment required to complete a project like this is greater than the monetary investment. More people need access to this new hybrid field to bring life into it. Conferences such as Siggraph and NIME show how experts in the fields of graphics and music are hard at work innovating and bringing something unique every year. Industrial robotics in performance is a novel 21st century concept with huge potential in the coming years. It requires deeper exploration and additional public awareness to bring it into the forefront of public interest. Another major factor that influences the direction of this new industry is how society sees robotics. The field is still relatively young and relegated mostly to utilitarian use. Robots, especially industrial robots, are surrounded by a stereotype that they belong inside a factory, welding pieces of cars together and taking away the jobs of factory workers. As we will discuss, robots are traditionally seen as threatening entities because they perform tasks with no perceived emotion or rest. At this time, it is difficult to predict how society will collectively adapt to the increasing presence of robots in daily life, but it is extremely important that society eventually sees robotics as a new medium for expression rather than simply a mechanical representation of our destruction. Industrial robots have the potential to fit into almost any role imaginable - any application in which the human arm or arms create something or solve a problem - art, music, architecture, engineering, medical and biomedical, etc. - the industrial robot can be trained to perform the same tasks with greater accuracy and repeatability. As a tool for human expression, the industrial robot can bring a seemingly fantastical feat into reality. This project aims to increase society’s awareness of industrial robots as tools of human expression and progress. Industrial robots are advanced but severely limited in real world use. They have the same potential to affect society as the internet did at its inception. We will create a kinetic performance in order to better understand how society perceives robotics as agents with emotional or creative capacity.
5 http://www.andyrobot.com/index.html
Creative Robotics Studio 8
1 Background
1.1 Relationship between Robots and Humans
In order to understand the more intricate interactions between humans and robots, the scope of the relationship between humans and robots must be defined beginning with the function of robots and generally how they currently fit into society. This is a broad topic, but it provides a useful framework for the basis of our project - the social climate has everything to do with how humans perceive and process the information from their senses. The primary use of robots in our society is to tackle the “three Ds:” tasks that are too dull, dirty, or dangerous for humans.6 Examples of these tasks include repetitive industrial, factory line jobs, sewer pipe inspections, and aerial military surveillance for hours longer than any human could perform in one sitting. These tasks are all undesirable to humans, and as such have been delegated to robots. More recently, as technology has advanced, robots have been used more broadly for human replacement. Robots can now tackle situations that they never could before due to technological restrictions. They fill certain roles that humans cannot for one reason or another, often with the intent to reduce the potential risk to humans.7 Robots can access locations too dangerous for humans, with more mechanical precision and processing power. In some cases, they can even tackle problems without emotional limitations. Robots are involved in nearly every area of our lives including labor, military, and entertainment. Examples like the Roomba show how they have even made their way into general labor at the consumer level. The media popularizes robotics as a result of their role in the military and scientific research. Additionally, hospitals and nursing homes use robots for medical and personal care, especially those in more robot accepting countries, such as Japan. They are on the verge of breaking into the entertainment industry as well. Robots are increasingly being used in the field of “Edutainment:" educational entertainment. For example, a KUKA industrial arm was programmed to "simulate" surfing by moving a surfboard around as a surfer rode it. The robot is necessary in these applications to hold the body weight of a person and react to the person’s movements, providing valuable interaction not available in, for instance, an arcade game. It can be used for amusement as well as for learning purposes. In the near future, robots could easily be used for tasks uncommon today8 In general, however, jobs involving labor seem to be the primary examples of how robots can perform human tasks, which often works against the notion that robots are designed to improve workplace safety and remove the need for humans to perform machine-like tasks. The industrial arm has the potential for more than work for work’s sake - as an artistic tool it can bring an entirely new realm of rapidly prototyped creativity and expression under the control of the artist, the musician, the scientist, or the architect.
1.1.1 “Human Replacement”
As much as robots may improve the standards of living by working efficiently, safely, and inexpensively, many people have an inherent fear of robots or robotic technology. Littered throughout history there are examples of artists’ and mechanics’ attempts to animate humanoid figures, as well as examples in fictitious literature of biomimetic creations going horribly wrong or whose existence is outside of our comprehension. Such examples include Da Vinci’s robotic
6 Lin, Abney, and Bekey, Robot Ethics. 7 Ibid. 8 Dautenhahn, “Roles and Functions of Robots in Human Society.”
Creative Robotics Studio 9 prototype, marionettes, and automatons. From literature and vernacular legend come the helpers of Hephaistos, clay golems, and the famous Frankenstein’s Monster.9 More recent works of science fiction like the movies Terminator or The Matrix even propose a future in which machines surpass humans and seek to become dominant. The real world equivalents of such examples are the predator drones employed to watch over and attack almost any target imaginable. The concept of strapping weaponry to an unmanned aerial vehicle and allowing it to act autonomously brings into question the concept of responsibility as it applies to robotics: who is responsible for the actions of the robot – the human creators and controllers, or the robot itself? Arguments about this topic include the apprehension that drones have “a capacity for insubordination” and a potential that the use of robots may allow human operators to more easily justify morally reprehensible decisions.10 While mimicry like this is classically viewed as a threat to humanity on many levels, a more immediate concern with robots is human replacement. Machines perform their intended task perfectly every time, tirelessly, and often much faster than a human counterpart. Industry workers are already facing replacement issues in higher-skilled craftsmanship positions. Because the development of robots shows no signs of slowing, and automation in higher-skilled industry is increasing, robotic replacement is more of threat now than ever.11 The 21st century grasp on material sciences, mathematics, and biological processes all contribute to deeper layers of realism and additional capabilities in robots that were previously impossible. Computer technology makes them even smarter. For instance, the Yaskawa Bushido Project uses motion capture software to reproduce a sword master’s movements using a Motoman robotic arm. With adjustment, it is able to perform all the tasks the sword master can, and towards the end of the performance its consistency contrasts with the sword master’s fatigue. The use of a weapon in the performance may further the perception of robots as “tireless killing machines.” However, robots are making their way into fields other than industry and labor and making names for themselves as a creative medium rather than machines with a set, repeating routine. For example, two ABB robots in Berlin and London replicated artist Alex Kiessling’s pen strokes in Vienna during the Long Distance Art event. The artist was not replaced, but enhanced by the robots as precise, extremely useful tools acting as a new medium of communication. However, the tendency humans have to focus heavily on dramatic and tragic occurrences has surrounded the industrial robotics field with skepticism. The divide between what can be considered art created by the robot, and art created using a robot becomes more and more difficult to distinguish and it remains extremely open to interpretation. Hopefully this works to show the benefits of incorporating robotic elements into artworks and adding another dimension to the creative process.
1.1.2 Humanizing Robots - Anthropomorphism
One of the earliest skills that humans learn is that of anthropomorphism: the tendency to ascribe human-like qualities to human-looking objects. As children, we learn that most things that look like humans are in fact humans and have a certain set of qualities attached to them: sentience, emotions, personalities, and so on. A side effect of this psychological shortcut is our tendency to unconsciously ascribe these qualities to anything that displays similarities to the human form, even if we can consciously recognize that the subject in question is non-human (or, indeed, not alive at all). This quirk has been exploited throughout human history in fields as varied as advertising, art, literature, and even religion.
9 Bar-Cohen, Hanson, and Marom, “Introduction,” 7,8. 10 Pasquale, “The Doubtful Ethics of Drone Warfare.” 11 correspondent, “Are the Robots Taking Enough Jobs?”
Creative Robotics Studio 10
A robot modelled after human anatomy will ideally appear as humanoid as technology allows; therefore, it follows that these robots are subject to a high degree of anthropomorphism. On the most basic level, this manifests as assumptions as to the functions of its more human- like features: “eyes” are assumed to be used in visual sensing, “hands” for use in tactile manipulation, etc. This phenomenon can be exploited to make robots seem more human than they are. Outfitting a robot with nonfunctional structures resembling eyes, lips, ears, or other sensory structures gives the impression that the robot is capable of processing the associated sense, even if the robot is incapable of these senses or the senses are processed in a separate device. RoboLounge’s tribute to Kraftwerk12 exploits this to make their robots seem more human. Their performance focuses on three robotic arms, each with an end effector composed of three lights in a triangle arrangement. The orientation of the robots, with their base grounded to the stage and their end effectors at the top, helps give them the impression of human form - the lights are perceived as eyes and mouths. These perceived facial features, the focus directed at the robots, and their synchronized choreographed movements to the music characterize the robots as dancers, a very human form of expression. People also respond socially to computers that show social cues, as shown in several studies involving computers giving positive comments, computer tutors, and computer generated speech. Although people may claim to not humanize computers, they still treat them as social beings in the right context.13 In “The Revenge: Timo Boll vs. KUKA Robot,” Timo Boll and a KUKA robot face off in a musical competition playing the glass harp. The robot’s only humanoid feature is a robotic hand as the end effector. Throughout the performance, the robot “taunts” and “challenges” Boll with humanlike hand gestures such as the “thumbs down” and the “fist pump.” While these gestures are merely programmed for the sake of the performance, the audience understands their meaning because they are widely used expressions. Since we recognize them as human expressions, the robot seems more human as well.14 Another component of anthropomorphism is the tendency for humans to evaluate humanoid animals or objects as humans in an ethical framework. For example, in a study by Riek et al, participants were mostly indifferent when clearly non-human robots such as a Roomba were kicked and insulted by the survey administrators, but expressed almost twice as much sympathy when the exact same treatment was administered to a humanoid android.15 A recent example shows how something as simple as balancing and adaptation benchmarks of Boston Dynamics ATLAS2 can easily be interpreted as abuse. Additionally, anthropomorphism is not purely based on something’s appearance – gestures and communication also play a large role. The way an object moves can influence the way people think about it: for example, a robotic arm moving between two points in a wild, fast arc may be perceived as aggressive and scary, while the same arm moving between the exact same points in a slow, deliberate fashion may be seen as peaceful and non-threatening. Additionally, the recreation of human nonverbal communication methods, like a thumbs-up or waving gesture, can make a clearly non-human robot more palatable to a human audience. Although humanoid robots such as Atlas and ASIMO are fairly recent developments, the idea of a robot that mimics the form and function of a human is nothing new. In fact, the idea is literally as old as the word "robot" itself, which was coined in 1921 by Czech writer Karel Čapek for his play Rossumovi Univerzální Roboti ("Rossum's Universal Robots"). R.U.R. tells the story of a factory that produces humanoid "robots" (from the Czech robota, meaning “forced labor”) using synthetic biological materials for use as slaves. These robots eventually overtake the
12 Robolounge, Robot Dance. 13 de Graaf, Allouch, and Klamer, “Sharing a Life with Harvey.” 14 Fussell et al., “How People Anthropomorphize Robots.” 15 Riek, Rabinowitch, Chakrabarti, and Robinson, “How Anthropomorphism Affects Empathy Toward Robots”
Creative Robotics Studio 11 world economy, mount a global rebellion, and exterminate humanity. This view of humanoid robots - that they are inherently dangerous tools intent on destroying and replacing humanity - set the tone for robots in science fiction for decades to come. Furthermore, since actual "robots" as we would recognize them were still almost a full century away, the word had the better part of the 20th century to become associated with genocidal killers in the public mind. Predictably, this has led to public opinion of humanoid robots being wary at best and outright distrustful at worst. This is partially due to the previously-mentioned fear of human replacement and partially due to the fact that real-life robots have only recently become advanced enough to approach the same level of anthropomorphism as their fictional, murderous pop-culture depictions. Recent features such as Wall-E and classics like Star Wars feature positive examples of anthropomorphism in robotics, but they are few and far between. Anthropomorphizing these more recent robots in a positive light is a necessary first step towards changing public opinions about advanced robots and their place in society. For our project, it is important that we see the robot as a tool controlled by an artist. It takes an artist to create something that appears to live and feel emotion. By anthropomorphizing a robot, the audience can find deeper meaning behind the robot’s actions and interact with the robot on a more human level.
1.1.3 Uncanny Valley
Questions about humanizing or anthropomorphizing robots cannot go without a discussion and understanding of the uncanny valley. Mediums that replicate something about humanity – movements, likeness, and behaviors – are in danger of approaching what is known as the “uncanny valley” to artists and roboticists alike. Roboticist Masahiro Mori coined the term in 1970 to refer to this phenomenon.16 It is difficult to properly describe the uncanny valley because it refers to an identification process that varies between people. However, we generally understand it as a point at which humans begin to identify less and less with a depiction of something as it becomes more realistic. Sometimes this phenomenon can adversely affect the creation, making it unsettling or upsetting to observe. The valley describes how an imitation can actually create a negative impact on the observer. Prosthetic limbs might produce such a sensation – when the viewer realizes the realistic, fleshy hand they are about to shake is lifeless, they develop an aversion to it.17 Robots that act human and look human are seen as, essentially, reanimated corpses. The sensation is at its worst when the doppelgänger is nearly perfect, but just slightly off – “monstrous doubles” according to the anthropologist René Girard.18 This commonly occurs with appearance alone, but movements and perceived behaviors can also invoke this sensation. This is why robot designers, for the most part, try to diverge from the human form in order to avoid the valley altogether. Industrial robots, while seemingly so different in design from any human forms, are modeled after the six degrees of freedom in the human arm. Even industrial robots, properly orchestrated and characterized in motion, have the potential to mimic any number of human motions and produce the unease described by the uncanny valley. For our considerations, the uncanny valley either obstructs or helps us accomplish some form of expression. In general we need to minimize the effect of the uncanny valley in our performance because we want to show the robot as a tool for human expression without alienating viewers. However, the uncanny valley does add a dimension to the medium. So, it may be possible to make use of the uncanny valley to invoke a certain response in viewers if
16 Mori, MacDorman, and Kageki, “The Uncanny Valley [From the Field].” 17 Ibid. 18 Grinbaum, “Uncanny Valley Explained by Girard’s Theory [Turning Point].”
Creative Robotics Studio 12 the performance calls for it. Otherwise it should be wisely avoided. We do not want the robot to pretend to be human because that defeats the purpose of using the robot in performance. Instead, we want the robot’s movements to appear natural without turning into a disturbingly mimetic performance. Girard claims that interaction between robots and humans must involve some appearance or transmission of emotion, otherwise the divide between us will always remain.19 This means we have to carefully articulate how we show off our industrial arm in performance. The animator mitigates the effect of the uncanny valley by embellishing the movements of the characters and props in such a way that appears fantastical to the viewer. The animation sells itself as intentionally divergent from the human standard which comes across as nonthreatening to the observer. Examples might be an over-exaggeration of secondary motions as the robot might appear to hold the weight of an item. Dramatic, sweeping motions can keep the arm from triggering aversion in the viewer.
1.1.4 Computers and Digital Technologies in Art
The emergence of the digital era in the mid-20th century with digital transistor-based computer systems began the digital artistic movement as well. One of the earliest examples of digital art was a vector-based pin-up image programmed by an anonymous IBM worker using a military computer between 1956 and 1958.20 In 1963 Ivan Sutherland invented the first rudimentary computer-based drawing program which signified the beginning of a more intuitive interface to create digital art.21 Up until that point, images had to be “coded” as sequences of vectors. However, the new medium created its fair share of controversy from conservative artists and critics in the same way that prints and photography did when they first appeared in publications. Critics of the time did not know how to judge this new medium because it was incomparable to classical examples using the “historical or psychological yardsticks”22 employed to review and understand artwork. Generally, based on unstable or unsupported reasoning, computer-art faced the criticism that it could not be art. Other reviews attempted to reconcile the execution and design choices behind the digital imagery with the traditional elements exemplified by those images – especially in choices of color and texture in physically based 3D rendering.23 The use of robotics and portrayal of robots as works of art mirrors the state of digital art when it was in its infancy. Just as computers created an entirely new medium that required many years of hard work from artists and programmers to realize greater expressive potentials and remove limitations, so too does robotics. We have to use this to our advantage, but we also have to understand that the medium is still greatly unexplored and therefore foreign to society. Because examples of robotics in art are still relatively few, people do not know to how to react to robotic performances in the same way they "know" how to react to classic examples of art and performance. As with the birth of digital computer art, the current social "yardsticks" for judging and responding to art do not apply.
19 Ibid. 20 Lee, “From a Pin-up Girl to Star Trek’s Holodeck,” 1. 21 Ibid. 22 Bentkowska-Kafel, Cashen, and Gardiner, Digital Art History, 153. 23 Elkins, “Art History and the Criticism of Computer-Generated Images,” 338.
Creative Robotics Studio 13
1.1.5 Robots in Art
So, with these issues in mind, what have artists and roboticists already accomplished and what do their accomplishments mean in the context of our project? A key starting point was to obtain some insights and perspective from Andy Flessas – the groundbreaking artist in robotics previously mentioned. Andy is an expert animator and roboticist who has brought industrial arms to public attention in performances with Bon Jovi and DeadMau5. He views robots as an important and natural technological advance for artists: “I think robots are as natural as, like, a bee making a honeycomb… robots are this natural progression of humans - blooming out of us; our expression.”24 He cautions, however, that robotic performances are not about the robot – “A lot of failures happen when people try to make it the robot show. It’s not the robot show.” Robots provide an opportunity for artists to engage with an audience in novel ways. When done well, Andy says “Robots and art unpack a joy...in people” with “young kids and old people just...loving the robot.” Key to Andy’s performances is Robot Animator – the software he developed that makes it possible to generate more fluid and graceful movements for industrial arms using Maya, an existing software commonly used by animators. He affirmed that “People do not like ballistic motions” and that he wanted to minimize audience fear by reducing such motions. This will be an important consideration as our project focuses on the interaction between the robot and the audience, and portraying the robot naturally and creatively. We subsequently analyzed performances that have used robots for creative expression. These demonstrations exemplify the larger concepts we examine in our project in an easily understandable and observable manner.
1.1.5.1 Yaskawa Bushido Project25
Figure 1. An industrial robotic arm replicates sword master Isao Machii's sword techniques.
As mentioned previously, the Bushido project uses motion capture software to reproduce a sword master’s movements using a Motoman robotic arm. This is a demonstration of how human motion capture can train a robot to perform mimetic actions in a parameterized,
24 Flessas, Interview: Andy Flessas. 25 【安川電機】e-メカサイト, YASKAWA BUSHIDO PROJECT / Industrial Robot vs Sword Master.
Creative Robotics Studio 14 adaptable fashion. The resulting robotic movements are smooth and precise, performing exactly as expected, however the lack of stylization and the perfect motionlessness at the end of each movement still appears very mechanical to the watcher. Directly mimicking a human falls into the uncanny valley and appears very stiff without an additional layer of tweaks to bring out the distinct character of the movements themselves – the slight errors and corrections from the human give life to a swing of the sword. Demonstrations like these, in which the robot faces off against a human opponent, contribute to the perpetuation of the “human replacement” fear that exists in society.26 Andy Flessas made the point in our interview that showing off a robot which effectively uses a dangerous weapon instills in the viewer a sense of unease or discomfort regarding replacement - “People are afraid of it.”27 In learning from this we have consciously steered away from the use of any prop which could be considered a violent or lethal weapon, or performance ideas that point out how the robot in some way usurps the humans in its apparent superiority. Replacement is not what this project is about, but rather unique creative utility. The prop itself, the sword in this case, has the unique ability to create a set of associations for the audience which we strive to minimize. The motion capture is a good prototype for other learning applications such as dancing or painting, which would better demonstrate how an artist might scale up their creations by taking advantage of the unique combination of precision and strength possessed by the robot – imagine a large industrial arm holding fifty gallons of paint, but still laying down the same flowing effortless brush strokes as the artist who taught it.
26 Bar-Cohen, Hanson, and Marom, “Introduction.” 27 Flessas, Interview: Andy Flessas.
Creative Robotics Studio 15
1.1.5.2 Robotic Artist at Long Distance Art event28
Figure 2. Alex Kiessling creating art while his strokes are sent to remote robots in real time.
The premise of this example is simple yet important: Alex Kiessling draws in Vienna while robots in London and Berlin copy his movements. This performance is a perfect example of how the robot can be applied as part of the artist’s medium of expression. Although the artist is only copied by the robot and not teaching the robot the movements, the resulting image is clearly recognizable as a work by artist Alex Kiessling. Application of the medium in this way is an extremely positive example of how industrial arms can make the experience of live art accessible to more people. Incorporating the industrial arm into the drawing experience introduces another layer of intrigue as well because even with its precision, the robot does not perfectly copy the artist as a result of errors in placement or delays in the code. The robot itself creates another layer of expression.
28 ABBRobotics, ABB Robotics - Robotic Artist at Long Distance Art Event.
Creative Robotics Studio 16
1.1.5.3 Robot Surf29
Figure 3. Programmer surfing a KUKA robot.
In its barest form, robot surf is an entertainment-based application for an industrial arm designed as more of a benchmark than a performance. It may not appear to be as important as the previous examples, but this demonstration shows the arm as a stable and safe platform capable of providing an experience the other examples cannot. Direct interaction between the audience and the robot like this is not generally feasible, especially in our case with the limited resources we possess, but it would be one of the fastest ways to change someone’s mind about the robot – assuming nothing goes wrong with the robot. The demonstration prototypes a performance that requires not only the robot, but the human participant to fully realize it. An exercise like this would very quickly build the audience’s trust in the robot as a readily available and interactive platform. Regardless of what the robot actually is, it is how the audience perceives the arm that matters because ultimately they control how the performance’s influence spreads and reaches out to others.
29 José Augusto Ribeiro Costa, Robot Surf.
Creative Robotics Studio 17
1.1.5.4 Blue Man Group and KUKA Industrial Robots for Factory Automation30
Figure 4. Blueman group performing with KUKA Industrial Robots.
In this example two automotive industrial arms are literally performers and characters on stage with the widely known and acclaimed Blue Man Group. The scale of this example is much larger than the others, and presents the industrial robots in a completely novel and compelling fashion. There is an irony that the robots begin by appearing next to the chassis of a car, in their intended working environment, but break away from this task and interact with the human performers up front, actively freeing themselves from the implied toil of the industrial setting to interact with the group in a performance setting. The robots practically disappear as their characters take over and we begin to anthropomorphize their actions and their appearances. Their “faces” and even the pop-culture reference to Slash, the easily recognizable guitar player for Guns n’ Roses, give the audience a focal point and a set of associations that have nothing to do with robotics, misleading their attention. And because the robotic arms playfully interact with the performers rather than compete for superiority, they come across as almost endearing as they move with the rhythm and contribute to the spectacle by dancing and playing music. Everything ties together in this performance. The percussive and raw nature of the music adds to the excitement and gives a solid rhythm for the performers and the robots. The gestures brought into the choreography mask the presence of a mechanical entity, but the audience remains aware that these are robots. We strive to achieve the same effect in our performance. In this way we can rearrange the associations the audience makes with regards to what the industrial robot fundamentally is. We will explore how we can control the perceived relationship between the arm and the performer to see how it affects the data. The level or depth to which the audience anthropomorphizes the robot is one of the most important aspects we can manipulate to control how the performance impacts the viewers.
30 andirobot, Blue Man Group and KUKA Industrial Robots for Factory Automation.
Creative Robotics Studio 18
1.2 Gesture
Gesture is a conceptual medium that defines the union between form and motif. Any observable element of performance such as poses and movements in dance, or filter sweeps in music become gestures when linked to abstract concepts like emotion or meaning. For example a gesture sketch in art is meant to capture the energy of a figure with as few lines as possible – the more the figure can be abstracted into indications of movement with lines, the easier it becomes to re-construct a rendered piece that emulates the energy of the original figure. In this example the observed elements capture the abstract idea of energy. One of our objectives in this project is to synthesize the data we gather from feedback on the performances and construct a library of gestures that classifies and organizes movements based on common themes in audience responses. This task is difficult because interpretation, especially emotion, is nearly impossible to solidify in a tangible form like the movements of the robot.
1.2.1 Using Props
A prop may be used to fit a certain theme, as shown in RoboLounge’s Electrobot.31 The lightsabers and white-painted robots resemble Stormtroopers in Star Wars, displaying a sci-fi and futuristic theme for their robots and evoking a sense of nostalgia in the viewer. In another case, the Yaskawa Bushido Project uses a robot with a sword to perform several sword techniques alongside a human sword master.32 The sword as a prop is used to demonstrate the robot’s precision and shows that a robot may copy a human’s skill accurately. For our presentation, we want to stress that robots can be used artistically and non-violently, so we will not use weapons as props. The Yaskawa Bushido Project addresses this with the robot’s “bowing” and sheathing of the sword to show that the skill must be used respectfully. A prop can also be used for personification and movement exaggeration. For instance, the Blue Man Group performance involved industrial robots with a clamp end-effector and red lights above it, resembling eyes and a mouth.33 One of the robots even “wore” a wig and hat at one point. The performance used the props to present the robots as actors, character and personality, in the show. They playfully hold and move objects just out of the person’s reach. The wig also emphasizes the robot’s movement; you can see its “head” shaking the hair as it moves up and down. A stronger example is a ventriloquist using a puppet, controlling their actions and speech to portray a particular character. In a similar way, a robot controlled by people (not necessarily in real time), should express a certain character if the aim is personification. In general, props give personality to the robot that would otherwise leave the audience without a familiar object on which to focus. In most cases, stage props provide a cheaper alternative to more expensive implements like monitors or projectors, but might constrain the expressive range of the artist based on the implications associated with the use of the prop. For example, the lightsaber creates a focal point and a set of associations while drawing attention away from the robots themselves. It depends on the artist’s intent, but in our case, we intend to feature the robot rather than the prop.
31 Robolounge, Robot Dance. 32 【安川電機】e-メカサイト, YASKAWA BUSHIDO PROJECT / Industrial Robot vs Sword Master. 33 andirobot, Blue Man Group and KUKA Industrial Robots for Factory Automation.
Creative Robotics Studio 19
1.2.2 Choreography
In the performance with one or more robots there exists a question of choreography for both the creator and the viewer: what about the choreography of a system makes the performance captivating? In conjunction with the audiovisual stimulation, there is also a distinct perception of coordination and rhythm based on natural pattern recognition tendencies in humans. Two primary principles of dance and movement in performance are synchronicity and divergence between the audio and visual stimuli presented. From a psychological standpoint, synchronicity is relaxing and “proper” to humans. The brain will devise a way to consume the least amount of energy to process incoming information, which means that any patterns between stimuli which can be recognized, will allow the brain to settle down from an alert state. Continuous and unchanging rhythms and synchronized movements allow the viewer to spend little effort processing the complexities of the music or the dancers moving out of time or sync with each other. We begin to see the unified nature of the performers across the stage rather than the multiplicity of the performers in individual locations.34 RoboLounge fully exploits the effects of synchronicity in their performances. Their choreographers paid close attention to moments at which they want the robots to diverge, holding the viewer’s attention before falling back into perfect synchrony.35 These motifs appear in modern dance as well, but with the robots it is much simpler to achieve. The designers of a robotic performance have to keep in mind that robots performing in synchrony have a much different effect on viewers than human performers. Human dancers always introduce their own element of chaos to the performance which means that they are never quite in synchrony with one another. To the audience, those little imperfections keep the mind alert to the movements. With robots, the movements are so precisely identical that they could easily become boring. We are used to seeing humans diverge in behavior and robots synchronize like machines. If we are to show how robots are a form a human expression then there needs to be some level of divergence involved. The accidents and mistakes from the artists and performers connect with their human audience more than the perfection of robots. From a choreographic standpoint, we should expect to see some kind of human choice behind each movement. In the same way that industrial arms in the factory have intent behind every maneuver, so does the dancer and the musician, and therefore so should the industrial arm in art. Beyond art, however, the choreography applies to other applications of the arms like architecture or medicine. We need to show that the robot works with humanity rather than against it or as its replacement.
1.2.3 Movement
In general, smooth movements appear more familiar to people, so the bridge between the audience and the robot can be dynamically controlled to match a performance aspect.36 If the performance calls for audience participation, then its movements will be smooth and graceful; non-threatening. Conversely if the performance needs to portray distance or alienation, the robot’s movements may be made jerky and sporadic. This effect may be amplified if any of the audience has not had a lot of time to interact with robot(s)37. How “sociable” or “familiar” the robot appears in performance could be an interesting experiment to include in this project, and may affect how the performance evolves during development.
34 Haas, “Synchronicity in the Performing Arts.” 35 Robolounge, Robot Dance. 36 Baddoura and Venture, “This Robot Is Sociable.” 37 Ibid.
Creative Robotics Studio 20
The “posture” of the robot must be taken into consideration for every movement after considering the quality of the movement itself. The jittery movements and the smooth movements can invoke one dimension of emotion, but the additional layer of posture reveals more dimensions for emotional expression. It is understood that closed-in hunching poses are a sign of sadness and that open and outstretched poses show happiness38. It is important, however, to make a choice about posture in performance: are there sections in which any emotion at all is to be invoked, thus humanizing the robot, or does the robot perform mechanically or neutral? This choice matters because it is perhaps the only way the robot can “communicate” with the audience. The robot and the music will “communicate” with each other much in the way that a conductor and an orchestra communicate. In this case, the robot’s movements and posture will be determined based on the music rather than directing the dynamics of the music as the conductor. However, the importance is in the exchange that occurs between the music and the robot – the effect of forte phrases may be amplified with quick, sweeping, and even angry movements as a visual metaphor for the increase in amplitude. Likewise softer phrases can be conveyed with lilting and calm movement39. By breaking out of one gesture suddenly or ceasing movement altogether the robot can create a visual distinction that the sound cannot. Intuitively, it is possible to describe sounds with identifiers used for visuals or movement. For example, saw waves and square waves sound “sharp”, and sine waves sound “smooth.” Likewise, music can be abstracted in the same way - techno is commonly “aggressive” where classical is more “lilting.” In the performance, the motions of our end effector and the robot itself have to match the way the audience expects to see movement based on what they hear. The viewer needs to see the artist behind the robot rather than fixate on the robot itself - gestures that appear natural can accomplish this.
1.2.4 Music and Emotion
Music can have a massive effect on our emotions. How can sound bring about these feelings in us? It cannot be that we are subconsciously associating the music with events; otherwise each person would have a unique reaction to a piece. It is very apparent that pieces tend to evoke the same or similar emotions in many different people. It could be that the words of the songs are bringing about certain feelings, but that would not explain how we react to songs in other languages, or even more oddly, songs without lyrics.40 Why music can bring about emotions in us is still relatively unknown. What can be explained is how the music creates the emotions. The main factors contributing to emotion elicited from a piece of music are the piece's tonal and rhythmic structure. This includes the piece's tempo, key, and pitch. In addition, a piece's timbre has a large impact on the emotions elicited. Dissonance, which is much harder to quantify, is the amount of disagreement, both rhythmic and tonal, between various parts of a piece, both rhythmic and tonal. Each of these factors can be linked to a specific feeling or set of emotions. For example, higher pitches, major keys, a soft timbre, faster tempos, and a lack of dissonance are associated with being happy. Lower pitches, minor keys, a soft timbre, slower tempos, and a slight presence of dissonance are generally associated with being sad. Different combinations of these qualities can result in different feelings.41 Having more dissonance, and lower pitches, for example, tend to lead to a more uneasy feeling. This can be sadness, as
38 Han et al., “Emotion Gesture Art.” 39 Stam and Ishino, Integrating Gestures : The Interdisciplinary Nature of Gesture, 350. 40 Krumhansl, “Music.” 41 Meyer, Emotion and Meaning in Music.
Creative Robotics Studio 21 mentioned earlier, or fear if there is a harsher timbre and a faster tempo. These factors are important to take into account when composing a piece or selecting one for use. What the intended emotion is should be a large determining factor in the creation or selection of a piece.42 For our presentation, techno or electronic dance music seems most appropriate. The reasons for this are twofold: first, the community of people interested in robotics and the community of people who are fans of techno have an extremely large overlap.43 The two communities tend to share an interest in the cutting edge of technology and how this technology can be applied in interesting ways. The second, and larger reason, is that as a genre techno does an excellent job of invoking a sense of excitement in the listener. One of the large goals of our project is to get people excited over the applications of an industrial robotics arm. Techno tends to have a large presence of both high and low pitches, a much faster tempo, a balance of soft and harsh timbres, and a decent amount of dissonance. These qualities all add up to produce excitement. The higher tempo leads to increased heart rate, a common sign of excitement. The high and low pitches as well as the balance of soft and harsh timbres appeal to people and can draw people in. The amount of dissonance keeps people engaged in the music, keeping the mind active.44 All of these qualities are key to helping the audience of our performance absorb as much of it as possible. While designing a robotic performance, the artist must stay constantly aware of how the robot “speaks” as it moves. This voice can characterize the robot as much as the movements themselves. The sounds created are unique and can be incredible assets to an entrepreneurial musician. The servos make a very whiny and mechanical sound that can be harnessed, as seen in Machine Yearning.45 These sounds provide a unique platform directly linked with specific actions for the viewer to see and hear. The audio accompaniment to any visual, or visual accompaniment to any audio should be coherently linked in the way one would expect to see from a musician. In the context of our performance and the purpose of this project, any emotion in the viewer from the experience has an effect on their perception of the robot itself.
1.2.5 Audio-visuals
The musical accompaniment to a video has a larger influence on the viewer’s emotions than the video itself. In addition, studies have shown that people prefer audio-video segments where the audio “matches” the video.46 This shows the importance of getting the music right for a performance. If the music doesn’t match, the audience will notice. This emphasizes the importance of correctly choosing a musical accompaniment. Mismatched music is distracting and, as a result, can easily detract from the visual aspect of a performance. The viewer’s interpretation is entirely dependent on how they perceive contextual elements of the performance. Juslin’s lens model portrays how the individual analyzes a number of “expressive cues” in a performance piece and combines them based on personal importance to come up with their own interpretation.47 In the interest of keeping our process as open-ended and adaptable as possible, we will not make any assumptions about the viewer’s reaction nor will we create a performance expecting to see a particular reaction. This is an iterative, data-driven process – for all intents
42 Evans and Schubert, “Relationships between Expressed and Felt Emotions in Music.” 43 Skelton and Valentine, Cool Places. 44 Eerola, “Are the Emotions Expressed in Music Genrespecific? An Audio-Based Evaluation of Datasets Spanning Classical, Film, Pop and Mixed Genres.” 45 Snyder et al., “Machine Yearning.” 46 Lipscomb, “Perceptual Measures of Visual and Auditory Cues in Film Music.” 47 Juslin, Handbook of Music and Emotion, 472.
Creative Robotics Studio 22 and purposes we are as naïve as the audience when creating and displaying these performances.
The aforementioned demonstrations and performances illustrate how industrial robots can exist within the context of human creative expression. Each performance has a different goal, ranging from sending a particular message about the robot and how it interacts with humans or society to simply showing a novel creative application for the robot that enables previously impractical or unachievable opportunities. For example, an experience like robot surf is entertainment-based, but could provide solutions to problems that no one can predict. Unfortunately, due to society’s inherent fear of robots, many of these solutions and opportunities will go unrealized. The goal of this project is to change that by using a robot- performed artistic display to show the public that industrial arms have creative potential rather than simply industrial. To facilitate this goal, we will develop an organized library of gestures for general use in future performances and demonstrations, constructed based on iterative audience feedback, allowing the robot to appear more human-like and accessible to the audience. Our project will address the lack of research regarding the interactions between humans and specifically industrial robots in a performance setting with a robotic performer. This Robot is Sociable examines the interaction between humans and a robot pretending to be a human.48 Machine Yearning addresses audience reaction briefly in its closing statements and Andy Flessas addressed this in his interview,49 but we need to know more from the audience itself. Machine Yearning gained the empathy of the audience, but the audience reaction was more negative – as if the robot was trapped.50 Andy’s artistic work does not involve robots as the performers themselves. We want to know how the audience sees gesture as expressed by the industrial arm in the hopes of creating new, unique opportunities where none existed.
2 Methodology
The overarching theme of this project, outlined above, is to positively bring the industrial robot into the public eye by displaying its artistic potential and by analyzing how audiences’ opinions change as a result. We decided to achieve this goal by designing a live, gesture-based and narrative-driven audiovisual performance for an industrial arm, showing our performance to an audience, and then analyzing their reactions and comments in relation to broader demographics. When designing the performance, we set out to not only demonstrate the visual and artistic potential of the robot, but also to convey as coherent of a narrative as possible. We wanted the robot to become a character with emotions and creativity. The more empathy the robot could gain from the audience, the clearer our message would become. As a character the robot has a better chance to cause our audience to begin to perceive the robot differently – and if we could accomplish this with our demonstration, then it could be possible for other artists with other audiences as well.
48 Baddoura and Venture, “This Robot Is Sociable.” 49 Flessas, Interview: Andy Flessas. 50 Snyder et al., “Machine Yearning.”
Creative Robotics Studio 23
Our objectives for this project are the following:
1. Better understand how robots can be perceived as agents that have emotional and creative capacity. 2. Design a performance consisting of a narrative sequence of robotic gestures that evoke human empathy and emotion. 3. Draft a toolset which allows artists to realize their creative vision using an industrial arm and animation software. 4. Execute a performance using our toolset to display how the robotic arm gives the artist new creative opportunities.
2.1 Surveys
Our live audience was specific and local, consisting entirely of WPI faculty and students. Given the prevalence of both robotics and engineering here, we anticipated that this audience might have some biases regarding the performance that would affect how well our data would extrapolate into general society. The first step, then, was to gather baseline data from people outside and inside WPI and compare the differences between the results. We hypothesized that WPI students are more comfortable with robots and robotics, so the background surveys should reflect that. Importantly, we needed a standard structure by which to evaluate or classify the responses of our audience members so that we could reasonably compare them with society as a whole. The three core questions we investigated through these performances and audience surveys were:
1. How does the audience perceive the creative potential of the robot? 2. How does the audience perceive the emotional expressiveness or emotional capacity of the robot? 3. How much does the audience trust the robot?
2.1.1 Questionnaire Design
We planned to use a questionnaire before and after our performance as one method to gauge the audience’s reaction. In the questionnaire we aimed to measure changes in the robot’s perceived danger and artistry. The questionnaire had to be designed to accurately and quantifiably determine an audience’s perception. We followed good design principles in our background survey to ensure that our participants have consistent interpretations of the questions. Questions needed to be clear and specific. The choice answers for closed-ended questions should be distinct and not overlap in any way. If possible, complex questions should be broken down into several simpler questions. The education level of the audience should be considered so that the questions are not difficult to understand.51 Double negatives, unfamiliar abbreviations, and slang should be avoided52. The connotations of the words can also have an effect (e.g. “welfare” vs. “assistance to the poor”).
51 Street et al., “Questionnaire Design.” 52 Ibid.
Creative Robotics Studio 24
Questions asked with the “agree-disagree” format (e.g. Do you agree or disagree with X?) should be reworded into a forced choice question (X or Y?). Less educated and less informed people, when given an “agree-disagree” question, are more likely to select “agree” (acquiescent bias).53 According to “social desirability bias”, people will answer questions in a way that makes them feel likable and accepted, making the answers inaccurate (e.g. How often do you go to class?)54. The effect is greater when an interviewer is present. Additional questions with choice answers reflecting the negative side (e.g. Have you ever missed class because of …?) should be asked to help the person give a more honest answer. Closed-ended questions should be answered after open-ended questions on the same topic are answered (to avoid “order effects”). If the closed-ended question is asked first, the person is more likely to bring up the topics in that question when responding to the open ended questions55. When a specific question is asked before a more general question a different response may occur (contrast effect). When responses in a questionnaire are conceptually connected because of the order of the questions in the survey, an assimilation effect occurs. The survey questions should be organized by topic and asked in a logical order that keeps the audience interested. Demographic questions should not be placed in the beginning unless they are used to determine eligibility for the survey or directing the person to certain parts of the survey.56
2.1.2 Background Survey Design
The background survey was designed to compare the demographic of our audience and the broader social demographic outside of WPI. In order to come to more accurate conclusions about the general opinion held by society regarding robotics based on our specific audience, we needed a background comparison between the WPI audience and the demographic outside WPI. In addition, by sending out a survey with questions related to the three core themes of our performance, we were able to become familiar with our limitations. For example, if our audience was more comfortable with robotic performances we would be able to take our performance in a different direction than previously imagined. We created a background survey with our three essential questions in mind, considering how each question would illuminate how a participant views robotics in varying contexts. The survey opened with some general literacy questions about technology, to associate whether or not individuals who have more experience with programming, social media, or modern personal computers are more likely to empathize with the robot. We extrapolated this data to show how the presence of technology in society has also increased the individual’s awareness and comfort with the concept of robots as an integral part of society. The body of the survey contained questions about hypothetical scenarios, and in some case very real scenarios, in which the robot assumes a role ordinarily foreign to it, such as the role of a musician or an artist. These are associative questions – creating an entire scenario with associated cultural and contextual expectations. The responses to these questions mostly test how much the participant views the creative potential and emotional expressiveness of the robot. In some cases we asked about trust, specifically for cases which require contact with the
53 Ibid. 54 Ibid. 55 Ibid. 56 Ibid.
Creative Robotics Studio 25 machine itself – we predicted that most people would be uncomfortable with this. Empathy and emotional expressiveness may be observed from a distance, but physical contact requires a level of trust we did not expect to find in general society. To start the survey design, we identified the key topics to address in the survey. We wanted to understand: how people perceive the creative potential of robots, how people perceive the emotional capacity of robots, how much people trust robots, and in what context(s) people typically imagine a robot. As an introduction, we asked the participant to name his or her favorite robot. This was used to get the participant thinking about robots and to check for a correlation with other responses in the survey. Then with a "technology literacy" test, we gauged the participant's grasp on the technological situation today. The General Technological Literacy questions test the participant on his or her involvement with current technology. The literacy questions show how willing the person accepts new technology and how "up-to-date" the person is with it. Next we asked the participant to estimate when certain technologies would become possible. The questions test the participant's estimate of technological progression. Some of these technologies – the robotic surgeon, autonomous industrial robots, and robotic artists and musicians – already exist today, and were used in the survey to further test the participant's awareness of robots in society. Since artistic robotics is a new concept, we needed to take into account the participant's biases toward new technologies. Afterwards, we presented the "how comfortable" questions, which ask the participant to rate their comfort in several situations. The first few questions in this area ask how much one might trust a robot with human tasks in a typically human context (as a maid, surgeon, or coworker).The responses indicate whether people are comfortable with the robot performing actions in contexts other than the stereotypical industrial setting. Next are situations that involve the expressive and creative capacity of robots – robotic performances in orchestra, electronic dance music, and plays. Positive answers to these questions would indicate that our audience could be more receptive to our robot performance. Then we presented the "robots and emotions" questions, which ask about the ability for a robot to express emotion. Answers to these questions may determine how difficult it is for an audience to empathize with the "emotions" portrayed by the robot. If it is difficult for the audience to empathize, the robot may take a less emotional role and the other characters will have to compensate for it through exaggerated elements in our narrative. The "robots in art" questions ask about the artistic capabilities of robots. Answers to these questions would reveal information about how people might perceive the robot as capable of producing genuine musical pieces or works of art rather than seeing the robot as a machine programmed to do so. They could reveal whether or not the autonomous nature of the robot causes its autonomously-created art to be considered of the same value as human-created art. We assumed based on our background research that this question might include some varied responses – some people may imagine that the robot is not an agent, because in reality the robot is not. Some, however, might subscribe to the idea that autonomous and procedurally- generated movements can literally make the robot into the artist, even though its behavior follows preset rules. The "technology in art" questions ask about the value of art made with more recent technology. Answers to these questions would reveal how people perceive the role of digital technology in the creation of art. There is controversy among people regarding the nature of creating art using digital mediums - and because we were using a primarily digital medium for our performance we needed to know how that may affect the audience's willingness to see the
Creative Robotics Studio 26 artwork produced by the unity of a number of digital mediums. Electronic music, especially, and digitally-generated animations were huge parts of this performance, so understanding whether or not people view digital mediums as obstacles to true artistic expression would greatly affect how they interpret the performance.
2.1.3 Survey Distribution
We made use of two online survey platforms in our outreach endeavors: SurveyMonkey and WPI Qualtrics. Both surveys were identical – the reason we separated the two platforms was to distinctly separate results from both platforms to make it simpler to compare. The Qualtrics version was sent just to WPI Students and faculty in order to gauge what kinds of opinions our audience members have. The SurveyMonkey version was given out to friends, acquaintances, and family outside of WPI who may not have been exposed to robotics to the same extent. Two different result sets allowed us to analyze the differences between them in order to extrapolate responses from our audience members, who could only be WPI students and faculty. If someone in our audience did have something different to say after they experienced the performance we might assume then, that someone more unfamiliar with robotics might also change their minds. A third survey was also created and sent out to WPI faculty through Qualtrics, giving us a different perspective from a generally different age group and a different WPI experience. Specific demographic data here, which we did not include in our survey to the students, could help us show connections between how age, department, and educational background might affect how someone perceives robotics. We hoped that our focus group audience would include at least one faculty member in addition to other students. Once we created the surveys, we sent them out to our target demographics and left them open for a week to give ample time for participants to take the survey.
2.1.4 Performance Group Discussion and Surveys
We considered a couple of different approaches to handling our live performance in terms of gathering research data for discussion. We had planned to conduct interviews personally, because we assumed that we would have a rather small audience. We eventually decided that this was out of scope and chose instead to hand out physical surveys to our audience members and facilitate a discussion with some post-performance questions. In the interest of keeping some responses unbiased by group opinion, we chose to have a group discussion for only one of the groups. Our performances were filmed in order to document, analyze, and distribute the performance outside of WPI. However, truly experiencing a performance first hand is much different than watching a video recording. A live performance gives the audience feedback to all five senses and allows them to interact with it without any limitations. A video has a limited angle and presents just audio and visual information without allowing the user to explore the space. Audience reactions to the performance count as points of discussion even though we could not quantify reactions specifically. Clear emotions like surprise or happiness are simpler to observe and we can talk about these reactions in the context of certain narrative elements. We did not expect all of our audience members to talk extensively about their reactions, but some data is better than no data. For this reason we chose to issue more concrete,
Creative Robotics Studio 27 numerical response style questions in our pre and post-performance surveys containing similar questions to those we had on our background survey. We had to know how they received the performance in order to determine if it would be possible to convince people, even partially, that the robot has agency; if not, then a semblance of agency. Extrapolating these interviews using the differences in the Qualtrics and SurveyMonkey surveys can reveal that someone who has spent less time around robotics perceives the robot as more of a “free-willed” agent.
2.2 Performance Design
This was the most technical leg of our project, encompassing everything we had learned from our background research and the background survey. We needed a relatively simple way to stay as flexible as possible with our design hardware so we could make significant changes to the performance if needed. The performance allowed the robot to interact with the audience from a safe distance through what is essentially interpretive dance, simple animated content, all choreographed to music.
2.2.1 Performance Hardware
The performance consisted of three main components: the robot itself, a display mounted to the end-effector of the robot, and the performance space surrounding the robot and the screen. The robot is an IRB 1600 – 6/1.45, which has a 6 kg maximum load specification for its end- effector and a 1.45m arm (see diagram below). Our hardware is important because it defines how we could use our space, how we could show visual data, and how much freedom we had when using the space.
Figure 5. Specifications for our ABB robot. All measurements shown are in mm.
The display was a 24-inch ViewSonic VX2452MH, which displays at 1920x1080 (1080p) resolution and has VESA-compatible mounting points. The size of the monitor is important
Creative Robotics Studio 28 because we needed every audience member to be able to see the screen from about 10 feet away during the entirety of the performance while still keeping most of the robot visible as well. The VESA mount compatibility allowed us to mount the monitor to the robot’s end effector by drilling holes into the VESA plate to align with the threaded holes on the end effector, screwing the plate onto the robot, and then mounting the monitor onto the plate as normal. The weight of the monitor was also relevant, as its 8.3lb (3.76kg) weight was under the limit for the arm. We needed to be careful with the velocities at the end of the arm to be sure the robot would not whip the screen around and perhaps eject it during moments of high inertia. Two 50 foot cables delivered visual data and power to the monitor with a huge margin of extra length to allow the robot to exercise its full range of motions without running out of cable. The space immediately surrounding the robot was composed of several elements. The largest of these elements was the wooden stage upon which the robot is mounted. This stage
stands approximately 2 feet off the ground and has the following dimensions:
^
Audience ^ ^
Figure 6. Performance space restrictions for our ABB robot.
The stage is surrounded by a “light curtain”, a vertically oriented line of IR sensors that automatically stop the robot if they are tripped by anything moving across them. To bend the light curtain around the irregular border of the stage, 3 mirror posts 1 transmitting, and 1 receiving post, have been mounted at precise angles on poles at strategic positions. Although these mirrors could potentially have blocked the audience’s view of the robot at certain positions, they were still necessary as the arm could not run in automatic mode without its light
Creative Robotics Studio 29 curtain activated and mirrors properly aligned. These are a required safety precaution which, for the audience safety, we were not allowed to remove. The space behind the robot is occupied by a CNC mill on the left (facing the robot from the door) and a CNC lathe on the right. Since our particular IRB 1600 was originally installed with the intention of picking up and transferring parts from the mill to the lathe and vice-versa, its full range of motion includes the insides of the both of these machines. Fortunately, this was not an issue for our performance, as we had decided to artificially limit the arm to a 180° range of motion centered on the projected location of the audience.
2.2.2 Storyboarding and Narrative
Our performance and gesture libraries would not be possible without a narrative through which to express emotions and show off creative communication. Otherwise, the robot simply moves through space without any particular continuity or apparent motivation to perform in the ways that it does. If the robot has a story to tell its audience, then the audience will recognize coherence and progression – concepts present in most popular literature, artworks, and music. We began our process with a narrative flow chart which outlined elements we considered to be important to show off in the performance. This includes, most importantly, descriptions of what the robot is supposed to show off in figurative terms e.g. “waking up,” and “inspecting.” These descriptors helped us visualize what we wanted to animate and compose, how each action relates to another, and what kind of emotions inspire the gestures. This stage was crucial because we could not proceed to animation and music composition without first understanding the underlying emotions to express. Gestures cannot stand alone in an appreciable manner in our context – we need to set up gestures in order, otherwise the performance becomes confusing even to us, the designers. If we cannot make sense of our own performance, then our audience would certainly become lost as well.
2.2.3 Animatics and Animation
This project made extensive use of animation software to choreograph the audio-visual components displayed on the end-of-arm screen alongside the movements of the robot itself. The rationale behind using animation software to "animate" this robot is based largely on a need for a fluid and adaptable workflow that allows for small changes to be quickly made and previewed. The animation software allows for smoother, natural looking motions because everything moves according to curves – accelerating and decelerating smoothly. We could modify these curves to control how much acceleration there is from one pose to another as well as disable acceleration altogether and make motion completely linear. As discussed above in section 2.3 of the background, smooth movement appears more natural and less threatening to humans. We recreated our stage virtually including important features like where the e-stops are, the milling machines in the background, and where the light curtain posts are. This way we could tell where the robot would and would not be obstructed by the light curtain posts. In the case of our narrative, we also needed the robot to be able to “inspect” the light posts as it “assessed” its environment. The virtual environment allowed us to understand how we should scale objects on the screen as well, and how the screen looks against the robot and within the context of its staging.
Creative Robotics Studio 30
We planned our animation beginning with quick, rough-draft preview animations in order to visually rough out a storyboard and make it easier to compose the score. Once we had the important keyframes down for the gestures and visual content based on our narrative, we then moved onto more nuanced movements and characterization.
2.2.3.1 Rigging and Inverse Kinematics
We cannot talk about animating this robot without discussing the importance of rigging and inverse kinematics. Both are crucial to freeing up the artist to focus on the animation without interrupting their creative flow. Rigging is the process of adding virtual bones and controls to a 3d model and then binding the model. This makes it much simpler to animate anything by simplifying how the animation software tracks and applies transformation data to a given 3d model. In addition, rigging allows for the user to create customizable controls which speed up the animation process by eliminating unnecessary elements and simplifying the selection and animation of bones. A well-made rig on a 3d model can make an enormous difference to the artist by reducing clutter and automating limitations so that the artist does not have to constantly monitor whether or not the pose they have is in some way impossible. Inverse kinematics refers to the process of orienting a chain of connected joints in a skeleton to reach a target location and orientation. This process models how humans interpret muscle control – when we reach to grab something we do not think about rotating our individual joints but rather aligning our hand to match a given orientation and position. Inverse kinematics does this mathematically, back-calculating each joint orientation in a skeleton so that the end of the chain matches the target controller. Inverse kinematics in animation is key to giving a character realism and personality as well as drastically reducing the time required to animate certain actions. We used Blender to rig and animate our virtual ABB IRB-1600 1.45, because the software performs well, allows for Python scripting, and – most importantly – implements SDLS (Selectively Damped Least Squares) and DLS (Damped Least Squares) inverse kinematic solvers for rigging. Damped Least Squares and Selectively Damped Least Squares inverse kinematics are alternative implementations to inverse kinematics which take into account physical limitations in order to determine a solution for the rig. Standard inverse kinematic solutions are closed-form and history-independent, which makes it possible for instantaneous and impossible joint reorientations to occur from one frame to another. These problems occur with the greatest frequency around axes 4 and 6 of the robot, which pass through singularity often, and require the artist to manually handle flipping. A singularity on the industrial arm occurs when two or more axes become collinear and the robot loses a degree of freedom. Singularity is enough of a problem to the animation process that it is well avoided as often as possible. The DLS and SDLS solutions make it a much easier task to animate the robot without worrying about unwanted joint flipping because they limit the rotation speed of the joints and eliminate impossible angular velocities. Both physically-based methods remain highly stable even when axes occasionally line up in singularity.57 The overhead on the animators part required to avoid joint flipping in animation is enough that using the SDLS method to
57 Buss, “Introduction to Inverse Kinematics with Jacobian Transpose, Pseudoinverse and Damped Least Squares Methods,” 9.
Creative Robotics Studio 31 automatically solve the majority of the problem is extremely convenient and dramatically decreases the time spent adjusting the animation to prevent joint flips. It is important to note that this is an artistic choice for an artistic application – the DLS and SDLS solutions sacrifice accuracy in order to solve near to singularity.
2.2.4 Exporting the Performance
The final step is to bake all of the joint data into a RAPID script using a python procedure running in Blender. We iterate over every single frame, copying the joint rotations into a large array. The RAPID script executes joint-only move commands by iterating through the large array until finally the program reaches its termination point and the performance is over. In addition to the audiovisual content on the screen and the RAPID script that controls the robot, we needed a way to keep the robots movements synchronized. We developed a syncing software using Java which could interface with RAPID to check where the robot was in its movements and adjust how quickly the robot ran the current move command in order to stay matched with the video playing on the monitor and the audio playing through the speakers. We drafted proprietary Java-based syncing software for this project which is essentially a media player wrapped with socket code and hosts a server with which the robot communicates. The application allows the user to load any media file and open a server on any port – however, JavaFX only supports .mp4 video files encoded with a H.264 codec and the port must be opened on 5515 in order to work with the robot controller.
2.2.5 Music
The first step in creating the music was identifying what styles of music were needed for the performance. While the goal was for the entire performance to feature Electronic music, we wanted to include a variety of styles to elicit different emotions throughout. Each different stylistic section was determined from the storyboard and roughly timed to match our animatic. Syncing the animation and music was an iterative process to properly align the robot movements with the intended musical segments. We used Ableton, Mixcraft, and Reaper to create and synchronize the music for the performance. Ableton is one of the leading pieces of software on the market for music creation using MIDI and sampling techniques.
2.2.6 Playing Back the Live Performance
Our staging options were limited in terms of location, but in some ways we could make it work in our favor. The performance space offered roughly 150 degrees of viewing angle. The audience members were free to move around during the performance to see how the robot moved from different angles. However, we opted to orient the performance generally in one direction, facing towards the greatest open space. We needed to control our staging somewhat because we wanted to keep the audience focused on the robot, and push the industrial setting into the background as much as possible. This project focuses on changing opinions regarding robots as more than just industrial machines, so the more we can eliminate the industrial setting from the scene, the more we assume the audience can focus on the robot as an agent rather than an industrial machine.
Creative Robotics Studio 32
We used a relatively lightweight 1080p computer monitor with VESA compatibility along with a modified VESA mounting plate in order to mount the screen to the robot’s end of arm tooling flange. We made sure that the 50’ power and HDMI cables were long enough to reach the monitor at any given position in the robot’s full range of motion without losing some amount of slack.
3 Data and Discussion
The data we obtained from our background surveys and performance show that the way people perceive robots is much more complex than we initially thought. In addition to generalized data from the background, responses from the live performance offered insight into the nuances of the audience’s interpretation. These findings can help future performances gauge how their audience might interpret a similar demonstration.
3.1 Background Survey Data
We received 294 total responses from our three demographics: 100 from WPI students, 89 from the faculty and staff, and 104 from the outside WPI group. We estimate that we reached around 3500 WPI undergraduates and graduates with our background survey – only 2.8 percent of whom responded. We estimate that we reached around 2000 faculty and staff with our survey which is a 4.5 percent response rate. These data are quite limited in scope and represent a very narrow portion of the WPI community, so statistical significance is difficult to measure. We cannot determine the scope of our outside-WPI demographic because our distribution methods were untracked. We discovered that even our non-WPI respondents were generally more comfortable with robots than we predicted (Figure 7). The means for the comfort level, robots and emotions, robots in art, and technology in art questions typically rested above the neutral value of 3 in the WPI students group, which included everyone from robotics majors to business majors. In most cases the WPI students were comfortable with the majority of our scenarios and agreed that robots have creative potential. In comparison with WPI students, the WPI faculty consistently scored lower in all areas, showing less comfort with robots and less conviction in the creative potential of robots. This may be a result of the age difference between the WPI faculty and the other two groups. Having grown up during times when robots were much less integrated in society might cause a person to be less comfortable with robots and have a lower expectation of their creative potential. It may also mean that they have greater room for change if they do not know what to expect going into a robotic performance. For the non-WPI group, the means for the comfort level questions generally fell in between those for the WPI student and the WPI faculty. However, the non-WPI group scored generally higher than both WPI students and WPI faculty in the emotions, art, and technology questions. It is possible that this group was able to perceive the more creative aspects of robots while WPI students are more focused on the technical aspects of robots.
Creative Robotics Studio 33
Figure 7. Participants were asked, “How comfortable are you in the following situations?” and scored each scenario from 1 (uncomfortable) to 5 (comfortable). For each comfort question, the mean score was plotted for each demographic.
4.1. Using a humanoid robotic android as a maid in my home 4.2. Being surgically operated upon by an entirely autonomous robotic surgical system 4.3. Working in a non-industrial job with robotic or completely autonomous coworkers 4.4. Watching a robot perform as the first chair violinist in an orchestra 4.5. Watching industrial robots perform as part of an Electronic Dance Music performance 4.6. Watching a dance performance in which a human dances with a robotic performer 4.7. Watching a play in which the entire cast is played by robots 4.8. A robot improvising in a jazz ensemble as an equal with humans 4.9. Receiving a massage from a robotic masseuse 4.10. Having a robotic teacher in a middle school classroom
Creative Robotics Studio 34
Figure 8. Participants were asked, “How comfortable are you in the following situations?” and scored each scenario from 1 (uncomfortable) to 5 (comfortable). The participants in the WPI students group and the WPI Faculty and Staff group were divided into three age groups: under 25 years old, 25-54 years old, and 55-84 years old. For each comfort question, the mean score was plotted for each age group.
There appears to be a correlation between recorded age and comfort levels as well, which implies that robots are becoming increasingly normal in society and that future generations will be more comfortable than the current one. It is also possible that comfort levels will dictate what kinds of roles become acceptable for robots in society going forward. In our case this means performance art, but this could apply to other fields including music and education.
Creative Robotics Studio 35
Figure 9. Participants scored each of the emotion questions from 1 (disagree) to 5 (agree). For each emotion question, the mean score was plotted for each demographic.
5.1. Machines are capable of expressing emotions 5.2. A performance making use of only robotic performers is capable of causing humans to experience strong emotions like sadness and elation 5.3. Machines are capable of recognizing and responding to human emotions 5.4. Humans can empathize with machines
Creative Robotics Studio 36
Figure 10. Participants scored each of the art questions from 1 (disagree) to 5 (agree). For each art question, the mean score was plotted for each demographic.
6.1. Robots are capable of assuming the lead performing role in a dance performance 6.2. Machines are capable of creative expression through visual mediums like painting 6.3. Machines are capable of creating music using physical instruments 6.4. Can a piece of music or art created autonomously by a robot be considered a true work of art?
Creative Robotics Studio 37
Figure 11. Participants scored each of the technology questions from 1 (disagree) to 5 (agree). For each technology question, the mean score was plotted for each demographic.
The data from WPI students held quite a few surprises, most of which involved counter-intuitive trends from those we would have expected to find based on our background research.
3.1.1 Comfort Scenarios
Of all the "how comfortable" questions, the "robotic teacher in a middle school classroom" question was rated the lowest by all three groups (Figure 7). This was surprising, as we expected that a person might be more afraid of situations where he or she is in close contact with robots, such as the "receiving a massage" and the "surgical robot" scenarios. This may suggest that empathy and emotion in a robot have greater influence than the need for safety. Or, people trust that a robot, if implemented, should already be safe. The next least comfortable scenario was the "surgical robot scenario." The most comfortable situation was the "electronic dance music performance," closely followed by the "human dancer with robotic performer." This was expected, since these scenarios portrayed robots in creative entertainment and did not imply any risks. The result supports our choice of using a robotic performance with electronic dance music to influence public opinion. It is best to start with the most comfortable situations to introduce a person to the creative potential of robotics.
Creative Robotics Studio 38
These data support the interpretation that generally people trust robots to perform technical tasks provided the robot does not directly interact with the observer. These tasks may even involve basic interactions between robots and other humans, such as dancing. However, it seems as though people trust robots less with direct-contact or social situations, or situations which involve more subjectivity and empathy. The emotion scores support this as well (Figure 9), showing how people across all demographics tended to disagree the most with the notion that humans can be empathetic with robots. So despite the general comfort observed across hypothetical scenarios, skepticism is still an important consideration.
3.1.2 Correlations with Literacy Scores
The total data from all three groups were divided into four quartiles, based on the sum of their “technological literary” scores. This analysis segregated participants into those least (Beginner) to most (Expert) up to date with current technology and associated these scores with how comfortable each quartile was with a hypothetical situation. There was a correlation between high literacy score sum and high comfort scores on every question. In other words, people more involved with technology and social media are generally more comfortable with robots, and more believing in or projecting onto the robot's artistic and emotional expressiveness.
Figure 12. For each comfort question, the average score for each technological literacy group was plotted. The “Expert”, “Advanced”, “Intermediate”, and “Beginner” groups correspond with participants whose sum score in the technological literacy questions were in the 4th, 3rd, 2nd, and 1st quartiles, respectively.
Creative Robotics Studio 39
Figure 13. For each emotion question, the average score for each technological literacy group was plotted. The “Expert”, “Advanced”, “Intermediate”, and “Beginner” groups correspond with participants whose sum score in the technological literacy questions were in the 4th, 3rd, 2nd, and 1st quartiles, respectively.
Creative Robotics Studio 40
Figure 14. For each art question, the average score for each technological literacy group was plotted. The “Expert”, “Advanced”, “Intermediate”, and “Beginner” groups correspond with participants whose sum score in the technological literacy questions were in the 4th, 3rd, 2nd, and 1st quartiles, respectively.
Creative Robotics Studio 41
Figure 15. For each technology question, the average score for each technological literacy group was plotted. The “Expert”, “Advanced”, “Intermediate”, and “Beginner” groups correspond with participants whose sum score in the technological literacy questions were in the 4th, 3rd, 2nd, and 1st quartiles, respectively.
3.1.3 Changes in Opinion
Our live performance was able to get a generally positive response from our 30 participants. If we assume our survey data accurately represents the opinions of those who watched our performance, then we can conclude that it did have an impact on how our viewers perceive, at the very least, our robot. By analyzing the difference between the post and pre survey numerical responses, it is clear that few people remained unchanged from the experience. In the context of our objective – explore how people perceive the creativity and emotional capabilities of robots – these data provide valuable insight into the variety of responses that people provide, given their attitude towards robotics. Statistically, however, due to our limited sample size – we cannot definitively conclude that we made a statistically significant difference because our p-value analyses from t-tests and least square difference tests consistently produced values well above 0.3. What we understand from these data is that interpretation is complex, and requires a much greater sample size. Though we cannot reject our null hypothesis that we were unable to change our audience’s opinion, we can still visualize the information and discuss small fluctuations in the data. The same comfort, emotion and art questions were asked before and after the performance. The sums of scores for each of these three sections were used for analysis. The post-performance sum minus the pre-performance sum was graphed in Figure 16, Figure 17, Figure 18, and Figure 19, and the values were sorted to better see the positive and negative effect. There were generally positive changes in the comfort scores and emotion scores. A
Creative Robotics Studio 42 larger number of people had increased score sums than those who had decreased score sums. Also, the average magnitude of increase was greater than the average magnitude of decrease. For the art questions however, the effect was more varied. After the performance, eight participants scored higher on the art questions while seven participants scored lower. All categories had 6-7 participants whose sum score did not change.
Figure 16. For each participant, the sum of scores on the comfort questions was calculated for the pre-performance survey and again for the post-performance survey. The post-performance sum minus the pre-performance sum was plotted to observe the change. These bars were then sorted to more easily view how the positive, negative, and no change distributions compare by volume.
That our performance was able to affect anyone’s opinion, either positive or negatively, is important to us and our objective with this project. In most cases we had a similar distribution of negative changes to positive changes, but in every case there were more participants who reported a positive change than negative ones. However, as shown in Figure 19 not all
Creative Robotics Studio 43 participants who seemed to become more comfortable with robots saw them as more creative or more emotionally expressive. We knew that our objective to explore how people perceive robots might result in rather ambiguous results, but based on these data we can infer that a similar audience in a similar performance scenario would leave with a more positive view of robots in their ability to be either more emotionally expressive or creative. Of course, there were quite a few participants who reported no change or negative change – we expected this to occur though not in such a high percentage. Each participant has a different interpretation of the performance context and the narrative as they observe it, which at any moment may affect how they perceive the entirety of the performance in retrospect. In addition, the difference between expectation and retrospection can heavily shape how a participant might report any self-change. In reality, we cannot expect the performance to fundamentally alter the way our audience members perceive a topic – especially not with a performance as short as ours or an audience as small as ours. These change data simply provide a good indication that we can show off the robot as a creative, emotionally expressive agent and expect that some of our audience will reevaluate the way they perceive our robot. This means that our design approach of creating a narrative performance accomplished at least one of its experience goals for each of our participants. Future endeavors should expect, given our results, to get a positive reaction by portraying the robot as a character in a narrative. There are definitely many more diverse applications for this system and our toolset, like robot surf and the Berlin Long Distance Art event, that would benefit from an animation software controller, but do not portray a narrative or a character. However, our successes with characterization and narrative make ours a viable method to evoke empathy in viewers.
Creative Robotics Studio 44
Figure 17. For each participant, the sum of scores on the emotion questions was calculated for the pre-performance survey and again for the post-performance survey. The post-performance sum minus the pre-performance sum was plotted to observe the change. These bars were then sorted to more easily view how the positive, negative, and no change distributions compare by volume.
Creative Robotics Studio 45
Figure 18. For each participant, the sum of scores on the art questions was calculated for the pre-performance survey and again for the post-performance survey. The post-performance sum minus the pre-performance sum was plotted to observe the change. These bars were then sorted to more easily view how the positive, negative, and no change distributions compare by volume.
Creative Robotics Studio 46
Figure 19. For each participant, the sum of scores of each category (comfort, emotion, art) of questions was calculated for the pre-performance survey and again for the post-performance survey. The post-performance sums minus the pre-performance sums were plotted to observe the change in each category.
Creative Robotics Studio 47
Figure 20. For each comfort, emotion, and art question, the pre-performance survey average minus the background survey average and the post-performance survey average minus the background survey average were graphed. Questions below.
4.1. Using a humanoid robotic android as a maid in my home 4.2. Being surgically operated upon by an entirely autonomous robotic surgical system 4.3. Working in a non-industrial job with robotic or completely autonomous coworkers 4.4. Watching a robot perform as the first chair violinist in an orchestra 4.5. Watching industrial robots perform as part of an Electronic Dance Music performance 4.6. Watching a dance performance in which a human dances with a robotic performer 4.7. Watching a play in which the entire cast is played by robots 4.8. A robot improvising in a jazz ensemble as an equal with humans 4.9. Receiving a massage from a robotic masseuse 4.10. Having a robotic teacher in a middle school classroom 5.1. Machines are capable of expressing emotions 5.2. A performance making use of only robotic performers is capable of causing humans to experience strong emotions like sadness and elation 5.3. Machines are capable of recognizing and responding to human emotions 5.4. Humans can empathize with machines 6.1. Robots are capable of assuming the lead performing role in a dance performance 6.2. Machines are capable of creative expression through visual mediums like painting 6.3. Machines are capable of creating music using physical instruments 6.4. Can a piece of music or art created autonomously by a robot be considered a true work of art?
Creative Robotics Studio 48
Figure 20 shows comparisons between the performance audience data and the broader “WPI student” data for the questions that lined up. We were surprised to see that our audience was generally uncomfortable with robots and dubious that robots could be creative or emotionally expressive. Although the audience was generally below the WPI Students average in the pre survey categories for comfort-based questions, creativity, and emotion questions, two important categories immediately jump out as well above average: 5.2 and 5.4, two questions related to how people perceive the emotional capacity of robots (Figure 20). This means that even before watching our performance, our audience members were predisposed to view robots as emotionally expressive, and capable of gaining empathy from a human observer. These two sections barely changed between the pre and post-performance responses. From these data we can infer that our audience was more empathetic to the robot and therefore more likely to anthropomorphize the robot into a character. For most of our other responses in which our audience fell below the WPI Student averages, we assume that our audience felt more uncomfortable with robots in more commonplace roles in society and were less willing to see the robot as an artist. Our performance was able to move these averages closer to the WPI Student baseline, which seems to be an important accomplishment relatively speaking. WPI Students were quite comfortable with most of the situations described in section 4 of our questions.
3.1.4 Open Ended Responses
Many of the participants interpreted our performance the way we intended – a robot breaking free from its work and discovering music. In addition, a few participants believed it was a form of interpretive dance, which shows that they thought of the robot as an expressive actor. One participant thought the robot was frightened by its own reflection, while we intended for the robot to be curious about its surroundings. This may have been an unintended result of the “shivering” screen during the performance. Some of the participants felt that the music was not synchronized well with the movements and worked as only background noise. This may have been a result of the speaker's limited apparent frequency range and the lacking capabilities of the computer we were using to synchronize the performance. This resulted in an extremely loud bass, not much treble, and occasional audio skips, all of which could have been distractors to our audience. Each participant had a different expectation for how the performance should have handled some moments of intentional audio and visual silence. Some of the participants felt disconnected from the robot when motion and visual animation were present but music was not or when motion and music were present but the visual animation was not. According to expectation confirmation theory, the participants’ expectations of the performance directly influence their perception of the performance, and their judgement of the performance is made in comparison to their original expectations, leading to decreased satisfaction if the performance did not meet the expectations.58 We attempted to portray the robot as a living character, but the audience saw a blank screen in the beginning of the performance when they might have expected a face, which contradicted the message and caused confusion.
58 Oliver, “Effect of Expectation and Disconfirmation on Postexposure Product Evaluations: An Alternative Interpretation.”
Creative Robotics Studio 49
As to be expected based on previous research, some of the participants felt emotional distance from the performance when the movements were too rigid or jerky. This was expected since rigid motions are known to cause unease59, but unavoidable because of the limits of the monitor mount – both the aluminum bracket we used and the monitor itself. The monitor’s anchor points are no more than threaded metal sleeves glued directly onto the plastic back plate from the inside. We had planned to create a much sturdier, custom mount for our purposes, but that would not have been able to eliminate the jittering motions from the plastic itself. Some of the participants felt empathy towards the robot while it was examining its confines and expressing its curiosity towards its surroundings and the musical ball. One participant felt empathy when the robot “shivered,” an unintended effect of the screen’s weak attachment to the robotic arm. Artefacts like these become part of the medium and the artist can learn to work with them to produce a certain effect. We are not entirely certain if we could have eliminated all monitor wobble, but it is something to be aware of when creating pieces. Slower movements or more space between move commands may mitigate this effect, but we did not have time to test this hypothesis. When asked about the robot’s creativity, some participants believed the robot expressed creativity in drawing shapes with the screen and having them show up on the visual animation. This is a much more literal interpretation of a gesture for what it displayed, and not necessarily for what it meant. However, we acknowledge that if the narrative was not entirely clear up until that point, the drawing gesture could easily be one of the more representative, recognizable gestures. On the other hand, one participant strongly expressed that the robot was incapable of expressing creativity because its movements and animations were made ahead of time, and any creativity was due to the programmer.
3.2 Developing an artist’s toolset
A significant portion of this project revolved around the creation and development of tools that could speed up the animation process, as well as provide a user-friendly way to control the live performance once it was ready for execution on the robot. These tools included a specialized animation rig in Blender with custom controls to make switching between IK and FK controls faster and more convenient, an easily-accessible and simple animation export procedure, and an easy-to-use user interface-based java application for socket management and media display.
59 Baddoura and Venture, “This Robot Is Sociable.”
Creative Robotics Studio 50
Close - invisible
UI Toggle Button - invisible Fullscreen - invisible
Frame offset Server port and restart
Figure 21. The GUI for the sync software without any media loaded – with annotations.
3.2.1 Function before Aesthetics
These tools were quickly drafted and implemented for the purpose of function before aesthetics – for our purposes we required only functional tools to increase production speed. The systems we devised can and should be expanded to a more complete and user-friendly array of tools across other platforms. For example, the media playback software included a number of debug controls and undocumented commands for our use because we just needed the functionality. We each have technical backgrounds so text and command based inputs were sometimes desirable over UI elements. Ultimately the few but important tools served our purpose. Development of the toolset gave us insight into how inextricably linked the hardware limitations of the robot are to how it displays gesture. Each axis of the robot has a different angle limitation which describes how the robot can physically move through gesture to another. We believe that these tools can be extended into proper user interfaces so that another artist may pick up our animation and exportation tools and quickly learn to use them. We hope that these tools are not simply restricted to this project as bi-products, but rather taken as prototypes for more generalized, robust solutions to the gap between animation and the industrial robot.
3.2.2 Future Endeavors with Our Toolset
With the basic goals of these tools in mind we can extrapolate how to improve what we currently have for future projects. The rig should be expanded to more platforms, including
Creative Robotics Studio 51 industry standards like Maya, so more artists can access it. We should also include a more concise set of rigging options to eliminate much of the extraneous information that did not require manipulation – some of which never needs to be controlled. The export script was never given a graphical interface, a feature it needed, and remained limited to console use only. For ease of use this script should also find its way onto the user interface and include some additional parameters such as a frame skipping option, or where the start and end frames should be, or even whether or not to intelligently overwrite the previous copy of the export if only small changes to the animation require updating. The concern we have regarding our toolset lies within the complexity that still exists in the animation and exportation process. We eliminated major problems when and where they occurred, but so much was left to manual correction. For example, no system was ever devised to automatically identify and fix axis inversions throughout the animation, a problem which encroached drastically on animation time. In addition, the exportation procedure is not automatic and requires the animator to execute a number of steps in a specific order, further slowing the creation process. An import future goal for the artistic toolset is real-time feedback with proper checks implemented so the animator never needs to break out of their creative flow. They could work live with the industrial arm, making changes to their performance or creative application on the fly, or even with an audience, modifying the performance itself based on how the audience reacts. The topics of more advanced, specialized IK solvers and animation tools need further exploration, as well as how to handle safe, efficient real-time control of the industrial arm from the animation software through a customizable server.
Creative Robotics Studio 52
3.3 The Live Performance60
Figure 22. A still from our live performance. The robot transitions from one gesture to another.
We were able to successfully create and execute a culminating performance which incorporated narrative and gesture into a live audiovisual performance art experience. The performance was shown to a small audience of roughly 30 WPI students and faculty. Our audience was generally diverse, including biomedical majors and aerospace majors, however robotics majors were among the most frequent in attendance (Figure 23). We will not make any sweeping conclusions about what being a robotics major means, but based on the types of classes robotics majors take and the kinds of projects they complete we can say that their perspective of our machine gives them unique insight into our performance that a student focused on another major might not have. Still, distribution of our audience is such that we can make tentative extrapolations to a larger WPI audience.
60 Held, Creative Robotics Studio - Final Performance.
Creative Robotics Studio 53
Figure 23. From those who chose to report their majors, these data show the distributions of majors present at the live performance. AE: Aerospace, BME: Biomedical, CS: Computer Science, ECE: Electric and Computer Engineering, ME: Mechanical Engineering, RBE: Robotics Engineering
3.3.1 The Animation Process
This system is not perfect and the animator must occasionally apply manual corrections in order to stop the solver from creating impossible motion. Inverse kinematics are certainly a topic for further exploration and development, especially in animation. However, the applications of customized solvers in the industry could potentially augment the efficiency of these robots both from a programming standpoint and an operational standpoint. A huge potential time save exists if the programmer and the machine never have to encounter issues near singularity.
3.3.2 Simulation vs. Reality
Before we attempted any real runs on the industrial arm we tested out the performance using Robot Studio to eliminate any unexpected movements and ensure that the sync software would work with the RAPID code. Smoothing over all of the glitches in the animation took an enormous amount of time for each revision, and ended up becoming quite a laborious task. On a technical level, the inverse kinematics solver had more trouble than we anticipated, and required a number of manual forward kinematic corrections in order to stop impossible axis flips and keep the movements smooth – this occurred nearly every time the robot’s end effector passed from a convex to a concave posture (Figure 24). This transition requires rotation about axis 4 that almost always encountered a singularity problem, despite the SDLS IK solver we used.
Creative Robotics Studio 54
Figure 24. The ABB IRB 1600 at the midpoint reset pose. Passing through this pose or its near derivatives causes singularity issues in most cases.
We ran tests before mounting performance hardware just to be sure that the robot would not execute any unwanted maneuvers, despite the fact that it could run perfectly in Robot Studio’s simulation environment. This turned out to be an important trouble-shooting stage because most of the problems transferring the performance from simulation to reality occurred at the communications level. We had to resolve our systems to static IP addresses and look up the correct port to use. It must be stated that participating in the live performance experience is vastly different than observing an animation of what the performance would look like. In much the same way that a live music concert is much more dynamic than an audio or video recording, so was our performance. In a live context the robot appears much larger and its movements more impactful. The animation and simulation separates the viewer from the robot by scaling it down to the size of a computer screen – in reality the robot has a 1.45 meter full reach, and in our lab the base of the robot is mounted about two feet above ground level. This means at full extension the robot appears seven feet tall or greater, giving it an imposing presence. The sense of scale and velocity realized in a live context can both give the robot a sense of vivacity and a sense of aggressiveness that we cannot fully appreciate through previsualization. Therefore, the performance becomes even more difficult to interpret because people react to the live presence of the robot differently than they would if presented an image of the same scenario. Occasionally the transfer of code from our simulation environment would not go over smoothly and we would neglect to make a system-specific change. An error on our part was not setting the starting frame to the correct array position causing the robot to execute two movement commands too quickly, offloading an enormous amount of force on our monitor. After two unexpected instances of this glitch we rectified the code. Another example: testing software on local machines required us to use a local socket, but running the software on the controller system needed some additional investigations and network privileges in order to work properly. However, the benefits of simulation were evident when, once we had pulled everything together,
Creative Robotics Studio 55 the first test of our performance went exactly as we planned. Then the following six runs over two performances followed without any further problems. We were somewhat limited by our controller computer which was much older and not particularly apt to handle the 1080p 30fps video playback of our performance. This introduced unwanted video and audio skips into the performance playback and unfortunately downgraded the quality of the music. We did not know that the audio quality was going to suffer as much as it did, but there was simply not enough time to make any changes to our hardware setup in the week leading up to the final performance.
3.3.3 The Synchronization Software
This synching software underwent multiple iterations and improvements to increase the quality of its communication techniques with the IRB1600. The RAPID code on the IRC5 robot controller began as a single-task, serial execution only style program, which required a break in movement in order to fetch data from the sync software. The controller would run a few move commands and then ask the server for the media playback position. The RAPID code then compared the local frame versus the server frame and adjusted the speed of the move commands accordingly. In simulation, the serial method worked perfectly because there was no socket latency between the server software and the simulator. In practice, however, the serial execution method worked but problems with socket latency caused jerky and inconsistent movements. Even a 10-100 ms delay in communication introduced unwanted hiccups in movement. In sacrificing update resolution for smoothness, the robot would drift more from the media playback position. Later, after doing research into ABB’s multi-tasking support, we implemented a parallel processing method with two tasks running simultaneously: one task handled the movement of the robot and the other handled communication with the server. This change vastly improved the quality and consistency of the robot’s movements, keeping a much tighter synchronization with the media as it played because the communication task could update as fast as possible, rather than over an interval.
Conclusion
We were largely successful in our goals to better understand how people perceive robots as agents through a dynamic industrial arm performance art experience. We showed how it was possible to create a toolset which offers artists additional control over the robot through an unconventional animation-based workflow. The ABB robotics system’s fully- functioned programming language, Blender’s extensible animation tools, and Java’s cross- platform capabilities make our toolset easy to expand and refine in the future. Issues with inverse kinematics require attention for the artist’s sake. Correcting impossible motions consumed so much time throughout the animation process that we believe artistic IK could definitely use a specialized implementation designed to avoid singularity-related issues. Live human-robot interaction is potentially the next topic to pursue. Studies exist which examine interactions between humans and robots, like This Robot is Sociable61, but hardware and software solutions which safely allow a potentially dangerous machine to interact with
61 Baddoura and Venture, “This Robot Is Sociable.”
Creative Robotics Studio 56 people is still sparse. Real-time reactions and procedural characterization of the robot might increase its ability to receive empathy from the audience and become an agent. Removing the artist “controlling” the performance entirely might offer another avenue of insight into the way people perceive robots. This project primarily focused on giving the artist additional creative control over the robot and determining whether the robot could appear as an agent in a performance. Observations we made from the surveys indicate that it is possible to positively affect people’s perceptions of a robot provided the artist creatively presents it. Ultimately, it is clear that industrial arms have enormous potential to affect how people perceive robotics – for the artist, who orchestrates the robot as a character, and for the audience who interprets the gestures of the robot. The industrial arm allows the artist to communicate emotion and draw empathy from their audience in the same way a musician or a painter might. It is also quite clear that additional investigations into the perception of these performances are required in order to discover trends in nuances. We have only scratched the surface of a vast and largely unexplored topic.
Creative Robotics Studio 57
Works Cited
ABBRobotics. ABB Robotics - Robotic Artist at Long Distance Art Event. Accessed December 17, 2015. https://www.youtube.com/watch?v=U8Iy0p7toeg. andirobot. Blue Man Group and KUKA Industrial Robots for Factory Automation. Accessed December 17, 2015. https://www.youtube.com/watch?v=KlXqRNLmV-s. Baddoura, Ritta, and Gentiane Venture. “This Robot Is Sociable: Close-up on the Gestures and Measured Motion of a Human Responding to a Proactive Robot - Springer.” Accessed October 10, 2015. doi:10.1007/s12369-015-0279-x. Bar-Cohen, Yoseph, David Hanson, and Adi Marom. “Introduction.” In The Coming Robot Revolution, 1–20. Springer New York, 2009. http://link.springer.com/chapter/10.1007/978-0-387-85349-9_1. Bentkowska-Kafel, Anna, Trish Cashen, and Hazel Gardiner. Digital Art History. Bristol, GB: Intellect Ltd., 2004. http://site.ebrary.com/lib/alltitles/docDetail.action?docID=10078137. Buss, Samuel R. “Introduction to Inverse Kinematics with Jacobian Transpose, Pseudoinverse and Damped Least Squares Methods.” IEEE Journal of Robotics and Automation 17, no. 1–19 (2004): 16. correspondent, Duncan Weldon Newsnight economics. “Are the Robots Taking Enough Jobs?” 11/17/2105. BBC News. Accessed February 10, 2016. http://www.bbc.com/news/business-34846356. Dautenhahn, Kerstin. “Roles and Functions of Robots in Human Society: Implications from Research in Autism Therapy.” Robotica 21, no. 04 (August 2003): 443–52. doi:10.1017/S0263574703004922. de Graaf, Maartje M.A., Somaya Ben Allouch, and Tineke Klamer. “Sharing a Life with Harvey: Exploring the Acceptance of and Relationship-Building with a Social Robot.” Computers in Human Behavior 43 (February 2015): 1–14. doi:10.1016/j.chb.2014.10.030. Eerola, Tuomas. “Are the Emotions Expressed in Music Genrespecific? An Audio-Based Evaluation of Datasets Spanning Classical, Film, Pop and Mixed Genres.” Journal of New Music Research 40, no. 4 (November 25, 2011): 349–66. Elkins, James. “Art History and the Criticism of Computer-Generated Images.” Leonardo 27, no. 4 (1994): 335–42. doi:10.2307/1576009. Evans, Paul, and Emery Schubert. “Relationships between Expressed and Felt Emotions in Music.” Musicae Scientiae 12, no. 1 (March 20, 2008): 75–99. doi:10.1177/102986490801200105. “Executive Summary.” World Robotics, 2015. http://www.worldrobotics.org/uploads/media/Executive_Summary__WR_2015.pdf. Flessas, Flessas. Interview: Andy Flessas. Interview by Graham Held, Walter Ho, Paul Raynes, and Harrison Vaporciyan, November 5, 2015. Fussell, Susan R., Sara Kiesler, Leslie D. Setlock, and Victoria Yew. “How People Anthropomorphize Robots.” Carnegie Mellon University, n.d. http://sfussell.hci.cornell.edu/pubs/Manuscripts/Fussell-HRI08.pdf. Grinbaum, A. “Uncanny Valley Explained by Girard’s Theory [Turning Point].” IEEE Robotics Automation Magazine 22, no. 1 (March 2015): 152–150. doi:10.1109/MRA.2014.2385568. Haas, G. J. F. de. “Synchronicity in the Performing Arts: Oscar Wilde’s Nightmare?” Clinical Rheumatology 32, no. 4 (February 17, 2013): 493–95. doi:10.1007/s10067-013-2200-z.
Creative Robotics Studio 58
Han, Eunjung, Chee-Onn Wong, Keechul Jung, and Kyung Ho Lee. “Emotion Gesture Art.” Leonardo 43, no. 3 (2010): 308–9. Held, Graham. Creative Robotics Studio - Final Performance. Accessed May 9, 2016. https://www.youtube.com/watch?v=LRD5o0pjOUA. José Augusto Ribeiro Costa. Robot Surf. Accessed December 17, 2015. https://www.youtube.com/watch?v=IYlu8px8b1I. Juslin, Patrik N. Handbook of Music and Emotion: Theory, Research, Applications. Oxford University Press, 1993. http://www.oxfordscholarship.com/view/10.1093/acprof:oso/9780199230143.001.0001/a cprof-9780199230143. Krumhansl, Carol L. “Music: A Link Between Cognition and Emotion.” Current Directions in Psychological Science 11, no. 2 (April 1, 2002): 45–50. doi:10.1111/1467-8721.00165. Lee, Newton. “From a Pin-up Girl to Star Trek’s Holodeck: Artificial Intelligence and Cyborgs.” In Digital Da Vinci, edited by Newton Lee, 1–21. Springer New York, 2014. http://link.springer.com/chapter/10.1007/978-1-4939-0965-0_1. Lin, Patrick, Keith Abney, and George A. Bekey. Robot Ethics: The Ethical and Social Implications of Robotics. MIT Press, 2011. Lipscomb, Scott D. “Perceptual Measures of Visual and Auditory Cues in Film Music.” State College, PA, 1997. http://www.lipscomb.umn.edu/JASA97/. Meyer, Leonard B. Emotion and Meaning in Music. University of Chicago Press, 2008. Mori, M., K.F. MacDorman, and N. Kageki. “The Uncanny Valley [From the Field].” IEEE Robotics Automation Magazine 19, no. 2 (June 2012): 98–100. doi:10.1109/MRA.2012.2192811. Oliver, Richard L. “Effect of Expectation and Disconfirmation on Postexposure Product Evaluations: An Alternative Interpretation.” Journal of Applied Psychology 62, no. 4 (August 1977): 480–86. Pasquale, Frank. “The Doubtful Ethics of Drone Warfare.” The Chronicle of Higher Education 62, no. 11 (November 13, 2015): B16. Robolounge. Robot Dance : roboLounge Electrobot Part II. Accessed November 20, 2015. https://www.youtube.com/watch?v=8KqrO_BQD8E. ———. Robot Dance : roboLounge Tribute to Kraftwerk. Accessed December 17, 2015. https://www.youtube.com/watch?v=hegEpbGjgnw. Skelton, Tracey, and Gill Valentine. Cool Places: Geographies of Youth Cultures. Routledge, 2005. Snyder, Jeff, Ryan Luke Johns, Gene Kogan, Charlie Avis, and Axel Kilian. “Machine Yearning: An Industrial Robotic Arm as a Performance Instrument.” Proceedings of the International Conference on New Interfaces for Musical Expression, Baton Rouge, LA, USA, May 31, 2015, 184–86. Stam, Gale, and Mika Ishino, eds. Integrating Gestures : The Interdisciplinary Nature of Gesture. Amsterdam, NLD: John Benjamins Publishing Company, 2011. http://site.ebrary.com/lib/wpi/reader.action?docID=10480788&ppg=9. Street, 1615 L., NW, Suite 800 Washington, and DC 20036 202 419 4300 | Main 202 419 4349 | Fax 202 419 4372 | Media Inquiries. “Questionnaire Design.” Pew Research Center. Accessed November 11, 2015. http://www.pewresearch.org/methodology/u-s-survey- research/questionnaire-design/.
Creative Robotics Studio 59
【安川電機】e-メカサイト. YASKAWA BUSHIDO PROJECT / Industrial Robot vs Sword Master. Accessed December 17, 2015. https://www.youtube.com/watch?v=O3XyDLbaUmU.
Creative Robotics Studio 60
Appendix 1 – Sync Software Code
QuickSync.java /* * To change this license header, choose License Headers in Project Properties. * To change this template file, choose Tools | Templates * and open the template in the editor. */ package quicksync; import javafx.application.Application; import javafx.fxml.FXMLLoader; import javafx.scene.Parent; import javafx.scene.Scene; import javafx.stage.Stage; import javafx.stage.StageStyle; import jdk.nashorn.internal.runtime.regexp.joni.Regex;
/** * * @author Graham Held */ public class QuickSync extends Application {
@Override public void start(Stage stage) throws Exception { Parent root = new FXMLMainController();
Scene scene = new Scene(root); scene.getStylesheets().add(getClass().getResource("def.css").toExternalForm());
stage.initStyle(StageStyle.UNDECORATED); stage.setScene(scene); stage.show();
//stage.setResizable(false); }
static boolean autoLaunchServer = false; static int serverPort = 5515; static boolean autoLoadFile = false; static String filePath = "F:\\Documents(E)\\WPI Junior\\IQP\\PQP\\previs5_screenOnly_0001- 7210.mp4"; static int frameOffset = 35;
/** * @param args the command line arguments */ public static void main(String[] args) { for (int a = 0; a Creative Robotics Studio 61 } System.out.println(filePath); } if (s.contains("-o")) { if (s.contains(":")){ frameOffset = Integer.parseInt(s.split(":")[1]); } System.out.println(frameOffset); } } System.out.println(args.length); launch(args); } } FXMLMainController.java /* * To change this license header, choose License Headers in Project Properties. * To change this template file, choose Tools | Templates * and open the template in the editor. */ package quicksync; import java.io.BufferedReader; import java.io.File; import java.io.IOException; import java.io.InputStreamReader; import java.io.PrintWriter; import java.net.ServerSocket; import java.net.Socket; import javafx.application.Platform; import javafx.event.ActionEvent; import javafx.fxml.FXML; import javafx.fxml.FXMLLoader; import javafx.scene.control.Button; import javafx.scene.control.Label; import javafx.scene.control.TextField; import javafx.scene.input.KeyEvent; import javafx.scene.input.MouseEvent; import javafx.scene.layout.AnchorPane; import javafx.scene.layout.Pane; import javafx.scene.media.Media; import javafx.scene.media.MediaPlayer; import javafx.scene.media.MediaView; import javafx.scene.paint.Color; import javafx.scene.shape.Circle; import javafx.stage.FileChooser; import javafx.stage.FileChooser.ExtensionFilter; import javafx.stage.Stage; import javafx.stage.StageStyle; /** * * @author Graham Held */ public class FXMLMainController extends AnchorPane { @FXML AnchorPane uiLayer; @FXML AnchorPane mainPane; @FXML Creative Robotics Studio 62 Button load; @FXML Button reset; @FXML Button playPause; @FXML Circle indicator; @FXML Label offsetInd; @FXML TextField portField; @FXML Pane dragBar; int currentFrame = 0; boolean playing = false; MediaView mView = new MediaView(); ShortServ server = new ShortServ(); double totalDuration = 1; int frameOffset = QuickSync.frameOffset; boolean guiOn = false; double initialX, initialY = 0.0; Stage stage; //MidiHandler cntrl = new MidiHandler(); public FXMLMainController() { FXMLLoader fxmlLoader = new FXMLLoader(getClass().getResource("FXMLMain.fxml")); fxmlLoader.setRoot(this); fxmlLoader.setController(this); try { fxmlLoader.load(); } catch (IOException exception) { throw new RuntimeException(exception); } mainPane.getChildren().add(0, mView); AnchorPane.setBottomAnchor(mView, 1.0); AnchorPane.setTopAnchor(mView, 1.0); AnchorPane.setLeftAnchor(mView, 1.0); AnchorPane.setRightAnchor(mView, 1.0); server.port = Integer.parseInt(portField.getText()); mainPane.setOnKeyTyped((KeyEvent ke) -> { handleKeyPressed(ke); }); if (QuickSync.autoLoadFile) { handleLoad(new ActionEvent()); } if (QuickSync.autoLaunchServer) { portField.setText("" + QuickSync.serverPort); server = new ShortServ(QuickSync.serverPort); Creative Robotics Studio 63 server.start(); } dragBar.setOnMousePressed((MouseEvent e) -> { initialX = e.getSceneX(); initialY = e.getSceneY(); }); dragBar.setOnMouseDragged((MouseEvent e) -> { getScene().getWindow().setX(e.getScreenX() - initialX); getScene().getWindow().setY(e.getScreenY() - initialY); stage.setFullScreen(false); }); Platform.runLater(() -> { stage = (Stage) getScene().getWindow(); }); updateOffsetLabel(); } public void handleKeyPressed(KeyEvent ke) { char kode = ke.getCharacter().charAt(0); System.out.println("key"); switch (kode) { case 'a': handleIncrFrameOffset(new ActionEvent()); break; case 'b': handleDecrFrameOffset(new ActionEvent()); break; case 'c': handlePlayPause(new ActionEvent()); break; case 'r': handleReset(new ActionEvent()); break; } } public int getCurrentFrame() { double result = 0; if (mView != null && mView.getMediaPlayer() != null) { result = mView.getMediaPlayer().getCurrentTime().toMillis() / (30.0); //System.out.println(result); } return (int) result; } @FXML public void handleFullScreen(ActionEvent e) { // do what you have to do stage.setFullScreen(!stage.isFullScreen()); } @FXML public void handleClose(ActionEvent e) { // do what you have to do stage.close(); } Creative Robotics Studio 64 @FXML public void handleGUIHide(ActionEvent e) { guiOn = !guiOn; uiLayer.setVisible(guiOn); } @FXML public void handleServerReset(ActionEvent e) { if (server != null) { try { Socket c = new Socket("localhost", server.port); PrintWriter out = new PrintWriter(c.getOutputStream(), true); BufferedReader in = new BufferedReader(new InputStreamReader(c.getInputStream(), "UTF-8")); out.println("4,0"); in.readLine(); server.terminate(); out.println("rektg"); //in.readLine(); } catch (IOException ex) { } } server = new ShortServ(Integer.parseInt(portField.getText())); server.start(); } @FXML public void handleIncrFrameOffset(ActionEvent e) { frameOffset += 1; updateOffsetLabel(); } @FXML public void handleDecrFrameOffset(ActionEvent e) { frameOffset -= 1; updateOffsetLabel(); } public void updateOffsetLabel() { offsetInd.setText("" + frameOffset); } @FXML public void handlePlayPause(ActionEvent e) { playing = !playing; if (playing) { playPause.setText("Pause"); try { //mView.getMediaPlayer().play(); server.currCmd = 'p'; } catch (Exception ex) { System.out.println("no file"); handleReset(e); } } else { playPause.setText("Play"); try { mView.getMediaPlayer().pause(); server.currCmd = 'q'; Creative Robotics Studio 65 } catch (Exception ex) { System.out.println("no file"); } } } @FXML public void handleReset(ActionEvent e) { try { mView.getMediaPlayer().stop(); currentFrame = 0; server.currCmd = 'r'; } catch (Exception ex) { System.out.println("no file"); } playing = false; playPause.setText("Play"); } @FXML public void handleLoad(ActionEvent e) { mView.setMediaPlayer(null); FileChooser fChooser = new FileChooser(); fChooser.setTitle("Open Media File"); fChooser.getExtensionFilters().addAll(new ExtensionFilter("Video Files", "*.mp4")); try { File f; if (QuickSync.autoLoadFile) { f = new File(QuickSync.filePath); } else { f = fChooser.showOpenDialog(getScene().getWindow()); } Media m = new Media(f.toURI().toString()); MediaPlayer mPlayer = new MediaPlayer(m); mView.setMediaPlayer(mPlayer); mPlayer.currentTimeProperty().addListener((observableValue, oldTime, newTime) -> { // int frame = (int) (newTime.toMillis() / 30.0); // if (frame % 10 == 0) { // System.out.println("time: " + frame); // } }); mPlayer.setOnReady(() -> { totalDuration = mPlayer.getMedia().getDuration().toMillis(); System.out.println("total duration: " + totalDuration); System.out.println("height: " + mPlayer.getMedia().getHeight() + "\nwidth: " + mPlayer.getMedia().getWidth()); mainPane.setPrefHeight(mPlayer.getMedia().getHeight()); mainPane.setPrefWidth(mPlayer.getMedia().getWidth()); double x = 41; double y = 18; if (stage.getStyle() == StageStyle.UNDECORATED) { x = 0; y = 0; } getScene().getWindow().setHeight(mPlayer.getMedia().getHeight() + x); getScene().getWindow().setWidth(mPlayer.getMedia().getWidth() + x); Creative Robotics Studio 66 }); } catch (Exception ex) { System.out.println("well it looks like you messed up: " + ex); } } private class ShortServ extends Thread { private boolean terminate = false; ServerSocket listener; Socket client; int port = 5515; BufferedReader in; PrintWriter out; char currCmd = '.'; public ShortServ(int port) { this.port = port; this.setDaemon(true); } public ShortServ() { this(5515); } public boolean terminate() { terminate = true; interrupt(); return true; } @Override public void interrupt() { super.interrupt(); } @Override public void run() { try { listener = new ServerSocket(port); System.out.println("The server is running on port: " + port); Platform.runLater(() -> { indicator.setFill(Color.web("FF0000", 0.8)); }); client = listener.accept(); BufferedReader in = new BufferedReader(new InputStreamReader(client.getInputStream(), "UTF-8")); PrintWriter out = new PrintWriter(client.getOutputStream(), true); // Send a welcome message to the client. String input = in.readLine(); int robotFrames = Integer.parseInt(input.split(",")[1]); System.out.println("robotFrames: " + robotFrames); out.println("server accepted"); Platform.runLater(() -> { indicator.setFill(Color.web("FF8800", 0.8)); }); Creative Robotics Studio 67 while (!terminate && currCmd != ';') { String str = in.readLine(); double fac = 0.0; try { fac = (mView.getMediaPlayer().getCurrentTime().toMillis() / totalDuration) * 100000.0; } catch (Exception ex) { System.out.println("no media loaded"); } String numPart = str.split("\\|")[1]; String robState = str.split("\\|")[2]; //System.out.println("reported: " + numPart + ",actual: " + (fac * robotFrames) / 100000.0 + ",robState: " + robState); if ((fac * robotFrames) / 100000.0 == robotFrames) { currCmd = 'q'; } int reportedFrame = Integer.parseInt(numPart); out.println(currCmd + "," + fac + "," + frameOffset + "|"); if (currCmd == 'p') { mView.getMediaPlayer().play(); } if (robState.equals("s")) { Platform.runLater(() -> { indicator.setFill(Color.web("FFFF00", 0.8)); }); } else if (robState.equals("p")) { Platform.runLater(() -> { indicator.setFill(Color.web("00FF00", 0.8)); }); } } in.readLine(); out.print(";"); client.close(); listener.close(); } catch (IOException e) { System.out.println(e); } finally { System.out.println("server terminated"); } } } } FXMLMain.fxml Creative Robotics Studio 68 Creative Robotics Studio 69 Creative Robotics Studio 70 Appendix 2 – RAPID Code Move Command Playback – note that the array has been abbreviated with an ellipsis MODULE BASICPLYBCK PERS loaddata monitor:=[9,[0,0,0],[1,0,0,0],0,0,0]; VAR jointTarget jt; PERS bool performReset; PERS string numPart; PERS bool terminate:=FALSE; PERS num currFrame:=2; PERS bool paused:=TRUE; PERS num currFac:=0; VAR num interpOffset:=0; !this value is reported as a double between 0 and 100000, so remember to divide by 100000 loser PERS string robState; !tell the server what the robot is currently doing: 'p' for playing, 's' for standby PERS num fOffset:=30; PERS num targLen; VAR num targs{7160,7}:=[ [0.00000,-138.64163,-13.18544,-20.54156,95.76308,39.84986,-71.72021], [0.03333,-110.04206,31.17579,19.19065,0.00000,23.30992,-0.00000], ... [0.03333,-102.49644,0.91456,-80.17713,-18.18201,-24.54285,18.45911], [0.03333,-102.49644,0.91456,-80.17713,-18.18201,-24.54285,18.45911]]; PROC main() !VAR jointTarget jt :=[[targs{a,2},targs{a,3},targs{a,4},targs{a,5},targs{a,6},targs{a,7}],[9E9,9E9,9E9,9E9,9E9,9 E9]]; currFrame:=fOffset; interpOffset:=0; targLen:=Dim(targs,1); robState:="s"; toStart; playForLoop; ENDPROC PROC playForLoop() VAR num divideBy; VAR num derivedFrame; VAR num a; WHILE NOT terminate DO IF currFrame>targLen THEN paused:=TRUE; robState:="s"; ENDIF IF paused=FALSE AND performReset=FALSE THEN derivedFrame:=targLen*(currFac/100000.0); divideBy:=33.0+(derivedFrame-currFrame); Creative Robotics Studio 71 IF NOT interpOffset = fOffset THEN interpOffset:= interpOffset + ((fOffset - interpOffset) / Abs(fOffset - interpOffset)); ENDIF currFrame:=currFrame+1; a:=currFrame+interpOffset; IF a>targLen THEN a:=targLen; ENDIF IF a<1 THEN a:=1; ENDIF IF divideBy<=0 THEN divideBy:=15.0; ENDIF jt:=[[targs{a,2},targs{a,3},targs{a,4},targs{a,5},targs{a,6},targs{a,7}],[9E9,9E9,9E9,9E9,9E9 ,9E9]]; MoveAbsJ jt,vmax\T:=1.0/divideBy,Z80,tool0\WObj:=wobj0; ENDIF IF performReset THEN toStart; ENDIF !MoveAbsJ jt,vmax\T:=targs{currFrame,1},Z10,tool0\WObj:=wobj0; ENDWHILE ENDPROC PROC toStart() jt:=[[targs{2,2},targs{2,3},targs{2,4},targs{2,5},targs{2,6},targs{2,7}],[9E9,9E9,9E9,9E9,9E9 ,9E9]]; MoveAbsJ jt,v500,fine,tool0\WObj:=wobj0; interpOffset:=0; ENDPROC ENDMODULE Server Communication Module – run in a separate task on the IRC5 controller. Note the shared PERS variables between the two modules. MODULE SOCKPOLL VAR socketdev socket1; VAR string received_string; PERS bool performReset; PERS string numPart; Creative Robotics Studio 72 PERS bool terminate:=FALSE; PERS num currFrame:=2; PERS bool paused:=TRUE; PERS num currFac:=0; !this value is reported as a double between 0 and 100000, so remember to divide by 100000 loser PERS string robState; !tell the server what the robot is currently doing: 'p' for playing, 's' for standby PERS num fOffset:=35; PERS num targLen; PROC main() runSockBk; ENDPROC PROC runSockBk() SocketCreate socket1; SocketConnect socket1,"127.0.0.1",5515; SocketSend socket1\Str:="4,"+NumToStr(targLen,0)+"\0A"; SocketReceive socket1\Str:=received_string; TPWrite "Server wrote - "+received_string; received_string:=""; ! Continue sending and receiving SocketSend socket1\Str:=GetSysInfo(\RobotType)+"|"+NumToStr(currFrame,0)+"|"+robState+"\0A"; WHILE NOT terminate DO ! in order to begin with a recieve statement there needs to be another send statement before hand to get things to line up on the server end SocketReceive socket1\Str:=received_string; robState:="s"; IF StrMemb(received_string,1,"p") THEN paused:=FALSE; performReset:=FALSE; robState:="p"; ENDIF IF StrMemb(received_string,1,"q") THEN paused:=TRUE; ENDIF IF StrMemb(received_string,1,"r") THEN paused:=TRUE; currFrame:=2; performReset:=TRUE; ENDIF IF StrMemb(received_string,1,";") THEN terminate:=TRUE; Creative Robotics Studio 73 ENDIF parseMsg; !playSingleFrame; SocketSend socket1\Str:=GetSysInfo(\RobotType)+"|"+NumToStr(currFrame,0)+"|"+robState+"\0A"; ENDWHILE ! Shutdown the connection !SocketSend socket1\Str:="Shutdown connection"; !SocketReceive socket1\Str:=received_string; !TPWrite "Server wrote - "+received_string; SocketClose socket1; ENDPROC PROC parseMsg() VAR bool ok; VAR String diff; VAR num out; VAR num len; VAR num found; VAR num found2; found:=StrFind(received_string,3,","); found2:=StrFind(received_string,3,"|"); len:=StrLen(received_string)-2; numPart:=StrPart(received_string,3,found-4); ok:=StrToVal(numPart,out); currFac:=out; numPart:=StrPart(received_string,found+1,found2-found-1); ok:=StrToVal(numPart,out); fOffset:=out; !RETURN out; ENDPROC ENDMODULE