Research showreel

1 First some personal timeline.

I’ve studied both at the Aalto schools of Science and Arts. Got my doctoral degree in 2007, then been working in the games industry.

2 3 There are many definitions. The image was selected because I remembered the second definition just a few days ago when my son decided to start walking only on the seams of the pavement at Helsinki railway station. In other word, he made the journey a game for himself by creating an artificial but fun challenge for himself.

4 Digital games are a medium that can encompass all others: text, visuals, audio, story, video, comics...

5 Games are also a growing business and the most significant Finnish cultural export area. In addition to the design and technology, the business side has been evolving rapidly.

The latest trend is Free to Play, which means that the game is a free download, and money is made through in app purchases, which help the player progress. One may decide whether to grind for hours or to skip the grind and pay up. The specifics of free to play mechanics are currently evolving.

One may also pay for the game with a variety of currencies. For example, Candy Crush let’s the player to decide whether to pay for new levels or advertise the game on facebook, which effectively means using up one’s social capital. Advertise too much and people will hide you from their feeds.

6 Currently in news: Rovio’s fall from the 100 top grossing list in App Store. On the other hand, Rovio just launched a video network – distributing an Angry Birds animated series through their games, reaching a wider audience than many television networks.

Mobile gaming future looking bright because it attracts new users – casual gamers who might have started playing games on Facebook and have migrated over to mobile thanks to the streamlined user experience, and mobile offers better tools for charging money and paying.

The statistics might not be fully accurate, they are based on what I heard from some industry executives a few days before making this talk. They are probably in the ballpark anyway.

7 The games business side of course comprises other things than games as a product that the customer pays for. Here’s an example of games in advertising, combining recent technology and gameplay (a Kinect dancing game) with advertising in a public space. It’s also a vending machine, rewarding players with a free bottle of coke.

8 9 Considering games as design, there’s four main components in the form of the Elemental Tetrad of Game Design (from the book The Art of Game Design by Jesse Schell).

The figure also shows the visibility of the components. On the surface there’s aesthetics, and as the important but less visible basis there’s technology.

10 11 There’s a variety of definition for a ”game mechanic”. For some authors (Rogers) it’s about equal to an interactive element such as a lever. Other authors consider mechanics equal to the rules of interaction, speaking of, e.g., the ”jumping mechanic” which includes when and how the character can jump, double jump and how it is implemented.

Game mechanics is also where a computer science student may excel over self-taught game programmers. They are often basically simple to implement, but polishing and finetuning may benefit from some science background. For example, physics simulation is not that trivial because it needs to have a solid basis in reality, but on the other hand, game design may require an artistically motivated deviation from reality. Understanding how to build or tweak physics engines benefits of an understanding of the underlying math.

Also, game camera movement is a challenging field where it is difficult to come up with simple and robust algorithms/rules that work in all the special conditions, such as player moving into a corner or inside crowded and tight spaces, below obstacles etc. Signal processing, e.g., knowledge of different low-pass filtering (smoothing) techniques helps in achieving a believable and pleasant camera movement. I’ve often used the following, which one will encounter on a typical audio or image processing course: a box filter (FIR with equal tap weights), a first order IIR lowpass, and a variable timestep double- exponential filter.

12 Other useful skills/technical skills: basics of machine learning (kernel regression, decision and regression trees/forests), numerical optimization (Newton, gradient, Jacobian transpose, stochastic optimization), Bayesian inference, particle filters (e.g., Sampling Importance Resampling)

12 From Level Up! by Scott Rogers. To illustrate that game mechanics are not invented out of thin air but the evolution and innovation is typically incremental, with totally new genres spawning only occasionally, often inspired by a new platform or technology.

13 From Theory of Fun by Raph Koster

14 Modern innovative gameplay (Portal by Valve)

15 The modern innovative indie platformer (Braid, inspired by Blinx: the Time Sweeper.)

16 Katamari Damacy. Both innovative gameplay (rolling things into a ball) and technology (dealing with scale changes from miniature to planetary on a PS2)

17 I agree with Rogers in that one needs to find just the right amount of story for each game. Some backstory helps in motivating the players, but too much dialogue or cut scenes can kill the interactivity. No story is better than a boring story.

Heavy rain: mechanics mostly serve the story, make a gesture to continue. Some players dislike, but has also its fans.

18 According to Rogers, Robocop is a good example of games missing the importance of the first act in the classic three act story structure. In the movie, the main character almost dies at 25 minutes, only to be resurrected as Robocop. By this point, the viewer has established a bond to the character’s human side and the love story has been set up, without which the movie would be a lot shallower.

19 However, all Robocop games skip this crucial part and start from the player as the Robocop – a huge missed opportunity considering the emotional depth of the playing experience.

20 Alyx from Half Life 2 as an example of a character with which the player can become emotionally attached to. For me, Alyx’s voice instructing me through headphones inside an enemy base, with her face occasionally showing up in a security monitor was the first moment of such gameplay intimacy that I begun to wonder if I had actually become at least a bit infatuated with a game character.

It was both embarrassing and awesome.

A deep and strong emotional experience is still more of an exception and not the norm in games, and remains a challenge for researchers and designers alike.

21 The importance of aesthetics must not be underestimated, although some game designers like to focus on the mechanics. There’s more and more evidence of humans perceiving the world through a holistic process that integrates various senses and cues, and our percepts of visual beauty can for example be affected by audio. Schell cites a study where the same game was ranked lower regarding the visuals when the players were listening to an audio track that lacked high frequencies.

From the study of human cognitive biases, we know that a single attribute may dominate our judgement, masking away deficiencies in other areas. For example, good looking politicians tend to get more votes even though they might be terrible candidates otherwise. Considering games, Trine 2 is an example of glorious visuals selling a quite traditional puzzle platformer.

22 Regarding visual design, many people recommend the book Drawing on the Right Side of the Brain, which is about learning to see and observe details that we don’t pay attention to.

23 An example of the effects of increasing photorealism in games.

24 Limbo is another example of a novel visual style as the selling point for a game with quite a traditional platform game mechanics. The approach also made it possible to produce the game with a small team and budget, because there was no need for texturing or highly detailed 3d geometry.

25 Fez has both unique style and mechanics.

26 Technology is what underlies and enables everything else in the elemental tetrad.

27 Having discussed the elemental tetrad, I’ll now focus on technology, which is the most relevant for most of you computer science students.

28 There’s been several transforming technological breakthroughs throughout the history of gaming. In the following, I’ll first show some examples from history and then look at some upcoming things

29 3d graphics and mouse are probably one of the most fundamental ones, spawning the first person shooter genre in the beginning of the 90’s when the PC started to have enough CPU power for textured 3d instead of just lines or colored polygons.

30 Nowadays graphics technology is mostly not the limiting bottleneck for gameplay innovation, and we have to look elsewhere for inspiration.

31 Having said that, there’s still interesting things happening if one looks at the demo scene, especially the 4k intros, which force the focus on procedural content generation instead of just rendering visuals and sound designed using standard artist sofware packages. Here’s Cdak by Quite & orange (http://pouet.net/prod.php?which=55758)

32 In addition to graphics, I think physics engines deserve a mention, as people have been innovating physics based gameplay for some decades now, and there’s still fresh ideas popping up, although not so often anymore. Getting started with physics based gameplay design is easy nowadays, as one can download a physics engine such as Box2D or Bullet and have an Angry Birds clone up and running in a matter of hours.

33 The Incredible Machine, from early 90’s. A direct ancestor of, e.g., Rovio’s Amazing Alex.

34 In iOs, we’ve seen a rapid evolution of physics based and hugely successful titles starting from Angry Birds

35 Cut the rope added rope physics

36 Disney’s Where’s my Water is heavily based on fluid simulation. The goal is to get the water to the crocodile so that it can take a bath.

37 Story is maybe an area where technical innovations have not had as big an impact, other than in procedural level creation. There’s also attempts in procedural story generation, but as far as I know, they haven’t produced big hits yet. With help of computer vision technology, we managed to create a new form of storytelling in Kung-Fu Live, using the images of the player embedded in an animated comic book.

38 There’s many other technologies that haven’t yet been fully utilized in games. Personally, I think that machine learning and AI may be the next big thing, because the range of methods that can be implemented has expanded because of parallel and cloud computing.

One particular important subfield of machine learning is generative statistical models, which can be used to synthesize both game assets or character behavior based on a dataset of training examples. The methods have progressed rapidly in the 2000’s, e.g., considering sparse kernel models, but game programmers are only now finding and studying them.

39 Using new input techologies, such as multitouch, computer vision and real- time analysis of the microphone signals, one can analyze and score many real-life skills in a game context. Here’s Draw Something – basically Pictionary utilizing Multitouch and Facebook.

40

I originally studied audio signal processing and electronics and ended up working in one of the world’s first singing games back in 1999 (on the left, the game was called Soittopeli). Later on, we’ve had the SingStart and Karaoke Revolution franchises. The games give immediate visual feedback of the user’s voice, which can help in singing in tune.

41 Ovelin, a Finnish startup takes the pitch detection to a new level. With good enough echo cancellation and polyphonic pitch detection they’re able to let the player play a real guitar to play an iPad game. Rock Prodigy also does this, but it doesn’t work as robustly without headphones. Using headphones prevents the game’s audio output from leaking back through the microphone and interfering with the pitch detection.

42 With Kinect, we can now track and analyze the full human body, although the technology is not perfect yet. Kinect tracks best when one stands facing the camera, and the tracking becomes unreliable when body parts occlude each other, e.g., when standing sideways or crouching or lying low on the floor.

43 Big sports brands like Nike have recently joined the motion gaming bandwagon.

44 45 With augmented reality, the motion/physical games can also be taken outside the living room. Here’s Nerf Lazer Tag that let’s one embed a smartphone into the gun, bringing the experience closer to a real-life FPS with augmented reality powerups etc.

46 One way into which mixed reality and physical games will probably evolve is utilizing more input (sensory) and output (motor) enhancement technologies. Here’s Brainport, a tongue display. The human tongue has such a high density of nerve endings that it’s possible to project an image to the tongue using an electrode plate. The human brain is plastic enough that with enough practice, one can begin to get an intuitive spatial view based on the display. There’s several possibilities of using this to augment the senses of blind people, soldiers, or gamers.

47 About motor enhancements: many perfectly healthy (physically…) people nowadays play and trick with power stilts, with which one can, e.g., backflip over a van. Don’t try this at home…

48 Slamball is an example of an extreme sport inspired by computer games and realized using trampolines as the motor enhancement technology.

49 Easy to see that Google’s self driving car and IBM’s Watson (natural language question answering system that understands puns and won a television quiz against humans) comprise algorithms that could be used for game NPC:s.

However, Watson may also have implications for the game development process. A big bottleneck is that often there’s only a few people who know it all, so new employees asking questions and learning about the codebase causes a penalty on the overall productivity. Within some years, we might have a natural language interface that could answer questions like ”I need to implement feature x. Can you suggest me what parts of the code I should look at or might be affected?”

50 Data mining, machine learning and information visualization are also used in improving games based on metrics. The figures show Halo heat maps from Microsoft’s usability labs. The heat maps show, e.g., where players die the most, which may help in spotting problems in the level design.

51 Parallel and grid computing is perhaps currently used more in movie production, e.g., in case of RenderFarm, which allows Blender users to share their computing power for rendering.

52 Better yet, we don’t have to limit ourselves to silicon processors, as network technology and parallel problem solving design can also exploit human wetware. Fittingly in this context, we can use game design to motivate human optimizers, as in the case of FoldIt.

FoldIt allows players to try to optimize and discover molecules, e.g., for new medicine, and there’s been some success too. The creators have also added an additional twist: they are now improving automatic protein folding algorithms based on the strategies of the players.

53 Thinking beyond games, one can apply gamification techniques such as leaderboards, badges, and achievements to, e.g., online communities, again using game design thinking to motivate sharing of human resources, as in the case of Sharetribe or Quora.

54 55 Continuing on a tangent about using games and gamification to control human behavior: I think game designers and all interaction designers have a lot to learn about how magicians hack the human brain, manipulating our attention and exploiting misdirection and cognitive illusions.

Games and magic are related as entertainment experiences, and also because magic is one of the oldest forms of interactive storytelling. In many tricks, the audience can choose options, such as a card, which creates a branch in the storyline of the show, but the magician is still able to guide the experience towards a dramatic and surprising finale.

Throughout history, magicians have based their powers on emerging technologies, such as magnetism. Nowadays, many claim to be basing their powers on psychology, which is actually partly true, considering their expertise in misdirection and cognitive illusions.

56 Books related to the science of magic and cognitive illusions.

57 Now a brief overview of current research

58 At Aalto Department of media technology, my group mainly tries to innovate gameplay and game production tools through utilizing or creating new technologies. We also use games as research instruments to study, e.g., user experience and psychology.

59 Currently, we try to take motion games to the next level by letting us not be confined to the living room. One particular topic is exaggeration of player’s movements and skills, which we have found to be motivating for players. This is in agreement with Malone’s intrinsic game motivations: curiosity, challenge and fantasy. The exaggeration adds a fantasy element. It has also been found the one’s perceived competence predicts sports training motivation.

We’ve previously successfully exaggerated player’s jump height etc. in the game world, but now we are experimenting on doing the same also in the real world, e.g., using trampolines.

One of our high level goals is to develop new and engaging half-digital extreme sports with a feeling of speed and danger but without actual risk of injury. Statistically speaking, the more new sports are developed, the more likely it is that everybody finds their own, which is of course desired from the point of view of national health.

60 Another research question is how to provide fast and effective feedback for learning motor skills through games, similar to the singing and guitar games discussed before.

Here’s an example of a cartwheel training interface that tries to present a simple numeric metric (flatness in depth direction) that is difficult to measure and display with traditional means.

61 Once the user starts the cartwheel, the system begins recording automatically.

62 After the cartwheel, the system presents feedback that shows the worst frame of the move, which helps in figuring out what to improve.

63 In addition to motion games, we are also developing an animation AI system that might distribute the computation similar to RenderFarm. The goal is to transform the animator into a choreographer that gives goals, instructions and examples, and AI (machine learning and optimization) then figures out how to reach the goals in a physically valid manner.

We also hope to get the system at least partly running in realtime, which would enable many novel game mechanics.

The research is also about choreographer’s interface to the characters – is it easier to use a 3d input device to give the instructions or define a target pose?

64 Jaakko Lehtinen is our new professor of computer graphics, with a long history from the games industry.

65 There’s also people who use high end research tools such as brain imaging in conjunction with games.

66 67 68 69 One could say that we are following the T-shaped skills paradigm, meaning that our graduates should have at least one strong speciality that lets them produce production quality, and a wide basis of other skills that helps in prototyping, communication and teamwork in general.

70 The last part of my talk will consider game design, programming and education as an iterative experiential learning process. The tools and methods for rapid iteration are and active research area, and understanding the importance of correct tools and feedback optimization can also have an impact on your personal learning.

Can’t stress this enough: Choose and built the right tools to get the most out of your time in Aalto.

As an example, if you want to become a game designer or programmer, learn or build an rapid iteration game engine and development environment like the ones described in the following. Currently, the students responsible for the programming side in our game projects have suffered because they decided to use ”old school” game engines that they had learned instead of modern rapid iteration engines like Unity3D. They have been able to build initial prototypes very quickly, but later on in the project, progress has stalled because 1) they have had to start building tools such as a level editor or 2) building and loading the game after code changes takes minutes, which slows down the gameplay iteration and polishing where one must try out various parameters and alternatives.

71 So, what is game design on a fundamental level? If I had to pick one characteristic thing, it would be the unpredictability, meaning that because a game designer can never fully predict how players will behave, there’s no quarantee that the game works the same way on paper and in practice, although of course you get better the more games you make.

72 Here’s one of my own experiences of the unpredictability. We were making a game where you fly as a dragon by moving in front of the webcam, and I learned the hard way that the technology has to be REALLY robust to cope with all the moves that children will try.

73 So, how do we fight it? Most designers and scholars agree that one must just acknowledge the unpredictability and try to deal with it using agile and iterative development processes. Since it’s impossible to know that an idea works for sure, it has to be tested in practice by creating a prototype, be it a paper prototype, pseudointeractive WoZ or an actual digital game.

74

The rule of the loop by Jesse Schell postulates that the more prototypes one can build and test given the time and money, the better the game will be.

75 This is also related to the concept of the design funnel (image here from Buxton’s Sketching User Experiences), meaning that in the beginning of a project, one should use cheap and efficient methods to create a lot of sketches and prototypes and then prune and discard the ideas until the design converges to the optimal one. The less ideas you are working on, the more expensive methods you can use.

76 So, what kind of people are ideal for rapid iteration, sketching and prototyping? One answer is that we need to educate designers who can program enough to try out their ideas in practice.

Delegating prototyping work to others takes time, and programmers hate when their they implement something that doesn’t necessarily work.

77 The designer has to either be able to create prototypes himself/herself or give considerable creative freedom to the programmers, which of course means that one needs programmers with good enough design sense and skills. If you want this career, I suggest you apply to the Game Design and Production minor subject / study module in Aalto Arts.

78 Now many of you may be thinking I’m crazy trying to teach programming to designers, because the truth is that programming can be difficult.

79 The great thing is that the rule of the loop applies to learning as well. The more you practice, the better you get, and the learning cycle can be optimized with proper tools.

80 Many real-life skills are not just memorization and recall of information, but about learning procedures and methods, and understanding how large systems work. This can be achieved by experiential learning, a dialogue with the tools, problem and creation, where one tries out things and figures out how to improve based on feedback from a teacher or the tools (such as a debugger in case of programming) or simply by observing the results.

The key here is drilling the skills so that it becomes automatic and requiress less conscious, analytic thought which is slow (see, e.g., Kahneman’s Thinking Fast and Slow about the division of work between the conscious mind and the adaptive subconsious/intuition) . Best athletes, chess players, programmers etc. can do their thing based on intuition, with the analytical mind only occasionally intervening. The way our brain works is that this is achieved through repetition.

Since one needs repetition, the essential questions are how to decrease the amount of repetitions needed and how to go through the cycle faster. The right feedback tools help in both, which applies to a wide range of skills from programming to sports. For example, many athletes nowadays use a video camera and a software that delays the video on screen so that one effectively gets an instant replay of a golf swing or a backflip. One doesn’t need to spend time operating, e.g., a video camera, and the video replay can help in spotting

81 errors. In programming, the tendency is to move away from the code, compile, build, load, test –cycle to making changes and compiling code live without noticeable build and load times.

81 I might be a bit out of my depth here, as I’m no pedagogy expert, but think of learning and intuition as function approximation or regression. You are training your neural circuitry with examples (the trials in the experiential learning iteration, the dots in the figure). You see that x (a procedure, an approach) produces y (the outcome, results), and with enough examples, you figure out the general relation. Though repetition you can learn to intuitively predict the y for a given x, as well as the inverse – what procedure or parameters do I need to get the desired results?.

Consider learning to throw a ball – you increase and decrease the amount of force x until you learn to hit the target at the distance y. By practicing with targets at different distances, you learn to apply the right x for the given y intuitively. Of course, you may learn about a relation by reading it from a book, and carry out an algorithm to get the right solution for a given problem, but practice and repetition helps in developing the intuitive hunches of the solutions, which is crucial for working rapidly enough in game design, programming, research or other creative and competitive areas.

A fascinating example of this intuitive, subconscious ”function approximation” or ”mapping” capability of us, from the book Thinking Fast and Slow: Sarah learned to read at the age of four. How tall is a man who is as tall as Sarah is smart? Most people produce answers of surprising coherence.

82 Bret Victor’s speech is mandatory for anyone learning programming (Don’t waste your time with inefficient tools!) or designing game development tools (Think big! Don’t compromise!).

83 Just to remind you, there’s many kinds of prototypes, most of which don’t require programming

84 Bill Buxton’s book has good ideas about efficient sketching and prototyping interactive software.

85 Khan Academy has implemented some of Bret Victor’s ideas, and also added a tutor soundtrack and the ability to pause lessons at any time and start tinkering with what the tutor has coded so far.

86 87 Considering commercial game development tools, Unity3d is perhaps closest to what Bret Victor is after. Highly recommended. The free version of Unity has some limitations but is perfectly suitable for learning.

88