When VR Really Hits the Street Panel presented at the 2014 SPIE “Engineering of ” Session. Session Chairs Ian McDowell and Margaret Dolinsky February 3, 2014

Panel Moderator Jacki Morie, All These Worlds, LLC (JFM) Panelists, VR Pioneers Brenda Laurel (BL) and Margaret Dolinsky (MD) Audience member participants (AM)

JFM: Welcome everybody. I am really excited to be doing this panel. My name is Jacki Morie and I have some illustrious VR people here with me today.

Here’s our schedule on the slide, which we’ll try to keep to so we can get a lot of information covered.

This is not only about us up here as panelists; it is also about you. We have an extra seat up here and there will times during the presentation that ask YOU to come up and be part of the panel.

This panel was actually inspired by recent events – that Silvia (Ruzanka) mentioned in her talk earlier – the new devices coming out, how inexpensive they are, and how much better they are than what we had 20-25 years ago.

So many of us who have worked in VR are looking at this and wondering: What is this? Is this the Second Coming, or is this the Second Coming of more hype? We have to figure this out. So my idea was to bring some experts in and have this conversation.

Do we really have what we dreamed of 25 years ago or is there still a big gap between what we need and what’s still coming in today?

So that’s what we are going to be talking about today. Now let me introduce my esteemed guests.

Brenda Laurel has been called a techno diva (audience laughs), which is one of the milder descriptions, but one that fits her very well.

And among her published books, Computers as Theater stands as a really foundational work that radically redirected our thinking about computers and our relationships to them. There is a new version out now – 22 years later.

Her tenure at the Banff Centre in Canada during their Art and Virtual Environment Project in the early 1990s resulted in a groundbreaking VR that pioneered a number of great techniques. That included networked participation and virtual objects where you could actually leave a voice message for somebody – which I thought was amazing and which haven’t seen much of since.

More than anything over the years, from creating a company to actually commercialize the technology – a company called Research – to doing the artwork, in all the work she has done in VR and other disciplines she has always presaged the future of what our technologies can do.

Our second illustrious panelist is Margaret Dolinsky, whom I am sure all of you know due to your association with this venue. Margaret is an Associate Professor and a Research scientist at Indiana University Bloomington. She is both a pioneer and one of the most prolific artists doing immersive VR art with a career that spans almost two decades now. Her work and her research has not only help push the practical technologies of VR like the CAVE but it has also, more importantly, pushed the aesthetic possibilities, and the way we think about what content we think could be in these immersive media.

For me I started in VR in 1989 when I wiggled my way into a research lab in Orlando that was doing some work for the military because I thought VR was just the coolest thing around and I had to get my hands on it. We were working on really basic stuff for the Army, like what kind of visual acuity do you actually have in the VR helmet and can we actually use VR to train people to become familiar with their surroundings. I thought that was boring. So with another colleague, after hours, we created VR environments that evoked emotional responses from participants. He was a psychology major turned computer scientist and I was an artist turned computer scientist, and together we made a number of these environments that evoked emotional responses.

So there you have 3 of us here. We all have some sort of Art connection as well as a connection to pushing technology in ways it wasn’t expected to go.

Now I want to take a couple of minutes to have our panelists make a couple of personal statements, answering the questions: What was your first VR experience? What HMD did you use? What was your first time? and: What were your thoughts about it? Did it change the way you thought about Life living and the nature of the universe or was it just another “oh this is cool.”

Let’s start with you Brenda.

BL: The first time I had my head in it, I believe was 1986 at NASA Ames Research in Scott Fisher’s lab and it was a Leap HMD with your basic data glove, which everyone invented. Eventually I got some funding, pulled Scott out of NASA and we started Telepresence Research.

We did some great work, but realized about 6 months in that we didn’t have a business plan. It was too early to monetize the technology in any reasonable way. So, I parted company with Scott and went on to university work.

I should say Telepresence was a dream team: Mark Bolas, Scott Foster (who made the Convolvotron), Scott Fisher, Michael Naimark, Rachel Strickland and Steve Saunders. These people keep showing up on my life in various disguises.

I think my biggest contribution to the field obviously was the Placeholder project at Banff. We had extra money from Interval Research. We had a million dollars from them to play with, more than Banff could provide. Our team included Rob Tow, John Harrison, Michael Naimark, Scott Foster, Steve Saunders, a lot of the Telepresence Research gang.

I think the reason we did Placeholder was as a design statement. It was to say that beyond these training simulations with drop down menus – which I never understood – why would you go to the trouble to produce a three-dimensional virtual environment with a pull down menu? We wanted to demonstrate that this could really be Through the Looking Glass, that we could explore environments of the imagination in an active way. And that there could be spiritual psychological and emotional value to that and it was a possibility of the medium.

That thread was certainly picked up by certainly picked up by Char Davies with Osmose. And Margaret, the queen of VR has given us another view of the kind of visceral emotional connections we can have. For me it was “dream soup” –

JFM: and you were swimming in it.

OK Margaret. Was your first time with an HMD or was the CAVE your first?

MD: The CAVE was definitely my first and something I really, really love. I was actually interested in going to grad school for painting and I found out about the EVL at the University of Illinois, Chicago. I went over for a visit and they were doing demonstrations, and the first thing I experienced was a roller coaster in the CAVE. There were a lot of people in the CAVE and someone yelled “Throw your hands up in the air.” And we all threw our hands up in the air, and we flew down this roller coaster and I was “Wow this is amazing!” It was this group experience in this 3D stereo space and it was very visceral

And I thought: This is a good place for my paintings I could finally be in a place where the characters from my paintings could be as well and I could actually walk through their world and get them to respond to me. So I was hooked. I have been doing VR ever since.

I still teach VR and my student s have VR exhibitions at the end of every semester, and people come from all over campus come to see them. I have been doing this for about 15 years.

JFM: It’s interesting that you had a very social experience for your first one, where you, Brenda probably had a much more insular experience. And the early days you pretty much by yourself in VR.

BL: Until Scott and I working together, and Michael Naimark who was doing other things with projection and presence and then we met Mark and the Fake Space Lab guys and saw all the commonalities we had.

JFM: My first time was using a Leap HMD with the breastplate counter weight you needed to wear to counterbalance the weight of the device on your head. I have no recollection of what environment I went into. I just knew I was there and that I would be working in this area.

MD: We should mention that Mark Bolas started this conference. BL: Really? So where the hell is he? JFM: And Ian is still carrying the torch here.

MD: The first fantastic HMD experience I had was when Ian brought me over to Fake Space Labs and put one of his new devices on my head and that was a very memorable moment. Fakespace was awesome. IS – There is still Fakespace!

JFM: I do want to ask the audience a few questions Because we talking about this new dawn, where there are now affordable devices out there, and we are in the age of the crowd sourced this and the crowd funded that, I want to ask how many people out in the audience have supported one of these new technology campaigns on something like Kickstarter or IndieGoGo within the past year. How many of you have put your money into that? ( little more than half)

Anyone want to say what they funded? How many backed the Rift? (about half of the audience). Anyone do a different device?

AM: The STEM a wireless version of the Razor Hydra from SIXSENSE.

JFM: How many of you plan to use some of these Kickstarter devices in your own work in the next year? (About 2/3 of the audience). Oh this is encouraging.

How many of you actually had the on and been able to get into an environment with it? (quite a few) Has anyone tried the latest Oculus Rift (higher res, less lag, more tracking points) unveiled at the CES show recently? (no one)

There are others, like the CastAR, and many more cool things to watch.

OK that gives us a sense of what the audience has been up to

And now I am going to ask the panelists a couple of questions, and after they have answered if the audience wants to weigh in you can come up to one of the empty chairs, because this is where you get to come up and be part of the panel.

So Brenda when did you stop working in VR and why?

BL: We ran out of money. We couldn’t make payroll and I didn’t feel right about continuing. So, I left Telepresence and ended up working at Interval Research. The Placeholder Project was actually the last thing I did.

I started working with critical theory when I saw Char’s stuff. So I have been more engaged in the discourse than the practice. And that is entirely because the institutions in which I have worked since Interval didn’t have that as part of the capability space. They weren’t hands-on labs.

Telepresence Research is still around though; Scott runs it as a consulting company, I believe. It was together in its original form for about a year and a half.

The Banff project was the real masterpiece for us. That sucker used 13 computers duct taped together. We had the number one Reality Engine off the assembly line. And on a good day it ran at 12 frames per second. But, it was persuasive. I have the video to prove it.

JFM: So I think you just answered one contributing reason VR didn’t happen – it used too much duct tape!

We all know – those of us who worked in the old days – that you can’t do throughput in a VR installation where somebody really gets a feel for what’s going on that’s less than five minutes that’s just a cheat, and even 5 min is pretty short to get a feel for what’s going on.

How can you stream people through and installation like that and how can you charge them for it so you can pay your bills. Disney tried, but it was a very difficult problem. It was simply unaffordable for these reasons and because of the dwell time that was needed to really experience the content.

MD: I have been working in these dark computer labs so long, nobody told me I should stop working in VR.

JFM: I don’t think you ever did stop.

MD: Like I said I have been teaching it. I now have an Oculus Rift. I had a big gallery exhibition that included a VR piece about a year ago (that was projection-based, not the Rift).

JFM: Are you planning on doing any that are more immersive that use an HMD over projection VR.

MD: I think I’ll have to because I do have this Oculus Rift. And the students are really excited about using it and they want to develop for it. I can’t get out of this game

JFM: You never got out of the game, but is there a reason you would point to why VR didn’t become a more popular medium?

MD: The biggest reason was because it was behind locked doors. It was in research laboratories that were controlled by scientists who had a lot of money and scientists who have a lot of money don’t have a lot of social skills.

They are writing grants all the time, trying to keep things funded and it’s hard to get conversations with them so that artists can get into these spaces. And once you do get in and get an art piece going, it’s really difficult to set up a time and a place where lots of people can come through and see it. And so, most people know about my work as an image here and an image there. It’s so ephemeral and you can’t really get a chance to see it. You can just hear about it or know about it.

And I think that when people don’t have accessibility to the medium, it really limits the conversation about it. Accessibility is everything.

JFM: Accessibility also affects and limits the way it can develop.

While you two did great content with your works, we have to admit that lots of what happens behind those “locked doors” is not necessarily great content as we think about media needing. It may be great for research or proving something or for training, but I would like to put one more reason why VR didn’t “make it”, and that is content.

So if someone wants to come up it is now time.

AM: You know when Gibson coined the term cyberspace and wrote his book, people said, “Oh cyberspace has arrived.” After Gibson said Cyberspace was everywhere it seems nobody wants to be in immersion. Even Gibson said “No.” They want to be in the world and to be not in a dark room. We need to be in the world. Even Howard Rheingold’s book indicates that these things don’t support what we need as humans.

BL: I think we need to remember that William Gibson is a fiction author and that when he invented the term cyberspace he was looking for a way to write science fiction that didn’t have to deal with outer space and he utterly made it up as he said in a recent interview with him in the New York Times about 7 months ago.

To me, what you are talking about is that same sort of oil slick that happened when everybody said desktop VR. Suddenly the word didn’t mean anything anymore. And I also think the kind of work that I have done, and Char and Margaret is often indexical to the natural world in a way that’s enlightening and delightful. So, I don’t see us as having to shut ourselves off from the natural world in VR. Au contraire. I think we can examine the natural world.

For example I think we can examine the natural world at scale; we didn’t play enough with scale in the old days. What does it mean to get in side the roots of the tree and go and go and go. How does that feel? What does that look like? We have enormous opportunities to explore the natural world with this instrumentality and with in ways we have not yet taken – in a loving Gaiam way or I wouldn’t be involved with it.

AM: I started my career early and at luckily Intel, where they talk about Moore’s law a lot. And I think that Moore actually told them it is a belief system with billions of dollars invested in it. But because of that enormous financial suspension of disbelief we have amazing tools

We could argue that as optimistic as people were 20 years ago, sometimes the sheer momentum of financial suspension of disbelief creates a thing. There was a dot com thing at Amazon and some people did really well taking that suspension of disbelief. The question is: is Oculus Rift a beginning or THE way, or just another device?

BL: Well it sure breaks down a major wall.

JFM: And think of this: In their first phase Kickstarter campaign, the Oculus team hoped to raise $250,000 and ended up with 2.5 million dollars. They sold 15,000 developer kits. And here’s a fact: there has ever never been 15000 HMDs in the entire world over the last 30 years. So, there’s a forcing function there in sheer numbers. Having that many out there will change the game.

AM: But we can also look at mobile. If you look at mobile technology today and you look at what Intel and Microsoft planned for, those are two very different things. The point is: It’s amazing how some people financially ran out of track for their railroad company. People always spend more money than they planned for than they already know how to justify and eventually somebody flies overhead in an airplane and it’s over.

The question is: where is this moment in VR? Is it going to the future that happened. Or is not yet there?

Another way to look at it. There’s so much stuff in the cloud it’s somewhere; it’s data, but there’s no way to use it. Google glasses can’t access all that visually. We need whole other user interfaces to do that. It’s certainly not a desktop metaphor.

JFM: So we will get to the question of what VR needs in a minute. Does anyone else want to speak to the question of “If you stopped working in VR why and what did you go to?

AM: I think it is a new tool with the HMD and users are so used to the 2D metaphor of the desktop and mouse now they won’t take the time to learn how to use the new tool. So there’s a big learning curve. They don’t t have time for a paradigm shift. The tool needs to be vital for their research. It needs to get to the point where they can’t do their research without this.

MD: I’d like to hear from Todd Margolis. He’s done a lot of artwork in VR and in many many different media.

TM: First of all I take exception of how these questions are phrased. MD: OK next person. JFM: They are meant to provoke.

TM: So what does it mean? VR didn’t happen. Obviously it has happened and it is happening. For me I look at – there’s a problem for me as a youngster. When I came unto my own as an artist, using VR as my primary medium, defining myself as a VR artist, I sort of got pigeonholed into this. And I felt that it was good and bad at the same time. On the one hand I looked at VR as the penultimate art form, where it was really a combination of many different types of art.

You had to be an expert visual artist; you had to an expert audio person. It was a lot of different things coming together. That being said, it was this highly technical expertise, and you had to be a programmer and you had to have access to these highly specialized systems. And so I co-formed a non-profit to bring VR out of the lab and to artists and galleries, and we did that to a certain extent But it was still very, very specialized to develop the content

As time went on and that situation didn’t really improve I found myself moving away from it. Now I found myself moving towards AR and for me that is kind of what I always wanted.

And the second thing I take exception to that you said before Jacki that I take exception to the statement that the Oculus Rift is more immersive than projection based VR. So this concept of immersion is really important and central to where we are at in culture, and why VR may or may not make it. It’s sort of wrestling with this idea of immersion, and this term is thrown out everywhere, now more than it’s ever has been and everyone uses it in subtly different ways.

And to me it’s really central to my work and my thinking: Is it a just sensorial thing or is there this social aspect to immersion? For example, we have this immersive experience where we go to a conference and we network at the conference we meet people and we develop ideas. That’s a form of immersion where you throw yourself into this community or culture and you thrive there, or not. That’s immersion too.

When Margaret mentioned how everyone threw up their arms together, that’s social immersion, not sensorial immersion. Sure they are all in this shared visual experience, and that what really impacted her is the social immersion. And it’s also what Silvia said before, about the game aspect of the Rift. I totally agree. There’s a lot of things converging with the Rift popular but the fact that they’re being smart about how they are bringing into the development aspect. They have all these really accomplished and experience game makers who know about social immersion, they know how to create experiences that bring people together.

That’s where I see VR going.

JFM: Would you say, Todd, that one contributing reason VR didn’t become this mass medium, was that VR wasn’t able to support the social aspect?

TM: I would say that it couldn’t, because that was a large aspect of my work; it was possible but it was very difficult. One thing that shockingly hasn’t come up in this discussion yet is . That is a VR type of headset – potentially. I think that people who don’t know what Glass really is think of it as that is a VR kind of head set – well, potentially. Some people will think of GG as VR. If you ask the average person on the street they will think it is a form of VR. They don’t know that it’s not really VR.

So Google Glass is related to these ideas, and that is thrilling because it does have that social immersion built into it. So for me that is a form of connecting people through multi- modal, multi-layered augmented experiences

MD: So are you still active in VR?

TM: Can’t I just say and leave it at that? I fluctuate back and forth along the Milgram spectrum.

AM: He’s a full spectrum man.

JFM: Maybe we need a new name because VR has connotations.

TM: I don’t want to be pigeonholed to that just what I do.

MD: I feel the same way. My last two exhibitions were sculpture with embedded electronics, and before that I was working with a biologist and we embedded my imagery into live leaves using chloroplast material.

BL: I work in Augmented Reality projects now as well. I see them, as you do, as living at ends of the continuum. AR, VR Mixed Reality. And in any case we are still concerned with this question of immersion. And you are still seeing the unseen.

One of my biggest concerns is that Oculus Rift and comparable technologies could vanish into the maw of games and we stop creating art and we don’t give ourselves the grounds to create emotionally engaging content that has spiritual value, that has personal value.

The game industry is a monster and it’s like feed the bulldog if you get involved with them. (Having been there since Atari in 1980, that’s a concern I have, that we disappear down the throat of the game business.)

JFM: Let’s go to the next couple of questions: Is it street ready yet? We can look at this from two points of view: First, is the technology getting to where it is accessible to people, and secondly, from what you said Brenda: What do we do for content to maximize and expand the way these things are used so they don’t fall into any one industry, like games.

BL: Going back to a conversation we had earlier, Jacki, a big near term challenge is the UI for the physical body in combination with the Oculus Rift. If you are not already familiar with the game controller and you can’t see the thing, then that’s not going to work.

We discovered early on that a key part of immersion is proprioception. It’s the ability to take action in a world. To move your body in space. And so we need two hands, for example, ideally two hands that can do a lot different things. One of the things we did with the Banff Project that I am very proud of is that we decoupled direction of gaze and direction of movement. (Demonstrates)

This gives you an entirely different physiological and emotional experience. So, an opportunity for what VR needs now, assuming we can continue to build affordable products, are UI devices that begin to engage the whole body again – and in a natural way. We can certainly engage the whole body now, but who wants to have to turn their head the direction they want to move.

JFM: Right now I don’t think that’s happening very much. We have one main tracking sensor. Another area we had a conversation about and I don’t know if you use this is your work or not, but spatialized audio.

BL: Where’s my Convolvotron

JFM: The Convolvotron was so far head of its time. You really could play sounds IN SPACE. The Rift uses ordinary stereo headsets, though I just heard yesterday that they might be adding much better audio.

But, we have neglected a number of things. Margaret did you use spatialized audio in your works?

MD: Yes I used spatialized audio quite a bit and developed audio sequencers for spatialized audio.

JFM: So right now you almost have to figure out how to do this yourself, or find a Convolvotron on eBay, which would still be hard to make work.

We have an impoverished way of doing navigation with our body, of doing interaction with our body, of doing the sounds. We have very little going on with smell for no good reason that I can see, because it is certainly possible to add smells to these experiences in fairly inexpensive ways.

BL: As long as you don’t have cut scenes because smells take a long time to dissipate.

JFM: Not if you do it right, it doesn’t.

MD: We did smells in Indianapolis with the CAVE with tubes to deliver the smells.

JFM: So I made a scent collar you wear around your neck, triggered by Bluetooth signals sent from the immersive VR, and it will send the right small to your nose. And because there is so little of it – it uses basically molecular drift – it goes away quickly and only you can smell it with none of that lingering stuff

So we can put this stuff in. What else does it still need?

AM: I can see my son at 8 years old. He says he needs optical fiber for the Kinect to play games with the friends with big projections. He expects more. He expects to be immersed. He plays games in a somewhat natural space, and the social aspect is important.

JFM: When we all have a CAVE then maybe we are there.

AM: Haptics is a serious problem, because you can have this chair with visualization, but as soon as you try to touch it, it disappears.

BL: There are some gloves that have haptic capabilities built in to them, but it is the lowest resolution of any of our bodily senses.

JFM: I have a solution for this I have been promoting for the last 13 years. Somebody once asked me how the Holodeck worked. I replied that is uses the nano-molecular display (NMD). This is where you have particles of matter that are very small but addressable and they can reconfigure into solid masses as they are rendered for you. We are just waiting for this NMD.

BL: It’s instant 3D printing. Audience: It’s the Grey goo. A non- disastrous grey goo.

JFM: OK then VR needs this Grey Goo. Let’s get a couple more audience contributions and then we will try to wrap up the ideas we have been discussing.

AM: It seems that with games there is a distinct lack of verbs in the content. There are a lot of adjectives.

BL: Oh that’s really interesting.

AM: Like you were saying with HMDs – we are all talking about displays and experiencing and exploring the world. And with games it is just shooting galleries. It’s just run and shoot. Just those verbs. We need more verbs to increase the amount of interaction and what you can do. We have a lot of adjectives for shooting. You can shoot in multiple different ways, but you can’t speak to anyone. There’s no negotiate button or debate gun.

JFM: You can’t leave them voice messages. Although some of the Indie games are starting on to be more expansive.

AM: Yes, I talked to Steve Gaynor at Fullbright who has been working on Gone Home, and they are all about exploring, and I know they have been talking to Oculus.

That’s been one of the primary aspects of VR, that we have something like a Holodeck. I thought of something like VR5 (TV series from the mid-1990s) It wasn’t about just being surrounded by something but about interacting – including that agency.

We never matched that agency the Holodeck led us to expect.

BC: The game thing is a critical issue. Because this technology and the games that are coming out are so tightly linked to it. One of the questions is how to you try to peel it away or how to dive into the maw (I want to get some advice on this later.) We can look at the range of stuff that is happening in the indie game space. Things like Kentucky Route Zero, Papers please, Journey, things that are doing very unusual kinds of experiences and activities – ways of interacting with game worlds or with other people.

JFM: So, do you think that as we get a bigger palette of ways of playing games and the type of games we can play beyond the vocabulary we have now – 20 ways to shoot, we might actually make that maw something new?

Ben: There is some creative foment happening in games and perhaps this is good opportunity. Looking back on the first wave of the work all of you did – all these seminal VR artworks, the way it’s always looked to me was taking this technology that is meant for something very, very different, and because there was no widespread agreement from the human race of what it had to be used for, it was an opportunity to get in the door and do the exact opposite. Maybe this is a similar chance with devices like the Oculus.

You have this very small early adopter community MD: Exactly, Right on.

You have a couple of people in the mainstream game industry who are very excited about it and are making a lot of hacks, while everyone else in the triple A game space is waiting to see if they can make money from it since it’s not even commercially released yet.

You could just make any weird thing and put it out there and have it be the top app for the Oculus. There is this space for this right now.

JFM: We have Palmer Luckey who came up with the Oculus Rift – he‘s 21 and just born when the first wave of VR happened. I am looking at the 8 year old kids of today playing Minecraft and wondering: “When you have some better, cheaper tools what are YOU going to do with this technology?”

And the other aspect we haven’t touched on here (but it is in the little paper that accompanies this panel in the proceedings) – one of the things that took over was social media and that whole social immersion which we couldn’t quite get to in a compelling way for a large number of people in VR, we took it out on social media. We embraced it and we became addicted to it. Is there going to be something where these things come back together – where we might have virtual worlds where we are in them with a lot of people and we can do all kinds of things there that are socially motivated but could be immersive in a spatial sense too

AM: We have it with the Google Glass. It’s there – to a certain extent.

JFM: To a certain extent, but it’s not widespread and ubiquitous. When I ask “Is it street ready?” I mean is it to the point where the street can find it’s own use for these things?

JFM: So we are going to close the panel now, and I just want to say, we need to watch these 10 year olds playing Minecraft. And when we reconvene this panel in 20 years, when those 10 year olds are 30, I can’t wait to see what they have done!

Thanks to all of you for your attention and for your contributions to the panel. And I especially want to thank Brenda and Margaret for their input and insights.