WEBVTT

1
00:00:00.819 --> 00:00:03.486
(melodic music)

2
00:00:10.911 --> 00:00:13.650
<v ->I'm John Philly, I'm an assistant professor in computer</v>

3
00:00:13.650 --> 00:00:18.485
science and member of the computer (mumbles) lab.

4
00:00:18.485 --> 00:00:21.057
It's my pleasure to introduce our distinguished

5
00:00:21.057 --> 00:00:22.886
speaker today, Doug Bowman.

6
00:00:22.886 --> 00:00:26.803
There's a little bit of room in the front here.

7
00:00:36.568 --> 00:00:40.485
For example, there's a chair by me that's open.

8
00:00:41.514 --> 00:00:44.129
Before we get started, I want to make a related announcement

9
00:00:44.129 --> 00:00:47.963
about the campus visualization partnership that started

10
00:00:47.963 --> 00:00:52.130
last year, the first conference is next week on Wednesday

11
00:00:53.180 --> 00:00:56.380
in Patterson Hall and this will be by the just hired

12
00:00:56.380 --> 00:00:59.735
information visualization professor Nikolas (mumbles)

13
00:00:59.735 --> 00:01:03.000
who's hired by the highest score.

14
00:01:03.000 --> 00:01:07.236
That's relevant because Doug does work in visualization,

15
00:01:07.236 --> 00:01:09.941
he's a professor of computer science at Virginia Tech

16
00:01:09.941 --> 00:01:12.075
where he directs a 3-D interaction group

17
00:01:12.075 --> 00:01:14.688
in the center for media and computer interaction.

18
00:01:14.688 --> 00:01:18.429
He does research on user interfaces, large, high-resolution

19
00:01:18.429 --> 00:01:21.857
displays and mixed reality systems, like virtual

20
00:01:21.857 --> 00:01:23.250
and augmented reality.

21
00:01:23.250 --> 00:01:26.095
He's very talented and talks about

22
00:01:26.095 --> 00:01:28.066
commodity devices, and consumer-related devices

23
00:01:28.066 --> 00:01:30.229
like Oculus Rift, we've started seeing this kind of

24
00:01:30.229 --> 00:01:34.396
renaissance around VR and AR and you saw that Samsung

25
00:01:35.680 --> 00:01:37.608
is coming out with a Gear VR system,

26
00:01:37.608 --> 00:01:39.396
and working with Oculus on that,

27
00:01:39.396 --> 00:01:41.352
so it's a very heavy kind of topic

28
00:01:41.352 --> 00:01:44.023
and a fun topic as well.

29
00:01:44.023 --> 00:01:45.826
Doug was named a distinguished scientist

30
00:01:45.826 --> 00:01:49.934
by ACN in 2010 and received in 2014 a high triple A

31
00:01:49.934 --> 00:01:52.029
visualization graphics special achievement award

32
00:01:52.029 --> 00:01:53.215
in virtual reality.

33
00:01:53.215 --> 00:01:56.048
So, with that, let's welcome Doug.

34
00:01:57.033 --> 00:01:59.450
(applauding)

35
00:02:01.429 --> 00:02:04.021
<v ->Thanks very much, thanks, John, for the introduction</v>

36
00:02:04.021 --> 00:02:06.648
and thanks to all of you and to the department

37
00:02:06.648 --> 00:02:07.991
for inviting me here.

38
00:02:07.991 --> 00:02:10.727
It's an honor to be here, talking with you.

39
00:02:10.727 --> 00:02:12.880
I apologize, I've got a cold today, so my voice

40
00:02:12.880 --> 00:02:15.861
may be weird or soft, I hope that those of you in the back

41
00:02:15.861 --> 00:02:18.773
there can hear, if I meet you later and I don't shake your

42
00:02:18.773 --> 00:02:21.995
hand, please forgive me for that.

43
00:02:21.995 --> 00:02:24.216
So, I understand that you're gonna be having

44
00:02:24.216 --> 00:02:27.305
actually a series of talks on virtual reality

45
00:02:27.305 --> 00:02:29.551
and augmented reality this year and it looks like

46
00:02:29.551 --> 00:02:31.938
there's some great folks coming to give their talks,

47
00:02:31.938 --> 00:02:36.527
so I wish I could come here and hear them talk and so on.

48
00:02:36.527 --> 00:02:39.569
But it looks like I'm the first one.

49
00:02:39.569 --> 00:02:41.555
So, I'll make sure that I do a thorough job

50
00:02:41.555 --> 00:02:44.651
of giving you a deep and scholarly treatment

51
00:02:44.651 --> 00:02:45.984
of these fields.

52
00:02:47.166 --> 00:02:50.857
So, with that in mind, I'm gonna start with Tony Stark.

53
00:02:50.857 --> 00:02:53.107
(laughing)

54
00:02:53.965 --> 00:02:58.143
So, you have to do that, Tony Stark is just cool, right?

55
00:02:58.143 --> 00:03:02.766
And being a cool guy, he has a really cool computer system

56
00:03:02.766 --> 00:03:06.103
and a really cool computer system has to have a really cool

57
00:03:06.103 --> 00:03:06.936
interface.

58
00:03:07.776 --> 00:03:12.131
So, he's got holographic projections, he's got the ability

59
00:03:12.131 --> 00:03:15.459
to manipulate these things in 3-D with his hands

60
00:03:15.459 --> 00:03:17.594
and his gestures, he's got this great

61
00:03:17.594 --> 00:03:19.347
artificial intelligence behind it all

62
00:03:19.347 --> 00:03:21.573
that kind of knows what he's thinking so that he doesn't

63
00:03:21.573 --> 00:03:25.636
even have to do thing manually himself.

64
00:03:25.636 --> 00:03:29.695
It's a really awesome in-vision sort of interface, right?

65
00:03:29.695 --> 00:03:33.862
We see this a lot in Hollywood, sci-fi movies and so on.

66
00:03:35.129 --> 00:03:38.125
Another example here is from Avatar, a few years back

67
00:03:38.125 --> 00:03:41.124
with the same sort of thing.

68
00:03:41.124 --> 00:03:44.497
It's 3-D, it's holographic, it's transparent,

69
00:03:44.497 --> 00:03:47.018
it's gesture-driven, you don't see a mouse or a keyboard

70
00:03:47.018 --> 00:03:50.158
anywhere and this is why a lot of folks envision

71
00:03:50.158 --> 00:03:53.857
kind of the future of interfaces to be.

72
00:03:53.857 --> 00:03:57.446
In particular, how to use VR and AR technologies

73
00:03:57.446 --> 00:04:00.696
for future sort of natural interaction.

74
00:04:01.584 --> 00:04:03.685
So, that's kind of the Hollywood vision,

75
00:04:03.685 --> 00:04:06.724
and with that said, there's a lot of excitement these days

76
00:04:06.724 --> 00:04:09.943
about new technologies that are coming out.

77
00:04:09.943 --> 00:04:13.000
So, Oculus is probably the one that most of us

78
00:04:13.000 --> 00:04:15.732
have heard the most about.

79
00:04:15.732 --> 00:04:19.282
So, this company, which nobody knew of

80
00:04:19.282 --> 00:04:20.899
a couple of years ago,

81
00:04:20.899 --> 00:04:23.498
just got bought by Facebook for two billion dollars,

82
00:04:23.498 --> 00:04:27.009
I think that was the figure, and they make this

83
00:04:27.009 --> 00:04:30.812
head-mounted display that looks like a large pair

84
00:04:30.812 --> 00:04:34.009
of ski googles and straps on your face,

85
00:04:34.009 --> 00:04:36.525
and it gives you a fairly wide field of view,

86
00:04:36.525 --> 00:04:39.455
a fairly high resolution view into a virtual world,

87
00:04:39.455 --> 00:04:41.959
it tracts your head movement, so that as you look around,

88
00:04:41.959 --> 00:04:44.364
you can see different parts to that virtual world.

89
00:04:44.364 --> 00:04:47.606
This, I guess is the artist rendition of what the final

90
00:04:47.606 --> 00:04:49.512
product might have looked like, but of course

91
00:04:49.512 --> 00:04:51.795
they do have actual products out there.

92
00:04:51.795 --> 00:04:54.285
I just received my developer kit too,

93
00:04:54.285 --> 00:04:56.618
my DK2 at my lab a week ago.

94
00:04:58.187 --> 00:05:01.410
And it's exciting to have that, it's a pretty interesting

95
00:05:01.410 --> 00:05:02.910
display certainly.

96
00:05:03.869 --> 00:05:06.748
And it's cool to see the sort of technology that we've

97
00:05:06.748 --> 00:05:09.493
been working with in my lab for 15 years or so

98
00:05:09.493 --> 00:05:12.982
finally get out there in the public eye and start getting

99
00:05:12.982 --> 00:05:16.033
used and to have developers really trying to build

100
00:05:16.033 --> 00:05:19.783
interesting content for this sort of display.

101
00:05:20.732 --> 00:05:24.899
Around this sort of new ecology of devices and displays

102
00:05:27.780 --> 00:05:31.057
are various input devices or peripherals that you can use

103
00:05:31.057 --> 00:05:33.242
with your VR headset.

104
00:05:33.242 --> 00:05:34.974
So, I don't know how many people have seen this,

105
00:05:34.974 --> 00:05:37.191
it hasn't got quite as much press as the Oculus,

106
00:05:37.191 --> 00:05:39.793
this is called the Virtuix Omni.

107
00:05:39.793 --> 00:05:43.082
So, the guy is wearing the Oculus as you can see here.

108
00:05:43.082 --> 00:05:47.244
This is a locomotion device to allow you to walk,

109
00:05:47.244 --> 00:05:50.681
run, jump, etc, through your virtual world

110
00:05:50.681 --> 00:05:52.723
by sliding your feet.

111
00:05:52.723 --> 00:05:55.419
So, you wear this special pair of shoes, makes you look

112
00:05:55.419 --> 00:05:59.777
really cool, and you slide your feet along this plastic,

113
00:05:59.777 --> 00:06:03.944
low-friction dish and you're held in place by some straps

114
00:06:05.490 --> 00:06:07.612
around your waist which I don't think the guy is wearing

115
00:06:07.612 --> 00:06:10.181
in this picture, but I think in the current version of this,

116
00:06:10.181 --> 00:06:12.970
it kind of holds you up so you can slide your feet.

117
00:06:12.970 --> 00:06:14.761
So, you can play your first-person shooter games

118
00:06:14.761 --> 00:06:17.935
by running through the world, by turning physically,

119
00:06:17.935 --> 00:06:21.344
by aiming directly with your gun prop and so on

120
00:06:21.344 --> 00:06:24.164
while you're wearing your head-mounted display.

121
00:06:24.164 --> 00:06:28.124
And the input device that I'm actually pretty excited about

122
00:06:28.124 --> 00:06:30.561
these days, which, many of you have probably heard of

123
00:06:30.561 --> 00:06:32.587
is the Leap motion device.

124
00:06:32.587 --> 00:06:35.702
This is another consumer-level device, it's quite cheap,

125
00:06:35.702 --> 00:06:38.774
I think it retails for 70-ish dollars, something

126
00:06:38.774 --> 00:06:42.071
like that, maybe something that's better than 80.

127
00:06:42.071 --> 00:06:45.216
But, this is a device that's basically a miniature

128
00:06:45.216 --> 00:06:48.339
Microsoft Kinect if you know what that is.

129
00:06:48.339 --> 00:06:52.139
It tracks your hands and fingers in 3-D space,

130
00:06:52.139 --> 00:06:55.704
and it can do a fairly precise job of knowing

131
00:06:55.704 --> 00:06:58.377
where your hands are, where your fingertips are

132
00:06:58.377 --> 00:07:00.785
and then reconstructing where all the joints

133
00:07:00.785 --> 00:07:04.051
of your hand must be, so that you can get this

134
00:07:04.051 --> 00:07:08.392
very expressive, rich sort of, hand-based input

135
00:07:08.392 --> 00:07:12.342
without wearing or holding any sort of device.

136
00:07:12.342 --> 00:07:14.713
So, this is a pretty unique development too.

137
00:07:14.713 --> 00:07:17.423
And I'll talk about some interfaces we've designed

138
00:07:17.423 --> 00:07:19.458
using this technology.

139
00:07:19.458 --> 00:07:22.613
So, it's an exciting time to be in the field of VR

140
00:07:22.613 --> 00:07:25.113
and AR and 3-D user interface.

141
00:07:25.990 --> 00:07:28.275
So, one thing I want to say, first of all,

142
00:07:28.275 --> 00:07:30.001
is really on a kind of a side note,

143
00:07:30.001 --> 00:07:32.336
but this is not new,

144
00:07:32.336 --> 00:07:36.503
this sort of excitement about VR and related technologies.

145
00:07:37.523 --> 00:07:42.193
On the other right here is (mumbles) wearing his first

146
00:07:42.193 --> 00:07:46.152
head-mounted display in about 1968 or so.

147
00:07:46.152 --> 00:07:48.749
This system was called the sort of Damocles,

148
00:07:48.749 --> 00:07:51.642
if you know the myth, you'll understand why,

149
00:07:51.642 --> 00:07:54.188
so he was connected to the ceiling in order to do

150
00:07:54.188 --> 00:07:56.456
mechanical tracking of the head.

151
00:07:56.456 --> 00:07:59.952
He had a CRT-based display, and he actually could do

152
00:07:59.952 --> 00:08:02.087
both virtual and augmented reality,

153
00:08:02.087 --> 00:08:04.435
so it had the capability of allowing him

154
00:08:04.435 --> 00:08:06.425
to see through into the real world and overlay

155
00:08:06.425 --> 00:08:08.689
graphics onto the real world.

156
00:08:08.689 --> 00:08:11.093
And these graphics weren't as good as ones that we can

157
00:08:11.093 --> 00:08:15.321
produce today, his tracking was quite good, actually,

158
00:08:15.321 --> 00:08:18.350
low latency, and fairly precise tracking,

159
00:08:18.350 --> 00:08:20.451
so we haven't kind of made as much progress

160
00:08:20.451 --> 00:08:22.078
in that direction.

161
00:08:22.078 --> 00:08:25.309
But even in the 60s, people were thinking about

162
00:08:25.309 --> 00:08:28.912
this sort of technology and what sort of applications

163
00:08:28.912 --> 00:08:30.860
it could enable.

164
00:08:30.860 --> 00:08:33.288
Fast-forward to the 80s, the other two picture here,

165
00:08:33.288 --> 00:08:37.940
so a company called VPL was created by Jaron Lanier,

166
00:08:37.940 --> 00:08:40.125
some of you might have heard that name.

167
00:08:40.125 --> 00:08:42.216
And they had head-mounted displays as well,

168
00:08:42.216 --> 00:08:45.799
they had head tracking and they had a glove

169
00:08:46.681 --> 00:08:49.635
that could track your hand, so similar to the Leap,

170
00:08:49.635 --> 00:08:53.501
except you had to wear this glove device,

171
00:08:53.501 --> 00:08:55.370
and this was on the cover of Scientific American

172
00:08:55.370 --> 00:08:58.537
in October of 1987, this glove device,

173
00:09:00.373 --> 00:09:02.709
and so the promise of virtual reality was touted

174
00:09:02.709 --> 00:09:06.200
as being not very far away in 1987.

175
00:09:06.200 --> 00:09:08.117
So, that was, let's do some math here, that was

176
00:09:08.117 --> 00:09:09.200
27 years ago.

177
00:09:10.144 --> 00:09:11.955
And now we're saying, now the virtual reality

178
00:09:11.955 --> 00:09:15.295
is not very far away, maybe this time it might be true.

179
00:09:15.295 --> 00:09:17.979
But this stuff is not new, we're kind of coming back

180
00:09:17.979 --> 00:09:21.978
to some places that we've gone before.

181
00:09:21.978 --> 00:09:23.811
Certainly though, our technologies these days

182
00:09:23.811 --> 00:09:26.846
are more robust and most importantly, they're cheaper,

183
00:09:26.846 --> 00:09:31.028
right, so many people can afford this sort of technology

184
00:09:31.028 --> 00:09:35.389
in their homes, in their offices, wherever they'd like

185
00:09:35.389 --> 00:09:38.266
to use it, and so that means that people will be developing

186
00:09:38.266 --> 00:09:42.215
content for this sort of thing and designing content for.

187
00:09:42.215 --> 00:09:45.430
So, why are VR and AR so compelling?

188
00:09:45.430 --> 00:09:48.878
These sorts of technologies give you an experience

189
00:09:48.878 --> 00:09:53.056
that's unlike any other experiences with computing systems.

190
00:09:53.056 --> 00:09:55.485
I remember the first time that I tried a head-mounted

191
00:09:55.485 --> 00:09:59.484
display, it was 1994, I was a brand new graduate student

192
00:09:59.484 --> 00:10:02.133
at Georgia Tech, at Larry Hodgins lab, and he said,

193
00:10:02.133 --> 00:10:04.350
okay, come and try our demo.

194
00:10:04.350 --> 00:10:07.353
And so, I put on this head-mounted display, and I was

195
00:10:07.353 --> 00:10:10.258
emersed into a virtual world that was a virtual

196
00:10:10.258 --> 00:10:13.843
glass elevator, that would go up and down inside the lobby

197
00:10:13.843 --> 00:10:15.287
of a virtual hotel.

198
00:10:15.287 --> 00:10:18.215
The graphics were terrible, there was probably 500

199
00:10:18.215 --> 00:10:21.517
polygons in the whole scene, there was no real lighting,

200
00:10:21.517 --> 00:10:23.693
the textures were bad, it didn't look anything

201
00:10:23.693 --> 00:10:25.659
like the games of today.

202
00:10:25.659 --> 00:10:27.778
But when I pressed the button, or actually I reached out

203
00:10:27.778 --> 00:10:30.712
and pressed the button on the virtual elevator

204
00:10:30.712 --> 00:10:33.411
and that thing started to go up virtually,

205
00:10:33.411 --> 00:10:36.954
I felt something in my knees, right?

206
00:10:36.954 --> 00:10:40.355
I don't feel that when I play a game on my desktop computer.

207
00:10:40.355 --> 00:10:43.604
There was this sense of embodiment in a virtual world

208
00:10:43.604 --> 00:10:46.445
that's a different experience than what you get with

209
00:10:46.445 --> 00:10:49.665
other sorts of computing systems.

210
00:10:49.665 --> 00:10:52.847
So, that's one reason why VR and AR are compelling

211
00:10:52.847 --> 00:10:55.009
and people use terms like, you know,

212
00:10:55.009 --> 00:10:56.540
it's just cool, right?

213
00:10:56.540 --> 00:10:58.350
It's expressive.

214
00:10:58.350 --> 00:11:01.824
So, if I've got the ability to use my bare hands and fingers

215
00:11:01.824 --> 00:11:06.297
to interact, that's richer in some way than using a mouse

216
00:11:06.297 --> 00:11:08.017
or a keyboard.

217
00:11:08.017 --> 00:11:11.100
It might feel magical, or futuristic.

218
00:11:12.222 --> 00:11:15.608
And one reason I think this is true is that we really kind

219
00:11:15.608 --> 00:11:19.380
of crave 3-D input and 3-D display because we live

220
00:11:19.380 --> 00:11:21.350
in a 3-D world, right?

221
00:11:21.350 --> 00:11:25.901
So, we are used to living in and interacting in

222
00:11:25.901 --> 00:11:28.842
this 3-dimensional space and when we can do that

223
00:11:28.842 --> 00:11:31.413
with a computer, it doesn't seem like a computer anymore,

224
00:11:31.413 --> 00:11:33.246
it seems like a world.

225
00:11:34.092 --> 00:11:35.766
It can also make me feel things, right,

226
00:11:35.766 --> 00:11:37.703
so it can make me feel like I'm powerful,

227
00:11:37.703 --> 00:11:41.378
or if I had a mastery or skill in some way.

228
00:11:41.378 --> 00:11:42.908
So, that's how Tony Stark looks, right?

229
00:11:42.908 --> 00:11:45.155
He looks like a virtuoso when he's interacting

230
00:11:45.155 --> 00:11:48.155
with his made-up computer interface.

231
00:11:49.285 --> 00:11:51.200
But the word that I hear most often about these

232
00:11:51.200 --> 00:11:53.759
technologies these days is natural.

233
00:11:53.759 --> 00:11:56.127
So, there is this whole field of natural user interfaces

234
00:11:56.127 --> 00:11:58.460
or NUIs that has cropped up.

235
00:11:59.299 --> 00:12:00.963
And so what I want to do in the talk today

236
00:12:00.963 --> 00:12:03.565
is really explore what we mean by natural

237
00:12:03.565 --> 00:12:06.648
and how close we are to being natural

238
00:12:07.499 --> 00:12:12.343
and whether we want to be natural in our VR interfaces.

239
00:12:12.343 --> 00:12:14.939
So, before I do that, just a little bit of background

240
00:12:14.939 --> 00:12:16.012
on VR and AR.

241
00:12:16.012 --> 00:12:18.897
So, if you were to ask me if I could give just a little

242
00:12:18.897 --> 00:12:23.064
context of what do we study in the fields of VR and AR,

243
00:12:24.160 --> 00:12:26.320
so the first thing is application areas

244
00:12:26.320 --> 00:12:28.820
and I've just listed a few here,

245
00:12:28.820 --> 00:12:31.627
the application areas are many, some of these,

246
00:12:31.627 --> 00:12:34.710
people are using VR and AR for today.

247
00:12:35.895 --> 00:12:38.598
Every day, so things like military training,

248
00:12:38.598 --> 00:12:41.711
entertainment certainly, scientific visualization,

249
00:12:41.711 --> 00:12:43.398
others are more of a promise,

250
00:12:43.398 --> 00:12:46.727
so education, for example, we'd like to have students

251
00:12:46.727 --> 00:12:50.577
in classes perhaps use virtual worlds to learn

252
00:12:50.577 --> 00:12:52.629
in an immersive sort of way.

253
00:12:52.629 --> 00:12:54.524
And I think Facebook is probably interested in the

254
00:12:54.524 --> 00:12:57.566
whole collaboration and communication application area

255
00:12:57.566 --> 00:12:59.786
where perhaps I could get together with my friends

256
00:12:59.786 --> 00:13:01.975
in a much more direct and compelling way

257
00:13:01.975 --> 00:13:04.725
than sharing posts on a web site.

258
00:13:05.750 --> 00:13:07.775
So, many application areas.

259
00:13:07.775 --> 00:13:10.424
Let me define my terms here, so I've been kind of throwing

260
00:13:10.424 --> 00:13:14.198
out these terms, willy-nilly so far.

261
00:13:14.198 --> 00:13:17.053
So, what's virtual reality, or VR,

262
00:13:17.053 --> 00:13:18.872
so this is the moment definition,

263
00:13:18.872 --> 00:13:23.005
you can take this from all the different definitions

264
00:13:23.005 --> 00:13:27.908
out there, I think of VR as 3-D environment first of all,

265
00:13:27.908 --> 00:13:30.317
so it needs to be 3-D, it's synthetic,

266
00:13:30.317 --> 00:13:33.415
so it's computer-generated, it's not real,

267
00:13:33.415 --> 00:13:36.096
and it's viewed interactively.

268
00:13:36.096 --> 00:13:38.650
So, the user has some control over the view

269
00:13:38.650 --> 00:13:41.418
and what happens in the world and that has to be

270
00:13:41.418 --> 00:13:44.497
rendered in real time in order for that to happen.

271
00:13:44.497 --> 00:13:47.245
So, that's a pretty broad definition,

272
00:13:47.245 --> 00:13:50.171
any 3-D computer game would fit into this definition,

273
00:13:50.171 --> 00:13:52.509
it doesn't require specialized display or tracking

274
00:13:52.509 --> 00:13:56.013
or whatever, so this is the baseline for VR.

275
00:13:56.013 --> 00:13:58.290
What we're gonna talk mostly about today is more immersive

276
00:13:58.290 --> 00:14:02.005
VR, ones that do use advanced displays or tracking systems

277
00:14:02.005 --> 00:14:03.005
or whatever.

278
00:14:04.280 --> 00:14:06.546
So, that's VR and then augmented reality,

279
00:14:06.546 --> 00:14:08.096
I did steal somebody else's definition here,

280
00:14:08.096 --> 00:14:12.088
so Ron Azuma in '97, in a pretty famous paper said

281
00:14:12.088 --> 00:14:15.820
that augmented reality had to also be a 3-D environment,

282
00:14:15.820 --> 00:14:18.488
it had to combine real and virtual imagery,

283
00:14:18.488 --> 00:14:21.960
so and that real and virtual imagery had to be registered,

284
00:14:21.960 --> 00:14:25.311
so, in other words, if I've got a virtual cup

285
00:14:25.311 --> 00:14:28.908
that's supposed to be sitting on top of that real table

286
00:14:28.908 --> 00:14:31.788
over there, when I move around, the cup stays on top

287
00:14:31.788 --> 00:14:35.288
of the table, so the worlds are registered

288
00:14:36.466 --> 00:14:38.163
and it's also interactive.

289
00:14:38.163 --> 00:14:40.157
So, that's AR.

290
00:14:40.157 --> 00:14:42.483
And then you might hear this term, mixed reality as well,

291
00:14:42.483 --> 00:14:44.658
mixed reality is just an umbrella term,

292
00:14:44.658 --> 00:14:46.917
it just refers to any combination of real and virtual

293
00:14:46.917 --> 00:14:50.834
imagery, so there's a framework for understanding

294
00:14:50.834 --> 00:14:54.024
the MR continuum, it was done by Paul Millgram

295
00:14:54.024 --> 00:14:58.422
and (mumbles) in '94 and they basically said

296
00:14:58.422 --> 00:15:00.402
that we can characterize any sort of mixed reality

297
00:15:00.402 --> 00:15:03.043
system by kind of the level of mixing of the two types

298
00:15:03.043 --> 00:15:04.725
of content.

299
00:15:04.725 --> 00:15:08.704
So, on the far left side, we've got pure reality,

300
00:15:08.704 --> 00:15:11.073
and that could just be my direct experience of the real

301
00:15:11.073 --> 00:15:14.085
world, like I'm directly experiencing you guys,

302
00:15:14.085 --> 00:15:16.749
or it could be something of a video conferencing system,

303
00:15:16.749 --> 00:15:19.335
where it's a computer system, but the imagery I'm looking at

304
00:15:19.335 --> 00:15:21.593
or the sensory data that is coming up

305
00:15:21.593 --> 00:15:23.510
is all real world data.

306
00:15:24.570 --> 00:15:27.303
And on the far right, we have pure virtuality

307
00:15:27.303 --> 00:15:29.768
where everything I'm looking at is computer-generated,

308
00:15:29.768 --> 00:15:31.600
everything is synthetic,

309
00:15:31.600 --> 00:15:34.036
and so this user is standing inside of a (mumbles),

310
00:15:34.036 --> 00:15:36.625
and experiencing mostly virtuality,

311
00:15:36.625 --> 00:15:38.749
although you could argue they can see their own body,

312
00:15:38.749 --> 00:15:41.738
so there is a little bit of real in there as well.

313
00:15:41.738 --> 00:15:44.372
And in between, you have these mixtures,

314
00:15:44.372 --> 00:15:47.680
so augmented reality is closer to this left-hand side,

315
00:15:47.680 --> 00:15:51.102
it's mostly the real world, but we augment it with some

316
00:15:51.102 --> 00:15:54.495
enhancements that are virtual, so we put labels

317
00:15:54.495 --> 00:15:56.253
on things that are virtual, or we add virtual

318
00:15:56.253 --> 00:15:59.364
objects to the real world or what have you,

319
00:15:59.364 --> 00:16:01.240
and the augmented virtuality would be the idea

320
00:16:01.240 --> 00:16:02.790
that I'm looking at a mostly virtual world,

321
00:16:02.790 --> 00:16:05.414
but I'm seeing some real-world imagery inside.

322
00:16:05.414 --> 00:16:07.580
So, these are students in a virtual classroom

323
00:16:07.580 --> 00:16:10.447
and they're seeing a video of a professor giving

324
00:16:10.447 --> 00:16:14.153
a lecture inside that virtual classroom.

325
00:16:14.153 --> 00:16:15.663
Okay, so those are some ways of kind of understanding

326
00:16:15.663 --> 00:16:19.033
the space of these technologies.

327
00:16:19.033 --> 00:16:23.389
So, in the end, VR and AR are just user interfaces, right,

328
00:16:23.389 --> 00:16:25.276
they're technologies that allow you to produce

329
00:16:25.276 --> 00:16:27.964
a user interface and so, we can also understand it

330
00:16:27.964 --> 00:16:32.005
using this kind of standard ACI loop, right?

331
00:16:32.005 --> 00:16:34.825
So, we have a user and we have a system

332
00:16:34.825 --> 00:16:36.773
and in-between the user and the system

333
00:16:36.773 --> 00:16:39.817
is the user interface and that user interface has hardware,

334
00:16:39.817 --> 00:16:42.137
that's the stuff below the line here,

335
00:16:42.137 --> 00:16:45.584
so that input device hardware and display devices,

336
00:16:45.584 --> 00:16:47.782
and it's got software for the user interface as well,

337
00:16:47.782 --> 00:16:50.406
so we interpret the data from those input devices

338
00:16:50.406 --> 00:16:54.073
and we take the world that needs to be shown

339
00:16:55.419 --> 00:16:57.529
to the user and we render it via software and put it

340
00:16:57.529 --> 00:16:58.564
on a display device.

341
00:16:58.564 --> 00:17:02.121
So, this is a standard computer interaction loop.

342
00:17:02.121 --> 00:17:04.294
I want to use this to kind of tell you

343
00:17:04.294 --> 00:17:06.627
a little bit about the types of research that goes on

344
00:17:06.627 --> 00:17:09.104
in VR and AR that basically corresponds to these

345
00:17:09.104 --> 00:17:12.360
different areas of the system.

346
00:17:12.360 --> 00:17:15.049
So, the first area would be, there is a top

347
00:17:15.049 --> 00:17:17.161
that just comprises of the computer system

348
00:17:17.161 --> 00:17:20.296
and the model with the data that is displayed,

349
00:17:20.296 --> 00:17:22.677
and so, some examples of the type of research that go on

350
00:17:22.677 --> 00:17:25.167
here in VR and AR, we might want to be able to produce

351
00:17:25.167 --> 00:17:28.639
more realistic virtual humans, for example, so that we

352
00:17:28.639 --> 00:17:31.544
could interact with virtual characters in the world.

353
00:17:31.544 --> 00:17:35.440
In order to get a very realistic model, we might want to

354
00:17:35.440 --> 00:17:37.119
capture the real world.

355
00:17:37.119 --> 00:17:39.415
So, do 3-D scanning for example of the real world

356
00:17:39.415 --> 00:17:43.771
in order to place that into a virtual world.

357
00:17:43.771 --> 00:17:45.983
(mumbles)

358
00:17:45.983 --> 00:17:48.049
So, all this stuff about the simulation, the model,

359
00:17:48.049 --> 00:17:52.324
the objects that are in the world fits up here

360
00:17:52.324 --> 00:17:53.791
in this piece.

361
00:17:53.791 --> 00:17:56.697
Second piece would be on the display side,

362
00:17:56.697 --> 00:17:59.282
so people researching computer graphics,

363
00:17:59.282 --> 00:18:02.247
like how do we do photorealistic lighting for example,

364
00:18:02.247 --> 00:18:05.382
or how do we do research on displays, like increasing

365
00:18:05.382 --> 00:18:08.112
the field of view or resolution of the display.

366
00:18:08.112 --> 00:18:11.349
Or how do we enable augmented reality by combining

367
00:18:11.349 --> 00:18:14.438
optical see-through displays, all these are examples

368
00:18:14.438 --> 00:18:18.678
of questions that might be asked on this side of the loop.

369
00:18:18.678 --> 00:18:21.439
On the other side of input, people are doing research

370
00:18:21.439 --> 00:18:23.860
on how do we design for example a bare-hand interaction

371
00:18:23.860 --> 00:18:27.162
technique, in fact, they'll make a motion device,

372
00:18:27.162 --> 00:18:29.542
then, what do I do with my hand in order to control

373
00:18:29.542 --> 00:18:31.415
something in the system, you might say,

374
00:18:31.415 --> 00:18:34.350
I'll attach an object to the hand and turn it around

375
00:18:34.350 --> 00:18:36.913
and look at it, so that's one interaction technique,

376
00:18:36.913 --> 00:18:38.649
but there's many others.

377
00:18:38.649 --> 00:18:40.533
How do we design those techniques to be effective,

378
00:18:40.533 --> 00:18:43.229
and I'll talk more about that later on in my talk.

379
00:18:43.229 --> 00:18:45.634
How do we do gesture recognition, how do we design

380
00:18:45.634 --> 00:18:49.695
locomotion devices, like Virtuix Omni that I talked about,

381
00:18:49.695 --> 00:18:52.250
those are the research questions on the input side.

382
00:18:52.250 --> 00:18:54.009
And then finally there's research questions that deal

383
00:18:54.009 --> 00:18:57.016
with the user, him or herself.

384
00:18:57.016 --> 00:19:00.157
So, a lot of what we do in my lab is we try to understand

385
00:19:00.157 --> 00:19:02.893
the effects of different system design choices.

386
00:19:02.893 --> 00:19:06.463
So, if I do increase the field of view of my display,

387
00:19:06.463 --> 00:19:09.256
what impact does that have on user performance

388
00:19:09.256 --> 00:19:11.970
or user satisfaction for example,

389
00:19:11.970 --> 00:19:14.575
we might also want to understand stuff just about human

390
00:19:14.575 --> 00:19:17.270
perception or cognition when they're using such systems,

391
00:19:17.270 --> 00:19:20.894
so a big topic in our field is depth perception.

392
00:19:20.894 --> 00:19:22.966
We're showing these 3-D environments, but we never

393
00:19:22.966 --> 00:19:26.753
fully replicate all the depth cues that humans use

394
00:19:26.753 --> 00:19:30.556
in order to understand depth in the real world

395
00:19:30.556 --> 00:19:32.823
and so, how does depth perception in VR differ

396
00:19:32.823 --> 00:19:35.802
from depth perception in the real world

397
00:19:35.802 --> 00:19:38.613
and how does that affect things like training.

398
00:19:38.613 --> 00:19:41.574
I'm training in a virtual environment and then I'm expected

399
00:19:41.574 --> 00:19:44.748
to go out and perform out in the real world,

400
00:19:44.748 --> 00:19:46.507
are those differences in depth perception gonna be

401
00:19:46.507 --> 00:19:50.161
important, I'll give an example of that research later.

402
00:19:50.161 --> 00:19:52.239
Okay, so I hope that gives you a flavor of the sorts

403
00:19:52.239 --> 00:19:55.462
of things that go on in this community.

404
00:19:55.462 --> 00:19:58.862
So, if you notice, a lot of these questions are

405
00:19:58.862 --> 00:20:02.857
about realism, they're about making the virtual world

406
00:20:02.857 --> 00:20:05.965
more realistic in some ways, so I want to increase

407
00:20:05.965 --> 00:20:10.134
the resolution of my display, or I want to allow natural

408
00:20:10.134 --> 00:20:13.326
gestures, so there's more realistic interaction,

409
00:20:13.326 --> 00:20:16.094
or I want to make virtual humans more realistic, right?

410
00:20:16.094 --> 00:20:19.179
So, a lot of the research topics in VR and AR

411
00:20:19.179 --> 00:20:21.882
are about increasing realism.

412
00:20:21.882 --> 00:20:24.609
So, we can also use this loop to think about different

413
00:20:24.609 --> 00:20:27.963
types of realism that there are.

414
00:20:27.963 --> 00:20:30.700
So, the first one is display realism,

415
00:20:30.700 --> 00:20:32.486
I'm gonna use the term fidelity.

416
00:20:32.486 --> 00:20:36.653
So, how close to a real-world view is the view that I get

417
00:20:37.832 --> 00:20:41.782
with my VR display, and so fidelity has levels, right?

418
00:20:41.782 --> 00:20:45.170
Here are two examples, a very low fidelity

419
00:20:45.170 --> 00:20:47.806
might be something like a laptop display, right?

420
00:20:47.806 --> 00:20:49.988
It's fairly low resolution, although they're pretty

421
00:20:49.988 --> 00:20:51.924
high-resolution these days, it's definitely

422
00:20:51.924 --> 00:20:55.080
low field of view, it doesn't change when I move my head,

423
00:20:55.080 --> 00:20:58.394
it's not stereoscopic, etc, on the high end of fidelity

424
00:20:58.394 --> 00:21:00.799
might be something like a Cave, where I've got

425
00:21:00.799 --> 00:21:04.012
high resolution, I've got head tracking, I've got stereo,

426
00:21:04.012 --> 00:21:07.398
the screens actually surround me, so that I can

427
00:21:07.398 --> 00:21:09.886
turn and look at different parts of the world.

428
00:21:09.886 --> 00:21:13.053
So, these are levels of display fidelity.

429
00:21:13.053 --> 00:21:15.386
On the other side, we've got levels of interaction fidelity,

430
00:21:15.386 --> 00:21:19.285
so this is how close my interface, my actions in the

431
00:21:19.285 --> 00:21:22.091
interface are the actions that I would do in the real world.

432
00:21:22.091 --> 00:21:25.299
So, if I want to move through a virtual scene,

433
00:21:25.299 --> 00:21:27.556
I could do it with arrow keys on my keyboard,

434
00:21:27.556 --> 00:21:29.782
that would mean very low fidelity, right,

435
00:21:29.782 --> 00:21:31.809
because that's nothing like what I do in the real world,

436
00:21:31.809 --> 00:21:35.749
or I can kind of walk through it, using a device like this,

437
00:21:35.749 --> 00:21:39.254
so this would be higher fidelity locomotion.

438
00:21:39.254 --> 00:21:42.630
Then, on the top, we have what we call scenario fidelity.

439
00:21:42.630 --> 00:21:44.672
So, this is everything to do with the simulation

440
00:21:44.672 --> 00:21:49.179
and the model and physics, etc, in your virtual world.

441
00:21:49.179 --> 00:21:51.660
So, here's just a simple example of one way of

442
00:21:51.660 --> 00:21:56.548
scenario fidelity changes in the detail in your model.

443
00:21:56.548 --> 00:22:00.129
So, on the left is a low fidelity version of actually

444
00:22:00.129 --> 00:22:03.401
a real-world scene, and on the right is a much higher

445
00:22:03.401 --> 00:22:04.469
fidelity version.

446
00:22:04.469 --> 00:22:07.777
Okay, so we can vary all these different types of fidelity

447
00:22:07.777 --> 00:22:12.510
and ask ourselves questions about how those changes

448
00:22:12.510 --> 00:22:15.849
affect the user of our VR system.

449
00:22:15.849 --> 00:22:17.494
So, I won't spend much time on this,

450
00:22:17.494 --> 00:22:19.347
but I did want to put the exact definitions

451
00:22:19.347 --> 00:22:22.046
that we use for these three terms in here.

452
00:22:22.046 --> 00:22:24.256
I'll go through one of them to get the idea.

453
00:22:24.256 --> 00:22:26.772
Display fidelity, which sometimes people call immersion,

454
00:22:26.772 --> 00:22:28.819
but I don't like that term anymore because too many

455
00:22:28.819 --> 00:22:30.770
people look at it in other ways, so I'm gonna call this

456
00:22:30.770 --> 00:22:32.855
display fidelity.

457
00:22:32.855 --> 00:22:35.422
We think of this as the objective degree of exactness

458
00:22:35.422 --> 00:22:38.269
with which real world sensory stimuli are reproduced

459
00:22:38.269 --> 00:22:40.330
by the display system.

460
00:22:40.330 --> 00:22:42.666
So, objective is important word here,

461
00:22:42.666 --> 00:22:44.904
this is, we're gonna look at the system

462
00:22:44.904 --> 00:22:48.375
that can tell you its level of display fidelity,

463
00:22:48.375 --> 00:22:50.962
it doesn't have anything to do with how the user perceives

464
00:22:50.962 --> 00:22:55.183
the realism of that display, it's based on the actual

465
00:22:55.183 --> 00:22:59.350
specs of that display, field of view, resolution, etc.

466
00:23:00.491 --> 00:23:03.992
And the gold standard is the real world, right,

467
00:23:03.992 --> 00:23:07.870
so the way that we sense things in the real world.

468
00:23:07.870 --> 00:23:10.037
(mumbles)

469
00:23:11.110 --> 00:23:12.443
<v Audience Member>Question.</v>

470
00:23:12.443 --> 00:23:13.431
<v ->Yes?</v>

471
00:23:13.431 --> 00:23:14.950
<v Audience Member>So, as far as virtual reality,</v>

472
00:23:14.950 --> 00:23:17.869
you can do for example virtual reality where you are

473
00:23:17.869 --> 00:23:21.336
pushed into a Bugs Bunny cartoon, and it's all

474
00:23:21.336 --> 00:23:24.611
clearly not real, but on the other hand,

475
00:23:24.611 --> 00:23:27.796
if you're accurately doing head-tracking, etc,

476
00:23:27.796 --> 00:23:32.020
you can still create an experience that is quite different

477
00:23:32.020 --> 00:23:35.700
that what you get from looking on something on the screen.

478
00:23:35.700 --> 00:23:37.172
<v ->Yeah, that's a good question.</v>

479
00:23:37.172 --> 00:23:38.761
So, I will talk about this definition then,

480
00:23:38.761 --> 00:23:41.777
our definition of scenario fidelity tries to capture that,

481
00:23:41.777 --> 00:23:44.591
so we say that scenario fidelity is the objective degree

482
00:23:44.591 --> 00:23:48.490
of exactness, but it's as compared to the real world

483
00:23:48.490 --> 00:23:50.079
intended experience.

484
00:23:50.079 --> 00:23:53.016
So, if my intended experience is a Bugs Bunny cartoon,

485
00:23:53.016 --> 00:23:56.105
then if it deals and looks just like a Bugs Bunny

486
00:23:56.105 --> 00:23:59.268
cartoon, then that would be a high scenario of fidelity,

487
00:23:59.268 --> 00:24:01.018
good question.

488
00:24:01.018 --> 00:24:03.805
Okay, so what sort of questions can we ask about

489
00:24:03.805 --> 00:24:07.972
VR fidelity, so many of you probably recognize this,

490
00:24:08.875 --> 00:24:12.155
this is the depiction of holodeck on Startrack

491
00:24:12.155 --> 00:24:14.898
The Next Generation, is that right?

492
00:24:14.898 --> 00:24:18.138
Is that the right one, I think it is,

493
00:24:18.138 --> 00:24:22.239
so the holodeck is this imagined virtual reality

494
00:24:22.239 --> 00:24:26.598
system that provides perfect fidelity, right?

495
00:24:26.598 --> 00:24:29.009
It's completely indistinguishable from the real world

496
00:24:29.009 --> 00:24:31.233
in all of these ways that I've just mentioned,

497
00:24:31.233 --> 00:24:34.272
display, interaction, scenario.

498
00:24:34.272 --> 00:24:38.394
So, this is a useful tool to think about research questions

499
00:24:38.394 --> 00:24:39.836
related to fidelity.

500
00:24:39.836 --> 00:24:41.970
So, the first question is, what level of fidelity

501
00:24:41.970 --> 00:24:44.086
can we achieve now, this is not a research question,

502
00:24:44.086 --> 00:24:47.137
it's just kind of an analysis of the field.

503
00:24:47.137 --> 00:24:50.952
In other words, how close are we to the holodeck right now?

504
00:24:50.952 --> 00:24:52.969
So, I'll talk about that and then I'll want to talk

505
00:24:52.969 --> 00:24:55.712
about a real research question, which is,

506
00:24:55.712 --> 00:24:58.423
is high-fidelity VR the right goal, in other words,

507
00:24:58.423 --> 00:25:02.062
do we want to be as realistic as possible all the time,

508
00:25:02.062 --> 00:25:06.229
do we want the holodeck, is that the ultimate goal?

509
00:25:08.247 --> 00:25:09.948
Okay, so let me address this first question,

510
00:25:09.948 --> 00:25:12.608
what level of fidelity can we currently achieve,

511
00:25:12.608 --> 00:25:15.546
and this is gonna be very kind of band-wavy,

512
00:25:15.546 --> 00:25:16.927
I don't have a lot of statistics,

513
00:25:16.927 --> 00:25:18.844
we haven't done lots of big data usages,

514
00:25:18.844 --> 00:25:23.586
this is our feeling on how VR is going right now.

515
00:25:23.586 --> 00:25:26.025
So, how are we doing in terms of visual display,

516
00:25:26.025 --> 00:25:28.328
I've got some examples up here, the Oculus Rift,

517
00:25:28.328 --> 00:25:31.843
we've already talked about, it's pretty good in a lot

518
00:25:31.843 --> 00:25:34.480
of ways, it's fairly lightweight which means I don't

519
00:25:34.480 --> 00:25:37.410
notice it very much, it's got a decent resolution,

520
00:25:37.410 --> 00:25:40.010
although that resolution is blown up so big

521
00:25:40.010 --> 00:25:42.605
that I can still see individual pixels,

522
00:25:42.605 --> 00:25:45.822
it's got a very wide field of view for a head-mounted

523
00:25:45.822 --> 00:25:48.055
display, which is a 100 degree field of view,

524
00:25:48.055 --> 00:25:50.010
whereas our human visual field is somewhere around

525
00:25:50.010 --> 00:25:53.160
180 degrees, so that's pretty good.

526
00:25:53.160 --> 00:25:56.709
And it's got pretty low latency and decent position

527
00:25:56.709 --> 00:26:01.251
head-tracking, so that's a pretty nice display.

528
00:26:01.251 --> 00:26:05.345
(mumbles), however, down here, on the bottom-right,

529
00:26:05.345 --> 00:26:08.637
something called the Dive, this is at Duke University,

530
00:26:08.637 --> 00:26:12.655
at North Carolina, and this is a six-sided display

531
00:26:12.655 --> 00:26:15.413
where all six sides of the room that you're standing in

532
00:26:15.413 --> 00:26:17.613
are projection surfaces, the entire resolution

533
00:26:17.613 --> 00:26:20.510
projectors, multiple projectors per wall,

534
00:26:20.510 --> 00:26:24.465
there's a high fidelity tracking system inside,

535
00:26:24.465 --> 00:26:26.599
it's stereoscopic, and you can physically walk

536
00:26:26.599 --> 00:26:28.430
around inside the space.

537
00:26:28.430 --> 00:26:30.422
So, this is a higher level of fidelity

538
00:26:30.422 --> 00:26:33.864
probably and then, we might not still be satisfied

539
00:26:33.864 --> 00:26:36.384
with the resolution of this thing, even with multiple

540
00:26:36.384 --> 00:26:40.304
projectors per wall, we still don't have enough pixels

541
00:26:40.304 --> 00:26:43.672
in some sense, and so the upper image is the image

542
00:26:43.672 --> 00:26:47.839
of the reality deck at Stony Brook up in Long Island,

543
00:26:48.880 --> 00:26:51.963
and this is a multi-gigapixel display

544
00:26:52.997 --> 00:26:56.544
made from tying LCD panels together

545
00:26:56.544 --> 00:26:58.368
and putting them in kind of a room configuration,

546
00:26:58.368 --> 00:27:01.124
this is much higher in terms of resolution fidelity,

547
00:27:01.124 --> 00:27:03.158
on the other hand, it's got some trade-offs,

548
00:27:03.158 --> 00:27:04.258
like it doesn't have a floor,

549
00:27:04.258 --> 00:27:05.499
it doesn't have a ceiling,

550
00:27:05.499 --> 00:27:07.599
it's got these seams between the displays,

551
00:27:07.599 --> 00:27:10.664
right, so it's not a continuous resolution.

552
00:27:10.664 --> 00:27:13.114
Overall I would say in terms of visual displays,

553
00:27:13.114 --> 00:27:16.469
VR is doing quite well in terms of fidelity,

554
00:27:16.469 --> 00:27:18.691
we're able to reproduce a lot of the stuff

555
00:27:18.691 --> 00:27:20.404
that we can see in the real world,

556
00:27:20.404 --> 00:27:24.257
but as I said, we can't always get all of the depth cues.

557
00:27:24.257 --> 00:27:26.840
What about non-visual displays?

558
00:27:27.729 --> 00:27:30.040
Remember that in VR we not only want to display

559
00:27:30.040 --> 00:27:33.333
to the visual sense, we want to display to the auditory

560
00:27:33.333 --> 00:27:37.363
sense, to the olfactory sense, the haptic sense,

561
00:27:37.363 --> 00:27:40.537
right so here are some examples of non-visual displays.

562
00:27:40.537 --> 00:27:43.358
The first one on the left is an auditory display,

563
00:27:43.358 --> 00:27:47.441
this is a 120-something speakers in a large space

564
00:27:48.288 --> 00:27:51.104
that can produce a 3-D sound field, there's also some

565
00:27:51.104 --> 00:27:55.271
ultrasonic beam speakers here which, technology I don't

566
00:27:56.449 --> 00:27:59.377
understand, that's the idea that you can only hear a sound

567
00:27:59.377 --> 00:28:01.555
when you're standing in a certain spot of the room

568
00:28:01.555 --> 00:28:04.904
because it beams the sound to that location.

569
00:28:04.904 --> 00:28:07.135
So, this is fairly high fidelity,

570
00:28:07.135 --> 00:28:09.988
on the other hand, we don't often see this sort of thing

571
00:28:09.988 --> 00:28:12.773
integrated with the ability of a visual display,

572
00:28:12.773 --> 00:28:15.102
so often people are either doing visual VR work,

573
00:28:15.102 --> 00:28:18.382
or they're doing auditory VR work and for some reason,

574
00:28:18.382 --> 00:28:19.380
sorry?

575
00:28:19.380 --> 00:28:23.090
(muffled speaking)

576
00:28:23.090 --> 00:28:24.757
Awesome, can't wait.

577
00:28:25.843 --> 00:28:29.230
Haptics, so right, this is a commercial haptic device

578
00:28:29.230 --> 00:28:31.297
or actually a pair of commercial haptic devices

579
00:28:31.297 --> 00:28:33.591
that you can use for gaming, it provides force feedback

580
00:28:33.591 --> 00:28:37.090
in a very limited workspace, the force feedback

581
00:28:37.090 --> 00:28:41.173
is okay, it can't provide all the ranges of force

582
00:28:42.206 --> 00:28:44.806
that we can experience in the real world, certainly.

583
00:28:44.806 --> 00:28:47.405
People have been working on haptics for a long time

584
00:28:47.405 --> 00:28:49.786
and there's some really nice success stories

585
00:28:49.786 --> 00:28:51.954
for specific types of haptic displays,

586
00:28:51.954 --> 00:28:54.597
but the general purpose haptic display

587
00:28:54.597 --> 00:28:56.952
that works anywhere in 3-D space, that can provide

588
00:28:56.952 --> 00:29:00.571
force and tactile and pressure and all the different

589
00:29:00.571 --> 00:29:03.538
haptic sensations doesn't exist and we're not really

590
00:29:03.538 --> 00:29:05.907
close to that, so we have a long way to go

591
00:29:05.907 --> 00:29:08.208
in terms of haptic fidelity.

592
00:29:08.208 --> 00:29:09.414
And my favorite one here,

593
00:29:09.414 --> 00:29:13.275
this is olfactory and gustatory fidelity,

594
00:29:13.275 --> 00:29:16.628
so this is a research group in Japan that has been

595
00:29:16.628 --> 00:29:19.795
looking at combing visuals with smell,

596
00:29:21.111 --> 00:29:24.763
they're actually doing a taste simulation here,

597
00:29:24.763 --> 00:29:27.105
but the idea is if we have good enough visuals

598
00:29:27.105 --> 00:29:29.512
and good enough smell, then we can make you taste

599
00:29:29.512 --> 00:29:32.228
something, we can fool you into tasting something,

600
00:29:32.228 --> 00:29:33.626
this experiment was great,

601
00:29:33.626 --> 00:29:36.963
they had these plain cookies, they were all the same,

602
00:29:36.963 --> 00:29:40.235
and they had these little AR markers on them,

603
00:29:40.235 --> 00:29:42.301
so you looked at the cookie, the system recognized

604
00:29:42.301 --> 00:29:44.377
which cookie it was, it contracted in space

605
00:29:44.377 --> 00:29:47.018
and they could overlay a texture on it,

606
00:29:47.018 --> 00:29:48.925
so you could make it into a strawberry cookie

607
00:29:48.925 --> 00:29:51.914
or chocolate cookie or what have you, visually,

608
00:29:51.914 --> 00:29:53.825
and then when you bite into the cookie,

609
00:29:53.825 --> 00:29:56.550
they pump a little smell into these tubes

610
00:29:56.550 --> 00:29:58.419
so that you get the right smell as well,

611
00:29:58.419 --> 00:30:01.323
and it turns out people think that they're eating

612
00:30:01.323 --> 00:30:04.457
different sorts of cookies which is pretty cool.

613
00:30:04.457 --> 00:30:07.673
However, this is very much a proof of concept,

614
00:30:07.673 --> 00:30:10.605
so a general purpose olfactory display certainly

615
00:30:10.605 --> 00:30:13.571
doesn't exist, there's a long way to go on that.

616
00:30:13.571 --> 00:30:16.523
All right, so that's kind of display fidelity,

617
00:30:16.523 --> 00:30:20.444
how about interaction fidelity, how are we doing on that?

618
00:30:20.444 --> 00:30:22.044
So, you can think about the different tasks

619
00:30:22.044 --> 00:30:24.996
that people might do in a virtual world,

620
00:30:24.996 --> 00:30:28.228
so the most basic task is just looking around, right?

621
00:30:28.228 --> 00:30:30.429
Seeing different parts of the virtual world,

622
00:30:30.429 --> 00:30:32.809
and the technology that that's based on of course

623
00:30:32.809 --> 00:30:37.024
is head tracking, and eye tracking would also be helpful,

624
00:30:37.024 --> 00:30:39.528
so head tracking is really good these days,

625
00:30:39.528 --> 00:30:41.992
head tracking is cheap, it's low latency,

626
00:30:41.992 --> 00:30:46.792
it's high precision, so getting the user's direction of gaze

627
00:30:46.792 --> 00:30:51.202
or their head's direction at least is really good.

628
00:30:51.202 --> 00:30:53.489
However, eye tracking still has some ways to go,

629
00:30:53.489 --> 00:30:56.100
eye tracking systems in general have come a long way

630
00:30:56.100 --> 00:30:58.481
from body movement head tracking and head-mounted displays

631
00:30:58.481 --> 00:31:01.529
and so on, but we've still got some work to do there.

632
00:31:01.529 --> 00:31:03.013
So, looking around in general,

633
00:31:03.013 --> 00:31:04.605
I think we're doing pretty well.

634
00:31:04.605 --> 00:31:06.088
Moving short distances is also good

635
00:31:06.088 --> 00:31:08.641
because that's just based on regular tracking technology

636
00:31:08.641 --> 00:31:11.272
as well, so if I can track the position of the user's head,

637
00:31:11.272 --> 00:31:13.905
I can get a different view of the world by making

638
00:31:13.905 --> 00:31:16.562
small movements like this, I don't have to cover

639
00:31:16.562 --> 00:31:18.727
the exact space to do that.

640
00:31:18.727 --> 00:31:20.994
What about walking long distances?

641
00:31:20.994 --> 00:31:22.847
How do we do that, well, you either need to track

642
00:31:22.847 --> 00:31:26.624
a really big space, right, or you need to have a locomotion

643
00:31:26.624 --> 00:31:30.219
device like Virtuix Omni that I showed before,

644
00:31:30.219 --> 00:31:32.636
or you need to do something in the middle.

645
00:31:32.636 --> 00:31:35.344
So, having a really big track space is just impractical

646
00:31:35.344 --> 00:31:37.347
for most applications, you can do it,

647
00:31:37.347 --> 00:31:41.347
but we're not really gonna do that anytime soon.

648
00:31:42.451 --> 00:31:44.466
Having some locomotion device like the Omni

649
00:31:44.466 --> 00:31:46.626
is not very high fidelity,

650
00:31:46.626 --> 00:31:48.861
so you actually walk very differently in a device

651
00:31:48.861 --> 00:31:51.792
like that, than you walk in the real world.

652
00:31:51.792 --> 00:31:54.210
So, to have the middle ground is this picture shown up here

653
00:31:54.210 --> 00:31:57.485
on the right which is something called redirected walking.

654
00:31:57.485 --> 00:32:00.436
And the idea here is that you've got a reasonably-sized

655
00:32:00.436 --> 00:32:02.120
track space, maybe the size of this room

656
00:32:02.120 --> 00:32:04.756
or a little bit smaller and you can make some

657
00:32:04.756 --> 00:32:07.585
manipulations to the user's view so that

658
00:32:07.585 --> 00:32:10.100
they stay within the room even though they think

659
00:32:10.100 --> 00:32:13.835
they can walk throughout a larger virtual environment,

660
00:32:13.835 --> 00:32:16.663
so getting close to the wall, and you do a slight turn

661
00:32:16.663 --> 00:32:19.565
in the virtual world and they just follow right along

662
00:32:19.565 --> 00:32:20.631
without noticing it.

663
00:32:20.631 --> 00:32:22.752
So, you can basically make people walk in circles

664
00:32:22.752 --> 00:32:24.814
when they think they're walking in a straight line,

665
00:32:24.814 --> 00:32:27.839
they have a much bigger virtual world than a physical world.

666
00:32:27.839 --> 00:32:29.684
So, that's pretty nice.

667
00:32:29.684 --> 00:32:32.050
Other tasks, picking up and manipulating objects,

668
00:32:32.050 --> 00:32:33.814
certainly we can track people's hands,

669
00:32:33.814 --> 00:32:36.061
we know when they're touching objects, we can attach

670
00:32:36.061 --> 00:32:39.589
objects to people's hands and have them manipulate them

671
00:32:39.589 --> 00:32:42.206
with certain degrees of freedom, that works pretty well.

672
00:32:42.206 --> 00:32:45.559
Finger tip type grasping, precision sorts of grasps,

673
00:32:45.559 --> 00:32:48.773
multiple hands passing back and forth, again this is

674
00:32:48.773 --> 00:32:51.935
still a research topic, we don't have robust systems

675
00:32:51.935 --> 00:32:55.382
that do that in a general way yet.

676
00:32:55.382 --> 00:32:57.230
And then final two things about giving commands

677
00:32:57.230 --> 00:32:58.858
and entering text and numbers,

678
00:32:58.858 --> 00:33:01.004
interestingly, I think our current user interfaces

679
00:33:01.004 --> 00:33:03.476
for those things are actually pretty good.

680
00:33:03.476 --> 00:33:06.078
Right, so the keyboard is probably one of the most

681
00:33:06.078 --> 00:33:09.699
expressive and high-performance ways to get text and numbers

682
00:33:09.699 --> 00:33:12.404
into a system, I doubt we're gonna come up with a better

683
00:33:12.404 --> 00:33:15.501
way to do that, it's more like what we could do in the real

684
00:33:15.501 --> 00:33:20.428
world, handwriting is definitely slower than the keyboard.

685
00:33:20.428 --> 00:33:23.395
So, interaction fidelity as a whole?

686
00:33:23.395 --> 00:33:25.810
Okay, we can replicate a lot of the interactions we do

687
00:33:25.810 --> 00:33:28.343
in the real world, a lot of the more subtle things

688
00:33:28.343 --> 00:33:30.093
still need some work.

689
00:33:31.653 --> 00:33:34.689
So, how are we doing overall in terms of level of fidelity

690
00:33:34.689 --> 00:33:35.913
in VR?

691
00:33:35.913 --> 00:33:39.719
So, Bret Brooks, one of the illuminaries in our field

692
00:33:39.719 --> 00:33:43.625
did a series of reports on kind of the state of VR

693
00:33:43.625 --> 00:33:44.720
back in the 90s.

694
00:33:44.720 --> 00:33:47.782
And in 1993, his kind of a conclusion was,

695
00:33:47.782 --> 00:33:49.728
VR almost works.

696
00:33:49.728 --> 00:33:53.895
That's a good thing, right, in 1999 he said VR barely works.

697
00:33:57.177 --> 00:34:02.171
So, there's still a lot of research challenges to address.

698
00:34:02.171 --> 00:34:06.455
In 2014, I'm gonna say that VR mostly works,

699
00:34:06.455 --> 00:34:09.817
but we still have all these limitations,

700
00:34:09.817 --> 00:34:12.115
so we can't replicate the real world,

701
00:34:12.115 --> 00:34:14.145
we can't do the holodeck certainly,

702
00:34:14.145 --> 00:34:16.243
we've got a long way to go in many of these areas,

703
00:34:16.243 --> 00:34:18.429
so since this is a university, I'm gonna give us

704
00:34:18.429 --> 00:34:21.843
a grade and I'm gonna say that we get a C plus.

705
00:34:21.843 --> 00:34:25.099
So, it's a passing grade, but you can improve,

706
00:34:25.099 --> 00:34:27.404
might need to work on this some more.

707
00:34:27.404 --> 00:34:30.084
All right, so that's how we're doing.

708
00:34:30.084 --> 00:34:33.746
The thing I'm gonna spend a little more time on is

709
00:34:33.746 --> 00:34:35.888
what we want to do.

710
00:34:35.888 --> 00:34:37.800
So, do we want the holodeck or not,

711
00:34:37.800 --> 00:34:40.537
is high-fidelity the right goal?

712
00:34:40.537 --> 00:34:42.622
So, there's a bunch of questions you can ask kind of

713
00:34:42.622 --> 00:34:44.644
under this heading.

714
00:34:44.644 --> 00:34:47.026
The first is, what are the effects of increasing realism

715
00:34:47.026 --> 00:34:50.706
and I'll show you some examples of studies around there.

716
00:34:50.706 --> 00:34:54.624
So, in terms of the experience of the user,

717
00:34:54.624 --> 00:34:57.461
how does higher fidelity make a difference,

718
00:34:57.461 --> 00:35:00.910
is a more natural user interface always desirable,

719
00:35:00.910 --> 00:35:03.223
I think you can probably guess from the question

720
00:35:03.223 --> 00:35:04.950
what the answer will be.

721
00:35:04.950 --> 00:35:07.213
(mumbles)

722
00:35:07.213 --> 00:35:10.735
And really something I want to focus on is

723
00:35:10.735 --> 00:35:14.106
if our goal is to make the interface indistinguishable

724
00:35:14.106 --> 00:35:16.368
from the real world, if our goal is to have a holodeck,

725
00:35:16.368 --> 00:35:19.384
then we really don't need UI design.

726
00:35:19.384 --> 00:35:23.632
So, if the template, the real world is already laid

727
00:35:23.632 --> 00:35:26.372
out for us and all we have to do is copy it,

728
00:35:26.372 --> 00:35:30.122
maybe UI designers would be out of their job.

729
00:35:31.174 --> 00:35:32.704
So, let me talk about this first question,

730
00:35:32.704 --> 00:35:35.369
so what are the effects of increasing realism?

731
00:35:35.369 --> 00:35:37.354
I'm just gonna give you a few examples of studies

732
00:35:37.354 --> 00:35:41.742
that we've run for over the years on this topic.

733
00:35:41.742 --> 00:35:44.285
One of the first things we looked at is gaming

734
00:35:44.285 --> 00:35:45.967
because it's fun.

735
00:35:45.967 --> 00:35:49.982
And so, we used the Dive display that I showed you before,

736
00:35:49.982 --> 00:35:53.611
with the six-sided projected display

737
00:35:53.611 --> 00:35:56.366
and we did a simple experiment on display fidelity

738
00:35:56.366 --> 00:35:58.376
and actually interaction fidelity as well,

739
00:35:58.376 --> 00:36:00.382
I'll tell you a little more about that later.

740
00:36:00.382 --> 00:36:04.136
So, we had people go through this kind of FPS game

741
00:36:04.136 --> 00:36:06.500
environment where they had to navigate,

742
00:36:06.500 --> 00:36:08.976
they had to aim at their enemies and shoot them,

743
00:36:08.976 --> 00:36:12.486
they had to hide from the enemies and win the game,

744
00:36:12.486 --> 00:36:14.664
basically the goal.

745
00:36:14.664 --> 00:36:17.186
And for display fidelity, we set two conditions,

746
00:36:17.186 --> 00:36:19.056
so we had a condition that was basically like a big

747
00:36:19.056 --> 00:36:22.717
desktop computer, we used one screen, one side of this

748
00:36:22.717 --> 00:36:26.550
display, so it's a really big desktop display,

749
00:36:27.696 --> 00:36:30.779
it was non-stereo and non-head track, it's just basically

750
00:36:30.779 --> 00:36:33.500
me standing in front of a big screen.

751
00:36:33.500 --> 00:36:37.134
We also used the complete Dive, so all six screens were on,

752
00:36:37.134 --> 00:36:40.482
they were stereoscopic and we had head tracking.

753
00:36:40.482 --> 00:36:42.692
And what we found was that people actually didn't

754
00:36:42.692 --> 00:36:45.945
do any better in the game or any worse in the game

755
00:36:45.945 --> 00:36:47.564
depending on which display they used,

756
00:36:47.564 --> 00:36:50.840
their performance was equivalent, based on display,

757
00:36:50.840 --> 00:36:52.546
but the high fidelity display did have

758
00:36:52.546 --> 00:36:54.425
a couple of advantages and they were more subjective

759
00:36:54.425 --> 00:36:56.963
advantages, so people found it more engaging

760
00:36:56.963 --> 00:36:59.830
and they found that they felt more present

761
00:36:59.830 --> 00:37:02.596
in the virtual world when they had a high fidelity display.

762
00:37:02.596 --> 00:37:04.455
That might be important for certain applications,

763
00:37:04.455 --> 00:37:07.078
for gaming, those are pretty important metrics,

764
00:37:07.078 --> 00:37:09.411
not just performance, right?

765
00:37:10.351 --> 00:37:13.307
What about a more serious application?

766
00:37:13.307 --> 00:37:15.824
So, scientific visualization,

767
00:37:15.824 --> 00:37:18.156
we did this study actually in a Cave,

768
00:37:18.156 --> 00:37:21.459
this is a four-sided display, not a six-sided display

769
00:37:21.459 --> 00:37:24.381
and we were looking at visualization of Cave system,

770
00:37:24.381 --> 00:37:28.013
so this is the caves in the cave study,

771
00:37:28.013 --> 00:37:30.212
this is the type of caves you crawl underground

772
00:37:30.212 --> 00:37:32.765
and you get muddy and so on.

773
00:37:32.765 --> 00:37:36.394
So, these visualizations show the layout structure

774
00:37:36.394 --> 00:37:38.835
of underground caves, and again we have these two

775
00:37:38.835 --> 00:37:41.537
conditions, one wall with no screen and no head

776
00:37:41.537 --> 00:37:44.531
tracking and all the walls with stereo and head tracking,

777
00:37:44.531 --> 00:37:48.837
we had people do tasks like estimate the distance from this

778
00:37:48.837 --> 00:37:51.804
point to this point or tell us if there's a connection

779
00:37:51.804 --> 00:37:54.879
between this passage and this passage.

780
00:37:54.879 --> 00:37:58.082
Or tell us what's the downward sloping angle

781
00:37:58.082 --> 00:38:01.492
of this foot passage, those sorts of tasks.

782
00:38:01.492 --> 00:38:03.268
What we found was that the high fidelity display

783
00:38:03.268 --> 00:38:05.876
here really did improve task performance, so people

784
00:38:05.876 --> 00:38:08.132
answered more quickly and more accurately,

785
00:38:08.132 --> 00:38:09.925
but not for all the tasks.

786
00:38:09.925 --> 00:38:13.619
It was only for the very spatially complex tasks.

787
00:38:13.619 --> 00:38:16.738
They did just fine on the simpler tasks with the lower

788
00:38:16.738 --> 00:38:18.687
fidelity display.

789
00:38:18.687 --> 00:38:22.510
So, we followed up on this study and we ran

790
00:38:22.510 --> 00:38:25.481
a second study that just looked at one task,

791
00:38:25.481 --> 00:38:28.545
it seemed to be a fairly spatially complex task,

792
00:38:28.545 --> 00:38:30.732
some of these cave structures are basically

793
00:38:30.732 --> 00:38:34.899
lattice like and there are levels, vertical connections

794
00:38:37.089 --> 00:38:39.795
between the levels and sometimes those vertical connections

795
00:38:39.795 --> 00:38:42.303
go all the way through and sometimes they don't,

796
00:38:42.303 --> 00:38:45.974
sometimes there's a gap where you can't basically

797
00:38:45.974 --> 00:38:47.950
complete a level, so here's an example of such a gap.

798
00:38:47.950 --> 00:38:50.127
There are small gaps that have to be in the right place

799
00:38:50.127 --> 00:38:54.505
in order to be able to see them and so here,

800
00:38:54.505 --> 00:38:56.388
we looked at this single task and we also looked

801
00:38:56.388 --> 00:38:59.065
at components of display fidelity,

802
00:38:59.065 --> 00:39:02.894
not just display fidelity as a holistic concept.

803
00:39:02.894 --> 00:39:05.030
So, we looked at a field of regard,

804
00:39:05.030 --> 00:39:09.272
that's how much of the space around you has graphics

805
00:39:09.272 --> 00:39:14.204
in it, in our case we had four screens, the complete cave

806
00:39:14.204 --> 00:39:15.975
versus one screen.

807
00:39:15.975 --> 00:39:18.759
And head tracking was on and stereo was on

808
00:39:18.759 --> 00:39:23.602
independently, and what we found was that field of regard

809
00:39:23.602 --> 00:39:27.057
and head tracking being at high levels reduced errors.

810
00:39:27.057 --> 00:39:30.293
Okay, so if you understood the whole space around you

811
00:39:30.293 --> 00:39:32.046
and where you might need to go,

812
00:39:32.046 --> 00:39:34.271
and you could do little movements like this

813
00:39:34.271 --> 00:39:36.525
to get your view just to the right place,

814
00:39:36.525 --> 00:39:38.937
then you could be more accurate in answering this question

815
00:39:38.937 --> 00:39:40.961
whether there is a gap or not.

816
00:39:40.961 --> 00:39:43.794
Stereo would reduce the task time,

817
00:39:44.653 --> 00:39:48.820
so stereo helped me to understand depth and understand

818
00:39:49.669 --> 00:39:52.757
where these gaps might be so that I could search for them

819
00:39:52.757 --> 00:39:54.295
more effectively.

820
00:39:54.295 --> 00:39:57.045
So, that's an interesting result.

821
00:39:58.972 --> 00:40:03.367
And the final for display fidelity is an example

822
00:40:03.367 --> 00:40:07.289
from training, so working with Offshore and Naval research

823
00:40:07.289 --> 00:40:10.212
on use of VR for military training,

824
00:40:10.212 --> 00:40:12.531
and one of the tasks that they're interested in

825
00:40:12.531 --> 00:40:14.141
is visual scanning.

826
00:40:14.141 --> 00:40:17.276
So, you've got some guys riding on the back of a humvee

827
00:40:17.276 --> 00:40:20.349
down the street somewhere on the other side of the world

828
00:40:20.349 --> 00:40:23.674
and they're looking carefully for threatening objects

829
00:40:23.674 --> 00:40:25.452
or people, okay?

830
00:40:25.452 --> 00:40:27.946
So, we replicated that in this study,

831
00:40:27.946 --> 00:40:30.310
and we tried to do is we tried to teach people

832
00:40:30.310 --> 00:40:34.477
how to scan the side of the street and to pick out

833
00:40:36.041 --> 00:40:40.982
these threatening objects and to mark them basically.

834
00:40:40.982 --> 00:40:42.482
And the thing I want to focus on here is that we

835
00:40:42.482 --> 00:40:44.869
varied the field of view of the training system, so you see

836
00:40:44.869 --> 00:40:46.690
these three different examples here,

837
00:40:46.690 --> 00:40:48.616
one with a very narrow field of view,

838
00:40:48.616 --> 00:40:50.861
medium field of view and a high field of view.

839
00:40:50.861 --> 00:40:52.966
You'll also notice some visual changes in the images,

840
00:40:52.966 --> 00:40:54.663
but just ignore those for now, that was another

841
00:40:54.663 --> 00:40:55.998
variable in our study.

842
00:40:55.998 --> 00:40:58.959
So, for here, we're just focusing on field of view,

843
00:40:58.959 --> 00:41:01.260
and what we found was that field of view affected training

844
00:41:01.260 --> 00:41:06.004
performance, so if I trained with the high field

845
00:41:06.004 --> 00:41:08.901
of view system, I found more targets during my training

846
00:41:08.901 --> 00:41:12.526
than if I trained with a low field of view system.

847
00:41:12.526 --> 00:41:15.917
However, when we then looked at training transfer,

848
00:41:15.917 --> 00:41:19.983
so we assessed people's performance in a realistic

849
00:41:19.983 --> 00:41:23.602
environment, in fact, that was this environment,

850
00:41:23.602 --> 00:41:25.931
the most realistic environment, we found that

851
00:41:25.931 --> 00:41:28.299
no matter which system you trained with,

852
00:41:28.299 --> 00:41:30.565
everybody did equally well.

853
00:41:30.565 --> 00:41:33.214
So, this is pretty interesting, so the level of display

854
00:41:33.214 --> 00:41:36.824
fidelity had some impact on performance, but it turns out

855
00:41:36.824 --> 00:41:39.679
that they got a much cheaper training system

856
00:41:39.679 --> 00:41:41.487
that was still training to do the task just as well

857
00:41:41.487 --> 00:41:43.971
in the real world and there's also speculation,

858
00:41:43.971 --> 00:41:47.009
I can show you the paper if you like about why that might be

859
00:41:47.009 --> 00:41:48.707
but this was an interesting result.

860
00:41:48.707 --> 00:41:51.492
So, high field of view, higher display fidelity,

861
00:41:51.492 --> 00:41:55.120
not always better and not always needed.

862
00:41:55.120 --> 00:41:56.603
Okay, so let me talk a little bit about studies

863
00:41:56.603 --> 00:41:58.443
of interaction fidelity.

864
00:41:58.443 --> 00:42:01.390
This was a manipulation study, so we had people

865
00:42:01.390 --> 00:42:05.039
grabbing these letter-shaped objects and trying

866
00:42:05.039 --> 00:42:07.267
to put them inside targets that were shaped like

867
00:42:07.267 --> 00:42:09.485
slightly bigger versions of those letters

868
00:42:09.485 --> 00:42:12.537
and we had two different conditions here.

869
00:42:12.537 --> 00:42:15.939
We had a (mumbles) technique where you just reached out

870
00:42:15.939 --> 00:42:18.442
and actually grabbed the object and moved it through space

871
00:42:18.442 --> 00:42:21.050
and placed it on target, and we had the desktop style

872
00:42:21.050 --> 00:42:23.179
technique where you might use a cab package,

873
00:42:23.179 --> 00:42:27.730
where you had widgets and a mouse to manipulate

874
00:42:27.730 --> 00:42:30.838
these targets and that's what we did here.

875
00:42:30.838 --> 00:42:33.256
And here the natural interaction was a clear winner,

876
00:42:33.256 --> 00:42:35.722
people were just much faster and more accurate with

877
00:42:35.722 --> 00:42:38.704
these integrated techniques than they were with the

878
00:42:38.704 --> 00:42:40.450
desktop style movements.

879
00:42:40.450 --> 00:42:42.740
On the other hand, interaction fidelity

880
00:42:42.740 --> 00:42:44.490
is not always better,

881
00:42:45.994 --> 00:42:49.071
so here's one that hopefully some of you can relate to,

882
00:42:49.071 --> 00:42:52.361
this is Mario Kart, we like games,

883
00:42:52.361 --> 00:42:56.494
so in Mario Kart, on the Wii, essentially

884
00:42:56.494 --> 00:42:58.482
there's several different control methods

885
00:42:58.482 --> 00:43:00.149
and one of them is more realistic

886
00:43:00.149 --> 00:43:02.875
with the wheel up here, so this is just a passive

887
00:43:02.875 --> 00:43:04.842
prop that you insert your Wii remote into

888
00:43:04.842 --> 00:43:06.495
and you steer by turning the wheel

889
00:43:06.495 --> 00:43:08.560
and one is more traditional,

890
00:43:08.560 --> 00:43:11.674
the less special game pad down here,

891
00:43:11.674 --> 00:43:14.623
and what we found is significantly faster times,

892
00:43:14.623 --> 00:43:17.256
in other words better performance on a time trial

893
00:43:17.256 --> 00:43:20.039
in this racing games, so without enemies

894
00:43:20.039 --> 00:43:22.164
or power ups or anything like that,

895
00:43:22.164 --> 00:43:24.107
better performance on a time trial with the less

896
00:43:24.107 --> 00:43:28.026
natural controller, okay, so this is interesting.

897
00:43:28.026 --> 00:43:30.428
You would think you could leverage those natural

898
00:43:30.428 --> 00:43:32.917
skills to drive more accurately,

899
00:43:32.917 --> 00:43:36.144
or more quickly with the natural controller,

900
00:43:36.144 --> 00:43:37.793
so we think the reason why this happened is

901
00:43:37.793 --> 00:43:39.520
that the natural controller is really not natural

902
00:43:39.520 --> 00:43:41.206
enough, right?

903
00:43:41.206 --> 00:43:43.325
So, this wheel is not attached to anything,

904
00:43:43.325 --> 00:43:45.103
it's kind of free floating in space,

905
00:43:45.103 --> 00:43:46.952
it doesn't give you any force feedback

906
00:43:46.952 --> 00:43:48.350
and it's got higher latency

907
00:43:48.350 --> 00:43:50.481
than the traditional controller,

908
00:43:50.481 --> 00:43:52.520
so it's higher interaction fidelity,

909
00:43:52.520 --> 00:43:56.687
but it's not higher performance for some of those reasons.

910
00:43:57.774 --> 00:44:00.667
<v Audience Member>Games aren't natural at all.</v>

911
00:44:00.667 --> 00:44:03.075
<v ->The game isn't natural, you mean the world?</v>

912
00:44:03.075 --> 00:44:08.038
That's true, that's true, so it's not a physics-accurate

913
00:44:08.038 --> 00:44:10.955
simulation of driving, that's true.

914
00:44:11.879 --> 00:44:14.262
I would claim that we would get probably the same

915
00:44:14.262 --> 00:44:16.326
result if we were using that Wii wheel

916
00:44:16.326 --> 00:44:20.715
with more accurate physics simulation racing games

917
00:44:20.715 --> 00:44:24.339
as well, I think it's just not high fidelity enough

918
00:44:24.339 --> 00:44:28.671
to allow you to use those skills from real life.

919
00:44:28.671 --> 00:44:31.477
So, that gives us mixed results,

920
00:44:31.477 --> 00:44:34.970
so back to this study of a first person shooter game,

921
00:44:34.970 --> 00:44:36.769
we not only varied the display fidelity, but we also

922
00:44:36.769 --> 00:44:39.639
varied interaction fidelity and we had one condition

923
00:44:39.639 --> 00:44:41.786
where people were using kind of the traditional

924
00:44:41.786 --> 00:44:44.381
mouse and keyboard controls, arrow keys, mouse to change

925
00:44:44.381 --> 00:44:47.870
your view, mouse button to fire and we had another

926
00:44:47.870 --> 00:44:50.978
division where people were using a gun prop, so they were

927
00:44:50.978 --> 00:44:53.830
aiming directly in 3-D world and pressing the trigger

928
00:44:53.830 --> 00:44:56.610
to fire and to navigate, they were using a technique

929
00:44:56.610 --> 00:44:59.541
which you can kind of see here which we call

930
00:44:59.541 --> 00:45:02.004
the human joystick, the idea is when you're standing

931
00:45:02.004 --> 00:45:04.215
on this little piece of carpet, you aren't moving

932
00:45:04.215 --> 00:45:05.688
in the virtual world

933
00:45:05.688 --> 00:45:07.477
and in order to move you must step off that carpet

934
00:45:07.477 --> 00:45:09.782
in a certain direction or certain distance

935
00:45:09.782 --> 00:45:11.789
and that would move you in that direction

936
00:45:11.789 --> 00:45:16.169
with velocity related to the distance from the center,

937
00:45:16.169 --> 00:45:18.511
so it's like a human joystick.

938
00:45:18.511 --> 00:45:21.477
So, that's more natural in some sense because you're using

939
00:45:21.477 --> 00:45:23.906
your legs, you're using your body and you're going

940
00:45:23.906 --> 00:45:26.449
actually physically in that direction.

941
00:45:26.449 --> 00:45:29.133
So, what we found was probably not surprising,

942
00:45:29.133 --> 00:45:30.899
the high fidelity interaction actually decreased

943
00:45:30.899 --> 00:45:33.607
performance per travel, per locomotion,

944
00:45:33.607 --> 00:45:36.563
this human joystick technique, although it's closer

945
00:45:36.563 --> 00:45:38.002
to what we do in the real world,

946
00:45:38.002 --> 00:45:40.219
it's not really close to what we do in the real world,

947
00:45:40.219 --> 00:45:42.287
right and so people are much better with arrow keys

948
00:45:42.287 --> 00:45:44.388
than they are with this.

949
00:45:44.388 --> 00:45:46.900
But high fidelity interaction increased performance

950
00:45:46.900 --> 00:45:48.478
for aiming because that was very natural,

951
00:45:48.478 --> 00:45:51.015
right, I really just saw the target,

952
00:45:51.015 --> 00:45:54.002
I swung and aimed right at him and that's faster

953
00:45:54.002 --> 00:45:58.169
than moving your mouse to put a crosshair over the target.

954
00:45:59.114 --> 00:46:02.324
So, to summarize this, higher display fidelity is often

955
00:46:02.324 --> 00:46:03.481
significantly better.

956
00:46:03.481 --> 00:46:05.624
We've got lots of results that show this.

957
00:46:05.624 --> 00:46:09.152
But, it's not always true, there are many conditions

958
00:46:09.152 --> 00:46:11.003
where we had lots of different results,

959
00:46:11.003 --> 00:46:13.606
so we couldn't find a difference between the two.

960
00:46:13.606 --> 00:46:16.370
Interestingly, we only had one negative result for

961
00:46:16.370 --> 00:46:19.454
display fidelity, so higher display fidelity in one of our

962
00:46:19.454 --> 00:46:22.684
studies led to more simulated sickness than lower

963
00:46:22.684 --> 00:46:24.038
display fidelity.

964
00:46:24.038 --> 00:46:28.549
So, having the stereo, having high quality graphics,

965
00:46:28.549 --> 00:46:31.915
having higher resolution actually made people more sick,

966
00:46:31.915 --> 00:46:35.082
so that's also related to interaction.

967
00:46:37.064 --> 00:46:39.920
On the interaction fidelity side, what we've shown

968
00:46:39.920 --> 00:46:42.564
is that higher interaction fidelity is better when

969
00:46:42.564 --> 00:46:46.875
that high interaction fidelity condition is very high,

970
00:46:46.875 --> 00:46:48.401
very high, when it's really close

971
00:46:48.401 --> 00:46:50.961
to the real world interaction and it's often more fun

972
00:46:50.961 --> 00:46:54.095
and more engaging and those are important metrics.

973
00:46:54.095 --> 00:46:56.047
But lower interaction fidelity is sometimes better

974
00:46:56.047 --> 00:46:58.973
than moderate fidelity and especially for travel,

975
00:46:58.973 --> 00:47:01.774
for moving around the virtual world.

976
00:47:01.774 --> 00:47:05.657
So, we have this hypothesis that there is this

977
00:47:05.657 --> 00:47:08.760
non-linear relationship between interaction fidelity

978
00:47:08.760 --> 00:47:13.198
and effectiveness, and that when we have very high

979
00:47:13.198 --> 00:47:15.834
interaction fidelity, we can get very high performance,

980
00:47:15.834 --> 00:47:17.450
very high satisfaction, etc,

981
00:47:17.450 --> 00:47:19.848
very close to the real world.

982
00:47:19.848 --> 00:47:22.157
When we have low interaction fidelity, but we design it well

983
00:47:22.157 --> 00:47:24.836
so that people can learn it and it's usable,

984
00:47:24.836 --> 00:47:26.748
and even though it's nothing like the real world,

985
00:47:26.748 --> 00:47:28.364
that also can be very effective, right?

986
00:47:28.364 --> 00:47:31.090
But when we have moderate interaction fidelity,

987
00:47:31.090 --> 00:47:32.056
what happens?

988
00:47:32.056 --> 00:47:34.928
People think, oh well, I can use this to kind of walk

989
00:47:34.928 --> 00:47:36.690
just like I can in the real world,

990
00:47:36.690 --> 00:47:39.008
on Virtuix Omni and just walk like in the real world,

991
00:47:39.008 --> 00:47:41.254
I can use all those existing skills and I don't have

992
00:47:41.254 --> 00:47:44.071
to learn any of them, but because it's different

993
00:47:44.071 --> 00:47:46.616
in some subtle ways than real walking,

994
00:47:46.616 --> 00:47:49.159
it ends up having much lower performance.

995
00:47:49.159 --> 00:47:51.048
And I'll show you another example of that,

996
00:47:51.048 --> 00:47:53.547
so we think that if you want to increase fidelity

997
00:47:53.547 --> 00:47:56.309
of interaction, you better make sure that you can

998
00:47:56.309 --> 00:47:58.921
increase it very close to the real world or you may

999
00:47:58.921 --> 00:48:01.421
actually get negative effects.

1000
00:48:03.140 --> 00:48:05.946
So, the current technology doesn't allow us

1001
00:48:05.946 --> 00:48:09.768
to get to the highest levels of fidelity all the time

1002
00:48:09.768 --> 00:48:12.757
for all these different components of fidelity,

1003
00:48:12.757 --> 00:48:14.483
but even if we could, let's speculate,

1004
00:48:14.483 --> 00:48:16.696
do we really want a Holodeck?

1005
00:48:16.696 --> 00:48:19.556
Well, the real world has a lot of limitations,

1006
00:48:19.556 --> 00:48:21.905
right, we are frustrated by real world limitations

1007
00:48:21.905 --> 00:48:24.919
every day, so some things are just hard to do.

1008
00:48:24.919 --> 00:48:28.119
If I want to travel from here to Los Angeles,

1009
00:48:28.119 --> 00:48:30.448
it's gonna take me a lot of time and a lot of effort

1010
00:48:30.448 --> 00:48:32.781
and it's not trivial, right?

1011
00:48:33.695 --> 00:48:36.160
I could create a magical interface that would allow me

1012
00:48:36.160 --> 00:48:38.979
to snap my fingers and in a virtual world

1013
00:48:38.979 --> 00:48:42.084
go from virtual DC to virtual Los Angeles

1014
00:48:42.084 --> 00:48:44.338
instantaneously, right?

1015
00:48:44.338 --> 00:48:45.878
Some things are impossible to do,

1016
00:48:45.878 --> 00:48:48.604
I cannot pick up the Moon in the real world,

1017
00:48:48.604 --> 00:48:51.569
but in a virtual world I could and some things

1018
00:48:51.569 --> 00:48:53.238
that we want to do just don't have a real world

1019
00:48:53.238 --> 00:48:54.990
analogue, right?

1020
00:48:54.990 --> 00:48:59.157
So making all interaction just mimicking natural interaction

1021
00:49:03.185 --> 00:49:06.741
is not gonna solve all our problems.

1022
00:49:06.741 --> 00:49:09.060
Does VR still need UI design?

1023
00:49:09.060 --> 00:49:11.479
What do you think my answer is gonna be?

1024
00:49:11.479 --> 00:49:12.801
<v Audience>Yes.</v>

1025
00:49:12.801 --> 00:49:13.634
<v ->No, yes!</v>

1026
00:49:15.455 --> 00:49:18.220
I think it does, phew, that was close.

1027
00:49:18.220 --> 00:49:20.062
Why do we still need UI design?

1028
00:49:20.062 --> 00:49:23.271
So, current technologies are limited as we saw

1029
00:49:23.271 --> 00:49:25.370
and they're gonna be for a long time,

1030
00:49:25.370 --> 00:49:27.656
there's still a lot of work before we get to

1031
00:49:27.656 --> 00:49:31.909
the highest possible fidelity for these technologies.

1032
00:49:31.909 --> 00:49:34.347
But even if we have those more realistic or natural

1033
00:49:34.347 --> 00:49:36.985
interactions are not always better

1034
00:49:36.985 --> 00:49:39.475
and the real world is not the ultimate interface.

1035
00:49:39.475 --> 00:49:41.354
So, in the time that I have left,

1036
00:49:41.354 --> 00:49:42.855
and I Want to leave time for questions,

1037
00:49:42.855 --> 00:49:45.048
but I want to give you one example

1038
00:49:45.048 --> 00:49:48.683
that brings a lot of these ideas together.

1039
00:49:48.683 --> 00:49:51.144
Talking about the Virtuix Omni locomotion device,

1040
00:49:51.144 --> 00:49:53.414
we have a similar thing called the Virtusphere,

1041
00:49:53.414 --> 00:49:55.410
I don't know if any of you have ever seen this,

1042
00:49:55.410 --> 00:49:57.970
but this is basically a human-sized hamster ball,

1043
00:49:57.970 --> 00:50:00.956
right, so you get inside and you walk

1044
00:50:00.956 --> 00:50:03.214
and it stays on rollers and you can move in any

1045
00:50:03.214 --> 00:50:05.850
direction, move the ball in any direction.

1046
00:50:05.850 --> 00:50:07.586
And it stays in the same place,

1047
00:50:07.586 --> 00:50:09.317
you wear a head-mounted display and that allows

1048
00:50:09.317 --> 00:50:12.603
you to walk through a virtual world infinitely, right?

1049
00:50:12.603 --> 00:50:16.364
So, this is a very interesting sort of locomotion

1050
00:50:16.364 --> 00:50:18.797
device that you might think allows people to walk

1051
00:50:18.797 --> 00:50:22.700
much more naturally than other interactions,

1052
00:50:22.700 --> 00:50:24.563
so we did a little experiment, we built

1053
00:50:24.563 --> 00:50:26.335
a course for people to run around,

1054
00:50:26.335 --> 00:50:29.421
this course had no walls, so you could fall off the side,

1055
00:50:29.421 --> 00:50:31.829
you had to shoot targets along the way,

1056
00:50:31.829 --> 00:50:33.697
just to give people something to do,

1057
00:50:33.697 --> 00:50:36.368
and when we first tested this,

1058
00:50:36.368 --> 00:50:37.885
we found that people were terrible

1059
00:50:37.885 --> 00:50:39.190
at the Virtusphere.

1060
00:50:39.190 --> 00:50:40.555
People were falling off all the time,

1061
00:50:40.555 --> 00:50:42.189
they can't go in the right direction,

1062
00:50:42.189 --> 00:50:45.191
they're not only virtually falling off the course,

1063
00:50:45.191 --> 00:50:48.388
but they're physically falling within the Virtusphere,

1064
00:50:48.388 --> 00:50:51.193
it's a very heavy device, it takes a lot of force

1065
00:50:51.193 --> 00:50:55.160
to get it going and a lot of force to stop it,

1066
00:50:55.160 --> 00:50:56.802
and when you try to walk just like you walk

1067
00:50:56.802 --> 00:51:00.526
in the real world, you fail miserably.

1068
00:51:00.526 --> 00:51:03.650
So, we thought okay, this is more interaction

1069
00:51:03.650 --> 00:51:05.997
fidelity, this is an example that we showed,

1070
00:51:05.997 --> 00:51:09.273
more interaction fidelity, but not enough,

1071
00:51:09.273 --> 00:51:12.568
so you get lower performance, but what can we do about

1072
00:51:12.568 --> 00:51:15.954
that, what can we do as designers to improve that situation

1073
00:51:15.954 --> 00:51:17.796
if we want to use this device?

1074
00:51:17.796 --> 00:51:21.132
So, very, very simple idea, this is like five

1075
00:51:21.132 --> 00:51:23.208
seconds of thought here,

1076
00:51:23.208 --> 00:51:26.791
let's teach people that they're not walking

1077
00:51:29.614 --> 00:51:31.124
in a virtual environment, but instead

1078
00:51:31.124 --> 00:51:32.886
that that they are rolling a ball

1079
00:51:32.886 --> 00:51:34.127
through a virtual environment.

1080
00:51:34.127 --> 00:51:36.267
So, if some of you ever played Super Monkey Ball,

1081
00:51:36.267 --> 00:51:38.687
just like that, let's make them think that they're

1082
00:51:38.687 --> 00:51:41.029
inside a ball and that their task is to roll

1083
00:51:41.029 --> 00:51:43.037
that ball through the environment,

1084
00:51:43.037 --> 00:51:46.653
so we give this very simple visualization of the ball,

1085
00:51:46.653 --> 00:51:49.890
it spins in the right direction and at the right speed

1086
00:51:49.890 --> 00:51:52.705
when the user walks in the Virtusphere and we train

1087
00:51:52.705 --> 00:51:54.026
people with that.

1088
00:51:54.026 --> 00:51:56.109
Some people trained with this and we had other people

1089
00:51:56.109 --> 00:52:00.109
that just trained with no visual cue whatsoever.

1090
00:52:01.012 --> 00:52:03.291
And here was the result, how many times did they

1091
00:52:03.291 --> 00:52:05.389
fall off the course in the virtual world.

1092
00:52:05.389 --> 00:52:08.739
We had four training sessions, and in the training

1093
00:52:08.739 --> 00:52:10.944
sessions, at least for the first three,

1094
00:52:10.944 --> 00:52:13.204
we see no difference, right, people were training

1095
00:52:13.204 --> 00:52:15.803
with visual cue, fell off just as many times as people

1096
00:52:15.803 --> 00:52:18.225
training with visual cue, and that wasn't very many,

1097
00:52:18.225 --> 00:52:20.081
so these were easy courses,

1098
00:52:20.081 --> 00:52:21.582
when we got to the fourth training course,

1099
00:52:21.582 --> 00:52:24.305
it got a lot harder, and now the people who are training

1100
00:52:24.305 --> 00:52:27.127
with the cue suddenly do a lot better than people

1101
00:52:27.127 --> 00:52:29.292
who are training without the cue.

1102
00:52:29.292 --> 00:52:31.772
And then we take the cue away for both groups,

1103
00:52:31.772 --> 00:52:35.556
right, and we just assess them as if neither of them

1104
00:52:35.556 --> 00:52:38.139
had a visual cue, and the people who trained

1105
00:52:38.139 --> 00:52:40.620
with the visual cue still do better.

1106
00:52:40.620 --> 00:52:42.992
Okay, so just by kind of rewiring their thinking about

1107
00:52:42.992 --> 00:52:45.479
what this sort of walking is like,

1108
00:52:45.479 --> 00:52:47.671
we can cause them to perform better.

1109
00:52:47.671 --> 00:52:49.589
Now, they're still not great, but it's still

1110
00:52:49.589 --> 00:52:51.339
a decent improvement.

1111
00:52:52.314 --> 00:52:56.346
Okay, so to conclude, natural interfaces are not

1112
00:52:56.346 --> 00:52:58.474
the answer to everything,

1113
00:52:58.474 --> 00:53:00.598
even natural Uis have to be designed.

1114
00:53:00.598 --> 00:53:04.765
So, UI design is a critical piece for VR and for AR.

1115
00:53:07.486 --> 00:53:09.382
And I want to conclude with this thought

1116
00:53:09.382 --> 00:53:12.101
that perhaps another definition of natural is more

1117
00:53:12.101 --> 00:53:14.953
appropriate, so we've been talking about natural

1118
00:53:14.953 --> 00:53:18.198
as if it means mimicking the real world, right?

1119
00:53:18.198 --> 00:53:20.139
And that's one definition of natural,

1120
00:53:20.139 --> 00:53:21.903
but I've been reading this book and I've been teaching

1121
00:53:21.903 --> 00:53:24.735
a class using this as a textbook, Brave NUI World

1122
00:53:24.735 --> 00:53:27.568
by Dennis Wixon and Daniel Wigdor,

1123
00:53:29.471 --> 00:53:32.032
and they think about natural user interfaces

1124
00:53:32.032 --> 00:53:33.954
in a different way, so here's their quite,

1125
00:53:33.954 --> 00:53:35.433
"Achieving the goal of creating a user interface that

1126
00:53:35.433 --> 00:53:37.357
"to its user feels natural",

1127
00:53:37.357 --> 00:53:40.589
so they're trying to feel natural, not be natural, right?

1128
00:53:40.589 --> 00:53:43.291
"Is not best achieved by mimicry of some other experience,

1129
00:53:43.291 --> 00:53:45.172
"by relying on familiar metaphors,

1130
00:53:45.172 --> 00:53:47.297
"or even by directly asking users what kind of experience

1131
00:53:47.297 --> 00:53:49.296
"they would like to have, we have to design,

1132
00:53:49.296 --> 00:53:53.126
"research and engineer these user interfaces."

1133
00:53:53.126 --> 00:53:55.252
They want to think about natural user interfaces

1134
00:53:55.252 --> 00:53:58.526
not as the user interface that is natural,

1135
00:53:58.526 --> 00:54:02.359
but as an interface that makes the user become

1136
00:54:05.170 --> 00:54:06.587
a natural, right?

1137
00:54:07.583 --> 00:54:11.118
So, you see an athlete or a musician who is highly

1138
00:54:11.118 --> 00:54:13.471
trained and highly skilled and they seem like

1139
00:54:13.471 --> 00:54:14.603
a natural, right?

1140
00:54:14.603 --> 00:54:17.211
They have full mastery over the situation and control

1141
00:54:17.211 --> 00:54:19.531
over everything that they're doing,

1142
00:54:19.531 --> 00:54:21.285
that's the type of interface that Wigdor and Wixon

1143
00:54:21.285 --> 00:54:24.806
argue that we should be designing.

1144
00:54:24.806 --> 00:54:28.054
So, I'm gonna finish up with one video if I still

1145
00:54:28.054 --> 00:54:29.637
have a minute here.

1146
00:54:31.589 --> 00:54:32.932
This is an interface that kind of illustrates

1147
00:54:32.932 --> 00:54:35.015
some of those principles,

1148
00:54:37.563 --> 00:54:38.696
and I won't show the whole video, but,

1149
00:54:38.696 --> 00:54:42.797
this was our entry to the 3-D UI contest this year,

1150
00:54:42.797 --> 00:54:46.964
(mumbles), and this year the task was to annotate

1151
00:54:48.590 --> 00:54:52.578
three points, so we have, this is a 3-D point

1152
00:54:52.578 --> 00:54:56.365
path of a beetle, and so we used the Leap motion device,

1153
00:54:56.365 --> 00:54:59.438
we wanted to do our natural user interface with this

1154
00:54:59.438 --> 00:55:02.454
and the techniques we came up with,

1155
00:55:02.454 --> 00:55:05.022
we called slice and swipe, so you're seeing what's

1156
00:55:05.022 --> 00:55:09.278
going on here, the metaphor is that of a chef's knife,

1157
00:55:09.278 --> 00:55:11.611
so instead of trying to exactly specify

1158
00:55:11.611 --> 00:55:14.934
each point in the set that we want to annotate,

1159
00:55:14.934 --> 00:55:18.574
instead we just do a series of kind of rough cuts,

1160
00:55:18.574 --> 00:55:22.894
so we cut through with the knife and then we swipe

1161
00:55:22.894 --> 00:55:25.097
away the parts that we don't want,

1162
00:55:25.097 --> 00:55:28.720
but that tool alone doesn't give us everything,

1163
00:55:28.720 --> 00:55:31.069
so we have a second tool, this is the bubble tool,

1164
00:55:31.069 --> 00:55:32.650
so using two fingers above the leaf,

1165
00:55:32.650 --> 00:55:35.233
we can control this bubble and we can control its size

1166
00:55:35.233 --> 00:55:38.601
and position and we brush the point cloud

1167
00:55:38.601 --> 00:55:41.289
and that gives us these two sets that we can then

1168
00:55:41.289 --> 00:55:43.306
choose one of them and finally, we have a lasso tool

1169
00:55:43.306 --> 00:55:46.925
which is even more precise, it works in screen space

1170
00:55:46.925 --> 00:55:51.612
instead of 3-D space, so you get the idea here.

1171
00:55:51.612 --> 00:55:54.349
It's not fully natural, it's not just like

1172
00:55:54.349 --> 00:55:55.692
what we would do in the real world,

1173
00:55:55.692 --> 00:55:57.441
in fact, we wouldn't do this in the real world,

1174
00:55:57.441 --> 00:56:01.608
so we have to design something that works for that.

1175
00:56:04.122 --> 00:56:06.862
So, the lessons from that were, using real world

1176
00:56:06.862 --> 00:56:09.040
metaphors, yes, like a chef's knife, but not mimicking

1177
00:56:09.040 --> 00:56:13.033
the real world, using expressive, but providing

1178
00:56:13.033 --> 00:56:15.823
multiple tools and using this idea of progressive

1179
00:56:15.823 --> 00:56:17.790
refinement, you don't have to get it precisely right

1180
00:56:17.790 --> 00:56:19.699
the first time, just do it a little bit at a time

1181
00:56:19.699 --> 00:56:22.577
and build up the final result.

1182
00:56:22.577 --> 00:56:24.233
Okay, so I'm gonna end there,

1183
00:56:24.233 --> 00:56:26.792
I'm looking forward to seeing the awesome VR and AR

1184
00:56:26.792 --> 00:56:29.441
interfaces that you guys design and others design

1185
00:56:29.441 --> 00:56:32.353
wit the new technology that's out there.

1186
00:56:32.353 --> 00:56:34.869
I want to thank all my students, and planning agencies

1187
00:56:34.869 --> 00:56:38.455
and colleagues and I'll be happy to take your questions.

1188
00:56:38.455 --> 00:56:40.872
(applauding)

1189
00:56:46.869 --> 00:56:50.656
<v Host>Okay, we have time for a few questions.</v>

1190
00:56:50.656 --> 00:56:52.469
<v Audience Member>I was a little curious,</v>

1191
00:56:52.469 --> 00:56:56.636
about 20 years ago when we were in the first application

1192
00:56:58.119 --> 00:57:02.266
of virtual reality, we found that people didn't like

1193
00:57:02.266 --> 00:57:06.433
to be inside this space for a non-trivial amount of time

1194
00:57:07.568 --> 00:57:09.818
(mumbles).

1195
00:57:14.824 --> 00:57:17.454
I don't think the difference is after so many years,

1196
00:57:17.454 --> 00:57:20.288
so how do you account for the fact that Oculus now

1197
00:57:20.288 --> 00:57:23.834
seems to be succeeding, is it just a new generation

1198
00:57:23.834 --> 00:57:27.108
of people who are 20 years younger?

1199
00:57:27.108 --> 00:57:28.969
<v ->I think that's part of it actually.</v>

1200
00:57:28.969 --> 00:57:32.004
The question was, you know, 20 years ago,

1201
00:57:32.004 --> 00:57:35.052
we found that people didn't like to be in head-mounted

1202
00:57:35.052 --> 00:57:37.089
displays for long periods of time,

1203
00:57:37.089 --> 00:57:39.947
and other sorts of displays were seen as preferable,

1204
00:57:39.947 --> 00:57:42.351
and has that changed now and why.

1205
00:57:42.351 --> 00:57:45.256
So, I think part of it is maybe generational,

1206
00:57:45.256 --> 00:57:49.561
so there are differences in the current digital natives

1207
00:57:49.561 --> 00:57:53.332
versus folks that didn't grow up with that,

1208
00:57:53.332 --> 00:57:55.996
but that's not certainly the whole thing.

1209
00:57:55.996 --> 00:57:59.398
So, wearing the current Oculus can still make you sick,

1210
00:57:59.398 --> 00:58:03.485
right, just like the older head-mounted displays

1211
00:58:03.485 --> 00:58:05.937
made you sick, they can still get some people

1212
00:58:05.937 --> 00:58:08.532
claustrophobia type of symptoms,

1213
00:58:08.532 --> 00:58:10.686
so I don't think that's the ultimate display

1214
00:58:10.686 --> 00:58:14.235
for everything and I imagine there is gonna be some

1215
00:58:14.235 --> 00:58:17.288
backlash in those directions when it becomes

1216
00:58:17.288 --> 00:58:19.818
a commercial product that everybody can buy,

1217
00:58:19.818 --> 00:58:22.807
that these limitations are gonna become more obvious.

1218
00:58:22.807 --> 00:58:25.340
So, I imagine we're still gonna see a range of different

1219
00:58:25.340 --> 00:58:27.241
types of displays.

1220
00:58:27.241 --> 00:58:30.309
The nice thing about the Oculus is that it's affordable

1221
00:58:30.309 --> 00:58:33.178
and portable among all of these things,

1222
00:58:33.178 --> 00:58:37.345
so I think it will be useful for a lot of applications.

1223
00:58:40.843 --> 00:58:44.103
<v Audience Member>So, you talked about applications</v>

1224
00:58:44.103 --> 00:58:46.366
in the space of gaming and military,

1225
00:58:46.366 --> 00:58:48.621
and you've drawn a lot of conclusions,

1226
00:58:48.621 --> 00:58:52.484
I'm wondering if you have any way to reflect

1227
00:58:52.484 --> 00:58:54.762
on the generality of what you've learned to see

1228
00:58:54.762 --> 00:58:57.464
if they would apply to other situations that might

1229
00:58:57.464 --> 00:59:01.978
have different technical and social requirements

1230
00:59:01.978 --> 00:59:06.558
and user interests, there's a lot of things that you said

1231
00:59:06.558 --> 00:59:09.976
that they might be constrained by these domains,

1232
00:59:09.976 --> 00:59:11.706
I wondered if you could clarify on that.

1233
00:59:11.706 --> 00:59:13.652
<v ->Yeah, I should've made that point,</v>

1234
00:59:13.652 --> 00:59:17.283
the question is, can we generalize these results

1235
00:59:17.283 --> 00:59:20.257
just based on a few studies in a few domains,

1236
00:59:20.257 --> 00:59:22.745
and I should've made that point,

1237
00:59:22.745 --> 00:59:25.278
we certainly have limitations in our knowledge,

1238
00:59:25.278 --> 00:59:29.281
but what we've tried to do in all our studies is say,

1239
00:59:29.281 --> 00:59:33.175
for these components of fidelity and for this task,

1240
00:59:33.175 --> 00:59:36.770
for this application area, this is the finding.

1241
00:59:36.770 --> 00:59:40.006
And we've got a pretty good breadth of findings

1242
00:59:40.006 --> 00:59:42.605
in different application areas, different tasks

1243
00:59:42.605 --> 00:59:45.238
at this point, we're actually building up an online

1244
00:59:45.238 --> 00:59:47.449
knowledge base of these findings where the task

1245
00:59:47.449 --> 00:59:49.283
and application area is specifically called out

1246
00:59:49.283 --> 00:59:52.616
as a parameter in each of these results.

1247
00:59:53.478 --> 00:59:55.600
So I think you have to keep that in mind,

1248
00:59:55.600 --> 00:59:58.221
the fidelity results may be tied to the task

1249
00:59:58.221 --> 01:00:01.534
in a lot of cases, so the results I had about

1250
01:00:01.534 --> 01:00:05.025
moderate interaction fidelity could be bad

1251
01:00:05.025 --> 01:00:08.198
mostly for these locomotion sorts of tasks

1252
01:00:08.198 --> 01:00:10.929
and not from other sorts,

1253
01:00:10.929 --> 01:00:14.346
so we definitely need to be careful about that.

1254
01:00:14.346 --> 01:00:15.925
<v Host>All right, let's thank Doug.</v>

1255
01:00:15.925 --> 01:00:18.425
(applauding)