WEBVTT

1
00:00:03.501 --> 00:00:05.918
(calm music)

2
00:00:10.842 --> 00:00:12.643
<v ->I think we'll get started.</v>

3
00:00:12.643 --> 00:00:14.917
So thanks to everyone for coming

4
00:00:14.917 --> 00:00:19.084
and thanks to Lillian for finding the time to come down.

5
00:00:20.038 --> 00:00:23.295
For those of you who are outside of the NLP field,

6
00:00:23.295 --> 00:00:26.218
Lillian has been sort of one of the powerhouses

7
00:00:26.218 --> 00:00:28.575
in the field for a while.

8
00:00:28.575 --> 00:00:30.697
I think one of the things

9
00:00:30.697 --> 00:00:33.704
that has always really impressed me about her work

10
00:00:33.704 --> 00:00:37.922
in general, is just that I pretty much know of

11
00:00:37.922 --> 00:00:41.819
like every student she's ever had and know what they've done

12
00:00:41.819 --> 00:00:44.456
and I think this is really rare

13
00:00:44.456 --> 00:00:47.292
and I think it's really sort of an amazing accomplishment

14
00:00:47.292 --> 00:00:51.508
that she's been able to have such a big impact on the field.

15
00:00:51.508 --> 00:00:55.879
Anyway, so you have a graph, what's she gonna tell us?

16
00:00:55.879 --> 00:00:58.327
And then I guess she liked the snow a lot

17
00:00:58.327 --> 00:01:00.475
because she went back to Cornell,

18
00:01:00.475 --> 00:01:02.910
where she's now a full professor.

19
00:01:02.910 --> 00:01:04.726
And today, she's going to be talking

20
00:01:04.726 --> 00:01:06.788
about language as influence.

21
00:01:06.788 --> 00:01:08.430
Thanks for coming.

22
00:01:08.430 --> 00:01:11.430
(audience applauds)

23
00:01:12.819 --> 00:01:15.874
<v ->Okay, I know it's a little hard to hear,</v>

24
00:01:15.874 --> 00:01:18.325
can you hear me in the back?

25
00:01:18.325 --> 00:01:20.588
(laughter)

26
00:01:20.588 --> 00:01:21.588
I'll try.

27
00:01:21.588 --> 00:01:23.528
Again, thanks for coming, thank you very much

28
00:01:23.528 --> 00:01:24.776
for the invitation.

29
00:01:24.776 --> 00:01:26.650
I'm really actually glad to be here

30
00:01:26.650 --> 00:01:28.913
with all your conversations today.

31
00:01:28.913 --> 00:01:30.311
For those of you

32
00:01:30.311 --> 00:01:32.424
who haven't had a chance to talk with me yet,

33
00:01:32.424 --> 00:01:35.490
I'm a very informal person, so I really encourage you

34
00:01:35.490 --> 00:01:39.735
to ask me questions, just let me finish my thought.

35
00:01:39.735 --> 00:01:43.291
So but no one usually has any questions now.

36
00:01:43.291 --> 00:01:46.232
And to warm things up, I have some questions for you,

37
00:01:46.232 --> 00:01:47.386
are you ready?

38
00:01:47.386 --> 00:01:48.792
<v Man>Yes.</v>

39
00:01:48.792 --> 00:01:51.496
<v ->All right, how many of you do AI for a living,</v>

40
00:01:51.496 --> 00:01:54.913
I just wanna get a feel for the audience.

41
00:01:57.215 --> 00:02:00.731
Yeah, okay, how many of you have seen Star Wars four,

42
00:02:00.731 --> 00:02:01.648
A New Hope.

43
00:02:03.321 --> 00:02:04.904
Not the same hands.

44
00:02:06.447 --> 00:02:08.447
<v Man>The first movie.</v>

45
00:02:10.573 --> 00:02:12.819
<v ->All right, and the final question,</v>

46
00:02:12.819 --> 00:02:16.092
how many of you are in professor Jordan Boyd Graber's class?

47
00:02:16.092 --> 00:02:18.342
(laughter)

48
00:02:24.805 --> 00:02:28.327
Okay, so the forms I'm gonna talk about today

49
00:02:28.327 --> 00:02:31.525
sort of in essence is what I did on my sabbatical

50
00:02:31.525 --> 00:02:35.089
and it includes the following observation, pretty clear.

51
00:02:35.089 --> 00:02:37.158
Now, there are lots of conversations available online

52
00:02:37.158 --> 00:02:38.306
these days.

53
00:02:38.306 --> 00:02:40.854
So they're sort of the things that are obviously

54
00:02:40.854 --> 00:02:43.145
holding conversations like Facebook and Twitter

55
00:02:43.145 --> 00:02:44.609
and these kind of things.

56
00:02:44.609 --> 00:02:45.931
Another key thing,

57
00:02:45.931 --> 00:02:48.267
those are obviously conversation oriented.

58
00:02:48.267 --> 00:02:51.662
So Reddit is supposedly sort of a content sharing site,

59
00:02:51.662 --> 00:02:54.355
but there's clearly a lot of conversations happening there.

60
00:02:54.355 --> 00:02:55.979
And then you have YouTube,

61
00:02:55.979 --> 00:02:58.462
ostensibly, a media sharing site,

62
00:02:58.462 --> 00:03:00.759
but then we also have conversations happening there,

63
00:03:00.759 --> 00:03:03.510
private conversions in the comment threads.

64
00:03:03.510 --> 00:03:04.991
And then you move even beyond that,

65
00:03:04.991 --> 00:03:06.789
you can think of things like review sites,

66
00:03:06.789 --> 00:03:08.335
Amazon or Yelp.

67
00:03:08.335 --> 00:03:10.971
And as the reviews actually often comment on each other

68
00:03:10.971 --> 00:03:12.693
or are talking about the same things,

69
00:03:12.693 --> 00:03:14.683
there's kind of latent conversations happening there

70
00:03:14.683 --> 00:03:15.712
as well.

71
00:03:15.712 --> 00:03:18.619
So we have all these sort of conversations happening online.

72
00:03:18.619 --> 00:03:21.353
And that means that we have many systems out there

73
00:03:21.353 --> 00:03:24.841
that are both humans and language, human language use

74
00:03:24.841 --> 00:03:26.363
as key components.

75
00:03:26.363 --> 00:03:30.319
And that means, ta-da, bold face, fantastic opportunities

76
00:03:30.319 --> 00:03:32.601
for the intersection of natural linear processing

77
00:03:32.601 --> 00:03:35.299
plus the social sciences if we got humans involved.

78
00:03:35.299 --> 00:03:38.195
That interaction could lead us to improve such systems

79
00:03:38.195 --> 00:03:40.271
and hopefully create new ones.

80
00:03:40.271 --> 00:03:42.002
So that's kind of the premise of the talk.

81
00:03:42.002 --> 00:03:44.569
The sort of way I'll try to convince you

82
00:03:44.569 --> 00:03:46.873
about these fantastic opportunities

83
00:03:46.873 --> 00:03:49.980
is by showing you two case studies I've been involved in.

84
00:03:49.980 --> 00:03:51.590
I'm gonna show you first, two kind of movie trailer

85
00:03:51.590 --> 00:03:52.744
versions of them.

86
00:03:52.744 --> 00:03:54.696
So I'll let you just pay attention

87
00:03:54.696 --> 00:03:56.894
to the introduction of the topic.

88
00:03:56.894 --> 00:03:58.896
So the first case study is motivated

89
00:03:58.896 --> 00:04:00.550
by the following question,

90
00:04:00.550 --> 00:04:02.284
I have a little picture here and I ask you

91
00:04:02.284 --> 00:04:03.787
who is in charge.

92
00:04:03.787 --> 00:04:05.870
What does this picture, this is an artist rendering

93
00:04:05.870 --> 00:04:07.620
of the supreme court.

94
00:04:08.595 --> 00:04:10.975
And if you know anything about this study,

95
00:04:10.975 --> 00:04:12.834
you should know who is in charge.

96
00:04:12.834 --> 00:04:15.934
Up here is Justice Scalia, he's up above the other person.

97
00:04:15.934 --> 00:04:17.785
He's sitting behind this bar that separates him

98
00:04:17.785 --> 00:04:20.546
from someone who is standing on lower ground.

99
00:04:20.546 --> 00:04:22.325
This guy is wearing fancy clothes,

100
00:04:22.325 --> 00:04:23.572
this guy is in regular clothes.

101
00:04:23.572 --> 00:04:25.525
A lot of visual stimulus makes it obvious

102
00:04:25.525 --> 00:04:27.666
who is in charge in that setting, right?

103
00:04:27.666 --> 00:04:29.925
But what if I didn't give you this visual scene,

104
00:04:29.925 --> 00:04:32.023
but if I only let you see what they said.

105
00:04:32.023 --> 00:04:33.424
And not even the speech signal,

106
00:04:33.424 --> 00:04:36.026
but I've written down transcripts of what they said.

107
00:04:36.026 --> 00:04:38.260
And what if I reduced that signal even further

108
00:04:38.260 --> 00:04:41.973
and say I'm also gonna hide away the content words as well.

109
00:04:41.973 --> 00:04:45.931
Could I, from these incredibly redacted transcripts

110
00:04:45.931 --> 00:04:48.068
from this very impoverished signal

111
00:04:48.068 --> 00:04:50.399
still be able to tell who is in charge?

112
00:04:50.399 --> 00:04:51.852
I think this is an interesting question

113
00:04:51.852 --> 00:04:53.420
because it tests scientifically

114
00:04:53.420 --> 00:04:56.255
how much of our social identities are we leaking

115
00:04:56.255 --> 00:04:57.915
in even these little teeny tiny

116
00:04:57.915 --> 00:05:00.412
sort of meaningless words.

117
00:05:00.412 --> 00:05:01.411
That's one of the questions

118
00:05:01.411 --> 00:05:04.507
that we're gonna address in this talk.

119
00:05:04.507 --> 00:05:06.919
Another reason, so why do we care about these

120
00:05:06.919 --> 00:05:08.340
sort of redacted transcripts?

121
00:05:08.340 --> 00:05:09.738
One, I just said it's interesting.

122
00:05:09.738 --> 00:05:11.659
I claim it is so and I'm the one who's up here,

123
00:05:11.659 --> 00:05:13.552
so I get to do that.

124
00:05:13.552 --> 00:05:15.897
A second reason is that if you think about privacy

125
00:05:15.897 --> 00:05:17.162
preserving data life,

126
00:05:17.162 --> 00:05:19.699
people may not want you to be mining

127
00:05:19.699 --> 00:05:22.147
their Facebook conversations or their email.

128
00:05:22.147 --> 00:05:24.394
But if we can just look at the functional roles

129
00:05:24.394 --> 00:05:25.859
without the content,

130
00:05:25.859 --> 00:05:27.540
maybe people would be more open to letting us

131
00:05:27.540 --> 00:05:31.265
do data mining on that kind of redacted transcript.

132
00:05:31.265 --> 00:05:33.454
And third, if you remove the content words

133
00:05:33.454 --> 00:05:35.878
and we're dealing with words that are debatably independent,

134
00:05:35.878 --> 00:05:37.944
we might hope to be able to transcribe

135
00:05:37.944 --> 00:05:39.085
different debates.

136
00:05:39.085 --> 00:05:40.632
So previously, I had a supreme court setting,

137
00:05:40.632 --> 00:05:43.802
what about if we have two people, these are Wikipedians,

138
00:05:43.802 --> 00:05:46.101
so that giant thing there is Wikipedia,

139
00:05:46.101 --> 00:05:47.165
that's the best way I could figure out

140
00:05:47.165 --> 00:05:49.199
how to present wikipedians.

141
00:05:49.199 --> 00:05:51.479
So if we have these--

142
00:05:51.479 --> 00:05:53.878
(sneeze)

143
00:05:53.878 --> 00:05:56.437
Defendant words, how can we tell who's in charge?

144
00:05:56.437 --> 00:05:59.027
Would it work on different people,

145
00:05:59.027 --> 00:06:01.907
such as Wikipedians, which are now getting an unlogged,

146
00:06:01.907 --> 00:06:04.371
asynchronous, non-Facebook based study.

147
00:06:04.371 --> 00:06:07.643
Okay, so that will be our first case study.

148
00:06:07.643 --> 00:06:11.969
Second case study, language and things you can learn

149
00:06:11.969 --> 00:06:15.239
from language and things that language does for us.

150
00:06:15.239 --> 00:06:17.590
It's a question that I've been thinking about

151
00:06:17.590 --> 00:06:19.346
for quite a while, which is whether phrasing

152
00:06:19.346 --> 00:06:21.591
affects the memorability of things.

153
00:06:21.591 --> 00:06:24.025
And you might say how could it not?

154
00:06:24.025 --> 00:06:27.429
Well, the other possibility is that lots of things,

155
00:06:27.429 --> 00:06:30.015
like context, affect the memorability of things.

156
00:06:30.015 --> 00:06:32.615
And the way we can actually think about this question,

157
00:06:32.615 --> 00:06:34.941
this just kind of stuck in our heads for a while.

158
00:06:34.941 --> 00:06:37.335
This is President Obama back in the rut

159
00:06:37.335 --> 00:06:39.697
of the 2008 elections, he said this.

160
00:06:39.697 --> 00:06:42.979
you can put lipstick on a pig, but it's still a pig.

161
00:06:42.979 --> 00:06:46.673
Many non-native speakers don't actually know what that means

162
00:06:46.673 --> 00:06:48.583
and I'm not gonna explain it to you here,

163
00:06:48.583 --> 00:06:49.580
but let's just say

164
00:06:49.580 --> 00:06:51.469
that that's a fairly striking thing to say.

165
00:06:51.469 --> 00:06:52.793
Lipstick on a pig.

166
00:06:52.793 --> 00:06:54.989
This phrase, believe it or not,

167
00:06:54.989 --> 00:06:57.193
got caught up by the popular press

168
00:06:57.193 --> 00:06:59.973
and discussed to quote a degree of detail,

169
00:06:59.973 --> 00:07:01.593
a lot of commenting about it.

170
00:07:01.593 --> 00:07:04.410
People were just talking about Obama said lipstick on a pig.

171
00:07:04.410 --> 00:07:06.843
So why did he do that?

172
00:07:06.843 --> 00:07:09.122
I, as a language person sort of thought

173
00:07:09.122 --> 00:07:11.790
you know, it's a really striking phrase,

174
00:07:11.790 --> 00:07:13.580
it's unusual, people don't say that kind of thing,

175
00:07:13.580 --> 00:07:15.980
it's very clunky and very evocative.

176
00:07:15.980 --> 00:07:17.512
Lipstick on a pig is a strange thing to say

177
00:07:17.512 --> 00:07:20.920
and it would be a strange thing to see, yes?

178
00:07:20.920 --> 00:07:24.084
On the other hand, so I always like to say

179
00:07:24.084 --> 00:07:26.279
this is my example of language having influence.

180
00:07:26.279 --> 00:07:30.503
I have this clunky phrase listening and discourse.

181
00:07:30.503 --> 00:07:32.653
When I tell my global scientist friends this,

182
00:07:32.653 --> 00:07:34.523
they'd be like no, you don't understand,

183
00:07:34.523 --> 00:07:35.890
there are other things going on.

184
00:07:35.890 --> 00:07:38.692
Earlier in the campaign, Sarah Palin had made a joke

185
00:07:38.692 --> 00:07:42.616
about what's the difference between a pitbull

186
00:07:42.616 --> 00:07:44.264
and a stick of lipstick.

187
00:07:44.264 --> 00:07:46.028
So Sarah Palin had made this remark

188
00:07:46.028 --> 00:07:48.046
about lipstick beforehand

189
00:07:48.046 --> 00:07:49.688
and people argued about Obama

190
00:07:49.688 --> 00:07:51.407
saying there's lipstick on a pig.

191
00:07:51.407 --> 00:07:53.320
The reason people were interested in talking about that

192
00:07:53.320 --> 00:07:55.169
was it seemed to be a reference

193
00:07:55.169 --> 00:07:56.714
to what Palin had said before.

194
00:07:56.714 --> 00:07:58.422
Hence, there wasn't anything interesting

195
00:07:58.422 --> 00:08:00.456
about the phrasing per se that causes a captive

196
00:08:00.456 --> 00:08:01.953
and national attention,

197
00:08:01.953 --> 00:08:03.894
it was the fact that it sort of intersected

198
00:08:03.894 --> 00:08:05.926
as a cultural coincidence, if you will.

199
00:08:05.926 --> 00:08:07.212
And that was what explained

200
00:08:07.212 --> 00:08:08.678
how this thing became well known.

201
00:08:08.678 --> 00:08:09.769
Let me argue with that.

202
00:08:09.769 --> 00:08:12.815
I can't go back in time, I wish I could,

203
00:08:12.815 --> 00:08:15.765
go back In history, make Palin not have said that

204
00:08:15.765 --> 00:08:17.393
and then run the experiment, right?

205
00:08:17.393 --> 00:08:19.844
And say okay, remove the cultural coincidence.

206
00:08:19.844 --> 00:08:23.315
So how do you, so even though I set up a lead

207
00:08:23.315 --> 00:08:25.471
there's something about language effect here,

208
00:08:25.471 --> 00:08:26.673
how could I prove that,

209
00:08:26.673 --> 00:08:28.086
how could I show you the language effect

210
00:08:28.086 --> 00:08:31.281
and somehow eliminate this sort of a cultural context?

211
00:08:31.281 --> 00:08:32.851
And that will be the question I consider

212
00:08:32.851 --> 00:08:35.065
in this presentation.

213
00:08:35.065 --> 00:08:37.179
All right, so two case studies sort of thinking about

214
00:08:37.179 --> 00:08:41.453
language and phrasing and the break down of power.

215
00:08:41.453 --> 00:08:43.979
Speaking of sort of power and stuff,

216
00:08:43.979 --> 00:08:45.598
there's something my advisor said,

217
00:08:45.598 --> 00:08:47.991
and I've said this to some people here before.

218
00:08:47.991 --> 00:08:49.893
My advisor said the following,

219
00:08:49.893 --> 00:08:52.339
which has had a lot of power and influence over me

220
00:08:52.339 --> 00:08:54.173
over many years.

221
00:08:54.173 --> 00:08:56.371
As I get older and older, I sort of see the wisdom

222
00:08:56.371 --> 00:08:58.006
of his words.

223
00:08:58.006 --> 00:09:00.188
This is sort of a paraphrase on what Stuart Shieber said

224
00:09:00.188 --> 00:09:03.188
because Stuart, is this being taped?

225
00:09:05.636 --> 00:09:07.708
He's extremely eloquent.

226
00:09:07.708 --> 00:09:10.250
His speech is Kind of manageable,

227
00:09:10.250 --> 00:09:12.215
but it kind of goes off on its own.

228
00:09:12.215 --> 00:09:15.017
So I had to paraphrase it.

229
00:09:15.017 --> 00:09:16.190
He said the following.

230
00:09:16.190 --> 00:09:18.555
Your goal, when you're presenting your work in a paper,

231
00:09:18.555 --> 00:09:22.585
is not to convince your audience that you are brilliant,

232
00:09:22.585 --> 00:09:26.476
but to convince your audience that your solution is trivial.

233
00:09:26.476 --> 00:09:28.334
That may sound very weird to you,

234
00:09:28.334 --> 00:09:29.948
well, many of you are graduate students

235
00:09:29.948 --> 00:09:31.346
and you're like you have to be kidding,

236
00:09:31.346 --> 00:09:32.707
the whole reason I'm in grad school

237
00:09:32.707 --> 00:09:34.015
is because how smart I am.

238
00:09:34.015 --> 00:09:37.091
But and indeed, it takes, as Stuart says,

239
00:09:37.091 --> 00:09:38.582
a certain strength of character,

240
00:09:38.582 --> 00:09:40.546
an achievement of one's goal.

241
00:09:40.546 --> 00:09:42.568
This is a sad fact, if you write your papers

242
00:09:42.568 --> 00:09:44.282
to make your solutions seem obvious,

243
00:09:44.282 --> 00:09:46.143
they will be rejected more often.

244
00:09:46.143 --> 00:09:48.987
If you tell people your solutions, such as a symbol,

245
00:09:48.987 --> 00:09:50.635
they're like you're not gonna do great,

246
00:09:50.635 --> 00:09:52.349
you didn't try, like, the smart thing.

247
00:09:52.349 --> 00:09:54.681
So why don't we take this advice apart.

248
00:09:54.681 --> 00:09:56.994
Well, you think about the goal.

249
00:09:56.994 --> 00:10:00.398
If your goal is not to convince people you're bright

250
00:10:00.398 --> 00:10:01.568
and brilliant and all that.

251
00:10:01.568 --> 00:10:03.709
And I've given up on that goal, I have to say,

252
00:10:03.709 --> 00:10:05.060
along with my other goals.

253
00:10:05.060 --> 00:10:07.181
Like, your goal is doing advanced science

254
00:10:07.181 --> 00:10:09.127
and to change the state of knowledge about the world

255
00:10:09.127 --> 00:10:11.365
and this thing of what people believe to be true,

256
00:10:11.365 --> 00:10:13.495
then you'll notice that if people think

257
00:10:13.495 --> 00:10:15.858
your findings are obvious, they must also believe

258
00:10:15.858 --> 00:10:18.023
that you are correct.

259
00:10:18.023 --> 00:10:21.052
And so if nothing else, if you go through this talk

260
00:10:21.052 --> 00:10:23.281
and you're like wow, she just took the dumbest approach,

261
00:10:23.281 --> 00:10:25.188
that may or may not be true,

262
00:10:25.188 --> 00:10:27.821
but know that I'm doing it on purpose.

263
00:10:27.821 --> 00:10:29.079
All right.

264
00:10:29.079 --> 00:10:31.425
So first part of the talk is study one,

265
00:10:31.425 --> 00:10:33.852
this is on joint work, my students,

266
00:10:33.852 --> 00:10:36.343
Cristian Danescu-Niculescu-Mizil.

267
00:10:36.343 --> 00:10:38.298
I can now say that.

268
00:10:38.298 --> 00:10:42.831
Former student Bo Pang, and Jon Kleinberg, 2012.

269
00:10:42.831 --> 00:10:46.498
And again, try to remember the little cartoon picture here,

270
00:10:46.498 --> 00:10:48.303
can we tell who is In charge

271
00:10:48.303 --> 00:10:51.957
even from extremely redacted transcripts?

272
00:10:51.957 --> 00:10:54.884
Okay, so why would we wanna do this kind of thing,

273
00:10:54.884 --> 00:10:56.371
I said before, it's just kind of,

274
00:10:56.371 --> 00:10:58.176
I think inherently interesting

275
00:10:58.176 --> 00:11:00.509
to understand how this information we're leaking

276
00:11:00.509 --> 00:11:04.558
about our sole positions in even like the trivial words

277
00:11:04.558 --> 00:11:05.892
that we say.

278
00:11:05.892 --> 00:11:08.078
But you talk about social applications,

279
00:11:08.078 --> 00:11:09.309
I think there are quite a few

280
00:11:09.309 --> 00:11:11.395
that I'm kind of looking forward to.

281
00:11:11.395 --> 00:11:15.332
I'm currently interested in proactive tasked

282
00:11:15.332 --> 00:11:17.843
discussion management system.

283
00:11:17.843 --> 00:11:20.311
So I've told some people this,

284
00:11:20.311 --> 00:11:22.093
I think about these kind of things a lot.

285
00:11:22.093 --> 00:11:25.029
These discussions are kind of meandering,

286
00:11:25.029 --> 00:11:26.647
I don't know if this happens here,

287
00:11:26.647 --> 00:11:28.371
but sometimes discussions get out of hand

288
00:11:28.371 --> 00:11:30.015
and it's not clear who is in charge

289
00:11:30.015 --> 00:11:32.077
or too many people seem to be In charge

290
00:11:32.077 --> 00:11:33.984
or no one seems to be in charge.

291
00:11:33.984 --> 00:11:37.456
The conversation has gotten so off track at that point.

292
00:11:37.456 --> 00:11:40.844
I mean great, in sort of these online settings,

293
00:11:40.844 --> 00:11:42.887
the assistants are able to correct for this

294
00:11:42.887 --> 00:11:44.778
because we got a solid thing saying

295
00:11:44.778 --> 00:11:46.526
you need to straighten it out.

296
00:11:46.526 --> 00:11:49.464
Sort of moving in that direction,

297
00:11:49.464 --> 00:11:51.732
a little farther in field, though,

298
00:11:51.732 --> 00:11:52.904
I'm interested in having

299
00:11:52.904 --> 00:11:55.224
conversational facilitation systems.

300
00:11:55.224 --> 00:11:56.641
You hear people say things like

301
00:11:56.641 --> 00:11:59.310
conversations online are getting more and more polarized,

302
00:11:59.310 --> 00:12:01.485
people just aren't liking each other anymore,

303
00:12:01.485 --> 00:12:04.141
the discourse isn't really working.

304
00:12:04.141 --> 00:12:06.773
And I think it'd be interesting if we had systems that,

305
00:12:06.773 --> 00:12:09.177
and I don't know totally how to do this,

306
00:12:09.177 --> 00:12:11.427
but systems that could somehow make conversations

307
00:12:11.427 --> 00:12:14.019
more productive, I think would be awesome.

308
00:12:14.019 --> 00:12:16.883
So what I did here, if you just defuse imminent flame wars,

309
00:12:16.883 --> 00:12:19.944
the sort of not inevitable ones,

310
00:12:19.944 --> 00:12:23.411
some of them are just gonna happen, you know at some point.

311
00:12:23.411 --> 00:12:24.876
And I've been thinking about this,

312
00:12:24.876 --> 00:12:26.180
if we can even just have a system

313
00:12:26.180 --> 00:12:27.809
where it can tell there's some imminent,

314
00:12:27.809 --> 00:12:29.724
some kind of tension in the room,

315
00:12:29.724 --> 00:12:31.286
and I don't know if this would work or not,

316
00:12:31.286 --> 00:12:33.220
but if you just slow down your typing rate,

317
00:12:33.220 --> 00:12:35.009
I was thinking maybe that could actually

318
00:12:35.009 --> 00:12:36.480
cause people to calm down

319
00:12:36.480 --> 00:12:38.449
because they just couldn't say things as fast.

320
00:12:38.449 --> 00:12:40.274
You know, just theoretically.

321
00:12:40.274 --> 00:12:41.947
I don't know exactly how we can make interventions,

322
00:12:41.947 --> 00:12:43.872
BHCI people would know much better than I.

323
00:12:43.872 --> 00:12:45.483
But the idea of being able to sense

324
00:12:45.483 --> 00:12:46.976
that things are happening beforehand

325
00:12:46.976 --> 00:12:48.089
and then do the intervention,

326
00:12:48.089 --> 00:12:50.715
whatever that should be, I think that would be really cool.

327
00:12:50.715 --> 00:12:52.599
MOOCs are big nowadays.

328
00:12:52.599 --> 00:12:54.848
So this kind of idea that people

329
00:12:54.848 --> 00:12:57.173
can be taking courses online

330
00:12:57.173 --> 00:12:59.217
and be very proud when you don't know your instructor

331
00:12:59.217 --> 00:13:01.027
and you don't even see an instructor at all.

332
00:13:01.027 --> 00:13:03.916
If you're gonna be any sort of distributed settings,

333
00:13:03.916 --> 00:13:06.562
I think it'd be really cool if even in those settings,

334
00:13:06.562 --> 00:13:09.221
you could kind of tell how engaged people were

335
00:13:09.221 --> 00:13:10.882
by the conversations that we're having.

336
00:13:10.882 --> 00:13:13.668
Same with the TA or whatever,

337
00:13:13.668 --> 00:13:15.249
I think that'd be a cool application.

338
00:13:15.249 --> 00:13:17.078
And let me skip the other things.

339
00:13:17.078 --> 00:13:19.008
I think there are some many conversations happening online

340
00:13:19.008 --> 00:13:22.669
and if we used them to make us better, it would be cool.

341
00:13:22.669 --> 00:13:24.868
And this idea of being able to tell who is in charge

342
00:13:24.868 --> 00:13:27.722
or if no one's in charge, I think is a step forward

343
00:13:27.722 --> 00:13:29.981
to these kinds of things.

344
00:13:29.981 --> 00:13:33.242
So the question is can I solve the language

345
00:13:33.242 --> 00:13:34.431
of who is in charge?

346
00:13:34.431 --> 00:13:36.610
Let me make sure we understand.

347
00:13:36.610 --> 00:13:39.714
So suppose I see these two people talking

348
00:13:39.714 --> 00:13:41.790
and this person says your honor, I agree,

349
00:13:41.790 --> 00:13:43.320
and this one says thank you.

350
00:13:43.320 --> 00:13:45.598
In this case, it is very obvious who is in charge.

351
00:13:45.598 --> 00:13:46.431
Why?

352
00:13:49.265 --> 00:13:52.652
Yep, this is the honor sitting here.

353
00:13:52.652 --> 00:13:55.235
Right, so sometimes, there are things like honor,

354
00:13:55.235 --> 00:13:57.395
extremely obvious cues as to who is In charge.

355
00:13:57.395 --> 00:13:59.809
I'm presenting this in a color coding scheme

356
00:13:59.809 --> 00:14:00.985
I'm gonna be using later.

357
00:14:00.985 --> 00:14:03.691
So this purple means I'm talking to someone

358
00:14:03.691 --> 00:14:06.170
who is in charge, the royal purple,

359
00:14:06.170 --> 00:14:08.485
I'm talking to somehow who on higher stance.

360
00:14:08.485 --> 00:14:12.446
Okay, this is someone talking on a lower status,

361
00:14:12.446 --> 00:14:16.443
you're talking to a green one, or someone who is green.

362
00:14:16.443 --> 00:14:18.560
But try to remember that, purple means you're talking

363
00:14:18.560 --> 00:14:22.143
upward and green means you're talking down.

364
00:14:23.736 --> 00:14:24.966
So if this is the case,

365
00:14:24.966 --> 00:14:26.651
in something like the judicial setting,

366
00:14:26.651 --> 00:14:29.108
very obvious via the cues as to who is in power.

367
00:14:29.108 --> 00:14:30.722
But that's obviously not as interesting.

368
00:14:30.722 --> 00:14:32.968
Other cases that have been considered,

369
00:14:32.968 --> 00:14:36.248
including the five people who work here.

370
00:14:36.248 --> 00:14:39.915
This is in the MROM setting, so this is corporate email.

371
00:14:39.915 --> 00:14:43.022
Here we have someone saying I'd love to get your thoughts

372
00:14:43.022 --> 00:14:44.749
on this when you are available.

373
00:14:44.749 --> 00:14:46.634
This person says let's discuss later.

374
00:14:46.634 --> 00:14:48.482
Who is in charge, you can tell from my color coding,

375
00:14:48.482 --> 00:14:51.144
but also things like the person who is in charge

376
00:14:51.144 --> 00:14:53.246
gets to say let's discuss later.

377
00:14:53.246 --> 00:14:57.413
The person who is here is like oh please, talk to me.

378
00:14:58.392 --> 00:15:00.831
So again, you see these key words.

379
00:15:00.831 --> 00:15:04.490
These were found by Eric Gilbert in 2012.

380
00:15:04.490 --> 00:15:06.763
So you can look at these kind of keyword phrases also.

381
00:15:06.763 --> 00:15:09.025
And those, again, are the easier cases.

382
00:15:09.025 --> 00:15:11.154
But these, again, I feel I said

383
00:15:11.154 --> 00:15:12.573
for being of a specific nature.

384
00:15:12.573 --> 00:15:14.509
And why are you forced to think about something,

385
00:15:14.509 --> 00:15:18.176
what are some general domain independencies.

386
00:15:19.500 --> 00:15:22.069
So how might you do this?

387
00:15:22.069 --> 00:15:24.054
So here is some inspiration you took.

388
00:15:24.054 --> 00:15:28.426
There's this phrasing done, I think is really quite,

389
00:15:28.426 --> 00:15:30.331
I really like how this was put.

390
00:15:30.331 --> 00:15:31.702
It's by--

391
00:15:31.702 --> 00:15:33.350
(blowing nose)

392
00:15:33.350 --> 00:15:36.247
He said communicative behaviors like conversations

393
00:15:36.247 --> 00:15:39.767
are quote, patterned and coordinated like a dance.

394
00:15:39.767 --> 00:15:43.114
And it's sort of a nice image to begin.

395
00:15:43.114 --> 00:15:46.500
So if we take the metaphor a little further

396
00:15:46.500 --> 00:15:49.198
and you say suppose two people are dancing

397
00:15:49.198 --> 00:15:50.948
and I wanna know who has the lead.

398
00:15:50.948 --> 00:15:53.919
Well, you can say who is it that seems to be initiating

399
00:15:53.919 --> 00:15:56.838
the actions and who is it who is kind of medium reacting?

400
00:15:56.838 --> 00:15:58.854
They don't necessarily mirror that person directly,

401
00:15:58.854 --> 00:16:01.169
but one person is sort of initiating,

402
00:16:01.169 --> 00:16:03.304
the other person is reacting immediately

403
00:16:03.304 --> 00:16:06.037
to what the person who has the lead.

404
00:16:06.037 --> 00:16:07.370
It's very clear.

405
00:16:08.987 --> 00:16:11.689
So why don't we look at this in language.

406
00:16:11.689 --> 00:16:16.070
So this is Uma Thurman, John Travolta, Pulp Fiction, yay.

407
00:16:16.070 --> 00:16:19.454
Suppose it's a case that every time Uma Thurman

408
00:16:19.454 --> 00:16:22.518
uses a preposition and a determiner,

409
00:16:22.518 --> 00:16:24.898
then John Travolta replies with a preposition

410
00:16:24.898 --> 00:16:25.982
and a determiner.

411
00:16:25.982 --> 00:16:28.042
Not necessarily the same preposition,

412
00:16:28.042 --> 00:16:32.388
but he is mirroring the class of function word

413
00:16:32.388 --> 00:16:34.099
that she is using.

414
00:16:34.099 --> 00:16:36.553
And yeah, here is a preposition, here is a preposition.

415
00:16:36.553 --> 00:16:38.635
If you saw that kind of behavior,

416
00:16:38.635 --> 00:16:42.027
you might say well, I'm confirmed as the conversational lead

417
00:16:42.027 --> 00:16:44.059
just as in a dance situation,

418
00:16:44.059 --> 00:16:47.324
you could say I confirm who has the lead in that setting.

419
00:16:47.324 --> 00:16:49.470
So that's the idea, we're gonna look for an asymmetric

420
00:16:49.470 --> 00:16:51.924
annotation of a linguistic style.

421
00:16:51.924 --> 00:16:54.167
To say a little more about this,

422
00:16:54.167 --> 00:16:57.857
I wanna contrast the idea of cell coordination

423
00:16:57.857 --> 00:17:00.754
with some other possibilities of coordination or mirroring

424
00:17:00.754 --> 00:17:02.754
that you might think of.

425
00:17:09.573 --> 00:17:12.607
So we prefer not to think about direct repetition

426
00:17:12.607 --> 00:17:13.856
or pairing.

427
00:17:13.856 --> 00:17:17.361
So you might be thinking oh, this is just a case

428
00:17:17.361 --> 00:17:20.613
of someone repeating directly what someone else said.

429
00:17:20.613 --> 00:17:22.837
And the reason we like to avoid thinking

430
00:17:22.837 --> 00:17:24.232
just about the repetition

431
00:17:24.232 --> 00:17:27.915
is that for one thing, that's under a speaker's control.

432
00:17:27.915 --> 00:17:30.627
So you can try to amp it, you might be trying

433
00:17:30.627 --> 00:17:33.835
to assert your power, say, in certain relationships

434
00:17:33.835 --> 00:17:36.164
and you might be doing more repetition on purpose.

435
00:17:36.164 --> 00:17:38.042
Whereas what we're interested in

436
00:17:38.042 --> 00:17:40.099
is the real power situation, if you will.

437
00:17:40.099 --> 00:17:42.165
So we'd like to get signals that are

438
00:17:42.165 --> 00:17:44.040
not under the speaker's conscious control, possibly,

439
00:17:44.040 --> 00:17:47.123
because that way, they can't mask it.

440
00:17:48.060 --> 00:17:51.307
Also, direct repetition could be just the choice of topic.

441
00:17:51.307 --> 00:17:54.061
So Hal does a lot of vlogging,

442
00:17:54.061 --> 00:17:56.220
so I'm gonna talk about a lot of vlogging,

443
00:17:56.220 --> 00:17:57.421
I'm gonna use the word vlog,

444
00:17:57.421 --> 00:17:58.421
he's gonna use the word vlog.

445
00:17:58.421 --> 00:18:01.225
That doesn't mean one of us is in power over each other,

446
00:18:01.225 --> 00:18:03.001
it just means the topic of vlogging

447
00:18:03.001 --> 00:18:05.465
requires us to both use the word vlog.

448
00:18:05.465 --> 00:18:08.040
Are there any student who vlog?

449
00:18:08.040 --> 00:18:10.633
<v Man>I have a vlog.</v>

450
00:18:10.633 --> 00:18:12.550
<v ->You've both got them.</v>

451
00:18:13.634 --> 00:18:15.552
Right, so we don't want topics

452
00:18:15.552 --> 00:18:17.689
to be confused with influence.

453
00:18:17.689 --> 00:18:19.431
And so that's another reason we're not gonna look

454
00:18:19.431 --> 00:18:21.767
for this direct repetition.

455
00:18:21.767 --> 00:18:23.070
Rather, we're gonna rely again

456
00:18:23.070 --> 00:18:25.669
On Diederhoffer and Pennebaker's intuition,

457
00:18:25.669 --> 00:18:27.976
that we can look at a function class matching

458
00:18:27.976 --> 00:18:29.962
like how does my rate of prepositions

459
00:18:29.962 --> 00:18:32.643
match your rate of prepositions when I talk.

460
00:18:32.643 --> 00:18:35.844
And the reason is it's unconscious, like we just argued,

461
00:18:35.844 --> 00:18:40.502
it's not something you can easily find or manipulate.

462
00:18:40.502 --> 00:18:42.365
Experiment, just prove this one to you.

463
00:18:42.365 --> 00:18:46.909
How many prepositions did I just use in the last sentence?

464
00:18:46.909 --> 00:18:49.518
Aren't you paying attention?

465
00:18:49.518 --> 00:18:51.447
I don't know either.

466
00:18:51.447 --> 00:18:53.113
Maybe I'm not paying attention.

467
00:18:53.113 --> 00:18:54.161
But the point is

468
00:18:54.161 --> 00:18:56.762
we can't consciously track these kind of things.

469
00:18:56.762 --> 00:18:58.867
So if you can't consciously track them,

470
00:18:58.867 --> 00:19:00.142
it's very hard to believe

471
00:19:00.142 --> 00:19:01.684
you're consciously controlling them

472
00:19:01.684 --> 00:19:03.863
to change your perceived control.

473
00:19:03.863 --> 00:19:05.837
And of course, there are frequent things

474
00:19:05.837 --> 00:19:08.282
like prepositions happen all the time In our life.

475
00:19:08.282 --> 00:19:10.103
So they're our most useful cue

476
00:19:10.103 --> 00:19:11.876
for this kind of thing.

477
00:19:11.876 --> 00:19:14.929
Again, some more just intuition

478
00:19:14.929 --> 00:19:17.395
as to what kind of matching you're talking about.

479
00:19:17.395 --> 00:19:19.325
And you'll notice, I kind of like movies,

480
00:19:19.325 --> 00:19:21.657
so this is Steve McQueen.

481
00:19:21.657 --> 00:19:23.554
This is a fake conversation.

482
00:19:23.554 --> 00:19:27.561
Someone imagined this conversation and wrote it down.

483
00:19:27.561 --> 00:19:31.182
And they had Steve McQueen say at least you're outside.

484
00:19:31.182 --> 00:19:32.592
And they had Ellen DuBurough reply

485
00:19:32.592 --> 00:19:34.833
it doesn't make much difference.

486
00:19:34.833 --> 00:19:36.905
Doesn't seem like there's much going on here,

487
00:19:36.905 --> 00:19:38.504
but notice the screenwriters could've had

488
00:19:38.504 --> 00:19:40.357
Ellen DuBurough say it doesn't really matter

489
00:19:40.357 --> 00:19:42.612
or it's not important.

490
00:19:42.612 --> 00:19:44.221
They also mean about the same nothing

491
00:19:44.221 --> 00:19:45.909
that the other thing means.

492
00:19:45.909 --> 00:19:49.004
But there's something kind of special about this,

493
00:19:49.004 --> 00:19:51.243
which is it turns out that this,

494
00:19:51.243 --> 00:19:52.894
it doesn't make much difference,

495
00:19:52.894 --> 00:19:56.393
has what the NLP system calls a modifier

496
00:19:56.393 --> 00:19:59.057
and there is a quantifier in what Steve McQueen says.

497
00:19:59.057 --> 00:20:02.175
So there's a match in word class.

498
00:20:02.175 --> 00:20:03.770
And again, to make this point again,

499
00:20:03.770 --> 00:20:05.316
it's very hard to look

500
00:20:05.316 --> 00:20:07.253
that the screenwriter at this point said gee,

501
00:20:07.253 --> 00:20:08.631
I better make sure Ellen DuBourough

502
00:20:08.631 --> 00:20:10.605
uses a quantifier right now.

503
00:20:10.605 --> 00:20:12.795
Even in this imagined conversation,

504
00:20:12.795 --> 00:20:15.010
somehow they unconsciously are putting this matching

505
00:20:15.010 --> 00:20:16.941
in function verb class.

506
00:20:16.941 --> 00:20:21.108
And that's the kind of thing we're gonna be looking for.

507
00:20:22.436 --> 00:20:25.551
All right, so how are you gonna measure this.

508
00:20:25.551 --> 00:20:28.061
We need some sort of numerical thing going on.

509
00:20:28.061 --> 00:20:30.295
Specifically what we're interested in

510
00:20:30.295 --> 00:20:32.614
is for a given function word class C,

511
00:20:32.614 --> 00:20:36.325
say, prepositions, and I have two people X1 and X2 talking,

512
00:20:36.325 --> 00:20:39.051
I'm interested in how much does speaker X1

513
00:20:39.051 --> 00:20:43.218
use prepositions and immediately trigger X2 to use it.

514
00:20:44.181 --> 00:20:46.281
Okay, so I'm looking for a very specific

515
00:20:46.281 --> 00:20:47.497
notion of influence.

516
00:20:47.497 --> 00:20:49.693
It's not that they're like I should use this word,

517
00:20:49.693 --> 00:20:52.281
right at that instant, I'm saying this preposition

518
00:20:52.281 --> 00:20:54.152
and you used a preposition in an immediate reply.

519
00:20:54.152 --> 00:20:56.985
So very fast, very sort of strong.

520
00:20:58.801 --> 00:21:01.418
I mean, and we had to use a little bit

521
00:21:01.418 --> 00:21:03.466
of a flipped terminology and rather than talk about

522
00:21:03.466 --> 00:21:04.936
X1 triggering X2,

523
00:21:04.936 --> 00:21:07.561
I'm gonna flip it and say how much does X2

524
00:21:07.561 --> 00:21:09.494
coordinate to X1,

525
00:21:09.494 --> 00:21:11.032
just a little terminological switch

526
00:21:11.032 --> 00:21:12.927
and I'm sure you guys are all into that.

527
00:21:12.927 --> 00:21:17.673
So here is a measure that Danescu-Niculescu-Mizil

528
00:21:17.673 --> 00:21:21.841
and a student named Dumais Gamon came up with in 2011.

529
00:21:21.841 --> 00:21:23.640
The first part is sort of obvious.

530
00:21:23.640 --> 00:21:25.474
How would measure this notion of triggering?

531
00:21:25.474 --> 00:21:28.132
Well, you say given that X1 used a preposition

532
00:21:28.132 --> 00:21:30.428
and that X2 is immediately replying,

533
00:21:30.428 --> 00:21:33.830
what's the probability that X2 will add a preposition.

534
00:21:33.830 --> 00:21:37.051
So that's usually like an immediate effect of influence.

535
00:21:37.051 --> 00:21:39.815
Obviously, you wanna know about this person

536
00:21:39.815 --> 00:21:43.195
and then about that person as well.

537
00:21:43.195 --> 00:21:45.716
However, we like to control for a bunch of things.

538
00:21:45.716 --> 00:21:48.833
So we'd like to control for the fact that maybe

539
00:21:48.833 --> 00:21:52.332
Hal uses prepositions all the time.

540
00:21:52.332 --> 00:21:55.362
So if I use a proposition, then Hal uses a preposition,

541
00:21:55.362 --> 00:21:57.456
but he's always using prepositions, god knows why,

542
00:21:57.456 --> 00:22:00.513
that's not me influencing it, right?

543
00:22:00.513 --> 00:22:03.753
So I need to subtract all of his sort of normal,

544
00:22:03.753 --> 00:22:05.113
if you call it normal in that case,

545
00:22:05.113 --> 00:22:06.676
rate of prepositions.

546
00:22:06.676 --> 00:22:08.155
Also, it could be

547
00:22:08.155 --> 00:22:10.293
that maybe Hal doesn't use prepositions all the time,

548
00:22:10.293 --> 00:22:12.570
but somehow, when people from Cornell come,

549
00:22:12.570 --> 00:22:14.943
there's something special about our relationship

550
00:22:14.943 --> 00:22:17.929
such that it's given me the use of prepositions a lot.

551
00:22:17.929 --> 00:22:20.127
That's still not my immediate speech

552
00:22:20.127 --> 00:22:22.179
triggering his immediate reply.

553
00:22:22.179 --> 00:22:25.445
And that's the kind of influence I'm looking at.

554
00:22:25.445 --> 00:22:28.023
So this is why we subtract off this term.

555
00:22:28.023 --> 00:22:29.620
What's the probability that X2,

556
00:22:29.620 --> 00:22:31.328
I guess this is now an email,

557
00:22:31.328 --> 00:22:35.567
uses a preposition given that we're in a situation

558
00:22:35.567 --> 00:22:38.879
where X2 is immediately replying to X1?

559
00:22:38.879 --> 00:22:41.628
I'll let you do a quick intuition check here.

560
00:22:41.628 --> 00:22:43.815
Can this probability be negative?

561
00:22:43.815 --> 00:22:44.815
<v Man>Yes.</v>

562
00:22:45.770 --> 00:22:47.603
<v ->I got one, yes, yes.</v>

563
00:22:48.623 --> 00:22:50.672
It can indeed be negative.

564
00:22:50.672 --> 00:22:53.676
Thus, we need a case for a diversion.

565
00:22:53.676 --> 00:22:57.843
So it could be every time I use a preposition normally,

566
00:22:58.698 --> 00:23:01.683
so Hal starts refusing to use prepositions

567
00:23:01.683 --> 00:23:04.774
when I talk to him and use prepositions,

568
00:23:04.774 --> 00:23:07.319
even though he always uses them all the time otherwise.

569
00:23:07.319 --> 00:23:09.717
That would be him diverging from me,

570
00:23:09.717 --> 00:23:11.142
and we actually see instances,

571
00:23:11.142 --> 00:23:12.860
there are some famous instances of this

572
00:23:12.860 --> 00:23:14.982
in the modern day transcripts, which I won't get into,

573
00:23:14.982 --> 00:23:17.443
but you can imagine that kind of thing happening

574
00:23:17.443 --> 00:23:19.045
and you'll get a negative score.

575
00:23:19.045 --> 00:23:20.468
So I just want people to understand what's the scales.

576
00:23:20.468 --> 00:23:23.641
<v Man>Is there a reason that you didn't make this</v>

577
00:23:23.641 --> 00:23:26.159
a difference of log cross and therefore ratio?

578
00:23:26.159 --> 00:23:27.304
<v ->It interprets better</v>

579
00:23:27.304 --> 00:23:29.720
so you don't have this infinite number versus,

580
00:23:29.720 --> 00:23:31.427
so it did have a lot.

581
00:23:31.427 --> 00:23:35.495
The qualities of interest are in, like point-wise

582
00:23:35.495 --> 00:23:36.865
neutral accumulation, right,

583
00:23:36.865 --> 00:23:38.584
you just need to log whenever.

584
00:23:38.584 --> 00:23:41.307
But it's just so much easier to scale the ones and zeros.

585
00:23:41.307 --> 00:23:43.052
So it's just more of a measure.

586
00:23:43.052 --> 00:23:46.887
It sort of a property of interpretability than anything.

587
00:23:46.887 --> 00:23:51.054
It's all about what's it mean to have a 35 coordination.

588
00:23:53.650 --> 00:23:56.884
Okay, so that was the measure that I just talked about.

589
00:23:56.884 --> 00:23:59.126
So it could go negative over positive,

590
00:23:59.126 --> 00:24:00.834
it's the longest axis.

591
00:24:00.834 --> 00:24:03.476
So what is this graph, so I wanna look at real data now

592
00:24:03.476 --> 00:24:05.839
and figure out if their are global coordination

593
00:24:05.839 --> 00:24:10.234
corresponds or correlates with power relations.

594
00:24:10.234 --> 00:24:11.379
So in this data,

595
00:24:11.379 --> 00:24:14.213
we're gonna look at supreme court transcripts,

596
00:24:14.213 --> 00:24:16.746
these are like oral arguments before the supreme court.

597
00:24:16.746 --> 00:24:18.825
You have a lawyer on one side, lawyer on the other,

598
00:24:18.825 --> 00:24:21.482
they talk to the justices, questions, things like that.

599
00:24:21.482 --> 00:24:23.170
There are no witnesses or whatever,

600
00:24:23.170 --> 00:24:25.818
it's just, these are all our members.

601
00:24:25.818 --> 00:24:27.744
So we have a whole bunch of this data.

602
00:24:27.744 --> 00:24:30.744
You can get it online, it's actually first developed

603
00:24:30.744 --> 00:24:33.316
by an industrious student.

604
00:24:33.316 --> 00:24:35.928
So I definitely agree with him.

605
00:24:35.928 --> 00:24:38.978
But you can get our version of the data online.

606
00:24:38.978 --> 00:24:41.366
So you have a lot of energy just between these justices.

607
00:24:41.366 --> 00:24:43.033
Who is not in there?

608
00:24:45.329 --> 00:24:46.162
Help out.

609
00:24:47.381 --> 00:24:48.548
He never said.

610
00:24:49.560 --> 00:24:52.142
In the New York Times, the one time he said something,

611
00:24:52.142 --> 00:24:56.309
they don't even know what he said In the New York times.

612
00:24:58.233 --> 00:25:01.802
This axis here is various function word classes

613
00:25:01.802 --> 00:25:05.086
such as prepositions are here, adverbs are here.

614
00:25:05.086 --> 00:25:07.547
So you can talk about different functional classes here.

615
00:25:07.547 --> 00:25:10.546
For the interest of this conversation,

616
00:25:10.546 --> 00:25:13.093
let's just talk about the stuff to the left of the bar.

617
00:25:13.093 --> 00:25:15.104
These are various abbreviations.

618
00:25:15.104 --> 00:25:17.777
Roughly speaking, averages, different ways

619
00:25:17.777 --> 00:25:21.258
of averaging results for the different word classes.

620
00:25:21.258 --> 00:25:23.304
So what are we seeing here, so this is coordination.

621
00:25:23.304 --> 00:25:26.515
In this case, all the coordination models are positive.

622
00:25:26.515 --> 00:25:28.856
So that means we didn't have any divergence

623
00:25:28.856 --> 00:25:30.606
on that coordination.

624
00:25:32.342 --> 00:25:35.326
So we see that everyone is coordinating everybody,

625
00:25:35.326 --> 00:25:37.341
but what are these different bars?

626
00:25:37.341 --> 00:25:40.568
So the purple bars here, remember my royal purple business?

627
00:25:40.568 --> 00:25:42.824
They are when someone of lower status,

628
00:25:42.824 --> 00:25:45.561
namely a lawyer, is speaking to someone of high status,

629
00:25:45.561 --> 00:25:47.660
namely, a supreme court justice.

630
00:25:47.660 --> 00:25:49.410
And there's some level of accommodation

631
00:25:49.410 --> 00:25:50.834
and these are the averages.

632
00:25:50.834 --> 00:25:52.133
First, it's the green bars.

633
00:25:52.133 --> 00:25:53.985
That's when someone of high status,

634
00:25:53.985 --> 00:25:56.064
a supreme court justice, is speaking to a lawyer,

635
00:25:56.064 --> 00:25:59.623
lower status, and you get some level of coordination there.

636
00:25:59.623 --> 00:26:01.768
In most cases, these levels of coordination are positive,

637
00:26:01.768 --> 00:26:05.780
but the purple bars are always above the green bars,

638
00:26:05.780 --> 00:26:09.574
meaning the people in power are accommodating,

639
00:26:09.574 --> 00:26:12.672
are coordinating less to people who are not in power

640
00:26:12.672 --> 00:26:13.922
and vice versa.

641
00:26:17.321 --> 00:26:19.025
Hold on one second.

642
00:26:19.025 --> 00:26:21.077
Because this point, for people paying attention,

643
00:26:21.077 --> 00:26:23.558
at this point, usually a couple of objections come up.

644
00:26:23.558 --> 00:26:24.772
And in the interest of time, I'll tell you

645
00:26:24.772 --> 00:26:27.139
what some of these objections are.

646
00:26:27.139 --> 00:26:29.546
Okay, well first of all, does anybody have any questions

647
00:26:29.546 --> 00:26:32.977
on interpreting the bars here, yeah?

648
00:26:32.977 --> 00:26:34.920
<v Man>How much do you have to say</v>

649
00:26:34.920 --> 00:26:36.587
before you can tell?

650
00:26:40.306 --> 00:26:41.611
<v ->All right, so by the way,</v>

651
00:26:41.611 --> 00:26:43.807
these are statistically significant.

652
00:26:43.807 --> 00:26:45.070
So I'm sorry, I should've mentioned this.

653
00:26:45.070 --> 00:26:48.570
This is sort of our p value, the .05, .01,

654
00:26:50.643 --> 00:26:52.393
.05, .01, .001, okay.

655
00:26:54.463 --> 00:26:55.534
How much do you need,

656
00:26:55.534 --> 00:26:57.769
you actually never know about the learning curves

657
00:26:57.769 --> 00:27:01.017
just because I think the point wasn't necessarily for

658
00:27:01.017 --> 00:27:03.313
we should do it or someone should do that,

659
00:27:03.313 --> 00:27:04.902
I don't have any now.

660
00:27:04.902 --> 00:27:06.849
There's a question, can you guys hear the questions?

661
00:27:06.849 --> 00:27:08.473
Okay.

662
00:27:08.473 --> 00:27:10.868
I've been coordinating to you,

663
00:27:10.868 --> 00:27:13.626
and you feel empowered and how about that?

664
00:27:13.626 --> 00:27:17.271
All right, so some projections I usually, oh, yes?

665
00:27:17.271 --> 00:27:21.438
<v Woman>Sorry, how did you account for the amount</v>

666
00:27:23.423 --> 00:27:26.590
of taxing human debt in the first one.

667
00:27:27.438 --> 00:27:29.907
<v Lecturer>So these are averages over every time</v>

668
00:27:29.907 --> 00:27:31.627
someone who is in power talks to someone

669
00:27:31.627 --> 00:27:32.825
who is not in power.

670
00:27:32.825 --> 00:27:34.850
So all the different participants are treated

671
00:27:34.850 --> 00:27:36.121
as one human being.

672
00:27:36.121 --> 00:27:38.848
<v Woman>Okay, so the fact that perhaps they're aware</v>

673
00:27:38.848 --> 00:27:43.015
of the comparison to one question, was the accounted for?

674
00:27:46.729 --> 00:27:50.630
<v ->So not per se, but it kind of varies.</v>

675
00:27:50.630 --> 00:27:52.143
So if you look at the transcripts,

676
00:27:52.143 --> 00:27:53.208
it's actually pretty interesting.

677
00:27:53.208 --> 00:27:54.527
Sometimes the justices go on for a while.

678
00:27:54.527 --> 00:27:58.107
And sometimes you have people like Scalia in there.

679
00:27:58.107 --> 00:28:00.202
And they put, like, talks in there.

680
00:28:00.202 --> 00:28:03.042
But I'm actually, so your question kind of comes up

681
00:28:03.042 --> 00:28:05.046
to some of the points in a direction

682
00:28:05.046 --> 00:28:06.441
that totally comes up

683
00:28:06.441 --> 00:28:08.349
because there's something about the supreme court

684
00:28:08.349 --> 00:28:10.453
that is giving these results.

685
00:28:10.453 --> 00:28:13.434
So you can say it's just a stylized reaction

686
00:28:13.434 --> 00:28:14.849
where it's easy to tell.

687
00:28:14.849 --> 00:28:16.546
You could say well, it's all Q and A

688
00:28:16.546 --> 00:28:18.709
so you're gonna get these kind of repetitions,

689
00:28:18.709 --> 00:28:20.423
they're not gonna correlate with the thing you're planning,

690
00:28:20.423 --> 00:28:21.877
stuff like that.

691
00:28:21.877 --> 00:28:24.110
So one objection does remain,

692
00:28:24.110 --> 00:28:26.048
that this is sort of too artificial.

693
00:28:26.048 --> 00:28:27.660
So it nicely wraps all of it

694
00:28:27.660 --> 00:28:29.560
and finally, a statement about real life.

695
00:28:29.560 --> 00:28:30.516
Are you gonna say something?

696
00:28:30.516 --> 00:28:33.221
<v Man>So we're not terribly keen on audience questions</v>

697
00:28:33.221 --> 00:28:35.791
and speaker responses?

698
00:28:35.791 --> 00:28:38.268
Can you flaunt audience questions

699
00:28:38.268 --> 00:28:40.391
and speaker responses again.

700
00:28:40.391 --> 00:28:42.250
Figure out who has the power.

701
00:28:42.250 --> 00:28:44.623
<v ->So in the supreme court, we don't do it</v>

702
00:28:44.623 --> 00:28:45.923
because there is no audience.

703
00:28:45.923 --> 00:28:47.613
This is another question about domains,

704
00:28:47.613 --> 00:28:49.610
so who are the participants.

705
00:28:49.610 --> 00:28:52.133
And so I wouldn't raise this objection

706
00:28:52.133 --> 00:28:54.716
in a courthouse to overrule it.

707
00:28:56.841 --> 00:28:58.508
This never gets old.

708
00:29:00.864 --> 00:29:02.292
So that's one objection.

709
00:29:02.292 --> 00:29:05.483
Another objection is that this actually isn't a, say,

710
00:29:05.483 --> 00:29:08.221
say about, this isn't really a finding about

711
00:29:08.221 --> 00:29:10.971
power per se, but of personality.

712
00:29:14.840 --> 00:29:16.818
It's a little bit of a solo argument, but it could be,

713
00:29:16.818 --> 00:29:20.131
theory goes, that people who become supreme court justices

714
00:29:20.131 --> 00:29:23.000
are not the type of people who spend their time repeating

715
00:29:23.000 --> 00:29:24.833
what other people say.

716
00:29:26.074 --> 00:29:27.916
So it's not really about the fact

717
00:29:27.916 --> 00:29:29.737
that there's a power relationship,

718
00:29:29.737 --> 00:29:31.097
it's about the kind of people

719
00:29:31.097 --> 00:29:32.932
that happen to be supreme court justices.

720
00:29:32.932 --> 00:29:35.849
So this is really about personality

721
00:29:36.934 --> 00:29:40.196
and not about the status or power in relationship.

722
00:29:40.196 --> 00:29:42.761
So that's another thing.

723
00:29:42.761 --> 00:29:44.489
And there are probably others, but I'm gonna go on,

724
00:29:44.489 --> 00:29:47.374
since it's 4:30 because I have another couple of,

725
00:29:47.374 --> 00:29:48.733
but I wanted to cover this

726
00:29:48.733 --> 00:29:50.066
because usually at this point,

727
00:29:50.066 --> 00:29:52.422
people are like you've gotta be kidding, I don't believe it,

728
00:29:52.422 --> 00:29:56.589
but I have more data that will hopefully allay your opinion.

729
00:29:58.234 --> 00:30:01.814
Okay so, one, I can totally switch domains.

730
00:30:01.814 --> 00:30:04.901
Okay, so now we can talk about Wikipedians

731
00:30:04.901 --> 00:30:07.362
where the domain is audience participation

732
00:30:07.362 --> 00:30:09.784
because any Wikipedia editors can edit it.

733
00:30:09.784 --> 00:30:11.430
So what's copying in Wikipedia,

734
00:30:11.430 --> 00:30:13.726
so I'm not talking about the sort of front page stuff,

735
00:30:13.726 --> 00:30:17.234
but it kind of seems there are discussion pages

736
00:30:17.234 --> 00:30:19.491
for each article where Wikipedia editors

737
00:30:19.491 --> 00:30:23.384
can talk about how they feel about the format

738
00:30:23.384 --> 00:30:25.156
or the content of an article.

739
00:30:25.156 --> 00:30:27.555
And there are also talk pages where Wikipedians

740
00:30:27.555 --> 00:30:29.298
can talk about sort of, with each other,

741
00:30:29.298 --> 00:30:32.116
policies about Wikipedia, questions about

742
00:30:32.116 --> 00:30:33.991
how some things should be handled.

743
00:30:33.991 --> 00:30:35.982
And there are things like elections on Wikipedia,

744
00:30:35.982 --> 00:30:38.824
and that's really interesting for those of you.

745
00:30:38.824 --> 00:30:41.036
The elections are really interesting

746
00:30:41.036 --> 00:30:43.225
because whoever the election is for,

747
00:30:43.225 --> 00:30:46.320
people become elected to admin status.

748
00:30:46.320 --> 00:30:48.548
And it's a totally open process

749
00:30:48.548 --> 00:30:51.774
so people have to give the reasons on Wikipedia

750
00:30:51.774 --> 00:30:53.896
as to why they're voting yes or no

751
00:30:53.896 --> 00:30:55.325
on this particular candidacy.

752
00:30:55.325 --> 00:30:57.860
The candidate waits and people can defend them.

753
00:30:57.860 --> 00:31:00.050
For people conversational data,

754
00:31:00.050 --> 00:31:02.142
I think Wikipedia is fascinating.

755
00:31:02.142 --> 00:31:03.768
It's also available for my webpage,

756
00:31:03.768 --> 00:31:05.121
I encourage you to look at it.

757
00:31:05.121 --> 00:31:07.190
Why am I spending some time talking about admins?

758
00:31:07.190 --> 00:31:09.934
Because that means something funny.

759
00:31:09.934 --> 00:31:12.684
It means status can change, okay.

760
00:31:13.782 --> 00:31:17.635
So I now make get addressed to the different domain name

761
00:31:17.635 --> 00:31:19.198
by going through a different than Wikipedia.

762
00:31:19.198 --> 00:31:22.050
But this thing about personality versus status,

763
00:31:22.050 --> 00:31:26.094
I can look at because people can have the same person,

764
00:31:26.094 --> 00:31:28.300
can change power status by becoming an admin.

765
00:31:28.300 --> 00:31:30.903
Okay, so can separate the personality

766
00:31:30.903 --> 00:31:32.820
from status hypothesis.

767
00:31:32.820 --> 00:31:35.840
So what does this graph show, this is time,

768
00:31:35.840 --> 00:31:37.898
and this little Wikipedian here,

769
00:31:37.898 --> 00:31:40.052
he or she is a big Wikipedian,

770
00:31:40.052 --> 00:31:42.380
this Wikipedian is going to stay on her election.

771
00:31:42.380 --> 00:31:45.654
The yellow bar is the time when they are crowned.

772
00:31:45.654 --> 00:31:47.154
That is the admin.

773
00:31:48.634 --> 00:31:51.739
These are regular old wikipedians.

774
00:31:51.739 --> 00:31:54.695
And there's a graph over here as well.

775
00:31:54.695 --> 00:31:57.042
What I'm graphing over time here,

776
00:31:57.042 --> 00:31:59.019
this line is how much does this Wikipedian

777
00:31:59.019 --> 00:32:01.686
coordinate to other wikipedians.

778
00:32:02.820 --> 00:32:06.518
And you can see that they coordinate just fine.

779
00:32:06.518 --> 00:32:08.422
But then they get elected as an admin

780
00:32:08.422 --> 00:32:10.566
and suddenly global coordination with other people

781
00:32:10.566 --> 00:32:11.637
has dropped.

782
00:32:11.637 --> 00:32:14.062
Same human being, status has changed,

783
00:32:14.062 --> 00:32:15.919
level of coordination going down.

784
00:32:15.919 --> 00:32:18.169
hey, for a hypothesis, yes,

785
00:32:19.224 --> 00:32:23.240
whereas you can see here in just a second

786
00:32:23.240 --> 00:32:25.284
that the other people, how much have they been coordinating

787
00:32:25.284 --> 00:32:26.282
to that person?

788
00:32:26.282 --> 00:32:27.783
Before they became an admin,

789
00:32:27.783 --> 00:32:29.548
they're not coordinating very much to them.

790
00:32:29.548 --> 00:32:30.964
But once they become an admin,

791
00:32:30.964 --> 00:32:33.410
suddenly everyone is coordinating with them.

792
00:32:33.410 --> 00:32:35.872
Same person, change of status, coordination changes

793
00:32:35.872 --> 00:32:37.382
as predicted.

794
00:32:37.382 --> 00:32:38.795
Yes, objection.

795
00:32:38.795 --> 00:32:42.545
(static drowns out dialogue)

796
00:32:48.032 --> 00:32:49.778
So the data is actually catatonic,

797
00:32:49.778 --> 00:32:53.433
so we're looking at secure ways of building conversations.

798
00:32:53.433 --> 00:32:55.300
So you only initiate once.

799
00:32:55.300 --> 00:32:56.473
So then every time you speak,

800
00:32:56.473 --> 00:33:01.265
there's both an initiation time and a response time.

801
00:33:01.265 --> 00:33:03.672
<v Man>So feel free to defer this,</v>

802
00:33:03.672 --> 00:33:07.845
in the earlier figures, it looked like a lot

803
00:33:07.845 --> 00:33:09.929
was varying by quantifiers.

804
00:33:09.929 --> 00:33:14.287
I mean, quantifiers would seem unexpectedly prominent

805
00:33:14.287 --> 00:33:15.467
in those graphs.

806
00:33:15.467 --> 00:33:18.368
Quantifiers are not usually thought of

807
00:33:18.368 --> 00:33:20.841
as like, one of your major categories of function,

808
00:33:20.841 --> 00:33:22.764
so what's the deal with quantifiers?

809
00:33:22.764 --> 00:33:24.436
<v ->So actually I think in the graph,</v>

810
00:33:24.436 --> 00:33:27.390
in this, quantifiers, are suddenly not in effect.

811
00:33:27.390 --> 00:33:30.479
So I actually, I'm not a huge believer,

812
00:33:30.479 --> 00:33:33.479
let me think about the taping again,

813
00:33:35.655 --> 00:33:37.743
so new categories are piling in,

814
00:33:37.743 --> 00:33:40.619
but I have no assurance that they're real categories.

815
00:33:40.619 --> 00:33:42.779
So that's one reason we just take it all,

816
00:33:42.779 --> 00:33:44.683
that's a little bit technical.

817
00:33:44.683 --> 00:33:47.421
Let me just say that I'm not saying that these categories

818
00:33:47.421 --> 00:33:49.237
are necessarily the right thing to look at,

819
00:33:49.237 --> 00:33:50.780
we just look at a bunch of them

820
00:33:50.780 --> 00:33:52.293
so that you could take some effect

821
00:33:52.293 --> 00:33:54.848
and hope you're not dependent too much on the category,

822
00:33:54.848 --> 00:33:57.023
precise category and definitions.

823
00:33:57.023 --> 00:33:58.509
One reason we're not talking about it

824
00:33:58.509 --> 00:34:00.664
is actually the relative performance of different domains,

825
00:34:00.664 --> 00:34:02.265
which is actually closer out.

826
00:34:02.265 --> 00:34:04.469
So this thing about quantifiers being powerful here,

827
00:34:04.469 --> 00:34:06.662
I'm sure there's some other graph in here.

828
00:34:06.662 --> 00:34:08.636
And I have no explanation for that.

829
00:34:08.636 --> 00:34:10.841
<v Man>Is there any meaning to</v>

830
00:34:10.841 --> 00:34:14.350
the magnitude of the difference between coordination.

831
00:34:14.350 --> 00:34:16.270
Is there some way to interpret that

832
00:34:16.270 --> 00:34:20.437
maybe it's immediate or more lasting power or energy?

833
00:34:22.217 --> 00:34:25.996
<v ->So we actually have, and I don't know the actual amount,</v>

834
00:34:25.996 --> 00:34:27.888
I mean, literally, the numbers are this probability

835
00:34:27.888 --> 00:34:29.179
minus that probability.

836
00:34:29.179 --> 00:34:31.596
But it's scaled to a hundred.

837
00:34:33.505 --> 00:34:35.627
But actually in the supreme court data,

838
00:34:35.627 --> 00:34:38.940
there's both associate justices and then the chief justice,

839
00:34:38.940 --> 00:34:40.729
so there are actually three levels.

840
00:34:40.729 --> 00:34:42.392
And there, we see people coordinate more

841
00:34:42.392 --> 00:34:45.409
to the chief justice than to the associate justice.

842
00:34:45.409 --> 00:34:47.226
So there is some kind of layering going on there.

843
00:34:47.226 --> 00:34:49.199
I don't know how far you should take that.

844
00:34:49.199 --> 00:34:50.693
People who do the end round data set

845
00:34:50.693 --> 00:34:52.730
are welcome to, you know, there's a higher level,

846
00:34:52.730 --> 00:34:53.730
like 40 levels.

847
00:34:53.730 --> 00:34:55.813
Are you still hearing me?

848
00:34:57.116 --> 00:34:58.699
I'm in trouble now.

849
00:35:03.345 --> 00:35:06.125
Okay, so hopefully, people are happy.

850
00:35:06.125 --> 00:35:10.292
I like this graph a lot, let me just put it out there.

851
00:35:12.446 --> 00:35:15.092
I'll also say this, that I Like this graph too.

852
00:35:15.092 --> 00:35:17.720
Real quick, here's another example

853
00:35:17.720 --> 00:35:19.708
that has the previous data set.

854
00:35:19.708 --> 00:35:22.198
He didn't really talk, he talked really about status.

855
00:35:22.198 --> 00:35:24.251
Now, this isn't necessarily the same as power.

856
00:35:24.251 --> 00:35:27.016
If you're playing Sandra Day O' Connor on the tennis court,

857
00:35:27.016 --> 00:35:29.318
maybe you actually have the power.

858
00:35:29.318 --> 00:35:33.248
So you're trying to disentangle status from power,

859
00:35:33.248 --> 00:35:36.335
and what we can do about, we can do this,

860
00:35:36.335 --> 00:35:37.794
is in the US supreme court,

861
00:35:37.794 --> 00:35:39.621
this is just kind of a requirement,

862
00:35:39.621 --> 00:35:44.295
it's generally thought that people know ahead of time

863
00:35:44.295 --> 00:35:46.756
the predisposition of each supreme court justice.

864
00:35:46.756 --> 00:35:48.911
They kind of have an inkling which way they're gonna go

865
00:35:48.911 --> 00:35:49.986
ahead of time.

866
00:35:49.986 --> 00:35:54.294
There's a lot of cues about their general policy, whatever.

867
00:35:54.294 --> 00:35:56.041
This means when a lawyer walks in there,

868
00:35:56.041 --> 00:35:58.782
they have an idea of who is probably on their side

869
00:35:58.782 --> 00:36:00.637
and who is probably not on their side.

870
00:36:00.637 --> 00:36:03.151
So you're facing justices, they all have the same status

871
00:36:03.151 --> 00:36:04.643
with respect to you,

872
00:36:04.643 --> 00:36:06.363
but some of them you need more.

873
00:36:06.363 --> 00:36:08.330
Because some of them, you need to change their vote,

874
00:36:08.330 --> 00:36:10.436
and some of them you don't.

875
00:36:10.436 --> 00:36:12.276
So this graph here shows

876
00:36:12.276 --> 00:36:15.060
how much they're coordinating to someone

877
00:36:15.060 --> 00:36:17.525
who already had your opinion, and so you need them less,

878
00:36:17.525 --> 00:36:21.145
against someone whose opinion you needed to change.

879
00:36:21.145 --> 00:36:23.633
So they have more power over you

880
00:36:23.633 --> 00:36:25.019
even though they have the same status,

881
00:36:25.019 --> 00:36:26.737
they're all supreme court justices.

882
00:36:26.737 --> 00:36:28.322
And again, you see that difference,

883
00:36:28.322 --> 00:36:30.835
statistically significant across.

884
00:36:30.835 --> 00:36:33.047
<v Man>Did you look at swaying justices versus--</v>

885
00:36:33.047 --> 00:36:34.500
<v ->We did.</v>

886
00:36:34.500 --> 00:36:38.935
So you know, you can imagine how many times we all did this.

887
00:36:38.935 --> 00:36:41.414
It turns out, much to our surprise,

888
00:36:41.414 --> 00:36:44.257
people do not coordinate more to Kennedy.

889
00:36:44.257 --> 00:36:46.053
We thought for sure they would.

890
00:36:46.053 --> 00:36:49.382
They don't, there's other things going on, obviously.

891
00:36:49.382 --> 00:36:51.446
There's actually a ton going on.

892
00:36:51.446 --> 00:36:53.402
So remember, I tried to draw a signal,

893
00:36:53.402 --> 00:36:54.923
and then we actually go into it

894
00:36:54.923 --> 00:36:56.935
and oh, this jack in the box kind of jumps out.

895
00:36:56.935 --> 00:37:00.366
There is so much more here besides just coordination.

896
00:37:00.366 --> 00:37:02.650
But the fact that we get these correlations,

897
00:37:02.650 --> 00:37:06.599
I think the correlations says very selective activities.

898
00:37:06.599 --> 00:37:08.993
Okay, one last round on this.

899
00:37:08.993 --> 00:37:11.638
So I showed you a correlation.

900
00:37:11.638 --> 00:37:15.408
Those of you who know machine learning are probably...

901
00:37:15.408 --> 00:37:18.499
How many are in competition here.

902
00:37:18.499 --> 00:37:21.707
Well, we couldn't have done this without machine results,

903
00:37:21.707 --> 00:37:23.492
let me tell you, so here they are.

904
00:37:23.492 --> 00:37:26.612
So what I'm most interested in showing you

905
00:37:26.612 --> 00:37:29.666
is that these features have predictive power

906
00:37:29.666 --> 00:37:32.416
and they have cross domain power.

907
00:37:34.761 --> 00:37:36.935
So that's the point I'm trying to prove here.

908
00:37:36.935 --> 00:37:38.514
So look at this graph here.

909
00:37:38.514 --> 00:37:41.658
This figure shows if you're trying to classify

910
00:37:41.658 --> 00:37:44.212
something between two people and what they said,

911
00:37:44.212 --> 00:37:46.706
can you tell which one is in power?

912
00:37:46.706 --> 00:37:49.456
50% is your random baseline.

913
00:37:50.454 --> 00:37:54.330
So this here is if I'm using supreme court data

914
00:37:54.330 --> 00:37:55.642
by training tests.

915
00:37:55.642 --> 00:37:59.877
And this one here is how low bag of words does.

916
00:37:59.877 --> 00:38:01.779
So I'm gonna apply an SVM classifier

917
00:38:01.779 --> 00:38:04.271
and this features 20,000 words.

918
00:38:04.271 --> 00:38:05.771
How is that doing?

919
00:38:06.612 --> 00:38:08.529
It's doing nearly perfect.

920
00:38:08.529 --> 00:38:11.544
Because this is the difference from 50%.

921
00:38:11.544 --> 00:38:14.978
So this is baseline, this is practically 100%.

922
00:38:14.978 --> 00:38:17.060
The star being statistically significant,

923
00:38:17.060 --> 00:38:18.897
meaning different choice.

924
00:38:18.897 --> 00:38:21.718
So why is bag of words biggest of all?

925
00:38:21.718 --> 00:38:25.744
Your honor, it's real super easy in the supreme court

926
00:38:25.744 --> 00:38:28.532
to use that word to tell who is in charge.

927
00:38:28.532 --> 00:38:30.773
Okay, are we gonna beat that baseline, no way.

928
00:38:30.773 --> 00:38:34.181
What should we do here, so this is using coordination.

929
00:38:34.181 --> 00:38:36.655
Nine features only because there's nine classes.

930
00:38:36.655 --> 00:38:39.097
And the third in here means were doing significantly better

931
00:38:39.097 --> 00:38:40.550
than the random baseline,

932
00:38:40.550 --> 00:38:42.321
but we're getting our pants eaten off,

933
00:38:42.321 --> 00:38:44.156
does that make sense?

934
00:38:44.156 --> 00:38:46.284
We're doing very, very poorly with this.

935
00:38:46.284 --> 00:38:50.517
So you sure as hell aren't gonna hear this.

936
00:38:50.517 --> 00:38:52.715
Oh, by the way, what's this thing,

937
00:38:52.715 --> 00:38:55.020
this thing is practically doing random choice.

938
00:38:55.020 --> 00:38:56.262
This is stylistic features,

939
00:38:56.262 --> 00:38:57.961
this is if you use those part of speech features,

940
00:38:57.961 --> 00:39:00.062
those part of speech classes,

941
00:39:00.062 --> 00:39:02.494
but we don't take into account the angle.

942
00:39:02.494 --> 00:39:04.504
So you could ask this question

943
00:39:04.504 --> 00:39:06.828
is it just the part of speeches that are features themselves

944
00:39:06.828 --> 00:39:07.920
or is it the equity.

945
00:39:07.920 --> 00:39:09.264
And you can see if you're gonna use this,

946
00:39:09.264 --> 00:39:11.290
you need better data.

947
00:39:11.290 --> 00:39:12.598
Okay, so it's useful for something.

948
00:39:12.598 --> 00:39:15.024
It's just way more spent down words.

949
00:39:15.024 --> 00:39:17.700
But see how this breaks up.

950
00:39:17.700 --> 00:39:20.114
All right, why don't we use Wikipedia.

951
00:39:20.114 --> 00:39:21.475
Hey, tables are turned.

952
00:39:21.475 --> 00:39:22.947
It turns out it's very hard,

953
00:39:22.947 --> 00:39:24.165
at least the way we did,

954
00:39:24.165 --> 00:39:26.248
to tell who is in charge.

955
00:39:27.201 --> 00:39:29.442
But our coordination features can at least do that

956
00:39:29.442 --> 00:39:31.273
with a baseline.

957
00:39:31.273 --> 00:39:32.559
Statistically significant,

958
00:39:32.559 --> 00:39:33.961
even though bag of words couldn't,

959
00:39:33.961 --> 00:39:37.179
oh, but the stylistic features are doing a little better.

960
00:39:37.179 --> 00:39:39.381
So here, we see the bag of words baseline,

961
00:39:39.381 --> 00:39:41.902
but now we're not doing any better than this.

962
00:39:41.902 --> 00:39:44.069
What's the next two lines.

963
00:39:45.057 --> 00:39:47.867
<v Man>Training court and test wiki.</v>

964
00:39:47.867 --> 00:39:49.156
<v ->Look at that.</v>

965
00:39:49.156 --> 00:39:51.989
So what's wrong with bag of words?

966
00:39:53.568 --> 00:39:55.485
It's overfitted, right?

967
00:39:57.364 --> 00:40:00.547
What's the only solid features of doing it this way

968
00:40:00.547 --> 00:40:03.137
and having it be the baseline, you all said it.

969
00:40:03.137 --> 00:40:04.701
<v Man>Diversity.</v>

970
00:40:04.701 --> 00:40:05.534
<v ->Oh, yay!</v>

971
00:40:07.551 --> 00:40:09.883
All right, see if I can convince you of speech assembly

972
00:40:09.883 --> 00:40:11.837
and the predictive power, sort of in the processing,

973
00:40:11.837 --> 00:40:14.135
and notice, there are only nine of them from 20,000.

974
00:40:14.135 --> 00:40:16.135
So even Gary might gawk.

975
00:40:18.939 --> 00:40:21.903
I know he's seen my mom before.

976
00:40:21.903 --> 00:40:23.525
Yay, okay.

977
00:40:23.525 --> 00:40:24.963
Next case study now.

978
00:40:24.963 --> 00:40:27.540
Right, so we can veer to the next thing here

979
00:40:27.540 --> 00:40:29.727
in case you're like oh my god, is this gonna stop?

980
00:40:29.727 --> 00:40:30.857
No more supreme court.

981
00:40:30.857 --> 00:40:32.274
But I'm not done.

982
00:40:36.052 --> 00:40:39.134
Yay, supreme court, I'm in favor of you.

983
00:40:39.134 --> 00:40:42.091
All right, so this one, now we're gonna work on--

984
00:40:42.091 --> 00:40:43.270
<v Man>What do you think they would say?</v>

985
00:40:43.270 --> 00:40:46.636
<v ->Would you like me to be a lawyer?</v>

986
00:40:46.636 --> 00:40:48.118
They are awesome.

987
00:40:48.118 --> 00:40:50.543
That's what the A stands for.

988
00:40:50.543 --> 00:40:53.469
So remember this, you can put those two kind of things,

989
00:40:53.469 --> 00:40:55.017
this must be kind of a big question.

990
00:40:55.017 --> 00:40:57.754
Is it the phrasing or is it the contextual advice,

991
00:40:57.754 --> 00:40:59.800
can you separate them apart and determine

992
00:40:59.800 --> 00:41:01.622
what makes something memorable.

993
00:41:01.622 --> 00:41:03.505
This is joint work again with Christian

994
00:41:03.505 --> 00:41:05.100
Danescu-Niculescu-Mizil

995
00:41:05.100 --> 00:41:06.870
and then Justin Cheng was an undergrad at the time

996
00:41:06.870 --> 00:41:09.537
and them finally, Jon Kleinberg.

997
00:41:10.633 --> 00:41:13.488
Okay, so I'm trying to decide

998
00:41:13.488 --> 00:41:15.113
do I even need motivation for this,

999
00:41:15.113 --> 00:41:16.522
but I think I should say this.

1000
00:41:16.522 --> 00:41:18.697
Watch and, like, ask the question

1001
00:41:18.697 --> 00:41:21.020
of whether phrasing affects memorability.

1002
00:41:21.020 --> 00:41:23.125
And this is getting into rhetorical questions

1003
00:41:23.125 --> 00:41:24.449
that I care about deeply.

1004
00:41:24.449 --> 00:41:26.831
It's whether phrasing affects influence.

1005
00:41:26.831 --> 00:41:30.061
So he's telling some people that he'd really say this.

1006
00:41:30.061 --> 00:41:32.086
And that would mean I'm very frustrated by the fact

1007
00:41:32.086 --> 00:41:34.896
that all of them say something and because of interaction,

1008
00:41:34.896 --> 00:41:37.442
and then someone also saying what I think is the same

1009
00:41:37.442 --> 00:41:39.509
but they asked him differently, I go oh,

1010
00:41:39.509 --> 00:41:41.698
that was so smart, let's do that,

1011
00:41:41.698 --> 00:41:43.692
why didn't anyone else think of that.

1012
00:41:43.692 --> 00:41:45.571
And it's just driving me up the wall.

1013
00:41:45.571 --> 00:41:46.962
So I'd really like to fix this.

1014
00:41:46.962 --> 00:41:49.054
And we've seen this in politics and stuff.

1015
00:41:49.054 --> 00:41:51.391
Right, when people are running against other people,

1016
00:41:51.391 --> 00:41:53.099
I think it's an important question

1017
00:41:53.099 --> 00:41:55.780
of whether the way they put it, not just their status

1018
00:41:55.780 --> 00:41:57.193
or things like that,

1019
00:41:57.193 --> 00:41:59.666
whether to some effect, the phrasing can change

1020
00:41:59.666 --> 00:42:02.186
how people are convinced by things

1021
00:42:02.186 --> 00:42:05.322
or how people change their mind.

1022
00:42:05.322 --> 00:42:08.887
Okay, so other motivations would be

1023
00:42:08.887 --> 00:42:11.275
if you know that something is gonna be more memorable,

1024
00:42:11.275 --> 00:42:13.862
you might promote it on things like Facebook and YouTube,

1025
00:42:13.862 --> 00:42:17.029
Reddit, or Twitter, whatever you want.

1026
00:42:18.257 --> 00:42:20.878
There's a lot of related work,

1027
00:42:20.878 --> 00:42:22.861
but I'll skip it in the interest of time

1028
00:42:22.861 --> 00:42:25.142
except to say that this work was heavily influenced

1029
00:42:25.142 --> 00:42:27.472
by the work of Gene Traver.

1030
00:42:27.472 --> 00:42:29.086
I don't know if people noticed,

1031
00:42:29.086 --> 00:42:31.302
but these are two Gene Traver references.

1032
00:42:31.302 --> 00:42:34.158
Those people were looking at how do,

1033
00:42:34.158 --> 00:42:37.669
sort of things like quotes propagate through the news.

1034
00:42:37.669 --> 00:42:39.975
Who picks it up, was it a blogger who was first,

1035
00:42:39.975 --> 00:42:41.959
was it the mainstream media, questions like that.

1036
00:42:41.959 --> 00:42:45.184
It's really forward, but I will go on.

1037
00:42:45.184 --> 00:42:49.468
Now, a very politicized idea of lipstick on a pig,

1038
00:42:49.468 --> 00:42:51.985
but let's think out of the box.

1039
00:42:51.985 --> 00:42:54.581
You know, I need more types of data,

1040
00:42:54.581 --> 00:42:56.786
more examples of things that could go that way.

1041
00:42:56.786 --> 00:43:00.845
These type of data are made for other adults

1042
00:43:00.845 --> 00:43:02.816
because I'm just not brave enough.

1043
00:43:02.816 --> 00:43:04.719
It was far too much to analyze this kind of thing

1044
00:43:04.719 --> 00:43:07.196
at the scale that you want lots and lots

1045
00:43:07.196 --> 00:43:08.364
and lots of examples.

1046
00:43:08.364 --> 00:43:11.431
Also, you need to figure out what counts as memorable.

1047
00:43:11.431 --> 00:43:12.977
It turns out it's very hard.

1048
00:43:12.977 --> 00:43:15.045
You can't just look at how many times something

1049
00:43:15.045 --> 00:43:16.140
show up on the web.

1050
00:43:16.140 --> 00:43:17.250
Because things like news wire

1051
00:43:17.250 --> 00:43:20.232
just get picked up and pounded on everybody.

1052
00:43:20.232 --> 00:43:22.274
So you have these artificially inflated counts.

1053
00:43:22.274 --> 00:43:24.076
It doesn't mean it's something memorable,

1054
00:43:24.076 --> 00:43:27.159
it just means everyone is copying it.

1055
00:43:28.159 --> 00:43:30.474
And then the big, let me list it, 64,000,

1056
00:43:30.474 --> 00:43:32.344
I don't even know, the big question is,

1057
00:43:32.344 --> 00:43:33.989
how do you tease apart the language effect

1058
00:43:33.989 --> 00:43:36.147
from the context effect, if it is possible.

1059
00:43:36.147 --> 00:43:39.147
Some pretty big problems to address.

1060
00:43:41.910 --> 00:43:43.735
So looking at the data,

1061
00:43:43.735 --> 00:43:47.277
so one thing like, I'm actually not sure who it was,

1062
00:43:47.277 --> 00:43:49.439
is that a movie related to politics,

1063
00:43:49.439 --> 00:43:52.417
so there's something else floating around there

1064
00:43:52.417 --> 00:43:54.573
where people care about the memorability of them,

1065
00:43:54.573 --> 00:43:55.790
and that's movie quotes.

1066
00:43:55.790 --> 00:43:57.373
So Americans, at least,

1067
00:43:57.373 --> 00:43:59.159
care so much about famous movie quotes

1068
00:43:59.159 --> 00:44:02.709
that there was a three hour television special on CBS

1069
00:44:02.709 --> 00:44:05.503
devoted simply to what are the 100 most famous movie quotes

1070
00:44:05.503 --> 00:44:07.070
of all time.

1071
00:44:07.070 --> 00:44:08.725
And you've probably heard someone use some

1072
00:44:08.725 --> 00:44:10.549
in the last week or so, right,

1073
00:44:10.549 --> 00:44:12.128
people love talking about them.

1074
00:44:12.128 --> 00:44:13.958
And there's sort of something nice,

1075
00:44:13.958 --> 00:44:15.832
really points out this cultural penetration

1076
00:44:15.832 --> 00:44:17.968
that political quotes don't necessarily have.

1077
00:44:17.968 --> 00:44:20.471
You know, lipstick on a pig, you know,

1078
00:44:20.471 --> 00:44:22.182
a certain percent don't.

1079
00:44:22.182 --> 00:44:23.479
But we all know

1080
00:44:23.479 --> 00:44:25.923
I'm gonna make you an offer you can't refuse.

1081
00:44:25.923 --> 00:44:27.670
So you can have this kind of even broader test bin

1082
00:44:27.670 --> 00:44:30.863
of things that you know, you're testing

1083
00:44:30.863 --> 00:44:34.019
when they entered the sort of cultural conversation or not.

1084
00:44:34.019 --> 00:44:35.286
So cool.

1085
00:44:35.286 --> 00:44:37.604
Now, how are you choosing stuff

1086
00:44:37.604 --> 00:44:39.359
and how are the contexts standardized.

1087
00:44:39.359 --> 00:44:42.776
So the audience is seeing Star Wars four,

1088
00:44:43.765 --> 00:44:46.994
this is a scene from it, that's Obi-Wan Kenobi

1089
00:44:46.994 --> 00:44:49.121
and he's doing this kind of Jedi mind trick

1090
00:44:49.121 --> 00:44:50.908
to the stormtrooper.

1091
00:44:50.908 --> 00:44:53.656
So he's saying you don't need to see his identification.

1092
00:44:53.656 --> 00:44:55.451
And then the stormtrooper repeats,

1093
00:44:55.451 --> 00:44:57.368
you can see who's in power here.

1094
00:44:57.368 --> 00:45:01.660
These aren't the droids you're looking for.

1095
00:45:01.660 --> 00:45:05.158
He says they can just move along.

1096
00:45:05.158 --> 00:45:06.959
First of all, that's my Jedi mind trick

1097
00:45:06.959 --> 00:45:08.459
role playing game.

1098
00:45:10.580 --> 00:45:12.543
Okay, which of these lines,

1099
00:45:12.543 --> 00:45:14.406
one of these lines went viral,

1100
00:45:14.406 --> 00:45:15.990
the other three did not.

1101
00:45:15.990 --> 00:45:16.990
Which is it?

1102
00:45:19.001 --> 00:45:22.001
Yeah, so it became just a huge thing,

1103
00:45:22.001 --> 00:45:23.676
everyone knows this line.

1104
00:45:23.676 --> 00:45:25.926
So here is just some visual memes.

1105
00:45:25.926 --> 00:45:27.837
Like here is a little baby stormtrooper

1106
00:45:27.837 --> 00:45:29.663
and she has her little doll here,

1107
00:45:29.663 --> 00:45:32.311
bye daddy, I hope you find the droids you're looking for.

1108
00:45:32.311 --> 00:45:34.168
Or here when he's searching,

1109
00:45:34.168 --> 00:45:36.793
he's searching on Google for the droids

1110
00:45:36.793 --> 00:45:38.777
they were looking for.

1111
00:45:38.777 --> 00:45:40.589
Okay, my personal favorite.

1112
00:45:40.589 --> 00:45:43.132
We do other things besides look for droids,

1113
00:45:43.132 --> 00:45:46.201
but that's all anyone ever remembers.

1114
00:45:46.201 --> 00:45:47.034
Poor guy.

1115
00:45:49.781 --> 00:45:53.979
This wouldn't be funny if that line hadn't gone viral.

1116
00:45:53.979 --> 00:45:56.000
You would never see him saying we do other things

1117
00:45:56.000 --> 00:45:58.961
besides look for their identifications.

1118
00:45:58.961 --> 00:46:03.052
So now, why did I tell you about this setup?

1119
00:46:03.052 --> 00:46:04.687
It's because notice that

1120
00:46:04.687 --> 00:46:06.470
in terms of controlling for things,

1121
00:46:06.470 --> 00:46:08.784
this is a lot of lines happening in the same movie.

1122
00:46:08.784 --> 00:46:11.427
They're all said by the same speaker,

1123
00:46:11.427 --> 00:46:13.664
all basically at the same time.

1124
00:46:13.664 --> 00:46:15.995
So we're controlling for things like different speaker,

1125
00:46:15.995 --> 00:46:17.801
different context, things like that.

1126
00:46:17.801 --> 00:46:20.178
Why did it become so one of them goes viral

1127
00:46:20.178 --> 00:46:21.673
and one of them doesn't.

1128
00:46:21.673 --> 00:46:23.504
So you can do this kind of pairing.

1129
00:46:23.504 --> 00:46:25.321
So that's where we're gonna get our data.

1130
00:46:25.321 --> 00:46:29.204
We're going to take movie scripts from the web.

1131
00:46:29.204 --> 00:46:31.910
And again, this data is available on my home page.

1132
00:46:31.910 --> 00:46:34.303
You can take the lines that people on IMDB

1133
00:46:34.303 --> 00:46:35.780
mark as memorable

1134
00:46:35.780 --> 00:46:38.120
and pair them with something that's close,

1135
00:46:38.120 --> 00:46:40.734
about the same length, said by the same speaker,

1136
00:46:40.734 --> 00:46:43.173
and that was not said to be memorable on IMDB.

1137
00:46:43.173 --> 00:46:46.653
And we can check also some google and bing counts

1138
00:46:46.653 --> 00:46:48.912
so that you can get this data.

1139
00:46:48.912 --> 00:46:50.871
Okay, so that's gonna be our test data.

1140
00:46:50.871 --> 00:46:52.650
It's memorable versus non-memorable pairing.

1141
00:46:52.650 --> 00:46:54.276
We tried to control as much as possible

1142
00:46:54.276 --> 00:46:55.982
for all these other possible factors besides the length.

1143
00:46:55.982 --> 00:46:58.958
So let me show you what these things look like,

1144
00:46:58.958 --> 00:47:00.530
they look like this.

1145
00:47:00.530 --> 00:47:03.310
So for our pilot study, we have viewers

1146
00:47:03.310 --> 00:47:05.426
look at stuff here.

1147
00:47:05.426 --> 00:47:08.212
And importantly, they have to be movies they haven't seen.

1148
00:47:08.212 --> 00:47:10.603
The idea is you only want them to see the language.

1149
00:47:10.603 --> 00:47:12.910
You don't know if anything memorable is going on.

1150
00:47:12.910 --> 00:47:15.864
So we've gotta strip it down to just the language.

1151
00:47:15.864 --> 00:47:17.697
So let's do a quick test here.

1152
00:47:17.697 --> 00:47:19.760
And I will let you take the quiz yourself.

1153
00:47:19.760 --> 00:47:22.709
These are two quotes from Jackie Brown.

1154
00:47:22.709 --> 00:47:24.661
Left hand, right hand.

1155
00:47:24.661 --> 00:47:25.839
How many people say the left hand

1156
00:47:25.839 --> 00:47:28.369
was the one that was memorable?

1157
00:47:28.369 --> 00:47:30.301
I gotta see you.

1158
00:47:30.301 --> 00:47:32.868
How many the right hand?

1159
00:47:32.868 --> 00:47:33.701
Only one?

1160
00:47:35.070 --> 00:47:35.903
Really?

1161
00:47:36.764 --> 00:47:38.640
All right, let's hope that you,

1162
00:47:38.640 --> 00:47:42.057
let's take the second one, how many left?

1163
00:47:43.301 --> 00:47:44.551
How many right?

1164
00:47:46.273 --> 00:47:47.880
Not many more to the right.

1165
00:47:47.880 --> 00:47:51.412
Okay, ordinary people, to the left?

1166
00:47:51.412 --> 00:47:52.495
To the right?

1167
00:47:55.134 --> 00:47:57.830
A lot of people stopped playing.

1168
00:47:57.830 --> 00:47:58.921
That's right, right?

1169
00:47:58.921 --> 00:48:01.463
Because you're either bored, which is okay,

1170
00:48:01.463 --> 00:48:04.989
that's fine, but it's also kind of hard.

1171
00:48:04.989 --> 00:48:06.526
I don't think it's probably,

1172
00:48:06.526 --> 00:48:08.041
there may be some people here

1173
00:48:08.041 --> 00:48:09.792
who think they are absolutely right each time.

1174
00:48:09.792 --> 00:48:11.683
Like, you'll notice when we insert the crowd

1175
00:48:11.683 --> 00:48:15.329
that you can't all be right because you disagree.

1176
00:48:15.329 --> 00:48:17.840
So one of the points I wanna get across here

1177
00:48:17.840 --> 00:48:20.084
is that it's not that easy.

1178
00:48:20.084 --> 00:48:22.531
On many of them, many of you do have a feeling

1179
00:48:22.531 --> 00:48:24.276
that there is something there,

1180
00:48:24.276 --> 00:48:26.077
I can kind of make that guess.

1181
00:48:26.077 --> 00:48:29.616
<v Man>Do you account how they vocalize these things</v>

1182
00:48:29.616 --> 00:48:30.905
when they hear them?

1183
00:48:30.905 --> 00:48:34.988
(dialogue drowned out by static)

1184
00:48:40.088 --> 00:48:41.340
<v ->If you remember them.</v>

1185
00:48:41.340 --> 00:48:42.834
So it's key that we're asking about

1186
00:48:42.834 --> 00:48:44.427
things you haven't seen before.

1187
00:48:44.427 --> 00:48:46.217
And you'll notice these quotes are Not so famous

1188
00:48:46.217 --> 00:48:49.001
that you've probably heard it out in the street.

1189
00:48:49.001 --> 00:48:50.628
I had never heard someone saying no cold feet

1190
00:48:50.628 --> 00:48:52.635
or any other parts of our anatomy.

1191
00:48:52.635 --> 00:48:55.250
So it's not an everyday thing.

1192
00:48:55.250 --> 00:49:00.232
There could be a thing, but I'm gonna count that out.

1193
00:49:00.232 --> 00:49:02.463
Okay, so I don't even have to tell you

1194
00:49:02.463 --> 00:49:03.554
what the right answer is

1195
00:49:03.554 --> 00:49:05.567
because all I wanted to get at is the notion

1196
00:49:05.567 --> 00:49:06.861
that it's kind of hard,

1197
00:49:06.861 --> 00:49:08.554
but you kind of have a feeling for it.

1198
00:49:08.554 --> 00:49:10.255
Okay, so how do people do it?

1199
00:49:10.255 --> 00:49:12.014
Let me talk about what the possibilities were.

1200
00:49:12.014 --> 00:49:13.568
People could've done 50%,

1201
00:49:13.568 --> 00:49:15.392
that would've been their advantage, yes,

1202
00:49:15.392 --> 00:49:17.333
which would mean it's impossible,

1203
00:49:17.333 --> 00:49:19.069
you cannot tell just from the language

1204
00:49:19.069 --> 00:49:21.319
which is gonna be the line.

1205
00:49:23.515 --> 00:49:26.618
What does 100% mean, if people were here,

1206
00:49:26.618 --> 00:49:29.970
it would say it's only the language, that's all you need

1207
00:49:29.970 --> 00:49:32.947
and you can predict what's gonna go viral.

1208
00:49:32.947 --> 00:49:35.867
Where did they fall, here.

1209
00:49:35.867 --> 00:49:38.054
Which is a really nice spot,

1210
00:49:38.054 --> 00:49:40.084
which I've already labeled as interesting.

1211
00:49:40.084 --> 00:49:42.336
What does it mean?

1212
00:49:42.336 --> 00:49:44.891
So if you're here, it means it's not trivial,

1213
00:49:44.891 --> 00:49:46.541
so it's not all language, which we know

1214
00:49:46.541 --> 00:49:49.190
since it's already a certain percent.

1215
00:49:49.190 --> 00:49:51.929
When Arnold Schwarzenegger says I'll be back,

1216
00:49:51.929 --> 00:49:53.807
you know, people remember this.

1217
00:49:53.807 --> 00:49:56.640
I've said I'll be back many times.

1218
00:49:57.997 --> 00:50:01.616
So it's not, right, so there's lots of stuff

1219
00:50:01.616 --> 00:50:03.564
other than the language.

1220
00:50:03.564 --> 00:50:05.638
On the other hand, there is a language effect,

1221
00:50:05.638 --> 00:50:07.112
it's not impossible.

1222
00:50:07.112 --> 00:50:09.000
But I think this is one of the key findings

1223
00:50:09.000 --> 00:50:10.450
that I'm very happy about, yeah.

1224
00:50:10.450 --> 00:50:12.670
<v Man>But didn't you sort of argue</v>

1225
00:50:12.670 --> 00:50:13.994
in the first part of the talk

1226
00:50:13.994 --> 00:50:16.037
that there's a lot of unconscious stuff

1227
00:50:16.037 --> 00:50:18.197
that we wouldn't necessarily pick up on.

1228
00:50:18.197 --> 00:50:20.031
So clearly it'd be a very hard task,

1229
00:50:20.031 --> 00:50:21.959
but it's still possible.

1230
00:50:21.959 --> 00:50:23.880
<v ->Yeah, so I'm just trying to argue</v>

1231
00:50:23.880 --> 00:50:25.421
that there is something in the language,

1232
00:50:25.421 --> 00:50:26.829
something is able to,

1233
00:50:26.829 --> 00:50:28.856
maybe we don't know that we're picking or something.

1234
00:50:28.856 --> 00:50:31.528
But remember the counter hypothesis is just

1235
00:50:31.528 --> 00:50:34.803
is it Schwarzenegger who says it or me?

1236
00:50:34.803 --> 00:50:36.720
I'm gonna frighten you.

1237
00:50:37.607 --> 00:50:39.764
Anyway, it's all right here.

1238
00:50:39.764 --> 00:50:41.525
Now, what's this dash.

1239
00:50:41.525 --> 00:50:43.478
It doesn't mean I can't count, okay,

1240
00:50:43.478 --> 00:50:45.446
although I do have trouble some days.

1241
00:50:45.446 --> 00:50:46.986
We ran this twice.

1242
00:50:46.986 --> 00:50:49.501
So this first number is one of the very controlled settings,

1243
00:50:49.501 --> 00:50:52.063
we knew that the people involved were the subjects,

1244
00:50:52.063 --> 00:50:53.089
et cetera.

1245
00:50:53.089 --> 00:50:57.013
After this paper got out, it itself went viral,

1246
00:50:57.013 --> 00:50:59.300
just some kind of meta-commentary.

1247
00:50:59.300 --> 00:51:02.700
So I have to say, although many good things

1248
00:51:02.700 --> 00:51:05.038
have happened to me, the highlight of my professional life

1249
00:51:05.038 --> 00:51:08.005
is we were made fun of Kathy on the Today Show.

1250
00:51:08.005 --> 00:51:12.172
So about 15,000 people ended up visiting this site.

1251
00:51:13.368 --> 00:51:17.357
And the average accuracy was 72%.

1252
00:51:17.357 --> 00:51:19.857
So I think we've got a winner.

1253
00:51:21.818 --> 00:51:24.285
All right, so now I hope alleviates

1254
00:51:24.285 --> 00:51:26.165
that there is something in the language

1255
00:51:26.165 --> 00:51:27.010
that can help you tell

1256
00:51:27.010 --> 00:51:28.331
whether something is gonna be viral or not,

1257
00:51:28.331 --> 00:51:29.934
although it's kind of hard to do.

1258
00:51:29.934 --> 00:51:32.827
We certainly didn't see certainty.

1259
00:51:32.827 --> 00:51:35.698
What are some things that one could look for?

1260
00:51:35.698 --> 00:51:38.858
So let me say off hand that we see some feature,

1261
00:51:38.858 --> 00:51:40.788
I mean, we'll talk about some features

1262
00:51:40.788 --> 00:51:42.421
which are intuitively plausible.

1263
00:51:42.421 --> 00:51:44.406
But I think probably if you guys worked on this hard enough,

1264
00:51:44.406 --> 00:51:46.618
you'd probably find features that we didn't.

1265
00:51:46.618 --> 00:51:48.725
That we just really,

1266
00:51:48.725 --> 00:51:52.475
well, any of them, they're totally all there.

1267
00:51:53.736 --> 00:51:55.882
So I'm not saying these are the features

1268
00:51:55.882 --> 00:51:59.347
or anything like that, but they're intuitively plausible

1269
00:51:59.347 --> 00:52:02.576
and I hope they provide evidence.

1270
00:52:02.576 --> 00:52:05.354
So one, on average if you compare the memorable quotes

1271
00:52:05.354 --> 00:52:07.203
versus their non-memorable twins,

1272
00:52:07.203 --> 00:52:09.125
we see that the memorable ones

1273
00:52:09.125 --> 00:52:11.383
tend to contain more surprising combinations of words,

1274
00:52:11.383 --> 00:52:14.323
as measured by you take the Brown method

1275
00:52:14.323 --> 00:52:16.696
and you look at the probabilities.

1276
00:52:16.696 --> 00:52:18.305
Just giving into your intuition,

1277
00:52:18.305 --> 00:52:20.906
so these aren't the droids you're looking for,

1278
00:52:20.906 --> 00:52:22.301
that's an unusual word.

1279
00:52:22.301 --> 00:52:26.019
It probably actually does occur in the realm of this,

1280
00:52:26.019 --> 00:52:27.205
anyway.

1281
00:52:27.205 --> 00:52:29.028
But although on the lexical level,

1282
00:52:29.028 --> 00:52:31.621
we've had some distinctiveness, something interesting

1283
00:52:31.621 --> 00:52:35.128
is that if you look at the part of speech sequences,

1284
00:52:35.128 --> 00:52:37.135
and you go along based on that,

1285
00:52:37.135 --> 00:52:40.794
we found that the memorable quotes tend to be more common

1286
00:52:40.794 --> 00:52:43.296
and more likely than a non-memorable quote

1287
00:52:43.296 --> 00:52:44.977
with respect to the part of speeches.

1288
00:52:44.977 --> 00:52:48.147
Okay, and you can read this and sort of say that,

1289
00:52:48.147 --> 00:52:49.927
also, here's an example here.

1290
00:52:49.927 --> 00:52:52.878
From Jaws, you're gonna need a bigger boat.

1291
00:52:52.878 --> 00:52:56.780
Compared to this, this just seems like a less memorable

1292
00:52:56.780 --> 00:52:59.499
way to say it, you're gonna need a boat that is bigger.

1293
00:52:59.499 --> 00:53:01.346
Syntax is a little weird.

1294
00:53:01.346 --> 00:53:03.572
So you know, you can read this slide as

1295
00:53:03.572 --> 00:53:05.865
memorable quotes have more surprising words

1296
00:53:05.865 --> 00:53:07.609
so that they're memorable,

1297
00:53:07.609 --> 00:53:09.632
but they have a more common syntactic scaffolding

1298
00:53:09.632 --> 00:53:11.905
so that they're easier to remember.

1299
00:53:11.905 --> 00:53:13.521
Let's adjust this, you know.

1300
00:53:13.521 --> 00:53:16.722
There's a lot of things we found,

1301
00:53:16.722 --> 00:53:19.837
I think I'm gonna skip them for the purposes of time.

1302
00:53:19.837 --> 00:53:22.752
But just to show you, again, to get an ACL,

1303
00:53:22.752 --> 00:53:25.433
we need prediction results, here they are.

1304
00:53:25.433 --> 00:53:28.606
So this time, random choice baseline of 50%.

1305
00:53:28.606 --> 00:53:31.172
You get to the data as you get too close,

1306
00:53:31.172 --> 00:53:33.772
you have to decide which one it is.

1307
00:53:33.772 --> 00:53:38.337
So here is bag of words doing a low chance at 60%.

1308
00:53:38.337 --> 00:53:41.268
But here is this distinctiveness feature.

1309
00:53:41.268 --> 00:53:43.034
So you've got 900 we did,

1310
00:53:43.034 --> 00:53:46.146
so that you have about 900 features for bag of words

1311
00:53:46.146 --> 00:53:48.509
versus just this distinctiveness feature.

1312
00:53:48.509 --> 00:53:52.592
(dialogue drowned out by static)

1313
00:53:53.822 --> 00:53:55.323
And somehow, our features are better

1314
00:53:55.323 --> 00:53:58.546
than bag of words, this is statistically significant.

1315
00:53:58.546 --> 00:54:00.838
And so this is where the human performance is.

1316
00:54:00.838 --> 00:54:03.503
So you can see we're falling.

1317
00:54:03.503 --> 00:54:06.999
This is this generality feature that I skipped.

1318
00:54:06.999 --> 00:54:08.854
I know there are people

1319
00:54:08.854 --> 00:54:10.925
who do information visualization here, so don't laugh.

1320
00:54:10.925 --> 00:54:13.988
But these stripes match this

1321
00:54:13.988 --> 00:54:15.866
and then the other stripes.

1322
00:54:15.866 --> 00:54:17.066
(laughter)

1323
00:54:17.066 --> 00:54:18.594
For some reason, everyone thinks twice is funny,

1324
00:54:18.594 --> 00:54:20.375
I'm not sure why.

1325
00:54:20.375 --> 00:54:22.744
But anyway, you find that if you combine the two,

1326
00:54:22.744 --> 00:54:24.887
two great tastes that taste good together,

1327
00:54:24.887 --> 00:54:27.331
you get your distance taking your performance,

1328
00:54:27.331 --> 00:54:31.795
how is it a statistically significant improvement.

1329
00:54:31.795 --> 00:54:35.828
Okay, real quick, you can do this to other things,

1330
00:54:35.828 --> 00:54:38.928
not just movie quotes, but also ad slogans

1331
00:54:38.928 --> 00:54:40.636
where people care very much

1332
00:54:40.636 --> 00:54:43.222
about what they say becoming memorable.

1333
00:54:43.222 --> 00:54:45.366
You can use our features to rate ad slogans.

1334
00:54:45.366 --> 00:54:48.687
And the number one that came out of the stuff that we had

1335
00:54:48.687 --> 00:54:52.290
was from Levi's, quality never goes out of style.

1336
00:54:52.290 --> 00:54:53.290
Pretty good.

1337
00:54:54.182 --> 00:54:56.551
Number two is come to Marlboro country.

1338
00:54:56.551 --> 00:54:57.945
There's that.

1339
00:54:57.945 --> 00:54:59.245
The first one, I have to look at my screen

1340
00:54:59.245 --> 00:55:01.201
because I can never remember it.

1341
00:55:01.201 --> 00:55:04.951
It's from Jaguar, it says grace, space, pace.

1342
00:55:08.878 --> 00:55:12.250
As they say, who was the marketing genius?

1343
00:55:12.250 --> 00:55:16.342
The second work was born and culled primal.

1344
00:55:16.342 --> 00:55:19.937
So I think our features are doing somewhat of a good job

1345
00:55:19.937 --> 00:55:21.871
of actually ranking these ad slogans.

1346
00:55:21.871 --> 00:55:24.477
I'm not gonna go too far down that road.

1347
00:55:24.477 --> 00:55:28.326
But just to give you an idea, this idea.

1348
00:55:28.326 --> 00:55:30.341
Another quick thing,

1349
00:55:30.341 --> 00:55:32.573
you can use this idea of distinctiveness

1350
00:55:32.573 --> 00:55:34.158
in totally other studies.

1351
00:55:34.158 --> 00:55:36.155
So we have this collaboration with Facebook,

1352
00:55:36.155 --> 00:55:38.186
this is just indicative of--

1353
00:55:38.186 --> 00:55:39.622
(sneeze)

1354
00:55:39.622 --> 00:55:42.673
(dialogue drowned out by static)

1355
00:55:42.673 --> 00:55:44.278
Here's the quick idea.

1356
00:55:44.278 --> 00:55:46.982
You can talk about the distinctiveness of a post

1357
00:55:46.982 --> 00:55:50.021
and how many comments does that post get.

1358
00:55:50.021 --> 00:55:53.261
And you can see that there's definitely, in Facebook,

1359
00:55:53.261 --> 00:55:55.180
an interesting correlation.

1360
00:55:55.180 --> 00:55:57.276
The more distinctive the text is,

1361
00:55:57.276 --> 00:56:00.939
a subject in certain conditions, which are here,

1362
00:56:00.939 --> 00:56:02.992
you tend to get more comments.

1363
00:56:02.992 --> 00:56:05.270
And that's if you get any comments at all,

1364
00:56:05.270 --> 00:56:07.155
whereas you don't see this at all on Wikipedia,

1365
00:56:07.155 --> 00:56:08.887
which I think also makes sense

1366
00:56:08.887 --> 00:56:11.055
because it's a more task based kind of thing.

1367
00:56:11.055 --> 00:56:15.191
You can answer hey, we need to delete this page.

1368
00:56:15.191 --> 00:56:16.856
But there are some applications here, too,

1369
00:56:16.856 --> 00:56:21.023
obviously ranking and social media, things like that.

1370
00:56:22.859 --> 00:56:25.930
Okay, so now we've got to here,

1371
00:56:25.930 --> 00:56:29.543
I want to close with this quote by William Carlos Williams,

1372
00:56:29.543 --> 00:56:31.088
a really good poet.

1373
00:56:31.088 --> 00:56:35.171
(dialogue drowned out by static)

1374
00:56:37.475 --> 00:56:39.298
He said the following.

1375
00:56:39.298 --> 00:56:41.371
I had not yet come to the conviction:

1376
00:56:41.371 --> 00:56:43.633
it is not what you say that matters,

1377
00:56:43.633 --> 00:56:45.579
but the manner in which you say it.

1378
00:56:45.579 --> 00:56:48.439
There lies the secret of the ages.

1379
00:56:48.439 --> 00:56:51.351
I have not, myself, necessarily come to this conviction,

1380
00:56:51.351 --> 00:56:53.295
but I have come into conclusions,

1381
00:56:53.295 --> 00:56:55.006
which is that social power relationships,

1382
00:56:55.006 --> 00:56:56.565
I think, is one body.

1383
00:56:56.565 --> 00:56:58.084
These affect language

1384
00:56:58.084 --> 00:56:59.907
even at the extremely fine-grained levels

1385
00:56:59.907 --> 00:57:01.321
that was saw in case study one.

1386
00:57:01.321 --> 00:57:03.341
And also, I think that we've shown that phrasing

1387
00:57:03.341 --> 00:57:05.501
does indeed have some effect on memorability,

1388
00:57:05.501 --> 00:57:07.321
on Facebook and stuff, and how immediate

1389
00:57:07.321 --> 00:57:09.918
a piece of language or text is.

1390
00:57:09.918 --> 00:57:11.397
Why do we care?

1391
00:57:11.397 --> 00:57:14.392
I think that it improves computational techniques

1392
00:57:14.392 --> 00:57:16.854
for these sort of socio-computational systems

1393
00:57:16.854 --> 00:57:19.991
that are in important direction for research.

1394
00:57:19.991 --> 00:57:22.971
I personally view more interplay between

1395
00:57:22.971 --> 00:57:25.916
constructing NLP-enabled systems,

1396
00:57:25.916 --> 00:57:28.231
things like Facebook, which aren't NLP systems per se,

1397
00:57:28.231 --> 00:57:30.906
but have NLP systems as a component,

1398
00:57:30.906 --> 00:57:33.591
interactions between these and how people use

1399
00:57:33.591 --> 00:57:35.137
and are affected language,

1400
00:57:35.137 --> 00:57:37.904
I think this is a huge opportunity for all concerned.

1401
00:57:37.904 --> 00:57:40.168
Now, I will totally amend

1402
00:57:40.168 --> 00:57:42.794
that this is maybe not so memorably phrased.

1403
00:57:42.794 --> 00:57:44.973
So here is another quote.

1404
00:57:44.973 --> 00:57:47.965
I think this is the beginning of a beautiful friendship.

1405
00:57:47.965 --> 00:57:50.215
(applause)

1406
00:57:57.587 --> 00:57:59.087
<v Man>Questions?</v>

1407
00:58:00.780 --> 00:58:01.982
<v Woman>So for memorability,</v>

1408
00:58:01.982 --> 00:58:03.589
do you think we'd also see the same kind of thing

1409
00:58:03.589 --> 00:58:05.074
for spontaneous things

1410
00:58:05.074 --> 00:58:08.491
rather than just things that are written?

1411
00:58:09.455 --> 00:58:10.466
<v ->I think so.</v>

1412
00:58:10.466 --> 00:58:14.676
I think, again, memes, lots of people repeat.

1413
00:58:14.676 --> 00:58:18.523
There's a certain phrase people sometimes use.

1414
00:58:18.523 --> 00:58:22.606
(dialogue drowned out by static)

1415
00:58:25.061 --> 00:58:26.596
Yeah, I think so.

1416
00:58:26.596 --> 00:58:27.679
I just asked.

1417
00:58:31.906 --> 00:58:35.989
(dialogue drowned out by static)

1418
00:58:48.162 --> 00:58:50.181
So we went through other features as well.

1419
00:58:50.181 --> 00:58:53.265
So the statistics, there's a lot of story of that.

1420
00:58:53.265 --> 00:58:57.348
(dialogue drowned out by static)

1421
00:59:00.025 --> 00:59:01.249
And if you guys know

1422
00:59:01.249 --> 00:59:03.239
how to conversationally characterize women,

1423
00:59:03.239 --> 00:59:04.906
that would be great.

1424
00:59:06.023 --> 00:59:08.002
So like I said, I'm not saying

1425
00:59:08.002 --> 00:59:10.182
that these features are the grandest ingredients.

1426
00:59:10.182 --> 00:59:13.691
(dialogue drowned out by static)

1427
00:59:13.691 --> 00:59:15.245
However, B, the second word.

1428
00:59:15.245 --> 00:59:17.767
I don't know, but I think,

1429
00:59:17.767 --> 00:59:20.786
it's the reason I showed you different processes.

1430
00:59:20.786 --> 00:59:22.690
You can just go see applications,

1431
00:59:22.690 --> 00:59:25.010
even if it's not...

1432
00:59:25.010 --> 00:59:29.344
(dialogue drowned out by static)

1433
00:59:29.344 --> 00:59:30.826
<v Man>All right, thanks for a wonderful talk,</v>

1434
00:59:30.826 --> 00:59:32.741
I mean, it was really impressive how you

1435
00:59:32.741 --> 00:59:35.149
braved or teased out these results

1436
00:59:35.149 --> 00:59:37.373
and really showed a strong effect.

1437
00:59:37.373 --> 00:59:41.860
But I'm not sure they provide actionable guidance for me.

1438
00:59:41.860 --> 00:59:44.069
I want to make a great phrase,

1439
00:59:44.069 --> 00:59:47.981
I'm not sure I'm gonna work it out that way.

1440
00:59:47.981 --> 00:59:51.974
I've been reading a book called words that work

1441
00:59:51.974 --> 00:59:54.094
and it's making it stick and guidance

1442
00:59:54.094 --> 00:59:57.104
and those provide much more explicit rules

1443
00:59:57.104 --> 00:59:59.083
for making memorable phrases.

1444
00:59:59.083 --> 01:00:01.391
Have you tested the principles of that?

1445
01:00:01.391 --> 01:00:03.544
<v ->No, so my student body, I haven't really decided.</v>

1446
01:00:03.544 --> 01:00:07.745
I definitely, maybe if somebody wanted me,

1447
01:00:07.745 --> 01:00:10.912
but you know, there is this creativity

1448
01:00:11.793 --> 01:00:14.388
that I think is an important component.

1449
01:00:14.388 --> 01:00:17.053
And then we have never made any plans that this is,

1450
01:00:17.053 --> 01:00:20.127
that we're gonna write conscriptive versions.

1451
01:00:20.127 --> 01:00:22.123
Other people get out and do that.

1452
01:00:22.123 --> 01:00:24.373
<v Man>Right, I mean, it's sort of a bumper sticker,</v>

1453
01:00:24.373 --> 01:00:27.378
or using alliteration or other things.

1454
01:00:27.378 --> 01:00:29.573
<v ->So we've been thinking about doing some of this,</v>

1455
01:00:29.573 --> 01:00:32.252
and I'm sort of two minds about it.

1456
01:00:32.252 --> 01:00:34.333
In some sense, I feel like the point that's saying

1457
01:00:34.333 --> 01:00:36.574
there is a language effect was the key thing.

1458
01:00:36.574 --> 01:00:38.699
It could've been, you know, that thing,

1459
01:00:38.699 --> 01:00:40.394
it could've been there was no language effect.

1460
01:00:40.394 --> 01:00:41.790
Once you see there's a language effect,

1461
01:00:41.790 --> 01:00:43.125
then I sort of welcome the community

1462
01:00:43.125 --> 01:00:44.931
to kind of go in there and figure out

1463
01:00:44.931 --> 01:00:46.173
what the language effect it.

1464
01:00:46.173 --> 01:00:48.174
And I sort of feel like,

1465
01:00:48.174 --> 01:00:51.476
I feel like there was a very important point to be made,

1466
01:00:51.476 --> 01:00:53.347
is there actually a language effect

1467
01:00:53.347 --> 01:00:57.090
that we sort of have that characterizing lots of things

1468
01:00:57.090 --> 01:00:59.090
for better or for worse.

1469
01:01:00.218 --> 01:01:01.718
So we could do it.

1470
01:01:05.733 --> 01:01:08.471
<v Man>I would argue you should do it.</v>

1471
01:01:08.471 --> 01:01:10.304
And I'll tell you why.

1472
01:01:11.785 --> 01:01:15.186
Because there is value in shining a light

1473
01:01:15.186 --> 01:01:19.039
on the unconscious processes that affect people.

1474
01:01:19.039 --> 01:01:21.477
<v Man 2>He's looking at Frank Watson's words that work.</v>

1475
01:01:21.477 --> 01:01:24.379
Where he says to listen to prescriptions

1476
01:01:24.379 --> 01:01:28.196
for manipulating people's perceptions, right?

1477
01:01:28.196 --> 01:01:31.515
And the students in my seminar and I have long had a dream

1478
01:01:31.515 --> 01:01:35.036
(dialogue drowned out by static)

1479
01:01:35.036 --> 01:01:37.500
You have a site where you put a piece of text

1480
01:01:37.500 --> 01:01:40.535
and it has something automatic, Cornell or whatever,

1481
01:01:40.535 --> 01:01:45.129
automatic that automatically shines a spotlight

1482
01:01:45.129 --> 01:01:47.986
on pieces of text and says you know,

1483
01:01:47.986 --> 01:01:49.751
this might be affecting you

1484
01:01:49.751 --> 01:01:52.181
because it's being framed in this way

1485
01:01:52.181 --> 01:01:53.957
or because they are using alliteration

1486
01:01:53.957 --> 01:01:55.153
or they're doing whatever

1487
01:01:55.153 --> 01:01:56.796
so that we can pay attention to the content.

1488
01:01:56.796 --> 01:01:58.753
Because when we see the understanding

1489
01:01:58.753 --> 01:02:00.799
that style has on people,

1490
01:02:00.799 --> 01:02:02.424
and so the people can control for it

1491
01:02:02.424 --> 01:02:04.454
when there's time to make decisions.

1492
01:02:04.454 --> 01:02:07.989
<v ->Although, given my choice between say,</v>

1493
01:02:07.989 --> 01:02:10.017
being a little more eloquent and being taller,

1494
01:02:10.017 --> 01:02:12.254
I might go with being taller.

1495
01:02:12.254 --> 01:02:14.504
(laughter)

1496
01:02:15.591 --> 01:02:16.877
there are these other factors

1497
01:02:16.877 --> 01:02:17.990
that may well matter.

1498
01:02:17.990 --> 01:02:19.586
I mean, there's obviously gender

1499
01:02:19.586 --> 01:02:20.963
and all sorts of things as well.

1500
01:02:20.963 --> 01:02:22.719
I don't know how much of an effect it has.

1501
01:02:22.719 --> 01:02:25.101
So I'm glad that there is a language effect.

1502
01:02:25.101 --> 01:02:28.047
I'm not sure, maybe I'm just kind of,

1503
01:02:28.047 --> 01:02:30.377
just kind of tested by my own laziness.

1504
01:02:30.377 --> 01:02:31.852
<v Man>Go for both.</v>

1505
01:02:31.852 --> 01:02:32.753
<v ->What's that?</v>

1506
01:02:32.753 --> 01:02:34.711
<v ->Go for both.
<v ->Go for both.</v>

1507
01:02:34.711 --> 01:02:36.211
That's true, right?

1508
01:02:36.211 --> 01:02:38.544
You know, I'm wearing heels.

1509
01:02:42.564 --> 01:02:46.647
(dialogue drowned out by static)

1510
01:02:50.322 --> 01:02:52.417
Right, so that's the feature I skipped,

1511
01:02:52.417 --> 01:02:53.667
was generality.

1512
01:02:56.354 --> 01:02:59.744
So the idea that there's a certain definite affableness

1513
01:02:59.744 --> 01:03:02.186
that life is like a box of chocolates.

1514
01:03:02.186 --> 01:03:04.452
There's a certain generalness that applies to many things.

1515
01:03:04.452 --> 01:03:08.528
And quick references like you're gonna need the bigger boat

1516
01:03:08.528 --> 01:03:11.492
versus you're gonna need a bigger boat.

1517
01:03:11.492 --> 01:03:12.968
If you're gonna need the bigger boat,

1518
01:03:12.968 --> 01:03:14.800
you're gonna be like what bigger boat,

1519
01:03:14.800 --> 01:03:16.210
maybe it's talking about a boat.

1520
01:03:16.210 --> 01:03:17.975
So you can't really use that phrase,

1521
01:03:17.975 --> 01:03:19.662
necessarily, so easily.

1522
01:03:19.662 --> 01:03:22.379
Although, they're obviously out there.

1523
01:03:22.379 --> 01:03:24.820
So we do see that there is a correlation

1524
01:03:24.820 --> 01:03:26.064
between our sort of,

1525
01:03:26.064 --> 01:03:30.231
our sort of semi-okay things that kind of get a generality.

1526
01:03:31.421 --> 01:03:35.504
(dialogue drowned out by static)

1527
01:03:42.989 --> 01:03:47.622
Maybe he knows, and all I do is validate it, I don't know.

1528
01:03:47.622 --> 01:03:51.223
This is not the rousing conclusion I thought of.

1529
01:03:51.223 --> 01:03:53.804
(laughter)

1530
01:03:53.804 --> 01:03:56.374
I think this is the beginning of a beautiful friendship.

1531
01:03:56.374 --> 01:03:57.963
Thank you very much.

1532
01:03:57.963 --> 01:04:00.213
(applause)

1533
01:04:03.404 --> 01:04:05.821
(calm music)