WEBVTT
1
00:00:00.779 --> 00:00:03.279
(light music)
2
00:00:11.166 --> 00:00:13.049
It is my pleasure to welcome
3
00:00:13.049 --> 00:00:15.708
Mike Kearn here today.
4
00:00:15.708 --> 00:00:17.654
Michael is a PhD at Harvard.
5
00:00:17.654 --> 00:00:20.571
His speeches won the award from ACM
6
00:00:21.765 --> 00:00:24.390
and was published by MIT Press as a book.
7
00:00:24.390 --> 00:00:27.735
And he consequently did a post-grad at MIT and Berkeley
8
00:00:27.735 --> 00:00:31.859
and then was at Bell Labs for a number of years
9
00:00:31.859 --> 00:00:34.849
before joining U Penn as a professor today.
10
00:00:34.849 --> 00:00:36.862
And he is extremely well known
11
00:00:36.862 --> 00:00:39.061
for his wealth of books concerning computer science
12
00:00:39.061 --> 00:00:42.675
as well AI, learning community, and social networks.
13
00:00:42.675 --> 00:00:44.197
And today he'll talk a little bit
14
00:00:44.197 --> 00:00:45.683
about his new directions,
15
00:00:45.683 --> 00:00:49.350
which are experiments in social computation.
16
00:00:50.491 --> 00:00:52.219
Okay, thank you for coming out
17
00:00:52.219 --> 00:00:54.889
and for the opportunity to speak.
18
00:00:54.889 --> 00:00:57.811
So I'm gonna tell you today about a
19
00:00:57.811 --> 00:00:59.975
somewhat unusual series of experiments
20
00:00:59.975 --> 00:01:01.827
that we've been running in my group
21
00:01:01.827 --> 00:01:03.907
since about 2006.
22
00:01:03.907 --> 00:01:06.278
These are experiments with human subjects,
23
00:01:06.278 --> 00:01:08.635
behavioral experiments on interactions
24
00:01:08.635 --> 00:01:09.763
in social networks.
25
00:01:09.763 --> 00:01:11.583
These are laboratory experiments.
26
00:01:11.583 --> 00:01:13.729
And there's several different ways of viewing
27
00:01:13.729 --> 00:01:14.596
these experiments.
28
00:01:14.596 --> 00:01:16.052
In some sense, they are an instance
29
00:01:16.052 --> 00:01:20.272
of network behavioral game theory and economics.
30
00:01:20.272 --> 00:01:22.623
But the particular angle or narrative
31
00:01:22.623 --> 00:01:24.863
that I'm gonna give today are as experiments
32
00:01:24.863 --> 00:01:26.696
in social computation.
33
00:01:28.704 --> 00:01:29.537
So
34
00:01:31.525 --> 00:01:33.741
I think everybody in the room is probably familiar
35
00:01:33.741 --> 00:01:36.591
with the recent phenomenon of both crowdsourcing
36
00:01:36.591 --> 00:01:38.872
and more generally of social computation
37
00:01:38.872 --> 00:01:40.121
of various kinds.
38
00:01:40.121 --> 00:01:43.422
And a sensible question to ask at this point is
39
00:01:43.422 --> 00:01:45.632
what do computer science and economics
40
00:01:45.632 --> 00:01:47.176
and related areas have to say
41
00:01:47.176 --> 00:01:49.196
about this empirical phenomenon,
42
00:01:49.196 --> 00:01:52.076
and more generally, what would be good design
43
00:01:52.076 --> 00:01:55.741
and engineering principles for building crowdsourcing
44
00:01:55.741 --> 00:01:57.910
or social computation systems?
45
00:01:57.910 --> 00:02:00.460
And if you kind of put your computer science glasses on
46
00:02:00.460 --> 00:02:03.314
and ask this question, I think you would agree with me
47
00:02:03.314 --> 00:02:06.287
that many of the yielded systems to date
48
00:02:06.287 --> 00:02:09.548
are what we might call embarrassingly parallelizable.
49
00:02:09.548 --> 00:02:11.897
That's not to say that they're uninteresting.
50
00:02:11.897 --> 00:02:13.868
They basically tackle problems in which
51
00:02:13.868 --> 00:02:16.281
there's very little or no dependency
52
00:02:16.281 --> 00:02:17.888
between the contributions that different
53
00:02:17.888 --> 00:02:19.217
participants make.
54
00:02:19.217 --> 00:02:21.009
And there are crude if any efforts
55
00:02:21.009 --> 00:02:24.243
to actually organize the reward course in a meaningful way.
56
00:02:24.243 --> 00:02:25.697
So to be more concrete, I have in mind
57
00:02:25.697 --> 00:02:29.192
things like the ESP game or Holden,
58
00:02:29.192 --> 00:02:31.086
the recent pro team polling game
59
00:02:31.086 --> 00:02:33.230
released by the University of Washington.
60
00:02:33.230 --> 00:02:36.227
Things like Galaxy Zoo, where people can go voluntarily
61
00:02:36.227 --> 00:02:39.477
sort of classify stellar constellations
62
00:02:40.326 --> 00:02:42.101
for scientific purposes,
63
00:02:42.101 --> 00:02:43.972
or even things that are much more open-ended,
64
00:02:43.972 --> 00:02:45.940
like Wikipedia, where it's harder to say
65
00:02:45.940 --> 00:02:47.956
what the goal of Wikipedia is
66
00:02:47.956 --> 00:02:51.119
in the sort of objective function sense.
67
00:02:51.119 --> 00:02:53.650
But there's definitely some obvious system
68
00:02:53.650 --> 00:02:57.010
that's going on where people are coming and voluntarily
69
00:02:57.010 --> 00:02:58.009
producing labor.
70
00:02:58.009 --> 00:03:00.124
And when I say embarrassingly parallelizable,
71
00:03:00.124 --> 00:03:02.436
what I mean is that among other things,
72
00:03:02.436 --> 00:03:04.071
some of the commonalities of the examples
73
00:03:04.071 --> 00:03:07.279
I just gave is that whatever the problem is,
74
00:03:07.279 --> 00:03:10.558
it's easily broken into many non-interacting,
75
00:03:10.558 --> 00:03:12.777
independent pieces, which can be given out
76
00:03:12.777 --> 00:03:15.595
to separate individuals or maybe pairs of individuals
77
00:03:15.595 --> 00:03:18.519
in the case of the ESP game.
78
00:03:18.519 --> 00:03:20.956
There's no problem assembling those answers back
79
00:03:20.956 --> 00:03:23.462
into some collective, like you have a big database
80
00:03:23.462 --> 00:03:25.943
of images that you want maybe for content.
81
00:03:25.943 --> 00:03:28.648
Just give those images out to individuals to do the job.
82
00:03:28.648 --> 00:03:32.119
And you might do simple tricks, like get robustness
83
00:03:32.119 --> 00:03:35.957
or error correction by redundancy or both.
84
00:03:35.957 --> 00:03:37.817
But what about harder problems?
85
00:03:37.817 --> 00:03:41.646
What about problems when really you have to coordinate
86
00:03:41.646 --> 00:03:43.168
a larger group of people,
87
00:03:43.168 --> 00:03:46.247
and the decision or contribution or labor
88
00:03:46.247 --> 00:03:49.056
that one party provides really influences
89
00:03:49.056 --> 00:03:51.158
or constrains what happens elsewhere
90
00:03:51.158 --> 00:03:53.539
in the same instance of the problem?
91
00:03:53.539 --> 00:03:54.372
And
92
00:03:55.650 --> 00:03:58.554
as mentioned, my origins were in the theory community.
93
00:03:58.554 --> 00:04:00.873
And so I can't look at these experiments
94
00:04:00.873 --> 00:04:03.017
without thinking about what might a theory
95
00:04:03.017 --> 00:04:04.625
of social computation look like,
96
00:04:04.625 --> 00:04:08.748
where you view human labor as a component
97
00:04:08.748 --> 00:04:11.548
of the system the same way you might view CPU cycles
98
00:04:11.548 --> 00:04:14.141
or primary or secondary memory,
99
00:04:14.141 --> 00:04:16.647
and you want to organize these resources
100
00:04:16.647 --> 00:04:19.082
in some sensible way that serves
101
00:04:19.082 --> 00:04:20.570
to solving some problem.
102
00:04:20.570 --> 00:04:24.245
So what would a theory of social computation look like?
103
00:04:24.245 --> 00:04:28.265
So I raise many good questions on this slide,
104
00:04:28.265 --> 00:04:29.952
and I don't plan to provide answers
105
00:04:29.952 --> 00:04:32.019
to any of them in the talk.
106
00:04:32.019 --> 00:04:34.162
But what I do think the talk contributes
107
00:04:34.162 --> 00:04:38.200
are kind of examples of the kind of phenomena
108
00:04:38.200 --> 00:04:40.828
that any good theory of social computation
109
00:04:40.828 --> 00:04:42.897
will at some point need to address,
110
00:04:42.897 --> 00:04:45.603
either explicitly or implicitly.
111
00:04:45.603 --> 00:04:47.490
So with that, let me tell you what we've been doing
112
00:04:47.490 --> 00:04:48.609
all these years.
113
00:04:48.609 --> 00:04:50.443
So these are human subject experiments
114
00:04:50.443 --> 00:04:53.510
that are basically very interdisciplinary.
115
00:04:53.510 --> 00:04:56.675
You'll see strands of thinking from computer science,
116
00:04:56.675 --> 00:04:59.797
economics and game theory, sociology,
117
00:04:59.797 --> 00:05:02.126
network science running throughout the examples
118
00:05:02.126 --> 00:05:03.714
I'm gonna talk about today.
119
00:05:03.714 --> 00:05:06.567
In these experiments, we simultaneously bring in
120
00:05:06.567 --> 00:05:08.835
groups of roughly three dozen human subjects
121
00:05:08.835 --> 00:05:12.203
into a lab network workstation at the same time.
122
00:05:12.203 --> 00:05:13.951
So these are real-time experiments
123
00:05:13.951 --> 00:05:16.255
with live participation from a group
124
00:05:16.255 --> 00:05:17.844
of about three dozen people.
125
00:05:17.844 --> 00:05:20.074
And in every little experiment,
126
00:05:20.074 --> 00:05:22.272
the experimental sessions are a few hours long,
127
00:05:22.272 --> 00:05:25.096
and a subject will participate in a series
128
00:05:25.096 --> 00:05:26.897
of rather short experiments.
129
00:05:26.897 --> 00:05:29.721
By short, I mean a couple of minutes on average.
130
00:05:29.721 --> 00:05:31.939
And in each experiment, the subject will be controlling
131
00:05:31.939 --> 00:05:33.880
some simple property or the state
132
00:05:33.880 --> 00:05:37.027
of a single vertex, namely their vertex
133
00:05:37.027 --> 00:05:39.575
in some underlying network structure,
134
00:05:39.575 --> 00:05:41.661
social network structure, if you will
135
00:05:41.661 --> 00:05:43.844
that we've exogenously imposed on them
136
00:05:43.844 --> 00:05:45.457
as experiments.
137
00:05:45.457 --> 00:05:46.859
So we've chosen these networks,
138
00:05:46.859 --> 00:05:48.678
and we're imposing them on the subjects.
139
00:05:48.678 --> 00:05:51.302
And basically these networks mediate all of,
140
00:05:51.302 --> 00:05:54.253
are the median through which communication takes place.
141
00:05:54.253 --> 00:05:57.027
And subjects only have a local view
142
00:05:57.027 --> 00:05:58.592
of the activity in the network.
143
00:05:58.592 --> 00:06:00.438
So you don't have a bird's eye view of the network.
144
00:06:00.438 --> 00:06:03.148
As a subject, you know the state of your own vertex
145
00:06:03.148 --> 00:06:05.259
and of the neighboring vertex.
146
00:06:05.259 --> 00:06:07.569
I'll show you GUI shots in a second.
147
00:06:07.569 --> 00:06:10.946
And following the principles of behavioral economics,
148
00:06:10.946 --> 00:06:14.743
we don't pay subjects for mere participation.
149
00:06:14.743 --> 00:06:17.474
We don't pay them for participation at all.
150
00:06:17.474 --> 00:06:19.371
In every short experiment, subjects have
151
00:06:19.371 --> 00:06:21.147
a real financial incentive
152
00:06:21.147 --> 00:06:25.070
to solve their piece of some collective task.
153
00:06:25.070 --> 00:06:28.620
And it's basically, you eat what you kill, right?
154
00:06:28.620 --> 00:06:29.989
You earn in these experiments
155
00:06:29.989 --> 00:06:33.428
whatever you earn according to the monetary payoffs
156
00:06:33.428 --> 00:06:36.000
that are clearly specified in each short experiment.
157
00:06:36.000 --> 00:06:40.065
So let me make this concrete with a simple example.
158
00:06:40.065 --> 00:06:41.867
So think about the graph coloring.
159
00:06:41.867 --> 00:06:43.779
Now, most of us are used to thinking
160
00:06:43.779 --> 00:06:46.350
about the graph coloring problem as a
161
00:06:46.350 --> 00:06:48.900
difficult optimization problem
162
00:06:48.900 --> 00:06:51.233
who's solution is useful for doing things
163
00:06:51.233 --> 00:06:54.028
like scheduling exams at large universities
164
00:06:54.028 --> 00:06:56.189
where there a limited number of rooms
165
00:06:56.189 --> 00:06:58.749
and a limited period in which to hold the exams.
166
00:06:58.749 --> 00:07:01.350
But it's actually a natural social problem as well.
167
00:07:01.350 --> 00:07:02.856
It's a natural cartoon, if you like,
168
00:07:02.856 --> 00:07:04.954
of a problem of social differentiation,
169
00:07:04.954 --> 00:07:07.672
where you simply want to distinguish yourself
170
00:07:07.672 --> 00:07:10.759
from your immediate neighbors in a social network.
171
00:07:10.759 --> 00:07:13.688
So when I was asked by the editor of the journal
172
00:07:13.688 --> 00:07:16.616
to give, you know, not a scheduling example
173
00:07:16.616 --> 00:07:19.479
of graph coloring but a real-world case
174
00:07:19.479 --> 00:07:21.184
where people might have to solve something
175
00:07:21.184 --> 00:07:24.355
like this problem, I offered the problem
176
00:07:24.355 --> 00:07:28.138
of picking a ringtone for your iPhone
177
00:07:28.138 --> 00:07:30.451
from a limited vocabulary, which is what you get
178
00:07:30.451 --> 00:07:32.479
when you get your iPhone.
179
00:07:32.479 --> 00:07:35.233
And one of the things that presumably we would like to do
180
00:07:35.233 --> 00:07:37.331
in selecting a ringtone for our iPhone
181
00:07:37.331 --> 00:07:39.276
is to pick one that's different from the people
182
00:07:39.276 --> 00:07:40.849
you spend a lot of time around
183
00:07:40.849 --> 00:07:42.074
so that we don't have this problem
184
00:07:42.074 --> 00:07:43.950
that every time somebody's iPhone rings,
185
00:07:43.950 --> 00:07:45.952
we're all looking into our pocket or our purse
186
00:07:45.952 --> 00:07:48.269
to figure out if it's a call for us.
187
00:07:48.269 --> 00:07:50.100
So think of graph coloring as a problem
188
00:07:50.100 --> 00:07:51.193
of social differentiation.
189
00:07:51.193 --> 00:07:53.556
You just have to choose a color for you vertex
190
00:07:53.556 --> 00:07:55.275
from some fixed set.
191
00:07:55.275 --> 00:07:58.122
In an experiment, we would incentivize you
192
00:07:58.122 --> 00:08:01.191
by basically paying you if, when the experiment ended
193
00:08:01.191 --> 00:08:02.507
and you were a different color
194
00:08:02.507 --> 00:08:03.587
than all of your neighbors,
195
00:08:03.587 --> 00:08:05.342
and otherwise you wouldn't be paid.
196
00:08:05.342 --> 00:08:06.856
So everybody has a local incentive
197
00:08:06.856 --> 00:08:10.201
to try to locally beat the graph coloring constraints
198
00:08:10.201 --> 00:08:14.104
and of course to introduce some lightweight economics
199
00:08:14.104 --> 00:08:17.604
planning, the maximum social welfare solution,
200
00:08:17.604 --> 00:08:20.993
to this problem, i.e. the configuration of play
201
00:08:20.993 --> 00:08:22.762
by the subjects that would collectively
202
00:08:22.762 --> 00:08:24.316
maximize their payoffs
203
00:08:24.316 --> 00:08:28.126
happens to coincide with the proper colorings of the graph.
204
00:08:28.126 --> 00:08:29.207
Okay?
205
00:08:29.207 --> 00:08:31.548
So what we've been doing across many, many different
206
00:08:31.548 --> 00:08:34.356
experimental sessions is deliberately varying
207
00:08:34.356 --> 00:08:35.616
a lot of things,
208
00:08:35.616 --> 00:08:39.137
but there are mainly two main design variables
209
00:08:39.137 --> 00:08:41.593
that we've been varying across these experiments.
210
00:08:41.593 --> 00:08:43.508
One is what's the network structure,
211
00:08:43.508 --> 00:08:44.938
what is the quality of the networks
212
00:08:44.938 --> 00:08:48.202
that we're imposing on the subjects?
213
00:08:48.202 --> 00:08:51.657
And what task or game, what collective problem
214
00:08:51.657 --> 00:08:54.122
are we trying to incentivize them to solve?
215
00:08:54.122 --> 00:08:58.126
And you'll see that, first of all let me point out
216
00:08:58.126 --> 00:09:00.638
that this is a huge design space, right?
217
00:09:00.638 --> 00:09:02.738
I mean, it's every possible problem
218
00:09:02.738 --> 00:09:04.736
you could think about solving on networks,
219
00:09:04.736 --> 00:09:08.386
and it's every possible network structure of 36 vertices.
220
00:09:08.386 --> 00:09:11.632
So there's sort of no hope to systematically explore
221
00:09:11.632 --> 00:09:14.377
this entire design space.
222
00:09:14.377 --> 00:09:16.166
And so what we've instead tried to do is make
223
00:09:16.166 --> 00:09:19.181
reasonably feasible choices.
224
00:09:19.181 --> 00:09:20.499
But there's always gonna be some amount
225
00:09:20.499 --> 00:09:22.953
of arbitrariness to it.
226
00:09:22.953 --> 00:09:24.612
But the networks we've largely chosen
227
00:09:24.612 --> 00:09:27.695
to be drawn from well-studied models,
228
00:09:28.818 --> 00:09:31.729
generative models for social networks,
229
00:09:31.729 --> 00:09:33.388
from the literature.
230
00:09:33.388 --> 00:09:36.012
So many of you probably know some of these models,
231
00:09:36.012 --> 00:09:39.397
like small-worlds models or preferential attachment models.
232
00:09:39.397 --> 00:09:41.977
These are all models that were designed,
233
00:09:41.977 --> 00:09:44.782
these are mathematical stochastic models
234
00:09:44.782 --> 00:09:46.562
for network formation that were designed
235
00:09:46.562 --> 00:09:49.917
to reliably generate certain topological properties
236
00:09:49.917 --> 00:09:52.828
that are frequently observed in real social networks,
237
00:09:52.828 --> 00:09:54.454
like having a small diameter
238
00:09:54.454 --> 00:09:56.733
or having a high cluster coefficient,
239
00:09:56.733 --> 00:09:59.850
having a heavy tail degree distribution.
240
00:09:59.850 --> 00:10:03.221
But I will towards the end talk about
241
00:10:03.221 --> 00:10:05.442
the artificiality of these network models
242
00:10:05.442 --> 00:10:06.694
and then try to address it,
243
00:10:06.694 --> 00:10:09.247
or show you some experiments that try to address it.
244
00:10:09.247 --> 00:10:11.587
And the tasks we've chosen for their diversity
245
00:10:11.587 --> 00:10:13.896
along a couple of different dimensions.
246
00:10:13.896 --> 00:10:17.956
One is whether the task is cooperative or competitive,
247
00:10:17.956 --> 00:10:21.410
which by I broadly mean is there some configuration
248
00:10:21.410 --> 00:10:23.588
of play by the subjects such that everybody
249
00:10:23.588 --> 00:10:25.671
gets the highest possible payoff?
250
00:10:25.671 --> 00:10:27.465
That's what I would call a cooperative game.
251
00:10:27.465 --> 00:10:30.739
And a competitive game is one where by definition,
252
00:10:30.739 --> 00:10:32.842
if some parties are making more,
253
00:10:32.842 --> 00:10:35.023
other parties must be making less.
254
00:10:35.023 --> 00:10:38.277
And we've also chosen tasks for their diversity
255
00:10:38.277 --> 00:10:42.988
according to what computational complexity would say
256
00:10:42.988 --> 00:10:45.541
about the difficulty of finding optimal
257
00:10:45.541 --> 00:10:47.602
or global solutions.
258
00:10:47.602 --> 00:10:50.928
I will be the first to admit that computational complexity
259
00:10:50.928 --> 00:10:54.928
is a woeful guide to what should be hard or easy
260
00:10:54.928 --> 00:10:57.965
in these experiments because computational complexity
261
00:10:57.965 --> 00:11:01.573
is about worst-case centralized computation
262
00:11:01.573 --> 00:11:04.020
where you have a global view of the network,
263
00:11:04.020 --> 00:11:07.455
whereas here we're talking about small-scale,
264
00:11:07.455 --> 00:11:10.230
non-asymptotic, non-worst-case computation
265
00:11:10.230 --> 00:11:12.876
by human subjects from only local information.
266
00:11:12.876 --> 00:11:15.977
But since complexity theory is really the only theory
267
00:11:15.977 --> 00:11:18.099
we have right now that's comprehensive
268
00:11:18.099 --> 00:11:20.684
and sort of in principle can assign
269
00:11:20.684 --> 00:11:23.796
a relative difficulty to any algorithmic
270
00:11:23.796 --> 00:11:28.446
problem you come across, it's the best starting point.
271
00:11:28.446 --> 00:11:30.820
And so there're several rules to this research
272
00:11:30.820 --> 00:11:33.314
that I hope you'll get a flavor for as I
273
00:11:33.314 --> 00:11:34.657
trundle along.
274
00:11:34.657 --> 00:11:37.613
The primary one is how is the structure
275
00:11:37.613 --> 00:11:41.067
of the underlying network and the task
276
00:11:41.067 --> 00:11:43.484
that subjects are trying to perform
277
00:11:43.484 --> 00:11:46.592
influence both collective performance
278
00:11:46.592 --> 00:11:49.565
and behavior and the individual performance and behavior.
279
00:11:49.565 --> 00:11:51.695
I'll mainly in this talk in the interest of time
280
00:11:51.695 --> 00:11:54.041
focus on collective performance and behavior.
281
00:11:54.041 --> 00:11:56.856
But the high-level question is,
282
00:11:56.856 --> 00:11:59.046
are there certain kinds of network structures
283
00:11:59.046 --> 00:12:02.301
on which solving certain types of problems are harder
284
00:12:02.301 --> 00:12:05.551
and other combinations that are easier?
285
00:12:06.485 --> 00:12:08.339
And are there systematic relationships
286
00:12:08.339 --> 00:12:10.068
between network structure and the ease
287
00:12:10.068 --> 00:12:11.597
or difficulty of problems,
288
00:12:11.597 --> 00:12:15.202
and how does that interact with the task in question?
289
00:12:15.202 --> 00:12:17.951
We're also interested in, and again I'll have
290
00:12:17.951 --> 00:12:19.751
a little less time to talk about this,
291
00:12:19.751 --> 00:12:21.690
is individual and collective modeling.
292
00:12:21.690 --> 00:12:24.173
So it would be nice if experiments like these
293
00:12:24.173 --> 00:12:26.457
generated data that would let you build
294
00:12:26.457 --> 00:12:29.386
individual models for subject play
295
00:12:29.386 --> 00:12:31.752
that when run in simulations on networks
296
00:12:31.752 --> 00:12:33.732
that we've never tried experiments on
297
00:12:33.732 --> 00:12:35.659
would make decently accurate predictions
298
00:12:35.659 --> 00:12:37.912
about how difficult those networks might be
299
00:12:37.912 --> 00:12:38.745
for the subjects.
300
00:12:38.745 --> 00:12:41.467
So actually generating not just a descriptive theory
301
00:12:41.467 --> 00:12:43.206
but a prescriptive one.
302
00:12:43.206 --> 00:12:46.465
And of course in the problems that I'll talk about
303
00:12:46.465 --> 00:12:49.180
that have a strong computational
304
00:12:49.180 --> 00:12:51.294
or you'd have to have the equilibrium theory
305
00:12:51.294 --> 00:12:52.851
about what should happen,
306
00:12:52.851 --> 00:12:57.347
so when complexity theory says a problem should be hard,
307
00:12:57.347 --> 00:13:00.427
does that mean it's hard for human subjects?
308
00:13:00.427 --> 00:13:03.628
When economics says that, well, at equilibrium
309
00:13:03.628 --> 00:13:06.150
this is what should happen in this experiment,
310
00:13:06.150 --> 00:13:08.809
do subjects play something close to an equilibrium?
311
00:13:08.809 --> 00:13:13.528
So we're interested in making those kinds of comparisons.
312
00:13:13.528 --> 00:13:16.624
Okay, so this is an exhaustive list
313
00:13:16.624 --> 00:13:18.652
of all the experiments we've run to date.
314
00:13:18.652 --> 00:13:21.738
You're not intended to read this slide.
315
00:13:21.738 --> 00:13:24.145
But it's meant to just give you kind of a broad sense,
316
00:13:24.145 --> 00:13:25.653
a sweep of what we've done.
317
00:13:25.653 --> 00:13:28.092
So the experiments are basically arranged
318
00:13:28.092 --> 00:13:31.086
by the problem type, and that's also the way
319
00:13:31.086 --> 00:13:32.516
we arrange the sessions, right,
320
00:13:32.516 --> 00:13:35.492
because the expensive thing is training the subjects
321
00:13:35.492 --> 00:13:39.868
to learn about, to learn what graph coloring is,
322
00:13:39.868 --> 00:13:41.575
to learn what the payoffs are,
323
00:13:41.575 --> 00:13:43.051
that we're incentivizing them to solve
324
00:13:43.051 --> 00:13:45.537
a graph coloring problem, to acclimate themselves
325
00:13:45.537 --> 00:13:47.967
to the particular GUI for that problem.
326
00:13:47.967 --> 00:13:49.624
And then once they've been trained to do that,
327
00:13:49.624 --> 00:13:52.113
it's relatively easy to run them through a series
328
00:13:52.113 --> 00:13:54.071
of short graph coloring experiments
329
00:13:54.071 --> 00:13:56.908
where the graph and other things may change
330
00:13:56.908 --> 00:13:58.521
from experiment to experiment.
331
00:13:58.521 --> 00:14:01.052
The high overhead thing is switching from task to task,
332
00:14:01.052 --> 00:14:05.227
so we tend to arrange the sessions based on task.
333
00:14:05.227 --> 00:14:08.084
So you can see here, so what it is here
334
00:14:08.084 --> 00:14:10.051
that I'm not asking you to read is for each one
335
00:14:10.051 --> 00:14:13.006
of these problems, I'm telling you
336
00:14:13.006 --> 00:14:16.980
what is player-controlled, like the color of their vertex.
337
00:14:16.980 --> 00:14:19.893
I'm basically telling you what are the maps
338
00:14:19.893 --> 00:14:21.231
that are involved, there are stages
339
00:14:21.231 --> 00:14:22.514
which you can think of as broadly
340
00:14:22.514 --> 00:14:24.525
what we were trying to collectively incentivize
341
00:14:24.525 --> 00:14:26.549
the subjects to do or to perform.
342
00:14:26.549 --> 00:14:28.809
And you can see here that
343
00:14:28.809 --> 00:14:30.790
if you look, for instance, just on the dimension
344
00:14:30.790 --> 00:14:33.586
of computational tractability
345
00:14:33.586 --> 00:14:35.279
vis a vis complexity theory,
346
00:14:35.279 --> 00:14:37.864
there's a huge range of difficulty here.
347
00:14:37.864 --> 00:14:40.309
So at the graph, you know, we've got experiments
348
00:14:40.309 --> 00:14:43.307
on graph coloring and also on independent set.
349
00:14:43.307 --> 00:14:46.751
These are two problems that are computationally challenging,
350
00:14:46.751 --> 00:14:48.341
formally, they're NP Hard.
351
00:14:48.341 --> 00:14:50.245
In fact, they're both NP Hard even to find
352
00:14:50.245 --> 00:14:53.396
lousy approximations of the optimal solution.
353
00:14:53.396 --> 00:14:55.863
At the other extreme, we have consensus.
354
00:14:55.863 --> 00:14:59.178
Consensus is the dual problem to coloring.
355
00:14:59.178 --> 00:15:01.220
Rather than wanting to be a different color
356
00:15:01.220 --> 00:15:02.772
than your neighbors, you're paying them
357
00:15:02.772 --> 00:15:04.588
only if you're the same color as your neighbors.
358
00:15:04.588 --> 00:15:06.142
So the max welfare solution, therefore,
359
00:15:06.142 --> 00:15:09.642
is just the entire network a single color.
360
00:15:11.295 --> 00:15:12.955
The computational complexity of that problem
361
00:15:12.955 --> 00:15:14.257
couldn't be more trivial, right?
362
00:15:14.257 --> 00:15:15.292
Just, whatever the network is,
363
00:15:15.292 --> 00:15:17.893
assign every vertex blue, okay?
364
00:15:17.893 --> 00:15:21.904
So we have a large range of computational difficulty.
365
00:15:21.904 --> 00:15:23.858
We have things in between, as well.
366
00:15:23.858 --> 00:15:24.910
I'll talk a little bit about this
367
00:15:24.910 --> 00:15:27.549
exchange economy, or trading game,
368
00:15:27.549 --> 00:15:30.033
where basically computing the equilibrium
369
00:15:30.033 --> 00:15:34.379
of that particular game can be done in polynomial time,
370
00:15:34.379 --> 00:15:37.623
but it requires linear programming as a subroutine.
371
00:15:37.623 --> 00:15:38.977
So you can think of this as a problem
372
00:15:38.977 --> 00:15:41.821
that's in P but barely in P.
373
00:15:41.821 --> 00:15:44.664
And then from a strategic standpoint,
374
00:15:44.664 --> 00:15:48.600
we have games that really are kind of coordination games
375
00:15:48.600 --> 00:15:50.501
that are in principle cooperative,
376
00:15:50.501 --> 00:15:52.248
meaning everybody can get our highest payoff,
377
00:15:52.248 --> 00:15:54.147
like the graph coloring and consensus,
378
00:15:54.147 --> 00:15:55.928
to other ones which are more competitive,
379
00:15:55.928 --> 00:15:57.163
like this independent set game
380
00:15:57.163 --> 00:15:59.471
that I won't have time to talk about
381
00:15:59.471 --> 00:16:01.021
and this bias loading game,
382
00:16:01.021 --> 00:16:03.043
which I probably will talk about.
383
00:16:03.043 --> 00:16:06.769
So these are the problems that we've been looking at
384
00:16:06.769 --> 00:16:10.616
and something about their computational complexity.
385
00:16:10.616 --> 00:16:13.996
And several of them have detailed equilibrium theory
386
00:16:13.996 --> 00:16:15.229
from economics.
387
00:16:15.229 --> 00:16:16.593
So for instance network bargaining
388
00:16:16.593 --> 00:16:20.793
and this exchange economy or trading problem
389
00:16:20.793 --> 00:16:23.111
have very clear equilibrium predictions
390
00:16:23.111 --> 00:16:24.408
coming from game theory.
391
00:16:24.408 --> 00:16:25.241
Yes?
392
00:16:25.241 --> 00:16:26.746
So I have a quick question.
393
00:16:26.746 --> 00:16:28.045
You mentioned that it's all a graph
394
00:16:28.045 --> 00:16:31.059
that all of these come from the social network models.
395
00:16:31.059 --> 00:16:33.277
Are these problems looked at on those models,
396
00:16:33.277 --> 00:16:34.756
and are the graphs on those models
397
00:16:34.756 --> 00:16:37.016
also very hard for coloring and things?
398
00:16:37.016 --> 00:16:40.035
So there's kind of a small literature
399
00:16:40.035 --> 00:16:44.198
starting to examine classical optimization problems
400
00:16:44.198 --> 00:16:47.255
when the graph is restricted to be
401
00:16:47.255 --> 00:16:49.425
from one of these families.
402
00:16:49.425 --> 00:16:53.189
That's very natural thing, sort of that you want to do,
403
00:16:53.189 --> 00:16:54.943
for instance, shortest path computations,
404
00:16:54.943 --> 00:16:57.520
and for Facebook you might wonder whether
405
00:16:57.520 --> 00:16:59.139
natural structures, social networks,
406
00:16:59.139 --> 00:17:00.768
would let you do something better than you can do
407
00:17:00.768 --> 00:17:02.718
in the worst case.
408
00:17:02.718 --> 00:17:06.346
I don't know that anybody's asked whether these
409
00:17:06.346 --> 00:17:09.443
problems, the problems that are intractable remain so
410
00:17:09.443 --> 00:17:13.450
when you look at these restricted classes of graph.
411
00:17:13.450 --> 00:17:17.074
My gut reaction would be probably not, right,
412
00:17:17.074 --> 00:17:19.034
because there's a lot of strong structure.
413
00:17:19.034 --> 00:17:22.902
It's hard to sort of see for many of these network models
414
00:17:22.902 --> 00:17:24.964
how you would program in a lot of the widgets
415
00:17:24.964 --> 00:17:27.626
that go into a traditional NP Completeness group.
416
00:17:27.626 --> 00:17:30.224
But so certainly the proofs would have to change,
417
00:17:30.224 --> 00:17:31.307
at a minimum.
418
00:17:32.866 --> 00:17:35.033
Okay, so what I want to do
419
00:17:36.942 --> 00:17:37.775
is just,
420
00:17:39.901 --> 00:17:42.196
so there are source papers for everything
421
00:17:42.196 --> 00:17:43.535
I'm gonna talk about here.
422
00:17:43.535 --> 00:17:47.053
And you can go and read, get any amount of detail you want.
423
00:17:47.053 --> 00:17:49.512
As of just last month in the October issue
424
00:17:49.512 --> 00:17:51.504
of Communications of the ACM,
425
00:17:51.504 --> 00:17:54.570
there was published a survey article
426
00:17:54.570 --> 00:17:56.716
covering all these experiments.
427
00:17:56.716 --> 00:17:59.521
So if you want sort of the companion paper to this talk
428
00:17:59.521 --> 00:18:02.790
that has a little bit more detail but is still shorter
429
00:18:02.790 --> 00:18:05.960
than reading eight individual papers, you can look at that.
430
00:18:05.960 --> 00:18:08.460
But in general and for the rest of the talk,
431
00:18:08.460 --> 00:18:09.960
I just want to give you feel for
432
00:18:09.960 --> 00:18:11.456
what these experiments are like.
433
00:18:11.456 --> 00:18:12.920
What happens in these experiments,
434
00:18:12.920 --> 00:18:15.601
what actual play looks like in these experiments,
435
00:18:15.601 --> 00:18:17.974
give you some hint for what results look like
436
00:18:17.974 --> 00:18:19.091
in these models, right,
437
00:18:19.091 --> 00:18:21.175
what kind of statement do we wanna make
438
00:18:21.175 --> 00:18:24.976
about these experiments, the data that they generate?
439
00:18:24.976 --> 00:18:27.608
And I'm gonna, in the interest of kind of keeping things
440
00:18:27.608 --> 00:18:30.495
raw, I'm not gonna spend a lot of time
441
00:18:30.495 --> 00:18:32.845
talking about statistical significance and the like.
442
00:18:32.845 --> 00:18:36.034
But all of those details are in the source papers.
443
00:18:36.034 --> 00:18:37.147
There was a question, yeah?
444
00:18:37.147 --> 00:18:38.567
So players are completely ignorant
445
00:18:38.567 --> 00:18:40.366
of the global state?
446
00:18:40.366 --> 00:18:43.569
That's right, you only see what's going on
447
00:18:43.569 --> 00:18:45.201
in your own network neighborhood.
448
00:18:45.201 --> 00:18:46.541
And while you're asking that question,
449
00:18:46.541 --> 00:18:49.806
let me take care of a couple of other details.
450
00:18:49.806 --> 00:18:52.803
So when subjects are brought into the laboratory
451
00:18:52.803 --> 00:18:55.454
for these experiments, they're generally students.
452
00:18:55.454 --> 00:18:59.172
So it's very easy to communicate to them
453
00:18:59.172 --> 00:19:01.896
what we want them to do and not do.
454
00:19:01.896 --> 00:19:05.345
We just tell them that hey, this is sort of a final exam.
455
00:19:05.345 --> 00:19:06.413
Don't try to communicate with anyone else.
456
00:19:06.413 --> 00:19:07.907
Don't talk with anyone else.
457
00:19:07.907 --> 00:19:10.133
Attend only your own workstation.
458
00:19:10.133 --> 00:19:11.474
We erect physical partitions
459
00:19:11.474 --> 00:19:13.874
so that people cannot see what's going on
460
00:19:13.874 --> 00:19:15.342
in other workstations.
461
00:19:15.342 --> 00:19:18.160
And there's no communication whatsoever
462
00:19:18.160 --> 00:19:21.724
allowed outside of the GUI that we provide in the system.
463
00:19:21.724 --> 00:19:23.646
So the only form of communication
464
00:19:23.646 --> 00:19:25.362
and the only knowledge that you have
465
00:19:25.362 --> 00:19:28.517
is through your GUI, which only is showing you
466
00:19:28.517 --> 00:19:30.642
your own network neighborhood,
467
00:19:30.642 --> 00:19:32.035
and I'll be precise about what I mean
468
00:19:32.035 --> 00:19:33.912
by a network neighborhood in a second.
469
00:19:33.912 --> 00:19:36.249
But nobody has a bird's eye view of the network.
470
00:19:36.249 --> 00:19:38.551
You only know what your own neighborhood looks like.
471
00:19:38.551 --> 00:19:42.440
Okay, and all moves in all of these games
472
00:19:42.440 --> 00:19:44.357
are asynchronous moves.
473
00:19:45.328 --> 00:19:48.228
Update your state as many times as you want, okay?
474
00:19:48.228 --> 00:19:50.846
So you can change your color as many times as you want,
475
00:19:50.846 --> 00:19:52.389
whenever you want.
476
00:19:52.389 --> 00:19:54.155
It's not synchronized in any sense.
477
00:19:54.155 --> 00:19:55.549
There's no clock.
478
00:19:55.549 --> 00:19:57.455
Whenever you change your color,
479
00:19:57.455 --> 00:19:59.517
it'll change on your GUI, and it'll change
480
00:19:59.517 --> 00:20:01.843
for anybody who's a neighbor of yours as well.
481
00:20:01.843 --> 00:20:03.444
They'll see that you changed your color.
482
00:20:03.444 --> 00:20:04.277
Yes?
483
00:20:04.277 --> 00:20:07.686
Isn't the implementation of communication...
484
00:20:07.686 --> 00:20:11.186
(student talking quietly)
485
00:20:12.254 --> 00:20:15.518
So as you'll see, there are many arbitrary aspects
486
00:20:15.518 --> 00:20:16.502
to these experiments.
487
00:20:16.502 --> 00:20:20.669
And there's a lot of artificialities and stylizations.
488
00:20:24.844 --> 00:20:29.011
So first of all, yes, there is arbitrariness to that.
489
00:20:31.875 --> 00:20:34.495
In order to kind of make experiments meaningful
490
00:20:34.495 --> 00:20:37.170
and have some hope of getting results out of them,
491
00:20:37.170 --> 00:20:38.735
yeah, you can't do everything.
492
00:20:38.735 --> 00:20:40.881
You can't explore everything simultaneously.
493
00:20:40.881 --> 00:20:43.216
You can't provide an open chat channel
494
00:20:43.216 --> 00:20:44.686
between network neighborhoods
495
00:20:44.686 --> 00:20:46.707
and hope to sort of meaningfully
496
00:20:46.707 --> 00:20:49.252
get clean results out of that.
497
00:20:49.252 --> 00:20:51.917
So one way that I like to think about these experiments,
498
00:20:51.917 --> 00:20:53.205
and I'll mention this specifically
499
00:20:53.205 --> 00:20:55.265
when I talk about one of the bias voting results,
500
00:20:55.265 --> 00:20:58.920
is that if some interesting phenomenon occurs
501
00:20:58.920 --> 00:21:03.352
in this model purely due to network structure
502
00:21:03.352 --> 00:21:05.685
and interactions with tasks,
503
00:21:06.592 --> 00:21:08.411
then in some sense you've identified
504
00:21:08.411 --> 00:21:11.143
at least the minimal conditions necessary
505
00:21:11.143 --> 00:21:13.834
for that phenomenon to happen in the real world, right?
506
00:21:13.834 --> 00:21:16.974
So if something can happen already in these models
507
00:21:16.974 --> 00:21:19.299
without any language provided, then at least
508
00:21:19.299 --> 00:21:21.490
we can sort of conclude that in general,
509
00:21:21.490 --> 00:21:23.545
language may not be required in order
510
00:21:23.545 --> 00:21:24.909
for that to take place.
511
00:21:24.909 --> 00:21:27.169
So in other words,
512
00:21:27.169 --> 00:21:29.044
I'm kind of jumping ahead of myself, but
513
00:21:29.044 --> 00:21:31.141
one of the experiments I'll describe
514
00:21:31.141 --> 00:21:32.554
is essentially designed to ask the question
515
00:21:32.554 --> 00:21:36.660
is it possible for a small but well-connected minority
516
00:21:36.660 --> 00:21:39.471
to systematically impose its preference
517
00:21:39.471 --> 00:21:44.021
on the collective against the greater goodwill,
518
00:21:44.021 --> 00:21:46.442
okay, sort of at the expense of social welfare.
519
00:21:46.442 --> 00:21:49.082
I would argue that you can show that the answer to that
520
00:21:49.082 --> 00:21:50.818
is definitively yes in a model
521
00:21:50.818 --> 00:21:53.293
where people don't even have language.
522
00:21:53.293 --> 00:21:55.921
But that's a worthwhile thing to know, right,
523
00:21:55.921 --> 00:21:57.632
because it sort of shows you that for instance
524
00:21:57.632 --> 00:21:59.062
when you're saying things like well,
525
00:21:59.062 --> 00:22:02.959
do political bloggers have an outside influence
526
00:22:02.959 --> 00:22:06.577
on national discourse and policy discussions,
527
00:22:06.577 --> 00:22:09.160
you can say, well, yes they do,
528
00:22:10.124 --> 00:22:14.095
and maybe that's because they're using rhetoric,
529
00:22:14.095 --> 00:22:16.826
passion, persuasion, language, et cetera,
530
00:22:16.826 --> 00:22:18.673
sort of everything you need to be human.
531
00:22:18.673 --> 00:22:20.757
But if I can show you that can already happen in a lab
532
00:22:20.757 --> 00:22:23.029
purely due to network structural effects,
533
00:22:23.029 --> 00:22:26.176
I think that's something worth knowing.
534
00:22:26.176 --> 00:22:28.716
But it's obviously not gonna study
535
00:22:28.716 --> 00:22:31.564
the language question specifically.
536
00:22:31.564 --> 00:22:33.247
By the way, one of the things that you'll see though,
537
00:22:33.247 --> 00:22:35.529
of course, is that in all of these,
538
00:22:35.529 --> 00:22:37.633
in every single experiment, the first thing
539
00:22:37.633 --> 00:22:40.461
subjects try to do, whether they succeed or not,
540
00:22:40.461 --> 00:22:42.812
is in some sense reintroduce language
541
00:22:42.812 --> 00:22:45.171
by various forms of signaling.
542
00:22:45.171 --> 00:22:47.161
Okay, so what I want to do is give you
543
00:22:47.161 --> 00:22:50.967
kind of a high-level tour of not all of the different
544
00:22:50.967 --> 00:22:53.602
experiments that I described on the last slide,
545
00:22:53.602 --> 00:22:55.189
but just on a sample of those
546
00:22:55.189 --> 00:22:58.522
so you get a feel for what happens in these experiments.
547
00:22:58.522 --> 00:23:00.690
So I'm gonna start by discussing two tasks
548
00:23:00.690 --> 00:23:04.184
kind of side by side, which are graph coloring
549
00:23:04.184 --> 00:23:05.177
and consensus.
550
00:23:05.177 --> 00:23:08.812
And remember this pair is nice because
551
00:23:08.812 --> 00:23:10.940
one of them is computationally intractable
552
00:23:10.940 --> 00:23:11.893
in the worst case.
553
00:23:11.893 --> 00:23:14.174
The other is always trivial.
554
00:23:14.174 --> 00:23:17.351
But both of them are cognitively,
555
00:23:17.351 --> 00:23:20.133
the two of them are cognitively incredibly similar.
556
00:23:20.133 --> 00:23:22.625
And in fact this might be the only pair of experiments,
557
00:23:22.625 --> 00:23:24.904
the only pair of tasks that we ran together
558
00:23:24.904 --> 00:23:26.510
in the same experimental session
559
00:23:26.510 --> 00:23:30.160
because it's so easy to get subjects to switch gears, right?
560
00:23:30.160 --> 00:23:32.418
You say, well, up until now you've been trying
561
00:23:32.418 --> 00:23:34.089
to be a different color than your neighbors.
562
00:23:34.089 --> 00:23:36.021
The GUI isn't gonna change at all.
563
00:23:36.021 --> 00:23:37.871
In the rest of the experiments, you're gonna be paid
564
00:23:37.871 --> 00:23:41.647
to be the same color as your neighbors, okay?
565
00:23:41.647 --> 00:23:43.605
Okay, so first of all, let me give you a sense
566
00:23:43.605 --> 00:23:45.862
of what GUIs look like in this experiment.
567
00:23:45.862 --> 00:23:48.008
This is a sample screenshot of a GUI
568
00:23:48.008 --> 00:23:50.193
for a coloring experiment.
569
00:23:50.193 --> 00:23:52.346
I'm not gonna describe all the annotation on it,
570
00:23:52.346 --> 00:23:53.623
just the main pieces.
571
00:23:53.623 --> 00:23:56.080
So in the middle, as in all of our experiments,
572
00:23:56.080 --> 00:23:59.063
there's an information panel
573
00:23:59.063 --> 00:24:00.799
which is showing you the current state
574
00:24:00.799 --> 00:24:02.300
of your network neighborhood.
575
00:24:02.300 --> 00:24:04.966
So here you are, clearly marked in the middle.
576
00:24:04.966 --> 00:24:07.372
And you have edges to each of your neighbors,
577
00:24:07.372 --> 00:24:09.621
and you're seeing what their colors are as well.
578
00:24:09.621 --> 00:24:11.823
And we're also choosing in this particular GUI
579
00:24:11.823 --> 00:24:14.856
to show you not just the edges from you to your neighbors
580
00:24:14.856 --> 00:24:17.900
but any edges that are present between your neighbors.
581
00:24:17.900 --> 00:24:20.740
So this red edge indicates that these two neighbors
582
00:24:20.740 --> 00:24:22.741
are connected, and these two neighbors are connected,
583
00:24:22.741 --> 00:24:25.177
and from that you can infer that there's not an edge
584
00:24:25.177 --> 00:24:27.113
between these two neighbors, okay?
585
00:24:27.113 --> 00:24:29.649
So this is what sociologists would sometimes call
586
00:24:29.649 --> 00:24:30.777
an ego network.
587
00:24:30.777 --> 00:24:33.117
Here I'm calling it the first neighborhood view.
588
00:24:33.117 --> 00:24:38.048
And it's basically motivated by this back of the envelope
589
00:24:38.048 --> 00:24:40.688
claim that people make that well, if I gave you
590
00:24:40.688 --> 00:24:42.206
a concrete definition of friendship
591
00:24:42.206 --> 00:24:44.390
and asked you to write down the names of all your friends,
592
00:24:44.390 --> 00:24:47.121
you hopefully would get most of your friends.
593
00:24:47.121 --> 00:24:49.789
And if I ask you, well, which pairs of your friends
594
00:24:49.789 --> 00:24:51.782
are friends with each other?
595
00:24:51.782 --> 00:24:53.079
Human beings are very good at tracking
596
00:24:53.079 --> 00:24:55.875
relationships of social relationships as well.
597
00:24:55.875 --> 00:24:58.290
Everybody has some sense of, for each pair of their friends,
598
00:24:58.290 --> 00:24:59.819
whether they're friends or not.
599
00:24:59.819 --> 00:25:01.255
This is especially important for not inviting
600
00:25:01.255 --> 00:25:02.836
to the same party two of your friends
601
00:25:02.836 --> 00:25:05.024
that can't stand each other.
602
00:25:05.024 --> 00:25:06.910
But certainly we can agree that if I ask you
603
00:25:06.910 --> 00:25:08.895
anything beyond this first neighborhood,
604
00:25:08.895 --> 00:25:10.283
we'd all be in deep trouble.
605
00:25:10.283 --> 00:25:11.424
If I asked you to write down the names
606
00:25:11.424 --> 00:25:13.464
of all the friends of your friends
607
00:25:13.464 --> 00:25:17.404
and social relationships between those step two neighbors,
608
00:25:17.404 --> 00:25:20.426
we would all do a very poor job.
609
00:25:20.426 --> 00:25:22.300
So this is the kind of information you're getting.
610
00:25:22.300 --> 00:25:23.856
And at the bottom there's an action panel,
611
00:25:23.856 --> 00:25:25.084
which indicates the coloring.
612
00:25:25.084 --> 00:25:26.853
This involves clicking on these buttons
613
00:25:26.853 --> 00:25:28.892
and changing your color when you want to.
614
00:25:28.892 --> 00:25:33.540
And then there's some sort of status information at the top.
615
00:25:33.540 --> 00:25:35.740
We always have to impose a hard time limit
616
00:25:35.740 --> 00:25:36.573
on these experiments.
617
00:25:36.573 --> 00:25:37.559
We would rather not.
618
00:25:37.559 --> 00:25:40.328
We would rather let every experiment run to completion
619
00:25:40.328 --> 00:25:42.251
because introducing a hard time limit
620
00:25:42.251 --> 00:25:45.067
gives clipping bias to the data.
621
00:25:45.067 --> 00:25:47.731
But if you don't do that, the problem is
622
00:25:47.731 --> 00:25:49.311
you can't keep people there indefinitely.
623
00:25:49.311 --> 00:25:51.609
And maybe the first graph coloring experiment you give them,
624
00:25:51.609 --> 00:25:54.154
they just never find a global solution,
625
00:25:54.154 --> 00:25:55.354
and they're there for three hours.
626
00:25:55.354 --> 00:25:56.965
And nobody gets paid anything,
627
00:25:56.965 --> 00:25:59.699
and you have no data, and your subjects hate you.
628
00:25:59.699 --> 00:26:01.868
So you have to have some short limit.
629
00:26:01.868 --> 00:26:04.926
Luckily, as I'll mention, people tend to be quite good
630
00:26:04.926 --> 00:26:06.169
at these sorts of things.
631
00:26:06.169 --> 00:26:10.070
And so this clipping bias isn't too bad.
632
00:26:10.070 --> 00:26:11.644
But many experiments are finishing
633
00:26:11.644 --> 00:26:13.207
well before the time limit.
634
00:26:13.207 --> 00:26:14.060
Yes?
635
00:26:14.060 --> 00:26:17.977
(student talking indistinctly)
636
00:26:21.884 --> 00:26:22.827
No, good question.
637
00:26:22.827 --> 00:26:24.580
So in all of the graph coloring experiments,
638
00:26:24.580 --> 00:26:27.947
we give them exactly the chromatic number of colors.
639
00:26:27.947 --> 00:26:31.062
So we don't give them any impossible tasks,
640
00:26:31.062 --> 00:26:34.410
but we also don't give them any approximation slack.
641
00:26:34.410 --> 00:26:36.963
We don't give them quite, offline we give them
642
00:26:36.963 --> 00:26:39.056
chromatic number, we give them that many colors.
643
00:26:39.056 --> 00:26:42.384
For consensus, of course, the number of colors to give
644
00:26:42.384 --> 00:26:44.379
isn't obvious, but there's some sense
645
00:26:44.379 --> 00:26:46.551
in which of course more is harder
646
00:26:46.551 --> 00:26:47.991
because it's a coordination problem.
647
00:26:47.991 --> 00:26:50.384
There's just more choices that we have to sift through.
648
00:26:50.384 --> 00:26:52.940
So in the graph coloring experiments,
649
00:26:52.940 --> 00:26:54.087
there's always a chromatic number.
650
00:26:54.087 --> 00:26:56.346
And the consensus experiments is all,
651
00:26:56.346 --> 00:26:58.777
we gave them nine, arbitrarily,
652
00:26:58.777 --> 00:27:00.774
as sort of something that seemed manageable
653
00:27:00.774 --> 00:27:03.849
but not so few that it would be too easy.
654
00:27:03.849 --> 00:27:06.830
And so we tell you how much time has elapsed
655
00:27:06.830 --> 00:27:10.651
so you know when the clock is running out.
656
00:27:10.651 --> 00:27:12.994
And in general, in all these experiments,
657
00:27:12.994 --> 00:27:15.106
if there ever comes a point,
658
00:27:15.106 --> 00:27:17.031
the incentives in general are local here, right?
659
00:27:17.031 --> 00:27:18.933
You're being paid if you're a different color
660
00:27:18.933 --> 00:27:20.184
than your neighbors.
661
00:27:20.184 --> 00:27:23.634
And so it's possible in a graph coloring experiment
662
00:27:23.634 --> 00:27:25.712
they don't reach a global solution
663
00:27:25.712 --> 00:27:27.104
but some people still get paid
664
00:27:27.104 --> 00:27:29.196
because locally, they're a different color,
665
00:27:29.196 --> 00:27:31.384
even though other parties don't have that property.
666
00:27:31.384 --> 00:27:35.255
But anytime the subjects reach a max welfare solution,
667
00:27:35.255 --> 00:27:36.710
i.e. in the case of graph coloring,
668
00:27:36.710 --> 00:27:38.693
if at some point before the time limit
669
00:27:38.693 --> 00:27:40.935
everybody's a different color than their neighbor,
670
00:27:40.935 --> 00:27:42.803
so they actually found a global solution,
671
00:27:42.803 --> 00:27:45.468
the system would freeze, everybody would be paid,
672
00:27:45.468 --> 00:27:47.283
and we would move on to the next experiment.
673
00:27:47.283 --> 00:27:48.116
Yes?
674
00:27:50.405 --> 00:27:53.588
Okay, and then you also are told what your payoffs are.
675
00:27:53.588 --> 00:27:55.751
You're basically told, in this case you'll be paid
676
00:27:55.751 --> 00:27:57.254
two dollars for this experiment
677
00:27:57.254 --> 00:27:59.282
if your color is different from all of your neighbors.
678
00:27:59.282 --> 00:28:01.736
Otherwise you get nothing.
679
00:28:01.736 --> 00:28:02.569
Yes?
680
00:28:02.569 --> 00:28:06.823
Isn't progress different than time?
681
00:28:06.823 --> 00:28:08.906
So game progress, yeah.
682
00:28:11.039 --> 00:28:13.066
So in this particular set of experiments,
683
00:28:13.066 --> 00:28:15.467
so in later experiments we took notions
684
00:28:15.467 --> 00:28:16.667
of game progress away.
685
00:28:16.667 --> 00:28:19.042
Game progress is sort of a global measure
686
00:28:19.042 --> 00:28:22.581
of how far we were from a global solution.
687
00:28:22.581 --> 00:28:23.414
And
688
00:28:24.732 --> 00:28:28.068
we had that in the initial, the very first experiment
689
00:28:28.068 --> 00:28:29.927
we did, which this screenshot is from,
690
00:28:29.927 --> 00:28:32.883
primarily because we were concerned,
691
00:28:32.883 --> 00:28:33.817
it's a cognitive thing.
692
00:28:33.817 --> 00:28:37.322
We were primarily concerned if subjects like this one
693
00:28:37.322 --> 00:28:39.967
might spend much of the experiment in a state
694
00:28:39.967 --> 00:28:44.142
where they're a different color from all their neighbors.
695
00:28:44.142 --> 00:28:46.775
And they might think that, sort of not attempt
696
00:28:46.775 --> 00:28:48.856
the experiment because it's so static.
697
00:28:48.856 --> 00:28:50.603
And by having this thing at the top,
698
00:28:50.603 --> 00:28:52.501
just sort of some indication that there's activity
699
00:28:52.501 --> 00:28:54.586
going on elsewhere in the network,
700
00:28:54.586 --> 00:28:56.989
to kind of keep them attentive to the experiment.
701
00:28:56.989 --> 00:28:59.359
We dropped that later because it gave them,
702
00:28:59.359 --> 00:29:01.185
even though it's a weak global signal,
703
00:29:01.185 --> 00:29:02.869
it is a global signal.
704
00:29:02.869 --> 00:29:04.449
And it didn't seem necessary
705
00:29:04.449 --> 00:29:06.306
because the money seemed to be enough
706
00:29:06.306 --> 00:29:09.306
to get people to attend to the task.
707
00:29:10.863 --> 00:29:11.696
Okay.
708
00:29:13.092 --> 00:29:15.759
So that's what a GUI looks like.
709
00:29:16.930 --> 00:29:21.581
And what I'm showing you in this visualization,
710
00:29:21.581 --> 00:29:23.809
so the GUI I actually showed you
711
00:29:23.809 --> 00:29:25.610
was both from the client you would see
712
00:29:25.610 --> 00:29:27.247
for graph coloring and also for consensus.
713
00:29:27.247 --> 00:29:28.380
And the only difference for consensus
714
00:29:28.380 --> 00:29:30.636
is that there'd be a line that says
715
00:29:30.636 --> 00:29:31.721
instead of you want to be a different color
716
00:29:31.721 --> 00:29:34.422
to get paid, you wanna be the same color to get paid, okay?
717
00:29:34.422 --> 00:29:37.422
So these diagrams are visualizations
718
00:29:38.915 --> 00:29:41.288
of actual play that I'll explain in a minute.
719
00:29:41.288 --> 00:29:42.660
But first let me tell you a little bit more
720
00:29:42.660 --> 00:29:44.705
about this joint session we ran
721
00:29:44.705 --> 00:29:47.320
with both graph coloring and consensus problems
722
00:29:47.320 --> 00:29:49.470
on the same set of underlying networks.
723
00:29:49.470 --> 00:29:50.910
And let me begin by describing
724
00:29:50.910 --> 00:29:52.503
what the underlying networks were.
725
00:29:52.503 --> 00:29:54.631
So we generated a bunch of networks
726
00:29:54.631 --> 00:29:56.736
from a model that I'm about to describe.
727
00:29:56.736 --> 00:29:58.208
And then for every single network,
728
00:29:58.208 --> 00:30:01.110
we ran a coloring experiment on that network
729
00:30:01.110 --> 00:30:03.630
and we ran a consensus experiment on that network
730
00:30:03.630 --> 00:30:05.469
to just sort of compare the difficulties
731
00:30:05.469 --> 00:30:07.259
of these two problems on these same
732
00:30:07.259 --> 00:30:08.844
underlying set of networks.
733
00:30:08.844 --> 00:30:10.448
So what were the networks?
734
00:30:10.448 --> 00:30:11.689
So the networks were generated
735
00:30:11.689 --> 00:30:13.859
according to a stochastic family
736
00:30:13.859 --> 00:30:15.916
that has a single parameter to it,
737
00:30:15.916 --> 00:30:18.371
which is a real number, let's call it Q,
738
00:30:18.371 --> 00:30:19.620
between zero and one.
739
00:30:19.620 --> 00:30:21.069
It's a probability, okay?
740
00:30:21.069 --> 00:30:24.756
Let's call it the rewiring parameter, okay?
741
00:30:24.756 --> 00:30:27.914
When Q is equal to zero, the network
742
00:30:27.914 --> 00:30:29.672
generated by the model is exactly
743
00:30:29.672 --> 00:30:32.370
the fixed network I am showing you here.
744
00:30:32.370 --> 00:30:36.965
It is a chain of six cliques of size six each, okay?
745
00:30:36.965 --> 00:30:40.552
But so each one of these little groups of size six
746
00:30:40.552 --> 00:30:42.150
has full interconnectivity.
747
00:30:42.150 --> 00:30:43.480
All edges are present.
748
00:30:43.480 --> 00:30:46.047
And then each group has a leader, if you'd like,
749
00:30:46.047 --> 00:30:49.107
that's connected in this chain of communication
750
00:30:49.107 --> 00:30:52.403
to the leaders to the left and to the right, okay?
751
00:30:52.403 --> 00:30:54.490
So you can think of this network
752
00:30:54.490 --> 00:30:57.744
as a very, very crude model, and for those of you
753
00:30:57.744 --> 00:30:59.283
that have seen these kinds of models before,
754
00:30:59.283 --> 00:31:01.583
you'll know that this is a common thing to do.
755
00:31:01.583 --> 00:31:04.501
You sort of design a model that lets you move
756
00:31:04.501 --> 00:31:07.153
between two extreme worlds in a smooth way.
757
00:31:07.153 --> 00:31:09.402
This extreme world is a crude model
758
00:31:09.402 --> 00:31:12.205
of a very tribal society, in which you have
759
00:31:12.205 --> 00:31:14.188
these tightly interconnected groups
760
00:31:14.188 --> 00:31:17.010
with full communication and then loose communication
761
00:31:17.010 --> 00:31:18.346
between the groups.
762
00:31:18.346 --> 00:31:21.921
And that's what the Q equals zero version is.
763
00:31:21.921 --> 00:31:24.873
If Q is positive, what you do is you go around
764
00:31:24.873 --> 00:31:27.798
to every clique edge in turn
765
00:31:27.798 --> 00:31:31.591
and you flip a coin with bias Q, independently,
766
00:31:31.591 --> 00:31:33.967
for each internal clique edge.
767
00:31:33.967 --> 00:31:38.207
If the coin comes up heads, with probability Q,
768
00:31:38.207 --> 00:31:40.215
sorry, if it comes up tails, which is probability
769
00:31:40.215 --> 00:31:42.582
one minus Q, you leave the edge in place
770
00:31:42.582 --> 00:31:44.326
and go on to the next edge.
771
00:31:44.326 --> 00:31:46.698
If it comes up heads with probability Q,
772
00:31:46.698 --> 00:31:50.528
you take this clique edge and you delete it.
773
00:31:50.528 --> 00:31:53.636
And you replace it with a random long-distance edge
774
00:31:53.636 --> 00:31:54.491
in the network.
775
00:31:54.491 --> 00:31:56.841
More specifically, you would take this edge,
776
00:31:56.841 --> 00:32:00.402
delete it, take one of its two endpoints at random,
777
00:32:00.402 --> 00:32:04.025
uniformly, and give that guy a random connection
778
00:32:04.025 --> 00:32:07.899
to an arbitrary vertex in the entire population.
779
00:32:07.899 --> 00:32:09.674
So you're replacing these clique edges
780
00:32:09.674 --> 00:32:12.367
with just sort of random pairs of connectivity
781
00:32:12.367 --> 00:32:14.518
across the entire population.
782
00:32:14.518 --> 00:32:17.709
So when Q equals zero, you get this network.
783
00:32:17.709 --> 00:32:20.770
When Q equals one, right, then this network
784
00:32:20.770 --> 00:32:22.154
is really a red herring.
785
00:32:22.154 --> 00:32:23.470
You're gonna erase the whole thing
786
00:32:23.470 --> 00:32:25.625
and replace it with the same number of edges
787
00:32:25.625 --> 00:32:27.528
but just between randomly chosen pairs
788
00:32:27.528 --> 00:32:28.901
of individuals in the network.
789
00:32:28.901 --> 00:32:31.494
So when Q equals one, you really have a random wrap
790
00:32:31.494 --> 00:32:33.584
or error readiness model.
791
00:32:33.584 --> 00:32:36.065
And in between, you have something in between, right?
792
00:32:36.065 --> 00:32:38.420
So when Q is in between, you have both,
793
00:32:38.420 --> 00:32:41.822
you have some of the original clique structure in place
794
00:32:41.822 --> 00:32:44.318
but some of it erased and replaced
795
00:32:44.318 --> 00:32:47.480
with this long-distance connective, okay?
796
00:32:47.480 --> 00:32:48.807
And so in a minute I'm gonna describe to you
797
00:32:48.807 --> 00:32:51.016
what happens when you run both coloring
798
00:32:51.016 --> 00:32:52.875
and consensus experiments on this same
799
00:32:52.875 --> 00:32:54.448
underlying set of networks.
800
00:32:54.448 --> 00:32:57.279
But first let me just tell you what these pictures show.
801
00:32:57.279 --> 00:32:59.778
These pictures show four consensus experiments.
802
00:32:59.778 --> 00:33:02.989
So everybody wants to be the same color as their neighbors.
803
00:33:02.989 --> 00:33:05.060
Consensus experiments on networks
804
00:33:05.060 --> 00:33:08.067
in which the value of Q was relatively small,
805
00:33:08.067 --> 00:33:12.085
like zero or 10%, maybe as much as 20% and not more,
806
00:33:12.085 --> 00:33:13.995
so that this internal clique structure
807
00:33:13.995 --> 00:33:16.944
that you see here is still largely in place, okay,
808
00:33:16.944 --> 00:33:21.279
and what each one of these visualizations is showing you
809
00:33:21.279 --> 00:33:25.710
is time of the experiment across the X axis.
810
00:33:25.710 --> 00:33:28.380
And then for every player in the experiment, there's a row.
811
00:33:28.380 --> 00:33:30.116
So there are 36 rows here.
812
00:33:30.116 --> 00:33:31.891
And the row is just showing you
813
00:33:31.891 --> 00:33:34.610
what was that player, what color was that player playing
814
00:33:34.610 --> 00:33:37.779
at that moment in the experiment?
815
00:33:37.779 --> 00:33:40.885
And because this clique structure is still largely in place
816
00:33:40.885 --> 00:33:43.625
at small values of Q, it makes sense, of course,
817
00:33:43.625 --> 00:33:47.396
to arrange the first six rows to be that clique,
818
00:33:47.396 --> 00:33:48.968
the members in that group.
819
00:33:48.968 --> 00:33:51.419
The next six rows will be the members of this group,
820
00:33:51.419 --> 00:33:52.674
et cetera, et cetera.
821
00:33:52.674 --> 00:33:55.027
Okay, so that's what you're seeing here.
822
00:33:55.027 --> 00:33:58.162
And you can see we're getting some both sensible
823
00:33:58.162 --> 00:34:00.513
but also very interesting collective
824
00:34:00.513 --> 00:34:01.877
and individual behavior.
825
00:34:01.877 --> 00:34:04.136
So first of all, in all these experiments,
826
00:34:04.136 --> 00:34:07.824
the first several milliseconds of the experiment,
827
00:34:07.824 --> 00:34:10.719
virtually all nine of the possible colors are used
828
00:34:10.719 --> 00:34:12.294
in each of the experiments.
829
00:34:12.294 --> 00:34:14.327
There's a great diversity in initial color.
830
00:34:14.327 --> 00:34:16.153
But then coordination begins to happen
831
00:34:16.153 --> 00:34:17.626
because of the network.
832
00:34:17.626 --> 00:34:19.812
And you start to see that pretty quickly
833
00:34:19.812 --> 00:34:21.896
in every experiment, we get down to
834
00:34:21.896 --> 00:34:23.684
at least the bulk of the population playing
835
00:34:23.684 --> 00:34:26.693
only a relatively much smaller group of colors.
836
00:34:26.693 --> 00:34:30.567
So here several seconds into the experiment,
837
00:34:30.567 --> 00:34:32.314
we're really down to three colors,
838
00:34:32.314 --> 00:34:34.427
the red, the brown, and the yellow.
839
00:34:34.427 --> 00:34:36.197
This one's more complicated, but you can see
840
00:34:36.197 --> 00:34:38.329
that a little ways in we're really down
841
00:34:38.329 --> 00:34:40.646
pretty much to blue and brown.
842
00:34:40.646 --> 00:34:42.582
Similarly here we get pretty quickly
843
00:34:42.582 --> 00:34:46.087
to this light green, blue, and orange.
844
00:34:46.087 --> 00:34:48.799
So you can see the network structure taking effect
845
00:34:48.799 --> 00:34:51.238
and people starting to coordinate with their neighbors.
846
00:34:51.238 --> 00:34:54.128
But then you see, also, that these groups
847
00:34:54.128 --> 00:34:55.849
are sort of behaving as blocks.
848
00:34:55.849 --> 00:34:58.328
And in particular, groups tend to drop one color
849
00:34:58.328 --> 00:34:59.705
and adopt another color
850
00:34:59.705 --> 00:35:02.658
more or less simultaneously, presumably along
851
00:35:02.658 --> 00:35:04.322
with the group that they're a part of.
852
00:35:04.322 --> 00:35:08.179
So you can see that clearly in this top experiment here.
853
00:35:08.179 --> 00:35:10.386
There's a group that largely starts on brown
854
00:35:10.386 --> 00:35:12.972
and gives it up at some point to yellow.
855
00:35:12.972 --> 00:35:15.995
The later, lower group down here sticks to the brown
856
00:35:15.995 --> 00:35:17.327
for a longer period of time,
857
00:35:17.327 --> 00:35:19.543
but then gives it up and joins the red group.
858
00:35:19.543 --> 00:35:22.683
So you can see lots of kind of block-sized structures,
859
00:35:22.683 --> 00:35:24.341
a size, sort of width
860
00:35:24.341 --> 00:35:28.157
of approximately six in these diagrams.
861
00:35:28.157 --> 00:35:31.275
You also see very interesting aspects
862
00:35:31.275 --> 00:35:33.467
of individual behavior.
863
00:35:33.467 --> 00:35:35.526
So in particular, you see many, many instances
864
00:35:35.526 --> 00:35:36.910
of signaling behavior.
865
00:35:36.910 --> 00:35:39.441
One would have to go look at the data directly,
866
00:35:39.441 --> 00:35:41.624
but presumably this is a vertex
867
00:35:41.624 --> 00:35:46.351
that is in a group that is converged locally on brown.
868
00:35:46.351 --> 00:35:50.743
But this is an individual who likely has connections,
869
00:35:50.743 --> 00:35:53.148
because of long-distance connectivity,
870
00:35:53.148 --> 00:35:55.843
to this group up here, who is trying to signal
871
00:35:55.843 --> 00:35:58.522
to their brown neighbors that all is not well
872
00:35:58.522 --> 00:35:59.706
elsewhere in the network.
873
00:35:59.706 --> 00:36:03.056
And in particular, this vertex is unhappy
874
00:36:03.056 --> 00:36:05.724
because they have both brown and yellow neighbors
875
00:36:05.724 --> 00:36:07.720
and can't be the same color as all their neighbors
876
00:36:07.720 --> 00:36:08.970
for that reason.
877
00:36:08.970 --> 00:36:10.213
So they're experimenting around,
878
00:36:10.213 --> 00:36:13.911
trying to get their group to switch to yellow.
879
00:36:13.911 --> 00:36:17.072
And maybe the signaling contributed to that later event.
880
00:36:17.072 --> 00:36:19.923
You also see very interesting acts
881
00:36:19.923 --> 00:36:22.681
of individual irrationality.
882
00:36:22.681 --> 00:36:23.514
So here's an individual
883
00:36:23.514 --> 00:36:24.683
(laughter)
884
00:36:24.683 --> 00:36:27.957
who's displaying this color, this blue,
885
00:36:27.957 --> 00:36:30.189
for a long period in which nobody else
886
00:36:30.189 --> 00:36:33.076
in the network is playing that color.
887
00:36:33.076 --> 00:36:36.858
And then back to the collective level,
888
00:36:36.858 --> 00:36:38.839
this is my favorite of these diagrams,
889
00:36:38.839 --> 00:36:40.940
this is the only consensus experiment
890
00:36:40.940 --> 00:36:43.374
in which the session did not end before the time limit
891
00:36:43.374 --> 00:36:45.495
with unanimity across the network.
892
00:36:45.495 --> 00:36:46.840
And you can see what happens here is,
893
00:36:46.840 --> 00:36:49.959
pretty quickly we get down to orange, blue,
894
00:36:49.959 --> 00:36:51.508
and this green color.
895
00:36:51.508 --> 00:36:53.363
At some point midway through the experiment,
896
00:36:53.363 --> 00:36:56.409
we're almost converged to blue, except for this one block
897
00:36:56.409 --> 00:36:57.921
that's still playing green.
898
00:36:57.921 --> 00:36:59.962
A short while later, they give it up
899
00:36:59.962 --> 00:37:01.653
and switch over to blue.
900
00:37:01.653 --> 00:37:04.291
But not just before they managed to cause
901
00:37:04.291 --> 00:37:07.159
a small trickle of their original color
902
00:37:07.159 --> 00:37:10.264
to kind of propagate to the other side of the network
903
00:37:10.264 --> 00:37:11.797
and then take a firm hold
904
00:37:11.797 --> 00:37:14.122
until at some later point where they actually
905
00:37:14.122 --> 00:37:16.278
had a majority playing their original color,
906
00:37:16.278 --> 00:37:18.016
and the whole network has swapped.
907
00:37:18.016 --> 00:37:21.310
And you can't tell that all these other experiments
908
00:37:21.310 --> 00:37:23.843
ended the moment when some last player switches
909
00:37:23.843 --> 00:37:26.260
to the color that makes everybody unanimous.
910
00:37:26.260 --> 00:37:28.464
This one actually ends because of the time limit.
911
00:37:28.464 --> 00:37:31.171
And we're still roughly at 50-50 at this point.
912
00:37:31.171 --> 00:37:34.175
This just makes you kind of a visceral sense
913
00:37:34.175 --> 00:37:35.694
for what happens in these experiments.
914
00:37:35.694 --> 00:37:38.652
Of course, none of the subjects know
915
00:37:38.652 --> 00:37:39.916
what this network looks like.
916
00:37:39.916 --> 00:37:42.135
None of them of course are seeing this picture.
917
00:37:42.135 --> 00:37:44.157
And none of them really know, I mean,
918
00:37:44.157 --> 00:37:46.200
so this Q equals zero network,
919
00:37:46.200 --> 00:37:49.672
these leaders have special roles in some sense.
920
00:37:49.672 --> 00:37:51.664
They have to somehow play a coordinating role
921
00:37:51.664 --> 00:37:52.972
if we're gonna reach unanimity.
922
00:37:52.972 --> 00:37:54.691
But if you were that leader,
923
00:37:54.691 --> 00:37:56.528
you have no particular reason to believe
924
00:37:56.528 --> 00:37:58.576
that you have special position in that network.
925
00:37:58.576 --> 00:38:01.075
You're just some vertex with some number of neighbors.
926
00:38:01.075 --> 00:38:04.647
And you wouldn't know that you had different
927
00:38:04.647 --> 00:38:07.942
local topology than anybody else.
928
00:38:07.942 --> 00:38:08.775
Yeah?
929
00:38:08.775 --> 00:38:10.536
So they had no idea what the network looked like?
930
00:38:10.536 --> 00:38:12.072
No idea other than what they can infer
931
00:38:12.072 --> 00:38:14.306
by looking at their local connectivity.
932
00:38:14.306 --> 00:38:16.565
And the Q's are uniform across these?
933
00:38:16.565 --> 00:38:18.881
These are different values of Q,
934
00:38:18.881 --> 00:38:20.958
but always on the low side, so that
935
00:38:20.958 --> 00:38:25.388
when you get to Q equals one or even Q equals a half,
936
00:38:25.388 --> 00:38:27.521
these diagrams wouldn't make sense anymore
937
00:38:27.521 --> 00:38:29.108
because first of all there's no clear
938
00:38:29.108 --> 00:38:32.383
natural arrangement of the rows in groups
939
00:38:32.383 --> 00:38:35.533
because the group structure has basically been erased.
940
00:38:35.533 --> 00:38:37.212
And there's no synchronization of time,
941
00:38:37.212 --> 00:38:38.887
at the start of the game, it's not taking rounds
942
00:38:38.887 --> 00:38:39.720
or something?
943
00:38:39.720 --> 00:38:42.532
No, no, everybody gets to do whatever they want
944
00:38:42.532 --> 00:38:44.732
whenever they want.
945
00:38:44.732 --> 00:38:48.232
(student talking quietly)
946
00:38:52.671 --> 00:38:54.534
Yeah, yeah, and for sort of everything else,
947
00:38:54.534 --> 00:38:55.425
you would have it.
948
00:38:55.425 --> 00:38:57.233
So plus your coefficient starts off
949
00:38:57.233 --> 00:38:59.176
very, very high.
950
00:38:59.176 --> 00:39:02.599
The diameter starts off high as well, for instance.
951
00:39:02.599 --> 00:39:05.066
As you increase Q, the diameter of course drops
952
00:39:05.066 --> 00:39:07.411
because you're adding long-distance connectivity,
953
00:39:07.411 --> 00:39:09.431
and the clustering coefficient's going down as well.
954
00:39:09.431 --> 00:39:11.133
So this is one of these models
955
00:39:11.133 --> 00:39:12.844
that has a knob you can turn,
956
00:39:12.844 --> 00:39:15.323
and you can ask, well, is there kind of
957
00:39:15.323 --> 00:39:17.178
a natural regime of this knob,
958
00:39:17.178 --> 00:39:20.393
where I can simultaneously achieve fairly high clustering
959
00:39:20.393 --> 00:39:22.328
and fairly low diameter.
960
00:39:22.328 --> 00:39:24.161
And the answer is yes.
961
00:39:25.633 --> 00:39:27.009
Yes?
962
00:39:27.009 --> 00:39:30.179
If it's natural, have you balanced for
963
00:39:30.179 --> 00:39:33.437
experience in your players, age, gender,
964
00:39:33.437 --> 00:39:34.679
those kinds of things?
965
00:39:34.679 --> 00:39:37.774
So first of all, the demographics of these students,
966
00:39:37.774 --> 00:39:39.836
except for the very first set of experiments we did,
967
00:39:39.836 --> 00:39:41.248
which was from the larger community,
968
00:39:41.248 --> 00:39:43.240
they're Penn undergraduates,
969
00:39:43.240 --> 00:39:45.257
which is not to say there's no variation.
970
00:39:45.257 --> 00:39:49.200
But they're all of approximately similar age
971
00:39:49.200 --> 00:39:52.485
and at least whatever is demographically applied
972
00:39:52.485 --> 00:39:54.733
by being a Penn undergraduate.
973
00:39:54.733 --> 00:39:56.422
In terms of experience, I mean,
974
00:39:56.422 --> 00:39:59.650
every subject in these experiments is,
975
00:39:59.650 --> 00:40:01.446
I don't think any of them have participated
976
00:40:01.446 --> 00:40:04.498
in anything like this before.
977
00:40:04.498 --> 00:40:08.745
And so we haven't controlled specifically,
978
00:40:08.745 --> 00:40:11.458
I wouldn't know what experience to measure.
979
00:40:11.458 --> 00:40:13.605
A related question is whether there appears
980
00:40:13.605 --> 00:40:17.495
to be learning occurring within an experimental session.
981
00:40:17.495 --> 00:40:20.495
And we haven't seen evidence for it,
982
00:40:21.424 --> 00:40:23.174
although this would strongly depend on
983
00:40:23.174 --> 00:40:24.914
what we defined to be learning.
984
00:40:24.914 --> 00:40:28.518
But one simple thing you can do is plot
985
00:40:28.518 --> 00:40:31.669
the cumulative payoffs to the subject
986
00:40:31.669 --> 00:40:34.091
as a function of the index of the experiment
987
00:40:34.091 --> 00:40:35.707
during the experimental session.
988
00:40:35.707 --> 00:40:37.773
And that basically, all it looks like
989
00:40:37.773 --> 00:40:39.294
is more or less a line.
990
00:40:39.294 --> 00:40:42.129
So you don't sort of see them starting to earn
991
00:40:42.129 --> 00:40:45.717
at a higher rate as they get better at playing, somehow.
992
00:40:45.717 --> 00:40:48.339
You also don't see them leveling off
993
00:40:48.339 --> 00:40:51.064
and doing worse due to fatigue.
994
00:40:51.064 --> 00:40:54.382
So there's no obvious learning effects to be seen.
995
00:40:54.382 --> 00:40:56.136
And you said that, I mean,
996
00:40:56.136 --> 00:40:57.336
motivation is intrinsic.
997
00:40:57.336 --> 00:40:58.877
It's winning the payment.
998
00:40:58.877 --> 00:41:01.666
But how do you get them there in the first place?
999
00:41:01.666 --> 00:41:04.027
We tell them we're gonna pay them.
1000
00:41:04.027 --> 00:41:04.860
If they win.
1001
00:41:04.860 --> 00:41:08.325
Yeah, we tell them, you know, money will be made.
1002
00:41:08.325 --> 00:41:09.747
Come play these games.
1003
00:41:09.747 --> 00:41:12.247
Some will win, some will lose.
1004
00:41:13.777 --> 00:41:14.610
Yes.
1005
00:41:14.610 --> 00:41:16.282
For some of the pathological cases,
1006
00:41:16.282 --> 00:41:18.157
do you ask them what was going on?
1007
00:41:18.157 --> 00:41:19.504
Were they using Facebook,
1008
00:41:19.504 --> 00:41:21.410
do they hate the graph coloring?
1009
00:41:21.410 --> 00:41:23.973
You mean sort of these irrational behaviors?
1010
00:41:23.973 --> 00:41:24.885
Yeah.
1011
00:41:24.885 --> 00:41:27.242
We kind of don't have the luxury
1012
00:41:27.242 --> 00:41:29.337
of kind of seeing the data and asking them,
1013
00:41:29.337 --> 00:41:30.820
hey, what are you doing here?
1014
00:41:30.820 --> 00:41:34.063
Pictures like these only emerge after an hour
1015
00:41:34.063 --> 00:41:35.641
of processing.
1016
00:41:35.641 --> 00:41:37.668
We do give every subject an exit survey
1017
00:41:37.668 --> 00:41:40.380
in which we ask them to comment generally
1018
00:41:40.380 --> 00:41:43.113
on what strategy they adopted,
1019
00:41:43.113 --> 00:41:46.401
what strategy they thought others were playing.
1020
00:41:46.401 --> 00:41:49.555
And in those self reports you will see
1021
00:41:49.555 --> 00:41:52.463
many are describing signaling strategies,
1022
00:41:52.463 --> 00:41:54.980
some of them actually quite subtle.
1023
00:41:54.980 --> 00:41:57.125
So for instance in graph coloring,
1024
00:41:57.125 --> 00:41:59.350
some people say, well, I would switch back and forth
1025
00:41:59.350 --> 00:42:01.454
between two colors when I perceived we were
1026
00:42:01.454 --> 00:42:04.609
kind of, you know, whatever their language is
1027
00:42:04.609 --> 00:42:06.050
for a local minimum,
1028
00:42:06.050 --> 00:42:07.713
just get the attention of your neighbors.
1029
00:42:07.713 --> 00:42:09.900
But others said things like, well,
1030
00:42:09.900 --> 00:42:12.384
if I could be either red or blue
1031
00:42:12.384 --> 00:42:15.437
and still satisfy the color constraints,
1032
00:42:15.437 --> 00:42:17.553
I would toggle back and forth between them
1033
00:42:17.553 --> 00:42:19.535
in case one of my neighbors would be able
1034
00:42:19.535 --> 00:42:21.479
to solve their problem locally
1035
00:42:21.479 --> 00:42:23.942
by being one or the other of those two colors
1036
00:42:23.942 --> 00:42:26.554
so that they could pick the one that they wanted to
1037
00:42:26.554 --> 00:42:28.327
and I would take the other one.
1038
00:42:28.327 --> 00:42:31.104
So that's a rather subtle signaling strategy.
1039
00:42:31.104 --> 00:42:34.092
And so there's no question that people self report
1040
00:42:34.092 --> 00:42:36.940
on these signaling strategies,
1041
00:42:36.940 --> 00:42:39.988
and then the harder thing to verify for the data
1042
00:42:39.988 --> 00:42:42.610
is whether anybody ever actually understood
1043
00:42:42.610 --> 00:42:45.344
those semantics and reacted to them
1044
00:42:45.344 --> 00:42:46.653
in a way that was beneficial,
1045
00:42:46.653 --> 00:42:49.320
either individually or globally.
1046
00:42:50.430 --> 00:42:52.934
Okay, so again this is a slide
1047
00:42:52.934 --> 00:42:55.474
that there's more on than I'm gonna say about.
1048
00:42:55.474 --> 00:42:57.045
I'm just gonna focus on this diagram
1049
00:42:57.045 --> 00:42:58.743
at the bottom left.
1050
00:42:58.743 --> 00:43:00.513
And this'll be one of the few points in the talk
1051
00:43:00.513 --> 00:43:02.269
that I'll try to get a sense of like,
1052
00:43:02.269 --> 00:43:04.045
what would a scientific finding
1053
00:43:04.045 --> 00:43:06.010
of these experiments look like?
1054
00:43:06.010 --> 00:43:08.017
So ignore the dashed lines.
1055
00:43:08.017 --> 00:43:09.576
Those are from a model.
1056
00:43:09.576 --> 00:43:12.527
The solid lines are for the actual experiments.
1057
00:43:12.527 --> 00:43:14.878
This is a plotting, which on the X axis
1058
00:43:14.878 --> 00:43:17.321
I have this rewiring probability Q.
1059
00:43:17.321 --> 00:43:19.937
Remember when Q equals zero, we really have that
1060
00:43:19.937 --> 00:43:21.285
chain of cliques.
1061
00:43:21.285 --> 00:43:23.711
When it equals one, we're at a truly random network.
1062
00:43:23.711 --> 00:43:25.202
In between we're in between.
1063
00:43:25.202 --> 00:43:29.016
And so each one of these values of Q,
1064
00:43:29.016 --> 00:43:31.120
we sampled several networks and random coloring
1065
00:43:31.120 --> 00:43:33.969
experiment and consensus experiment on them.
1066
00:43:33.969 --> 00:43:38.019
And this is showing the average time of subjects
1067
00:43:38.019 --> 00:43:41.885
at each value of Q to find a global solution
1068
00:43:41.885 --> 00:43:43.469
to either the coloring problem
1069
00:43:43.469 --> 00:43:46.552
or the consensus problem respectively
1070
00:43:47.434 --> 00:43:50.245
for networks generated at that value of Q.
1071
00:43:50.245 --> 00:43:51.514
Okay, so lower is better here
1072
00:43:51.514 --> 00:43:53.370
because it means they're finding a global solution
1073
00:43:53.370 --> 00:43:54.906
more rapidly.
1074
00:43:54.906 --> 00:43:57.155
And of course the thing I want to highlight
1075
00:43:57.155 --> 00:44:00.474
about this diagram is that there's monotone behavior
1076
00:44:00.474 --> 00:44:04.110
for each problem with respect to this parameter Q.
1077
00:44:04.110 --> 00:44:05.979
So there's clear scuffle effects.
1078
00:44:05.979 --> 00:44:08.693
So even before talking about the relative performance,
1079
00:44:08.693 --> 00:44:12.956
the fact that this curve is consistently decreasing with Q
1080
00:44:12.956 --> 00:44:14.720
and this one is consistently increasing
1081
00:44:14.720 --> 00:44:17.333
establishes perhaps not a mind-bending
1082
00:44:17.333 --> 00:44:20.007
but an important fact, which is human subjects
1083
00:44:20.007 --> 00:44:22.321
really do, on these experiments,
1084
00:44:22.321 --> 00:44:25.261
respond in systematic ways to systematic
1085
00:44:25.261 --> 00:44:27.526
structural changes within the network, right?
1086
00:44:27.526 --> 00:44:29.375
So in a simple model, when there's only one
1087
00:44:29.375 --> 00:44:30.879
structural parameter Q,
1088
00:44:30.879 --> 00:44:34.038
both of these curves are behaving monotonically
1089
00:44:34.038 --> 00:44:36.621
with respect to that parameter.
1090
00:44:37.492 --> 00:44:39.420
And of course the more interesting thing about this plot
1091
00:44:39.420 --> 00:44:41.990
is that they're responding in opposite ways.
1092
00:44:41.990 --> 00:44:45.654
As I replace clique structure with long-distance structure,
1093
00:44:45.654 --> 00:44:48.227
consensus is becoming considerably easier,
1094
00:44:48.227 --> 00:44:51.673
and as I do that same thing, coloring is becoming harder.
1095
00:44:51.673 --> 00:44:56.340
And if you want to know, do these curves really cross,
1096
00:44:56.340 --> 00:45:00.693
you can go do a test for significance for the data
1097
00:45:00.693 --> 00:45:03.896
at this end and establish, yes, this really is above that,
1098
00:45:03.896 --> 00:45:05.936
and at the other end, this really is above that.
1099
00:45:05.936 --> 00:45:07.489
So these curves really are crossing
1100
00:45:07.489 --> 00:45:10.344
in a statistically significant sense.
1101
00:45:10.344 --> 00:45:13.982
And I think the important thing about this diagram is
1102
00:45:13.982 --> 00:45:17.827
that it shows the dangers of spending too much time
1103
00:45:17.827 --> 00:45:21.288
analyzing social network topology and structure
1104
00:45:21.288 --> 00:45:24.826
in isolation without thinking about what's going on
1105
00:45:24.826 --> 00:45:25.904
on the network.
1106
00:45:25.904 --> 00:45:27.333
Because here we have two problems
1107
00:45:27.333 --> 00:45:29.323
that are cognitively incredibly similar
1108
00:45:29.323 --> 00:45:31.310
from the subject's point of view.
1109
00:45:31.310 --> 00:45:33.530
And one of them, if I ask the question,
1110
00:45:33.530 --> 00:45:36.777
well, if I don't talk about the task,
1111
00:45:36.777 --> 00:45:39.039
and I say, well, what are good network structures
1112
00:45:39.039 --> 00:45:42.466
for social collaboration or problem solving,
1113
00:45:42.466 --> 00:45:44.205
or what are bad network structures?
1114
00:45:44.205 --> 00:45:46.198
This diagram shows you that even within
1115
00:45:46.198 --> 00:45:49.326
a very restricted setting, that question may be meaningless
1116
00:45:49.326 --> 00:45:51.595
if we don't talk about the particular task
1117
00:45:51.595 --> 00:45:54.012
because in this simple model,
1118
00:45:55.272 --> 00:45:57.652
what's a good network for consensus
1119
00:45:57.652 --> 00:46:00.633
is a bad network for coloring and vice versa.
1120
00:46:00.633 --> 00:46:05.057
And so in some sense this plot has an editorial comment
1121
00:46:05.057 --> 00:46:08.998
contained in it, which is a critique of a literature
1122
00:46:08.998 --> 00:46:12.078
that I myself will admit to have participated in,
1123
00:46:12.078 --> 00:46:14.490
where you kind of go out and get network data
1124
00:46:14.490 --> 00:46:18.498
from a collaboration network or a co-authorship network,
1125
00:46:18.498 --> 00:46:21.203
and you marvel at, oh, look the degree distribution
1126
00:46:21.203 --> 00:46:22.569
looks like such and such
1127
00:46:22.569 --> 00:46:24.366
and this clustering coefficient.
1128
00:46:24.366 --> 00:46:27.851
And one kind of output of this is sort of, so what?
1129
00:46:27.851 --> 00:46:29.272
What did you really learn?
1130
00:46:29.272 --> 00:46:32.691
Is that structure helpful to the task
1131
00:46:32.691 --> 00:46:34.948
that this network or organization is trying to solve
1132
00:46:34.948 --> 00:46:36.381
or is it harmful?
1133
00:46:36.381 --> 00:46:39.411
And this is to show that you can't understand that
1134
00:46:39.411 --> 00:46:41.128
by looking at the network structure alone.
1135
00:46:41.128 --> 00:46:44.465
You really have to think about the dynamics, so to speak.
1136
00:46:44.465 --> 00:46:45.298
Okay.
1137
00:46:48.079 --> 00:46:51.273
So I'm kind of debating what to do with time here.
1138
00:46:51.273 --> 00:46:53.826
I think what I want to do is
1139
00:46:53.826 --> 00:46:56.600
skip this, because it's a more technical part,
1140
00:46:56.600 --> 00:47:00.168
and it has to do with a much more financial model.
1141
00:47:00.168 --> 00:47:02.283
The thing that's most nice about it,
1142
00:47:02.283 --> 00:47:05.828
which makes it easier for looking at the survey paper
1143
00:47:05.828 --> 00:47:08.736
from the CACM, the thing that's nice about it
1144
00:47:08.736 --> 00:47:11.154
is that if I take a game theoretic view
1145
00:47:11.154 --> 00:47:12.907
of the problems I talked about so far,
1146
00:47:12.907 --> 00:47:14.800
like coloring and consensus, I mean,
1147
00:47:14.800 --> 00:47:17.262
I can talk about the equilibria of graph coloring
1148
00:47:17.262 --> 00:47:18.548
or consensus, right?
1149
00:47:18.548 --> 00:47:20.301
Let's take graph coloring for a second.
1150
00:47:20.301 --> 00:47:23.755
So certainly if everybody plays,
1151
00:47:23.755 --> 00:47:26.220
if we all collectively play a proper coloring
1152
00:47:26.220 --> 00:47:29.427
of the network, that's a Nash equilibrium, right?
1153
00:47:29.427 --> 00:47:31.811
Because we're all getting, there's two rates of pay,
1154
00:47:31.811 --> 00:47:36.288
zero for being the same color as one of your neighbors,
1155
00:47:36.288 --> 00:47:39.167
or plus for being differentiated.
1156
00:47:39.167 --> 00:47:40.767
So for all to get the payoff,
1157
00:47:40.767 --> 00:47:42.075
that's gotta be a Nash equilibrium.
1158
00:47:42.075 --> 00:47:44.193
And in fact that's the maximum social welfare solution
1159
00:47:44.193 --> 00:47:46.204
as well, right?
1160
00:47:46.204 --> 00:47:50.336
But there are lots of, so notice that this thing
1161
00:47:50.336 --> 00:47:52.913
when it's made has nothing to do with network structure.
1162
00:47:52.913 --> 00:47:54.138
Whatever the network structure is,
1163
00:47:54.138 --> 00:47:55.819
as long as we have enough colors,
1164
00:47:55.819 --> 00:47:57.637
there is one Nash equilibrium
1165
00:47:57.637 --> 00:47:59.407
in which we all get paid.
1166
00:47:59.407 --> 00:48:01.661
And pretty much for any other network structure
1167
00:48:01.661 --> 00:48:03.300
you give me, there will be all kinds
1168
00:48:03.300 --> 00:48:05.944
of other Nash equilibria as well
1169
00:48:05.944 --> 00:48:09.576
that basically correspond to local minima, right?
1170
00:48:09.576 --> 00:48:12.413
So by local minima I mean a configuration of colors
1171
00:48:12.413 --> 00:48:14.783
that's not a proper coloring but in which
1172
00:48:14.783 --> 00:48:18.866
no single, every player that has a color conflict
1173
00:48:20.010 --> 00:48:21.447
doesn't have an available color
1174
00:48:21.447 --> 00:48:22.444
that they can change to.
1175
00:48:22.444 --> 00:48:24.343
So in particular if there are three colors
1176
00:48:24.343 --> 00:48:27.468
and all three of those colors are used by your neighbors,
1177
00:48:27.468 --> 00:48:29.377
there's just nothing you can do.
1178
00:48:29.377 --> 00:48:32.532
There's no unilateral deviation you can effect
1179
00:48:32.532 --> 00:48:34.673
that will now cause you to get payoff.
1180
00:48:34.673 --> 00:48:37.535
And it's easy to find, for really any network,
1181
00:48:37.535 --> 00:48:40.259
configurations where you have these local minima.
1182
00:48:40.259 --> 00:48:43.794
So we can take an economic or game theoretic view
1183
00:48:43.794 --> 00:48:45.826
of the graph coloring problem, but the equilibrium theory
1184
00:48:45.826 --> 00:48:47.592
isn't especially interesting.
1185
00:48:47.592 --> 00:48:49.926
In particular, it has nothing to do with network structure.
1186
00:48:49.926 --> 00:48:51.851
It just tells you there are equilibria.
1187
00:48:51.851 --> 00:48:53.804
There's a good equilibria that corresponds
1188
00:48:53.804 --> 00:48:54.888
to a proper coloring.
1189
00:48:54.888 --> 00:48:57.420
And there's all these local minima equilibria.
1190
00:48:57.420 --> 00:48:59.659
The thing that's nice about this network trading model
1191
00:48:59.659 --> 00:49:03.310
is that it has an equilibrium in which
1192
00:49:03.310 --> 00:49:06.214
different players are going to make more or less money
1193
00:49:06.214 --> 00:49:08.900
depending exactly on their net position,
1194
00:49:08.900 --> 00:49:11.340
depending entirely on their position in the network.
1195
00:49:11.340 --> 00:49:14.817
So the high level is the trading model,
1196
00:49:14.817 --> 00:49:16.528
and again, I'm not gonna talk about
1197
00:49:16.528 --> 00:49:18.393
the details of the results, but it's a trading model
1198
00:49:18.393 --> 00:49:20.101
where you have two types of players.
1199
00:49:20.101 --> 00:49:23.811
One type of player starts with, let's say milk.
1200
00:49:23.811 --> 00:49:25.792
The other type of player starts with wheat.
1201
00:49:25.792 --> 00:49:29.203
And both players have, the only way they get payoff
1202
00:49:29.203 --> 00:49:32.146
is by trading what they started with for the other thing.
1203
00:49:32.146 --> 00:49:34.145
So you want to trade as much of the milk
1204
00:49:34.145 --> 00:49:35.661
you started with for as much of the wheat
1205
00:49:35.661 --> 00:49:38.505
that you can get from the other side.
1206
00:49:38.505 --> 00:49:40.986
And so the network's sort of bipartite,
1207
00:49:40.986 --> 00:49:43.371
the milk and wheat players are on opposite sides
1208
00:49:43.371 --> 00:49:44.531
of this network.
1209
00:49:44.531 --> 00:49:45.364
And
1210
00:49:46.331 --> 00:49:48.090
because everything's symmetric,
1211
00:49:48.090 --> 00:49:50.125
everybody's starting with one unit
1212
00:49:50.125 --> 00:49:52.213
of their starting good, and there's the same number
1213
00:49:52.213 --> 00:49:54.660
of each type of player, if at equilibrium
1214
00:49:54.660 --> 00:49:57.054
somebody's gonna make more or somebody's gonna make less,
1215
00:49:57.054 --> 00:50:00.739
it has to be due to entirely the network structure.
1216
00:50:00.739 --> 00:50:02.075
And in this particular model,
1217
00:50:02.075 --> 00:50:04.409
what ends up being important is not so much
1218
00:50:04.409 --> 00:50:05.953
how many trading partners you have
1219
00:50:05.953 --> 00:50:07.220
but what their status is.
1220
00:50:07.220 --> 00:50:09.587
So in particular, you and I might both be
1221
00:50:09.587 --> 00:50:11.330
on the same side of the network
1222
00:50:11.330 --> 00:50:13.596
and have let's say five trading partners.
1223
00:50:13.596 --> 00:50:15.903
Okay, so our degree is the same.
1224
00:50:15.903 --> 00:50:16.736
But
1225
00:50:19.956 --> 00:50:21.744
my five trading partners might have
1226
00:50:21.744 --> 00:50:24.478
lots of other partners back on our side of the network,
1227
00:50:24.478 --> 00:50:25.687
and your five trading partners
1228
00:50:25.687 --> 00:50:28.403
might only have you as a trading partner.
1229
00:50:28.403 --> 00:50:30.691
Well, then your five trading partners
1230
00:50:30.691 --> 00:50:32.444
are way more valuable than mine, right,
1231
00:50:32.444 --> 00:50:34.940
because they're essentially a captive of you, okay?
1232
00:50:34.940 --> 00:50:37.481
So there's a very nice, very efficient equilibrium theory
1233
00:50:37.481 --> 00:50:40.684
that basically involves looking for sets of vertices
1234
00:50:40.684 --> 00:50:42.400
on one side of the network
1235
00:50:42.400 --> 00:50:45.493
that collectively only have a small number of neighbors
1236
00:50:45.493 --> 00:50:46.765
on the other side.
1237
00:50:46.765 --> 00:50:48.834
And that's what determines who gets richer or poorer
1238
00:50:48.834 --> 00:50:49.966
in this game.
1239
00:50:49.966 --> 00:50:52.350
And so you have a very detailed equilibrium theory.
1240
00:50:52.350 --> 00:50:55.417
And what's nice about that is you can then go,
1241
00:50:55.417 --> 00:50:58.367
for every network that we run such an experiment on,
1242
00:50:58.367 --> 00:51:01.450
you can go to, we can offline compute
1243
00:51:02.845 --> 00:51:04.569
what the equilibrium theory predicts
1244
00:51:04.569 --> 00:51:06.912
would be the outcome and ask, well,
1245
00:51:06.912 --> 00:51:09.017
do collectively the subjects play
1246
00:51:09.017 --> 00:51:11.517
something close to the computed equilibrium theory?
1247
00:51:11.517 --> 00:51:13.916
And the short answer is that yes,
1248
00:51:13.916 --> 00:51:15.328
they play something that's quite close
1249
00:51:15.328 --> 00:51:18.730
but is also kind of warped back towards equality.
1250
00:51:18.730 --> 00:51:23.081
So what I mean by that is, maybe to say the punchline,
1251
00:51:23.081 --> 00:51:26.919
the best model we found for how the human subjects play
1252
00:51:26.919 --> 00:51:29.349
is to take the wealth distribution
1253
00:51:29.349 --> 00:51:31.717
predicted by equilibrium, which might predict
1254
00:51:31.717 --> 00:51:34.206
that some parties are gonna get much more than others,
1255
00:51:34.206 --> 00:51:37.662
and mixing that distribution with the uniform distribution.
1256
00:51:37.662 --> 00:51:40.168
So there's a sort of inequality aversion phenomenon
1257
00:51:40.168 --> 00:51:41.530
going on.
1258
00:51:41.530 --> 00:51:43.645
So go look at it if you're interested,
1259
00:51:43.645 --> 00:51:45.890
especially if you have an interest in microeconomics
1260
00:51:45.890 --> 00:51:48.810
because there's this very tight tie
1261
00:51:48.810 --> 00:51:51.862
between equilibrium theory and that.
1262
00:51:51.862 --> 00:51:54.233
So let me finish up in the last five or 15 minutes or so
1263
00:51:54.233 --> 00:51:57.631
talking about these voting experiments.
1264
00:51:57.631 --> 00:52:00.881
Biased voting is exactly like consensus
1265
00:52:01.973 --> 00:52:04.507
but with a crucial strategic twist.
1266
00:52:04.507 --> 00:52:08.674
So like consensus, the goal is to be the same color
1267
00:52:09.783 --> 00:52:12.329
as your neighbors, or really to be more clear,
1268
00:52:12.329 --> 00:52:13.730
the same color locally.
1269
00:52:13.730 --> 00:52:15.428
So now there's only gonna be two colors.
1270
00:52:15.428 --> 00:52:16.877
And the payoffs are as follows:
1271
00:52:16.877 --> 00:52:20.460
if within the allotted time all 36 subjects
1272
00:52:22.010 --> 00:52:25.427
unanimously select red or all 36 subjects
1273
00:52:26.718 --> 00:52:28.718
unanimously select blue,
1274
00:52:30.082 --> 00:52:31.969
then everybody gets paid something,
1275
00:52:31.969 --> 00:52:34.489
and I'll return to the something in a second.
1276
00:52:34.489 --> 00:52:36.181
If that condition doesn't hold,
1277
00:52:36.181 --> 00:52:37.503
everybody gets nothing.
1278
00:52:37.503 --> 00:52:39.919
If, when time runs out, 35 of us are red
1279
00:52:39.919 --> 00:52:42.049
and one of us is playing blue, everybody
1280
00:52:42.049 --> 00:52:43.670
in that experiment gets nothing.
1281
00:52:43.670 --> 00:52:46.766
So now the views in information are still gonna be local,
1282
00:52:46.766 --> 00:52:48.803
but now the payoffs are really collective.
1283
00:52:48.803 --> 00:52:51.036
It's not just we want the same color as your neighbors.
1284
00:52:51.036 --> 00:52:52.436
You're really wanting the same color
1285
00:52:52.436 --> 00:52:54.799
across the entire network, okay?
1286
00:52:54.799 --> 00:52:57.328
So there's a very, very strong course
1287
00:52:57.328 --> 00:52:59.211
toward unanimity.
1288
00:52:59.211 --> 00:53:01.500
And if we're not unanimous, we don't get anything.
1289
00:53:01.500 --> 00:53:04.505
But the strategic twist is that in the same experiment,
1290
00:53:04.505 --> 00:53:07.913
some of us may get paid more for unanimity to red,
1291
00:53:07.913 --> 00:53:10.708
and others may be paid more for unanimity to blue.
1292
00:53:10.708 --> 00:53:13.186
So for instance, my payoffs might tell me,
1293
00:53:13.186 --> 00:53:16.715
well, if we reach unanimity to red, I get a dollar 50.
1294
00:53:16.715 --> 00:53:19.263
If we reach unanimity in blue, I only get 50 cents.
1295
00:53:19.263 --> 00:53:21.537
And your incentives might be exactly the opposite.
1296
00:53:21.537 --> 00:53:23.137
So you have two types of players now,
1297
00:53:23.137 --> 00:53:25.259
depending on their higher payoff color.
1298
00:53:25.259 --> 00:53:28.466
So now there's this tension between our need
1299
00:53:28.466 --> 00:53:31.404
to be unanimous, but the fact that, of course,
1300
00:53:31.404 --> 00:53:33.786
each one of us would rather reach unanimity
1301
00:53:33.786 --> 00:53:37.709
so the payoff, the color that gets the higher payoff.
1302
00:53:37.709 --> 00:53:39.595
And so now the high level experimental design
1303
00:53:39.595 --> 00:53:42.249
falls not just in choice of network structure
1304
00:53:42.249 --> 00:53:45.386
but actually how you arrange these two types of players
1305
00:53:45.386 --> 00:53:46.927
on the network structure.
1306
00:53:46.927 --> 00:53:48.260
And of course you can do all kinds
1307
00:53:48.260 --> 00:53:50.315
of interesting things there.
1308
00:53:50.315 --> 00:53:52.871
So in particular, just look at this
1309
00:53:52.871 --> 00:53:55.034
far right-hand column, okay?
1310
00:53:55.034 --> 00:53:56.368
So we ran a series of experiments
1311
00:53:56.368 --> 00:53:58.837
that we called the minority power experiments.
1312
00:53:58.837 --> 00:54:00.705
In these experiments, we generated networks
1313
00:54:00.705 --> 00:54:03.539
with 36 vertices via preferential attachment,
1314
00:54:03.539 --> 00:54:05.919
which is a well-studied stochastic model.
1315
00:54:05.919 --> 00:54:08.339
And the salient property of preferential attachment
1316
00:54:08.339 --> 00:54:11.337
is if you look at the distribution of connectivity,
1317
00:54:11.337 --> 00:54:15.137
the histogram of how many neighbors each vertex has,
1318
00:54:15.137 --> 00:54:16.845
it has a very long tail.
1319
00:54:16.845 --> 00:54:19.134
So you get a small number of vertices
1320
00:54:19.134 --> 00:54:20.853
which have a number of connections
1321
00:54:20.853 --> 00:54:23.249
which is much, much higher than the average,
1322
00:54:23.249 --> 00:54:25.390
these sort of connectors or hubs, okay?
1323
00:54:25.390 --> 00:54:28.682
So what we did was we as in this network,
1324
00:54:28.682 --> 00:54:30.237
we took a network and generated it
1325
00:54:30.237 --> 00:54:31.721
according to preferential attachment,
1326
00:54:31.721 --> 00:54:35.316
and we chose the incentive so that the vast majority
1327
00:54:35.316 --> 00:54:37.743
of the subjects had a higher payoff with blue.
1328
00:54:37.743 --> 00:54:39.597
So in this case, 30 out of the 30 subjects
1329
00:54:39.597 --> 00:54:41.648
get a higher payoff for blue, and only six
1330
00:54:41.648 --> 00:54:43.719
get a higher payoff for red.
1331
00:54:43.719 --> 00:54:46.970
But the red-preferring subjects
1332
00:54:46.970 --> 00:54:48.763
were not chosen to be arbitrary.
1333
00:54:48.763 --> 00:54:50.245
They were chosen to be the six vertices
1334
00:54:50.245 --> 00:54:52.984
with the highest degree, with the most connectivity.
1335
00:54:52.984 --> 00:54:56.707
So this is a stylized test of the question
1336
00:54:56.707 --> 00:55:00.711
of whether a small but well-connected minority
1337
00:55:00.711 --> 00:55:04.856
can systematic impose its preference on the majority.
1338
00:55:04.856 --> 00:55:06.972
And notice that it really is an imposition here,
1339
00:55:06.972 --> 00:55:09.333
because the maximum social welfare solution
1340
00:55:09.333 --> 00:55:11.477
in this network, of course, is everybody plays blue
1341
00:55:11.477 --> 00:55:14.044
because then 30 people get their higher payoff
1342
00:55:14.044 --> 00:55:15.548
and only six get their lower,
1343
00:55:15.548 --> 00:55:16.814
rather than six getting their higher
1344
00:55:16.814 --> 00:55:19.564
and 30 getting their lower, okay?
1345
00:55:20.608 --> 00:55:24.171
So the answer to this question is an overwhelming yes.
1346
00:55:24.171 --> 00:55:28.584
Out of the 27 experiments of this kind that we ran,
1347
00:55:28.584 --> 00:55:30.968
24 of them reached unanimity, which was much higher
1348
00:55:30.968 --> 00:55:34.491
than the overall rate over the entire experimental session,
1349
00:55:34.491 --> 00:55:36.000
which includes lots of other network structures
1350
00:55:36.000 --> 00:55:38.396
that I'm not describing.
1351
00:55:38.396 --> 00:55:40.629
So the vast majority of these experiments
1352
00:55:40.629 --> 00:55:42.140
reached unanimity.
1353
00:55:42.140 --> 00:55:45.616
Every single one of those 24 successful experiments
1354
00:55:45.616 --> 00:55:49.783
converged to unanimity to the minority preference.
1355
00:55:53.200 --> 00:55:55.355
And if you think about it, one of the things
1356
00:55:55.355 --> 00:55:56.699
that's kind of surprising about this
1357
00:55:56.699 --> 00:55:59.660
is again, this well-connected, powerful minority
1358
00:55:59.660 --> 00:56:01.660
has no particular reason to think
1359
00:56:01.660 --> 00:56:03.408
that they have a special position.
1360
00:56:03.408 --> 00:56:05.002
And in fact because they're well connected
1361
00:56:05.002 --> 00:56:06.710
and therefore have a very good sample
1362
00:56:06.710 --> 00:56:08.413
of behavior in the rest of the network,
1363
00:56:08.413 --> 00:56:10.092
what happens in these experiments is that
1364
00:56:10.092 --> 00:56:13.037
at the very beginning of course what everybody does
1365
00:56:13.037 --> 00:56:15.211
is initially select their higher payoff color.
1366
00:56:15.211 --> 00:56:17.603
This is, they always do this.
1367
00:56:17.603 --> 00:56:20.568
And so these well connected red players will wake up
1368
00:56:20.568 --> 00:56:23.768
and see, oh my god, the vast majority of the players
1369
00:56:23.768 --> 00:56:25.804
have a preference for blue.
1370
00:56:25.804 --> 00:56:28.311
And of course it's quite natural for such a player
1371
00:56:28.311 --> 00:56:31.271
to shortly thereafter acquiesce to blue.
1372
00:56:31.271 --> 00:56:35.068
Yet somehow, despite this, the dynamics always,
1373
00:56:35.068 --> 00:56:37.969
cause these are actual screenshots from stages of play
1374
00:56:37.969 --> 00:56:40.113
in an experiment, the dynamics always end up
1375
00:56:40.113 --> 00:56:43.537
coming back to all red, eventually.
1376
00:56:43.537 --> 00:56:45.208
And to get to this earlier question,
1377
00:56:45.208 --> 00:56:50.053
so to me the fact that this result can be arrived at
1378
00:56:50.053 --> 00:56:51.833
purely from network structure,
1379
00:56:51.833 --> 00:56:54.662
without any language, explicit sort of language
1380
00:56:54.662 --> 00:56:57.988
or other types of non-network influences
1381
00:56:57.988 --> 00:57:01.296
is sort of worth identifying when debating things like
1382
00:57:01.296 --> 00:57:05.754
the ways in which political influence spreads
1383
00:57:05.754 --> 00:57:09.587
or people influence voting in social networks.
1384
00:57:10.946 --> 00:57:11.823
Do I have like two minutes?
1385
00:57:11.823 --> 00:57:12.656
Sure.
1386
00:57:12.656 --> 00:57:13.489
Okay, so
1387
00:57:16.221 --> 00:57:19.888
in all the experiments we ran up until 2011,
1388
00:57:22.014 --> 00:57:24.915
most of which i haven't talked about here,
1389
00:57:24.915 --> 00:57:27.192
human subject performance was remarkably good
1390
00:57:27.192 --> 00:57:29.020
across all the network structures
1391
00:57:29.020 --> 00:57:31.108
and all the different tasks for the game subjects.
1392
00:57:31.108 --> 00:57:33.432
And of course every one of these experiments
1393
00:57:33.432 --> 00:57:34.861
is apples and oranges.
1394
00:57:34.861 --> 00:57:36.861
It's hard to make direct comparisons.
1395
00:57:36.861 --> 00:57:40.877
But they are unified by a single measure,
1396
00:57:40.877 --> 00:57:42.259
which is money, okay?
1397
00:57:42.259 --> 00:57:45.009
So here's a way of quantifying performance, right?
1398
00:57:45.009 --> 00:57:47.014
In every single experiment I run,
1399
00:57:47.014 --> 00:57:48.874
I know what the network structure was,
1400
00:57:48.874 --> 00:57:50.576
I know what all the incentives were.
1401
00:57:50.576 --> 00:57:53.264
So I can compute offline what would have been
1402
00:57:53.264 --> 00:57:54.722
the max welfare solution.
1403
00:57:54.722 --> 00:57:57.512
What would have been the configuration subjects play
1404
00:57:57.512 --> 00:57:59.862
that would have maximized the payoffs
1405
00:57:59.862 --> 00:58:02.320
I had to make to the subjects?
1406
00:58:02.320 --> 00:58:04.771
And I can sum that quantity up over every little experiment
1407
00:58:04.771 --> 00:58:07.821
I've ever done and put it in the denominator.
1408
00:58:07.821 --> 00:58:09.319
And in the numerator, I can sum up
1409
00:58:09.319 --> 00:58:10.970
the actual amount of money that they made
1410
00:58:10.970 --> 00:58:13.210
in all of the experiments of all time.
1411
00:58:13.210 --> 00:58:15.544
And I can look at that ratio as sort of a measure
1412
00:58:15.544 --> 00:58:16.942
of performance or efficiency.
1413
00:58:16.942 --> 00:58:18.527
That number over all of the experiments
1414
00:58:18.527 --> 00:58:22.720
that we ran up til 2011 was close to 90%.
1415
00:58:22.720 --> 00:58:24.370
It's about 0.88, okay?
1416
00:58:24.370 --> 00:58:27.022
So only 10% of the money available to them
1417
00:58:27.022 --> 00:58:29.251
in principle was left on the table
1418
00:58:29.251 --> 00:58:31.738
by the subjects across all of these experiments.
1419
00:58:31.738 --> 00:58:35.905
Now, of the many artificialities in these experiments,
1420
00:58:38.223 --> 00:58:40.564
the one that always bothered me the most
1421
00:58:40.564 --> 00:58:42.488
was not really the lack of language
1422
00:58:42.488 --> 00:58:45.673
but the fact that we were exogenously imposing
1423
00:58:45.673 --> 00:58:47.716
the network structure on the subjects
1424
00:58:47.716 --> 00:58:50.814
rather than letting the network structure evolve
1425
00:58:50.814 --> 00:58:54.195
or be built by the subjects themselves.
1426
00:58:54.195 --> 00:58:58.028
So there is a small but interesting literature
1427
00:58:59.220 --> 00:59:01.502
in organizational behavior and elsewhere
1428
00:59:01.502 --> 00:59:03.809
and also many popular articles
1429
00:59:03.809 --> 00:59:06.226
that suggest that, well, if you really want
1430
00:59:06.226 --> 00:59:08.528
an organization of people, you wanna task them
1431
00:59:08.528 --> 00:59:10.528
to solve some problem,
1432
00:59:10.528 --> 00:59:12.847
you shouldn't organize them yourself
1433
00:59:12.847 --> 00:59:15.091
in some, say, hierarchical fashion.
1434
00:59:15.091 --> 00:59:16.867
You should let them self-organize
1435
00:59:16.867 --> 00:59:18.670
in service of solving the task.
1436
00:59:18.670 --> 00:59:21.280
So especially during the dot com era, right,
1437
00:59:21.280 --> 00:59:24.888
there were lots of cases where consulting companies
1438
00:59:24.888 --> 00:59:27.477
were hired to go into a large corporation,
1439
00:59:27.477 --> 00:59:31.428
and they would study who communicated with whom.
1440
00:59:31.428 --> 00:59:32.506
They'd look at email records,
1441
00:59:32.506 --> 00:59:33.660
they talked to employees,
1442
00:59:33.660 --> 00:59:35.777
they documented who talked to whom.
1443
00:59:35.777 --> 00:59:38.081
And from that, invariably the result of this
1444
00:59:38.081 --> 00:59:41.156
was the consulting company kind of go in and say,
1445
00:59:41.156 --> 00:59:43.695
well, here's the org chart you gave me
1446
00:59:43.695 --> 00:59:45.149
at the start of the project.
1447
00:59:45.149 --> 00:59:48.149
And here's the network of which work
1448
00:59:49.230 --> 00:59:53.543
and communication actually take place in your organization.
1449
00:59:53.543 --> 00:59:55.896
And somehow the suggestion of this literature
1450
00:59:55.896 --> 00:59:58.246
is that the latter is better than the former, right?
1451
00:59:58.246 --> 01:00:02.133
I mean, invariably these studies would identify,
1452
01:00:02.133 --> 01:00:03.854
for instance, somebody at the organization
1453
01:00:03.854 --> 01:00:06.008
who's at the bottom of the org chart
1454
01:00:06.008 --> 01:00:08.163
in an apparently unimportant role
1455
01:00:08.163 --> 01:00:10.663
who turns out to be crucial because they know everybody
1456
01:00:10.663 --> 01:00:13.112
or they know people well in two different divisions,
1457
01:00:13.112 --> 01:00:14.517
and so on and so forth.
1458
01:00:14.517 --> 01:00:15.774
So I was kind of thinking, well,
1459
01:00:15.774 --> 01:00:18.607
given that, isn't it kind of weird
1460
01:00:19.973 --> 01:00:23.898
that I'm picking the networks separate from the task
1461
01:00:23.898 --> 01:00:25.331
and sort of saying, here's the network,
1462
01:00:25.331 --> 01:00:27.544
here's the task, go solve it.
1463
01:00:27.544 --> 01:00:30.106
Shouldn't subjects do better if I let them
1464
01:00:30.106 --> 01:00:32.571
build the network in service of the task?
1465
01:00:32.571 --> 01:00:34.323
Okay, so we tried that.
1466
01:00:34.323 --> 01:00:37.104
And I'll be quick in the interest of time.
1467
01:00:37.104 --> 01:00:38.863
The task we gave them was exactly
1468
01:00:38.863 --> 01:00:41.592
the bias voting task that I just described.
1469
01:00:41.592 --> 01:00:44.148
The goal is reach unanimity of color,
1470
01:00:44.148 --> 01:00:46.748
but you still have a preference for which color you reach.
1471
01:00:46.748 --> 01:00:49.467
But now the twist is that at the beginning
1472
01:00:49.467 --> 01:00:51.709
of the experiment, there's no network at all.
1473
01:00:51.709 --> 01:00:54.203
Every vertex is an isolated singleton.
1474
01:00:54.203 --> 01:00:55.729
So at the beginning of this experiment,
1475
01:00:55.729 --> 01:00:57.966
you can't see anybody's color but yourself.
1476
01:00:57.966 --> 01:01:01.816
Your GUI looks like you in the middle of nowhere, okay?
1477
01:01:01.816 --> 01:01:04.400
But throughout the experiment, anybody who wanted to
1478
01:01:04.400 --> 01:01:08.627
could choose to purchase edges to other players
1479
01:01:08.627 --> 01:01:10.647
at a cost that would be subtracted
1480
01:01:10.647 --> 01:01:12.576
from any eventual earnings, right?
1481
01:01:12.576 --> 01:01:16.908
So you're trying to solve this coordination problem,
1482
01:01:16.908 --> 01:01:19.979
this bias voting problem, but you need
1483
01:01:19.979 --> 01:01:22.452
to sort of see what other people are choosing.
1484
01:01:22.452 --> 01:01:24.595
And so we need to buy some edges.
1485
01:01:24.595 --> 01:01:26.445
So there was basically a price for edges,
1486
01:01:26.445 --> 01:01:27.963
we'd vary the price for edges,
1487
01:01:27.963 --> 01:01:30.968
and basically the only information you have
1488
01:01:30.968 --> 01:01:33.453
in selecting who to build an edge to
1489
01:01:33.453 --> 01:01:35.539
was what their current degree was,
1490
01:01:35.539 --> 01:01:37.228
how many neighbors they currently had,
1491
01:01:37.228 --> 01:01:40.823
and also what their shortest path distance was to you
1492
01:01:40.823 --> 01:01:42.832
in the network at the current moment.
1493
01:01:42.832 --> 01:01:45.952
So unless the GUI lets you do things like,
1494
01:01:45.952 --> 01:01:48.491
well, I'd like to buy an edge to somebody
1495
01:01:48.491 --> 01:01:50.297
who's currently far from me in the network
1496
01:01:50.297 --> 01:01:52.083
and has a lot of neighbors.
1497
01:01:52.083 --> 01:01:54.445
Maybe on the logic that that individual is aggregating
1498
01:01:54.445 --> 01:01:56.232
information in another part of the network
1499
01:01:56.232 --> 01:01:58.412
that you could use in sort of solving
1500
01:01:58.412 --> 01:01:59.780
this coordination problem.
1501
01:01:59.780 --> 01:02:02.205
Or you might choose, you might say,
1502
01:02:02.205 --> 01:02:04.401
well, look, time is running out in this experiment.
1503
01:02:04.401 --> 01:02:06.541
Here's this vertex that has very low degree.
1504
01:02:06.541 --> 01:02:08.833
I'm gonna buy an edge to them, not for information,
1505
01:02:08.833 --> 01:02:11.166
but on the logic that maybe I'll influence them
1506
01:02:11.166 --> 01:02:14.613
in their color choice towards my favorable direction.
1507
01:02:14.613 --> 01:02:17.240
Maybe if the vote is very close for instance.
1508
01:02:17.240 --> 01:02:20.816
So the system gave you some salient information
1509
01:02:20.816 --> 01:02:23.870
for the problem at hand in selecting
1510
01:02:23.870 --> 01:02:25.763
what edges you would buy.
1511
01:02:25.763 --> 01:02:29.347
And long story, so this is what the GUI looked like.
1512
01:02:29.347 --> 01:02:31.897
This is the same GUI for bias voting,
1513
01:02:31.897 --> 01:02:34.096
except now when you buy an edge, right,
1514
01:02:34.096 --> 01:02:36.860
a new vertex will sort of animate over here
1515
01:02:36.860 --> 01:02:38.953
and be incorporated in your network neighborhood.
1516
01:02:38.953 --> 01:02:41.302
By the way, edge purchase was unilateral,
1517
01:02:41.302 --> 01:02:43.508
but edge benefits were bilateral.
1518
01:02:43.508 --> 01:02:45.493
So if I buy an edge to you, I pay for it,
1519
01:02:45.493 --> 01:02:48.172
but then you get to see my color as well, okay?
1520
01:02:48.172 --> 01:02:49.756
And so here's this GUI at the bottom
1521
01:02:49.756 --> 01:02:52.073
which shows you for all for your non-neighbors
1522
01:02:52.073 --> 01:02:53.919
what their current distance is to you
1523
01:02:53.919 --> 01:02:55.271
and what their degree is.
1524
01:02:55.271 --> 01:02:56.680
And you can kind of click on one of these
1525
01:02:56.680 --> 01:02:58.528
and then it would be incorporated
1526
01:02:58.528 --> 01:03:00.054
into your network neighborhood.
1527
01:03:00.054 --> 01:03:02.054
Okay, this is a high-level one, and I'm rushing,
1528
01:03:02.054 --> 01:03:03.810
but that's a good high-level idea.
1529
01:03:03.810 --> 01:03:06.477
So the punchline of this is that
1530
01:03:09.151 --> 01:03:10.875
this is the first task we gave subjects
1531
01:03:10.875 --> 01:03:14.345
that they were really bad at, okay?
1532
01:03:14.345 --> 01:03:17.147
And they did much, much worse in these experiments
1533
01:03:17.147 --> 01:03:19.218
than they did in bias voting experiments
1534
01:03:19.218 --> 01:03:22.939
where we exogenously imposed the wacky artificial networks,
1535
01:03:22.939 --> 01:03:24.770
deliberately setting tensions up
1536
01:03:24.770 --> 01:03:27.139
between a minority and a majority.
1537
01:03:27.139 --> 01:03:29.372
When we let the subject build the network themselves,
1538
01:03:29.372 --> 01:03:31.093
they did a terrible job.
1539
01:03:31.093 --> 01:03:34.325
And if you remember this 90% efficiency figure I gave,
1540
01:03:34.325 --> 01:03:37.769
here the figure was below, depending on the exact setting,
1541
01:03:37.769 --> 01:03:41.533
well below 50% all the way down to 35%.
1542
01:03:41.533 --> 01:03:45.025
And just to finish with an interesting punchline,
1543
01:03:45.025 --> 01:03:47.938
you know, we're so alarmed by these experiments.
1544
01:03:47.938 --> 01:03:49.475
We had three sessions planned.
1545
01:03:49.475 --> 01:03:51.577
The first two were quite similar.
1546
01:03:51.577 --> 01:03:53.924
And after seeing how poorly they did
1547
01:03:53.924 --> 01:03:55.378
on those first two sessions, we decided
1548
01:03:55.378 --> 01:03:56.885
we have to scrap the third session
1549
01:03:56.885 --> 01:03:59.512
and investigate why they're doing so poorly
1550
01:03:59.512 --> 01:04:01.179
at this task because
1551
01:04:02.149 --> 01:04:03.616
here are three possible reasons
1552
01:04:03.616 --> 01:04:06.074
why they were poor at this task.
1553
01:04:06.074 --> 01:04:08.533
One is a simple information overload.
1554
01:04:08.533 --> 01:04:11.427
They can solve bias voting, and they can build
1555
01:04:11.427 --> 01:04:14.160
good networks for solving it,
1556
01:04:14.160 --> 01:04:16.334
but if they have to do those same tasks together,
1557
01:04:16.334 --> 01:04:18.467
in a limited period of time, it's just too hard
1558
01:04:18.467 --> 01:04:20.877
to think about building a good network for the task
1559
01:04:20.877 --> 01:04:23.489
and also solving the problem on the fly.
1560
01:04:23.489 --> 01:04:26.007
It's kind of like trying to build a plane and fly it.
1561
01:04:26.007 --> 01:04:27.505
And they have a very short time to do it.
1562
01:04:27.505 --> 01:04:30.403
So maybe it's not that they can't solve this problem well
1563
01:04:30.403 --> 01:04:32.014
but they just need a little bit more time.
1564
01:04:32.014 --> 01:04:33.686
That's sort of hypothesis one.
1565
01:04:33.686 --> 01:04:37.894
Hypothesis two is sort of an incentives reason,
1566
01:04:37.894 --> 01:04:41.311
which is, so let's take a common example.
1567
01:04:42.327 --> 01:04:45.757
Suppose you start off with a dollar 50 for unanimity
1568
01:04:45.757 --> 01:04:47.907
to red and 50 cents for blue.
1569
01:04:47.907 --> 01:04:51.297
And suppose you then bought 45 cents of edges.
1570
01:04:51.297 --> 01:04:54.253
So now your adjusted payoffs are a dollar five
1571
01:04:54.253 --> 01:04:55.575
and five cents.
1572
01:04:55.575 --> 01:04:57.628
So maybe at that point you're like, you know what?
1573
01:04:57.628 --> 01:05:00.294
Five cents just isn't worth it to me.
1574
01:05:00.294 --> 01:05:03.694
I'm gonna hold out, I'm gonna be much more stubborn
1575
01:05:03.694 --> 01:05:05.725
than I was with my original incentives.
1576
01:05:05.725 --> 01:05:09.389
And you might see, and furthermore,
1577
01:05:09.389 --> 01:05:11.472
I paid 45 cents on edges.
1578
01:05:12.861 --> 01:05:15.138
So I'm entitled, I'm more entitled
1579
01:05:15.138 --> 01:05:17.203
to have my higher payoff color
1580
01:05:17.203 --> 01:05:19.630
because I contributed to this collective resource
1581
01:05:19.630 --> 01:05:22.234
that we needed to build in order to solve this problem.
1582
01:05:22.234 --> 01:05:25.278
So maybe they were building good networks,
1583
01:05:25.278 --> 01:05:27.009
but once the networks were built,
1584
01:05:27.009 --> 01:05:28.059
and of course these two things
1585
01:05:28.059 --> 01:05:30.207
are going on simultaneously, maybe at that point
1586
01:05:30.207 --> 01:05:32.479
some fraction of the population has sort of
1587
01:05:32.479 --> 01:05:35.150
painted themselves into a corner of stubbornness
1588
01:05:35.150 --> 01:05:37.457
and then we were failing for that reason.
1589
01:05:37.457 --> 01:05:40.521
And then option three is that, no, no, no,
1590
01:05:40.521 --> 01:05:42.895
actually people, when given the ability
1591
01:05:42.895 --> 01:05:44.977
to build these networks, just build networks
1592
01:05:44.977 --> 01:05:47.963
on which this problem is hard, okay?
1593
01:05:47.963 --> 01:05:50.379
And luckily there's an easy way of testing
1594
01:05:50.379 --> 01:05:53.792
these three hypotheses, which is to take the networks
1595
01:05:53.792 --> 01:05:56.541
built by the first two sessions' subjects,
1596
01:05:56.541 --> 01:05:59.543
on which they failed to reach unanimity,
1597
01:05:59.543 --> 01:06:03.289
and giving those networks as fixed exogenous networks
1598
01:06:03.289 --> 01:06:04.908
to a third group of subjects
1599
01:06:04.908 --> 01:06:07.669
who only have to solve the bias voting problem
1600
01:06:07.669 --> 01:06:12.127
using the original incentives pre-edge purchases.
1601
01:06:12.127 --> 01:06:15.387
So you just take these apparently hard networks,
1602
01:06:15.387 --> 01:06:17.210
give them to a third set of subjects,
1603
01:06:17.210 --> 01:06:20.716
and restore the dollar five and five cents
1604
01:06:20.716 --> 01:06:23.463
back to a dollar 50, 50 cents.
1605
01:06:23.463 --> 01:06:24.755
And now you run an experiment
1606
01:06:24.755 --> 01:06:26.448
with no edge purchases whatsoever.
1607
01:06:26.448 --> 01:06:28.904
So it's just like the original bias voting experiments
1608
01:06:28.904 --> 01:06:31.321
but on networks built by previous groups
1609
01:06:31.321 --> 01:06:32.613
of human subjects.
1610
01:06:32.613 --> 01:06:35.081
And that gave the worst performance of all.
1611
01:06:35.081 --> 01:06:35.963
(laughing)
1612
01:06:35.963 --> 01:06:36.903
35%.
1613
01:06:36.903 --> 01:06:39.371
So the conclusion seems to be that
1614
01:06:39.371 --> 01:06:41.433
given the ability to build the network
1615
01:06:41.433 --> 01:06:44.504
on which they solve the task that they clearly understood,
1616
01:06:44.504 --> 01:06:45.916
they ended up building a network
1617
01:06:45.916 --> 01:06:49.050
that was hard for human subjects.
1618
01:06:49.050 --> 01:06:50.259
So I'm out of time.
1619
01:06:50.259 --> 01:06:52.453
I'll just put the conclusions up for you to read.
1620
01:06:52.453 --> 01:06:54.370
And I'll hang out and take any questions.
1621
01:06:54.370 --> 01:06:55.203
But thank you.
1622
01:06:55.203 --> 01:06:57.453
(applause)
1623
01:07:03.481 --> 01:07:06.898
(person talking quietly)
1624
01:07:16.782 --> 01:07:18.862
If I scaled these experiments up?
1625
01:07:18.862 --> 01:07:22.385
So first of all as a general matter,
1626
01:07:22.385 --> 01:07:24.907
one thing that, it is from some of these years,
1627
01:07:24.907 --> 01:07:28.493
is that if you ask me a question about,
1628
01:07:28.493 --> 01:07:30.802
do my results generalize or what will happen
1629
01:07:30.802 --> 01:07:33.208
if I try something that wasn't literally
1630
01:07:33.208 --> 01:07:37.365
what I tried, the best answer is I have no idea.
1631
01:07:37.365 --> 01:07:39.032
I mean, I think yes, it is.
1632
01:07:39.032 --> 01:07:43.393
But behavioral experiments are so sensitive
1633
01:07:43.393 --> 01:07:46.583
to so many things that I'm extremely reluctant
1634
01:07:46.583 --> 01:07:49.666
to try to make large generalizations.
1635
01:07:50.978 --> 01:07:53.575
Now, the question is a good one, though.
1636
01:07:53.575 --> 01:07:55.558
And I'd like to scale these experiments up.
1637
01:07:55.558 --> 01:07:57.946
And I think there's a lot of methodological problems
1638
01:07:57.946 --> 01:08:00.453
with doing that, at least in terms of doing so
1639
01:08:00.453 --> 01:08:04.137
in a way that makes it directly comparable.
1640
01:08:04.137 --> 01:08:07.031
I believe that human subjects could solve,
1641
01:08:07.031 --> 01:08:09.366
let's say very, very large-scale graph colorings
1642
01:08:09.366 --> 01:08:11.449
in short amounts of time.
1643
01:08:12.677 --> 01:08:16.005
But they would look like different experiments.
1644
01:08:16.005 --> 01:08:18.783
So first of all you have to worry about,
1645
01:08:18.783 --> 01:08:19.656
I mean, many things.
1646
01:08:19.656 --> 01:08:23.156
One is, in our experiments we sort of know
1647
01:08:24.907 --> 01:08:26.047
what everybody's doing.
1648
01:08:26.047 --> 01:08:28.041
We know that they're attending to the task.
1649
01:08:28.041 --> 01:08:29.732
In an online experiment, you have to deal with
1650
01:08:29.732 --> 01:08:31.877
people who sign up to do the experiment
1651
01:08:31.877 --> 01:08:36.579
and then don't play or go away or aren't attending.
1652
01:08:36.579 --> 01:08:38.829
You have much less control.
1653
01:08:39.891 --> 01:08:41.490
You have to of course worry about things
1654
01:08:41.490 --> 01:08:43.257
like collusion, like is there one vertex
1655
01:08:43.257 --> 01:08:46.933
or one algorithm who's playing multiple vertices.
1656
01:08:46.933 --> 01:08:51.100
So I think the path is clear to do large-scale versions
1657
01:08:52.259 --> 01:08:54.956
of these things, but I need to design them
1658
01:08:54.956 --> 01:08:55.789
to be efficient.
1659
01:08:55.789 --> 01:08:57.877
So for instance, people playing attrition
1660
01:08:57.877 --> 01:08:59.735
and they have better ideas instead of having
1661
01:08:59.735 --> 01:09:03.194
a single human subject play the same vertex
1662
01:09:03.194 --> 01:09:06.159
til you reach your solution.
1663
01:09:06.159 --> 01:09:09.209
As subjects, if we have a graph coloring online,
1664
01:09:09.209 --> 01:09:12.426
say, well, anybody who wants to, as soon as you arrive,
1665
01:09:12.426 --> 01:09:15.367
I'm gonna put you down on some vertex, let's say,
1666
01:09:15.367 --> 01:09:17.539
that has a current color and let you make
1667
01:09:17.539 --> 01:09:21.704
a color change, and there'll be a $10,000 prize
1668
01:09:21.704 --> 01:09:25.003
for that individual who makes the final change
1669
01:09:25.003 --> 01:09:27.178
that causes there to be a global solution.
1670
01:09:27.178 --> 01:09:28.712
So that's the way I would do it in practice,
1671
01:09:28.712 --> 01:09:31.468
but of course now you don't have any notion of
1672
01:09:31.468 --> 01:09:34.936
individuals kind of attending to a vertex over time,
1673
01:09:34.936 --> 01:09:37.750
working out a solution in their local neighborhood,
1674
01:09:37.750 --> 01:09:38.583
et cetera.
1675
01:09:40.824 --> 01:09:41.657
Yeah.
1676
01:09:42.910 --> 01:09:43.951
Could you say more about
1677
01:09:43.951 --> 01:09:47.055
the new theory that you were using?
1678
01:09:47.055 --> 01:09:49.722
I mean, this just goes back to
1679
01:09:51.840 --> 01:09:55.923
what I hinted at at the very beginning, which is,
1680
01:09:57.581 --> 01:09:59.378
and there's starting to be interesting work on this.
1681
01:09:59.378 --> 01:10:02.839
There's this fellow at U Mass who I met recently
1682
01:10:02.839 --> 01:10:04.124
who's doing this kind of work.
1683
01:10:04.124 --> 01:10:06.041
But the kind of idea is
1684
01:10:08.533 --> 01:10:11.897
let's take seriously the idea that we're in an era
1685
01:10:11.897 --> 01:10:15.517
where one of the resources that we have available
1686
01:10:15.517 --> 01:10:18.533
to us in designing algorithms for systems
1687
01:10:18.533 --> 01:10:19.909
is human labor.
1688
01:10:19.909 --> 01:10:22.690
And so you can, in the same way that
1689
01:10:22.690 --> 01:10:26.820
we have CPUs and we have memory and we have storage,
1690
01:10:26.820 --> 01:10:29.044
we sort of know the strengths of those devices,
1691
01:10:29.044 --> 01:10:30.579
and we know how they'll respond
1692
01:10:30.579 --> 01:10:33.115
under different circumstances.
1693
01:10:33.115 --> 01:10:35.018
We now have this much messier resource,
1694
01:10:35.018 --> 01:10:37.478
which is human cognitive abilities.
1695
01:10:37.478 --> 01:10:39.963
And the constraints on that are first of all
1696
01:10:39.963 --> 01:10:41.539
what kinds of things can people do
1697
01:10:41.539 --> 01:10:43.221
in reasonable amounts of time?
1698
01:10:43.221 --> 01:10:45.315
And what will they do for you,
1699
01:10:45.315 --> 01:10:47.154
and how do you incentivize them, right?
1700
01:10:47.154 --> 01:10:48.913
But we're sort of starting to see
1701
01:10:48.913 --> 01:10:51.393
outlines of answers to these things.
1702
01:10:51.393 --> 01:10:54.298
So one thing we know is that people
1703
01:10:54.298 --> 01:10:56.615
seem to respond to being paid money.
1704
01:10:56.615 --> 01:11:00.088
So Mechanical Turk is an example of a
1705
01:11:00.088 --> 01:11:03.901
computational resource that you can go pay money for
1706
01:11:03.901 --> 01:11:07.002
and get cycles out the subjects.
1707
01:11:07.002 --> 01:11:09.041
Gaming or entertainment seems to be
1708
01:11:09.041 --> 01:11:11.178
another way of motivating subjects.
1709
01:11:11.178 --> 01:11:14.113
So the high-level idea would be to
1710
01:11:14.113 --> 01:11:16.756
try to quantify what the tolerance is
1711
01:11:16.756 --> 01:11:19.017
and abilities of these resources are
1712
01:11:19.017 --> 01:11:20.512
under different circumstances
1713
01:11:20.512 --> 01:11:22.537
so that you can start to actually start to have
1714
01:11:22.537 --> 01:11:24.752
some kind of engineering theory about
1715
01:11:24.752 --> 01:11:27.398
how you would design a system.
1716
01:11:27.398 --> 01:11:29.370
So that's kind of what I mean by this idea
1717
01:11:29.370 --> 01:11:32.528
of crowdsourcing compiler, that it would be,
1718
01:11:32.528 --> 01:11:34.249
when I write a high-level programming language
1719
01:11:34.249 --> 01:11:35.788
these days, right, when I don't have to say,
1720
01:11:35.788 --> 01:11:38.160
well, which of these variables should you put in
1721
01:11:38.160 --> 01:11:40.726
for a fast register and what's okay
1722
01:11:40.726 --> 01:11:42.379
to just keep in the main memory?
1723
01:11:42.379 --> 01:11:47.308
And I don't say exactly how these should be implemented.
1724
01:11:47.308 --> 01:11:50.053
You could imagine a future where I still program
1725
01:11:50.053 --> 01:11:52.152
in some high-level language and the compiler is just
1726
01:11:52.152 --> 01:11:54.152
like, you know, what, for this thing,
1727
01:11:54.152 --> 01:11:56.825
what I think we should do is go from
1728
01:11:56.825 --> 01:11:59.669
in real time a population of people
1729
01:11:59.669 --> 01:12:01.241
on Amazon Mechanical Turk,
1730
01:12:01.241 --> 01:12:02.878
and this should be what we pay them,
1731
01:12:02.878 --> 01:12:04.829
and this should be how long we should wait
1732
01:12:04.829 --> 01:12:06.217
to get the answers from them
1733
01:12:06.217 --> 01:12:07.560
on these problems.
1734
01:12:07.560 --> 01:12:09.108
And this is how we should get a grade back.
1735
01:12:09.108 --> 01:12:10.597
And then for this other part of the problem,
1736
01:12:10.597 --> 01:12:12.847
the machine is gonna do it.
1737
01:12:15.437 --> 01:12:16.985
It's very blue sky, but I think
1738
01:12:16.985 --> 01:12:19.964
it's where we're headed, eventually.
1739
01:12:19.964 --> 01:12:23.205
I think it's a related question.
1740
01:12:23.205 --> 01:12:26.392
Do you ever experiment with intervention?
1741
01:12:26.392 --> 01:12:30.206
Where somebody says to a user, you should be blue.
1742
01:12:30.206 --> 01:12:32.137
No, we haven't.
1743
01:12:32.137 --> 01:12:33.672
And we haven't also experimented with
1744
01:12:33.672 --> 01:12:36.862
a related idea of mixing machines with people.
1745
01:12:36.862 --> 01:12:38.974
For instance in bias voting,
1746
01:12:38.974 --> 01:12:42.224
having algorithms play certain vertices
1747
01:12:43.524 --> 01:12:45.993
in order to try to effect certain outcomes.
1748
01:12:45.993 --> 01:12:48.043
Maybe in graph coloring, actually,
1749
01:12:48.043 --> 01:12:50.278
if you just have it peppered throughout,
1750
01:12:50.278 --> 01:12:53.566
just some heuristics that try to keep activity
1751
01:12:53.566 --> 01:12:56.799
going in areas where there were too many constraints
1752
01:12:56.799 --> 01:12:58.229
and the like.
1753
01:12:58.229 --> 01:13:00.312
So we haven't tried that.
1754
01:13:02.057 --> 01:13:05.589
So what's the times in these experiments?
1755
01:13:05.589 --> 01:13:08.179
They vary from five minutes for the very initial
1756
01:13:08.179 --> 01:13:11.594
graph coloring experiments down to
1757
01:13:11.594 --> 01:13:14.807
I think we were at two minutes for these bias voting
1758
01:13:14.807 --> 01:13:15.998
experiments.
1759
01:13:15.998 --> 01:13:17.713
Okay, how about the the one
1760
01:13:17.713 --> 01:13:21.180
where you allow the subjects to do the network?
1761
01:13:21.180 --> 01:13:22.372
Also two minutes.
1762
01:13:22.372 --> 01:13:23.205
Also two minutes?
1763
01:13:23.205 --> 01:13:24.357
Yeah, yeah.
1764
01:13:24.357 --> 01:13:27.353
So that's why this information overload hypothesis
1765
01:13:27.353 --> 01:13:30.248
came about that we wanted to refute,
1766
01:13:30.248 --> 01:13:32.412
which is saying, well, they did pretty good
1767
01:13:32.412 --> 01:13:33.956
at bias voting in two minutes, but now
1768
01:13:33.956 --> 01:13:35.791
we're giving them an additional task
1769
01:13:35.791 --> 01:13:37.158
and the same amount of time.
1770
01:13:37.158 --> 01:13:38.603
So maybe we blew it.
1771
01:13:38.603 --> 01:13:40.848
And that's what's making them do poorly.
1772
01:13:40.848 --> 01:13:44.566
So that's why we went through this third session.
1773
01:13:44.566 --> 01:13:45.712
Yes?
1774
01:13:45.712 --> 01:13:48.654
Have you thought about making the payment
1775
01:13:48.654 --> 01:13:50.512
for buying an edge bilateral,
1776
01:13:50.512 --> 01:13:52.227
maybe like for two parties?
1777
01:13:52.227 --> 01:13:53.060
No.
1778
01:13:53.060 --> 01:13:54.256
And that's only if they agree?
1779
01:13:54.256 --> 01:13:56.838
Yeah, I haven't thought about it.
1780
01:13:56.838 --> 01:13:59.004
That would be another even added complexity,
1781
01:13:59.004 --> 01:14:02.356
so now there's this sort of bargaining aspect as well.
1782
01:14:02.356 --> 01:14:03.939
Haven't tried that.
1783
01:14:06.905 --> 01:14:10.072
(man talking quietly)
1784
01:14:22.123 --> 01:14:24.991
So we did that, too, actually.
1785
01:14:24.991 --> 01:14:25.979
I didn't mention it.
1786
01:14:25.979 --> 01:14:28.646
We did that too, and we also did
1787
01:14:30.841 --> 01:14:34.931
both of the above with allowing subjects
1788
01:14:34.931 --> 01:14:38.130
to have one edge purchase, just to make it
1789
01:14:38.130 --> 01:14:40.137
cognitively comparable to having
1790
01:14:40.137 --> 01:14:42.339
to take a little bit of value at first so they're not,
1791
01:14:42.339 --> 01:14:43.981
and basically none of it matters,
1792
01:14:43.981 --> 01:14:47.560
by which I mean performance in all these scenarios
1793
01:14:47.560 --> 01:14:49.871
is still very, very poor.
1794
01:14:49.871 --> 01:14:52.528
(man talking quietly)
1795
01:14:52.528 --> 01:14:54.410
Yeah, or depending on exactly which
1796
01:14:54.410 --> 01:14:57.128
cross-section we're taking, it might be somewhere
1797
01:14:57.128 --> 01:15:00.228
between 35% up to 45%.
1798
01:15:00.228 --> 01:15:02.675
But still way, way below
1799
01:15:02.675 --> 01:15:06.592
performance on exogenously determined networks.
1800
01:15:07.699 --> 01:15:09.567
I guess sort of my question is
1801
01:15:09.567 --> 01:15:11.902
on price estimate, just saying,
1802
01:15:11.902 --> 01:15:15.769
okay, we're gonna use 23 or five or whatever,
1803
01:15:15.769 --> 01:15:19.528
and you can buy extras, if that would be,
1804
01:15:19.528 --> 01:15:22.012
I mean, you mentioned some of the ways.
1805
01:15:22.012 --> 01:15:24.344
Yeah, it is the case, right, right.
1806
01:15:24.344 --> 01:15:27.253
So yeah, this is a good example of something
1807
01:15:27.253 --> 01:15:28.730
that's interesting that I think
1808
01:15:28.730 --> 01:15:31.452
you probably can't guess what would happen.
1809
01:15:31.452 --> 01:15:33.695
Cause that's very different.
1810
01:15:33.695 --> 01:15:35.197
And if you look at the data, right,
1811
01:15:35.197 --> 01:15:36.907
you will definitely see
1812
01:15:36.907 --> 01:15:39.903
that it's not like everybody bought
1813
01:15:39.903 --> 01:15:41.412
the same number of edges.
1814
01:15:41.412 --> 01:15:43.469
There's quite a bit of free riding going on.
1815
01:15:43.469 --> 01:15:45.809
So if you look at the distribution of edge purchases,
1816
01:15:45.809 --> 01:15:48.496
in every experiment there's some
1817
01:15:48.496 --> 01:15:50.996
healthy group of people that basically
1818
01:15:50.996 --> 01:15:53.321
bought no edges whatsoever.
1819
01:15:53.321 --> 01:15:56.108
And by the way, the other question one might wonder
1820
01:15:56.108 --> 01:15:58.331
about these experiments is, maybe people
1821
01:15:58.331 --> 01:16:01.316
are just cheap and didn't buy enough edges, right?
1822
01:16:01.316 --> 01:16:03.772
So and the minimum, right, we have to buy
1823
01:16:03.772 --> 01:16:05.365
a spanning group, right?
1824
01:16:05.365 --> 01:16:09.103
If we have two disconnected components,
1825
01:16:09.103 --> 01:16:12.149
there's no communication between those two,
1826
01:16:12.149 --> 01:16:14.460
there's just a pretty good likelihood
1827
01:16:14.460 --> 01:16:16.443
that even if each group reaches unanimity,
1828
01:16:16.443 --> 01:16:17.904
it's gonna be to opposite colors
1829
01:16:17.904 --> 01:16:18.978
and there won't be a game.
1830
01:16:18.978 --> 01:16:22.002
So we need to buy at least like 35 edges
1831
01:16:22.002 --> 01:16:24.343
in order to have global connectivity.
1832
01:16:24.343 --> 01:16:25.888
So you might wonder if maybe the problem with this,
1833
01:16:25.888 --> 01:16:28.251
you know, people didn't buy the networks,
1834
01:16:28.251 --> 01:16:30.281
so consistent with what I said,
1835
01:16:30.281 --> 01:16:31.814
they build hard networks.
1836
01:16:31.814 --> 01:16:33.450
Well, maybe the reason the networks were hard
1837
01:16:33.450 --> 01:16:36.635
is they were disconnected or maybe just incredibly sparse.
1838
01:16:36.635 --> 01:16:37.970
This is not true.
1839
01:16:37.970 --> 01:16:41.339
There's a great deal of edge purchasing that goes on.
1840
01:16:41.339 --> 01:16:45.506
And since you're sitting here, I'll show you a movie.
1841
01:16:46.635 --> 01:16:47.802
This shows you
1842
01:16:49.322 --> 01:16:53.304
an actual visualization of the edge purchases
1843
01:16:53.304 --> 01:16:56.054
and the color choices being made.
1844
01:16:56.907 --> 01:16:58.907
I think this is the one.
1845
01:17:00.394 --> 01:17:01.823
Okay, so here we are.
1846
01:17:01.823 --> 01:17:03.594
There are no edges yet.
1847
01:17:03.594 --> 01:17:05.449
All the players with the higher payoff for red
1848
01:17:05.449 --> 01:17:06.352
are at the bottom.
1849
01:17:06.352 --> 01:17:07.650
All the players with a higher payoff for blue
1850
01:17:07.650 --> 01:17:09.483
are at the top.
1851
01:17:09.483 --> 01:17:11.008
The outer ring, when it is animated,
1852
01:17:11.008 --> 01:17:13.495
will indicate the current color choice of that vertex.
1853
01:17:13.495 --> 01:17:15.058
And these inner half moons represent
1854
01:17:15.058 --> 01:17:17.616
your payoff for red and your payoff for blue.
1855
01:17:17.616 --> 01:17:20.207
And those shrink as you make edge purchases.
1856
01:17:20.207 --> 01:17:21.578
Basically the area of this thing
1857
01:17:21.578 --> 01:17:23.661
is proportional to your remaining payoffs
1858
01:17:23.661 --> 01:17:26.068
post your edge expenditures.
1859
01:17:26.068 --> 01:17:28.165
And this is actually an experiment
1860
01:17:28.165 --> 01:17:30.365
in which the edge costs were the highest.
1861
01:17:30.365 --> 01:17:33.302
So I think the edge costs here were like 25 cents.
1862
01:17:33.302 --> 01:17:35.376
So your getting payoffs sort of on the order
1863
01:17:35.376 --> 01:17:36.892
of a dollar 50, you know.
1864
01:17:36.892 --> 01:17:41.059
So you really can't buy many, the edges are expensive
1865
01:17:41.940 --> 01:17:45.161
relative to your overall payoff.
1866
01:17:45.161 --> 01:17:48.661
And this is actually real time, I believe.
1867
01:17:49.899 --> 01:17:52.250
So connectivity's not the problem here.
1868
01:17:52.250 --> 01:17:54.045
Very early in the experiment, we definitely have
1869
01:17:54.045 --> 01:17:56.933
global connectivity and much, much more than a span.
1870
01:17:56.933 --> 01:17:59.101
There's way more than 35 edges here.
1871
01:17:59.101 --> 01:18:01.933
You can see what I'm talking about here.
1872
01:18:01.933 --> 01:18:03.705
Here's somebody who's basically spent
1873
01:18:03.705 --> 01:18:06.007
almost as much as they can on edges.
1874
01:18:06.007 --> 01:18:07.821
By the way, we maintain the invariant
1875
01:18:07.821 --> 01:18:11.081
that you couldn't buy more edges than,
1876
01:18:11.081 --> 01:18:13.090
you couldn't buy edges to make your lower payoff
1877
01:18:13.090 --> 01:18:14.416
go negative, right?
1878
01:18:14.416 --> 01:18:18.398
You had to at least have a positive lower payoff.
1879
01:18:18.398 --> 01:18:20.030
So now we're done.
1880
01:18:20.030 --> 01:18:21.800
And this is a rather dense graph.
1881
01:18:21.800 --> 01:18:24.230
This is even when the edges were expensive.
1882
01:18:24.230 --> 01:18:27.123
So you can see the variation between people
1883
01:18:27.123 --> 01:18:30.096
who basically spent the maximum amount on edges
1884
01:18:30.096 --> 01:18:32.825
to other complete free riders over here.
1885
01:18:32.825 --> 01:18:33.883
(laughing)
1886
01:18:33.883 --> 01:18:35.535
And you can see that this is an experiment
1887
01:18:35.535 --> 01:18:37.242
that ended unsuccessfully.
1888
01:18:37.242 --> 01:18:38.953
Most people have adopted blue,
1889
01:18:38.953 --> 01:18:42.367
but there's still this group of red holdouts
1890
01:18:42.367 --> 01:18:46.534
that are preventing, nobody got payoffs in this experiment.
1891
01:18:51.350 --> 01:18:54.099
So what role did time play in this?
1892
01:18:54.099 --> 01:18:56.552
I would have thought that if this was real time,
1893
01:18:56.552 --> 01:18:58.343
everybody would have sat around
1894
01:18:58.343 --> 01:19:00.667
hoping somebody would have connected to them
1895
01:19:00.667 --> 01:19:02.106
so they wouldn't have had to pay.
1896
01:19:02.106 --> 01:19:05.148
Yeah, there's two minutes total.
1897
01:19:05.148 --> 01:19:06.576
And when those two minutes are up,
1898
01:19:06.576 --> 01:19:10.006
either you've reached unanimity or you haven't.
1899
01:19:10.006 --> 01:19:12.339
And you can see, right, that
1900
01:19:14.661 --> 01:19:16.911
you saw, I won't replay it,
1901
01:19:18.048 --> 01:19:21.717
even though kind of, there was a long period of time left,
1902
01:19:21.717 --> 01:19:23.937
people weren't waiting around to buy edges.
1903
01:19:23.937 --> 01:19:27.880
People were doing it rather quickly from the start.
1904
01:19:27.880 --> 01:19:29.760
So it doesn't seem like people
1905
01:19:29.760 --> 01:19:31.731
were kind playing the chicken game
1906
01:19:31.731 --> 01:19:33.632
where they're like, well, I'm not gonna build the edges.
1907
01:19:33.632 --> 01:19:36.330
I'm gonna see if everybody else builds the edges.
1908
01:19:36.330 --> 01:19:37.693
Only you do of course have free riders
1909
01:19:37.693 --> 01:19:39.470
who never bought edges.
1910
01:19:39.470 --> 01:19:40.361
In that last example,
1911
01:19:40.361 --> 01:19:41.997
was there any cards going around?
1912
01:19:41.997 --> 01:19:43.557
Did people have any idea that they
1913
01:19:43.557 --> 01:19:44.530
would just be people?
1914
01:19:44.530 --> 01:19:46.830
No, no, you just know how much time is left,
1915
01:19:46.830 --> 01:19:49.403
and you know that you haven't reached unanimity yet.
1916
01:19:49.403 --> 01:19:53.540
But of course, if you look at the network towards the end,
1917
01:19:53.540 --> 01:19:56.707
most players had a rather large degree
1918
01:19:57.918 --> 01:20:01.552
and to probably see, at least in their sampling
1919
01:20:01.552 --> 01:20:04.481
of neighbors, that there was at least one red and one blue.
1920
01:20:04.481 --> 01:20:07.438
So they might be able to guess from the proportion
1921
01:20:07.438 --> 01:20:09.756
that they were pretty close to unanimity
1922
01:20:09.756 --> 01:20:11.256
but not there yet.
1923
01:20:13.357 --> 01:20:14.422
Yes.
1924
01:20:14.422 --> 01:20:17.415
So I guess in most other games
1925
01:20:17.415 --> 01:20:21.083
that you're listing here, do you have the notion of
1926
01:20:21.083 --> 01:20:24.166
making the best chance from the game?
1927
01:20:26.886 --> 01:20:29.164
So if you have a sense about how much
1928
01:20:29.164 --> 01:20:32.664
chance was in a player, had the best shot,
1929
01:20:35.678 --> 01:20:40.072
is there even an option of choosing some...
1930
01:20:40.072 --> 01:20:43.072
Are you asking me what fraction of
1931
01:20:45.838 --> 01:20:48.171
the player across timestamps
1932
01:20:50.608 --> 01:20:52.804
such that that player is best responding
1933
01:20:52.804 --> 01:20:54.892
to their neighborhood at that moment?
1934
01:20:54.892 --> 01:20:57.833
Yes, and given the time the player has,
1935
01:20:57.833 --> 01:21:02.418
there's the notion that there is a best option, right,
1936
01:21:02.418 --> 01:21:04.896
so what percentage of the players-
1937
01:21:04.896 --> 01:21:06.933
So we haven't systematically studied that
1938
01:21:06.933 --> 01:21:09.072
across all experiments.
1939
01:21:09.072 --> 01:21:11.445
And in some cases it's not clear how to define that.
1940
01:21:11.445 --> 01:21:13.554
So the bias voting is a good example.
1941
01:21:13.554 --> 01:21:16.895
So my higher payoff is for red,
1942
01:21:16.895 --> 01:21:20.672
and 60% of my neighbors are playing blue.
1943
01:21:20.672 --> 01:21:22.672
What's my best response?
1944
01:21:25.399 --> 01:21:26.483
It's not clear, right?
1945
01:21:26.483 --> 01:21:29.370
Maybe I should continue to hold out for red
1946
01:21:29.370 --> 01:21:32.479
and cause everybody to come to my higher-payoff color,
1947
01:21:32.479 --> 01:21:37.335
or maybe I should acquiesce and play with the majority.
1948
01:21:37.335 --> 01:21:39.097
But by the way, you can formulate,
1949
01:21:39.097 --> 01:21:42.159
for all these games, something that we often do
1950
01:21:42.159 --> 01:21:46.049
is formulate some simple heuristic model for play
1951
01:21:46.049 --> 01:21:49.264
and ask how correlated it is with the various aspects
1952
01:21:49.264 --> 01:21:50.702
of the actual subject play.
1953
01:21:50.702 --> 01:21:54.057
So for instance in bias voting,
1954
01:21:54.057 --> 01:21:56.557
the natural thing to do is do,
1955
01:21:57.980 --> 01:22:01.286
let's scale my payoffs to a number between zero and one
1956
01:22:01.286 --> 01:22:04.553
that indicates my strength of preference for red, let's say.
1957
01:22:04.553 --> 01:22:07.465
So for instance 0.5 if my payoffs are equal.
1958
01:22:07.465 --> 01:22:09.437
If it's approaching one,
1959
01:22:09.437 --> 01:22:12.801
if my payoffs were a dollar 50 for red
1960
01:22:12.801 --> 01:22:15.547
and 50 cents for blue, this value would be
1961
01:22:15.547 --> 01:22:18.066
a dollar 50 divided by two dollars, okay?
1962
01:22:18.066 --> 01:22:20.392
So everybody gets a number between zero and one
1963
01:22:20.392 --> 01:22:22.815
that indicates their
1964
01:22:22.815 --> 01:22:25.123
relative preference for red.
1965
01:22:25.123 --> 01:22:28.306
And a natural thing to do is to say, well,
1966
01:22:28.306 --> 01:22:31.464
at each moment in time, suppose what each agent does
1967
01:22:31.464 --> 01:22:33.826
is to take that static number,
1968
01:22:33.826 --> 01:22:35.602
which has just to do with their payoffs,
1969
01:22:35.602 --> 01:22:37.663
and multiply it by the fraction of players
1970
01:22:37.663 --> 01:22:39.538
in their neighborhood playing red
1971
01:22:39.538 --> 01:22:43.545
and to normalize this quantity into a probability, okay?
1972
01:22:43.545 --> 01:22:46.841
So now I'm deliberately balancing two concerns.
1973
01:22:46.841 --> 01:22:50.272
One, if my payoff for red is very high,
1974
01:22:50.272 --> 01:22:53.053
that's gonna make this probability higher in general,
1975
01:22:53.053 --> 01:22:56.070
but I'm not completely ignoring my neighborhood, right?
1976
01:22:56.070 --> 01:22:58.104
If my payoff is higher for red
1977
01:22:58.104 --> 01:23:00.048
but the vast majority of my neighbors
1978
01:23:00.048 --> 01:23:01.960
were playing blue, that's gonna lower this probability
1979
01:23:01.960 --> 01:23:03.568
of playing red.
1980
01:23:03.568 --> 01:23:05.856
So you can take a model like that and run it
1981
01:23:05.856 --> 01:23:08.697
in simulation, independently on every vertex.
1982
01:23:08.697 --> 01:23:11.334
And let's say run it until you've reached unanimity
1983
01:23:11.334 --> 01:23:13.545
on each of the networks that we actually
1984
01:23:13.545 --> 01:23:16.027
gave the subjects in the experiments.
1985
01:23:16.027 --> 01:23:19.525
And you could ask, how long is the time to completion
1986
01:23:19.525 --> 01:23:21.614
of that simulation correlated,
1987
01:23:21.614 --> 01:23:22.915
how strongly is that correlated
1988
01:23:22.915 --> 01:23:26.052
with the actual subjects' time to unanimity?
1989
01:23:26.052 --> 01:23:28.532
And if you do that, these two numbers
1990
01:23:28.532 --> 01:23:29.892
are highly correlated.
1991
01:23:29.892 --> 01:23:32.514
The correlation coefficient is about 0.6
1992
01:23:32.514 --> 01:23:35.538
and so highly significant statistically.
1993
01:23:35.538 --> 01:23:38.819
And so it sort of explains 60% of the variants
1994
01:23:38.819 --> 01:23:40.208
of this model, okay?
1995
01:23:40.208 --> 01:23:42.875
And this is very typical, right?
1996
01:23:43.839 --> 01:23:48.104
It's often more conforming to some reasonable heuristic
1997
01:23:48.104 --> 01:23:52.595
that's captured, there's something real about it
1998
01:23:52.595 --> 01:23:56.124
that captures a lot of the variants in the subject play.
1999
01:23:56.124 --> 01:23:57.378
But it's not strong enough.
2000
01:23:57.378 --> 01:24:00.355
So 60% is a strong correlation coefficient,
2001
01:24:00.355 --> 01:24:02.451
but it's not enough to be predictive, right?
2002
01:24:02.451 --> 01:24:04.407
It's not like I can build a regression
2003
01:24:04.407 --> 01:24:07.538
from this simulation quantity and make an accurate
2004
01:24:07.538 --> 01:24:11.062
R kind of squared error of prediction
2005
01:24:11.062 --> 01:24:13.534
of this time the subjects take to complete.
2006
01:24:13.534 --> 01:24:16.769
It wouldn't be much better than random guessing.
2007
01:24:16.769 --> 01:24:20.331
So it's kind of interesting that you can find
2008
01:24:20.331 --> 01:24:23.437
heuristics that model something real,
2009
01:24:23.437 --> 01:24:25.604
even though not in detail.
2010
01:24:26.832 --> 01:24:29.588
I think getting to better, detailed models
2011
01:24:29.588 --> 01:24:31.755
is the next open question.
2012
01:24:33.544 --> 01:24:34.411
Yeah.
2013
01:24:34.411 --> 01:24:36.011
I was wondering if you ever included
2014
01:24:36.011 --> 01:24:38.511
some sort of model for selling
2015
01:24:41.001 --> 01:24:44.629
influence, so for instance if I buy an edge to you
2016
01:24:44.629 --> 01:24:47.232
then your color reverts to my color
2017
01:24:47.232 --> 01:24:51.149
or selling your fealty so, I buy an edge to you
2018
01:24:52.013 --> 01:24:54.564
and I see your connections, I want to buy an edge to you,
2019
01:24:54.564 --> 01:24:58.064
and I'm gonna sell my color to your color.
2020
01:25:00.096 --> 01:25:02.554
Yeah, that's an interesting idea.
2021
01:25:02.554 --> 01:25:04.810
So we're kind of looking at more,
2022
01:25:04.810 --> 01:25:06.367
even though there's money in our experiments,
2023
01:25:06.367 --> 01:25:09.273
we're looking at non-monetary forms of diffusion
2024
01:25:09.273 --> 01:25:12.207
or influence rather than direct monetary ones
2025
01:25:12.207 --> 01:25:13.790
where I'll pay you.
2026
01:25:15.063 --> 01:25:17.128
The closest we came to that were these independent
2027
01:25:17.128 --> 01:25:18.461
set experiments,
2028
01:25:19.931 --> 01:25:23.279
which I'll just briefly mention, since you asked.
2029
01:25:23.279 --> 01:25:25.888
So in these experiments, we created
2030
01:25:25.888 --> 01:25:28.445
the independent set model with kings and pawns.
2031
01:25:28.445 --> 01:25:30.461
So basically over the experiment,
2032
01:25:30.461 --> 01:25:32.629
you either declare yourself to be a king
2033
01:25:32.629 --> 01:25:34.685
or you declare yourself to be a pawn.
2034
01:25:34.685 --> 01:25:36.372
If you declare yourself to be a pawn,
2035
01:25:36.372 --> 01:25:37.998
no matter what's going on in your neighborhood,
2036
01:25:37.998 --> 01:25:40.114
you get 50 cents per minute.
2037
01:25:40.114 --> 01:25:41.690
These are the only experiments we did
2038
01:25:41.690 --> 01:25:43.849
where the payments were prorated by the task.
2039
01:25:43.849 --> 01:25:45.377
Whatever rate of rate you're getting,
2040
01:25:45.377 --> 01:25:46.938
as long as you're in that state,
2041
01:25:46.938 --> 01:25:48.258
you get that rate of pay.
2042
01:25:48.258 --> 01:25:51.675
So the rate if you were a pawn was 50 cents a minute,
2043
01:25:51.675 --> 01:25:53.358
no matter what.
2044
01:25:53.358 --> 01:25:55.133
If you declared yourself to be a king
2045
01:25:55.133 --> 01:25:57.504
and all of your neighbors were pawns,
2046
01:25:57.504 --> 01:26:00.542
then while that state holds, you got a dollar per minute.
2047
01:26:00.542 --> 01:26:01.733
This is like electing yourself to be
2048
01:26:01.733 --> 01:26:03.674
in the independent set, okay?
2049
01:26:03.674 --> 01:26:06.310
But if you declared yourself to be a king
2050
01:26:06.310 --> 01:26:10.668
and you had a king neighbor, you got nothing.
2051
01:26:10.668 --> 01:26:13.085
So this is sort of a tension.
2052
01:26:16.785 --> 01:26:19.459
And so the maximum social welfare solutions
2053
01:26:19.459 --> 01:26:20.972
are actually the independent sets.
2054
01:26:20.972 --> 01:26:22.601
You basically have, you take a maximum
2055
01:26:22.601 --> 01:26:24.749
independent set, call those the kings.
2056
01:26:24.749 --> 01:26:27.519
That's the highest, you get the dollar.
2057
01:26:27.519 --> 01:26:28.976
Everybody else gets 50 cents.
2058
01:26:28.976 --> 01:26:33.966
So sort of the only kind of influence trading game
2059
01:26:33.966 --> 01:26:37.633
we gave is that we ran all those experiments
2060
01:26:39.709 --> 01:26:42.861
with a variation where you kind of had
2061
01:26:42.861 --> 01:26:45.757
a slider bar that went from zero to 100%
2062
01:26:45.757 --> 01:26:47.748
in 10% increments.
2063
01:26:47.748 --> 01:26:49.820
And what that slider indicated was
2064
01:26:49.820 --> 01:26:53.925
if you were a king that only had pawn neighbors,
2065
01:26:53.925 --> 01:26:56.645
that fraction of your prorated earnings
2066
01:26:56.645 --> 01:26:58.557
would be divided amongst your neighbors.
2067
01:26:58.557 --> 01:27:00.077
The idea here is that, well,
2068
01:27:00.077 --> 01:27:03.899
if I'm a king and all of my neighbors are pawns,
2069
01:27:03.899 --> 01:27:05.501
then I'm enjoying the highest rate of pay.
2070
01:27:05.501 --> 01:27:07.531
Well, it's my neighbors' pawn-ness
2071
01:27:07.531 --> 01:27:10.396
that makes my king-ness possible.
2072
01:27:10.396 --> 01:27:12.585
And maybe I should be grateful to them
2073
01:27:12.585 --> 01:27:15.727
and share some of my earning with them, right,
2074
01:27:15.727 --> 01:27:18.666
to encourage them to stay in the pawn state.
2075
01:27:18.666 --> 01:27:21.442
So this is sort of an example of what you're saying
2076
01:27:21.442 --> 01:27:23.175
where I'm actually gonna pay somebody
2077
01:27:23.175 --> 01:27:25.438
for not influence in this case
2078
01:27:25.438 --> 01:27:28.167
but for remaining in this lower payoff state.
2079
01:27:28.167 --> 01:27:29.953
And one of the interesting things is
2080
01:27:29.953 --> 01:27:31.749
that now there can be network structures
2081
01:27:31.749 --> 01:27:34.308
where you could actually get higher than the king
2082
01:27:34.308 --> 01:27:36.733
rate of pay by being a pawn
2083
01:27:36.733 --> 01:27:39.569
because for instance think of a hub and spoke, right?
2084
01:27:39.569 --> 01:27:42.736
So in a hub and spoke, by being a pawn
2085
01:27:44.167 --> 01:27:47.642
at the hub and letting all of the spokes be kings,
2086
01:27:47.642 --> 01:27:52.190
well, you might get side payments from all of them
2087
01:27:52.190 --> 01:27:54.790
and make much more than the dollar a minute for a king.
2088
01:27:54.790 --> 01:27:57.098
And the interesting thing in those experiments
2089
01:27:57.098 --> 01:27:59.750
is in every single experiment we ran like that,
2090
01:27:59.750 --> 01:28:02.058
introducing this side payment mechanism
2091
01:28:02.058 --> 01:28:03.563
increased the social welfare.
2092
01:28:03.563 --> 01:28:04.966
It increased the total payoffs
2093
01:28:04.966 --> 01:28:08.122
because basically now instead of there being conflicts
2094
01:28:08.122 --> 01:28:10.440
like where two players are like, well,
2095
01:28:10.440 --> 01:28:12.084
you've been a king for a while now,
2096
01:28:12.084 --> 01:28:17.024
and I've been a pawn, so now I have to conflict with you
2097
01:28:17.024 --> 01:28:19.354
in order to get you to back off and sort of share
2098
01:28:19.354 --> 01:28:20.840
that higher rate.
2099
01:28:20.840 --> 01:28:22.554
Now that higher rate can be shared directly
2100
01:28:22.554 --> 01:28:23.983
through this tipping mechanism,
2101
01:28:23.983 --> 01:28:25.262
side payment mechanism.
2102
01:28:25.262 --> 01:28:27.612
And that made everybody make more money.
2103
01:28:27.612 --> 01:28:30.632
I mean, they're collectively making more money.
2104
01:28:30.632 --> 01:28:31.496
We better wrap this up.
2105
01:28:31.496 --> 01:28:33.848
Yeah, thank you, thanks a lot.
2106
01:28:33.848 --> 01:28:35.888
(applause)
2107
01:28:35.888 --> 01:28:38.388
(light music)