Frederic Vernier
"Multimodality and Visualization"
October 16, 2:00pm , A.V. Williams 3258
Abstract
After saying a few words about the lab where I did my work
in Grenoble, I will present my thesis two main research
axes: Multimodality and Visualization. The goal of my
research is to apply multimodal interactions to the design
of graphical user interfaces for large information spaces.
The large variety of multimodal interfaces stems from the
diversity of available modalities AND on the many ways of
combining them.
For in put user interfaces, the "put that there" paradigm
illustrates through the usage of different modalities
(speech and mouse) the variety of interaction means.
This richness stems not only from the different available
modalities but also comes from the multiple ways of
combining them: There are spatial, temporal, syntactic
and semantic aspects involved in the combination of modalities.
For output multimodality, the "look at that" paradigm will be
introduced. For example a system can display a graphic and
play a sound "this graphic has been updated 15 minutes ago".
The aspects of the combination for output are similar to
those for input. The main difference is that the system must
be able to combine the modalities (selection, synchronization,
etc.): that is the combination is no longer performed by the user.
Screen-based modalities, play a central role in my work because
my research focuses on the design of output multimodal user
interfaces for large information spaces. Graphical modalities
define a vast range of possibilities. A graphical multimodal
user interface can for example provide to the user several
views of different data at the same time or different
complementary views of the same data such as the focus+context
design rule. In these two cases, the richness of the interface
is due to the use of different graphical modalities and the
various ways of combining them. To better understand this
combination I will present several existing taxonomies in
visualization and our design space, which contains various
combinations of output modalities.
At last I will present three systems or will make three
demonstrations depending on the possibilities. These three
visualization systems illustrate the concepts of the taxonomies
and raise new chalenges and new problems. The talk will surely
finishes with informal discutions about implementing visualization
systems in JAVA, adding new features to my systems, exchanging
piece of code