Creativity Support Tools for and by the New Media Arts Community

Pamela Jennings Human Computer Interaction Institute, School of Art Carnegie Mellon University Pittsburgh, PA 15213 412-268-5273 pamelaj@andrew.cmu.edu

Elisa Giaccardi Center for LifeLong Learning and Design (L3D) University of Colorado Boulder, CO 80309-0430, USA 303-492-4147 elisa.giaccardi@colorado.edu

INTRODUCTION

The new media arts are a particularly fertile domain for the development of creativity support tools that both supplement creative practices and contribute valuable research methodologies for other disciplines. Many parallel research concerns of new media art practitioners and researchers are found in the Human Computer Interaction and software engineering communities, including: education technology, computer supported collaborative work, data visualization, database architecture, and tools development research in pervasive computing, tangible interfaces, emotion and context aware interaction, and so on. New media arts practitioners and researchers should be regarded as valuable contributors not only as users needing better creativity support tools (CST) to enhance their own creative process, but also as the designers of experimental and innovative creativity support tools capable of providing insights and indications for:

1. Categorizing what features of the interface and what system components engender and satisfy the requirements of these multiple forms of creativity. Section 1, Creativity Support Tools development for New Media Arts Curriculum presents examples of tools that have been designed for new media art production, and potential tool features, that could empower the creative potentials of practitioners across all fields of creative production.

2. Defining the range of potential “creativities” (Sternberg, 2005) for which new technologies and tools may be developed. Section 2, Research-In- Practice, introduces works by new media artists that have developed creativity support tools that are used by a large user-base of artists and other professionals as a result of their own creative practices. Sometimes this is a primary outcome, often a residual effect from developing robust tools for one’s own art practice that can withstand a variety of user interaction and manipulation. This section also presents cases of new media arts research-in-practice that hold the similar broad reaching potentials.

3. Developing more comprehensive and appropriate evaluation methodologies grounded in the “research-in-practice” approach. Section 3, Policy Making and New Media Arts, is a brief overview of international policies for new media arts practices. Many of these policies recognize the innovative potentials for effecting research in science and technology including and beyond the arts.

This document, generated from presentations and conversations at the NSF Creativity Support Tools Workshop held in Washington D.C. in June, 2005, presents example cases of the contributions that new media art pedagogy, practice and research can provide in the development of support tools that promote the situated, affective and social aspects of creativity.

1. CST DEVELOPMENT FOR NEW MEDIA ARTS CURRICULUM

Education technology research by information technology, education, and public policy researchers has grown tremendously as the Internet has become the de facto platform for the dissemination of information, and platform for community-based collaborations. From component based authoring environments to cognitive tutors, much of the research in this area has been funded by the National Science Foundation to support STEM (Science, Technology, Engineering, and Mathematics) disciplines. Our tendency to separate the STEM disciplines from creativity, culture and humanities sets forth a pattern of missed opportunities to develop new technologies that could help to solve research problems for disciplines that present alternative perspectives on data acquisition, analysis, and manipulation. In recent years new media arts practices have entered the mainstream. Many international universities are actively creating new media arts programs geared towards artists that both use and create technologies (Jaimes & Jennings, 2004). NSF has recently funded a few research initiatives to develop curriculum modules that integrate computer science and new media arts (Integration Digital Media Curriculum Development (NSF DUE- 0340969) and the Digital Media Curriculum Development Project (NSF DUE-0127280). Pedagogical practices in new media arts are based on problem solving through open exploration of conceptual ideas that sometimes conform to, but mostly challenge, the intended functionalities of technology-based tools. This pedagogical practice of open tools usage sits in contrast to the typical computer science curriculum that is based on learning through constrained problem solving. In the later case, students are typically given assignments that have a limited number of acceptable solution variations. The open and indeterminate nature of new media arts practice presents unique opportunities for developers of creativity support tools to incorporate complex programming abstractions, relational databases, integrated search functionalities, and scaffolded interfaces as a means to create more flexible creativity support tools for new media arts students and practitioners. (Bransford & Brown, 1999)

1.1. NEW MEDIA ARTS CURRICULUM CASE #1:

Design of Appropriate Tools Features for Open Ended Creative Production

Fig. 1 3. Human Computer Interaction and Interaction Design students working with electronics in the Physical Computing: Wearables offered by the School of Art at CMU.

Students in the Introduction to New Media Arts class at Carnegie Mellon University were given an assignment to create a sound self-portrait five minutes in length. The main rule was a restriction on the length of a sound bite to ten seconds. This rule was given to encourage the exploration of composition by editing multiple sound layers. PEAK, a sound editing application, was selected because of its relative easy to use and happens to be the introductory application installed on all of our computers. An advanced sound editing application, though more flexible, was not introduced at this stage to give students, of all levels, a sense of efficacy that would be difficult with a more advanced application. PEAK is also an application that replaces Sound Edit Pro, a Macintosh application from the early 1990’s. Both applications are simple to use but have a couple of profound differences. Sound Edit Pro enabled artists to approach sound editing in a much more fluid and conceptual manner with an interface that did not assume that the end product was to be a perfectly balanced stereo recording. Thus it did not put constraints on how the user could combine multiple layers of sound across multiple sound channels. PEAK, on the other hand comes with more sound synthesis filters and plug-ins than any practitioner may ever want to use. It has savvy widgets for editing within a sound channel, but has very limited flexibility for mixing more than two mono tracks at a time, making the combination of multiple sound layers an unnecessarily arduous task. The students persevered through the assignment and produced wonderfully complex sound compositions, because of their deep interest in contemporary sound culture. Benefits they gained from working with this inflexible interface included the realization that working with computer-based tools requires, focus, patience, and the willingness to fail often before producing a satisfactory product. However, they could have learned these lessons with an application better attuned to alternative interpretations of assumed use by its developers. This would be an interface with features for the beginning professional that respect her developing aesthetic voice.

1.2. NEW MEDIA ARTS CURRICULUM CASE #2:

Tools to Support Mixed Skills Level Classes

Upon completion of the compulsory introductory classes in new media arts, the Carnegie Mellon University art student can enroll in intermediate and advance elective classes that range from video and sound production and animation, to interactive programming and physical computing. These classes are also very popular for students in the Human Computer Interaction Institute, Electrical and Computing engineering, Entertainment Technology Center, College of Humanities and Social Sciences, and School of Design. This mix of students enables opportunities for cross-disciplinary learning and collaboration. The classes attract a broad range of students for several reasons. Students learn specific technology-based tools for creative arts and interactive design prototypes. Students can explore these techniques in a curriculum that encourages freedom of exploration and expression. Many of our computer science and engineering students enjoy these classes because they can engage in open-ended learning, which is generally a different approach than the typical exercise or lab-based technology-learning pedagogy.

The broad range of students’ skills in these classes presents a few creativity support tool development opportunities The first is the design of tools that can accommodate a range of skills from beginner to advance with proper scaffold features and functionalities for each user level. The second is the development of multi-level help modules that can assist students and instructors in facilitating a broad range of questions and possible solutions. For example, it would be wonderful to have a cognitive tutor that notices that a student with beginning skills levels continuously gets a syntax error in her code which she is having difficulty correcting. After analyzing and finding no errors in the code structure, the help module recommends that she check her spelling. Another example if a student wants multiple sprites on the stage to design a Tetris game like interface should he be instructed to create each sprite as a unique entity a long and tedious process that uses a minimum of code, or should he delve into property lists and object oriented programming far more efficient but requires a basic knowledge of programming. Or, should he be encouraged to implement sorting and advanced search algorithms that allow him to develop an intelligent self-playing game. These are not hypothetical examples, but the range of students I have seen in new media art classes taught in the School of Art at Carnegie Mellon University and surely a case that is replicated at other universities.

1.3 NEW MEDIA ARTS CURRICULUM CASE #3:

Holistic Presentation Tools

Students taking interaction design classes are often required to work in groups to solve a particular design /human computer interaction interface problem. They are typically given a hypothetical situation to ideate, visualize, and present. Students are encouraged to select very creative means to explore and represent their ideas including text, graphics, photographs, video, theater, 3D animation, sound, programming examples, etc…. It is the student team’s task to come up with methods to formalize their needs analysis research, aggregate and negotiate their ideas and produce a visual or functional demo of the interface, application, or gadget. Unfortunately, all of the effort in this work is often reduced to a power point presentation for class-wide discussion and critique. Here is a tremendous opportunity to develop a tool, or set of interoperable tools to support all the requirements as described above without forcing the final presentation to be formatted into bullet point lists.

Fig. 4 7. Storyboard excerpts from Critical Interaction Design class. Students involved were from Human Computer Interaction, Architecture, Entertainment Technology Center and School of Design at Carnegie Mellon University.

2. “RESEARCH-IN-PRACTICE”

Creativity Support Tools for and by the New Media Arts Community

The new media arts are characterized by what is usually referred to as “research-in-practice”: an experimentalism and reflexivity that bring artists to link creative research and practice in a “highly responsive, iterative process where new insights are fed back quickly into the development process” (Candy & Edmonds, 2005). The new media arts express a risk-taking and subversive attitude, ultimately seeking cultural acts through which to provide society with entry points for change (for example, in the very definition of what is creativity and how it can be supported). Stephen Wilson’s “Information Arts: Intersections of Art, Science and Technology” (Wilson, 2002) is an encyclopedic archive of arts, technology and science collaborations exploring new ideas that challenge and contribute alternative perspectives on research practice and tools development across nearly every category of science and technology research from microbiology to nanotechnology to augmented reality.

Fig. 8 . David Rokeby, Very Nervous System (1990); Fig. 9. Paul Kaiser with Bill T. Jones, Ghost Catching (1999); Fig. 10. Screen shot of Cycling 74 MAX / Jitter interface; Fig. 11. Char Davies, navigation technologies developed for Osmose virtual environment (1995).

Examples of innovative tools created within the new media arts community include, David Rokeby's "Very Nervous System", a computer vision system used by many installation artists and stage performers. Miller Puckett's MAX, distributed by Cycling 74 and his open source version Pure Data (PD) has opened the door to real-time audio and video synthesis and analysis as well as controlling external equipment for theatrical performances for students and professionals working in a variety of media-based fields. Char Davies influence on the graphical user interface and aesthetic filters for SoftImage 3D rendering software represents another interesting case illustrating how novel ideas from the new media arts have influenced the aesthetics of mass media and Hollywood cinema, as well as physical navigation of virtual environments. Paul Kaiser’s work with choreographer Bill T. Jones and computer programmer Shelley Eshkar has produced new techniques for real-time motion capture and visual processing. Donna Cox’s visualizations of the universe have aided school children and scientist to understand phenomena like the “big bang.”

We live in a culture that tends to separate research and acquisition of new knowledge into two general camps, applied technical research and aesthetic and social research. The HCI community has made great progress in reuniting social and technical inquiry. Inclusion of new media arts practices and research presents the opportunity to not only integrate aesthetic inquiry with the socio-technical platform of HCI, but also deliver an influential impact on domains outside of the arts, for example, investigating the relationships between metadata, multimedia content, and culture; developing novel forms and tools for interaction with data; understanding the influence of different narrative traditions on data collection and presentation and on the design of novel forms of digital representations that extend beyond the pervasive WIMP model (Jaimes & Jennings, 2004).

2.1. CASE STUDIES

This section presents innovative principles and interface features created by new media artists that have impact on the future development of creativity support tools within and outside of the new media arts research and practice aligned with pervasive, tangible and collaborative screen-based development methods. These principles and features support: (a) temporal, spatial and conceptual distribution across multiple interaction spaces; (b) emotion and context aware interaction to nourish participation in the creative process; (c) use of generative elements to evoke surprise and provoke user reactions; and (d) integration of applied research and production methods from art, design and technology-based fields.

2.1.1. NEW MEDIA ART RESEARCH IN PERVASIVE COMPUTING

Because creative activities take place in a context in which interactions with other people and artifacts are essential contributors (Harrington, 1990; Mockros & Csikszentmihályi, 1999; Fischer et al., 2005), some new media artists have started to think of creativity support tools as distributed structures that mutually reinforce both individual and social creativity. This approach implies a shift from the idea of tool—or set of tools—to the notion of a socio-technical architecture deeply interwoven with the physical environment and social fabric of local communities, based on mobile and ubiquitous computing, and focused on the transmission of data and information among different interaction spaces. This line of inquiry, which we might call pervasive creativity, appears to be a relevant context for investigating and promoting situated and distributed aspects of creativity, particularly in relation to temporal, spatial and conceptual distribution across multiple interaction spaces.

INNOVATIVE CREATIVITY SUPPORT TOOL CASE #1:

THE SILENCE OF THE LANDS
Developed at the Center for LifeLong Learning & Design (L3D), University of Colorado, Boulder by Elisa Giaccardi and Hal Eden in collaboration with Gianluca Sabena, Politecnico di Torino, Italy.

Case #1: Research Description

The Silence of the Lands (SOL) is a combined social game and an information-gathering tool, inspired by the vision and principles of the EDC (Arias et al., 2000) and the emerging use of pervasive computing. SOL supports the collection, interpretation, and visualization of Global Positioning Systems data that have been recorded directly from the members of a local community in order to address some of the societal problems in the definition of policies for the protection and enjoyment of natural quiet (Fig. 12). In its initial application, these data are "ambient sounds" and represent subjective interpretations of the "soundscape" of urban or natural settings that affect everyday life. By means of social participation and engagement, the project promotes a model for preservation, experience, and renewal that empowers the active and constructive role of local communities in the process of interpretation of natural quiet. This model embodies an approach to interaction design (viz., metadesign, see Giaccardi & Fischer, 2005) as a form of cultural intervention aimed to support creative and sustainable solutions to complex societal problems.

SOL enables people with different, sometimes competing visions to communicate and coordinate their different knowledge and perspectives about natural quiet. This is accomplished by using sounds (rather than words) as the conversation pieces of a social game about preservation and enjoyment of natural quiet in urban or natural settings. The goal is to create a living space inside the local community by engaging participants in the recording and mapping of their own, experienced soundscape and in the construction of an idealized, virtual one. In order to support the social dialogue and the soundscapes’ collaborative design, the project combines pervasive computing and tangible interfaces in a socio-technical architecture of disinct, but integrated interaction spaces.

Fig. 12. Overview of the socio-technical architecture of SOL: A combination of multiple interaction spaces and social practices mediated by pervasive computing and tangible interfaces: (1) satellite (GPS signal); (2) participants (sound walks); (3) mobile interface (sound catching and geo-referencing); (4) antenna (wifi connection); (5) server (database management); (6) web interface (visualization and description of individual soundscapes); (7) tangible interface (face-to-face interaction and collective soundscape interpretation).

By providing different entry points, promoting the different properties of each interaction space, and supporting different interaction roles over a sustained period of time, such an architecture aims to: (a) empower the creative interaction between current and future interpretations engendered by collaborative design, (b) enable participation and collaboration that fits more naturally with existing social practices and the way in which people act and interact with their local environment, and (c) support processes of social awareness and informal learning. The collective conversation produced by the collaborative design of the participants is expected to create an “affective geography” of natural quiet and transform such an abstract concept into a living entity that changes according to current and future interpretations. In this way, The Silence of the Lands not only increases sensitivity and social awareness about ambient sounds, but it also provides a tool for the visualization of collective perception and public trends.

Case #1: Design Principles

Experimental design principles for pervasive creativity deriving from the new media arts suggest that—in order to activate the collective stock of ideas and visions that belong to an environmental setting—tools and spaces must be woven into the existing social fabric and physical environment of the urban setting or community by means of mobile and ubiquitous computing.

The Silence of the Lands addresses the design of creativity support tools from an “ecological” perspective: that is, as a set of multiple tools and interaction spaces promoting the environmental setting as a “creative milieu” (Cliche et al., 2002) and composing a distributed, socio-technical infrastructure capable of mediating and linking the ideas, visions, people, places, production processes, and values that pertain to a specific environmental setting.

According to initial studies on public authoring (Silverston & Zoe, 2004), place-based content facilitates memory, association, and connotation. Furthermore, the shift from the single-desktop tool to multiple tools and interaction spaces (separated physically but seamlessly integrated virtually) promotes the integration of individual and social creativity. The flowing and manipulation of data throughout multiple interaction spaces (including the natural environment) enables users to promote not only spatial and temporal distribution, but also the distribution of ideas and visions (Fischer et al., 2005). Moreover, it produces what we might call information enrichment, that is, the engaging possibility of collecting and reinterpreting both individual and collective data over a sustained period of time, according to the different properties of the space with which a user is interacting and through which data is traveling (Fig. 13).

Fig. 13. Data Flow in SOL: Ambient sounds are collected from the natural environment by means of handheld devices. Each sound is linked to the user that collected it and is associated with GPS data (which determine its location in space and time). Sounds and walks (i.e. the paths followed by participants during recording sessions) are stored on the web server and visualized on the web site as individual soundscapes, one for each participant. On the web, users can access and manage their individual soundscapes (eventually modifying and changing them). They can also visualize the collective soundscape resulting and growing from the overlap of all individual soundscapes. Such a collective soundscape represents the starting point for participants in the community to collaborate on the creation of the virtual soundscape (i.e. the ideal soundscape). In the public space, both old and new participants can interact with the collective soundscape by means of tangible interfaces. Each public session produces a temporary soundscape, reflecting the understanding and creativity of the people that participated in that session. All temporary soundscapes are then composed in a historical soundscape on the basis of purposely-designed algorithms. A visualization of the historical soundscape is provided both on the web site of the project and in the public interactive space.

Case #1: Tools Development and Evaluation Methodology

This result is obtained by combining direct experience, cognitive mapping, and face-to-face interaction; that is, by combining: (a) data catching (individual sound collection and geo-referencing by mobile computing); (b) data description (individual soundscape management by web tools); and (c) data interpretation (collective interaction and social negotiation by tangible interfaces in a public space).

Sounds are geo-referenced and visualized on a GIS map as evolving matrices of audio objects aimed to reveal areas of dissension, consensus, and uncertainty by means of color-coded attributes and descriptors (Fig. 14 and Fig. 15).

Figg. 14-15. Web visualization of the collective soundscape and color-coded audio objects at different zoom levels.

2.2. NEW MEDIA ART RESEARCH IN TANGIBLE SOCIAL INTERFACES

Design thinking, which has been categorized into available design, design, and re-design, is integral to the development of meta-cognitive and meta-linguistic abilities. (New London Group, 1996) Re-design, the most transformative of the categories, supports the generation of new knowledge from current discourse by supporting the process of inquiry, discourse and negotiation. This process can be facilitated by convivial tools that enable “users to invest the world with their meaning, to enrich the environment with the fruits of their vision and to use them for the accomplishment of a purpose they have chosen” is a method by which to incorporate the transformative process of re-design (Illich, 1973). In our increasingly media-rich environment, marked by pervasive and ubiquitous computing and wireless devices, practices in new media culture are no longer limited to screen-based, audiovisual and interactive media content but address the wider social, urban and global context of the information environment, through novel approaches to process-based networked projects. Many new media artists have taken on the challenge to design systems that foster “the diversity of the public actors and terrains and…develop strategies [for] articulating the new public domains that connect physical urban spaces and potential public sphere of electronic networks.” (Broeckmann, 2000) Convivial systems, such as the tangible social interface, encourage users to be actively engaged in computer-mediated open generative systems, designed to support intersubjective experiences that encourage, provoke and support debate, discussion for the construction of new knowledge and understanding in our shared worlds. Intersubjectivity is a theoretical concept used to understand how individuals can interact and produce consensual interpretations about a shared experience which can be other people, objects, or events (Thompson, 2001).

INNOVATIVE CREATIVITY SUPPORT TOOL CASE #2

CONSTRUCTED NARRATIVES

Principle Investigator: Pamela Jennings, Assistant Professor School of Art and the Human Computer Interaction Institute, Carnegie Mellon University.

Fig. 16. Potential target user audience for the Constructed Narratives project in an airport waiting lounge , Zurich, Switzerland. Fig. 17. Demonstration of the Constructed Narratives project at the Kiasma Museum for Contemporary Art in Helsinki, Finland 2004.

Case #2: Research Description

The Constructed Narratives project is a tangible social interface (TSI) a physical interface designed to enable users to collaboratively construct and negotiate their social and knowledge networks based upon their unique preferences and user profiles. This on-going project is comprised of a set of physical blocks that when connected form an open topology network. Construction patterns in the emerging collaboratively built structure are tracked and analyzed. This analysis is used to seed a search for text which is revealed in a 3D screen-based navigable replica of the physical structure. The collaboratively built construction is a socio-technical architecture, similar in goals to projects described earlier in this document, built from the development and repurposing of information technologies to explore physical environments and the social networks among people they support. This computer supported collaborative play project is being designed for public spaces to enable dialogue between builders (users) in environments where such communicative acts are less likely to occur. The project, and overall research inquiry, was inspired by the principle investigators countless hours watching people watch people in international airports and wondering how information technologies could be used to make communicative connections between people who are co-located in a public space. Though inspired by airports, the project is envisioned for any public space where a large number of people are facilitated -- from an informal science center to a Cineplex.

Constructed Narratives is a platform being developed as a common-ground mediator that incorporates play and problem solving as a means for enhancing informal learning and knowledge networking in situations or about relational topics between participants that are unlikely to happen without a mediator to prompt contact. A key principle for learning, as articulated by the National Research Council Committee on Developments in the Science of Learning, is the ability for an individual “to engage in the mental work of making inferences,” as a means to make relationships between available information for resolving an inquiry, problem, or task. (Bransford & Brown, 1999) The Constructed Narratives system prompts the builder to incorporate inferential problem solving techniques to understand and manipulate the relationships between the physical construction and the text output. The builders’ actions of arranging and rearranging the physical blocks artifacts supports a process of empowerment where the builder negotiates structural solutions simultaneously with her collaborator and the topic of discussion as revealed through the semantic layer. The builders are co-constructing a world in which they have ultimate design authority. This is a world in which they are the very material of which that that world is made. Topics of discussion are prompted by a text layer to the construction made visible in a 3D navigable screen-projection of the physical construction. The semantic layer is determined by an underlying software engine that examines in real-time the emerging construction, ownership of the blocks and a few simple questions each builder answers prior to game play. Topics of the semantic layer include the relationship of each builder to other’s at the construction table, (e.g. self-identity, origins, environments of work and play and belief constructs,) or a domain specific topic such as environmental science, issues and their relationships to communities familiar to the builders.

Case #2: Design Principles

The design of the Constructed Narratives block is based on George Stiny's shape grammars, a computational design methodology. Stiny’s was greatly influenced by Froebel’s Kindergarten Gifts philosophy of learning through play for his design methodology. Constructed Narratives also references and a lineage of research based on the work of Architect Jonathon Frazer and his Universal Constructor generative system. (Stiny 1980, Frazer, 1995; Jennings, 2005b) The Constructed Narratives research project was developed by a team of Carnegie Mellon University students from eight schools on campus including the School of Art, Human Computer Interaction Institute, Electrical and Computer Engineering, Computer Science, School of Drama, School of Design, Cognitive Science, and Information Systems Management. Working on this project continues to be an interdisciplinary and collaborative effort requiring each research team member to quickly become adept with negotiating their discipline specific knowledge-base while learning new technology platforms for the development of the system. Several types of boundary objects have been used as tangible aids in the exploration of ideas and development of system solutions, as illustrated in figure 18 -21. As Gerhard Fischer has noted in his research, the boundary object is a mediator that enables the exploitation of problem solving opportunities afforded by the “symmetry of ignorance” in an interdisciplinary research team (Fischer, 1999). They provide a means to support dialogue between team members who may not be accustomed to the discipline specific language of other team members.