"I Hear the Pattern" - Listening to Geographical Data
By Gabe Goldberg, HCIL Media Fellow
Have you smelled any bright colors lately? Or seen pleasant odors? Of course not; our senses are specialized and not interchangeable. They are cooperative, though, in that a deficiency in one can heighten others.
And sometimes technology helps the process by translating one sense's language to another's -- for example, by reading words on a computer screen to someone visually impaired. But while this is useful for conveying text and picture captions (provided by HTML "alt" tags), it breaks down when handling data such as numeric tables. Without the aural equivalent of a photographic memory, listening to tabular information is overwhelming.
Reader software with good table-handling tools might perform simple tasks such as finding the state with highest unemployment. However, it won't identify global patterns, exceptions, or clusters. Even worse, graphical (e.g., map) data can't be read at all.
A research project (www.cs.umd.edu/hcil/audiomap) at the University of Maryland's Human Computer Interaction Laboratory (HCIL) investigated interactive sonification (user-controlled data-driven non-speech sound) to present the geographical distribution pattern of statistical data. In addition to helping blind users, this will benefit low-vision and sighted users in visually overloaded situations. The immediate goal is improving access to geo-referenced data. In a larger context, this will improve citizen access to government statistics for queries, problem solving, and decision making.
This effort extends previous work in several ways. First, it allows users to actively explore data instead of just passively listening. Second, rather than exploring regular grids, geographic data scatters on map regions with irregular shapes and sizes, imposing special design challenges. Third, while focusing on standard devices (e.g., keyboard) and sound resources (e.g., stereo MIDI), the investigation also exploits other common devices such as tablet and advanced techniques such as virtual spatial sound.
While vision impaired users typically access data in a linear or tabular form, research has shown them benefiting from other representations. For geographical data, a map is a natural form to show locations and adjacencies of geographical regions. While such geographical knowledge can be added to tables, previous study subjects strongly preferred (and performed better with) maps. Typical data sets are population, unemployment, and cancer deaths -- all of which can be presented as maps (representational rather than navigational) on which each region (state, county) is a uniform color with hue/intensity proportional to variables of interest. Color/hue intensity is conveyed by musical tones with higher pitch signifying higher value.To convey geographical data distribution patterns via interactive sonification, several user actions were supported. Gist plays a short audio message giving a quick grasp of overall data patterns or trends; it also guides further exploration and often reveals anomalies and outliers. Navigation allows moving within data, selecting and listening to portions of it. Details-on-demand provides extended information on a data element or aggregate.
Computer usage involves issuing commands and receiving feedback. For sighted users, the visual interface sustains continuous near-instantaneous interaction. By contrast, auditory feedback presents information over time. So the sonification system must help the user create and maintain a mental representation of the data space.
Earlier digital map exploration combined tactile feedback and sound to convey spatial information. While tactile feedback is potentially useful, it limits interfaces' accessibility and portability because of dependence on special devices or hard-to-find tactile maps.
The standard computer keyboard is widely used by blind users. Its arrow keys are natural means for relative movement in up/down/left/right directions; the numeric keypad supports movement in eight directions.And with remapping, the 3x3 numeric keypad can be a low-resolution 2-D pointing device; users can track grid recursion for three levels, providing resolution of 27x27 grid cells.
Touch tablets, though not standard input devices, are increasingly common. A 14" device costing less than $150 provides high-resolution 2-D pointing and continuous movement by finger pressure. Feedback from arm and finger movement, combined with the tablet frame as position reference, may help user orientation during data space navigation and mental-mapping.
Audible feedback describes data items activated, presenting both statistical values and 2-D locations. Pitch most effectively represents values while regions are mapped to perceived sound locations via standard MIDI hardware and specialized software. For a gist, a rapid sequence of musical tones sings out the geographical data pattern.
During summer 2005, intensive case studies were conducted with seven local blind users. All possessed basic computer skill and relied on screen readers to access computer information. They were all comfortable with maps and tables, had experience with numerical data sets, and used government statistical data at work. The subjects used the table "view" for most value comparisons; they used the map to compare items in the geographical context or to acquire/confirm region locations.
Users liked the ability to quickly change between table and map views. One evaluator said that playing a whole table column in pitch only was like "sitting in the opera". He noted comparing two numeric values is easy, but that it's hard to look at patterns of more numbers. So music makes a lot of sense in this case, he continued, "Tones help me do things I used to do with graphs".
With advice from Prof. Ben Shneiderman and Research Scientist Catherine Plaisant, doctoral candidate Haixia Zhao continues her research with design of two new interfaces, one each for keyboard and tablet. On the keyboard, the 3x3 numeric keypad activates a horizontal sweep of regions in each of the corresponding nine map areas. Each region's value sound is played, with distinctive sounds indicating end of column and end of sweep. Users can zoom for details and recursively explore ranges in the 3x3 style.
On the tablet, users drag fingers or press spots to activate regions. The value sound for a region continues until the finger lifts or changes region. Again, distinctive sounds alert for location and status changes. With both interfaces, users can request a region's spoken name and value.
Researcher and doctoral candidate Haixia Zhao reports that blind users recognized patterns on familiar and unknown maps using both interfaces. In addition, a low-vision user found that sound cues augmented what he could see, especially in small map areas.
A user described avoiding tables in his work because they're often not accessible or are hard to use, though his usual information source is text information from other people. He commented, "This tool allows me to drill down into data and find interesting facts, and provide them to other people."
Zhao's work demonstrates systematic investigation of interactive auditory display design, guided by a design space framework. Lessons and insights obtained can benefit non-visual interface design for other user groups.
