Coherent multi-modal instrument design in digital media

In nature, sound results from the vibration of the medium, and vision results from the reflection of lights. On the other hand, digital mediums are not constrained by the materials and laws of physics. Then how do we sculpt and relate to the digital multi-modal information and expand it into immersive media? What opportunities does it allow? And what types of skills and insights are required? This talk brings these questions to light and shares my experience exploring the answers with scientific discovery, data visualization and sonification, music composition, signal processing, machine learning, and gestural interface. Particularly, I will introduce the design processes of selected art-science collaborative works including “Coexistence with the SARS-CoV-2 virus” and how their multi-modal experience aroused interesting discussions among the artists, scientists, engineers, and general audience. I will also share my 4 years of teaching experience with students from diverse backgrounds and the evolution of a new multi-modal composition class from a purely audio synthesis class.