PhD Proposal: Towards Generalizable Computational Representations for Data Visualizations

Various data visualization downstream applications, including interactive authoring/editing and reverse engineering, require a vocabulary that describes the semantic structure of visualization scenes (i.e., representations) and how the scenes shall be manipulated (i.e., computations on the representations). The representations in which visualizations are rendered, i.e., Bitmap and SVG, are too low-level to work with effectively. Programs, as higher-level representations, do not provide adequate native support for scene manipulation computations because they either lack high-level semantic abstractions or conceal the scene representation from the user. Researchers thus have been proposing new visualization representations. However, those representations are either designed for specific applications, thereby hard to generalize to other tasks, or restricted to a small set of visualization types, limiting their expressiveness. To address this gap, we first introduce Manipulable Semantic Components (MSC), a computational representation of data visualization consisting of a unified object model describing the visualization scene structure in terms of semantic components and an operation set for modifying the scene components. Secondly, we present Mystique, a mixed-initiative authoring tool that decomposes an input SVG chart into semantic components compatible with MSC and allows reusing its structure and visual styles on new data through operations in MSC, demonstrating MSC’s usability in chart reverse engineering. Thirdly, to pave the road to the integration of MSC and AI, we have conducted a state-of-the-art survey to understand current practices and limitations of researchers creating visualization corpora, and then curated VISANATOMY, an expert-annotated diverse chart corpus with rich, fine-grained labels that reflect the semantic components defined in MSC. Through four different use cases, we demonstrate that MSC-informed semantic labels can support various downstream tasks on a wide range of visualizations. Lastly, we propose to investigate how the semantic components defined in MSC can help synthesize meaningful vector representations for data visualizations with modern learning-based methods (e.g., graph neural networks) to facilitate downstream tasks such as visualization retrieval and interaction modeling.