PhD Proposal: The Evolution of Cloud Data Architectures: Storage, Compute, and Migration

Gang Liao
10.25.2021 12:00 to 14:00

IRB 5165

Recent advances in data architectures have shifted from on-premises to the cloud. New challenges emerge as data explosion continues to expand at an exponential rate. As a result, my Ph.D. research focuses on addressing the following challenges.First, cloud data warehouses such as Snowflake, BigQuey, Redshift often rely on storage systems such as distributed file systems or object stores to store massive amounts of data. The growth of data volumes is accompanied by an increase in the number of objects stored and the amount of metadata such systems must manage. By treating metadata management similar to data management, we built FileScale, an HDFS-based file system that replaces metadata management in HDFS with a three-tiered distributed architecture that incorporates a high throughput, distributed main-memory database system at the lowest layer, along with distributed caching and routing functionality above it to enable linear scalability as the file system metadata increases.Second, Function as a Service, or FaaS, is a new system architecture that allows you to execute code in response to events without the complex infrastructure typically associated with building and launching microservices applications. FaaS such as AWS Lambda, GCP Function, and Azure Functions offer cloud functions with millisecond billing granularity to be scaled automatically, independently, and instantaneously as needed. We built Squirtle, the first practical cloud-native SQL query engine that supports event stream processing on cloud function services. We show that this architecture is more cost-effective than traditional systems, especially for dynamic workloads and continuous queries.Third, Software as a Service, or SaaS, is a method of software product delivery to end-users over the internet and via pay-as-you-go pricing in which the software is centrally hosted and managed by the cloud service provider. Continuous Deployment (CD) in SaaS, an aspect of DevOps, is the increasingly popular practice of frequent, automated deployment of software changes. To realize the benefits of CD, it must be straightforward to deploy updates to both front-end code and the database, even when the database’s schema has changed. Unfortunately, this is where current practices run into difficulty. We built BullFrog, a PostgreSQL extension that is the first system to use lazy schema migration to support single-step, online schema evolution without downtime, which achieves efficient, exactly-once physical migration of data under contention.Examining Committee:

Chair:Department Representative:Members:

Dr. Daniel Abadi Dr. Peter Keleher Dr. Amol Deshpande