Designing efficient and scalable cache management systems

Software-managed caches have been ubiquitously deployed in today's system infrastructure. From personal devices to servers on the edge and the cloud, these caches speed up data access, reduce data movement, and avoid repeated computation. However, they consume a huge amount of resources, i.e., DRAM and CPUs. In this talk, I will discuss how to design efficient and scalable cache systems. In the first part, I will demonstrate that the efficiency of a key-value cache is not only determined by the eviction algorithm but also by other components, e.g., storage layout and expiration design. I will then describe how I designed Segcache to reduce memory footprint by up to 60% and increase throughput by 8x compared to state-of-the-art systems. Segcache has been adopted for production at Twitter and Momento. In the second part, I will introduce a surprising new finding from our largest-scale eviction algorithm study: FIFO queues are all we need for cache eviction. I will then describe S3-FIFO, a new cache eviction algorithm that is simpler, more scalable, and more efficient than state-of-the-art algorithms. S3-FIFO has been adopted for production at Google, VMWare, Redpanda, and several others. Finally, I will describe my future work on building efficient, performant, and robust data systems.