Abstract
iii Modern datacenters utilize traditional Ethernet networks to connect hundreds or thousands of machines. Although inexpensive and ubiquitous, Ethernet imposes design constraints on datacenter-scale distributed storage systems that use traditional client-server architectures. Round-trip latency around 100µs means that minimizing per-operation round trips and maximizing data locality is vital to ensure low application latency. To maintain liveness, sufficient server CPU cores must be provisioned to satisfy peak load, or clients need to wait for additional server CPU resources to be spun up during load spikes. Recent technological trends indicate that future datacenters will embrace interconnects with ultra-low latency, high bandwidth, and the ability to offload work from servers to clients. Interconnects like Infiniband found in traditional supercomputing clusters already provide these features, including < 3µs latency, 40 − 80 Gbps throughput, and Remote Direct Memory Access (RDMA), which allows access to another machine’s memory without involving its CPU. Future datacenter-scale distributed storage systems will need to be designed specifically to exploit these features. This thesis explores what these features mean for large-scale in-memory distributed storage systems, and derives two key insights for building