Flash-based storage management in cloud computing datacenter infrastructure

of the Dissertation Flash-based Storage Management in Cloud Computing Datacenter Infrastructures by Zhengyu Yang Doctor of Philosophy in Electrical and Computer Engineering Northeastern University, July 2018 Dr. Ningfang Mi, Advisor A basic credendum of cloud computing can be summarized as: user devices are light terminals to assign jobs and gather results, while those heavy computations are conducted on remote distributed server clusters. This light-terminal-heavy-server structure makes high availability no longer an option, but a requirement in today’s datacenters. Furthermore, when bringing compute and storage capabilities into balance, we find that the biggest challenge here is closing the gap between computing and storage performance to shift storage’s curve back towards Moore’s law [1]. In detail, the time consumed to wait for I/Os is the main cause of idling and wasting CPU resources, since lots of popular cloud applications are I/O intensive, such as video streaming, file sync and backup, and data iteration for machine learning, etc [2]. Thus, storage I/O is the biggest bottleneck in large-scale datacenters. To address this bottleneck, Solid State Drives (SSDs) are widely being deployed as a per-virtual disk, second-level cache of Hard Disk Drives (HDDs) in SSD-HDD hybrid storage systems to improve I/O access performance, ascribing to SSDs’ high I/O throughput and low I/O latency and power consumption. Recently, with the capital expenditure of Flash-based SSDs keeps decreasing and the storage capacity of SSDs keeps increasing, the “sales pitch” of traditional HDDs as backend storage – low cost and large capacity – is no longer that unique, and eventually they will be replaced by low-end SSDs which have large capacity but perform orders of magnitude better than HDDs. As a result, it is widely believed that all-flash multi-tier storage systems will be adopted in the enterprise datacenters running big data platforms in the near future. Therefore, in this dissertation, we focus on flash-based storage resource management for SSD-HDD hybrid, all-flash storage systems and storage optimization in big data platforms, and we aim to achieve high availability by improving both performance and reliability in these storage systems.