NSF Award
2412437
With the rapid growth of cloud computing, artificial intelligence (AI), machine learning (ML), and scientific computing, a massive amount of unstructured data is being created. To store and access unstructured data, Log-Structured-Merge tree-based Key-Value Stores (LSM-KVS) have become essential data storage systems, widely deployed by most of the large IT service companies. As application workloads vary and continuously scale up, existing LSM-KVS systems, designed based on monolithic servers and shared-nothing architectures, face numerous issues, including resource inefficiency, difficulty in load balancing, low scalability, and poor elasticity. This project aims to develop and optimize LSM-KVS-based systems within disaggregated infrastructure environments, consisting of multiple compute servers and heterogeneous memory and storage farms connected via fast networks. The project will address several fundamental issues, including heavy network traffic between resource pools caused by compaction and shard-migration, memory limitations of read and write buffers, tightly coupled control of otherwise decoupled resource-intensive modules by LSM-KVS, and more frequent and complex transient errors. <br/><br/>The goal of this project is to redesign and optimize an LSM-KVS architecture for disaggregated infrastructure, called Decoupled-LSM, to achieve higher performance, better resource utilization, and improved management. The Decoupled-LSM project will devise new techniques that decouple data-intensive modules from the control of LSM-KVS, execute the modules in different resource pools efficiently, and attain high performance, better resource utilization, and greater flexibility in disaggregated environments. Decoupled-LSM will lay the foundation for a new LSM-KVS architecture optimized over disaggregated infrastructure for many critical applications, such as cloud computing, AI, ML, and scientific computing, which impact daily life. Overall, this project can help to better store, use, and manage extremely large-scale unstructured data, used by government, industry, and individuals. The proposed methodologies, system designs, and implemented components will benefit the storage and networking research communities in further developing storage systems for disaggregated infrastructure and cloud computing. The project will also involve students from underrepresented groups and outreach to high schools, along with collaboration with industry partners.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
