z-logo
Premium
Robust and efficient memory management in Apache AsterixDB
Author(s) -
Kim Taewoo,
Behm Alexander,
Blow Michael,
Borkar Vinayak,
Bu Yingyi,
Carey Michael J.,
Hubail Murtadha,
Jahangiri Shiva,
Jia Jianfeng,
Li Chen,
Luo Chen,
Maxon Ian,
Pirzadeh Pouria
Publication year - 2020
Publication title -
software: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.437
H-Index - 70
eISSN - 1097-024X
pISSN - 0038-0644
DOI - 10.1002/spe.2799
Subject(s) - computer science , memory management , memory map , big data , flat memory model , interleaved memory , cache only memory architecture , extended memory , shared memory , operating system , semiconductor memory
Summary Traditional relational database systems handle data by dividing their memory into sections such as a buffer cache and working memory, assigning a memory budget to each section to efficiently manage a limited amount of overall memory. They also assign memory budgets to memory‐intensive operators such as sorts and joins and control the allocation of memory to these operators; each memory‐intensive operator attempts to maximize its memory usage to reduce disk I/O cost. Implementing such memory‐intensive operators requires a careful design and application of appropriate algorithms that properly utilize memory. Today's Big Data management systems need the ability to handle large amounts of data similarly, as it is unrealistic to assume that truly big data will fit into memory. In this article, we share our memory management experiences in Apache AsterixDB, an open‐source Big Data management software platform that scales out horizontally on shared‐nothing commodity computing clusters. We describe the implementation of AsterixDB's memory‐intensive operators and their designs related to memory management. We also discuss memory management at the global (cluster) level. We conducted an experimental study using several synthetic and real datasets to explore the impact of this work. We believe that future Big Data management system builders can benefit from these experiences.

This content is not available in your region!

Continue researching here.

Having issues? You can contact us here