English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Nowadays, Hadoop distributed file system as a distributed storage system, has a good effect on the storage of large files. However, there is a natural flaw in the storage of small files: storing a large number of small files will produce excessive metadata, resulting in namenode memory bottlenecks; frequent RPC communications will cause time consumption due to over-provisioning. To solve these problems, this paper presents a merging algorithm based on two factors: the distribution of files and the correlation of files. The algorithm can not only reduce the HDFS blocks, but also make relevant files close. Experimental results show that the algorithm effectively improves the storage efficiency of HDFS on small files and help to optimize the access of small files. Introduction With the rapid development of Internet technology in recent years, the era of big data has arrived. Due to the growing demands for data storage and access, many distributed technologies and frameworks emerge as the times require. One of them is Apache Hadoop[1]. Because of its advantages on robustness, scalability and cost to other platforms, Hadoop has become the most widely used Big Data analytics platform today. Hadoop Distribute File System (HDFS)[2] has the characteristics of high throughput and high fault tolerance, and has become the most important data storage framework in the industry. The HDFS framework is a master-slave architecture. The master node Namenode is responsible for providing the metadata service; the slave node Datanodes is responsible for storing the data by blocks. However, HDFS can not cope with the storage and access of a large number of small files: when storing small files, it generates too much metadata information and runs out of memory of the Namenode; when accessing small files, clients frequently make calls through RPC Getting metadata about the block where the small file resides adds significant time costs. In addition, HDFS allocates 128MB of data blocks for each file for storage, and the independent occupation of blocks will causes storage space fragmentation. At present, one of the solutions to the small file problem on the HDFS is to merge the files into bigger files and then store them. Two factors need to be considered in the process of merge: 1) the relevance of small files, to ensure the efficiency of small files; 2) the distribution of small files, to ensure maximum use of storage space, optimization the processing efficiency of platform. This paper fully considers and weighs the above two factors. Based on the quantitative analysis of file relevance, this paper proposes a uniform merge algorithm based on file relevancy.

An optimization strategy of massive small files storage based on HDFS