English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

This paper focuses on the low efficiency of Spark SQL reading massive small files on HDFS in 4G Industry Application Card (IAC) business analysis system. To solve this issue, we propose a Local Merge Storage Model (LMSM) for 4G IAC small files. In this model, locality is enhanced by exploring the type and time of small files. Then, Spark is used to merge small files into the Parquet column storage file and store them to HDFS. Finally, according to the experimental results, after merging partitioned storage of small files, Spark SQL query efficiency increases up to 60 times higher.

Research of Spark SQL Query Optimization Based on Massive Small Files on HDFS