Distributed indexing of web scale datasets for the cloud | Zendy

Ioannis Konstantinou | Zendy; Evangelos Angelou | Zendy; Dimitrios Tsoumakos | Zendy; Nectarios Koziris | Zendy

AI Assistant Blog Pricing

Home ZAIA Blog

Open Access

Distributed indexing of web scale datasets for the cloud

Author(s) -

Ioannis Konstantinou,

Evangelos Angelou,

Dimitrios Tsoumakos,

Nectarios Koziris

Publication year - 2010

Publication title -

citeseer x (the pennsylvania state university)

Language(s) - English

Resource type - Conference proceedings

DOI - 10.1145/1779599.1779600

Subject(s) - nosql , computer science , search engine indexing , cloud computing , robustness (evolution) , database , flexibility (engineering) , distributed data store , distributed computing , distributed database , information retrieval , data mining , big data , operating system , biochemistry , chemistry , statistics , mathematics , gene

In this paper, we present a distributed architecture for indexing and serving large and diverse datasets. It incorporates and extends the functionality of Hadoop, the open source MapReduce framework, and of HBase, a distributed, sparse, NoSQL database, to create a fully parallel indexing system. Experiments with structured, semi-structured and unstructured data of various sizes demonstrate the flexibility, speed and robustness of our implementation and contrast it with similarly oriented projects. Our 11 node cluster prototype managed to keep full-text indexing time of 150GB raw content in less than 3 hours, whereas the system's response time under sustained query load of more than 1000 queries/sec was kept in the order of milliseconds. © 2010 ACM

The content you want is available to Zendy users.

Already have an account? Click here to sign in.

Having issues? You can contact us here

Accelerating Research