z-logo
Premium
Scalable, asynchronous, distributed eigen monitoring of astronomy data streams
Author(s) -
Bhaduri Kanishka,
Das Kamalika,
Borne Kirk,
Giannella Chris,
Mahule Tushar,
Kargupta Hillol
Publication year - 2011
Publication title -
statistical analysis and data mining: the asa data science journal
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.381
H-Index - 33
eISSN - 1932-1872
pISSN - 1932-1864
DOI - 10.1002/sam.10120
Subject(s) - computer science , scalability , data stream mining , asynchronous communication , terabyte , data mining , large synoptic survey telescope , sky , real time computing , astronomy , database , physics , computer network , operating system
In this paper, we develop a distributed algorithm for monitoring the principal components (PCs) for next generation of astronomy petascale data pipelines such as the Large Synoptic Survey Telescopes (LSST). This telescope will take repeated images of the night sky every 20 s, thereby generating 30 terabytes of calibrated imagery every night that will need to be co‐analyzed with other astronomical data stored at different locations around the world. Event detection, classification, and isolation in such data sets may provide useful insights to unique astronomical phenomenon displaying astrophysically significant variations: quasars, supernovae, variable stars, and potentially hazardous asteroids. However, performing such data mining tasks is a challenging problem for such high‐throughput distributed data streams. In this paper, we propose a highly scalable and distributed asynchronous algorithm for monitoring the PCs of such dynamic data streams and discuss a prototype web‐based system PADMINI (Peer‐to‐Peer Astronomy Data Mining) which implements this algorithm for use by the astronomers. We demonstrate the algorithm on a large set of distributed astronomical data to accomplish well‐known astronomy tasks such as measuring variations in the fundamental plane of galaxy parameters. The proposed algorithm is provably correct (i.e., converges to the correct PCs without centralizing any data) and can seamlessly handle changes to the data or the network. Real experiments performed on Sloan Digital Sky Survey (SDSS) catalogue data show the effectiveness of the algorithm. © 2011 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 2011

This content is not available in your region!

Continue researching here.

Having issues? You can contact us here