MataNui–A Distributed Storage Infrastructure for Scientific Data

In science and engineering the problem of «sanity» in data management is quite common. Particularly, if partners within a project are geographically distributed, and require access to this data. These partners would ideally like to access or store data using a local server, while still retaining remote access. Furthermore, data does not «live alone,» but has a «partner»: Meta- data. Currently available solutions cater for different needs, and do not fully address the problems in our own environment. Particularly, seamless integration within a Grid infrastructure is essential.For these reasons, the presented MataNui data server infrastructure offers full storage of files along with arbitrarily exten- sive meta-data for each stored item. The server can replicate its content between geographically distributed locations, making each usable for full read/write access, enabling researchers to use the one closest to their own network. Files are accessible through GridFTP as the predominant protocol for Grid Computing. For full access to meta-data and server side search queries a (RESTful) Web Service is provided. The current implementation features very high performance in data throughput on storage, retrieval and query operations. Through the Web Service front end it is straight forward to integrate it with scientific workflow environments or scientific data management applications