arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · Computer Networks and Communications

Benchmarking the Performance of Hadoop Triple Replication and Erasure Coding\non a Nation-Wide Distributed Cloud

Lakshmi J Mohan (20070738), Renji Harold (20095443), Pablo Caneleo (20070741), Udaya Parampalli (19125562), Aaron Harwood (20070744)

· Figshare

See on the map

Abstract

Large Scale distributed storage systems play a vital role in maintaining data across storage locations globally. These systems use replication as the default mechanism for providing fault-tolerance. Recently, erasure codes are being used as a viable alternative to replication, since they provide the same fault-tolerance for reduced storage overhead. However, their performance is unclear in a geographically diverse distributed storage system. This paper compares the performance of triple replication with the erasure coding (Reed-Solomon codes) used in Apache Hadoop's implementation of a distributed file system, on a cluster distributed across Australia that runs on the NeCTAR research cloud. Our results show that using erasure coding does not degrade the read performance in such a setting. We also compare the Hadoop's code with a local reconstruction code, implemented in the XORBAS version of Hadoop. These codes perform well in our clusters but the performance gain observed in our results does not conform to the results reported. Hence, we need new codes that perform better, addressing the geographical diversity issue. We believe that our framework is readily usable to test a range of novel erasure codes that are being introduced in the literature.

More like this