Testing of Information Storage System Using Multidimensional Parity Algorithms Resistant to Partial Loss of Storage Locations

Abstract

The data of testing the distributed storage system with data splitting approved in the internal network of the organization are given. One of the main parameters of the tested distributed storage system is the splitting level, which is responsible for the parity dimension generated in the splitting process. In the course of tests, in which the levels of splitting gradually increased, the optimal value for the information network of the organization was established. With the increase in the level of splitting and the associated parity dimension, the resistance to losses of split parts of files distributed among the nodes involved in storing information increases. At the same time, with the increase in the level of splitting, the time of information processing in the system also increases. Testing has shown that the most optimal levels of splitting are the levels from the second to the third, at which there is a reasonable compromise between the reliability of storage and latency of the system. At these levels, up to 33% of nodes are guaranteed to recover files 100% in case of failure, and the probability of information loss was 0.07 in the case of 44% of nodes failure. The most relevant distributed storage of information with data splitting for large volumes that require the use of multiple storage locations in which it is known that virtually the only method to achieve reliability is multiple redundancy and replication (Hadoop FS, ZFS). The described storage method, having less redundancy with comparable reliability, can be recommended for use in Big Data systems.