Fault-tolerant technology for big data cluster in distributed flow processing system

Issue title: Big Data Networking-Challenges and Applications

Guest editors: Zhixiang Hou and Chong Shen

Article type: Research Article

Authors: Jia, Zhicheng^{; *}

Affiliations: Gansu University of Political Science and Law, Lanzhou 730070, China. E-mail: [email protected]

Correspondence: [*] Corresponding author. E-mail: [email protected].

Abstract: In order to overcome the problem of poor recovery and stability of traditional big data fault-tolerant technology, a fault-tolerant technology of big data cluster in distributed flow processing system is proposed. Using the target protocol to build the cache data fault-tolerant mechanism of the distributed flow processing system and the disk data fault-tolerant mechanism, so as to build the internal data fault-tolerant mechanism of the system. By using spark application framework, a fault-tolerant model of big data cluster is built to realize the fault-tolerant of big data cluster in distributed flow processing system. The experimental results show that compared with the traditional methods, the recovery rate of big data cluster fault-tolerant technology proposed in this paper is higher, the highest recovery efficiency is 99.83%, the stability is stronger, and it is more suitable for big data fault-tolerant processing of distributed flow processing system.

Keywords: Distributed flow processing system, big data, cluster fault-tolerant technology, disk data fault-tolerant mechanism

DOI: 10.3233/WEB-200432

Journal: Web Intelligence, vol. 18, no. 2, pp. 101-110, 2020