IAHR Document Library

« Back to Library Homepage « Proceedings of the 1st International Symposium on Water Syst...

Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop

Author(s): Jia-Ke Lv; Yang Li; Xuan Wang

Linked Author(s):

Keywords: No keywords

Abstract: The log data real-time processing platform which is built using Storm On YARN integrated MapReduce and Storm that use MapReduce to complete large-scale off-line data global knowledge extraction, sudden knowledge extraction of small-scale data in Kafka buffers through Storm, and continuous real-time calculation of streaming data in combination with global knowledge. We tested our technique with the well-known KDD99 CUP data set. The experimentation results prove the system to be effective and efficient.


Year: 2018

Copyright © 2024 International Association for Hydro-Environment Engineering and Research. All rights reserved. | Terms and Conditions