Lambda Architecture Used Apache Kudu and Impala


KIPS Transactions on Computer and Communication Systems, Vol. 9, No. 9, pp. 207-212, Sep. 2020
https://doi.org/10.3745/KTCCS.2020.9.9.207,   PDF Download:
Keywords: Apache Hadoop, HDFS, Apahce Kudu, Apache Impala, Lambda Architecture, IoT
Abstract

The amount of data has increased significantly due to advances in technology, and various big data processing platforms are emerging, to handle it. Among them, the most widely used platform is Hadoop developed by the Apache Software Foundation, and Hadoop is also used in the IoT field. However, the existing Hadoop-based IoT sensor data collection and analysis environment has a problem of overloading the name node due to HDFS’ Small File, which is Hadoop’s core project, and it is impossible to update or delete the imported data. This paper uses Apache Kudu and Impala to design Lambda Architecture. The proposed Architecture classifies IoT sensor data into Cold-Data and Hot-Data, stores it in storage according to each personality, and uses Batch-View created through Batch and Real-time View generated through Apache Kudu and Impala to solve problems in the existing Hadoop-based IoT sensor data collection analysis environment and shorten the time users access to the analyzed data.


Statistics
Show / Hide Statistics

Statistics (Cumulative Counts from September 1st, 2017)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.


Cite this article
[IEEE Style]
Y. Hwang, P. Lee, Y. Shin, "Lambda Architecture Used Apache Kudu and Impala," KIPS Transactions on Computer and Communication Systems, vol. 9, no. 9, pp. 207-212, 2020. DOI: https://doi.org/10.3745/KTCCS.2020.9.9.207.

[ACM Style]
Yun-Young Hwang, Pil-Won Lee, and Yong-Tae Shin. 2020. Lambda Architecture Used Apache Kudu and Impala. KIPS Transactions on Computer and Communication Systems, 9, 9, (2020), 207-212. DOI: https://doi.org/10.3745/KTCCS.2020.9.9.207.