Applying big data warehousing and visualization techniques on pingER data

ABSTRACT

Nowadays, the Internet has turned into a crucial piece of our cutting edge society. It is a stage of exploration, financial development, democratic participation and speech. The operations of the Internet have prompted a huge development and collection of information known as Big Data. Therefore, it is important to monitor and measure the Quality of Service (QoS) of Internet traffic. The SLAC National Accelerator Laboratory started the PingER project in 1995 to measure the End-to-End Internet performance history of servers and routers worldwide. The project involves measurements of the 700 monitored sites in over 160 countries. PingER Monitoring Agents (MAs) ping a list of monitored sites after every 30 minutes to obtain Round Trip Time (RTT) values revealing interesting information about Internet performance (e.g., RTT, jitter, packet loss and unreachability) major events (e.g., fiber cuts, earthquakes, and social upheavals). Thus, the project has collected a vast amount of historical Internet Performance data worldwide since 1995. Currently, the data is stored in flat text files, making it difficult to analyze collectively. In addition, this simplistic format limits the analytical potential of this data. In this paper, we propose an approach to process, store, analyze and visualize PingER data. A Data warehouse is created which combines Hadoop Big Data techniques. The data are processed by using Sci-cumulus MR workflow, stored in HDFS, analyzed by Impala queries and visualized by using Google API's. This approach makes PingER data more accessible and enhances its potential contribution to ongoing research and application development.

References

1. R. L. Cottrell, "Tutorial on Internet Monitoring & PingER at SLAC" {Online} Available at http://pinger.seecs.edu.pk/tutorial/tutorial.html 07 july,2015.
2. W. Fan and A. Bifet, "Mining Big Data: Current Status, and Forecast to the Future," Special Interest Group on Knowledge Discovery in Data (SIGKDD), vol. 14, no. 2, pp. 1--5.
3. Q. Liu, B. Ribeiro, A. H. Sung, and D. Suryakumar, "Mining the Big Data: The Critical Feature Dimension Problem," 2014 IIAI 3rd Int. Conf. Adv. Appl. Informatics, no. 2010, pp. 499--504, Aug. 2014.
4. S.-H. Liao, P.-H. Chu, and P.-Y. Hsiao, "Data mining techniques and applications - A decade review from 2000 to 2011," Expert Syst. Appl., vol. 39, no. 12, pp. 11303--11311, Sep. 2012.
5. Y. Qi, "Data Analysis Analysis Visualization in Media Big Data", 2015 IEEE/ACIS 14th Int. Conf. on Comp. and Info. Science (ICIS), pp. 0--3, 2015.
6. X. Bai, D. White, and D. Sundaram, "Context adaptive visualization for effective business intelligence," 2013 15th IEEE Int. Conf. Commun. Technol., pp. 786--790, Nov. 2013.
7. R. F. Souza, L. Cottrell, B. White, M. L. Campos, and M. Mattoso, "Linked Open Data Publication Strategies: Application in Networking Performance Measurement Data," 2014 ASE BIGDATA/SOCIALCOM/CYBERSECURITY conf. stanford university, pp. 1--7, 2014.
8. Nabi, "Implementation of Relational archive site for PingER" {Online}. Available at https://confluence.slac.stanford.edu/display/IEPM/Implementation+of+Relational+archive+site+for+PingER. 26 July, 2011.
9. T. M. S. Barbosa, R. Souza, S. M. S. Cruz, M. L. Campos, and R. Les Cottrell, "Applying Data Warehousing and Big Data Techniques to Analyze Internet Performance," 2015 NETAPPS 4th Int. Conf. on Internet Applications, Protocols and Services, pp. 31--36, 2015.
10. R. Kimball, "The Data Warehouse Lifecycle Toolkit: Expert Methods for Designing, Developing, and Deploying Data Warehouses Architecture," p. 771, 1998.
11. A. Zoss, "Introduction to Data Visualization: Visualization Types" {Online}. Available at http://guides.library.duke.edu/datavis/vis_types. 8 December, 2015.
12. S. Chaudhuri, and U. Dayal, "An overview of data warehousing and OLAP technology," ACM SIGMOD Special Interest Group on Management Of Data, vol. 26, no 1, pp. 65--74, 1997.
13. X. Wu, X. Zhu, and S. Member, "Data Mining with Big Data," IEEE trans. on knowl. and data eng., vol. 26, no. 1, pp. 97--107, 2014.
14. Burbank, D., The 5 V's of Big Data {Online}. Available at http://enterprisearchitects.com/the-5v-s-of-big-data/ (May, 2016).
15. Q. Fu, W. Liu, T. Xue, H. Gu, S. Zhang, and C. Wang, "A BIG DATA PROCESSING METHODS FOR VISUALIZATION," 2014 IEEE 3rd Int. Conf. on Cloud Comp. and Intelligence Systems, 2014.
16. K. Krishnan, "Big Data Processing Architectures", In Data Warehousing in the Age of Big Data. P.29--42 Part 1 (2nd Ed.) Elsevier.2013
17. B. Hedlund, "Understanding Hadoop Clusters and the Network" {Online}. Available at http://bradhedlund.com/2011/09/10/understanding-hadoop-clusters-and-the-network/. 10 September, 2011.
18. R. Vijayakumari, R. Kirankumar, and K. G. Rao, "Comparative analysis of Google File System and Hadoop Distributed File System," Int. Journal of Advanced Trends in Computer Science and Engineering, vol. 3, no. 1, pp. 553--558, 2014.
19. Apache HDFS, "HDFS Architecture Guide" {Online}. Available at http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html.
20. E. Indarto, "Data Mining" {Online}. Available at http://recommendersystems.readthedocs.io/en/latest/datamining.html. 5 July, 2013.
21. SciCumulus, "SciCumulus/C2 - Parallel Scientific Workflow Management System" {Online}. Available at https://scicumulusc2.wordpress.com/starter-guide-2/. 20 May, 2016.
22. Cloudera, "Cloudera Data Management" {Online}. Available at http://www.cloudera.com/documentation/enterprise/latest/topics/datamgmt.html. 20 May, 2016.
23. Cloudera, "Cloudera Impala" {Online}. Available at http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html. 20 May, 2016.
24. Cloudera, "Using the Parquet File Format with Impala Tables" {Online}. Available at http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/latest/topics/impala_parquet.html. 20 May, 2016.