Yong Ge
Abstract
The management of large-scale distributed information systems relies on the effective use and modeling of monitoring data collected at various points in the distributed information systems. A traditional approach to model monitoring data is to discover invariant relationships among the monitoring data. Indeed, we can discover all invariant relationships among all pairs of monitoring data and generate invariant networks, where a node is a monitoring data source (metric) and a link indicates an invariant relationship between two monitoring data. Such an invariant network representation can help system experts to localize and diagnose the system faults by examining those broken invariant relationships and their related metrics, since system faults usually propagate among the monitoring data and eventually lead to some broken invariant relationships. However, at one time, there are usually a lot of broken links (invariant relationships) within an invariant network. Without proper guidance, it is difficult for system experts to manually inspect this large number of broken links. To this end, in this article, we propose the problem of ranking metrics according to the anomaly levels for a given invariant network, while this is a nontrivial task due to the uncertainties and the complex nature of invariant networks. Specifically, we propose two types of algorithms for ranking metric anomaly by link analysis in invariant networks. Along this line, we first define two measurements to quantify the anomaly level of each metric, and introduce the mRank algorithm. Also, we provide a weighted score mechanism and develop the gRank algorithm, which involves an iterative process to obtain a score to measure the anomaly levels. In addition, some extended algorithms based on mRank and gRank algorithms are developed by taking into account the probability of being broken as well as noisy links. Finally, we validate all the proposed algorithms on a large number of real-world and synthetic data sets to illustrate the effectiveness and efficiency of different algorithms.
- L. Akoglu, M. Mcglohon, and C. Faloutsos. 2009. Anomaly detection in large graphs. In CMU-CS-09-173 Technical Report.Google Scholar
- S. Brin and L. Page. 1998. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems 30, 1--7. Google ScholarDigital Library
- H. Chen, H. Cheng, G. Jiang, and K. Yoshihira. 2008. Exploiting local and global invariants for the management of large scale information systems. In Poceedings of the 8th IEEE International Conference on Data Mining. 113--122. Google ScholarDigital Library
- H. Chen, G. Jiang, and K. Yoshihira. 2010. Invariants based failure diagnosis in distributed computing systems. In Proceedings of the 29th IEEE International Symposium on Reliable Distributed Systems. 160--166. Google ScholarDigital Library
- W. Eberle, L. Holder, and D. Cook. 2009. Identifying threats using graph-based anomaly detection. Machine Learning in Cyber Trust 2, 73--108.Google Scholar
- W. Eberle, L. Holder, and B. Massengill. 2012. Graph-based anomaly detection applied to homeland security cargo screening. In Proceedings of the 25th International Florida Artificial Intelligence Research Society Conference. 382--387.Google Scholar
- W. Eberle and L. B. Holder. 2009. Applying graph-based anomaly detection approaches to the discovery of insider threats. In Proceedings of IEEE Intelligence and Security Informatics. 206--208. Google ScholarDigital Library
- J. Gertler. 1998. Fault Detection and Diagnosis in Engineering Systems. Marcel Dekker.Google Scholar
- S. Ghanbari and C. Amza. 2002. Semantic-driven model composition for accurate anomaly diagnosis. In Proceedings of the 24th International Conference on Software Engineering. 291--301.Google Scholar
- S. Hangal and M. S. Lam. 2008. Tracking down software bugs using automatic anomaly detection. In Proceedings of the International Conference on Autonomic Computing. 35--44.Google Scholar
- K. Inoue, R. Yokomori, H. Fujiwara, T. Yamamoto, M. Matsushita, and S. Kusumoto. 2003. Component rank: Relative significance rank for software component search. In Proceedings of the 25th International Conference on Software Engineering. 14--24. Google ScholarDigital Library
- R. Isermannn and P. Balle. 1997. Trends in the application of model-based fault detection and diagnosis of industrial process. Control Engineering Practice 5, 5, 709--719.Google ScholarCross Ref
- K. Jarvelin and J. Kekalainen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems 20, 4, 422--446. Google ScholarDigital Library
- G. Jiang, H. Chen, and K. Yoshihira. 2006a. Discovering likely invariants of distributed transaction systems for autonomic system management. In Proceedings of the 3rd International Conference on Autonomic Computing. 199--208. Google ScholarDigital Library
- G. Jiang, H. Chen, and K. Yoshihira. 2006b. Modeling and tracking of transaction flow dynamics for fault detection in complex systems. IEEE Transactions on Dependable and Secure Computing 3, 4, 312--326. Google ScholarDigital Library
- G. Jiang, H. Chen, and K. Yoshihira. 2007. Efficient and scalable algorithms for inferring likely invariants in distributed systems. Transactions on Knowledge and Data Engineering 19, 11, 1508--1523. Google ScholarDigital Library
- M. Jiang, M. A. Munawar, T. Reidemeister, and P. A. S. Ward. 2008. Detection and diagnosis of recurrent faults in software systems by invariant analysis. In Proceedings of the 11th IEEE High Assurance Systems Engineering Symposium. 323--332. Google ScholarDigital Library
- M. Jiang, M. A. Munawar, T. Reidemeister, and P. A. S. Ward. 2009. System monitoring with metric-correlation models: problems and solutions. In Proceedings of the International Conference on Autonomic Computing. 13--22. Google ScholarDigital Library
- M. Jiang, M. A. Munawar, T. Reidemeister, and P. A. S. Ward. 2011. Efficient fault detection and diagnosis in complex software systems with information-theoretic monitoring. IEEE Transactions on Dependable and Secure Computing 8, 4, 510--522. Google ScholarDigital Library
- D. Q. Le, T. Jeong, H. E. Roman, and J. W.-K. Hong. 2011. Traffic dispersion graph based anomaly detection. In Proceedings of the 2nd Symposium on Information and Communication Technology. 36--41. Google ScholarDigital Library
- C. Liu, X. Yan, H. Yu, J. Han, and P. S. Yu. 2005. Mining behavior graphs for backtrace of noncrashing bugs. In Proceedings of SIAM International Conference on Data Ming. 286--297.Google Scholar
- L. Ljung. 1998. System Identification: Theory for the User, 2nd ed. Prentice Hall PTR. Google ScholarDigital Library
- H. D. K. Moonesinghe and P.-N. Tan. 2008. Outrank: a graph-based outlier detection framework using random walk. International Journal on Artificial Intelligence Tools 17, 1, 19--36.Google ScholarCross Ref
- C. C. Noble and D. J. Cook. 2003. Graph-based anomaly detection. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 631--636. Google ScholarDigital Library
- H. Shan, G. Jiang, and K. Yoshihira. 2010. Extracting overlay invariants of distributed systems for autonomic system management. In Proceedings of the 4th IEEE International Conference on Self-Adaptive and Self-Organizing Systems. 41--50. Google ScholarDigital Library
- H. Valizadegan, R. Jin, R. Zhang, and J. Mao. 2009. Learning to rank by optimizing n measure. In Proceedings of the 23rd Annual Conference on Neural Information Processing Systems.Google Scholar
- S. Wei, Y. Zhao, Z. Zhu, and N. Liu. 2010. Multimodal fusion for video search reranking. Transactions on Knowledge and Data Engineering 22, 8, 1191--1199. Google ScholarDigital Library
- S. A. Yemini, S. Kliger, E. Mozes, Y. Yemini, and D. Ohsie. 1996. High speed and robust event correlation. IEEE Communications Magazine 34, 5, 82--90. Google ScholarDigital Library
Index Terms
- Ranking Metric Anomaly in Invariant Networks
Recommendations
Metric Ranking of Invariant Networks with Belief Propagation
ICDM '14: Proceedings of the 2014 IEEE International Conference on Data MiningThe management of large-scale distributed information systems relies on the effective use and modeling of monitoring data collected at various points in the distributed information systems. A promising approach is to discover invariant relationships ...
Generic radial orthogonal moment invariants for invariant image recognition
As the variation of parameters in Jacobi polynomial, Jacobi-Fourier moments can form various types of orthogonal moments: Legendre-Fourier moments, Orthogonal Fourier-Mellin moments, Zernike moments, pseudo-Zernike moments, and so on. In this paper, we ...
3D radial invariant of dual Hahn moments
In this work, we propose new sets of 2D and 3D rotation invariants based on orthogonal radial dual Hahn moments, which are orthogonal on a non-uniform lattice. We also present theoretical mathematics to derive them. Thus, this paper presents in the ...
Comments