Aleksandar Milenkoski
Marco Vieira
Skip Abstract Section
Skip Supplemental Material SectionAbstract
The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this article, we survey and systematize common practices in the area of evaluation of such systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.
Supplemental Material
Available for Download
zip
milenkoski.zip (143.1 KB)
Supplemental movie, appendix, image and software files for, Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices
- Abdulbasit Ahmed, Alexei Lisitsa, and Clare Dixon. 2011. A misuse-based network intrusion detection system using temporal logic and stream processing. In Proceedings of the 5th International Conference on Network and System Security (NSS’11). 1--8.Google Scholar
Cross Ref
- Miriam Allalouf, Muli Ben-Yehuda, Julian Satran, and Itai Segall. 2010. Block storage listener for detecting file-level intrusions. In Proceedings of the 26th Symposium on Mass Storage Systems and Technologies (MSST’10). 1--12. Google ScholarDigital Library
- Faeiz Alserhani, Monis Akhlaq, Irfan U. Awan, Andrea J. Cullen, and Pravin Mirchandani. 2010. MARS: Multi-stage attack recognition system. In Proceedings of the 24th IEEE International Conference on Advanced Information Networking and Applications (AINA’10). IEEE, Los Alamitos, CA, 753--759. Google ScholarDigital Library
- Alberto Avritzer, Rajanikanth Tanikella, Kiran James, Robert G. Cole, and Elaine J. Weyuker. 2010. Monitoring for security intrusion using performance signatures. In Proceedings of the 1st Joint WOSP/SIPEW International Conference on Performance Engineering (WOSP/SIPEW’10). 93--104. Google ScholarDigital Library
- Stefan Axelsson. 2000. The base-rate fallacy and its implications for the difficulty of intrusion detection. ACM Transactions on Information and Systems Security 3, 3, 186--205. Google ScholarDigital Library
- Saketh Bharadwaja, Weiqing Sun, Mohammed Niamat, and Fangyang Shen. 2011. Collabra: A Xen hypervisor based collaborative intrusion detection system. In Proceedings of the 8th International Conference on Information Technology: New Generations (ITNG’11). IEEE, Los Alamitos, CA, 695--700. Google ScholarDigital Library
- Tsung-Huan Cheng, Ying-Dar Lin, Yuan-Cheng Lai, and Po-Ching Lin. 2012. Evasion techniques: Sneaking through your intrusion detection/prevention systems. IEEE Communications Surveys and Tutorials 14, 4, 1011--1020.Google ScholarCross Ref
- Chien-Yi Chiu, Yuh-Jye Lee, Chien-Chung Chang, Wen-Yang Luo, and Hsiu-Chuan Huang. 2010. Semi-supervised learning for false alarm reduction. In Advances in Data Mining: Applications and Theoretical Aspects. Lecture Notes in Computer Science, Vol. 6171. Springer, Berlin, 595--605. Google ScholarDigital Library
- Chun-Jen Chung, Khatkar Pankaj, Tianyi Xing, Jeongkeun Lee, and Dijiang Huang. 2013. NICE: Network intrusion detection and countermeasure selection in virtual network systems. IEEE Transactions on Dependable and Secure Computing 10, 4, 198--211. Google ScholarDigital Library
- Simon P. Chung and Aloysius K. Mok. 2006. On random-inspection-based intrusion detection. In Proceedings of the 8th International Conference on Recent Advances in Intrusion Detection (RAID’06). 165--184. Google ScholarDigital Library
- Scott E. Coull, Charles V. Wright, Fabian Monrose, Michael P. Collins, and Michael K. Reiter. 2007. Playing devils advocate: Inferring sensitive information from anonymized network traces. In Proceedings of the Network and Distributed System Security Symposium. 35--47.Google Scholar
- Robert K. Cunningham, R. P. Lippmann, D. J. Fried, S. L. Garfinkel, I. Graf, K. R. Kendall, S. E. Webster, D. Wyschogrod, and M. A. Zissman. 1999. Evaluating intrusion detection systems without attacking your friends: The 1998 DARPA Intrusion Detection Evaluation. In Proceedings of the SANS 1999 Workshop on Securing Linux.Google Scholar
- Hervé Debar, Marc Dacier, and Andreas Wespi. 1999. Towards a taxonomy of intrusion-detection systems. Computer Networks 31, 9, 805--822. Google ScholarCross Ref
- Hervé Debar, Marc Dacier, Andreas Wespi, and Stefan Lampart. 1998. An Experimentation Workbench for Intrusion Detection Systems. Technical Report. IBM Research, Zurich Research Laboratory.Google Scholar
- Alex Dehnert. 2012. Intrusion detection using VProbes. VMware Technical Journal 1, 2, 28--31.Google Scholar
- Holger Dreger, Anja Feldmann, Vern Paxson, and Robin Sommer. 2008. Predicting the resource consumption of network intrusion detection systems. In Proceedings of the 2008 International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS’08). ACM, New York, NY, 437--438. Google ScholarDigital Library
- Tudor Dumitras and Darren Shou. 2011. Toward a standard benchmark for computer security research: The worldwide intelligence network environment (WINE). In Proceedings of the 1st Workshop on Building Analysis Datasets and Gathering Experience Returns for Security. ACM, New York, NY, 89--96. Google ScholarDigital Library
- George W. Dunlap, Samuel T. King, Sukru Cinar, Murtaza A. Basrai, and Peter M. Chen. 2002. ReVirt: Enabling intrusion analysis through virtual-machine logging and replay. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI’02). ACM, New York, NY, 211--224. Google ScholarDigital Library
- Robert Durst, Terrence Champion, Brian Witten, Eric Miller, and Luigi Spagnuolo. 1999. Testing and evaluating computer intrusion detection systems. ACM Communications 42, 7, 53--61. Google ScholarDigital Library
- Y. Frank Jou, Fengmin Gong, Chandru Sargor, Xiaoyong Wu, Shyhtsun F. Wu, Heng-Chia Chang, and Feiyi Wang. 2000. Design and implementation of a scalable intrusion detection system for the protection of network infrastructure. In Proceedings of DARPA Information Survivability Conference and Exposition, Vol. 2. 69--83.Google Scholar
- Josè Fonseca, Marco Vieira, and Henrique Madeira. 2014. Evaluation of Web security mechanisms using vulnerability and attack injection. IEEE Transactions on Dependable and Secure Computing 11, 5, 440--453.Google ScholarCross Ref
- Romain Fontugne, Pierre Borgnat, Patrice Abry, and Kensuke Fukuda. 2010. MAWILab: Combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking. In Proceedings of the 6th International Conference on Emerging Networking Experiments and Technologies (CoNEXT’10). Article No. 8. Google ScholarDigital Library
- James C. Foster. 2007. Metasploit Toolkit for Penetration Testing, Exploit Development, and Vulnerability Research. Syngress Publishing. Google ScholarDigital Library
- Mohammed Gad El Rab. 2008. Evaluation des systèmes de détection d’intrusion. Ph.D. Dissertation. Université Paul Sabatier—Toulouse III.Google Scholar
- John E. Gaffney and Jacob W. Ulvila. 2001. Evaluation of intrusion detectors: A decision theory approach. In Proceedings of the 2001 IEEE Symposium on Security and Privacy. 50--61. Google ScholarDigital Library
- Tal Garfinkel and Mendel Rosenblum. 2003. A virtual machine introspection based architecture for intrusion detection. In Proceedings of the Network and Distributed Systems Security Symposium. 191--206.Google Scholar
- John Linwood Griffin, Adam Pennington, John S. Bucy, Deepa Choundappan, Nithya Muralidharan, and Gregory R. Ganger. 2003. On the Feasibility of Intrusion Detection Inside Workstation Disks. Research Paper. Carnegie-Mellon University, Pittsburgh, PA.Google Scholar
- Guofei Gu, Prahlad Fogla, David Dagon, Wenke Lee, and Boris Skorić. 2006. Measuring intrusion detection capability: An information-theoretic approach. In Proceedings of the 2006 ACM Symposium on Information, Computer, and Communications Security (ASIACCS’06). ACM, New York, NY, 90--101. Google ScholarDigital Library
- Mike Hall and Kevin Wiley. 2002. Capacity verification for high speed network intrusion detection systems. In Proceedings of the 5th International Conference on Recent Advances in Intrusion Detection (RAID’02). Springer-Verlag, Berlin, 239--251. Google ScholarDigital Library
- Amin Hassanzadeh and Radu Stoleru. 2011. Towards optimal monitoring in cooperative IDS for resource constrained wireless networks. In Proceedings of 20th International Conference on Computer Communications and Networks (ICCCN’11). 1--8.Google ScholarCross Ref
- Hai Jin, Guofu Xiang, Feng Zhao, Deqing Zou, Min Li, and Lei Shi. 2009. VMFence: A customized intrusion prevention system in distributed virtual computing environment. In Proceedings of the 3rd International Conference on Ubiquitous Information Management and Communication (ICUIMC’09). ACM, New York, NY, 391--399. Google ScholarDigital Library
- Hai Jin, Guofu Xiang, Deqing Zou, Song Wu, Feng Zhao, Min Li, and Weide Zheng. 2011. A VMM-based intrusion prevention system in cloud computing environment. Journal of Supercomputing 66, 1133--1151. Google ScholarDigital Library
- Pradeep Kannadiga and Mohammad Zulkernine. 2005. DIDMA: A distributed intrusion detection system using mobile agents. In Proceedings of the Sixth International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. 238--245. Google ScholarDigital Library
- Christopher Kruegel, Fredrik Valeur, and Giovanni Vigna. 2005. Intrusion Detection and Correlation: Challenges and Solutions. Advances in Information Security, Vol. 14. Springer. Google ScholarDigital Library
- Marcus Laureano, Carlos Maziero, and Edgard Jamhour. 2007. Protecting host-based intrusion detectors through virtual machines. Computer Networks 51, 5, 1275--1283. Google ScholarDigital Library
- Hoang Le and Viktor K. Prasanna. 2013. A memory-efficient and modular approach for large-scale string pattern matching. IEEE Transactions on Computers 62, 5, 844--857. Google ScholarDigital Library
- Po-Ching Lin and Jia-Hau Lee. 2013. Re-examining the performance bottleneck in a {NIDS} with detailed profiling. Journal of Network and Computer Applications 36, 2, 768--780. Google ScholarDigital Library
- Richard Lippmann, Joshua W. Haines, David J. Fried, Jonathan Korba, and Kumar Das. 2000. The 1999 DARPA off-line intrusion detection evaluation. Computer Networks 34, 4, 579--595. Google ScholarDigital Library
- Flavio Lombardi and Roberto Di Pietro. 2011. Secure virtualization for cloud computing. Journal of Network and Computer Applications 34, 4, 1113--1122. Google ScholarDigital Library
- John McHugh. 2000. Testing intrusion detection systems: A critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory. ACM Transactions on Information and System Security 3, 4, 262--294. Google ScholarDigital Library
- Peter Mell, Vincent Hu, Richard Lippmann, Josh Haines, and Marc Zissman. 2003. An Overview of Issues in Testing Intrusion Detection Systems. NIST Interagency/Internal Report. National Institute of Standards and Technology.Google Scholar
- Yuxin Meng. 2012. Measuring intelligent false alarm reduction using an ROC curve-based approach in network intrusion detection. In Proceedings of the IEEE International Conference on Computational Intelligence for Measurement Systems and Applications (CIMSA’12). 108--113.Google ScholarCross Ref
- Yuxin Meng and Wenjuan Li. 2012. Adaptive character frequency-based exclusive signature matching scheme in distributed intrusion detection environment. In Proceedings of the IEEE 11th International Conference on Trust, Security, and Privacy in Computing and Communications (TrustCom’12). 223--230. Google ScholarDigital Library
- Chirag Modi and Dhiren Patel. 2013. A novel hybrid-network intrusion detection system (H-NIDS) in cloud computing. In Proceedings of the IEEE Symposium on Computational Intelligence in Cyber Security (CICS’13). 23--30.Google ScholarCross Ref
- Khalid Nasr, AnasAbou-El Kalam, and Christian Fraboul. 2012. Performance analysis of wireless intrusion detection systems. In Internet and Distributed Computing Systems. Springer, 238--252. Google ScholarDigital Library
- Swapnil Patil, Anand Kashyap, Gopalan Sivathanu, and Erez Zadok. 2004. FS: An in-kernel integrity checker and intrusion detection file system. In Proceedings of the 18th USENIX Conference on System Administration (LISA’04). 67--78. Google ScholarDigital Library
- Nicholas Puketza, Mandy Chung, Ronald A. Olsson, and Biswanath Mukherjee. 1997. A software platform for testing intrusion detection systems. IEEE Software 14, 5, 43--51. Google ScholarDigital Library
- Nicholas J. Puketza, Kui Zhang, Mandy Chung, Biswanath Mukherjee, and Ronald A. Olsson. 1996. A methodology for testing intrusion detection systems. IEEE Transactions on Software Engineering 22, 10, 719--729. Google ScholarDigital Library
- Kannaiya N. Raja, Srinivasan Arulanandam, and Raja Rajeswari. 2012. Two-level packet inspection using sequential differentiate method. In Proceedings of the International Conference on Advances in Computing and Communications (ICACC’12). 42--45. Google ScholarDigital Library
- Marcus J. Ranum. 2001. Experiences Benchmarking Intrusion Detection Systems. White Paper. NFR Security Technical Publications.Google Scholar
- Jason Reeves, Ashwin Ramaswamy, Michael Locasto, Sergey Bratus, and Sean Smith. 2012. Intrusion detection for resource-constrained embedded control systems in the power grid. International Journal of Critical Infrastructure Protection 5, 2, 74--83.Google ScholarCross Ref
- Ryan Riley, Xuxian Jiang, and Dongyan Xu. 2008. Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection (RAID’08). Springer-Verlag, Berlin, 1--20. Google ScholarDigital Library
- Martin Roesch. 1999. Snort—lightweight intrusion detection for networks. In Proceedings of the 13th USENIX Conference on System Administration (LISA’99). 229--238. Google ScholarDigital Library
- Karen Scarfone and Peter Mell. 2007. Guide to Intrusion Detection and Prevention Systems (IDPS). Report 900-94. NIST Special Publication.Google Scholar
- Vidar Evenrud Seeberg and Slobodan Petrovic. 2007. A new classification scheme for anonymization of real data used in IDS benchmarking. In Proceedings of the 2nd International Conference on Availability, Reliability, and Security (ARES’07). 385--390. Google ScholarDigital Library
- Jaydip Sen, Arijit Ukil, Debasis Bera, and Arpan Pal. 2008. A distributed intrusion detection system for wireless ad hoc networks. In Proceedings of the 16th IEEE International Conference on Networks (ICON’08). 1--6.Google ScholarCross Ref
- Ali Shiravi, Hadi Shiravi, Mahbod Tavallaee, and Ali Ghorbani. 2012. Toward developing a systematic approach to generate benchmark datasets for intrusion detection. Computers and Security 31, 3, 357--374. Google ScholarDigital Library
- Sushant Sinha, Farnam Jahanian, and Jignesh M. Patel. 2006. WIND: Workload-aware INtrusion Detection. In Proceedings of the 9th International Conference on Recent Advances in Intrusion Detection (RAID’06). Springer-Verlag, Berlin, 290--310. Google ScholarDigital Library
- Robin Sommer and Vern Paxson. 2010. Outside the closed world: On using machine learning for network intrusion detection. In Proceedings of the 2010 IEEE Symposium on Security and Privacy (SP’10). 305--316. Google ScholarDigital Library
- Abhinav Srivastava, Kapil Singh, and Jonathon Giffin. 2008. Secure Observation of Kernel Behavior. Retrieved July 28, 2015, from http://hdl.handle.net/1853/25464.Google Scholar
- William Stallings. 2002. Cryptography and Network Security: Principles and Practice. Pearson Education. Google ScholarDigital Library
- Bo Sun, Xuemei Shan, Kui Wu, and Yang Xiao. 2013. Anomaly detection based secure in-network aggregation for wireless sensor networks. IEEE Systems Journal 7, 1, 13--25.Google ScholarCross Ref
- Chris Wright, Crispin Cowan, Stephen Smalley, James Morris, and Greg Kroah-Hartman. 2002. Linux security modules: General security support for the Linux kernel. In Proceedings of the 11th USENIX Security Symposium. 17--31. Google ScholarDigital Library
- Senhua Yu and Dipankar Dasgupta. 2011. An effective network-based intrusion detection using conserved self pattern recognition algorithm augmented with near-deterministic detector generation. In Proceedings of the 2011 IEEE Symposium on Computational Intelligence in Cyber Security (CICS’11). 17--24.Google ScholarCross Ref
- Yuxin Meng and Lam-For Kwok. 2013. Towards an information-theoretic approach for measuring intelligent false alarm reduction in intrusion detection. In Proceedings of the 12th IEEE International Conference on Trust, Security, and Privacy in Computing and Communications (TrustCom’13). 241--248. Google ScholarDigital Library
- Youhui Zhang, Hongyi Wang, Yu Gu, and Dongsheng Wang. 2008. IDRS: Combining file-level intrusion detection with block-level data recovery based on iSCSI. In Proceedings of the 3rd International Conference on Availability, Reliability, and Security (ARES’08). 630--635. Google ScholarDigital Library
Index Terms
- Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices
Recommendations
A hybrid intrusion detection system design for computer network security
Intrusions detection systems (IDSs) are systems that try to detect attacks as they occur or after the attacks took place. IDSs collect network traffic information from some point on the network or computer system and then use this information to secure ...
A Survey on Intrusion Detection and Prevention Systems
AbstractIn the digital world, malicious activities that violate the confidentiality, integrity, or availability of data and devices are known as intrusions. An intrusion detection system (IDS) analyses the activities of a single system or a network to ...
Comments