Cyber Defence Library

Methods for Detecting Important Events and Knowledge from Data Security Logs

Methods for Detecting Important Events and Knowledge from Data Security Logs

In modern computer networks and IT systems, event logging is commonly used for collecting system health information, in order to ease the system management process. For example, many sites are collecting events and network flow records from their applications, servers, and network devices over protocols like syslog, SNMP and Netflow, and analyze these data at central monitoring server(s). Among collected data, many events and records provide information about security incidents. Unfortunately, during the last decade security logs have grown rapidly in size, making the manual analysis extremely labor intensive task. This task is further complicated by the large number of irrelevant records and false positive alerts in security logs. For this reason, the development of methods for detecting important events and knowledge from security logs has become a key research issue during the recent years. In this paper some methods are proposed for tackling this issue in the context of IDS and Netflow logs from an organizational network. The first contribution of this paper is the study of important properties of IDS and Netflow logs. The author has conducted an analysis on a number of production system logs obtained from a large financial institution, and some of their findings are supported by results from other researchers. The second contribution of the paper is the proposal of several data mining based and heuristic methods for event and knowledge detection from security logs. These data mining methods are based on frequent itemset mining for identifying regularities in IDS alert sets and network traffic. These regularities are then used for finding unexpected IDS alert patterns and prominent network traffic flows. In this paper, the author also discusses the implementations of the proposed methods in a production environment, and provide performance estimates for implementations. The paper is concluded with a short discussion on some promising directions for further research.

Published in: Proceedings of the 10th European Conference on Information Warfare and Security at the Tallinn University of Technology Tallinn, Estonia 7-8 July 2011.

Vaarandi, R. (2011). Methods for Detecting Important Events and Knowledge from Data Security Logs. Proceedings of the 10th European Conference on Information Warfare and Security at the Tallinn University of Technology Tallinn, Estonia 7-8 July 2011, pp. 261-267.