A Hybrid Flow-based Intrusion Detection System Incorporating Uncertainty

The advances of today’s cyberattacks threatening network infrastructures are both versatile and alarming. This requires thoroughly planned security solutions to spot malicious behavior in those networks. Systems serving this duty are intrusion detectors commonly relying on deep packet inspection, which come up with high resource consumption because network traffic is observed at a very fine granularity. With increasing link speeds of current and future networks, this situation is becoming a serious affair for operational staff. These circumstances are further fueled by the rise of end-to-end encryption preventing deeper insights to packet content. To absorb these drawbacks, we investigate alternative roads and propose a new hybrid flow-based intrusion detection system in this work. It rests upon flow data as primary entity to monitor network sites, which is enabled by the established flow export protocols NetFlow/IPFIX. As opposed to packet data, flows elevate network activities to a much coarser format posing several practical benefits. Yet, it is unclear to which degree flows can contribute to a broad attack coverage with a low false alarm rate realized through a single detection system. On this account, a feature analysis is conducted on newly compiled benchmark data to expose meaningful flow features coupled with other supplemental information that are incorporated into our intrusion detector. Moreover, the system adapts the essential idea of combining misuse and anomaly detection techniques based on machine learning principles towards a hybrid solution following a two-step inspection attempt. In the first step, the stream of incoming flows is examined against a repository of known patterns. If no pattern match can be identified at this point, flows are directed to the anomaly detector for a final examination. From there, missing knowledge in the pattern repository is complemented gradually by a new pattern building mechanism employing in-database analytics, i.e. an undertaking to lift database systems beyond traditional data management tasks. A key asset of this cascading design is transparency as black box classifications at the anomaly detector are immediately turned into human readable patterns serving follow-up actions for responsible personnel. Additionally, our system architecture aims at scalability and adaptivity to address network dynamics. Empirical assessments under very realistic circumstances reveal interesting insights. In particular, they confirm that the proposed solution can compensate increasing workloads by appending more hardware resources permitting to monitor medium to large production networks. It can also handle simple concept drift scenarios self-sufficiently but minor manual intervention is required for more rigorous drifts. Furthermore, results document a baseline protection against several attack types. This outcome is paired with few false alarms and a high chance for explainable predictions. These and further findings demonstrate that our approach is a step in the right direction to safeguard network systems without cumbersome packet analysis leaving ample room for further research.