Collaborative Research: Real-Time Data-Driven Anomaly Detection for Complex Networks

Anomaly detection is an important problem dealing with the detection of abnormal data patterns. Importance of anomaly detection lies in the fact that an anomaly in the observed data may be a sign of an unwanted and often actionable event such as failure, malicious activity, etc. in the underlying system. In many real-time systems, timely and accurate detection of abnormal data patterns is crucial, and will allow proper countermeasures to be taken in a timely manner, to counteract any possible harm. Although anomaly detection has long been studied, today's complex networks exhibit new challenges, such as: low latency requirements, data size, system dynamics, unknown distributions, distributed nature, and privacy. The objective of this proposal is to investigate effective and scalable approaches for real-time data-driven anomaly detection in complex systems with these challenges. The main themes of this proposal address multiple important problems in the early detection of anomalies and attacks in a general complex network setting. Considering the importance of cybersecurity in today's world, methodologies to understand and forewarn changes in the organizational dynamics of such complicated networks is of immense significance. This proposal directly addresses these issues by bringing a fresh and novel set of engineering tools and ideas.

Following a systematic approach, this project first considers (1) how to timely detect anomalies in centralized high-dimensional systems with dynamicity and hidden anomaly challenges; (ii) how to deal with resource constraints in monitoring distributed systems; and (iii) how to enable privacy-preserving solutions for real-time anomaly detection in distributed systems. These challenges and the solution methods presented in this project are generally applicable to a variety of complex systems. To be specific, this project focuses on two challenging IoT networks: surveillance camera network and smart home network. The proposed approaches exploit an array of advanced techniques including sequential change detection, deep reinforcement learning, event-triggered processing, and differential privacy, and will bring significant innovations to the theory and applications of anomaly detection. In particular, the practical use of proposed algorithms will be demonstrated and their performance will be evaluated with respect to the state of the art using hardware implementations of two IoT networks - a surveillance camera network and a smart home network.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.