CAREER: Efficient Fuzzing with Neural Program Smoothing

Fuzzing is an automated software testing technique that involves feeding a stream of invalid, unexpected, or rare data as inputs to a computer program for discovering bugs leading to crashes, assertion failures, or memory corruption. Fuzzing is the de facto standard technique for finding software vulnerabilities. However, despite their tremendous promise, popular fuzzers, especially for large programs, often tend to get stuck trying redundant test inputs and struggle to find security vulnerabilities hidden deep into the program logic. To find interesting test inputs, most popular fuzzers use evolutionary algorithms, which start from a set of inputs, apply random mutations and crossovers on these inputs to generate new inputs, and apply a fitness function (e.g., achieved code coverage) to select the most promising new inputs for the next set of mutations. The key insight of this research is that an approach using continuous optimization techniques can do better, by more efficiently using the structure of the underlying functions (e.g., gradients or higher-order derivatives). The key benefit of this approach is that continuous gradient-guided optimization can efficiently generate new promising inputs with a few targeted mutations based on the gradient value and the step size rather than random unguided mutations used in evolutionary techniques. Gradient-guided optimizations have already been shown to significantly outperform evolutionary techniques in popular tasks like training of neural networks. Better fuzzers will significantly improve the security, reliability, and robustness of critical infrastructure software used by billions of users across the world. Data and tools will be made available through open source. Curriculum and training will be developed to disseminate the results.

This project will develop a set of novel techniques and tools that will enable fuzzers to fully exploit the power of continuous optimization techniques like gradient descent. One of the key challenges behind applying continuous optimization for fuzzing is that real-world program behaviors often contain many discontinuities and thus are not directly amenable to smooth optimization. Therefore, the target programs must be smoothed before performing continuous optimization. This project will develop a new technique involving surrogate neural networks and graybox instrumentation that will automatically learn smooth approximations of discontinuous program behaviors. This project will further use global optimization schemes like branch-and-bound and cutting plane algorithms to avoid getting stuck at local optima due to the non-convexity of the target program.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.