CAREER: Systematic Software Testing for Deep Learning Applications

A paradigm shift is underway in software development, where decision making is increasingly shifting from hand-coded program logic to a reliance on Deep Learning (DL) --- popular applications of Speech Processing, Image Recognition, Robotics, etc. are using DL to implement their core components. Deep Neural Networks (DNNs), a widely used form of DL, is a key behind much of this progress. With such spectacular growth in traditional applications, DNNs and other DL technologies are also increasingly being used in safety-critical systems such as autonomous cars, medical diagnosis, malware detection, and aircraft collision avoidance systems. Such a wide adoption of DL techniques carries with it concerns about the reliability of these systems, as several high-profile instances of DL-based behavior have already been reported. Thus, it has become crucial to rigorously test these applications with realistic corner cases to ensure high reliability. However, due to the fundamental architectural differences between DL implementations such as DNNs and traditional software, existing software testing techniques do not apply to them in any obvious way. In fact, companies like Google, Tesla, etc. are increasingly confronting software testing challenges to ensure reliable and safe DL applications. Therefore, systematically testing DL-based software systems will be a significant step towards increasing safety and reliability of sensitive and safety-critical DL systems.

This project will design, implement, and evaluate a novel software testing framework to assess the reliability of the Deep Learning applications and detect buggy behaviors during the application development and maintenance phase. In particular, the proposed framework will develop novel white-box testing strategies, realistic test-case generation techniques, and regression testing techniques to assess DL applications. A unique characteristic of the DL-based programming paradigm is that the end applications highly depend on the training data. Therefore, the research will build novel white-box testing strategies to evaluate both the model and the training data together as a whole system. In addition, this research will design and deploy techniques to generate new test cases that capture the real-world corner-case behavior where the DL applications may fail. The project will also investigate how any changes in data or model architecture can impact a pre-trained model in order to guide regression test case selection and prioritization process.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.