Collaborative Research: SHF: Medium: Causal Performance Debugging for Highly-Configurable Systems

Software performance is critical for most software systems to achieve scale and limit operating costs and energy consumption. As modern software systems, such as big data and machine-learning systems, are increasingly built by composing many reusable infrastructure components and deployed on distributed and heterogeneous hardware, developers have powerful tools and abstractions at their fingertips, and as a result face immense configuration complexity. Software and hardware need to be selected and configured carefully to achieve high performance for a given system and task. Unfortunately, in practice, performance faults and misconfigurations are common, where a system performs much worse than expected, not achieving its mission or simply wasting cost and energy. In large configuration spaces, end-users and developers face severe challenges in understanding and fixing performance faults by changing software configuration, changing hardware deployment, or modifying the software's code itself. Current approaches that model system performance by analyzing correlations among performance measurements and options are slow and may produce misleading results, obfuscating the actual causes of performance faults. Even if they can fix the problem, most of them cannot explain why (1) the obtained configurations are the real cause of the problem, and (2) a user/developer should consider the proposed recommendations. In both cases, the lack of explainability is a big issue. The project is intended to initiate a paradigm shift in today's testing and debugging methodology for complex, highly configurable systems, thereby positively impacting a broad range of industrial sectors relying on complex, highly configurable systems. Specifically, the project contributes to substantial energy savings and reduced carbon emissions, especially for the many big-data and machine-learning systems that operate at a massive scale. Finally, the research is providing valuable training for involved students from diverse backgrounds in research and generating high-quality researchers and practitioners for society.

This project develops and evaluates foundations and tools for a causal approach to performance modeling and performance debugging. This project introduces the new concept of causal performance models that are learned using causal structure learning by intervening over configuration options and observing system performance regarding (multiple) performance objectives, rather than just analyzing correlations. Causal models enable causal inference and counterfactual reasoning for numerous tasks, including debugging performance faults and misconfigurations. Based on a solid technical foundation of causal modeling and extensive experience with performance modeling for configurable systems, this project develops innovations in three thrusts: (1) It designs and refines a causal modeling approach for software performance of systems composed of multiple configurable components, using innovations in sampling strategies, code analysis, compositional reasoning, and transfer learning to build accurate causal models efficiently. (2) It develops and evaluates user-facing tool support, based on causal models, to help users select well-performing configurations for their specific tasks and hardware and resolve misconfiguration faults with configuration changes, highlighting the (causal) performance impact of configuration decisions and providing a Pareto analysis of involved tradeoffs. (3) It develops and evaluates developer-facing tool support to foster code-level debugging and documentation. Finally, all contributions are being evaluated end-to-end with developers on real performance faults, showing how both users and developers benefit from causal models and related tools.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.