CAUSAL REINFORCEMENT LEARNING: DISCOVERY AND DECISION MAKING.

Artificial Intelligence (AI) plays an increasingly prominent role in society since decisions that were once made by humans are now delegated to automated systems. However, these systems are typically not statistically efficient, robust, explainable, nor generalizable. Principled methods for robust decision-making rely on causal understanding of the mechanisms of the underlying system, typically encoded as a causal diagram (Pearl, 2000). The emerging formalism in AI known as Causal Reinforcement Learning (CRL) generalizes RL for performing decision-making by leveraging the causal invariances of the environment. However, CRL is still in its infancy, and the current methods available in the field all require an explicit characterization of the true causal diagram in order to improve over its non-causal RL counterparts. There is a growing literature about learning causal structures from data, which is also known as causal discovery. However, the disciplines of RL and Causal discovery evolved independently, with virtually no interaction between them. The goal of this project is to marry the capabilities of these fields towards what we call data-driven causal reinforcement learning. In particular, we will advances structure learning results and combine them with i) the general language of causal inference (CI) and ii) the interventional capabilities of RL. Structure learning will allow us to learn causal structure without assuming it apriori, CI allows us to leverage existing theory to piece together disparate fragments of causal knowledge, and RL provides sample-efficient interventional capabilities to leverage this knowledge. Data-driven CRL will ultimately provide a principled framework for allowing agents, natural or artificial, to systematically learn imperfect causal knowledge to improve their learning and subsequently their decision-making capabilities, which will then in turn also improve their causal knowledge. For additional information about this larger initiative and the current state of the field, please refer to https://crl.causalai.net.