Causal Reinforcement Learning Theory Algorithms and Applications

Artificial Intelligence (AI) plays an increasingly prominent role in society since decisions thatwere once made by humans are now de,legated to automated systems. These systems are expectedto be efficient, robust, explainable, and generalizable. It is formally unde,rstood that the mereaccumulation of data, unfortunately, does not immediately translate into new insights about theunderlying data-g,enerating mechanisms, better predictions about the effects of new interventions,and more efficient decision-making. Principled metho,ds for decision-making rely on causal understandingof the mechanisms of the underlying system (Pearl, 2000). The current formalism i,nAI responsible for performing decision-making, known as Reinforcement Learning (RL), does notemploy any explicit causal model of th,ed as do(X =x), wheneveran agent wants to optimize an outcome measure Y, for every possible action X = x. From a causalstandpoint, t,his is a somewhat naive approach.In most practical settings, gratuitous experimentation is undesirable ? interventions can beexpensi,ve, unethical, technically challenging, and lead to unintended, possibly catastrophic sideeffects. The need to avoid experimentation, and to leverage causal relations to support effectivedecision-making is one of the central requirements found throughout high-stake,s domains, includingnational security and defense. In practice, massive amounts of non-interventional data areavailable for relative,ly lower cost, which allows one to estimate the observational distribution. Unfortunately,this distribution is not immediately usefu,l for making causal and counterfactual claimssince it is often plagued with confounding bias, which arises due to the non-random nat,ure of thedecision-making procedure. In fact, causal capabilities are achieved by (i) formally integrating andencoding assumptions d,erived from a combination of domain knowledge and data, (ii) distinguishingcausal from associational relationships, and (iii) genera,lizing causal findings across differentdomains and populations.Research efforts on Causal Inference (CI) culminated in a general fra,mework for understanding,representing, and algorithmizing causal findings and generalizations from observational andinterventional d,ata (Bareinboim and Pearl, PNAS?16). However, the disciplines of CI and RL haveevolved independently, with virtually no interaction,between them. The goal of this project is tomarry the capabilities of these fields towards what we call causal reinforcement learnin,g (CRL,for short). In particular, we will combine the general language of CI with the interventional capabilitiesof RL such that the, former allows us to piece together disparate fragments of causalknowledge, while the latter provides sample-efficient interventiona,al, to systematically use imperfect causal knowledge to improve their learning and subsequentlytheir decision-making capabilities. F,or additional information about this larger initiative,please refer to https://crl.causalai.net.The Department of Defense (DoD) face,s constant decision-making pressures given its operationsin highly complex and structurally uncertain environments. We believe that,CRL ? bothfoundational and algorithmic developments ? can have a significant impact on the DoD AI operationssince the current genera,tion of decision-making formalisms are virtually causally-blind,which means that decisions are possibly suboptimal and not causally-,explainable. The substantiveknowledge about specific domains accumulated by the DoD could then be leveraged, and translatedto suppor,t more transparent, robust, and efficient decis