SCH: INT: An Adaptive Robotic Hand Orthosis with Multimodal Sensing and Continuous Learning

We propose to develop a wearable robotic hand orthosis, able to act as a multimodal sensory platform and to physically assist functional hand movement. Working with stroke patients, we will deploy this orthosis for extended use in retraining the upper limb to improve the ability to perform activities of daily living in a simulated domestic environment. In these sessions, we will test our platform in two possible roles: an active orthosis as well as a rehabilitation device. To enable extended use of the orthosis for functional tasks, we rely on two novel approaches. The first is that of a robotic hand orthosis seen as a platform for both actuation and multimodal sensing. This will be the first device to collect multimodal data on the assisted impaired hand during functional tasks, combining information such as applied motor force, finger joint angles, fingertip pose, applied contact pressure, muscle activity, etc. Based on this data, we propose a multimodal learning mechanism for inferring user intent in order to operate the device. As a significant innovation, this approach lends itself to continuous learning: different sensing modalities can inform and train each other during use, in the absence of manually provided ground truth. Thus, we aim to remove a longstanding challenge for user-controlled assistive devices: the need for repeated calibration and supervision. The second novel approach is that of an assistive platform used to characterize and develop mitigation strategies for unwanted and abnormal muscle activations, an important and currently unaddressed challenge for achieving functional movement of the combined proximal and distal upper limb. These phenomena, which directly oppose training movements, have generally been ignored in the design of robotic devices, or addressed through blunt methods such as a general increase of the applied force in order to overcome spasticity, or exclusion of the proximal upper limb from the task in order to avoid synergies. Our sensing capabilities will allow us to study how applied assistive forces affects these phenomena, and develop targeted strategies to mitigate them.