Action Learning in an Integrated Model of Basal Ganglia and its Application in Control Systems

Abstract

This thesis is motivated by the need to comprehend the complex functionalities of the nervous system, specifically the basal ganglia (BG), due to its significant role in neurological disorders and behavioral processes. Traditional experimental approaches have fallen short in explaining the contributions of brain structures to intricate behaviors, thus necessitating alternative approaches like computational modeling. This thesis aims to bridge this gap by employing biologically-inspired computational models, with a focus on spiking neural networks (SNNs) due to their ability to emulate the temporal dynamics of biological neurons, to simulate the basal ganglia. The simulation aims to understand the basal ganglia's role in action selection and the influence of neuromodulators, particularly dopamine and acetylcholine, on learning and decision-making. The ultimate objective is to link and integrate insights from neuroscience to embodied agents, by applying the knowledge of motor learning processes within the basal ganglia to the development of action selection in control systems. ¶ ¶ This thesis presents several contributions to the field of computational neuroscience, focusing on models of the basal ganglia. First, it introduces a computational model to elucidate the paradoxical sensorial improvement observed in patients with Huntington's disease, suggesting that both dopamine levels and the early stage of affliction may independently play significant roles. Secondly, it undertakes a thorough examination of methods for tuning spiking neural models of striatum plasticity, highlighting the effectiveness of automatic optimization algorithms in calibrating the models. Third, it constructs a biologically-inspired network model of the striatum integrating features such as spike-timing-dependent plasticity, homeostatic mechanisms, and lateral inhibitory connections, capable of recognizing complex patterns and choosing rewarded actions. Additionally, it develops a functional striosome model for reward prediction error (RPE) in the basal ganglia. Finally, it refines an existing striatal reinforcement learning model by incorporating acetylcholine as a local population feedback, which demonstrates proficiency in pattern recognition and action selection, while offering insights into the brain's learning mechanisms and the role of neuromodulators. ¶ ¶ The research presented in this thesis has explored the neural mechanisms involved in action selection, learning, and decision-making, with a particular focus on the striatum and the basal ganglia. Through computational models and novel methodologies, this work has contributed to an enhanced understanding of how neuronal populations and neuromodulators interact within the basal ganglia, and potentially with other brain regions. While the insights gained are promising, the study acknowledges certain limitations, notably the sensitivity of the model's internal dynamics to the form of input representation. Future research is encouraged to further validate these findings and explore additional biological factors, such as the basal ganglia-cortex loop and the role of interneurons. Moreover, the use of high-performance computing platforms and the development of novel optimization techniques are suggested as avenues for refining the computational models. This could have practical applications, including advancing knowledge on neurological disorders like Huntington's disease and fostering the development of bio-inspired reinforcement agents.