Reinforcement Learning in Autonomous Robots: An Empirical Investigation of the Role of Emotions

Abstract

This thesis presents a study of the provision of emotions for artificial agents with the ultimate aim of enhancing their autonomy, i.e. making them more exible, robust and self-sufficient. In recent years, the importance of emotions and their assistance to cognition has been increasingly acknowledged. Emotions are no longer considered undesirable or simply useless. Their role in various aspects of human and animal cog- nition like perception, attention, memory, decision-making and social interaction has been recognised as essential. The importance of emotions is much more evident in social interaction and therefore much of the emotions research done in artificial systems focuses on the expression and recognition of emotions. However, recent neurophysiological research suggests that emotions also play a crucial part in cognition itself. This thesis investigates ways in which artificial emotions can improve autonomous behaviour in the domain of a simple, but complete, solitary learning agent. For this purpose, a non-symbolic emotion model was designed and implemented. It takes the form of a recurrent artificial neural network where emotions influence the perception of the state of the world, on which they ultimately depend. This is done through a hormone system that acts as a persistence mechanism. This model is somewhat more sophisticated than those usually found in equivalent non-symbolic systems, yet the emotions themselves were restricted to a few simplified emotions that do not try to mimic the complexity of the human counterparts, but are afforded by the agent's interaction with the environment. Several hypotheses were investigated of how the emotion model above could be integrated in a reinforcement learning framework which, by itself, provides the base for the adaptiveness necessary for autonomy. Experiments were carried out in a realistic robot simulator that compared the performance of emotional with non-emotional agents in a survival task that consists of maintaining adequate energy levels in an environment with obstacles and energy sources. One of the most common roles attributed to emotions is as source of reinforcement and was therefore examined first. In experiments with a controller that selects between primitive actions, the reinforcement provided by emotions was found inappropriate because of the time scale discrepancies introduced by the emotion model. The reinforcement provided by emotions proved to be much more successful when used by a controller that selects between behaviours rather than actions, achieving equivalent performance to that of a standard reinforcement function. One of the crucial issues for efficient and productive learning, highlighted by the latter experiments, is to determine exactly when the controller should re-evaluate its decision concerning which behaviour to activate. The emotions proved to be particularly helpful in this role, enabling better performance with substantially less computational effort than the best suited interruption mechanism using regular time intervals. The modulation of learning parameters such as learning rate and the exploration vs. exploitation ratio was also explored. Experiments suggested that emotions might also be useful for this purpose. This research led to the conclusion that artificial emotions are a useful construct to have in the domain of behaviour-based autonomous agents, because they provide a unifying way to tackle different issues of control, analogous to natural systems' emotions.