http://hdl.handle.net/1842/3390 2013-05-23T16:54:52Z 2013-05-23T16:54:52Z Milner, Robin http://hdl.handle.net/1842/6050 2012-07-06T10:33:31Z 1991-01-01T00:00:00Z

Title: The Polyadic pi-Calculus: A Tutorial Authors: Milner, Robin Abstract: The pi-calculus is a model of concurrent computation based upon the notion of naming. It is first presented in its simplest and original form, with the help of several illustrative applications. Then it is generalized from monadic to polyadic form. Semantics is done in terms of both a reduction system and a version of labelled transitions called commitment; the known algebraic axiomatization of strong bisimilarity is given in the new setting, and so also is a characterization in modal logic. Some theorems about the replication operator are proved. Justification for the polyadic form is provided by the concept of sort, sorting and sort discipline which it supports. Several illustrations of different sortings are given. One example is the presentation of data structures as processes which respect a particular sorting; another is the sorting for a known translation of the lambda-calculus in to pi-calculus. For this translation, the equational validity of beta-conversion is proved with the help of replication theorems. The paper ends with an extension of the pi-calculus to w-order processes, and a brief account of the demonstration by Davide Sangiorgi that higher-order processes may be faithfully encoded at first-order. This extends and strengthens the original result of this kind given by Bent Thomsen for second-order processes. Description: This report was published in F. L. Hamer, W. Brauer and H. Schwichtenberg, editors, Logic and Algebra of Specification. Springer-Verlag, 1993.

1991-01-01T00:00:00Z Sugiyama, Masashi Hachiya, Hirotaka Towell, Christopher Vijayakumar, Sethu http://hdl.handle.net/1842/3714 2010-08-31T15:11:26Z 2007-04-01T00:00:00Z

Title: Value Function Approximation on Non-Linear Manifolds for Robot Motor Control Authors: Sugiyama, Masashi; Hachiya, Hirotaka; Towell, Christopher; Vijayakumar, Sethu Abstract: The least squares approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in realworld reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in a simulated robot arm control and Khepera robot navigation.

2007-04-01T00:00:00Z Vijayakumar, Sethu Peters, Jan Schaal, Stefan http://hdl.handle.net/1842/3710 2010-08-31T15:09:07Z 2004-07-01T00:00:00Z

Title: Reinforcement Learning for Humanoid Robots - Policy Gradients and Beyond Authors: Vijayakumar, Sethu; Peters, Jan; Schaal, Stefan Abstract: Reinforcement learning offers one of the most general frameworks to take traditional robotics towards true autonomy and versatility. However, applying reinforcement learning to high dimensional movement systems like humanoid robots remains an unsolved problem. In this paper, we discuss different approaches of reinforcement learning in terms of their applicability in humanoid robotics. Methods can be coarsely classified in to three different categories, i.e., greedy methods, ’vanilla’ policy gradient methods, and natural gradient methods. We discuss that greedy methods are not likely to scale into the domain humanoid robotics as they are problematic when used with function approximation. Vanilla’ policy gradient methods on the other hand have been successfully applied on real-world robots including at least one humanoid robot [3]. We demonstrate that these methods can be significantly improved using the natural policy gradient instead of the regular policy gradient. A derivation of the natural policy gradient is provided, proving that the average policy gradient of Kakade[10] is indeed the true natural gradient. A general algorithm for estimating the natural gradient, the Natural Actor-Critic algorithm, is introduced. This algorithm converges to the nearest local minimum of the cost function with respect to the Fisher information metric under suitable conditions. The algorithm outperforms non-natural policy gradients by far in a cart-pole balancing evaluation, and for learning non-linear dynamic motor primitives for humanoid robot control. It offers a promising route for the development of reinforcement learning for truly high-dimensionally continuous state-action systems.

2004-07-01T00:00:00Z Howard, Matthew Vijayakumar, Sethu http://hdl.handle.net/1842/3709 2010-08-31T15:03:06Z 2007-01-01T00:00:00Z

Title: Reconstructing Null-space Policies Subject to Dynamic Task Constraints in Redundant Manipulators Authors: Howard, Matthew; Vijayakumar, Sethu Abstract: We consider the problem of direct policy learning in situations where the policies are only observable through their projections into the null-space of a set of dynamic, non-linear task constraints. We tackle the issue of deriving consistent data for the learning of such policies and make two contributions towards its solution. Firstly, we derive the conditions required to exactly reconstruct null-space policies and suggest a learning strategy based on this derivation. Secondly, we consider the case that the null-space policy is conservative and show that such a policy can be learnt more easily and robustly by learning the underlying potential function and using this as our representation of the policy.

2007-01-01T00:00:00Z