Resumo
O objetivo deste trabalho é analisar os efeitos da definição de parâmetros do Aprendizado por Reforço no desempenho de um manipulador robótico. Para isso, são realizados experimentos com um manipulador simulado do tipo SCARA. Além disso, é adotada a modelagem matemática via Metodologia de Superfície de Resposta. Os resultados apontam uma tendência no aumento de desempenho do manipulador quando a taxa de aprendizado e o fator de desconto se aproximam simultaneamente de zero.
Palavras-chave: Aprendizado por Reforço. Manipulador Robótico. Metodologia de Superfície de Resposta.
Analysis of the defition of reinforcement learning parameters in the robotic manipulator performance
Abstract
The objective of this work is to analyze the effects of the definition of Reinforcement Learning parameters on the performance of a robotic manipulator. For this experiments are performed with a simulated manipulator of the type SCARA. Furthermore it adopted the mathematical modeling via Response Surface Methodology. The results point to a tendency in the manipulator performance increase when the learning rate and discount factor are simultaneously approaching zero.
Keywords: Reinforcement Learning. Robotic Manipulator. Response Surface Methodology.
Referências
BECK, C.; SRIKANT, R. Error bounds for constant step-size Q-learning. Systems and Control Letters, v. 61, n. 12, p. 1203–1208, 2012.
DENAVIT, J.; HARTENBERG, R. S. A kinematic notation for lower pair mechanisms based on matrices. Journal of Applied Mechanics, v. 77, n. 2, p. 215–221, 1955.
EVEN-DAR, E.; MANSOUR, Y. Learning Rates for Q-learning. Journal of Machine Learning Research, v. 5, p. 1–25, 2003.
HERNANDEZ, J. A. M. Software tools for reinforcement learning, artificial neural networks and robotics (matlab and python). Disponível em: <https://jamh-web.appspot.com/download.htm>. Acesso em: 03 ago. 2017
HERNANDEZ, J. A. M.; LOPE, J. A distributed reinforcement learning control architecture for multi-link robots: Experimental validation. ICINCO 2007 - International Conference on Informatics in Control, Automation and Robotics, p. 192–197, 2007.
KIM, B. et al. Impedance learning for robotic contact tasks using natural actor-critic algorithm.
IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), v. 40, n. 2, p. 433–443, April 2010. ISSN 1083-4419.
LENTH, R. V. Response-Surface Methods in R, using rsm. Journal of Statistical Software, v. 32, n. 7, p. 1–17, 2009.
LIN, C.-K. H∞ reinforcement learning control of robot manipulators using fuzzy wavelet networks. Fuzzy Sets and Systems, v. 160, n. 12, p. 1765 – 1786, 2009. ISSN 0165-0114.
LOPES, R. H. C. Kolmogorov-smirnov test. In: . International encyclopedia of statistical science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011. p. 718–720. ISBN 978-3-642-04898-2.
MILJKOVIc, Z. et al. Neural network reinforcement learning for visual control of robot manipulators. Expert Systems with Applications, v. 40, n. 5, p. 1721 – 1736, 2013. ISSN 0957-4174.
MURAKOSHI, K.; MIZUNO, J. A parameter control method in reinforcement learning to rapidly follow unexpected environmental changes. Biosystems, v. 77, n. 1-3, p. 109 – 117, 2004. ISSN 0303-2647.
MYERS, R. H.; MONTGOMERY, D. C.; Anderson-Cook, C. M. Response surface methodology: process and product optimization using designed experiments. [S.l.]: John Wiley & Sons, 3 ed, 2009.
OTTONI, A. L. C.; NEPOMUCENO, E. G.; OLIVEIRA, M. S. Análise de sensibilidade dos parâmetros do aprendizado por reforço na solução do problema do caixeiro viajante: modelagem via superfície de resposta. In: CONGRESSO BRASILEIRO DE AUTOMÁTICA, 21., 2016, Vitória.ES. Anais... Vitória, ES: SBA. [S.l.: s.n.], 2016. p. 513–518.
OTTONI, A. L. C. et al. Análise da influência da taxa de aprendizado e do fator de desconto sobre o desempenho dos algoritmos Q-learning e SARSA: aplicação do aprendizado por reforço na navegação autônoma. Revista Brasileira de Computação Aplicada, v. 8, n. 2, p. 44–59, 2016.
PARK, J.-J.; KIM, J.-H.; SONG, J.-B. Path planning for a robot manipulator based on probabilistic roadmap and reinforcement learning. International Journal of Control Automation and Systems, Korean Institute of Electrical Engineers, v. 5, n. 6, p. 674–680, 2007.
PELLEGRINI, J.; WAINER, J. Processos de Decisão de Markov: um tutorial. RITA - Revista de Informática Teórica e Aplicada, v. 14, n. 2, p. 133–179, 2007.
R Core Team. R: A language and environment for statistical computing. Vienna, Austria, 2013.
SCHWEIGHOFER, N.; DOYA, K. Meta-learning in reinforcement learning. Neural Networks, v. 16, n. 1, p. 5–9, 2003.
SUTTON, R.; BARTO, A. Reinforcement learning: an introduction. 1st. ed. [S.l.]: Cambridge, MA: MIT Press, 1998.
TANG, L.; LIU, Y.-J.; TONG, S. Adaptive neural control using reinforcement learning for a class of robot manipulator. Neural Computing and Applications, Springer, v. 25, n. 1, p. 135–141, 2014.
THAM, C. K.; PRAGER, R. W. Reinforcement learning methods for multi-linked manipulator obstacle avoidance and control. In: ASIA-PACIFIC WORKSHOP ON ADVANCES IN MOTION CONTROL, 1993. Proceedings... [S.l.: s.n.], 1993. p. 140–145.
VISIOLI, A.; LEGNANI, G. On the trajectory tracking control of industrial scara robot manipulators. IEEE Transactions on Industrial Electronics, v. 49, n. 1, p. 224–232, Feb 2002. ISSN 0278-0046.
WATKINS, C. J.; DAYAN, P. Technical note Q-learning. Machine Learning, v. 8, n. 3, p. 279–292, 1992.