How to Cite
Abstract
O objetivo deste trabalho é analisar os efeitos da definição de parâmetros do Aprendizado por Reforço no desempenho de um manipulador robótico. Para isso, são realizados experimentos com um manipulador simulado do tipo SCARA. Além disso, é adotada a modelagem matemática via Metodologia de Superfície de Resposta. Os resultados apontam uma tendência no aumento de desempenho do manipulador quando a taxa de aprendizado e o fator de desconto se aproximam simultaneamente de zero.
Palavras-chave: Aprendizado por Reforço. Manipulador Robótico. Metodologia de Superfície de Resposta.
Analysis of the defition of reinforcement learning parameters in the robotic manipulator performance
Abstract
The objective of this work is to analyze the effects of the definition of Reinforcement Learning parameters on the performance of a robotic manipulator. For this experiments are performed with a simulated manipulator of the type SCARA. Furthermore it adopted the mathematical modeling via Response Surface Methodology. The results point to a tendency in the manipulator performance increase when the learning rate and discount factor are simultaneously approaching zero.
Keywords: Reinforcement Learning. Robotic Manipulator. Response Surface Methodology.
References
BECK, C.; SRIKANT, R. Error bounds for constant step-size Q-learning. Systems and Control Letters, v. 61, n. 12, p. 1203–1208, 2012.
DENAVIT, J.; HARTENBERG, R. S. A kinematic notation for lower pair mechanisms based on matrices. Journal of Applied Mechanics, v. 77, n. 2, p. 215–221, 1955.
EVEN-DAR, E.; MANSOUR, Y. Learning Rates for Q-learning. Journal of Machine Learning Research, v. 5, p. 1–25, 2003.
HERNANDEZ, J. A. M. Software tools for reinforcement learning, artificial neural networks and robotics (matlab and python). Disponível em: <https://jamh-web.appspot.com/download.htm>. Acesso em: 03 ago. 2017
HERNANDEZ, J. A. M.; LOPE, J. A distributed reinforcement learning control architecture for multi-link robots: Experimental validation. ICINCO 2007 - International Conference on Informatics in Control, Automation and Robotics, p. 192–197, 2007.
KIM, B. et al. Impedance learning for robotic contact tasks using natural actor-critic algorithm.
IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), v. 40, n. 2, p. 433–443, April 2010. ISSN 1083-4419.
LENTH, R. V. Response-Surface Methods in R, using rsm. Journal of Statistical Software, v. 32, n. 7, p. 1–17, 2009.
LIN, C.-K. H∞ reinforcement learning control of robot manipulators using fuzzy wavelet networks. Fuzzy Sets and Systems, v. 160, n. 12, p. 1765 – 1786, 2009. ISSN 0165-0114.
LOPES, R. H. C. Kolmogorov-smirnov test. In: . International encyclopedia of statistical science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011. p. 718–720. ISBN 978-3-642-04898-2.
MILJKOVIc, Z. et al. Neural network reinforcement learning for visual control of robot manipulators. Expert Systems with Applications, v. 40, n. 5, p. 1721 – 1736, 2013. ISSN 0957-4174.
MURAKOSHI, K.; MIZUNO, J. A parameter control method in reinforcement learning to rapidly follow unexpected environmental changes. Biosystems, v. 77, n. 1-3, p. 109 – 117, 2004. ISSN 0303-2647.
MYERS, R. H.; MONTGOMERY, D. C.; Anderson-Cook, C. M. Response surface methodology: process and product optimization using designed experiments. [S.l.]: John Wiley & Sons, 3 ed, 2009.
OTTONI, A. L. C.; NEPOMUCENO, E. G.; OLIVEIRA, M. S. Análise de sensibilidade dos parâmetros do aprendizado por reforço na solução do problema do caixeiro viajante: modelagem via superfície de resposta. In: CONGRESSO BRASILEIRO DE AUTOMÁTICA, 21., 2016, Vitória.ES. Anais... Vitória, ES: SBA. [S.l.: s.n.], 2016. p. 513–518.
OTTONI, A. L. C. et al. Análise da influência da taxa de aprendizado e do fator de desconto sobre o desempenho dos algoritmos Q-learning e SARSA: aplicação do aprendizado por reforço na navegação autônoma. Revista Brasileira de Computação Aplicada, v. 8, n. 2, p. 44–59, 2016.
PARK, J.-J.; KIM, J.-H.; SONG, J.-B. Path planning for a robot manipulator based on probabilistic roadmap and reinforcement learning. International Journal of Control Automation and Systems, Korean Institute of Electrical Engineers, v. 5, n. 6, p. 674–680, 2007.
PELLEGRINI, J.; WAINER, J. Processos de Decisão de Markov: um tutorial. RITA - Revista de Informática Teórica e Aplicada, v. 14, n. 2, p. 133–179, 2007.
R Core Team. R: A language and environment for statistical computing. Vienna, Austria, 2013.
SCHWEIGHOFER, N.; DOYA, K. Meta-learning in reinforcement learning. Neural Networks, v. 16, n. 1, p. 5–9, 2003.
SUTTON, R.; BARTO, A. Reinforcement learning: an introduction. 1st. ed. [S.l.]: Cambridge, MA: MIT Press, 1998.
TANG, L.; LIU, Y.-J.; TONG, S. Adaptive neural control using reinforcement learning for a class of robot manipulator. Neural Computing and Applications, Springer, v. 25, n. 1, p. 135–141, 2014.
THAM, C. K.; PRAGER, R. W. Reinforcement learning methods for multi-linked manipulator obstacle avoidance and control. In: ASIA-PACIFIC WORKSHOP ON ADVANCES IN MOTION CONTROL, 1993. Proceedings... [S.l.: s.n.], 1993. p. 140–145.
VISIOLI, A.; LEGNANI, G. On the trajectory tracking control of industrial scara robot manipulators. IEEE Transactions on Industrial Electronics, v. 49, n. 1, p. 224–232, Feb 2002. ISSN 0278-0046.
WATKINS, C. J.; DAYAN, P. Technical note Q-learning. Machine Learning, v. 8, n. 3, p. 279–292, 1992.
Authors who post on ForScience agree to the following terms:
- The said work is of own authorship and has never been published or sent to another magazine, this magazine having the right of exclusivity on the edition and publication, under the license Creative Commons Attribution License 3.0 (CC-BY-NC) which allows anyone to share to adapt the work according to the following terms:
Atribuição — You must give the appropriate credit, provide a link to the license and indicate if the changes have been made. You must do so in any reasonable circumstance, but in no way do you suggest to the licensor to support you or your use.
Non-Commercial — You may not use the material for commercial purposes.
- If plagiarism has been proven in any published work, this Journal is free of any responsibility, and its author (s) shall bear the penalties provided by law.
- Authors are authorized to take additional contracts separately, for non-exclusive distribution of the version of the work published in this journal (eg, publish in institutional repository or as a book chapter), with acknowledgment of authorship and initial publication in this journal.
- Authors are allowed and encouraged to publish and distribute their work online (eg in institutional repositories or on their personal page), as this can generate productive changes, as well as increase the impact and citation of the published work. However, it should be noted that once approved by the evaluators, the manuscript can not be changed. If the author wishes to do so, he should restart the submission process.