Spacecraft relative on-off control via reinforcement learning
Рубрика:
| 1Khoroshylov, SV, 2Wang, Chanangoin 1Institute of Technical Mechanics of the National Academy of Sciences of Ukraine and the State Space Agency of Ukraine, Dnipro, Ukraine 2Northwestern Polytechnical University, Xi'an Shaanxi, P. R. China |
| Space Sci. & Technol. 2024, 30 ;(2):03-14 |
| https://doi.org/10.15407/knit2024.02.003 |
| Язык публикации: English |
Аннотация: The article investigates the task of spacecraft relative control using reactive actuators, the output of which has two states "on" or "off". For cases where the resolution of the thrusters does not provide an accurate approximation of linear control laws using a pulse-width thrust modulator, the possibility of applying reinforcement learning methods for direct finding of control laws that map the state vector and the on-off thruster commands has been investigated. To implement such an approach, a model of controlled relative motion of two satellites in the form of a Markov decision process was obtained. The intelligent agent is presented in the form of "actor" and "critic" neural networks, and the architecture of these modules is defined. It is proposed to use a cost function with variable weights of control actions, which allows optimizing the number thruster firings explicitly.
To improve the control performance, it is proposed to use an extended input vector for "actor" and "critic" neural networks of the intelligent agent, which, in addition to the state vector, also includes information about the control action on the previous control step and the control step number. To reduce the training time, the agent was pretrained on the data obtained using conventional control algorithms. Numerical results demonstrate that the reinforcement learning methodology allows the agent to outperform the results provided by the linear controller with pulse-width modulator in terms of control accuracy, response time, and numbers of thruster firings.
|
| Ключевые слова: actor, critic, neu-ral network, on-off control, reinforcement learning, spacecraft relative control, thruster firing |
References:
1. Alpatov A. P., Cichocki F., Fokov A. A., Khoroshylov S. V., Merino M., Zakrzhevskii A. E. (2015). Algorithm for determination of force transmitted by plume of ion thruster to orbital object using photo camera. 66th Int. Astronautical Congress, Jerusalem, Israel, 2239-2247.
2. Alpatov A., Khoroshylov S., Lapkhanov E. (2020). Synthesizing an Algo-rithm to Control the Angular Motion of Spacecraft Equipped with an Aeromagnetic Deorbiting System. Eastern-European Journal of Enterprise Technologies. 5 (103), 37-46.
https://doi.org/10.15587/1729-4061.2020.192813
3. Anthony T., Wie B., Carroll S. (1989). Pulse-Modulated Control Synthesis for a Flexible Spacecraft. Journal of Guidance, Control, and Dynamics. Vol 13 (6), 1014-1022.
https://doi.org/10.2514/6.1989-3433
4. Artificial intelligence: a modern approach (2010). Eds. S. J. Russell, P. Norvig. Pearson education. Inc. ISBN-13: 978-0134610993.
5. Bernelli-Zazzera F., Mantegazza P., Nurzia V. (1998). Multi-Pulse-Width Modulated Control of Linear Systems. Journal of Guidance, Control, and Dynam-ics. Vol 21 (1), 64-70.
https://doi.org/10.2514/2.4198
6. Deep Learning (2016). Eds. I. Goodfellow, Y. Bengio, A. Courville. The MIT Press. ISBN 978-0262035613.
7. Gaudet B., Linares R., Furfaro R. (2020). Adaptive guidance and integrated navigation with reinforcement meta-learning. Acta Astronautica, 169, 180-190.
https://doi.org/10.1016/j.actaastro.2020.01.007
8. Gaudet B., Linares R., Furfaro R. (2020). Seeker based adaptive guidance via reinforcement meta-learning applied to asteroid close proximity operations. Acta Astronautica, 171, 1-13.
https://doi.org/10.1016/j.actaastro.2020.02.036
9. Golubek A. V., Dron M. M., Petrenko O. M. (2023). Estimation of the pos-sibility of using electric propulsion systems for large-sized orbital debris post-mission disposal. Space Science and Technology., 29, № 3 (142), 34-46.
https://doi.org/10.15407/knit2023.03.034
10. Hovell K., Ulrich S. (2020). On deep reinforcement learning for spacecraft guidance. AIAA SciTech Forum, 6-10 January 2020, Orlando, FL.
https://doi.org/10.2514/6.2020-1600
11. Ieko T., Ochi Y., Kanai K. (1997) A New Digital Redesign Method for Pulse-Width Modulation Control Systems. AIAA proceedings AIAA-97, 3700.
https://doi.org/10.2514/6.1997-3770
12. Izzo D., Märtens M., Pan B. (2019). A survey on artificial intelligence trends in spacecraft guidance dynamics and control. Astrodyn., 3, 287-299.
https://doi.org/10.1007/s42064-018-0053-6
13. Khoroshylov S. V. (2018). Relative motion control system of spacecraft for contactless space debris removal. Nauka innov., 14, № 4, 5-16.
https://doi.org/10.15407/scin14.04.005
14. Khoroshylov S. V., Redka M. O. (2019). Relative control of an underactuat-ed spacecraft using reinforcement learning. Тechnical Mechanics, 4, 43-54.
https://doi.org/10.15407/itm2020.04.043
15. Khoroshylov S. V., Redka M. O. (2021). Deep learning for space guidance, navigation, and control. Space Science and Technology. Vol. 27, № 6 (133), 38-52.
https://doi.org/10.15407/knit2021.06.038
16. Khosravi A., Sarhadi P. (2016). Tuning of pulse-width pulse-frequency modulator using PSO: An engineering approach to spacecraft attitude controller de-sign. Automatika. № 57, 212-220.
https://doi.org/10.7305/automatika.2016.07.618
17. Lapkhanov, E., Khoroshylov, S. (2019). Development of the aeromagnetic space debris deorbiting system. Eastern-European Journal of Enterprise Technolo-gies. 5 (101), 30-37.
https://doi.org/10.15587/1729-4061.2019.179382
18. Lewis F. L., Vrabie D., Syrmos V.L., Optimal Control, 3rd Edition. John Wiley & Sons, Inc., New York, USA (2012).
https://doi.org/10.1002/9781118122631
19. Li W., Cheng D., Liu X., at al. (2019). On-orbit service (OOS) of spacecraft: A review of engineering developments, Progress in Aerospace Sciences, Volume 108, 32-120.
https://doi.org/10.1016/j.paerosci.2019.01.004
20. Machine Learning (1997). Ed. T. Mitchell. New York: McGraw Hill. ISBN 0070428077.
21. Mnih V., Badia A., Mirza M., Graves A., Lillicrap T., Harley T., Silver D. (2016). Asynchronous Methods for Deep Reinforcement Learning. arXiv preprint, ArXiv:1602.01783.
22. Oestreich C.E., Linaresy R., Gondhalekarz R. (2021). Autonomous six-degree-of-freedom spacecraft docking maneuvers via reinforcement learning. J. Aerospace Inform. Syst., 18, № 7.
https://doi.org/10.2514/1.I010914
23. Redka M. O., Khoroshylov S. V. (2022). Determination of the force impact of an ion thruster plume on an orbital object via deep learning // Space Science and Technology. 28, № 5 (138), 15-26.
https://doi.org/10.15407/knit2022.05.015
24. Reinforcement learning: an introduction (1998). Eds. R. S. Sutton, A. G. Barto. MIT press. ISBN 978-0262193986.
25. Robinett R. D., Parker G. G., Schaub H., Junkins J. (1997). Lyapunov Opti-mal Saturated Control for Nonlinear Systems. Journal of Guidance, Control, and Dynamics. Vol 20 (6), 1083-1088.
https://doi.org/10.2514/2.4189
26. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. (2017). Proxi-mal policy optimization algorithms. arXiv preprint, arXiv:1707.06347.
27. Silver D., Schrittwieser J., Simonyan K. (2017). Mastering the game of Go without human knowledge. Nature, 550, 354-359.
https://doi.org/10.1038/nature24270
28. Song G., Buck N. V., Agrawal B. N. (1999). Spacecraft Vibration reduction using pulse-width pulse-frequency modulated input shaper. Journal of Guidance, Control, and Dynamics. Vol 22 (6), 433-440.
https://doi.org/10.2514/2.4415
29. Yamanaka K., Ankersen F. (2002). New State Transition Matrix for Relative Motion on an Arbitrary Elliptical Orbit. Journal of Guidance, Control, and Dynamics. 25 (1), 60-66.
https://doi.org/10.2514/2.4875