Spacecraft relative on-off control via reinforcement learning
Heading:
| 1Khoroshylov, SV, 2Wang, C 1Institute of Technical Mechanics of the National Academy of Sciences of Ukraine and the State Space Agency of Ukraine, Dnipro, Ukraine 2Northwestern Polytechnical University, Xi'an Shaanxi, P. R. China |
| Space Sci. & Technol. 2024, 30 ;(2):01-01 |
| Publication Language: English |
Abstract: The article investigates the task of spacecraft relative control using reactive actuators, the output of which has two states "on" or "off". For cases where the resolution of the thrusters does not provide an accurate approximation of linear control laws using a pulse-width thrust modulator, the possibility of applying reinforcement learning methods for direct finding of control laws that map the state vector and the on-off thruster commands has been investigated. To implement such an approach, a model of controlled relative motion of two satellites in the form of a Markov decision process was obtained. The intelligent agent is presented in the form of "actor" and "critic" neural networks, and the architecture of these modules is defined. It is proposed to use a cost function with variable weights of control actions, which allows optimizing the number thruster firings explicitly.
To improve the control performance, it is proposed to use an extended input vector for "actor" and "critic" neural networks of the intelligent agent, which, in addition to the state vector, also includes information about the control action on the previous control step and the control step number. To reduce the training time, the agent was pretrained on the data obtained using conventional control algorithms. Numerical results demonstrate that the reinforcement learning methodology allows the agent to outperform the results provided by the linear controller with pulse-width modulator in terms of control accuracy, response time, and numbers of thruster firings.
|
| Keywords: actor, critic, neu-ral network, on-off control, reinforcement learning, spacecraft relative control, thruster firing |
Full text (PDF)
References:
1. Alpatov A. P., Cichocki F., Fokov A. A., Khoroshylov S. V., Merino M., Zakrzhevskii A. E. (2015). Algorithm for determination of force transmitted by plume of ion thruster to orbital object using photo camera. 66th Int. Astronautical Congress, Jerusalem, Israel, 2239—2247.
2. Alpatov A., Khoroshylov S., Lapkhanov E. (2020). Synthesizing an Algo-rithm to Control the Angular Motion of Spacecraft Equipped with an Aeromagnetic Deorbiting System. Eastern-European Journal of Enterprise Technologies. 5 (103), 37-46.
3. Anthony T., Wie B., Carroll S. (1989). Pulse-Modulated Control Synthesis for a Flexible Spacecraft. Journal of Guidance, Control, and Dynamics. Vol 13 (6), 1014–1022.
4. Artificial intelligence: a modern approach (2010). Eds. S. J. Russell, P. Norvig. Pearson education. Inc. ISBN-13: 978-0134610993.
5. Bernelli-Zazzera F., Mantegazza P., Nurzia V. (1998). Multi-Pulse-Width Modulated Control of Linear Systems. Journal of Guidance, Control, and Dynam-ics. Vol 21 (1), 64–70.
6. Deep Learning (2016). Eds. I. Goodfellow, Y. Bengio, A. Courville. The MIT Press. ISBN 978-0262035613.
7. Gaudet B., Linares R., Furfaro R. (2020). Adaptive guidance and integrated navigation with reinforcement meta-learning. Acta Astronautica, 169, 180—190.
8. Gaudet B., Linares R., Furfaro R. (2020). Seeker based adaptive guidance via reinforcement meta-learning applied to asteroid close proximity operations. Acta Astronautica, 171, 1—13.
9. Golubek A. V., Dron M. M., Petrenko O. M. (2023). Estimation of the pos-sibility of using electric propulsion systems for large-sized orbital debris post-mission disposal. Space Science and Technology., 29, № 3 (142), 34—46. https://doi.org/10.15407/knit2023.03.034
10. Hovell K., Ulrich S. (2020). On deep reinforcement learning for spacecraft guidance. AIAA SciTech Forum, 6—10 January 2020, Orlando, FL. DOI: 10.2514/6.2020-1600.
11. Ieko T., Ochi Y., Kanai K. (1997) A New Digital Redesign Method for Pulse-Width Modulation Control Systems. AIAA proceedings AIAA-97, 3700.
12. Izzo D., Märtens M., Pan B. (2019). A survey on artificial intelligence trends in spacecraft guidance dynamics and control. Astrodyn., 3, 287—299. DOI: 10.1007/s42064-018-0053-6.
13. Khoroshylov S. V. (2018). Relative motion control system of spacecraft for contactless space debris removal. Nauka innov., 14, № 4, 5—16.
14. Khoroshylov S. V., Redka M. O. (2019). Relative control of an underactuat-ed spacecraft using reinforcement learning. Тechnical Mechanics, 4, 43—54.
15. Khoroshylov S. V., Redka M. O. (2021). Deep learning for space guidance, navigation, and control. Space Science and Technology. Vol. 27, № 6 (133), 38—52.
16. Khosravi A., Sarhadi P. (2016). Tuning of pulse-width pulse-frequency modulator using PSO: An engineering approach to spacecraft attitude controller de-sign. Automatika. № 57, 212–220.
17. Lapkhanov, E., Khoroshylov, S. (2019). Development of the aeromagnetic space debris deorbiting system. Eastern-European Journal of Enterprise Technolo-gies. 5 (101), 30–37.
18. Lewis F. L., Vrabie D., Syrmos V.L., Optimal Control, 3rd Edition. John Wiley & Sons, Inc., New York, USA (2012).
19. Li W., Cheng D., Liu X., at al. (2019). On-orbit service (OOS) of spacecraft: A review of engineering developments, Progress in Aerospace Sciences, Volume 108, 32-120.
20. Machine Learning (1997). Ed. T. Mitchell. New York: McGraw Hill. ISBN 0070428077.
21. Mnih V., Badia A., Mirza M., Graves A., Lillicrap T., Harley T., Silver D. (2016). Asynchronous Methods for Deep Reinforcement Learning. arXiv preprint, ArXiv:1602.01783.
22. Oestreich C.E., Linaresy R., Gondhalekarz R. (2021). Autonomous six-degree-of-freedom spacecraft docking maneuvers via reinforcement learning. J. Aerospace Inform. Syst., 18, № 7. DOI: 10.2514/1.I010914.
23. Redka M. O., Khoroshylov S. V. (2022). Determination of the force impact of an ion thruster plume on an orbital object via deep learning // Space Science and Technology. 28, № 5 (138), 15—26.
24. Reinforcement learning: an introduction (1998). Eds. R. S. Sutton, A. G. Barto. MIT press. ISBN 978-0262193986.
25. Robinett R. D., Parker G. G., Schaub H., Junkins J. (1997). Lyapunov Opti-mal Saturated Control for Nonlinear Systems. Journal of Guidance, Control, and Dynamics. Vol 20 (6), 1083–1088.
26. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. (2017). Proxi-mal policy optimization algorithms. arXiv preprint, arXiv:1707.06347.
27. Silver D., Schrittwieser J., Simonyan K. (2017). Mastering the game of Go without human knowledge. Nature, 550, 354—359. DOI:10.1038/nature24270.
28. Song G., Buck N. V., Agrawal B. N. (1999). Spacecraft Vibration reduction using pulse-width pulse-frequency modulated input shaper. Journal of Guidance, Control, and Dynamics. Vol 22 (6), 433–440.
29. Yamanaka K., Ankersen F. (2002). New State Transition Matrix for Relative Motion on an Arbitrary Elliptical Orbit. Journal of Guidance, Control, and Dynam-ics. 25 (1), 60–66.
2. Alpatov A., Khoroshylov S., Lapkhanov E. (2020). Synthesizing an Algo-rithm to Control the Angular Motion of Spacecraft Equipped with an Aeromagnetic Deorbiting System. Eastern-European Journal of Enterprise Technologies. 5 (103), 37-46.
3. Anthony T., Wie B., Carroll S. (1989). Pulse-Modulated Control Synthesis for a Flexible Spacecraft. Journal of Guidance, Control, and Dynamics. Vol 13 (6), 1014–1022.
4. Artificial intelligence: a modern approach (2010). Eds. S. J. Russell, P. Norvig. Pearson education. Inc. ISBN-13: 978-0134610993.
5. Bernelli-Zazzera F., Mantegazza P., Nurzia V. (1998). Multi-Pulse-Width Modulated Control of Linear Systems. Journal of Guidance, Control, and Dynam-ics. Vol 21 (1), 64–70.
6. Deep Learning (2016). Eds. I. Goodfellow, Y. Bengio, A. Courville. The MIT Press. ISBN 978-0262035613.
7. Gaudet B., Linares R., Furfaro R. (2020). Adaptive guidance and integrated navigation with reinforcement meta-learning. Acta Astronautica, 169, 180—190.
8. Gaudet B., Linares R., Furfaro R. (2020). Seeker based adaptive guidance via reinforcement meta-learning applied to asteroid close proximity operations. Acta Astronautica, 171, 1—13.
9. Golubek A. V., Dron M. M., Petrenko O. M. (2023). Estimation of the pos-sibility of using electric propulsion systems for large-sized orbital debris post-mission disposal. Space Science and Technology., 29, № 3 (142), 34—46. https://doi.org/10.15407/knit2023.03.034
10. Hovell K., Ulrich S. (2020). On deep reinforcement learning for spacecraft guidance. AIAA SciTech Forum, 6—10 January 2020, Orlando, FL. DOI: 10.2514/6.2020-1600.
11. Ieko T., Ochi Y., Kanai K. (1997) A New Digital Redesign Method for Pulse-Width Modulation Control Systems. AIAA proceedings AIAA-97, 3700.
12. Izzo D., Märtens M., Pan B. (2019). A survey on artificial intelligence trends in spacecraft guidance dynamics and control. Astrodyn., 3, 287—299. DOI: 10.1007/s42064-018-0053-6.
13. Khoroshylov S. V. (2018). Relative motion control system of spacecraft for contactless space debris removal. Nauka innov., 14, № 4, 5—16.
14. Khoroshylov S. V., Redka M. O. (2019). Relative control of an underactuat-ed spacecraft using reinforcement learning. Тechnical Mechanics, 4, 43—54.
15. Khoroshylov S. V., Redka M. O. (2021). Deep learning for space guidance, navigation, and control. Space Science and Technology. Vol. 27, № 6 (133), 38—52.
16. Khosravi A., Sarhadi P. (2016). Tuning of pulse-width pulse-frequency modulator using PSO: An engineering approach to spacecraft attitude controller de-sign. Automatika. № 57, 212–220.
17. Lapkhanov, E., Khoroshylov, S. (2019). Development of the aeromagnetic space debris deorbiting system. Eastern-European Journal of Enterprise Technolo-gies. 5 (101), 30–37.
18. Lewis F. L., Vrabie D., Syrmos V.L., Optimal Control, 3rd Edition. John Wiley & Sons, Inc., New York, USA (2012).
19. Li W., Cheng D., Liu X., at al. (2019). On-orbit service (OOS) of spacecraft: A review of engineering developments, Progress in Aerospace Sciences, Volume 108, 32-120.
20. Machine Learning (1997). Ed. T. Mitchell. New York: McGraw Hill. ISBN 0070428077.
21. Mnih V., Badia A., Mirza M., Graves A., Lillicrap T., Harley T., Silver D. (2016). Asynchronous Methods for Deep Reinforcement Learning. arXiv preprint, ArXiv:1602.01783.
22. Oestreich C.E., Linaresy R., Gondhalekarz R. (2021). Autonomous six-degree-of-freedom spacecraft docking maneuvers via reinforcement learning. J. Aerospace Inform. Syst., 18, № 7. DOI: 10.2514/1.I010914.
23. Redka M. O., Khoroshylov S. V. (2022). Determination of the force impact of an ion thruster plume on an orbital object via deep learning // Space Science and Technology. 28, № 5 (138), 15—26.
24. Reinforcement learning: an introduction (1998). Eds. R. S. Sutton, A. G. Barto. MIT press. ISBN 978-0262193986.
25. Robinett R. D., Parker G. G., Schaub H., Junkins J. (1997). Lyapunov Opti-mal Saturated Control for Nonlinear Systems. Journal of Guidance, Control, and Dynamics. Vol 20 (6), 1083–1088.
26. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. (2017). Proxi-mal policy optimization algorithms. arXiv preprint, arXiv:1707.06347.
27. Silver D., Schrittwieser J., Simonyan K. (2017). Mastering the game of Go without human knowledge. Nature, 550, 354—359. DOI:10.1038/nature24270.
28. Song G., Buck N. V., Agrawal B. N. (1999). Spacecraft Vibration reduction using pulse-width pulse-frequency modulated input shaper. Journal of Guidance, Control, and Dynamics. Vol 22 (6), 433–440.
29. Yamanaka K., Ankersen F. (2002). New State Transition Matrix for Relative Motion on an Arbitrary Elliptical Orbit. Journal of Guidance, Control, and Dynam-ics. 25 (1), 60–66.