Closed-loop Flow Control Method Based on Deep Reinforcement Learning using a Co-flow Jet

Document Type : Regular Article


National Key Laboratory of Science and Technology on Aerodynamic Design and Research, Northwestern Polytechnical University, Xi’an, 710072, China



A closed-loop control framework is developed for the co-flow jet (CFJ) airfoil by combining the numerical flow field environment of a CFJ0012 airfoil with a deep reinforcement learning (DRL) module called tensorforce integrated in Python. The DRL agent, which is trained through interacting with the numerical flow field environment, is capable of acquiring a policy that instructs the mass flow rate of the CFJ to make the stalled airfoil at an angle of attack (AoA) of 18 degrees reach a specific high lift coefficient set to 2.0, thereby effectively suppressing flow separation on the upper surface of the airfoil. The subsequent test shows that the policy can be implemented to find a precise jet momentum coefficient of 0.049 to make the lift coefficient of the CFJ0012 airfoil reach 2.01 with a negligible error of 0.5%. Moreover, to evaluate the generalization ability of the policy trained at an AoA of 18 degrees, two additional tests are conducted at AoAs of 16 and 20 degrees. The results show that, although using the policy gained under another AoA cannot help the lift coefficient of the airfoil reach a set target of 2 accurately, the errors are acceptable with less than 5.5%, which means the policy trained under an AoA of 18 degrees can also be applied to other AoAs to some extent. This work is helpful for the practical application of CFJ technology, as the closed-loop control framework ensures good aerodynamic performance of the CFJ airfoil, even in complex and changeable flight conditions. 


Main Subjects

Ahuja, S., & Rowley, C. W. (2010). Feedback control of unstable steady states of flow past a flat plate using reduced-order estimators. Journal of Fluid Mechanics, 645, 447–478.
Akbiyik, H., & Yavuz, H. (2021). Artificial neural network application for aerodynamics of an airfoil equipped with plasma actuators. Journal of Applied Fluid Mechanics, 14(4), 1165–1181.
Franklin, G. F., Powell, J. D., & Emami-Naeini, A. (2019). Feedback Control of Dynamic Systems (8th ed.). Boston, MA: Pearson.
Gan, W. B., Zhou, Z., Xu, X. P., & Wang, R. (2013). Delayed detached-eddy simulation and application of a coflow jet airfoil at high angle of attack. Applied Mechanics and Materials, 444–445, 270–276.
Glauser, M., Higuchi, H., Ausseur, J., Pinier, J., & Carlson, H. (2004, June 28). Feedback control of separated flows. 2nd AIAA Flow Control Conference. 2nd AIAA Flow Control Conference, Portland, Oregon.
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. 2017 IEEE International Conference on Robotics and Automation (ICRA), 3389–3396.
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Pérez, P. (2022). Deep reinforcement learning for autonomous driving: a survey. IEEE Transactions on Intelligent Transportation Systems, 23(6), 4909–4926.
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90.
Lefebvre, A. M., & Zha, G. (2014, June 16). Co-flow jet airfoil trade study part I: Energy consumption and aerodynamic efficiency. 32nd AIAA Applied Aerodynamics Conference, Atlanta, GA.
Lei, Z., & Zha, G. (2021). Lift enhancement of supersonic thin airfoil at low speed by co-flow jet active flow control. AIAA AVIATION 2021 FORUM.
Lou, J., Chen, R., Liu, J., Bao, Y., You, Y., & Chen, Z. (2023). Aerodynamic optimization of airfoil based on deep reinforcement learning. Physics of Fluids, 35(3), 037128.
Ma, C. Y., & Xu, H. Y. (2022). Parameter-based design and analysis of wind turbine airfoils with conformal slot co-flow jet. Journal of Applied Fluid Mechanics, 16(2), 269–283.
Ma, C. Y., Xu, H. Y., & Qiao, C. L. (2023). Comparative study of two combined blowing and suction flow control methods on pitching airfoils. Physics of Fluids, 35(3), 035120.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013, December 19). Playing atari with deep reinforcement learning. ArXiv.Org.
Moshtaghzadeh, M., & Aligoodarz, M. R. (2022). Prediction of wind turbine airfoil performance using artificial neural network and CFD approaches. International Journal of Engineering and Technology Innovation, 12(4), 275–287.
Paris, R., Beneddine, S., & Dandois, J. (2021). Robust flow control and optimal sensor placement using deep reinforcement learning. Journal of Fluid Mechanics, 913, A25.
Rabault, J., & Kuhnle, A. (2019). Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Physics of Fluids, 31(9), 094105.
Rabault, J., Kuchta, M., Jensen, A., Réglade, U., & Cerardi, N. (2019). Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. Journal of Fluid Mechanics, 865, 281–302.
Ren, F., Rabault, J., & Tang, H. (2021a). Applying deep reinforcement learning to active flow control in weakly turbulent conditions. Physics of Fluids, 33(3), 037121.
Ren, F., Wang, C., & Tang, H. (2021b). Bluff body uses deep-reinforcement-learning trained active flow control to achieve hydrodynamic stealth. Physics of Fluids, 33(9), 093602.
Samy, I., Postlethwaite, I., Gu, D. W., & Green, J. (2010). Neural-Network-Based flush air data sensing system demonstrated on a mini air vehicle. Journal of Aircraft, 47(1), 18–31.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., & Moritz, P. (2015). Trust region policy optimization. Proceedings of the 32nd International Conference on Machine Learning.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017, July 20). Proximal Policy Optimization Algorithms. ArXiv.Org.
Shimomura, S., Sekimoto, S., Oyama, A., Fujii, K., & Nishida, H. (2020). Closed-Loop flow separation control using the deep q network over airfoil. AIAA Journal, 58(10), 4260–4270.
Siegel, S., Cohen, K., & McLaughlin, T. (2003, June 23). Feedback control of a circular cylinder wake in experiment and simulation (Invited). 33rd AIAA Fluid Dynamics Conference and Exhibit, Orlando, Florida.
Silver, D., Huang, A., Maddison, C. Guez, A., Sifre, L., Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Kalchbrenner, N., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489.
Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3(1), 9–44.
Tang, H., Rabault, J., Kuhnle, A., Wang, Y., & Wang, T. (2020). Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning. Physics of Fluids, 32(5), 053605.
Wang, B., & Zha, G. C. (2011). Detached-Eddy simulation of a coflow jet airfoil at high angle of attack. Journal of Aircraft, 48(5), 1495–1502.
Wang, B., Haddoukessouni B., Levy, J., & Zha, G. C. (2008). Numerical investigations of injection-slot-size effect on the performance of coflow jet airfoils. Journal of Aircraft, 45(6), 2084-2091.
Xu, K., & Zha, G. (2021). High control authority three-dimensional aircraft control surfaces using coflow jet. Journal of Aircraft, 58(1), 72-84.
Yang, Y., & Zha, G. (2019, January 7). Conceptual design of the co-flow jet hybrid electric regional airplane. AIAA Scitech 2019 Forum. AIAA Scitech 2019 Forum, San Diego, California.
Zha, G. C., Paxton, C. D., Conley, C. A., Wells, A., & Carroll, B. F. (2006a). Effect of injection slot size on the performance of coflow jet airfoil. Journal of aircraft, 43(4), 987-995.
Zha, G., Gao W., & Paxton, C. (2006b, January 9). Numerical simulation of co-flow jet airfoil flows. 44th AIAA Aerospace Sciences Meeting and Exhibit. Reno, Nevada.
Zha, G., Yang, Y., Ren, Y., & McBreen, B. (2018, June 25). Super-lift and thrusting airfoil of coflow jet actuated by micro-compressors. 2018 Flow Control Conference, Atlanta, Georgia.
Zha, G. C., & Paxton, C. (2004, June 28). A novel airfoil circulation augment flow control method using CFJ. 2nd AIAA Flow Control Conference, Portland, Oregon.
Zha, G. C., Carroll, B. F., Paxton, C. D., Conley, C. A., & Wells, A. (2007a). High-performance airfoil using coflow jet flow control. AIAA Journal, 45(8), 2087–2090.
Zha, G. C., Gao, W., & Paxton, C. D. (2007b). Jet effects on coflow jet airfoil performance. AIAA Journal, 45(6), 1222–1231.