[Paper Review] How a DQN Tamed Airfoil Vortices with a Synthetic Jet — Reinforcement-Learning Flow Control

At a high angle of attack, vortices peel off the airfoil in turn. Lift swings every cycle and the wing shakes. The usual fix is a shape change or a frozen periodic forcing. Hammouda et al. (2026) left the jet speed free. A synthetic jet (a zero-net-mass actuator that blows, then sucks) sits near the leading edge, and a DQN picks one of 21 speeds from 0 to 20 m/s.

Set the ε slider near 0 and the bars lock onto whichever speed scored high first. At 0.2 the yellow exploration picks thin out and the cyan exploitation picks gather near 12 m/s.

Hit Train and watch whether the yellow picks fade as the bar heights update. Push ε to 0.8 and the agent keeps sampling other speeds after it already knows a good bin.

Where this paper sits#

• Title: Application of deep reinforcement learning for aerodynamic control around an angled airfoil via synthetic jet
• Authors: N. Ghezaiel Hammouda, R. Khan, L. Mostafa, et al. (Scientific Reports, 2026)
• Setting: Weakly compressible laminar flow at Reynolds number (inertia over viscosity) 100 and Mach 0.2. A high-angle airfoil with a synthetic jet near the leading edge.
• Result: Dueling DQN converged most steadily, cutting vortex shedding while raising lift and lowering drag.

At Re 100 the flow is laminar. A high angle of attack still sheds vortices behind the wing. That shedding is what swings lift and drag.

Translating vortices into a reinforcement-learning problem#

Reinforcement learning (a policy learned by trial to grow a reward) starts once three things are named.

• State: pressure and velocity from virtual sensors around the airfoil and in the wake. The paper reports faster learning when velocity is added to pressure.
• Action: jet speed $U_a$. Integers from 0 to 20 m/s in steps of 1 m/s, 21 bins, because DQN wants a discrete set.
• Reward: one line that cuts drag and raises lift.

$\langle C_D \rangle_{ac}$ and $\langle C_L \rangle_{ac}$ are drag and lift averaged over one action window. $R_1$ and $R_2$ keep the reward positive and set the weight of lift against drag. The paper used $R_1=3$, $R_2=0.2$. One action is one shedding period. Training runs 300 episodes of 25 periods each.

ε-greedy: between exploration and exploitation#

The agent stores action value as $Q(s,a)$. The update is the Bellman step.

$\alpha$ is the learning rate, $\gamma$ discounts future reward, and $\max_{a'}Q(s',a')$ is the best value expected from the next state.

The open question is how to try an action whose value is still unknown. ε-greedy picks the current best (exploitation) with probability $1-\epsilon$ and a random bin (exploration) with probability $\epsilon$. Large $\epsilon$ wanders. Small $\epsilon$ sits down fast. That probability is why the bars above stall near 0 and walk toward 12 m/s near 0.2.

The synthetic jet as an action#

A synthetic jet shakes a membrane, blows out of one hole, and sucks back in. Net mass is zero. Momentum still enters the boundary layer. The dimensionless size of that push is the momentum coefficient.

$\rho_j$, $U_a$, $d_j$ are jet density, speed, and orifice diameter. $\rho_\infty$, $U_\infty$, $c$ are freestream density, speed, and chord. The hole is on the suction side near the leading edge at $x/c=0.1$, diameter 0.2 mm. Extra momentum delays separation and weakens shedding.

At Mach 0.2 the acoustic waves tighten the time step before the agent ever sees the wake. How to integrate that low-Mach problem without deleting the waves is the subject of an asymptotic-preserving IMEX scheme. The wake picture sits after the printed reward.

Hands-on: turning the jet on with Q-learning#

The paper's network does not get ported whole. A table of Q-values keeps the binned amplitude and the discrete speeds. The state is a bin of lift-oscillation amplitude. The action is jet speed.

import numpy as np
 
class SyntheticJetEnv:
    """1D phenomenological airfoil-wake environment.
 
    State  : binned lift-oscillation amplitude (0..n_bins-1)
    Action : jet speed level {0,1,...,20} m/s
    Reward : R1 - <Cd> + R2*<Cl>  (paper Eq. 4)
    """
    def __init__(self, n_bins=6, peak=12, R1=3.0, R2=0.2, seed=0):
        self.n_bins, self.peak = n_bins, peak
        self.R1, self.R2 = R1, R2
        self.rng = np.random.default_rng(seed)
        self.amp = 1.0  # normalized shedding amplitude (1 = uncontrolled)
 
    def reset(self):
        self.amp = 1.0
        return self._bin()
 
    def _bin(self):
        return min(self.n_bins - 1, int(self.amp * self.n_bins))
 
    def step(self, action):
        ctrl = action / 20.0                       # control authority 0..1
        target = max(0.05, 1.0 - 0.8 * ctrl)       # jet damps the amplitude
        self.amp += 0.5 * (target - self.amp)      # first-order relaxation
        cl = 1.8 + 0.2 * ctrl - 0.4 * self.amp     # lift coefficient
        cd = 0.085 - 0.006 * ctrl + 0.02 * self.amp  # drag coefficient
        waste = 0.01 * max(0, action - self.peak)  # penalty for over-blowing
        reward = self.R1 - cd + self.R2 * cl - waste
        reward += self.rng.normal(0, 0.05)
        return self._bin(), reward
 
def epsilon_greedy(q_row, eps, rng):
    if rng.random() < eps:
        return int(rng.integers(len(q_row)))      # explore
    return int(np.argmax(q_row))                  # exploit
 
def train_jet_controller(episodes=300, steps=25, alpha=0.1, gamma=0.9, eps0=0.3):
    env = SyntheticJetEnv()
    n_actions = 21
    Q = np.zeros((env.n_bins, n_actions))
    rng = np.random.default_rng(1)
    history = []
    for ep in range(episodes):
        s = env.reset()
        eps = eps0 * (1 - ep / episodes)          # linear decay
        total = 0.0
        for _ in range(steps):
            a = epsilon_greedy(Q[s], eps, rng)
            s2, r = env.step(a)
            Q[s, a] += alpha * (r + gamma * Q[s2].max() - Q[s, a])
            s, total = s2, total + r
        history.append(total / steps)
    best = int(np.argmax(Q.sum(axis=0)))
    return Q, history, best
 
if __name__ == "__main__":
    Q, hist, best = train_jet_controller()
    print(f"episode   1 avg reward = {hist[0]:.3f}")
    print(f"episode 300 avg reward = {hist[-1]:.3f}")
    print(f"learned jet velocity   = {best} m/s")

The output looks like this.

episode   1 avg reward = 3.12
episode 300 avg reward = 3.25
learned jet velocity   = 12 m/s

Leave the jet-speed slider at 0 and vortices peel off in turn, with a wide $C_L$ swing. Move it to 15–20 m/s and the vortices fade as the wake settles. The printed 12 m/s is not the right-hand end of that slider. Pause the run and park the slider near 12 to see whether the quiet wake the reward wants is the same bin the table stopped on.

The printed 3.25 is equilibrium. At action 12 the control fraction is 0.6 and the amplitude settles at $1 - 0.8 \times 0.6 = 0.52$. Lift is $1.8 + 0.2 \times 0.6 - 0.4 \times 0.52 = 1.71$ and drag is $0.085 - 0.006 \times 0.6 + 0.02 \times 0.52 = 0.092$. Reward $3 - 0.092 + 0.2 \times 1.71 = 3.25$ matches episode 300. Waste turns on only past action 12, so 12 m/s is the last zero-waste bin, not the peak of the lift formula.

At 20 m/s the amplitude is 0.2, lift is 1.92, drag is 0.083, and waste $0.01 \times (20 - 12) = 0.08$ drops the reward to 3.22. Jet off sits at reward 3.18. The learned gain is 0.07. The printed 3.12 is noise, not that baseline.

Paper lift rises from 1.79 to about 2.0. Times $R_2 = 0.2$ the reward moves only 0.042 beside $R_1 = 3$. The toy's lift at 12 m/s is 1.71, and even 20 m/s reaches only 1.92. The printed speed marks where the penalty starts, not the lift target.

The DQN siblings: Double vs Dueling#

The paper ran three DQN variants side by side.

• Vanilla DQN: the $\max$ leans toward overestimating value.
• Double DQN: one network picks the action, another scores it, which cuts that overestimate.
• Dueling DQN: splits $Q$ into a state value $V(s)$ and an advantage $A(s,a)$.

$V(s)$ is how good the state is. $A(s,a)$ is how much this action beats the average inside it. When 11 m/s and 13 m/s look alike, the state value is learned once. That is why the Dueling curve in the paper shook the least.

Five layers of 128 neurons settled inside 300 episodes. With control on, $C_L$ rose from 1.79 to about 2.0 and the wake calmed. That is a different number from the lift of 1.71 the table produces at 12 m/s.

Why 12 m/s Is Not a Lift Optimum#

The 0.13 gained over 300 episodes is a small move beside $R_1$. The paper's lift rise contributes 0.042 to the reward. Whether that gap is enough to trust a 1 m/s bin is a separate call.

Ranking 11 m/s against 13 m/s in a real CFD loop means integrating 25 shedding periods per episode. A scheme that stretches the step and still loses $L_\infty$ stability is the subject of the bill for a 1001× time step. Can that bill tell the waste penalty at 12 m/s apart from the paper's lift of 2.0?