Capstone L3 — Bursty flow: managing an inventory spike you can’t see coming

Capstone C2 modelled toxic flow: informed clients, a mid that drifts against you, losses from adverse selection. This one models the danger you are far more likely to actually meet on a market-making desk — and it needs no informed clients at all.

The setup

The mid is a fair, driftless random walk. Clients are innocent. But flow is bursty: most intervals are calm, then — an index rebalance, a stop cascade, one real-money order sliced across dealers — a batch of up to max_simultaneous clients (13 here) lifts your one streamed price at once, before you can re-quote. You are now holding a big one-sided book through no fault of the price. Expected PnL is zero, but the variance of that book is large and getting flat costs the spread. The loss is risk, not edge.

Two honest constraints, two levers

You cannot see the burst coming — the observation is only recent mid moves and your own inventory. So you manage the book you already hold, with two levers:

• Size — showing less as inventory grows limits how fast a burst loads you. But on a fair mid there is no drift to bail you out: inventory is a one-way ratchet, and sizing down only slows the climb — it cannot bring you to flat.
• Hedge — pay the spread to offload into the market. This is how you actually get down. Sizing without hedging just digs slower.

So the real answer is both: size off inventory to limit intake, hedge off inventory to work it back down. That subtlety — that sizing alone can’t cap a one-way spike — is the point of this lesson.

Three dealers, on the same held-out market

• constant-size — full size always, never hedges. Gets lifted for the lot and its book ratchets to the worst spike of the three.
• inventory-manager — a hand rule: size and hedge, both scaled to how heavy it is. The explicit version of the two-lever answer.
• PPO — sees only its inventory and recent mid moves, and must discover how to keep the book bounded. Watch which lever it leans on.

The bar is risk-adjusted, not gross PnL: keep the inventory spike small without throwing away the spread.

Run order matters. The cells below build on each other. Run them top to bottom (Jupyter: Run → Run All Cells; VS Code: Run All). If you hit NameError (or similar), you skipped the Setup cell — run it first.

Setup — imports & configuration

Run this cell first. It imports every library and defines the module-level constants the rest of the notebook relies on.

import numpy as np
import matplotlib.pyplot as plt
from stable_baselines3 import PPO
from stable_baselines3.common.monitor import Monitor
from stable_baselines3.common.callbacks import BaseCallback
from common.fx_burst_flow import (
    BurstMarketMakingEnv, constant_size_policy, inventory_manager_policy,
)
from common import sharpe_ratio, plot_learning_curve
SEED = 0
MAX_SIMULTANEOUS = 13
EVAL_SEED = 12345
N_TRAIN = 6000
N_TEST = 4000
TOTAL_STEPS = 300_000

make_env (function)

Build a bursty-flow market. The episode is fixed by the reset seed, so training (varied seeds) and evaluation (one held-out seed) are cleanly separated and every policy is scored on the identical market + bursts.

def make_env(n):
    return BurstMarketMakingEnv(n_steps=n, max_simultaneous=MAX_SIMULTANEOUS)

rollout (function)

Run one held-out pass under a FIXED seed; collect per-step PnL and the inventory path. The same seed for every policy is what makes it fair.

def rollout(env, act_fn, seed):
    obs, _ = env.reset(seed=seed)
    pnls, inv = [], []
    done = False
    while not done:
        obs, reward, term, trunc, info = env.step(act_fn(env, obs))
        pnls.append(info["step_pnl"])
        inv.append(info["inventory"])
        done = term or trunc
    inv_arr = np.asarray(inv)
    return {
        "pnls": pnls, "inventory": inv,
        "net_pnl": float(env.cash), "sharpe": sharpe_ratio(pnls),
        "inv_std_m": float(inv_arr.std()) / 1e6,
        "inv_max_m": float(np.abs(inv_arr).max()) / 1e6,
        "n_fills": env.fills,
    }

LearningCurve (class)

Log episode rewards during training for the learning-curve plot.

class LearningCurve(BaseCallback):

    def __init__(self):
        super().__init__()
        self.rewards = []

    def _on_step(self):
        for info in self.locals.get("infos", []):
            ep = info.get("episode")
            if ep is not None:
                self.rewards.append(ep["r"])
        return True

train_ppo (function)

Train PPO. It sees only recent mid moves and its own inventory, so any discipline it shows is learned from the book it holds — it cannot anticipate a burst.

def train_ppo(train_env, total_steps):
    model = PPO("MlpPolicy", train_env, seed=SEED, device="cpu", verbose=0,
                n_steps=4096, batch_size=256, gae_lambda=0.95, gamma=0.99,
                ent_coef=0.01, learning_rate=3e-4, n_epochs=10)
    cb = LearningCurve()
    model.learn(total_timesteps=total_steps, callback=cb, progress_bar=False)
    return model, cb.rewards

main (function)

def main():
    train_env = Monitor(make_env(N_TRAIN))
    print(f"Training PPO on bursty flow (max_simultaneous={MAX_SIMULTANEOUS}) "
          f"for {TOTAL_STEPS:,} steps...")
    model, curve = train_ppo(train_env, TOTAL_STEPS)

    def ppo_act(env, obs):
        action, _ = model.predict(obs, deterministic=True)
        return action

    test_env = make_env(N_TEST)   # every policy scored on the SAME EVAL_SEED episode
    results = {
        "constant-size":      rollout(test_env, lambda e, o: constant_size_policy(e, 1.0), EVAL_SEED),
        "inventory-manager":  rollout(test_env, lambda e, o: inventory_manager_policy(e, 1.0), EVAL_SEED),
        "PPO (learned)":      rollout(test_env, ppo_act, EVAL_SEED),
    }

    print("\nHeld-out performance (same market + bursts for every policy)")
    print(f"{'policy':18s}{'net PnL':>11s}{'Sharpe':>8s}{'inv std':>9s}"
          f"{'MAX inv':>9s}{'fills':>7s}")
    for name, r in results.items():
        print(f"{name:18s}{r['net_pnl']:>11,.0f}{r['sharpe']:>8.2f}"
              f"{r['inv_std_m']:>8.1f}M{r['inv_max_m']:>8.1f}M{r['n_fills']:>7d}")

    # --- plots ---------------------------------------------------------------
    fig, axes = plt.subplots(2, 2, figsize=(13, 9))
    plot_learning_curve(curve, window=20, title="PPO learning curve (episode reward)", ax=axes[0, 0])

    ax = axes[0, 1]
    for name, r in results.items():
        ax.plot(np.asarray(r["inventory"]) / 1e6, label=name, lw=0.8)
    ax.axhline(0, color="k", lw=0.6)
    ax.set_title("Inventory path — constant-size ratchets up; managed books stay bounded")
    ax.set_xlabel("step"); ax.set_ylabel("inventory (M units)"); ax.legend(); ax.grid(alpha=0.3)

    # Inventory DISTRIBUTION — the risk lives in the tails.
    ax = axes[1, 0]
    for name, r in results.items():
        ax.hist(np.asarray(r["inventory"]) / 1e6, bins=50, histtype="step", lw=1.6, label=name)
    ax.set_yscale("log")
    ax.set_title("Inventory distribution — the naive dealer's fat tail is the risk")
    ax.set_xlabel("inventory (M units)"); ax.set_ylabel("steps (log)"); ax.legend(); ax.grid(alpha=0.3)

    # Risk/return frontier: net PnL against the worst inventory spike.
    ax = axes[1, 1]
    for name, r in results.items():
        ax.scatter(r["inv_max_m"], r["net_pnl"] / 1e3, s=90)
        ax.annotate(name, (r["inv_max_m"], r["net_pnl"] / 1e3),
                    textcoords="offset points", xytext=(6, 4), fontsize=9)
    ax.set_title("Risk/return frontier: PnL vs worst inventory spike")
    ax.set_xlabel("max |inventory| reached (M units)"); ax.set_ylabel("net PnL ($k)"); ax.grid(alpha=0.3)

    plt.tight_layout(); plt.show()

    c, im, p = results["constant-size"], results["inventory-manager"], results["PPO (learned)"]
    print(
        f"\nTakeaway: on a fair mid the danger is not the price, it is the book a "
        f"burst hands you. The constant-size dealer shows full size and never "
        f"hedges, so a run of one-sided bursts ratchets its inventory to "
        f"{c['inv_max_m']:.0f}M (Sharpe {c['sharpe']:.2f}) — a fat-tailed book "
        f"(bottom-left) that swings hard on ordinary volatility. Sizing off "
        f"inventory limits how fast you dig, but on a driftless mid it cannot get "
        f"you flat — you also have to HEDGE to work the book down. The "
        f"inventory-manager does both and holds the spike to {im['inv_max_m']:.0f}M "
        f"(Sharpe {im['sharpe']:.2f}); PPO, seeing only its own inventory, learns "
        f"to keep the book bounded at {p['inv_max_m']:.0f}M for a net "
        f"${p['net_pnl']:,.0f}. The lesson the naive dealer misses: sizing caps "
        f"intake, hedging caps the book — and with max_simultaneous="
        f"{MAX_SIMULTANEOUS} clients able to lift one quote, you need both, "
        f"because you never see the burst coming."
    )

Run the lesson

Execute everything above, then run main().

main()

Training PPO on bursty flow (max_simultaneous=13) for 300,000 steps...

Held-out performance (same market + bursts for every policy)
policy                net PnL  Sharpe  inv std  MAX inv  fills
constant-size         416,488    2.25    13.3M    45.0M   1161
inventory-manager     347,099    2.78     6.7M    31.8M   1114
PPO (learned)         150,934    1.64     2.2M    23.8M    199

Takeaway: on a fair mid the danger is not the price, it is the book a burst hands you. The constant-size dealer shows full size and never hedges, so a run of one-sided bursts ratchets its inventory to 45M (Sharpe 2.25) — a fat-tailed book (bottom-left) that swings hard on ordinary volatility. Sizing off inventory limits how fast you dig, but on a driftless mid it cannot get you flat — you also have to HEDGE to work the book down. The inventory-manager does both and holds the spike to 32M (Sharpe 2.78); PPO, seeing only its own inventory, learns to keep the book bounded at 24M for a net $150,934. The lesson the naive dealer misses: sizing caps intake, hedging caps the book — and with max_simultaneous=13 clients able to lift one quote, you need both, because you never see the burst coming.