import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from stable_baselines3 import PPO
from stable_baselines3.common.monitor import Monitor
from stable_baselines3.common.callbacks import BaseCallback
from common.fx_market_making import (
generate_market, generate_requests, FXMarketMakingEnv,
fixed_spread_policy, inventory_skew_policy, PIP,
)
from common import sharpe_ratio
SEED = 0
N_TRAIN = 8000
N_TEST = 4000
TOTAL_STEPS = 300_000Capstone — FX RFQ market making: learning to quote
Everything in Track 2 was building to this. We are a dealer streaming prices in EUR/USD. Clients send us Requests (“I want to trade 5M, what’s your price?”); we answer with a quote; the ones they accept become Fills — our trades. We never chose a direction; the client did. What we control is how we price, and the risk we take on as a result.
The dataset, three interleaved step-indexed streams
- Market — a level-1 book (bid/ask/sizes) whose mid random-walks. Our reference. Steps, not clock time.
- Requests — client RFQs:
side(their direction),qty. - Fills — the quotes they lifted. No partial fills: all-or-nothing.
The decision: quote width and skew
Each RFQ, the agent picks two numbers:
- width — the half-spread it charges. Wide earns more per fill but wins fewer; tight wins flow but on a thin edge.
- skew — a directional shading. Long and a client wants to buy (flattening us)? Skew tight to win it. A trade that piles on more inventory? Skew wide.
Why this is a real MDP, and genuinely hard
Inventory is the state. Every fill leaves us holding risk that marks to market as the mid moves — and on a driftless mid, that PnL has zero mean but large variance. So the objective is not “earn spread”; it is “earn spread per unit of inventory risk”. The agent must learn the Avellaneda–Stoikov instinct — skew to offload, widen when nervous — from reward alone.
We train PPO (our Track 2 workhorse) on synthetic data, then judge it on a held-out market against two baselines: a fixed spread, and a hand-crafted inventory-skewing dealer. Beating the skewing baseline on risk-adjusted PnL is the bar.
Plugging in real data. The env consumes a
marketDataFrame (step, bid_px, ask_px, bid_sz, ask_sz, mid) and arequestsDataFrame (step, req_id, client_id, side, qty). Swap the synthetic generators for your own files with those columns and everything below runs unchanged.
Run order matters. The cells below build on each other. Run them top to bottom (Jupyter: Run → Run All Cells; VS Code: Run All). If you hit
NameError(or similar), you skipped the Setup cell — run it first.
Setup — imports & configuration
Run this cell first. It imports every library and defines the module-level constants the rest of the notebook relies on.
show_sample_data (function)
Print a small slice of the three streams so the schema is concrete.
We generate a short market, a few RFQs, run a fixed-spread dealer over it, and show the fills its quotes produced. This is exactly the data shape the environment consumes — and the shape your real files should have.
def show_sample_data():
rng = np.random.default_rng(42)
mkt = generate_market(12, rng, vol_pips=0.5)
req = generate_requests(mkt, rng, arrival_rate=0.7)
# window=1 so even the earliest RFQs are quoted; a tight 0.3-pip quote wins
# most of them, so the fills table is populated for illustration.
env = FXMarketMakingEnv(mkt, req, window=1)
env.reset()
done = False
while not done:
_, _, term, trunc, _ = env.step(fixed_spread_policy(env, 0.3, 0.0))
done = term or trunc
print("MARKET (level-1 book we quote off)")
print(mkt.to_string(index=False))
print("\nREQUESTS (client RFQs)")
print(req.to_string(index=False))
print("\nFILLS (quotes the client accepted = our trades)")
print(pd.DataFrame(env.fills).to_string(index=False) if env.fills else "(none)")make_env (function)
Build a market-making env from a fresh synthetic market + RFQ stream.
A different seed gives a different price path and client flow, which is how we get a genuinely held-out test set — the single most important guard against fooling ourselves in trading RL.
def make_env(seed, n):
rng = np.random.default_rng(seed)
market = generate_market(n, rng)
requests = generate_requests(rng=rng, market=market)
return FXMarketMakingEnv(market, requests)rollout (function)
Run one full pass of env under an action function; collect traces.
Returns per-step PnL (edge + mark-to-market), the inventory path, and the width/skew the agent quoted — enough to score it and to see whether it learned to skew with inventory.
def rollout(env, act_fn):
obs, _ = env.reset()
pnls, inv_path, widths, skews = [], [], [], []
done = False
while not done:
action = act_fn(env, obs)
widths.append(env.min_hs + (np.clip(action[0], -1, 1) + 1) / 2 * (env.max_hs - env.min_hs))
skews.append(np.clip(action[1], -1, 1) * env.max_skew)
obs, reward, term, trunc, info = env.step(action)
pnls.append(info["edge_usd"] + info["inv_mtm"])
inv_path.append(info["inventory"])
done = term or trunc
return {
"pnls": pnls, "inventory": inv_path, "widths": widths, "skews": skews,
"cash_pnl": float(env.cash), "n_fills": len(env.fills),
"inv_std_m": float(np.std(inv_path)) / 1e6,
"sharpe": sharpe_ratio(pnls),
}LearningCurve (class)
Log each episode’s reward during training for a learning-curve plot.
class LearningCurve(BaseCallback):
def __init__(self):
super().__init__()
self.rewards = []
def _on_step(self):
for info in self.locals.get("infos", []):
ep = info.get("episode")
if ep is not None:
self.rewards.append(ep["r"])
return Truetrain_ppo (function)
Train PPO on the market-making env.
PPO is on-policy and robust — a sensible default when the reward is as noisy as inventory mark-to-market makes this one. We give it a slightly larger rollout (n_steps) so each update sees many RFQs and inventory cycles.
def train_ppo(train_env, total_steps):
model = PPO(
"MlpPolicy", train_env, seed=SEED, device="cpu", verbose=0,
n_steps=4096, batch_size=256, gae_lambda=0.95, gamma=0.99,
ent_coef=0.005, learning_rate=3e-4, n_epochs=10,
)
cb = LearningCurve()
model.learn(total_timesteps=total_steps, callback=cb, progress_bar=False)
return model, cb.rewardsmain (function)
def main():
print("=" * 68, "\nA sample of the FX RFQ dataset\n", "=" * 68, sep="")
show_sample_data()
# --- train on one market, hold out another --------------------------------
train_env = Monitor(make_env(SEED, N_TRAIN))
print(f"\nTraining PPO for {TOTAL_STEPS:,} steps on the market-making env...")
model, curve = train_ppo(train_env, TOTAL_STEPS)
test_env = make_env(SEED + 999, N_TEST)
# --- score PPO and the two baselines on the SAME held-out market ----------
def ppo_act(env, obs):
action, _ = model.predict(obs, deterministic=True)
return action
results = {
"PPO (learned)": rollout(test_env, ppo_act),
"fixed 1.0 pip": rollout(test_env, lambda e, o: fixed_spread_policy(e, 1.0, 0.0)),
"inventory-skew (AS)": rollout(test_env, lambda e, o: inventory_skew_policy(e, 1.0)),
}
print("\nHeld-out performance (higher cash & Sharpe, lower inv-risk = better)")
print(f"{'policy':22s}{'cash PnL':>12s}{'fills':>8s}{'inv std':>10s}{'Sharpe':>9s}")
for name, r in results.items():
print(f"{name:22s}{r['cash_pnl']:>12,.0f}{r['n_fills']:>8d}"
f"{r['inv_std_m']:>9.1f}M{r['sharpe']:>9.2f}")
# --- plots ----------------------------------------------------------------
fig, axes = plt.subplots(2, 2, figsize=(13, 9))
from common import plot_learning_curve
plot_learning_curve(curve, window=20, title="PPO learning curve (episode reward)", ax=axes[0, 0])
ax = axes[0, 1]
for name, r in results.items():
ax.plot(np.cumsum(r["pnls"]) / 1e3, label=name)
ax.axhline(0, color="k", lw=0.6); ax.set_title("Cumulative PnL, held-out ($k)")
ax.set_xlabel("step"); ax.set_ylabel("cum PnL ($k)"); ax.legend(); ax.grid(alpha=0.3)
ax = axes[1, 0]
for name, r in results.items():
ax.plot(np.asarray(r["inventory"]) / 1e6, label=name, lw=0.9)
ax.axhline(0, color="k", lw=0.6); ax.set_title("Inventory path (lower variance = better risk control)")
ax.set_xlabel("step"); ax.set_ylabel("inventory (M units)"); ax.legend(); ax.grid(alpha=0.3)
ax = axes[1, 1]
ppo = results["PPO (learned)"]
inv_m = np.asarray(ppo["inventory"]) / 1e6
ax.scatter(inv_m, ppo["skews"], s=4, alpha=0.3)
ax.set_title("Did PPO learn to skew? (quoted skew vs inventory)")
ax.set_xlabel("inventory (M units)"); ax.set_ylabel("quoted skew (pips)"); ax.grid(alpha=0.3)
plt.tight_layout(); plt.show()
ppo, as_, fx = results["PPO (learned)"], results["inventory-skew (AS)"], results["fixed 1.0 pip"]
print(
f"\nTakeaway: the bar is Avellaneda-Stoikov — a hand-built dealer that "
f"skews with inventory (Sharpe {as_['sharpe']:.2f}). PPO reached Sharpe "
f"{ppo['sharpe']:.2f}, essentially matching it, and did so holding LESS "
f"risk (inventory std {ppo['inv_std_m']:.1f}M vs {as_['inv_std_m']:.1f}M) "
f"while the naive fixed-spread dealer languishes at {fx['sharpe']:.2f} with "
f"a reckless {fx['inv_std_m']:.0f}M inventory swing. PPO earns less gross "
f"cash than the fixed dealer precisely because it declines the toxic flow "
f"that would load it up — the bottom-right scatter shows it learned to skew "
f"quotes against its inventory purely from reward, never told the AS "
f"formula. That is the whole point: RL rediscovers the market-making "
f"instinct from data, so on REAL flow — where AS's Brownian/Poisson "
f"assumptions break — it has room to do better than the closed form. Swap "
f"the synthetic generators in make_env() for your market/requests files "
f"(same columns) to run this on live data."
)Run the lesson
Execute everything above, then run main().
main()====================================================================
A sample of the FX RFQ dataset
====================================================================
MARKET (level-1 book we quote off)
step bid_px ask_px bid_sz ask_sz mid
0 1.084985 1.085045 6000000 7000000 1.085015
1 1.084933 1.084993 5000000 6000000 1.084963
2 1.084971 1.085031 3000000 2000000 1.085001
3 1.085018 1.085078 6000000 5000000 1.085048
4 1.084920 1.084980 4000000 2000000 1.084950
5 1.084855 1.084915 4000000 6000000 1.084885
6 1.084862 1.084922 4000000 5000000 1.084892
7 1.084846 1.084906 2000000 3000000 1.084876
8 1.084845 1.084905 1000000 1000000 1.084875
9 1.084802 1.084862 4000000 7000000 1.084832
10 1.084846 1.084906 7000000 4000000 1.084876
11 1.084885 1.084945 1000000 7000000 1.084915
REQUESTS (client RFQs)
step req_id client_id side qty
1 RFQ000001 BANK_X BUY 3000000
2 RFQ000002 ACME_FUND SELL 5000000
5 RFQ000005 HEDGE_Y BUY 1000000
6 RFQ000006 BANK_X BUY 1000000
7 RFQ000007 BANK_X BUY 3000000
8 RFQ000008 RETAIL_AGG BUY 10000000
10 RFQ000010 HEDGE_Y SELL 3000000
11 RFQ000011 HEDGE_Y SELL 2000000
FILLS (quotes the client accepted = our trades)
step req_id client_id side qty markup_pips edge_usd inventory_after
1 RFQ000001 BANK_X BUY 3000000 0.3 90.000006 -3000000.0
5 RFQ000005 HEDGE_Y BUY 1000000 0.3 30.000002 -4000000.0
6 RFQ000006 BANK_X BUY 1000000 0.3 30.000002 -5000000.0
8 RFQ000008 RETAIL_AGG BUY 10000000 0.3 300.000021 -15000000.0
10 RFQ000010 HEDGE_Y SELL 3000000 0.3 90.000006 -12000000.0
Training PPO for 300,000 steps on the market-making env...
Held-out performance (higher cash & Sharpe, lower inv-risk = better)
policy cash PnL fills inv std Sharpe
PPO (learned) 125,865 409 4.6M 1.85
fixed 1.0 pip 200,600 514 50.1M 0.20
inventory-skew (AS) 197,126 533 10.0M 1.64
Takeaway: the bar is Avellaneda-Stoikov — a hand-built dealer that skews with inventory (Sharpe 1.64). PPO reached Sharpe 1.85, essentially matching it, and did so holding LESS risk (inventory std 4.6M vs 10.0M) while the naive fixed-spread dealer languishes at 0.20 with a reckless 50M inventory swing. PPO earns less gross cash than the fixed dealer precisely because it declines the toxic flow that would load it up — the bottom-right scatter shows it learned to skew quotes against its inventory purely from reward, never told the AS formula. That is the whole point: RL rediscovers the market-making instinct from data, so on REAL flow — where AS's Brownian/Poisson assumptions break — it has room to do better than the closed form. Swap the synthetic generators in make_env() for your market/requests files (same columns) to run this on live data.