RL-Starter: from bandits to FX trading

A hands-on reinforcement-learning course. Every lesson teaches one algorithm on a clean, intuitive toy problem, then immediately re-applies it to a trading environment of matching difficulty — so the curriculum ladders from multi-armed bandits all the way to FX algorithmic trading.

Deep-RL lessons use Gymnasium for the environment API and Stable-Baselines3 for battle-tested algorithm implementations, so the notebooks stay focused on environments, rewards, and evaluation rather than re-deriving optimisers.

The ladder

#	Lesson	Algorithm (learned on)	Trading application
L1	Bandits → allocation	ε-greedy / UCB / Thompson (k-armed bandit)	Pick among strategies; regret & exploration
L2	Contextual bandits	LinUCB / neural bandit	Features → trade signal, no inventory yet
L3	DQN on CartPole	DQN	First real MDP: delayed reward, replay, target net
L4	DQN trading	Double / Dueling DQN	`{long/flat/short}` on a synthetic series with costs
L5	Policy gradients	REINFORCE → A2C	Stochastic policies, advantage, GAE
L6	PPO trading	PPO	The workhorse on the discrete trading env
L7	Continuous sizing	SAC / PPO-continuous	Position sizing — where FX lives
C1	FX RFQ market making	PPO	Quote width + skew, inventory risk, vs Avellaneda–Stoikov
C2	Toxic flow (informed)	PPO	Adverse selection: mid drifts against you; agent widens/hedges
C3	Bursty flow (innocent)	PPO	Inventory-spike risk on a fair mid; size + hedge off inventory, no burst detection

Why trading is hard as an RL problem

Worth stating up front, because it shapes every lesson: FX has a terrible signal-to-noise ratio, is non-stationary (the “MDP” drifts under you), rewards are noisy and delayed, and transaction costs punish churn. The classic failure is backtest overfitting — an agent that memorises one price path. So the course builds not just the algorithms but the evaluation discipline: held-out data, cost modelling, and baselines you cannot beat by luck.

How to run locally

pip install -e .
python _py_to_notebook.py          # scripts -> notebooks
./run_all_notebooks.ps1            # execute on GPU, write outputs in place
quarto render                      # build the HTML site into _site/

--- title: "RL-Starter: from bandits to FX trading" --- A hands-on reinforcement-learning course. Every lesson teaches one algorithm on a clean, intuitive toy problem, then immediately re-applies it to a trading environment of matching difficulty — so the curriculum *ladders* from multi-armed bandits all the way to FX algorithmic trading. Deep-RL lessons use [Gymnasium](https://gymnasium.farama.org/) for the environment API and [Stable-Baselines3](https://stable-baselines3.readthedocs.io/) for battle-tested algorithm implementations, so the notebooks stay focused on *environments, rewards, and evaluation* rather than re-deriving optimisers. ## The ladder | # | Lesson | Algorithm (learned on) | Trading application | |---|--------|------------------------|---------------------| | L1 | Bandits → allocation | ε-greedy / UCB / Thompson (k-armed bandit) | Pick among strategies; regret & exploration | | L2 | Contextual bandits | LinUCB / neural bandit | Features → trade signal, *no inventory yet* | | L3 | DQN on CartPole | DQN | First real MDP: delayed reward, replay, target net | | L4 | DQN trading | Double / Dueling DQN | `{long/flat/short}` on a synthetic series with costs | | L5 | Policy gradients | REINFORCE → A2C | Stochastic policies, advantage, GAE | | L6 | PPO trading | PPO | The workhorse on the discrete trading env | | L7 | Continuous sizing | SAC / PPO-continuous | Position *sizing* — where FX lives | | C1 | FX RFQ market making | PPO | Quote width + skew, inventory risk, vs Avellaneda–Stoikov | | C2 | Toxic flow (informed) | PPO | Adverse selection: mid drifts against you; agent widens/hedges | | C3 | Bursty flow (innocent) | PPO | Inventory-spike risk on a fair mid; size + hedge off inventory, no burst detection | ## Why trading is hard as an RL problem Worth stating up front, because it shapes every lesson: FX has a terrible signal-to-noise ratio, is non-stationary (the "MDP" drifts under you), rewards are noisy and delayed, and transaction costs punish churn. The classic failure is *backtest overfitting* — an agent that memorises one price path. So the course builds not just the algorithms but the **evaluation discipline**: held-out data, cost modelling, and baselines you cannot beat by luck. ## How to run locally ```bash pip install -e . python _py_to_notebook.py # scripts -> notebooks ./run_all_notebooks.ps1 # execute on GPU, write outputs in place quarto render # build the HTML site into _site/ ```