The Structural Ceiling of Supervised Learning

Every machine learning architecture deployed across this series so far has shared a common epistemic foundation. The LightGBM ensemble for high-frequency arbitrage, the LSTM sequential model for pattern recognition, the meta-model stacked on top of both, and the purged k-fold cross-validation framework that validated them all. Each of these is a supervised learning system. They consume a vector of features, emit a prediction, and are trained against a labeled ground truth. The training objective is the minimization of prediction error against a fixed target.

This paradigm has carried the analytical stack to a substantial level of capability. It also has a structural ceiling that is now becoming visible. Supervised learning predicts. It does not act. The translation from prediction to action, in every system constructed so far, has been performed by hand-coded rule layers that sit downstream of the model output. If predicted return exceeds threshold X, allocate Y percent. If predicted volatility exceeds threshold Z, reduce position by W percent. These rules are static. They were calibrated against historical market regimes and they do not adapt when the regime shifts. The model learns. The execution logic does not.

The cost of this architectural separation is not theoretical. It is measurable in the persistent gap between backtested and live performance, the gap that this entire series has periodically returned to as a reference point. A supervised model trained on three years of bull-leaning crypto data will produce excellent predictions during a continuation of that regime and structurally degraded decisions during a regime change, because the rule layer that converts prediction to position size was calibrated against the same regime. The model and the policy are not jointly optimized. They are sequentially calibrated, which is a fundamentally weaker structural property.

Reinforcement learning collapses this separation. Instead of learning to predict and then bolting on rules, an RL agent learns the policy directly. The objective function is no longer prediction accuracy. It is cumulative risk-adjusted return measured against the agent’s own actions in the environment. The agent learns simultaneously what to predict, when its prediction matters, and how aggressively to act on it. The supervised stack is replaced by a single optimization problem, and the rule layer disappears entirely because the rules become part of what is learned.

Why PPO and Not Q-Learning

The reinforcement learning literature offers several algorithmic families, and the choice among them is not cosmetic. For portfolio allocation, the action space is continuous. The agent must emit a vector of allocation weights summing to one, with each weight a real number in a bounded range. Discrete-action algorithms such as DQN and its variants are structurally inappropriate here. Discretizing a continuous allocation space to make DQN applicable destroys the geometric structure of the problem and produces an exponential blow-up in the action dimensionality as the asset universe grows.

The relevant algorithm class is policy gradient methods, and within that class, Proximal Policy Optimization is the load-bearing default. PPO solves two failure modes that plague vanilla policy gradient methods. The first is the destructive policy update, where a single gradient step takes the policy too far from its current behavior and collapses learned performance. PPO’s clipped surrogate objective bounds the per-update policy change to a trust region, which makes training stable across the long horizons required for financial environments. The second is sample inefficiency. PPO performs multiple epochs of optimization on each batch of collected experience, extracting more signal per environment interaction than methods that discard each batch after a single update. In a financial environment where each environment step is a real or simulated market interaction, sample efficiency is not an academic concern. It is the difference between a tractable training run and one that requires months of compute.

The mathematical core of PPO is the clipped objective:

L^CLIP(θ) = E_t[ min( r_t(θ) · A_t, clip(r_t(θ), 1 − ε, 1 + ε) · A_t ) ]

Here r_t(θ) is the probability ratio between the new policy and the old policy at timestep t, A_t is the advantage estimate from generalized advantage estimation, and ε is the clip range, typically 0.2. The minimum operator over the clipped and unclipped terms is the structural mechanism that prevents catastrophic policy updates. When the advantage is positive and the new policy is making the chosen action substantially more likely, the clip caps the update. When the advantage is negative, the clip allows the policy to move freely away from the bad action. This asymmetry is what makes PPO simultaneously stable and capable of meaningful learning.

The Environment: Where the Architecture Actually Lives

The structural mistake most retail attempts at RL-based trading make is to treat the algorithm as the centerpiece. The algorithm is the easiest part. Stable Baselines 3 provides a production-quality PPO implementation in a few lines of code. The hard part, the part that determines whether the trained agent will survive contact with live markets, is the environment. The environment is where the simulation of reality lives, and an environment that misrepresents reality will produce an agent that performs spectacularly in training and disintegrates in live deployment.

A correctly specified portfolio environment must encode four structural elements with mathematical precision. First, the observation space must include not only price history but the microstructure features and sentiment vectors constructed earlier in this series. An agent trained on price alone will learn to act on price patterns alone, and will be blind to the regime signals that the rest of the analytical stack exists to provide. Second, the action space must be a continuous simplex over the asset universe plus a cash position, allowing the agent to express any valid allocation including full risk-off. Third, the reward function must be risk-adjusted, not raw return. A reward function that pays for raw return will produce an agent that maximizes leverage and dies in the first drawdown. Fourth, transaction costs must be modeled with realistic slippage and fee assumptions, because an agent trained against frictionless execution will learn to churn the portfolio at a frequency that real execution costs make catastrophically unprofitable.

The Reward Function

The single most consequential design decision is the reward function. The naive choice is portfolio return per step. This produces an agent that maximizes leverage and concentrates positions in whatever asset has recently risen. The correct choice is a differential Sharpe ratio approximation evaluated incrementally:

R_t = (ΔP_t / P_t-1) − λ · σ_rolling − κ · turnover_t

The first term is portfolio return for the step. The second term is a volatility penalty, where σ_rolling is a trailing window standard deviation of returns and λ is the risk aversion coefficient. This term punishes the agent for generating returns through volatility expansion rather than skill. The third term is a turnover penalty, where turnover_t is the L1 norm of the change in allocation weights between steps and κ is the cost coefficient. This term forces the agent to internalize transaction costs into its policy, producing portfolios that turn over only when the marginal expected return justifies the realized friction.

The choice of λ and κ is not arbitrary. They are calibrated against the actual cost structure of the deployment venue. For a Binance or Bybit perpetual futures deployment, κ should reflect the realized round-trip cost including taker fee, expected slippage at the agent’s typical order size, and a funding rate cost component. For a spot deployment with maker rebates, the calibration is different. An agent trained against the wrong cost structure will produce a policy that is structurally misaligned with its deployment venue.

Production-Grade Environment Implementation

The environment implementation below extends the Gymnasium API and is compatible with Stable Baselines 3 out of the box. It loads features from the ClickHouse warehouse constructed earlier, supports a configurable asset universe, and implements the reward function with explicit cost modeling. Type annotations, structured logging, and defensive numerical handling are integrated from the start.

import logging
from dataclasses import dataclass
from typing import Optional, Tuple, Dict, Any

import numpy as np
import gymnasium as gym
from gymnasium import spaces

logger = logging.getLogger("nql.rl.environment")
logger.setLevel(logging.INFO)

@dataclass(slots=True)
class EnvConfig:
    n_assets: int
    feature_dim: int
    lookback: int
    initial_capital: float = 100_000.0
    max_leverage: float = 1.0
    cost_per_turnover: float = 0.0006
    risk_aversion: float = 0.5
    volatility_window: int = 20
    episode_length: int = 2_048

class PortfolioAllocationEnv(gym.Env):
    metadata = {"render_modes": []}

    def __init__(
        self,
        feature_tensor: np.ndarray,
        return_tensor: np.ndarray,
        config: EnvConfig,
    ) -> None:
        super().__init__()
        if feature_tensor.shape[0] != return_tensor.shape[0]:
            raise ValueError("feature and return tensors must align on time axis")
        if feature_tensor.shape[1] != config.n_assets:
            raise ValueError("feature tensor asset dim must match config")

        self._features = feature_tensor.astype(np.float32)
        self._returns = return_tensor.astype(np.float32)
        self._config = config

        n_dims = config.n_assets + 1
        self.action_space = spaces.Box(
            low=-1.0, high=1.0, shape=(n_dims,), dtype=np.float32,
        )
        obs_dim = (config.lookback * config.n_assets * config.feature_dim) + n_dims + 1
        self.observation_space = spaces.Box(
            low=-np.inf, high=np.inf, shape=(obs_dim,), dtype=np.float32,
        )

        self._step_idx: int = 0
        self._episode_start: int = 0
        self._weights: np.ndarray = np.zeros(n_dims, dtype=np.float32)
        self._weights[-1] = 1.0
        self._equity: float = config.initial_capital
        self._return_history: list = []

    def _normalize_action(self, action: np.ndarray) -> np.ndarray:
        shifted = action - action.min()
        total = shifted.sum()
        if total <= 1e-8:
            uniform = np.ones_like(action) / len(action)
            return uniform.astype(np.float32)
        normalized = shifted / total
        normalized = normalized * self._config.max_leverage
        return normalized.astype(np.float32)

    def _build_observation(self) -> np.ndarray:
        start = self._step_idx - self._config.lookback
        window = self._features[start:self._step_idx]
        flat_window = window.flatten()
        equity_normalized = np.array(
            [self._equity / self._config.initial_capital], dtype=np.float32,
        )
        obs = np.concatenate([flat_window, self._weights, equity_normalized])
        if not np.isfinite(obs).all():
            logger.warning("non-finite observation step=%d", self._step_idx)
            obs = np.nan_to_num(obs, nan=0.0, posinf=0.0, neginf=0.0)
        return obs.astype(np.float32)

    def reset(
        self, seed: Optional[int] = None, options: Optional[Dict[str, Any]] = None,
    ) -> Tuple[np.ndarray, Dict[str, Any]]:
        super().reset(seed=seed)
        max_start = len(self._features) - self._config.episode_length - 1
        if max_start <= self._config.lookback:
            raise RuntimeError("dataset too short for configured episode length")
        self._episode_start = self.np_random.integers(
            self._config.lookback, max_start,
        )
        self._step_idx = self._episode_start
        self._weights = np.zeros(self._config.n_assets + 1, dtype=np.float32)
        self._weights[-1] = 1.0
        self._equity = self._config.initial_capital
        self._return_history = []
        return self._build_observation(), {}

    def step(
        self, action: np.ndarray,
    ) -> Tuple[np.ndarray, float, bool, bool, Dict[str, Any]]:
        new_weights = self._normalize_action(action)
        turnover = float(np.abs(new_weights - self._weights).sum())
        cost = turnover * self._config.cost_per_turnover

        asset_returns = self._returns[self._step_idx]
        portfolio_return = float(np.dot(new_weights[:-1], asset_returns))
        net_return = portfolio_return - cost

        self._equity *= (1.0 + net_return)
        self._return_history.append(net_return)
        if len(self._return_history) > self._config.volatility_window:
            self._return_history = self._return_history[-self._config.volatility_window:]

        if len(self._return_history) >= 5:
            vol = float(np.std(self._return_history, ddof=1))
        else:
            vol = 0.0
        reward = net_return - self._config.risk_aversion * vol

        self._weights = new_weights
        self._step_idx += 1

        terminated = self._equity <= self._config.initial_capital * 0.5
        truncated = (self._step_idx - self._episode_start) >= self._config.episode_length
        truncated = truncated or self._step_idx >= len(self._features) - 1

        info = {
            "equity": self._equity,
            "turnover": turnover,
            "cost": cost,
            "net_return": net_return,
            "rolling_vol": vol,
        }
        return self._build_observation(), float(reward), terminated, truncated, info

Several design decisions warrant explicit note. The action normalization uses a shifted softmax-like construction rather than a raw softmax because the latter saturates at the boundaries and produces vanishing gradients during training. The shifted construction preserves gradient flow across the action space. The episode termination on a fifty percent equity drawdown is a hard structural guardrail that prevents the agent from learning policies that survive only on long backtest windows; an agent that cannot avoid catastrophic drawdowns within a 2048-step episode is not a viable agent regardless of its terminal return. The observation includes both the recent feature window and the current weight vector, which gives the agent direct visibility into its own state and is essential for learning policies that respect transaction costs.

Training the Agent

With the environment defined, the training loop is a thin layer over Stable Baselines 3. The non-trivial decisions here are hyperparameter selection and parallelization. Financial RL training requires a substantial number of environment interactions, typically in the millions, and serial environment execution is structurally too slow. The standard pattern is vectorized environments running in parallel processes, sharing a single policy network on the GPU.

from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import SubprocVecEnv
from stable_baselines3.common.callbacks import CheckpointCallback, EvalCallback
from stable_baselines3.common.monitor import Monitor

def make_env_factory(features, returns, config, env_id: int):
    def _init():
        env = PortfolioAllocationEnv(features, returns, config)
        env = Monitor(env, filename=f"./logs/monitor_{env_id}.csv")
        return env
    return _init

def train_ppo_agent(
    train_features: np.ndarray,
    train_returns: np.ndarray,
    eval_features: np.ndarray,
    eval_returns: np.ndarray,
    config: EnvConfig,
    n_envs: int = 8,
    total_timesteps: int = 5_000_000,
) -> PPO:
    env_fns = [
        make_env_factory(train_features, train_returns, config, i)
        for i in range(n_envs)
    ]
    train_env = SubprocVecEnv(env_fns)

    eval_env = SubprocVecEnv([
        make_env_factory(eval_features, eval_returns, config, 999),
    ])

    eval_callback = EvalCallback(
        eval_env=eval_env,
        best_model_save_path="./models/ppo_best/",
        log_path="./logs/eval/",
        eval_freq=20_000,
        n_eval_episodes=10,
        deterministic=True,
    )
    checkpoint_callback = CheckpointCallback(
        save_freq=100_000,
        save_path="./models/checkpoints/",
        name_prefix="ppo_portfolio",
    )

    model = PPO(
        policy="MlpPolicy",
        env=train_env,
        learning_rate=3e-4,
        n_steps=2_048,
        batch_size=256,
        n_epochs=10,
        gamma=0.99,
        gae_lambda=0.95,
        clip_range=0.2,
        ent_coef=0.01,
        vf_coef=0.5,
        max_grad_norm=0.5,
        policy_kwargs={"net_arch": [256, 256]},
        verbose=1,
        tensorboard_log="./logs/tensorboard/",
    )

    model.learn(
        total_timesteps=total_timesteps,
        callback=[eval_callback, checkpoint_callback],
        progress_bar=True,
    )
    return model

The hyperparameters above are not arbitrary defaults. The learning rate of 3e-4 is the empirical sweet spot for PPO in continuous-control environments. The clip range of 0.2 is the standard PPO trust region. The entropy coefficient of 0.01 is critical for financial environments because it forces the agent to maintain exploration of the allocation space rather than collapsing prematurely to a deterministic policy. A premature collapse is a common failure mode in financial RL because the early-training reward signal is dominated by the volatility penalty, and an agent that has not yet learned to generate alpha will rationally collapse to all-cash to minimize the penalty term, after which it never explores its way out of that local optimum.

The Validation Imperative

An RL agent that performs well on its training data is meaningless. The only test that matters is out-of-sample performance on a temporally disjoint validation window, and even that is insufficient by itself. A complete validation protocol for a financial RL agent requires three structural elements. First, walk-forward evaluation across multiple market regimes, which means the validation windows must include at least one bull regime, one bear regime, and one sideways regime. An agent that has only been validated on one regime has not been validated. Second, transaction cost stress testing, where the cost parameter is increased by a factor of two or three during evaluation to verify that the policy remains profitable under realistic friction. A policy that survives only at the calibrated cost level is structurally fragile and will fail on any execution venue with worse fills. Third, action distribution analysis, where the actual allocation patterns produced by the trained agent are inspected for pathological behavior such as extreme concentration, excessive turnover, or refusal to use cash. A policy that mathematically performs well but exhibits structurally unreasonable behavior is not deployable, because the failure mode is unknown.

The integration with the purged k-fold cross-validation framework constructed earlier in this series is direct. Each fold becomes a separate training run with disjoint train and validation periods, and the aggregate performance across folds is the meaningful metric. A single fold result is statistical noise. The distribution across folds is signal.

The Verdict

Reinforcement learning is not a silver bullet, and its introduction does not retire the supervised stack. The supervised models continue to produce high-quality features that feed the RL observation space. The microstructure analytics continue to provide regime signals. The ClickHouse warehouse continues to deliver the data substrate. The RL agent is the policy layer that sits on top of these inputs and replaces the static rule layer that previously mediated between prediction and action.

The structural argument for this transition is that supervised learning has reached the natural ceiling of what a separation between prediction and action can achieve. Every additional unit of supervised model capacity produces a smaller marginal improvement in live performance because the bottleneck has shifted from prediction quality to policy quality. The frontier moves to where the bottleneck is, and the bottleneck is now the policy. PPO and its successors are the structural answer to that bottleneck. They produce agents that learn the joint optimization of what to observe, what to predict, and how to act, and they do so with stability properties that make production deployment tractable.

The transition from supervised prediction to learned policy is the same kind of architectural shift that the migration from PostgreSQL to ClickHouse represented at the data layer, and that the migration from cloud LLM APIs to local inference represented at the language layer. In each case, the transition is not an optimization. It is a recognition that the previous architecture has reached its structural ceiling and that further progress requires a different foundation. An institutional-grade quant stack is the cumulative result of having made each of these transitions at the correct time, neither prematurely nor belatedly. The reinforcement learning transition is the current frontier. The operators who internalize it now will compound that structural advantage across every subsequent market regime. The operators who continue to bolt rule layers onto supervised models will find that their gap to the frontier widens with each passing quarter, and that gap, once opened, does not close.