Backtest a Betting Model Against Historical Odds Data

Almost every betting model that looks profitable in a backtest is not. Not because the modeller is dishonest, but because backtesting betting markets contains a set of traps that each independently manufacture edge out of nothing — and most people hit several at once.

This guide builds a backtest properly. It covers data assembly, the four biases that will fake an edge for you, realistic execution modelling, walk-forward validation, and how to report a result honestly enough that you would bet real money on it.

What a backtest actually has to prove

Not "my model picks winners". A model that picks 70% winners at $1.20 loses money. The claim you are testing is narrower:

At the prices I could actually have taken, at the moments I could actually have taken them, my selections returned more than they cost, by a margin larger than chance.

Every word in that sentence corresponds to a way backtests go wrong.

Step 1: assemble the data

You need odds and results joined on a stable event identifier. Getting the join right is 90% of the work.

# fetch.py
import os
import httpx
import pandas as pd

BASE = "https://krokodds.com.au/api/v1"
HEADERS = {"X-API-Key": os.environ["KROK_API_KEY"]}


def fetch_odds_history(sport: str, limit: int = 1000) -> pd.DataFrame:
    r = httpx.get(f"{BASE}/odds-history",
                  params={"sport_key": sport, "limit": limit,
                          "include_snapshots": "true"},
                  headers=HEADERS, timeout=60)
    r.raise_for_status()

    rows = []
    for rec in r.json()["data"]:
        for snap in rec.get("snapshots", []):
            rows.append({
                "event_id":    rec["event_id"],
                "sport":       rec.get("sport_key", sport),
                "commence":    pd.to_datetime(rec["commence_time"], utc=True),
                "market":      snap["market"],
                "selection":   snap["selection"],
                "bookmaker":   snap["bookmaker"],
                "price":       float(snap["price"]),
                "captured_at": pd.to_datetime(snap["captured_at"], utc=True),
            })
    return pd.DataFrame(rows)


def fetch_results(sport: str, limit: int = 1000) -> pd.DataFrame:
    r = httpx.get(f"{BASE}/results",
                  params={"sport_key": sport, "limit": limit},
                  headers=HEADERS, timeout=60)
    r.raise_for_status()
    return pd.DataFrame([{
        "event_id":   x["event_id"],
        "home_score": x.get("home_score"),
        "away_score": x.get("away_score"),
        "winner":     x.get("winner"),
        "settled_at": pd.to_datetime(x["settled_at"], utc=True),
    } for x in r.json()["data"]])

Validate the join before you do anything else

odds = fetch_odds_history("aussierules_afl", limit=5000)
results = fetch_results("aussierules_afl", limit=5000)

merged = odds.merge(results, on="event_id", how="left", indicator=True)

unmatched = merged[merged["_merge"] == "left_only"]["event_id"].nunique()
total = merged["event_id"].nunique()
print(f"unmatched events: {unmatched}/{total} ({unmatched / total:.1%})")

assert unmatched / total < 0.05, "join rate too low — fix identifiers first"

That assertion is not decoration. A silent 30% join failure does not produce an error; it produces a backtest on a biased subsample. And the subsample is rarely random — missing results skew toward postponed, abandoned and rescheduled fixtures, which are exactly the events where odds behaved unusually.

Step 2: the four biases that fake an edge

Stack all four and a model with genuinely zero edge can backtest at +15% ROI. This is not hypothetical — it is the default outcome of a naive implementation.

Step 3: build a point-in-time price view

The core primitive: given an event and a moment, what prices were available?

import pandas as pd


def prices_as_of(odds: pd.DataFrame, event_id: str, market: str,
                 as_of: pd.Timestamp) -> pd.DataFrame:
    """Latest price per (selection, bookmaker) strictly before as_of."""
    sub = odds[
        (odds["event_id"] == event_id)
        & (odds["market"] == market)
        & (odds["captured_at"] < as_of)
    ]
    if sub.empty:
        return sub
    return (sub.sort_values("captured_at")
               .groupby(["selection", "bookmaker"], as_index=False)
               .last())

The strict < matters. Using <= lets a price captured in the same millisecond as your decision leak in, which sounds pedantic until you realise many feeds timestamp on the minute and you have just given yourself the whole minute.

Simulating when you would have bet

BET_LEAD = pd.Timedelta(hours=4)   # bet 4h before jump

def bet_time(commence: pd.Timestamp) -> pd.Timestamp:
    return commence - BET_LEAD

Pick a lead time and hold it constant. "Bet whenever the price was best" is not a strategy you can execute — it requires knowing the future — and it is the single most common way an otherwise careful backtest becomes fiction.

Step 4: execution model

Three scenarios, all worth reporting.

def execute(prices: pd.DataFrame, selection: str, mode: str,
            book: str | None = None) -> tuple[float, str] | None:
    """Return (price, bookmaker) you would realistically have got."""
    sub = prices[prices["selection"] == selection]
    if sub.empty:
        return None

    if mode == "best":                 # optimistic: every account, always top price
        row = sub.loc[sub["price"].idxmax()]
    elif mode == "single":             # pessimistic: one book only
        one = sub[sub["bookmaker"] == book]
        if one.empty:
            return None
        row = one.iloc[0]
    elif mode == "second_best":        # realistic: you missed the outlier
        ranked = sub.sort_values("price", ascending=False)
        row = ranked.iloc[1] if len(ranked) > 1 else ranked.iloc[0]
    else:
        raise ValueError(mode)

    return float(row["price"]), str(row["bookmaker"])

The second_best mode is underrated. In practice the single best price on a market is frequently either a stale quote from a broken feed, a book that will limit you immediately, or a price that vanished before you could click. Second-best is a decent proxy for what a real punter with a good set of accounts actually achieves.

Step 5: the backtest loop

from dataclasses import dataclass


@dataclass
class Bet:
    event_id: str
    selection: str
    price: float
    bookmaker: str
    stake: float
    model_prob: float
    won: bool | None = None

    @property
    def edge(self) -> float:
        return self.model_prob * self.price - 1

    @property
    def pnl(self) -> float:
        if self.won is None:
            return 0.0
        return self.stake * (self.price - 1) if self.won else -self.stake


def run_backtest(odds, results, model, *, mode="second_best",
                 book=None, min_edge=0.03, stake=1.0) -> list[Bet]:
    bets: list[Bet] = []
    events = odds[["event_id", "commence"]].drop_duplicates()

    for _, ev in events.sort_values("commence").iterrows():
        as_of = bet_time(ev["commence"])
        prices = prices_as_of(odds, ev["event_id"], "h2h", as_of)
        if prices.empty:
            continue

        # Model sees ONLY data available at as_of.
        probs = model.predict(ev["event_id"], as_of)
        if probs is None:
            continue

        for selection, p in probs.items():
            got = execute(prices, selection, mode, book)
            if got is None:
                continue
            price, bm = got
            if p * price - 1 < min_edge:
                continue

            res = results[results["event_id"] == ev["event_id"]]
            won = None if res.empty else bool(res.iloc[0]["winner"] == selection)
            bets.append(Bet(ev["event_id"], selection, price, bm, stake, p, won))

    return bets

Step 6: report it honestly

A single ROI number is not a result. Report the interval.

import numpy as np


def summarise(bets: list[Bet], bootstrap: int = 10_000) -> dict:
    settled = [b for b in bets if b.won is not None]
    if not settled:
        return {}

    pnl = np.array([b.pnl for b in settled])
    turnover = sum(b.stake for b in settled)
    roi = pnl.sum() / turnover

    # Bootstrap CI — betting P&L is far too skewed for a normal approximation.
    rng = np.random.default_rng(42)
    boot = np.array([
        rng.choice(pnl, size=len(pnl), replace=True).sum() / turnover
        for _ in range(bootstrap)
    ])

    return {
        "bets": len(settled),
        "turnover": turnover,
        "profit": float(pnl.sum()),
        "roi": float(roi),
        "roi_ci_95": (float(np.percentile(boot, 2.5)),
                      float(np.percentile(boot, 97.5))),
        "strike_rate": float(np.mean([b.won for b in settled])),
        "avg_price": float(np.mean([b.price for b in settled])),
        "max_drawdown": float(_max_dd(pnl)),
    }


def _max_dd(pnl: np.ndarray) -> float:
    equity = np.cumsum(pnl)
    peak = np.maximum.accumulate(equity)
    return float((peak - equity).max())

Here is why the confidence interval is non-negotiable, at a typical 55% strike rate on even-money style prices:

Note the last row. A genuine 2% edge — which is a very good result in Australian markets — is not statistically distinguishable from zero at 1,000 bets. That is the reality of the signal-to-noise ratio in betting, and it is why closing line value is used as a leading indicator: it converges far faster than profit does.

Step 7: walk-forward validation

A single train/test split lets you tune until the test set looks good. Walk-forward does not.

def walk_forward(odds, results, model_factory,
                 train_days=180, test_days=30) -> list[dict]:
    start = odds["commence"].min()
    end = odds["commence"].max()
    out = []

    cursor = start + pd.Timedelta(days=train_days)
    while cursor + pd.Timedelta(days=test_days) <= end:
        train_mask = (odds["commence"] >= cursor - pd.Timedelta(days=train_days)) \
                   & (odds["commence"] < cursor)
        test_mask = (odds["commence"] >= cursor) \
                  & (odds["commence"] < cursor + pd.Timedelta(days=test_days))

        model = model_factory()
        model.fit(odds[train_mask], results)

        fold = summarise(run_backtest(odds[test_mask], results, model))
        fold["window_start"] = cursor
        out.append(fold)

        cursor += pd.Timedelta(days=test_days)
    return out

What you are looking for is consistency across folds, not a high average. A model returning +3%, +2%, +4%, +1%, +3% across five folds is far more trustworthy than one returning +22%, −6%, +1%, −3%, +19% at the same mean. The second is a model that found one exploitable period and nothing else.

Step 8: the sanity checks

Before you believe any positive result, run these:

• Shuffle the results. Randomly permute the outcome column and re-run. If the model still shows edge, you have a leak.
• Bet the opposite. Invert every selection. You should get roughly the negative of your edge minus double the margin. Anything else means an asymmetry bug.
• Check edge by price band. Edge concentrated entirely in $15+ selections is almost always a data artefact, because longshot prices are the noisiest and the most likely to be stale.
• Check edge by bookmaker. If all your profit comes from one book, ask whether that book's feed had a capture bug rather than whether they price badly.
• Check edge over time. An edge that disappeared eighteen months ago is a historical fact, not a strategy.

# Leakage check — the most important twenty lines in the whole file.
shuffled = results.copy()
shuffled["winner"] = shuffled["winner"].sample(frac=1, random_state=0).values
placebo = summarise(run_backtest(odds, shuffled, model))
print(f"placebo ROI: {placebo['roi']:+.2%}  (expect ≈ −margin)")

From backtest to deployment

A backtest that survives all of the above still is not a live edge. Two things change on deployment:

• You get restricted. The books offering the prices your model likes will limit you, often within a few hundred bets. Model this by re-running with your best books removed and seeing whether the edge survives.
• The market adapts. Any inefficiency large enough to find in historical data is visible to others. Expect measured edge to decay, and re-fit on a schedule.

The most reliable bridge between backtest and live is closing line value. Track it from your first live bet — if your model's selections consistently beat the close, the backtest is telling the truth about the market even before the profit arrives.

The historical archive behind all of this is available through the Krok Odds API, with captured per-bookmaker odds, settled results, and Betfair closing prices for racing. The backtest tool runs the same methodology in the browser if you want a reference result to compare your own implementation against.

About the author

David Koval

Senior Advantage Betting Contributor

David contributes long-form retrospectives, case studies, and operational pieces on AU advantage betting markets. His writing focuses on the realities of running a sustainable AU advantage operation — what works, what fails, and the operational details most blogs gloss over.