Video: Your Backtesting Results Are LYING to You (Here’s Why)
Creator: Jason Graystone
Original Video: Watch the original video
Difficulty: Advanced
The Big Idea
A backtest records one sequence of trades. Its deepest drawdown describes what happened in that sequence, rather than setting a ceiling on what can happen next.
Jason Graystone’s central lesson is that traders often size positions around a historical experience that was unusually comfortable. A strategy can produce a worse losing stretch without an obvious change to its rules. The useful response is to examine alternative paths and choose a risk budget before those paths become real.
That requires separating three questions: how uncomfortable the journey could become, whether the strategy has evidence of positive expectancy, and how much money to expose. Each needs a different calculation.
Same Trades, Different Drawdown
Let 1R represent a trade’s planned initial risk. A result of +2R earns twice that amount; −1R loses the planned amount. The following original example uses six trades with identical outcomes in different orders. Assume fixed cash risk, results net of costs, and an equity curve measured at each trade close.
| Measure | Sequence A | Sequence B |
|---|---|---|
| Trade order | +2, −1, +2, −1, −1, +2 | +2, +2, +2, −1, −1, −1 |
| Cumulative R after each trade | 2, 1, 3, 2, 1, 3 | 2, 4, 6, 5, 4, 3 |
| Total return | +3R | +3R |
| Maximum drawdown | 2R | 3R |
| Longest losing streak | 2 trades | 3 trades |
Both paths finish at +3R. Sequence A falls from a peak of +3R to +1R. Sequence B falls from +6R to +3R. A trader who planned around the first path’s 2R drawdown would experience a larger decline on the second path despite having exactly the same winners and losers.
Drawdown is the decline from a running equity peak. Include the starting balance as the initial peak. Losing-streak length is a different statistic: small winners can interrupt a streak without restoring the previous equity high.
Trade-close drawdown also misses losses that develop while positions remain open. An account can approach a margin or loss limit before the trade ledger records its next closed result. Evaluate marked-to-market equity when that distinction matters.
What Monte Carlo Actually Changes
The video describes repeatedly reordering recorded trades. This is useful for asking how sensitive a strategy’s historical drawdown is to trade order. It does not create new market regimes or reveal losses absent from the original sample.
| Method | What changes? | Useful question | Important limitation |
|---|---|---|---|
| Random permutation | Order changes; every recorded trade appears once | How much can drawdown change with these same outcomes? | Fixed-risk total R and mean R remain unchanged |
| Bootstrap sampling with replacement | Some trades repeat and others are omitted | How variable might estimated mean return or a sampled path be? | Independent sampling can erase clustering and dependence |
| Block bootstrap | Groups of consecutive observations are resampled | How do results change when some local dependence is preserved? | Block length matters; unseen regimes remain unseen |
NIST’s bootstrap explanation describes sampling with replacement to estimate the variability of a statistic. That distinction matters because a shuffled ledger and a resampled ledger answer different questions.
For each simulated path, calculate maximum drawdown, longest losing streak, recovery time in trades, and whether the chosen account limit was breached. Keep the number of trades per path explicit. A drawdown estimate over 100 trades is not directly interchangeable with one over 1,000 trades.
If risk changes with equity, simulate dollars or account percentage at each step. Under constant fractional sizing with identical trade returns and no path-dependent rules, the final wealth multiplier is also unchanged by order. Drawdown still changes. Once sizing, pauses, withdrawals, or account limits depend on the path, the terminal outcome can change too.
A 95th-Percentile Drawdown Is Conditional
Graystone distinguishes a typical simulated decline, a more adverse outcome, and the worst outcome observed in the simulations. These summaries can help planning, provided their boundaries are understood.
A 95th-percentile drawdown means approximately 95% of simulated paths had drawdowns at or below that value under the chosen model, sample, and horizon. It does not establish a dependable 5% real-world chance of exceeding it.
The worst observed simulation is also not the maximum possible loss. Running more simulations may discover a worse path. A model based on ordinary historical losses cannot represent an unprecedented gap, disappearing liquidity, or a structural change in the strategy’s edge unless those risks are introduced separately.
Independent shuffling can be particularly optimistic when losses cluster. Several bad trades during the same news event may share a cause. Reshuffling them across the sample can make the strategy look easier to tolerate than it would be during another concentrated bad period.
Drawdown Confidence and Edge Confidence Are Different
The video also discusses an interval around estimated expectancy. An interval that includes zero is compatible with no positive edge under the selected statistical model. A positive lower bound is stronger evidence, but it remains dependent on the data and method.
Pure reshuffling cannot produce a meaningful distribution of mean trade expectancy. Reordering the same trades leaves their sum and average unchanged. Estimating uncertainty in the mean requires a separate method, such as bootstrap resampling with replacement, with appropriate treatment of dependence.
The transcript does not fully specify the software’s inference method, so this is a distinction to verify rather than a conclusion that its implementation is faulty. Check exactly how an “edge confidence” figure is calculated.
Even a well-calculated interval cannot repair an overfit backtest. If you searched hundreds of parameter combinations and retained the best one, ordinary uncertainty estimates on that selected sample can overstate the evidence. Test unchanged rules on held-out data, include realistic costs, and examine stability across periods.
Convert a Drawdown Budget Into Position Risk
Suppose, in an original planning example, a trader allocates $1,500 to a strategy’s drawdown budget. Its historical maximum is 8R, while a selected stress scenario is 15R. With constant cash risk per trade:
Cash risk per trade = drawdown budget ÷ stress drawdown in R.
| Sizing basis | Calculation | Cash risk per trade | Loss during a 15R decline |
|---|---|---|---|
| Historical 8R only | $1,500 ÷ 8 | $187.50 | $2,812.50 |
| Selected 15R stress scenario | $1,500 ÷ 15 | $100.00 | $1,500.00 |
This calculation translates a scenario into size; it does not guarantee the budget will hold. Extra costs, larger-than-planned losses, and a worse sequence can exceed it. Leaving a buffer below the budget is a separate decision.
Translate cash risk into tradable quantity using stop distance, instrument value, and estimated execution costs. Round down to permitted lot or contract increments. If the smallest position exceeds the budget, the proposed trade does not fit that budget. CME’s position and risk management guidance emphasizes planning exposure in relation to available capital.
For fractional sizing, use a compounded equity path. Fifteen consecutive full −1R losses at 1% of current equity lose approximately 1 − 0.9915 = 14%, rather than exactly 15%. Mixed outcomes and size changes need their own path calculation. Concurrent positions also require portfolio-level analysis because correlated losses can arrive together.
A Practical Research Workflow
- Clean the ledger. Record timestamps, instrument, setup, original planned risk, realized net result, costs, and overlapping positions. Normalize R consistently; never redefine initial risk after a loss to improve the statistics.
- Measure the observed path. Calculate expectancy, drawdown, streaks, and recovery time. Compare closed-trade and marked-to-market drawdown where possible.
- Run distinct experiments. Compare shuffled paths, independent resampling, and blocks. Use the same path length and report simulation count and assumptions.
- Test the actual sizing policy. Compare fixed cash risk with fixed fractional risk. If size changes after wins or losses, apply that exact rule throughout every simulated path.
- Stress execution separately. Increase costs and adverse fills; examine plausible gap losses and correlated exposure. Label these as assumptions rather than observed results.
- Set review triggers in advance. Define when to inspect execution, rule adherence, market conditions, and drawdown. A trigger can justify reducing or pausing exposure without proving the edge has disappeared.
The video promotes Trader Performance OS and mentions an accelerated sizing framework, but does not provide enough detail to reproduce that policy here. Any sizing system needs its complete rules tested across the same adverse paths. A “cleared to trade” label cannot establish that a strategy is suitable for an individual account.
Use the Statistics to Improve Decisions
A deeper-than-historical drawdown can reflect ordinary sequence variation. It can also reflect deteriorating execution or a changed market. Those explanations require investigation rather than an automatic assumption of eventual recovery.
Likewise, changing size emotionally after every loss can obscure what the strategy is doing. A predetermined sizing policy makes the process easier to evaluate, but consistency alone does not make an unprofitable strategy profitable.
For a useful exercise, calculate both six-trade paths above, then place the three losses at the beginning. Include the starting equity peak. You will find another 3R maximum drawdown, showing why a profitable ending does not erase the difficulty of reaching it.
Final Takeaway
Historical drawdown is one observed path. Alternative-path testing helps expose how much that path influenced your expectations, while separate inference methods assess uncertainty in expectancy. Position sizing should connect those findings to an explicit loss budget, execution assumptions, and a review policy. The goal is a process that remains manageable when the next sequence is less comfortable than the backtest.
Risk Disclosure: This material is for educational and research purposes only and is not financial or investment advice. Trading can result in substantial losses. Backtests and simulations are hypothetical, depend heavily on their assumptions, and cannot reproduce every aspect of live execution. Commissions, spreads, slippage, liquidity, gaps, order behavior, market regime changes, and discretionary decision-making can materially affect real-world results. Historical performance and simulated confidence estimates do not guarantee future outcomes.
This article is an independent educational companion to the video and is not affiliated with, sponsored by, endorsed by, or authorized by Jason Graystone.