Before You Backtest: Turn a Market Hunch into a Testable Factor

“Stocks that have fallen sharply should bounce.” It sounds like a strategy. It is still only a hunch: sharply compared with what, over which interval, and when could you have acted?

This mini-course turns that sentence into a research specification. You will define a factor, separate prediction from portfolio construction, and diagnose a result that looks better than its evidence. All numbers below are invented for teaching; they are not historical returns or a suggested trade.

1. Give the hunch a mechanism and a rival

Our proposed mechanism is temporary selling pressure: some large short-term declines might reverse once that pressure fades. A rival explanation is deteriorating business prospects, in which case falling prices may continue falling. Neither story establishes which explanation dominates.

A useful hypothesis is narrower: Within a predefined liquid-stock universe, a larger five-session decline predicts a higher return over the next five sessions. Decide the universe, sample dates and minimum data history before inspecting results. Record a failure condition: no stable ordering of future returns across signal groups, or an apparent advantage that disappears under plausible costs.

A factor here means a measurable characteristic used to rank observations. It does not automatically mean a priced risk factor or an independent source of alpha. S&P Dow Jones Indices’ factor compendium explains why benchmark exposure and stock-selection skill need to be separated.

2. Fix the information clock

Let P be a consistently adjusted closing-price series. At the end of session t, calculate:

five_session_return = P[t] / P[t-5] - 1
reversal_score = -five_session_return

The negative sign makes a bigger fall a higher score. This is a definition, not evidence of predictive power. A stock rising 5% receives -0.05; a stock falling 5% receives +0.05.

The closing price at t becomes known only after that close. Our proposed test therefore enters at the next session’s open, t+1, and measures through the close of t+5. That is one explicit holding convention; changing it creates a different experiment. It also means the score window and outcome window use different price endpoints.

For real data, document exchange calendars, missing opens, dividends, splits and delistings. Use point-in-time universe membership, rather than only today’s surviving stocks. Check how the vendor adjusts historical prices and whether any inputs require information unavailable at the decision time. A clean-looking spreadsheet can still contain hindsight.

3. Calculate one small example

Here are four fictional stocks observed at the same decision time. The final column uses the next open and later close specified above, expressed as a percentage.

The top half by score is A and B. Its equal-weight average outcome is (2% - 1%)/2 = 0.5%. The bottom half, C and D, averages (1% + 3%)/2 = 2.0%. The high-minus-low difference is -1.5 percentage points.

On this observation, the ranking goes against our hypothesis. Looking only at A’s bounce would hide that. The difference is a diagnostic of the score’s ordering; it is not a realized long-short return with financing, borrow fees and execution included. Four stocks on one date cannot establish statistical significance.

4. Test a prediction before optimizing a portfolio

Across many decision dates, group stocks by score and compare subsequent outcomes. Report the number of observations, mean, median and dispersion. Examine whether a few outliers create the average and whether the ordering changes across subperiods or sectors.

Repeated daily five-session outcomes overlap. They share market moves, so treating every row as independent can exaggerate certainty. Non-overlapping sampling is a simple first diagnostic; formal inference should account for dependence. A result that survives this check still needs a comparison with relevant market and sector exposures.

Keep portfolio decisions separate. Equal weights, volatility scaling, sector constraints and turnover controls answer implementation questions. Trying every version and reporting only the strongest curve turns a modest experiment into a search over many experiments.

Bailey and co-authors’ backtest-overfitting paper explains why selection among alternatives can produce misleading simulated performance. A reserved later sample helps assess a frozen specification, but it is not a cure if you repeatedly inspect it and revise the model.

5. Write the execution bill

Suppose a later, entirely hypothetical portfolio test produces a gross 0.20% per holding period. If the assumed combined entry-and-exit cost is 0.25% of the full position value, the simplified net result is -0.05%, before financing or borrow costs. Cost conventions matter: state whether figures are one-way, round-trip, or applied to traded turnover.

The SEC’s order-type bulletin notes that market-order prices are not guaranteed and limit orders may not execute. Consequently, a closing-price signal is not a promise of a closing-price fill. Include spreads, fees, slippage, capacity and, for any short positions, availability and borrow assumptions.

Exercise: find the hidden revision

A researcher calculates the score using Friday’s close, assumes execution at that same close, tests five different lookbacks, and presents only the winner. On one toy date, the two highest-score stocks return +4% and -2%; the two lowest-score stocks return +1% and +1%. Combined entry-and-exit costs are assumed to be 0.30% for a fully invested top-half portfolio.

Calculate the high-minus-low difference and the top-half net return. Identify two problems that the arithmetic does not fix.

Worked solution

The high-score group averages 1%. The low-score group also averages 1%, so the difference is zero. The simplified top-half net return is 0.70%, but that positive number does not show that the score beat the comparison group.

Using the completed Friday close to obtain a fill at that same close needs an executable, timestamp-consistent process; our proposed design instead enters at the next open. Selecting the best of five lookbacks also introduces selection bias. Report all trials and assess a frozen choice on later data. Correct arithmetic cannot repair either problem.

Your final deliverable before a full backtest is a one-page specification: mechanism, rival, universe, formula, decision time, entry and exit, comparison, costs, sample split and failure condition. That document gives the eventual chart something concrete to answer.