A backtest is an experiment run on data that has already happened. The rule is fixed, the series is fixed, the costs are assumed, and the engine reports what the rule would have done. That is a computation, and it is worth doing: it turns a hand-waved idea into a set of trades with dates, prices and costs attached, and a set of trades can be checked.
The trouble is that the same computation will produce a pleasant answer for a rule with no content at all, if the experiment is built carelessly. Five defects do most of the damage, and none of them requires bad faith. A test that executes on the bar whose close generated the signal, a universe assembled from today’s listings, a search over two hundred variations of the rule, a cost assumption of zero, and a single four-year window will together turn noise into something that looks like a finding. Each defect is arithmetic. Each has a check. Each can be reversed, and the reversal is where the useful work is.
This piece does five things: it names the five defects and what each does to a number, it puts them in a table with the check that catches each one, it audits one illustrative test line by line so that the corrections add up, it sets out what to do instead, and it ends with the sentence that no amount of care removes: a clean backtest is not evidence of future returns. There is no strategy here and no result presented as evidence that a method works.
What a backtest is, and what it is not
A backtest answers one question: given this rule, this series and these costs, what would the arithmetic have produced? Everything in it is an assumption except the prices, and even the prices are a choice — a venue, a session, an adjustment for splits, a treatment of the roll in a futures series.
What a backtest is not is a forecast. It is one path through one sample under one set of assumptions, and the number it reports is a description of that path. The gap between a description of the past and a statement about the future is where nearly every overfitting error begins, because the second sentence is easier to write than the first and reads better.
An engine can enforce part of the discipline. The backtest operation on the trade service takes a declarative rule, executes on the next bar’s open rather than the signal bar’s close, uses whole-share accounting, marks to market daily, and accepts an optional cost model. Those four choices remove several defects by construction and make others visible: if the run used the next bar’s open, the number no longer contains the fill-at-close assumption. The rest of the discipline is the analyst’s, and it is mostly a matter of what gets written down before the run starts.
Look-ahead: the test that knows the answer
Look-ahead is any use of information in the test that would not have been available at the moment the decision was made. The crudest form is a signal computed from a bar’s close and executed at that same close: the rule learns the answer and then buys the answer. It is also the most common, because it is the natural way to write the loop.
The consequence is measurable, and the size of it is the size of the move between the signal bar’s close and the next bar’s open. A rule that detects momentum fires after the move it detects, so a fill at the signal close captures part of the gap that a fill at the next open does not. In the audit below that difference is 0.035 per cent of notional per round trip, which on 200 round trips is 7.00 percentage points of cumulative return over four years — 1.75 points a year — and it is the smallest of the five defects in that example.
Look-ahead has quieter forms. A universe defined by today’s index membership uses knowledge of which companies survived. A fundamental series restated after an accounting change uses figures nobody had at the time. A volatility filter calibrated on the full sample and then applied from the first bar uses the future to decide the past. A dividend-adjusted price series reconstructs history as of today’s adjustment factors. In each case the test is not wrong about the arithmetic; it is answering a question that could only be asked after the fact.
The check is mechanical. For every input, ask whether the value was knowable on the date the decision is stamped with. For prices, that means the next bar’s open. For fundamentals, the filing date rather than the period end, plus a publication lag. For the universe, a point-in-time membership file. For a series, the version as it existed, not the current adjustment. Where any of those cannot be established, the run reports the assumption instead of burying it.
Survivorship: the sample assembled afterwards
A universe drawn from today’s listings excludes everything that failed and everything that was acquired. It is a sample selected on an outcome — the outcome being that the instrument was still there to be selected. The bias is not subtle: the excluded names are, by construction, the ones that did badly enough to disappear.
The mechanism is worth stating in units, because it is not a rounding error. In the audit below the universe is 120 names as listed at the end of the window; 18 of them — 15.0 per cent — were delisted or absorbed during it and were therefore absent from the test. Those 18 carried 32 of the 200 round trips, and the rule’s mean on those trades was −0.55 per cent. Restoring them subtracts 17.60 percentage points of cumulative return: 32 × 0.55. That is 4.40 points a year, and it is the largest of the defects in that example.
The check is a point-in-time universe: the names that were listed and tradable on each date, including the ones that left. Where a survivorship-free file is unavailable, the run can still state the exclusion, and a test that says “18 of 120 names were dropped because they delisted” is honest in a way that a clean-looking equity curve is not. The related defect is a universe that is defined by the same data the rule sorts on — a screen on the last five years of returns, run on the last five years of returns.
Parameter search: the winner of two hundred tries
Every rule has settings. A moving average has a length, a threshold has a level, a stop has a multiple, an exit has a holding period. Choosing those settings by finding the combination that scored best on the sample is not fitting a model; it is selecting the maximum of a set of random variables, and the maximum of a large set of random variables is large whatever the underlying distribution.
The arithmetic is standard, and it is the most useful number in this piece. If a test statistic is roughly normal under the null and the trials are independent, the best of n trials has an expected value of about the square root of twice the natural log of n. Ten variants: 2.15. Fifty: 2.80. Two hundred: 3.26. A thousand: 3.72. Meanwhile the threshold that keeps the family-wise error at 5 per cent across all n trials — the Bonferroni threshold, the point at which the best result is unlikely to be chance — is 1.96 for one trial, 3.29 for fifty, 3.66 for two hundred and 4.06 for a thousand.
Read the two columns together and the consequence is blunt: a search over two hundred variants is expected to produce a winner at a statistic of 3.26 by chance alone, and the threshold for treating that winner as something other than chance is 3.66. A statistic of 2.9 from that search is below the expected maximum of the search, which is to say it is less impressive than the noise it was selected from. Any test that reports a statistic of 2.9 without reporting the two hundred variants behind it has reported the wrong number.
The check has two parts, and the second is the one usually skipped. Report the number of variants tested — all of them, including the ones that were abandoned quietly — and then deflate the result by that count rather than comparing it with the threshold for a single test. If the count cannot be reconstructed, that is itself the finding: a search whose size is unknown cannot be assessed.
Costs: the trade the test did not pay for
A cost assumption of zero is the most optimistic single number in a backtest, and it is the easiest to fix. The components are commission, the half-spread crossed on entry and again on exit, slippage from the size of the order against the depth available, and, for instruments that carry them, funding, borrow and the roll. A round trip that pays three basis points a leg pays six, and a rule that trades often pays it constantly.
The drag is linear in turnover and therefore easy to state in a table: at 6 basis points a round trip, 25 round trips a year costs 1.50 per cent of notional a year, 50 costs 3.00, 100 costs 6.00, and 200 costs 12.00. A rule whose edge is smaller than its turnover cost is a rule that loses money at the assumed costs, and the ordering of those two numbers is the first thing to check rather than the last.
In the audit below the test charged nothing, and 200 round trips at 6 basis points restore 12.00 percentage points of cumulative return to the cost column — 3.00 points a year. Note what that arithmetic implies about frequency: the same rule at 100 round trips would have cost half as much, and a test that reports a gross figure without its turnover has not reported the number that decides whether the rule is viable.
There is a second, subtler cost defect: a cost model that is applied but not scaled. A fixed 5 basis points on every trade ignores that a small order crosses less spread than a large one, that a thin instrument costs more than a liquid one, and that costs rise precisely when the market is stressed, which is when the rule is most likely to be trading. The check is a cost per trade that varies with the order’s size relative to the depth available, or at minimum a stated assumption with its sensitivity: what happens to the result if costs double.
One sample: the number and its error bar
The fifth defect is not a bias like the other four; it is the absence of an error bar. A backtest reports a statistic computed from a finite sample of trades, and a finite sample of trades has a standard error. The standard error of the mean result per trade is the standard deviation of the per-trade results divided by the square root of the number of trades, and it is usually large enough to swallow the finding.
In the audit below the per-trade standard deviation is 2.80 per cent and the sample is 200 trades, so the standard error of the mean is 0.198 per cent. A mean of 0.097 per cent per trade — the corrected figure — is 0.49 standard errors from zero. At 60 trades instead of 200, the same mean would be 0.27 standard errors from zero. Neither is a finding, and no correction to the other four defects changes that, because the error bar is not a defect in the test. It is the test.
The check is to report the standard error beside every statistic, and to state the sample in trades rather than in years. Four years of daily data is 1,000 bars, and a rule that trades four times a month has about 200 trades in that window; one that trades every session has 1,000. The years are not the sample size. The trades are, and the effective sample is smaller still when the trades overlap or when they cluster in one regime.
Table 1. The five defects, what each does to a reported figure, the check that catches it, and what the test should report. Source: constructed for this piece from the audit in Table 2.
| Defect | What it does | The check | What to report |
|---|---|---|---|
| Look-ahead | Executes at a price that was generated by the signal; imports restated, revised or adjusted data | Was every input knowable on the decision date? Execute on the next bar’s open; use filing dates with a publication lag; use point-in-time series | The execution convention, the data vintage, and the size of the gap between signal and fill |
| Survivorship | Selects the sample on an outcome: the names that were still listed | Point-in-time universe including delisted and acquired names | How many names entered and left the universe, and what the dropped names contributed |
| Parameter search | Reports the maximum of many trials as though it were a single test | Count every variant tried, then deflate by that count | The number of variants, the expected maximum for that count, and the family-wise threshold |
| Costs ignored | Charges nothing for crossing the spread, commission, slippage, funding or the roll | Recompute at a stated cost per round trip, then double it | Turnover, cost per round trip, and the result at zero and at double the assumed cost |
| One sample | Reports a statistic with no error bar, from a sample counted in years rather than trades | Standard deviation of per-trade results ÷ √number of trades | The number of trades, the standard error, and the statistic measured in standard errors |
One test, audited
The five defects are easier to see when they are subtracted one at a time from a single figure. Table 2 does that on an illustrative test: a daily rule on a 120-name universe, 1,000 bars — four years at 250 sessions — 200 round trips, and a reported cumulative result of 56.00 per cent, which is 11.76 per cent a year compounded and 14.00 per cent arithmetic.
Every line of that table is constructed for this piece. The point of the exercise is the shape of the arithmetic, not the numbers: a reported figure decomposes into the defects that produced it, and what remains is smaller than the error bar around it.
Table 2. An illustrative backtest audited defect by defect. The reported result is cumulative over four years; each correction is the amount that defect contributed to that figure; the residual is what remains, with its standard error. Source: constructed for this piece; no market, instrument or strategy is described.
| Line | Computation | Value | Per year |
|---|---|---|---|
| Reported result | stated, cumulative | 56.00% | 14.00 points |
| Less look-ahead | 200 round trips × 0.035% | −7.00% | −1.75 points |
| Less survivorship | 32 trades × 0.55% | −17.60% | −4.40 points |
| Less costs | 200 round trips × 0.06% | −12.00% | −3.00 points |
| Residual | 56.00 − 7.00 − 17.60 − 12.00 | 19.40% | 4.85 points |
| Residual as an annual rate | 1.194^(1/4) − 1 | — | 4.53% |
| Standard error of the sample | 2.80% × √200 | ±39.60% | — |
| Residual in standard errors | 0.097 ÷ 0.198 | 0.49 σ | — |
Three observations follow from the table, and they are the reason to do the exercise rather than argue about it.
The first is that the defects are not small. The three that can be subtracted account for 36.60 of the 56.00 points — 65 per cent of the reported figure — and the largest of them is a universe that was built after the fact rather than a subtlety of execution.
The second is that the residual is not a finding. At 19.40 points cumulative and 4.53 per cent a year it looks like something. It is 0.49 standard errors from zero, on a sample whose standard error is 39.60 points, and the honest reading of “0.49 standard errors” is that the corrected result is indistinguishable from no result at all. Nothing about the correction procedure turned noise into evidence; it removed the part of the number that was manufactured.
The third is that the parameter search is not in the table, because it cannot be expressed as a subtraction from a result. A search over 200 variants produces a best-of-200 statistic whose expected value is 3.26 under the null; if the test’s reported statistic was 2.9, the search explains it entirely and there is nothing left to explain. The correction for that defect is not a haircut to the return; it is a refusal to treat the maximum of a search as a measurement. Table 3 sets out the arithmetic that makes the refusal checkable.
Table 3. The expected maximum of n independent standard normal trials, the Bonferroni threshold that keeps the family-wise error at 5 per cent across all n, and the per-test significance that threshold implies. Source: √(2 ln n) for the expected maximum; the threshold is the two-sided normal quantile at 0.05 ÷ n.
| Variants tested | Expected maximum | Family-wise threshold | Per-test p |
|---|---|---|---|
| 1 | — | 1.96 | 0.050000 |
| 5 | 1.79 | 2.58 | 0.010000 |
| 10 | 2.15 | 2.81 | 0.005000 |
| 50 | 2.80 | 3.29 | 0.001000 |
| 200 | 3.26 | 3.66 | 0.000250 |
| 1,000 | 3.72 | 4.06 | 0.000050 |
| 10,000 | 4.29 | 4.56 | 0.000005 |
What to do instead
The remedies are procedural, which is why they are unpopular: they cost time before the result is known, and they constrain the freedom to keep trying.
Write the specification before the run. Rule, universe, period, execution price, cost model, and the statistic that will decide. A specification written afterwards is a description of what was found, not a test of what was proposed, and the difference is the whole of the parameter-search problem.
Hold out a period and touch it once. Split by time, not at random: train on the first two-thirds, and reserve the last third. Once the holdout has been looked at, it is part of the search — a holdout used four times is a training set with extra steps.
Use a point-in-time universe and point-in-time data. Names as they were listed, figures as they were filed, series as they were published. Where that is impossible, report what was dropped and what its absence is worth, as Table 2 does.
Charge the costs, then double them. A result that survives 6 basis points and fails at 12 was never a result; it was a cost assumption. Report both.
Count the trials and deflate. The number of variants is a required field, and Table 3 is the arithmetic that makes it useful. Where the count is unknown, the test cannot be read.
Report the error bar, and count trades rather than years. The statistic in standard errors is the only version of the number that can be compared with zero.
Then run it forward. A shadow period — the rule fixed, the data arriving — is the only sample that was not available when the rule was chosen. It is also slow, small, and the only one that tests the thing the reader cares about.
What a clean backtest still is not
Suppose every defect above is corrected: the execution is right, the universe is point-in-time, the costs are charged, the trials are counted, the holdout was untouched. The output is still one path through one sample under one set of assumptions, and it remains a description of the past.
A backtest cannot see a change in the market’s structure: a different participant mix, a different tick size, a different cost of leverage, a different set of rules. It cannot see a change in its own effect: a rule that is published, or widely adopted, meets different fills than the sample it was tested on. It cannot see capacity, because the sample’s depth is not the depth the rule would have faced at the size the rule would need to be worth running. And it cannot see the cost of being wrong in the way the reader will be wrong — a drawdown that is survivable in a spreadsheet and not in a mandate.
The honest statement is short: a clean backtest is not evidence of future returns. It is evidence that a rule was specified precisely enough to be computed, that its inputs were available when the decision would have been made, and that its costs were charged. Those are necessary conditions for a serious test and they are not sufficient for a return. A backtest can fail and be informative; it can pass and mean nothing. The only claim the arithmetic supports is the narrow one: if the rule is run again on data from the same process at the same costs, the result is expected to be smaller than the sample’s, because the sample’s maximum is the reason the rule was chosen.
What a test costs
Three engine calls produce the audit: backtest, for the run itself with the cost model applied and execution on the next bar’s open; indicator, for the per-trade statistics that produce the standard error; and verify, to cross-check the price series against a second observation — a series that has been adjusted differently is a different sample. At the published $0.006125 on Base that is $0.018375, published rather than measured while the analyst services are pending review.
Five model calls draft the work at DeepSeek’s measured $0.00700: the specification, the defect checklist, the reconciliation table, the paragraph that reads the residual, and the one that says what the test does not establish. That is $0.035000, so one audited test is $0.053375.
Six a month — one rule, six specifications, which is a realistic number for a question worth testing — is 18 engine calls ($0.110250) and 30 model calls ($0.210000): $0.320250, or $0.053375 a test and $3.84 over twelve months. At the metered ceiling of $0.00955 the model line rises to $0.286500 and the month to $0.396750, 23.9 per cent more; on Qwen at a flat $0.00875 the model line is $0.262500 and the month $0.372750, 16.4 per cent more. A test is eight paid calls, so the 0.50 USDC opening balance — seventy-one calls at $0.00700 — holds eight tests. The document parts that render the reconciliation carry no published per-call price and sit outside every total here.
A decision rule
The output of a backtest is a specification and a reconciliation, not a result. Four things go in the note, and the fourth is the one that keeps the other three honest.
Publish the specification before the result. Rule, universe, period, execution price, cost model, number of variants, and the statistic that decides. A reader who has the specification can re-run the test; a reader who has only the result cannot.
Publish the reconciliation, defect by defect. Each correction with its arithmetic, as Table 2 does, so that the reader can see what the reported figure was made of. A corrected number with no audit trail is a second unsupported number.
Publish the error bar and the trade count. The statistic in standard errors, beside the number of trades rather than the number of years. If the statistic is under one standard error, say so in those words.
Publish the refusal. One sentence, in the body rather than a footnote: this is a description of one sample under stated assumptions, it is not evidence of future returns, and the number it reports is expected to be smaller the next time it is computed, because the sample’s maximum is why the rule was chosen.
That is a smaller deliverable than a strategy with a track record, and it is the only one the arithmetic supports. A test that reports all four is useful even when its residual is zero — especially then, because the reader learns that the rule has no content before paying for it in a live account.
Method note
The rule, the universe, the 1,000-bar window, the 200 round trips and every figure in Table 2 are constructed for this piece and describe no market, no instrument and no strategy; they exist to show the arithmetic of a reconciliation. The look-ahead correction is 200 × 0.035 = 7.00 points cumulative; the survivorship correction is 32 × 0.55 = 17.60, where 18 of 120 names (15.0 per cent) were dropped and carried 32 of the trades at a mean of −0.55 per cent; the cost correction is 200 × 0.06 = 12.00 at 6 basis points a round trip, being 1.0 basis point of commission, 1.5 of half-spread and 0.5 of slippage on each leg. The residual is 56.00 − 36.60 = 19.40 points, which is 4.53 per cent a year compounded and 4.85 arithmetic. The standard error of the mean is 2.80 ÷ √200 = 0.198 per cent per trade, against a corrected mean of 19.40 ÷ 200 = 0.097, so the residual is 0.49 standard errors from zero; the cumulative standard error is 2.80 × √200 = 39.60 points. Table 3’s expected maximum is √(2 ln n), an approximation for independent normal trials that is optimistic when the variants are correlated, as they usually are; the thresholds are two-sided normal quantiles at 0.05 ÷ n. Engine prices are the published $0.006125 per basic operation on Base from catalog.json, read 31 October 2026, published rather than measured because the analyst services are pending review and return no 402 quote. Model prices are measured from the live 402 handshake’s exact amount on Base on 31 October 2026 at a short prompt and max_tokens: 1000: $0.00700 for deepseek-v4-flash, $0.00955 at about 3,000 tokens, $0.00875 for qwen3.8-flash at both lengths. Monthly counts are planning assumptions. No result in this piece is presented as evidence that a method works, no strategy is recommended, and nothing here is investment advice.
The pillar is Finance and markets on x402 — three engines, four jobs, one price. The role that runs a test is trade_analyst, whose backtest part executes on the next bar’s open and reports the period, the costs assumed and the overfitting caveat, and which will not call a strategy profitable. The compute is bought at x402.press/services/deepseek/ with a second model at x402.press/services/qwen/; the measured handshake is reviewed at DeepSeek: per-call chat completions over x402, and the two endpoints against each other at Inference endpoints compared. The same discipline — a sensitivity grid that shows which assumption the answer depends on — is the subject of The DCF model structure checklist, and the version of it that reads a document rather than a series is in Financial statement red flags, where the checks are designed to fail loudly rather than to confirm.
x402.review is written and edited by N.P. Vincent. The publication is connected to x402.press, which sells the services reviewed here; that interest is disclosed on every piece that touches a service, and it is stated here rather than left to inference.
Filed under Jobs · Tags: backtesting, overfitting, validation, trading
