Skip to main content
|View as Markdown|

How we test

Every trading idea is tested in our edge lab before any EA or strategy is built on it. Most ideas fail, and the lab keeps those too: the hypothesis log lists every idea tested, with its verdict. This page explains how an idea is tested and what counts as a pass.

The test​

  • The idea and its pass mark are written down first. Each hypothesis is recorded with the score it must reach before it is tested. The pass mark is never moved after the result is known.
  • It is scored against random entries. The rule's trades are compared with random entries that keep the same side, stop and target. A rule passes only if it does clearly better than chance.
  • It pays the broker's spread and commission. Every trade is charged the broker's own costs, at the time it would have been filled.
  • It is checked on a second data feed. A result that appears in one supplier's quotes and not in another's is treated as a data artifact, not an edge.
  • A holdout is kept back and scored once. Some markets or years are set aside while the rule is being worked on. When the rule is fixed, the holdout is scored once. A rule that needs changing after that is a new hypothesis.
  • The bar is t > 3. Many ideas are tried, and some will look good by chance. A t-statistic above 3 is a higher bar than the usual one, set for that reason.

Lab tests and tester runs​

A lab test is the verdict on a rule. A tester run is the EA in a platform's Strategy Tester on a shipped preset. It shows that the EA trades the rule and how the platform fills it. It is never the verdict.

On a strategy page, the Evidence section shows the lab test first and the tester run second. A rule that passed or is on the watch list has its own research page with the full lab test, and links to the product that trades it, if there is one.

Next​