Backtesting has many inherent limitations. The chief among them is that backtesting can only eliminate systems that did not work in the past. In fact, it cannot even do that fully. It can only eliminate a specific form of proposition which is based on the form of “Doing X was profitable in the past and thus if I do X in the future it will be profitable”. Backtesting can not really say whether or not X will profitable in the future, in either case. However, it can reveal the case where a pattern is falsely assumed to be predictive in the past and is therefore traded like it was predictive.
The more backtests that a developer runs then the more likely they are to find results that appear statistically valid but won’t hold up. Generative system development processes often attempt to use advanced statistical measures. But, this just leads to more random systems developed that pass the tests by pure chance.
What’s the solution to the problem? The solution is that one must be able to predict which systems will work before backtesting them. The very first system I developed was profitable and has continued to profit well over the past 5 years. How can one explain it?
The explanation is the same reason behind that discretionary traders can be profitable in the first place. The reason is that traders with high market cognition have a higher probability of understanding the dynamics that drive markets and thus their ideas have higher probability of being predictive.
This realization is not really anything new. Many system developers believe backtesting is the process by which they falsify their ideas. But, the deeper realization is that traders with higher market cognition will be more successful as system developers because their ideas will be more likely to work without falsification. Stated another way, the developer who can generate a higher percentage of systems that pass a given level of statistical testing has higher market cognition. The developer who needs to generate more systems to pass the same statistical tests has lower cognition and will have more failed systems.
This demonstrates, also, that for generative system developers that the goal is not merely to produce the best metrics for eliminating bad systems but the goal is to also produce methods that generate a higher proportion of systems that pass. Methods that generate systems with higher passing rates for equivalent statistical measures are more likely to be valid.
Some immediate take aways are to be careful getting into a “rut” of simply backtesting ideas or concepts because they are convenient to test and to instead try to come up with creative discretionary trading ideas. This might be accomplished by the following ideas:
- Watch your market(s) at least some amount everyday to come up with new ideas. I recommend tracking your market at least in the first 1-2 hours each day to come up with new ideas.
- Tracking and watching new markets.
- Do not settle for convenience bias. Example, moving averages are readily available in most trading platforms. They might serve a purpose but the best insights are more likely to derive from “market cognition” versus patterned or rut thinking.
- Be willing to work harder to test non “traditional” systems. For example, a system that combines price action with fundamentals or sentiment might yield a different sort of profit factor potential then either in isolation.