Backtesting has many inherent limitations. The chief among them is that backtesting can only eliminate systems that did not work in the past. In fact, it cannot even do that fully. It can only eliminate a specific form of proposition which is based on the form of “Doing X was profitable in the past and thus if I do X in the future it will be profitable”. Backtesting can not really say whether or not X will profitable in the future, in either case. However, it can reveal the case where a pattern is falsely assumed to be predictive in the past and is therefore traded like it was predictive.
The more backtests that a developer runs then the more likely they are to find results that appear statistically valid but won’t hold up. Generative system development processes often attempt to use advanced statistical measures. But, this just leads to more random systems developed that pass the tests by pure chance.
What’s the solution to the problem? The solution is that one must be able to predict which systems will work before backtesting them. The very first system I developed was profitable and has continued to profit well over the past 5 years. How can one explain it?
The explanation is the same reason behind that discretionary traders can be profitable in the first place. The reason is that traders with high market cognition have a higher probability of understanding the dynamics that drive markets and thus their ideas have higher probability of being predictive.
This realization is not really anything new. Many system developers believe backtesting is the process by which they falsify their ideas. But, the deeper realization is that traders with higher market cognition will be more successful as system developers because their ideas will be more likely to work without falsification. Stated another way, the developer who can generate a higher percentage of systems that pass a given level of statistical testing has higher market cognition. The developer who needs to generate more systems to pass the same statistical tests has lower cognition and will have more failed systems.
This demonstrates, also, that for generative system developers that the goal is not merely to produce the best metrics for eliminating bad systems but the goal is to also produce methods that generate a higher proportion of systems that pass. Methods that generate systems with higher passing rates for equivalent statistical measures are more likely to be valid.
Some immediate take aways are to be careful getting into a “rut” of simply backtesting ideas or concepts because they are convenient to test and to instead try to come up with creative discretionary trading ideas. This might be accomplished by the following ideas:
The author is passionate about markets. He has developed top ranked futures strategies. His core focus is (1) applying machine learning and developing systematic strategies, and (2) solving the toughest problems of discretionary trading by applying quantitative tools, machine learning, and performance discipline. You can contact the author at firstname.lastname@example.org.
Please log in again. The login page will open in a new tab. After logging in you can close it and return to this page.