Tradestation and Multicharts support a form of optimization known as Matrix WFO. The Matrix WFO runs through many combinations of WFO in and out of sample periods to try to determine the best setting to use for re-optimization. As well, some traders use Matrix WFO to robustness test a strategy.
Recently, I implemented my “Proven Winner WFO” technique. This allowed me to do something unique. I was able to run a WFO Matrix optimization on the in-sample data but only for already pre-determined high performing strategies. In other words, the WFO can only select from winning strategies (see Proven Winner). There are many potential insights to be gained from such an optimization: such as seeing how well a strategy walk-forward optimizes in the “optimal” working case.
In my case, the performance across my matrix space was uniform positive except for one losing grid cell. Now, for that grid cell think about what the optimizer had to do: it had to create a losing equity curve from a selection of very high performing systems! Beyond that, the performance among the positive optimization varied significantly.
And, that’s when I realized a simple but perhaps not obvious insight: smaller sample sizes will exhibit greater deviations in the mean. You can prove it to yourself with a simple Excel experiment. Generate 1,000 random numbers in excel from 1 to 100 and take the average. Now, take samples at 5 or 10 from that population and average them. While sometimes you will get close to the true mean– often your results will be much higher or lower. For example, I got 38 and 65 when I performed this over about 5-10 samples at the extremes. Remember, we know the true mean was 50 in this case from the larger sample. Now, think about the implications for a trading system parameters such as average profit per trade.
The implication is clear much of the variation in performance observed in the matrix optimization is going to be caused due to variation in smaller samples sizes and not by any difference in actual performance. The understanding of this has implications as to how one should properly structure a matrix optimization.
The author is passionate about markets. He has developed top ranked futures strategies. His core focus is (1) applying machine learning and developing systematic strategies, and (2) solving the toughest problems of discretionary trading by applying quantitative tools, machine learning, and performance discipline. You can contact the author at firstname.lastname@example.org.
Please log in again. The login page will open in a new tab. After logging in you can close it and return to this page.