WFA Failure Scenario

System Trading

Feb 09

We will introduce a somewhat hypothetical, but nonetheless realistic, walk forward analysis (WFA or WFO) scenario that returns poor results, and then we will suggest alternative strategies that might reveal much better results.  There are two common types of WFA; anchored which can use all the data and rolling. In this scenario, we are only considering the rolling scenario. In the rolling scenario, a trading system is optimized  over in-sample periods (IS) and verified against out-of-sample data (OOS). The previous OOS sample becomes the next in-sample and this process is carried out iteratively over the entire history. The resulting slices of OOS equity curves can be stitched together to create a single OOS equity curve that is used to validate or invalidate the system.

The walk forward optimizer selects the best values to trade according to some heuristic or rule. There is some discretion in determining what values the walk forward optimizer can select amongst. A key and underlooked component of the WFA cycle is the selection criteria. Most WFA use simple selection criteria such as maximum profit or max profit to drawdown. This contrast with the process that a machine learning expert might use, which would be to seek to understand the underlying relationships before making a determination– but, that is another topic for another time.

In our hypothetical scenario, let’s imagine that for whatever reason that during every IS period there are some input settings that strongly outperform. But, that during the next OOS period, those same settings turn out to be poor. It might look something like the following:

Optimal Found1075911
Optimal Used10759
Profit IS$8,000$12,000$9,000$14,000$8,000
Profit OOS($5,000)($3,000)($2,000)($200)

The rational conclusion suggested from this WFA is that the system is a failure and doesn’t work. You shouldn’t trade it. However, what if in every sample there was another optimal value that produced a consistent but lower return, and looked something like this:

Ignored Optimal1414141414
Profit IS$7,000$11,000$8,000$12,000$6,000

In other words, this indicator has two areas or humps that yield outperformance. One of those humps is settings from 5 to 11 which yield the best outperformance but that outperformance is completely inconsistent and cycles from out performance to under performance with unpredictable shifts. The second hump has a lower return but has been stable over a very long time.

The big waves for the returns from 5 to 11 drown out the consistent and stable returns of the system when used with a return of 14. Now imagine, even if the system had been profitable with shifting values from 5 to 11, which would type of game would you rather play, an unproven combination with a bigger return or something that has always worked? Of course, there are no guarantees but the second table is much more appealing to me.

As far as suggestions for alternative strategies for addressing this type of scenario, one possibility would be to reexamine the walk forward optimizer’s fitness function. Examining all the equity curves for all settings over time instead relying on a single snapshot of performance in time would likely reveal the consistent outperformer, as well.  The dropout technique, discussed in my most recent post, could be adapted to find stable parameters instead of for OOS verification. Allowing both the randomized dropout selection and the optimal indicator parameter values to vary together should reveal the settings that are most consistent through time because each time the dropout selection changes, at least, a portion of the data remains OOS. The analysis is a bit complicated but if any optimal value performs well across all dropout sections then it confers a degree of stability and consistency. Finally, the anchored WFO technique seems like it might be designed for this sort of scenario.

What do you think? Ideas? Suggestions? Leave a comment or shoot me an email.

About the Author

The author is passionate about markets. He has developed top ranked futures strategies. His core focus is (1) applying machine learning and developing systematic strategies, and (2) solving the toughest problems of discretionary trading by applying quantitative tools, machine learning, and performance discipline. You can contact the author at

  • >