WFA Failure Scenario

We will introduce a somewhat hypothetical, but nonetheless realistic, walk forward analysis (WFA or WFO) scenario that returns poor results, and then we will suggest alternative strategies that might reveal much better results.  There are two common types of WFA; anchored which can use all the data and rolling. In this scenario, we are only considering the rolling scenario. In the rolling scenario, a trading system is optimized  over in-sample periods (IS) and verified against out-of-sample data (OOS). The previous OOS sample becomes the next in-sample and this process is carried out iteratively over the entire history. The resulting slices of OOS equity curves can be stitched together to create a single OOS equity curve that is used to validate or invalidate the system.

The walk forward optimizer selects the best values to trade according to some heuristic or rule. There is some discretion in determining what values the walk forward optimizer can select amongst. A key and underlooked component of the WFA cycle is the selection criteria. Most WFA use simple selection criteria such as maximum profit or max profit to drawdown. This contrast with the process that a machine learning expert might use, which would be to seek to understand the underlying relationships before making a determination– but, that is another topic for another time.

In our hypothetical scenario, let’s imagine that for whatever reason that during every IS period there are some input settings that strongly outperform. But, that during the next OOS period, those same settings turn out to be poor. It might look something like the following:

IS OS IS OS IS
Optimal Found 10 7 5 9 11
Optimal Used 10 7 5 9
Profit IS $8,000 $12,000 $9,000 $14,000 $8,000
Profit OOS ($5,000) ($3,000) ($2,000) ($200)

The rational conclusion suggested from this WFA is that the system is a failure and doesn’t work. You shouldn’t trade it. However, what if in every sample there was another optimal value that produced a consistent but lower return, and looked something like this:

IS OS IS OS IS
Ignored Optimal 14 14 14 14 14
Profit IS $7,000 $11,000 $8,000 $12,000 $6,000

In other words, this indicator has two areas or humps that yield outperformance. One of those humps is settings from 5 to 11 which yield the best outperformance but that outperformance is completely inconsistent and cycles from out performance to under performance with unpredictable shifts. The second hump has a lower return but has been stable over a very long time.

The big waves for the returns from 5 to 11 drown out the consistent and stable returns of the system when used with a return of 14. Now imagine, even if the system had been profitable with shifting values from 5 to 11, which would type of game would you rather play, an unproven combination with a bigger return or something that has always worked? Of course, there are no guarantees but the second table is much more appealing to me.

As far as suggestions for alternative strategies for addressing this type of scenario, one possibility would be to reexamine the walk forward optimizer’s fitness function. Examining all the equity curves for all settings over time instead relying on a single snapshot of performance in time would likely reveal the consistent outperformer, as well.  The dropout technique, discussed in my most recent post, could be adapted to find stable parameters instead of for OOS verification. Allowing both the randomized dropout selection and the optimal indicator parameter values to vary together should reveal the settings that are most consistent through time because each time the dropout selection changes, at least, a portion of the data remains OOS. The analysis is a bit complicated but if any optimal value performs well across all dropout sections then it confers a degree of stability and consistency. Finally, the anchored WFO technique seems like it might be designed for this sort of scenario.

What do you think? Ideas? Suggestions? Leave a comment or shoot me an email.

2 thoughts on “WFA Failure Scenario

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s