A reader, Z.S has a question on WFA. His question and my response follows:
I struggling with WFO similarly to you in your article:
https://beyondbacktesting.com/2017/02/09/wfa-failure-scenario/Despite the fact WFO should bring verification to the development process it rather yields more uncertainty.
Simply changing WFO insample and outsample periods or fitness functions the outsample changes wildly, sometimes I feel it is simply random.I tried Pessimistic Return on Capital from Pardo, but it does not help in this regard at all.
I even tried to rank and select the best N based on fitnessA, then select N based on fitnessB, etc. then intersect them to find the ones which is good enough by several fitness, but this did not result more consistency than I hoped.
I tried to filter out the best N trades to avoid overfitting for some good outlyer, but did not help either.
Now I’ll try to examine the chosen parameter surroundings to avoid selecting a spiky result, we will see.
Did you come up with something usable for WFO? Currently it ruins most of my algs, even the live profitable ones. Maybe this is just a fact of WFO, I don’t know. What do you think?
Thanks,
Z.S (name abbreviated for privacy)
First, thank you for contacting me Z.S. I have a bit of a confession to make, as well. One of my most profitable and successful systems was backtested over a long history and did well. It provided mixed to failed results on WFA. However, it continued to perform in live for years and years with results in-line with the historical results.
One problem with WFA is the “selection” of the parameters which is not at all like most developers would select the parameters. If you think about it, most developers wouldn’t just pick some random best performing value but would look for an area of stability. They would prefer to pick the more stable value.
There are some solutions you can use to “assist” your WFA. By doing so, you will be trading off some the validation benefits of WFA in-exchange for greater intelligence. Let’s go back to the idea that markets are efficient. This means all systems are “fit” to the market to some degree or they couldn’t work. There are two ideas on this and they conflict somewhat but I can explain where they originate. See my recent post on “Market Cognition” for more.
I suggest the following:
The author is passionate about markets. He has developed top ranked futures strategies. His core focus is (1) applying machine learning and developing systematic strategies, and (2) solving the toughest problems of discretionary trading by applying quantitative tools, machine learning, and performance discipline. You can contact the author at curtis@beyondbacktesting.com.
Session expired
Please log in again. The login page will open in a new tab. After logging in you can close it and return to this page.