System Developer Tip: Get Better Stats!

System Trading

Mar 16

Traditionally developed trading systems are made of up of rules. A common example, would be a moving average cross over system that buys when a fast moving average crosses above a slow moving average. The downside to developing a system with the “rules first” approach is that it is not unlikely that whatever results the system developer sees in the backtest is somewhat random.

The trader will try one rule or combination and find it doesn’t yield good results and then move on to another and another. Eventually, the trader is likely to hit upon a combination that looks good by random chance. Previously, I shared that one way around this was not to look at individual backtests but rather to run an optimization and look at the entire range of possibilities. While it makes sense, it is not the way most developers prefer to work– at least initially.

In this post, I wanted to share another technique I developed or rediscovered for learning from market data. But, first let me credit Dr. Howard Bandy who shared with me an almost identical concept years ago. The technique is to convert the binary signal into a continuous state and buy or sell every bar for the entire state. Doing this allows the system developer to better explore the statistical properties of your data and gives greater confidence that whatever feature you have discovered about the market is more likely valid.

See the example below:


// Beyondbacktesting.com

//For example the signal code commented out can be converted to the continuous state code below
{
if Average(Close, 20) cross above Average(Close, 50) then buy next bar at market;
if Average(Close, 20) cross below Average(Close, 50) then sell next bar at market;
}


if Average(Close, 20) > Average(Close, 50) then buy next bar at market;
if marketposition <> 0 then 
sell next bar at market;


Now, you can start to ask questions about how the market really generally behaves in relation to your state indicator. Usefully, the system performance report also reports the bar-by-bar statistics. For example, if your winning ratio is positive then it tells you that up bars are more likely then down bars. This is useful information for developing filters.

While we have increased the sample size, the relationship is still binary. You can also explore the data across multiple spectrums by converting the state signal into a power signal. Below, I change the conditional to a difference, and I use the min/max scaling with range -1 to 1. This effectively converts the trend conditional indicator into a trend strength indicator:


value10 = Average(Close,20) - Average(Close,50);

if (highest(value10,200) - lowest(value10,200)) > 0 then
value11 = (2 * (value10 - lowest(value10,200)) / (highest(value10,200) - lowest(value10,200)))-1;


if value11 > .5 then buy next bar at market;

By utilizing the simple techniques shared above, it opens up the ability to more effectively learn from market data. Of course, the developer should hold an out-of-sample segment to validate any relationships. Because the statistics are derived from many more instances, it is less likely that the developer happens upon a good looking backtest by random chance. Only after the general properties of the data are understood should the developer seek to refine the system into the final optimized form.

Hope it helps!

About the Author

The author is passionate about markets. He has developed top ranked futures strategies. His core focus is (1) applying machine learning and developing systematic strategies, and (2) solving the toughest problems of discretionary trading by applying quantitative tools, machine learning, and performance discipline. You can contact the author at curtis@beyondbacktesting.com.

  • Neil says:

    Hi Curtis. I just found out about your site. Thank you for this creative idea. I appreciate you sharing it.

  • >