A Reader’s Question on Backtesting and Discretionary Trading

John writes,

I saw the emphasis on this site and was a bit intrigued. As of late, I have been thinking of writing off back testing as something that is not required for my method of trading as I am about as a discretionary trader as one can get – I use support and resistance and a few other price points to influence my trading, but overall, it is just about taking “promising” set-ups I am familiar with. I am beginner so my question is am I missing something, i.e. should I still not give up on backtesting because it could be of some intrinsic value? Thank you much, John

This is a very excellent question. First, if you are getting the results you want then obviously it is not required. However, utilizing backtesting, quantitative studies, algorithmic edge, and simulations can elucidate specific, unique, non-obvious insight into markets generally referred to as edge. The greatest difficulty for integrating these sorts of processes as an active process for the day trader is that many forms of day trading are facilitated best when the trader is in a well-rested open minded and peak performance state. A typical process might be for the day trader to observe certain patterns of interest or ideas during the morning or trading day and then run studies at night. However, if one doesn’t have the resources for a dedicated researcher then it can create a time crunch and fatigue and exhaustion can easily set in. On the other hand, if you don’t test your best insights and observations then you give up many opportunities for developing edge. One possible solution to the fatigue/performance split might be to avoid being too reactive in your trading. This might be accomplished by trading on the simulator and developing your insights more slowly.

Many think of the term “backtesting” as some a singular process that results in a system but it can refer to different processes. Below, I suggest some of the benefits and limitations of the processes,

• Simulations. Running simulations is a little different then what most people think of when they hear “backtesting”. Simulations can allow for testing and understanding variances. Simulations are most useful for deriving insight into position sizing and risk management. As an example, one early simulation I ran was a perfect trader variation, and I have ran many of these and like the idea. I wanted to see what happened if a swing trader caught the swing lows within some low tolerance. This study really opened my eyes because the perfect trader was having these really high drawdowns. It shown to me right there that swing trading might not be able to be leveraged.
• Quantitative studies. One can think of quantitative studies as a singular backtest. These are often useful for elucidating general insights into markets. These help to elucidate general insight into your market. Often they may help you disprove a wrong belief or thesis. But, they do not form an actual system. These may be run on an ad-hoc basis. A good example is a “runs study”, where we test what tends to happen after moves up or down, to learn if your market tends to have momentum or mean reversion properties.
• Graybox, algorithmic edge, semi-systematic, and quantamental. Finding areas where edge can be blended is where much of my focus is. The goal is generally to improve an existing capability. For example, let’s say you have a pattern or setup you look for in the market. It might not have enough edge to actually automate. However, there may be aspects you can automate. A simple example might be using an algorithm to recognize the particular conditions you want to trade under. Alternatively, you might have good entries but know that your exits could be optimized. You could keep the discretionary entries and find the optimal exits. There might be other cases where you can see the edge but have trouble executing the ability to simplify the execution or partially automate it may help. Another very key benefit and concept is “multiple factor models”. My bias for a trade by itself or a systematic edge might be prerequisite but insufficient to trigger an actual trade– instead it might require both.
• Systematic and automated system development. This is where we take the insights derived from quantitative studies and go the next step toward codifying all of the rules. The benefit is a much higher level of confidence and ability to automate many strategies versus just running a few.
• Bayesian, If-Then, and Relational Analysis. I have not did much in these areas. But, a basic concept might be to create various models for how the market might behave if certain things happen. For example, if there are new sanctions on Iran then what will happen to the price of oil? You can create probabilities for certain events and then use those as a basis for your trading.
• Automated System Development, Generative Development, and Reverse Engineering. Many tools exist today that can use generative processes to create trading systems. Often, the rules generated may be overly complex but sometimes can be reverse engineered to elucidate simpler generalized and unique insights into markets. For example, one system had this really strong performance but as I broke down the rules, I wasn’t finding anything of real value until I realized that it one very basic rule of avoiding extremely overbought markets. It didn’t generate its edge so much from finding some rare opportunity as it it did from avoiding risky markets.
• Limitations and Risk. One of my thesis for higher levels of trading performance is that it comes from higher levels of “market cognition”. Market cognition is basically the ability to understand the factors, sentiment, other traders, etc. driving the market today. It is suspected that markets tend to have properties similar to what are known as Markov or memory less processes. The basic gist is that older data is less relevant. I think that a general problem of backtesting as a development process is that backtesting can easily re-orient ones focus from seeking opportunity and understanding markets to “throwing everything at the wall” or only testing what is available for testing. Both false positives (edges that don’t exist) and false negatives (excluding real insights) are a distinct possibility. For myself, watching markets in real-time and tape reading seems to enhance my “market cognition”.  Now, another trap is that it is easy to find patterns that show up over a small data set but don’t extrapolate. This is especially a risk when running “quantitative studies” that only contain a few instances.