Paradox of Predictability

Trader's Mindset

Nov 26

I recognized early on that you must be able to predict markets to profit from them (at least for traditional directional trading as opposed to liquidity provision–even then one must attempt to avoid the jumps). Some people think markets are completely unpredictable. It surprises me still that some traders like to think they can profit from directional trading without being able to predict the market. Now one might consider it empirically or as a deterministic process with high variability but the basic fact is not up for debate. It is not really even interesting enough to bother correcting such notions.

However, what is interesting is this notion that some traders have that it can only be predicted typically in the long term but more rarely you might hear a trader refer to it in the short term. The common refrain is “markets are random in the short term but predictable in the long term”.

Let’s look at the typical arguments.

Long term predictable thesis

  • Markets have a trend or drift components
  • Deviations from this represent opportunity and the rest is noise

Short term predictable thesis:
The problem is most things that can be predicted should virtually always be able to be predicted more accurately in the short term. Take weather forecasting, that’s a dynamic, difficult and complex system like markets, and weather forecasts tend to become more accurate the less far out they are predicted. Imagine how absurd it would be if it were the opposite: older forecasts would be more accurate then newer forecasts. Taken to the extremes, our ability to predict the weather would depend on how far back we started predicting. However, we know more recent data should be more relevant.

The reality is, if you can predict anything, then the forecast should become less accurate, never more accurate, the farther out in time you go. However, this long term thesis seems to be, at least, on the surface somewhat plausible. You can look at a long term chart and see whether or not there is a long term trend.

In order to resolve the paradox, we need to introduce a few more notions. The notions we need to introduce are precision, accuracy, and frequency. Accuracy can be thought of as our winning percentage. We can increase our winning percentage by making our prediction less precise. Precision can be thought of as leverage or return. The more precisely we can predict the market then the more we can make above market returns. On the other hand, a low precision might not allow for any profit opportunity. It is the combination of winning percentage and leverage that will determine how much we make.

The final piece needed to resolve the paradox is the information frequency or wavelength of our predictive information. Let’s imagine for a moment our predictive information is simply some sort of price information. If we are tracking longer periodic charts, say 1 hour charts, then it is not unlikely that we might disregard most movements on the 1 minute chart as noise. In reality, the information change is taking place below our precision, that is below our level of measurement or concern.

Finally, I should address the notion that market movements are random. First, a true uniform random process would be easy to predict because we’d know any deviation from the median would be a profit opportunity. I would argue that virtually no market movements are random. All market movements are created by traders. Traders have reasons for trading: namely the making of money. However, whether or not they are predictable is what we really want to know.

The answer to that is more complicated. For trading systems, we’re interested in whether or not we can statistically profit from an opportunity. A discretionary trader is not restricted to statistical profitability for any given trade idea but still must be able to discern the winners from losers. In my opinion, markets have variable degrees of predictability: sometimes they can be predicted using certain methods but other times those methods fail. But, there are no easy predictions. But, let me suggest that markets must make themselves predictable enough in order to entice trade.

Going back to the notion of markets being predictable in the long term and not the short term: there is some “truth” to that “naive” notion. The truth is that yes anyone can see a trend and know that okay the prices are more likely to be higher or lower in the future. However, if one isn’t able to do call such things with sufficient accuracy and precision then such knowledge doesn’t constitute any advantage. On the other hand, if one is trading on short term patterns then their results are going to be very subject to who is trading. A single pattern can be created by different participants. The risk becomes the “shocks” to the systems. Overall, there is a sort of logical “price discovery” process or “market logic” that can be used to understand markets. But, achieving this in real-time and profiting over costs is no easy task.

About the Author

The author is passionate about markets. He has developed top ranked futures strategies. His core focus is (1) applying machine learning and developing systematic strategies, and (2) solving the toughest problems of discretionary trading by applying quantitative tools, machine learning, and performance discipline. You can contact the author at