When designing a trading system there are a number of things traders need to be aware of if they want to avoid building a curve-fit system. Trading system selection bias is one of the most dangerous and seemingly misunderstood.
I spoke previously about the dangers of survivorship bias and the relevance of start date. I also spoke about the need to keep data out of sample and to keep the number of optimisable parameters as low as possible in order to avoid data mining. As well, I mentioned how the only true out of sample data is future data since we already have an awareness of what the market has done in the past.
Selection bias in trading systems
Recently, I have been reading David Hand’s Improbability Principle: Why coincidences, miracles, and rare events happen every day.
It’s a great book and reminded me of the problems of selection bias in trading systems.
Put simply, David’s book reminds us that improbable things happen all the time even though they have a statistically minute chance of transpiring.
In trading system development, this is a very real danger.
Optimising the system
Consider the process of testing and optimising a trading system. Using a computer and a trading platform it is possible to test hundreds and thousands of different combinations of settings.
Clearly, if you test enough different strategies you will eventually find one that gives huge returns and exhibits an equity curve to die for.
The problem is that this equity curve may only have come about by pure chance. The profit potential of the system is actually random, however, the inexperienced trader may believe that she has just stumbled across the ultimate winning trading system.
Eye-balling equity curves
Let’s say that you are in the business of trading systems and you’re looking for a system with low drawdowns, positive expectancy and a smooth equity curve.
If so, what do you think of the equity curve below?
Pretty nice chart I think you’ll agree.
But what if I told you that the curve was actually a result of 10,000 random coin tosses?
That’s right, random outcomes can generate beautiful looking equity curves. Just take a look at this coin toss generator and see for yourself all the types of outcomes that can result from randomness…
The problem is that an attractive equity curve can fool a trader into believing they have a profitable trading system that is not random. This is sometimes enhanced when the trader gets a positive equity curve in out-of-sample too, which may also have come about by pure chance!
Is your trading system random?
Knowing this information is extremely important but how can traders build trading systems that are not random?
Well the answer is not always simple and has a lot to do with statistics. But for starters we can make sure to employ some basic principles:
1) Make sure to base your trading system on theoretical principles before you begin to build it. Don’t search for profitable patterns within the data as you will be sure to find plenty of profitable patterns that are in fact random.
2) Always keep some data out-of-sample for cross-validation of the system. Once you have tested the system in-sample, test it just once on the out-of-sample data to see if it performs in a similar way.
Never go back to the in-sample data and adjust your system based on the out-of-sample results. This guarantees curve-fit results.
3) Keep the number of parameters as small as possible. Increasing the number of adjustable parameters increases the chance of data-ming exponentially. There are profitable trading systems out there that consist of just one adjustable parameter.
4) When optimising, don’t choose the best performing variable. Look for ranges of variables where the system performs well and pick something in the middle.
For example, if a system performs well on a 100 day breakout, it should also perform well on a 98 or 95 day breakout. Or a 102 and 110 day breakout. If it doesn’t perform with very similar variables then there’s a good chance the result is random.
Trading system selection bias: conclusions
In general then, it is important to realise that there are very few ‘edges’ available in the market that are exploitable for the average trader. Staying away from the dangers of curve-fitting is therefore a systems trader’s major concern. Trading systems should be simple and robustness should be preferred over outlandish returns.
I will be looking at some more ways to reduce the chance of curve-fitting in due course, just remember if something seems too good to be true, it almost always is.