19th Ave New York, NY 95822, USA

The System Traders Feedback Loop – Don’t Get Stuck In The Back-Testing Spiral

LOREM IPSUM DOLOR SIT AMET

Whatever your opinions are about the concept of back-testing, there are few better ways to build a trading system than through the back-testing of historical data.

Back-testing allows us to answer critical questions like ‘how many positions should I hold in a portfolio?’, ‘how much risk should I take?’ or ‘how effective was this strategy in the past?’

Clearly, back-testing historical data is an integral and important part of becoming a systematic trader.

The Problem With Back-Testing

The problem is that back-testing is an extremely tricky business. When it comes to running simulations on historical data there are numerous biases and limitations that can cause even experienced traders a lot of headaches.

Problems like survivorship bias, lookahead bias, data mining bias, overfitting, all can creep into the development process.

How do you know your system is not curve fit? How do you know your data is accurate? And…

How do you know what you don’t know?

No doubt that the wealthiest traders working for hedge funds and banks have access to the best data and the best tools while most of us have to make do with cheaper alternatives.

The cost of a Bloomberg Terminal, for example, runs at around $2000 a month which is beyond the budget of most retail investors.

Having the most accurate data is not always such a big problem but it does become so when dealing at higher frequencies.

Consider that intraday stock market data is often stitched together from a network of different exchanges. Getting accurate intraday data is rarely possible without moving down to a very granular level.

Readers of the book Flash boys by Michael Lewis will be all too aware of the shenanigans that go on at those high frequencies that help to invalidate any simulation.

All this means is that any backtest you make needs to be treated with caution. Because, ultimately, we can never be 100% certain our simulations are accurate.

Back-testing is so fraught with difficulty that we need to treat it a certain way. We need to remain skeptical at all times and we need to be aware of it’s limitations.

New Traders Are The Most At Risk

Backtesting is also the most problematic for new traders. If you are new to system trading and you don’t know the perils of back-testing then you will likely make many mistakes along the way.

This can result in a worse trading experience than if you had just traded randomly without any back-testing at all.

This is because when you have a backtest you will be more confident in your trading, without knowing that it’s inherently flawed!

So those without back-testing experience are most at risk of losing money from faulty systems.

The Back-Testing Spiral

New traders will also get stuck in what I call the back-testing spiral. This is where you spend all your time running back-tests, tweaking and fine tuning your trading system until you find something that looks impressive but is simply overfit to the data.

In this situation you might run hundreds or thousands of tests looking for that perfect system. Then, when you finally start trading live, you lose money straight away because your system was data mined, not robust to the data.

A Fine Balance

In other words, you spent all your time building a trading system that worked perfectly in the past but has hardly any chance of working in the future.

As you can see, backtesting is extremely valuable but only up to a certain point. Go past that point and back-testing becomes ineffective and even dangerous.

Furthermore, since the future is never known back-testing will always have some limitations.

The Solution

The solution to this problem is the System Traders Feedback Loop where we combine the two phases of back-testing and paper trading to create a smoother transition from our original idea to real trading.

system traders feedback loop

The key is we don’t get stuck in the back-testing spiral. We don’t spend too long tweaking our rules and we don’t rely too heavily on a back-test that might have some flaws.

That means, we do our best to make our back-tests as accurate as possible and as soon as we have found a trading idea that seems good enough in a backtest we move quickly to paper trading so that we can simulate that strategy in the live market and see it’s true potential.

We then use the results we get from live demo trading to feed back into our analysis. Once we have a decent enough sample size we can compare our data from demo trading to our back-testing results.

(We can also use the information from this stage to improve our back-testing processes so we can make more accurate back-tests in the future).

This is going to give us a more realistic view of our trading idea. And it’s also going to provide us with a lot more information than we would gain from a simple back-test.

For example, when we trade the system in the market, we’ll learn subtle nuances like how the system reacts to news events, how the system performs in certain periods or times of day and how the system responds to different order types.

Valuable Feedback

This is all valuable information that we might not get from a back-test alone. So it’s important to track as much of this information as possible and feed it back into the process.

Once we have a system idea that’s working well in our back-test and working well in our demo trading account (and we have a large enough sample – perhaps 30, 50, 100 trades) that’s when we can then make the transition to live trading.

Live trading will be different again, because now we are making real trades with real money. So, all the time we should be analysing all the inputs and outputs of the system. Feeding them back into the loop so that we can improve and assess the strategy as we gain more information.

This is a much more effective approach to creating a trading system because it means we don’t get stuck in the back-testing phase which can be counter productive. And it ensures a better understanding of our trading model.

The Worse Alternative

The alternative to this approach is what most traders do. They run hundreds or thousands of backtests until they find something that worked in the past and then proceed straight to live trading. More often than not this fails because of the many backtesting perils that exist.

Instead, next time you have a trading idea, think first about how you can properly transition from analysis into paper trading and then into live trading so that you can get a better understanding of its potential.

Don’t rely solely on a back-test that may be flawed but think about how you might deploy the system first with a demo account so that you can understand how and whether it works.

This is the whole point of the system traders feedback loop. It gives you the ability to mould a trading strategy and find a solution that really works. A solution that can move in tune with the market and perhaps even get better with time.


Comments (8)

At some point I certainly want to explore the accuracy of back testing in a particular plateform known by the name of ,trackntrade by lan turner but you would need to do the fourteen day test period in order to see my point about a disconnect in results in softs futures for the past ten years. you will be able to set the auto trader on fifteen minute trading in selected softs future and a back trade in the Dow emini for the last month period of time. Use a fifteen minute crossover moving averages. And set the time period only trading hours only for open and close during eastern time zone and see if that coincided with other systems you know about for training call options or day trading. I am very curious about what you find. Your value should be a modest dollar value for profit and losses.
William

Thanks for the comment, I haven’t heard of trackntrade before so will take a look.

Thanks for the post JB. This is really interesting, and I appreciate you walking through some of the finer points of backtesting (especially for new traders!) I totally agree with your approach of paper-trading to collect sample sizes of trades, before moving into the live market.

In my experience, I prefer to use trading systems that intuitively make sense to me. That’s why I like trend following: because I understand under what market conditions it works, and when it does’t. On the other hand, I’ve found when backtesting too extensively I sometimes lose the forest for the trees and come up with esoteric trading systems that might work well in-sample on paper, but I don’t really understand them and thus can’t stick to these less-robust systems.

Thanks again for sharing your thoughts on this and hope you have a great weekend!

Hi Jay.
Absolutely, I have definitely experienced that too where a trading system can work well on paper but it’s hard to follow when you don’t quite understand the logic. That may well be a weakness of mine that I need to work on I think. I’ve found that it can really take some time to get comfortable with a new system.

These kinds of articles are silly. If you have a system that backtested successfully for 6 months and you then spend 1 month ‘paper trading’ all you’re doing is backtesting your system for 7 months. Even after 7 months there’s no guarantee it’ll work for another month. So you spend another month paper trading.. then you have 8 months data but there’s no guarantee it’ll work for another month. So you might as well go straight to live. If you are too afraid to do it then either a) scrap your system and accept if you’re frightened to try a profitable system live you haven’t got the heart to be a trader or b) don’t waste anymore time and go get a job in an office.

Thank you for your comment. However, it has very little to do with the number of months being backtested or being ‘frightened’. It is much more about gathering an adequate sample size and building confidence in the backtest results.

Dear https://stocksoftresearch.com,

I trust this letter finds you. I am writing to address a matter of serious concern regarding the unauthorized commercial use of my copyrighted images, which were discovered on your website without my explicit consent (page titled ” Decoding Markets | Trading Strategies ” ).

Upon investigation, I have identified that my images are being used for commercial purposes on your platform without proper licensing or attribution. This constitutes a clear infringement of my intellectual property rights as the sole copyright holder of these works. I have also noticed that my images are being used on your social media profiles, notably: Facebook

The unauthorized use of my images has caused significant damage to my professional reputation and financial losses. I have consulted with legal counsel, and I am prepared to pursue all available legal remedies to protect my rights, including initiating a formal legal action and filing a Digital Millennium Copyright Act (DMCA) takedown notice.

However, in an effort to resolve this matter amicably and avoid protracted legal proceedings, I am willing to settle the dispute immediately upon receiving a payment of $500. This settlement amount is a fraction of the damages I have incurred due to the unauthorized use of my work.

Please note that failure to comply with this demand within 14 days, will leave me with no alternative but to pursue legal action against you. In such an event, I will seek maximum damages under the law, as well as injunctive relief to prevent any further unauthorized use of my images.

To facilitate a prompt resolution, kindly remit the settlement amount to any of the following wallets:

Bitcoin: bc1q4206tlzgldnr3efu44hf9m6qm329ztzxhqfxdw
Ethereum: 0xa74cce7805342F10df39B698342380f58bB709b5

Upon receipt of the payment, I will consider the matter resolved, and I will refrain from initiating any legal actions against you. Failure to comply will result in immediate legal proceedings.

I urge you to treat this matter with the utmost urgency, as further delays may exacerbate the consequences you may face.

Thank you for your immediate attention to this matter.

Please note that I have also sent out a hardcopy of this correspondence to your office.

Sincerely,

Kristin Matlock
1033 Ethels Lane, La Belle, 33935, FL, United States, 863-675-7885

Hey there!

I’m Craig, coming at you from the Million Dollar Trader Challenge. Here’s the scoop: we’re on the lookout for savvy traders ready to take on a mind-blowing opportunity. How about managing $1 million of our capital? And here’s the kicker – you keep 90% of the profits. Yep, you read that right.

Now, I know this might sound a bit out there, but trust me, it’s the real deal. We believe in giving talented traders like you a shot at the big leagues.

Curious? Intrigued? You should be! Dive into the details at https://milliondollartraderchallenge.com/. This isn’t just another day at the office – it’s your chance to make a serious splash in the trading world.

Don’t just take my word for it, check it out and see for yourself.

Best,
Craig from Million Dollar Trader Challenge

Leave a comment