David Bailey, Research Fellow, Department of Computer Science, University of California
A group of researchers have taken issue with the misuse of quantitative techniques, claiming there is a lack of stringent mathematical diligence in testing investment strategies. The paper, "Pseudo- mathematics and financial charlatanism", goes so far as to equate the effects of backtest overfitting on out of sample performance to fraud, though with the caveat that such actions could be inadvertent.
Irrespective of its economic and financial viability, a strategy in its first stages is unproven. To help determine real-world performance, it undergoes backtesting: multiple simulations of the strategy are run against historical financial to derive a host of performance metrics. In the absence of scrupulous due diligence, the metrics may not indicate an optimal strategy at all, rather are biased by statistical flukes or false positives.
When a strategy or its parameters are tweaked multiple times against the same initial data set, or the in sample set, the problem of overfitting, or confusing noise with signal, arises. David Bailey, a research fellow at the Department of Computer Science in the University of California and one
of the four authors of the paper, explained that "when a statistical test is applied multiple times, the probability of obtaining false positives increases as a function of the number of trials".
ELEPHANT IN THE ROOM
Similarly, model complexity - when multiple parameters are added to the strategy in order to achieve the most favourable result - is also a concern. That's because by using enough input variables, any result can be obtained; a fact described by renowned mathematician John von Neumann, who said: "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk."
Even if the basis of the strategy is false, there exists a configuration for that strategy which yields an outstanding performance with in sample data. Any model that is specifically tailored for such a data set is unlikely to perform similarly against different out of sample data or in live markets.
And so, many strategies fail on go live.
It is imperative, once a model and its parameters are extrapolated from the in sample set, to run through an out of sample test to ensure that the observations pertaining to the model can be generalised. But even out of sample testing does not prevent overfitting. The reason is that one can repeat those tests as many times as needed to obtain a desirable outcome.
The key is to control for the number of trials involved in a particular finding to determine whether the result is a fluke.
Marcos Lopez de Prado, a research affiliate at Lawrence Berkeley National Laboratory, also author of the paper, adds another dimension to the problem of obtaining trustworthy results...