Low vol strategies
8 May 2020
Of Julia and R
8 May 2020
How to manipulate risk forecasts 101
30 April 2020
The five principles of correct riskometer use
27 April 2020
The problem with Backtesting
25 April 2020
The magic of riskometers
24 April 2020
Risk and scientific socialism
23 April 2020
Financial crises and epidemics
19 April 2020
Hayek and Corona
17 April 2020
Hayek et Corona
17 April 2020
Ignoring the Corona analysis
15 April 2020
The coronavirus crisis is no 2008
26 March 2020
Artificial intelligence as a central banker
6 March 2020
Systemic consequences of outsourcing to the cloud
2 December 2019
The dissonance of the short and long term
12 August 2019
Central banks and reputation risk
6 August 2019
The Brexit culture war
5 May 2019
All about BoB — The Bank of England Bot
29 April 2019
My tiny, tiny contribution to Apple's fall in profits
6 January 2019
The 2018 market in a 250 year context
1 January 2019
Short and long-term risk
3 December 2018
Perceived and actual risk
2 December 2018
Cryptocurrencies: Financial stability and fairness
9 November 2018
The October 2018 stock market in a historical context
1 November 2018
The hierarchy of financial policies
12 September 2018
Which numerical computing language is best: Julia, MATLAB, Python or R?
9 July 2018
Cryptocurrencies
26 June 2018
What are risk models good for?
3 June 2018
The McNamara fallacy in financial policymaking
1 June 2018
VIX, CISS and all the political uncertainty
20 May 2018
Here be dragons
30 March 2018
Low risk as a predictor of financial crises
26 March 2018
Cryptocurrencies don't make sense
13 February 2018
Yesterday's mini crash in a historical context
6 February 2018
Artificial intelligence and the stability of markets
15 November 2017
European bank-sovereign doom loop
30 September 2017
Do the new financial regulations favour the largest banks?
27 September 2017
The ECB Systemic Risk Indicator
24 September 2017
Finance is not engineering
22 September 2017
University of Iceland seminar
14 June 2017
Brexit and systemic risk
31 May 2017
Should macroprudential policy target real estate prices?
12 May 2017
Learning from history at LQG
13 April 2017
Is Julia ready for prime time?
12 March 2017
With capital controls gone, Iceland must prioritise investing abroad
12 March 2017
Competing Brexit visions
25 February 2017
Systemic consequences of Brexit
23 February 2017
Why macropru can end up being procyclical
15 December 2016
The fatal flaw in macropru: It ignores political risk
8 December 2016
Why it doesn't make sense to hold bonds
27 June 2016
On the financial market consequences of Brexit
24 June 2016
Cyber risk as systemic risk
10 June 2016
Big Banks' Risk Does Not Compute
24 May 2016
Interview on þjóðbraut on Hringbraut
21 May 2016
Farewell CoCos
26 April 2016
Will Brexit give us the 1950s or Hong Kong?
18 April 2016
Of Brexit and regulations
16 April 2016
IMF and Iceland
12 April 2016
Stability in Iceland
7 April 2016
Everybody right, everybody wrong: Plural rationalities in macroprudential regulation
18 March 2016
Of tail risk
12 March 2016
Models and regulations and the political leadership
26 February 2016
Why do we rely so much on models when we know they can't be trusted?
25 February 2016
Does a true model exist and does it matter?
25 February 2016
The point of central banks
25 January 2016
The macro-micro conflict
20 October 2015
Volatility, financial crises and Minsky's hypothesis
2 October 2015
Impact of the recent market turmoil on risk measures
28 August 2015
Iceland, Greece and political hectoring
13 August 2015
A proposed research and policy agenda for systemic risk
7 August 2015
Are asset managers systemically important?
5 August 2015
Objective function of macro-prudential regulations
24 July 2015
Risky business: Finding the balance between financial stability and risk
24 July 2015
Regulators could be responsible for next financial crash
21 July 2015
How Iceland is falling behind. On Sprengisandur
12 July 2015
Greece on Sprengisandur
12 July 2015
Why Iceland can now remove capital controls
11 June 2015
Market moves that are supposed to happen every half-decade keep happening
14 May 2015
Capital controls
12 May 2015
What do ES and VaR say about the tails
25 April 2015
Why risk is hard to measure
25 April 2015
Post-Crisis banking regulation: Evolution of economic thinking as it happened on Vox
2 March 2015
The Danish FX event
24 February 2015
On the Swiss FX shock
24 February 2015
Europe's proposed capital markets union
23 February 2015
What the Swiss FX shock says about risk models
18 January 2015
Model risk: Risk measures when models may be wrong
8 June 2014
The new market-risk regulations
28 November 2013
Solvency II: Three principles to respect
21 October 2013
Political challenges of the macroprudential agenda
6 September 2013
Iceland's post-Crisis economy: A myth or a miracle?
21 May 2013
The capital controls in Cyprus and the Icelandic experience
28 March 2013
Towards a more procyclical financial system
6 March 2013
Europe's pre-Eurozone debt crisis: Faroe Islands in the 1990s
11 September 2012
Countercyclical regulation in Solvency II: Merits and flaws
23 June 2012
The Greek crisis: When political desire triumphs economic reality
2 March 2012
Iceland and the IMF: Why the capital controls are entirely wrong
14 November 2011
Iceland: Was the IMF programme successful?
27 October 2011
How not to resolve a banking crisis: Learning from Iceland's mistakes
26 October 2011
Capital, politics and bank weaknesses
27 June 2011
The appropriate use of risk models: Part II
17 June 2011
The appropriate use of risk models: Part I
16 June 2011
Lessons from the Icesave rejection
27 April 2011
A prudential regulatory issue at the heart of Solvency II
31 March 2011
Valuing insurers' liabilities during crises: What EU policymakers should not do
18 March 2011
Risk and crises: How the models failed and are failing
18 February 2011
The saga of Icesave: A new CEPR Policy Insight
26 January 2010
Iceland applies for EU membership, the outcome is uncertain
21 July 2009
Bonus incensed
25 May 2009
Not so fast! There's no reason to regulate everything
25 March 2009
Modelling financial turmoil through endogenous risk
11 March 2009
Financial regulation built on sand: The myth of the riskometer
1 March 2009
Government failures in Iceland: Entranced by banking
9 February 2009
How bad could the crisis get? Lessons from Iceland
12 November 2008
Regulation and financial models: Complexity kills
29 September 2008
Blame the models
8 May 2008

The problem with Backtesting

25 April 2020

Backtesting is the magical technique that tells us how well a forecast model works. Test the model on history, and we have an objective way to evaluate how good the model is. But does it really work in practice?

How well do riskometers perform?

Suppose you are a junior analyst in a bank and you go to your boss with your new fancy riskometer.

Boss: “Hey Junior analyst, how good is your riskometer?”

You: “Hey, boss. Wait five years, we will observe how it works operationally and then we will know.”

I think the next thing you will see is a pink slip.

So instead, we resort to backtesting. If you want a more technical discourse, hop over to my lecture slides on backtesting from my book Financial risk forecasting.

Imagine you have daily data on a stock from 1 January 2000, and you are running a riskometer that needs 1000 observations or four years of data. You then use the 1000 days from 1 January 2000 until 31 December 31 2003 to forecast risk for 1 January 2004. You can now compare how accurate the forecast is to actually what happened that day.

You then move up by one day, to forecast risk for 2 January 2004, and repeat until today, giving you 4330 days of risk forecasts and prices.

That gives you a large sample to test how good your riskometer is. Perhaps use Peter Cristoffersen’s coverage test if you are using Value-at-risk, and simply count the number of violations and compare them to what you expect. If you don’t like the coverage test, there are many others to choose from.

If doing 99% VaR, 4330 observations mean expected 43 violations, plenty for a robust test.

So all is good? No. There are two problems:

  1. Backtests look at how a riskometer forecasts history when the operator knows the history and has a vested interest in either passing or rejecting a riskometer.

  2. We can backtest a riskometer relative to a benchmark. However, we have to assume the benchmark, and it has to be relevant for our objective.

The first problem is that anybody who designs a riskometer knows what happened. If I’m testing it over the 4330 days since 1 January 2004, I know that 2004, 2005 and 2006 were low volatility — low risk periods, and a financial crisis hit in autumn of 2008. I can tailor pick a riskometer that will perform particularly well over this testing period. I don’t have to run it to find out, any experienced designer of riskometers will just know which to pick.

And that automatically biases the backtest.

It’s a problem that would be familiar to the historians of the Soviet Union, who become experts in forecasting history because history justified decisions and the survival of the historian depended on how well their history supported the Kremlin.

The second problem is more complex. We have to test the riskometer relative to a benchmark, but what is to say the benchmark is relevant to us? If it is a coverage test, we count violations, but maybe what matters more is the clustering of violations or magnitude or something else.

Ultimately, backtests aren’t scientific as I discussed recently

That doesn’t mean they aren’t useful, just take them with a grain of salt, and please don’t base your market risk regulations on them.

© All rights reserved, Jon Danielsson, 2020