What do EWIs tell us?
(Extract from pages 42-43 of BIS Quarterly Review, March 2018)
This box explains how to read the table that assesses current vulnerabilities based on the set of early warning indicators (EWIs). Then it explains the limitations of those indicators in the context of a broader analysis of vulnerabilities,
To interpret the table entries, it helps to understand the methodology used to derive the critical thresholds that - if crossed - lead to a warning signal. For any indicator, we start off with a large sample spanning countries and time that ideally contains as many crises and non-crisis periods as possible. After checking whether the indicator has more EWI power than a coin toss,
we search over a range of potential thresholds that, when breached, issue a warning signal. We judge a crisis as correctly predicted if there is a warning signal at least once in the 12 quarters preceding the crisis, ie if the crisis occurs anytime within the three years following the breach. If a signal is issued but no crisis occurs within that time frame, we count this as a false alarm.
We choose two different thresholds to identify amber and red "alert zones". In both cases the calibration, drawing on historical experience, minimises the ratio of false alarms to correct warning signals (the "noise-to-signal ratio"). But one threshold is chosen so as to predict at least two thirds of the crises (red), and the other at least 90% (amber). The red threshold is more stringent (higher) in the sense that it is exceeded less often.
The cells also include asterisks (*). These refer to instances in which the combined behaviour of the corresponding debt and property price indicators signal vulnerabilities. For this debt-cum-property price combined indicator we follow a similar logic to the one above. We keep the property price gap threshold constant at its optimal standalone value and then optimise over the debt indicator threshold, so as to capture at least 66% of crises while minimising the noise-to-signal ratio. In other words, for a warning signal to be issued, we require that (i) the debt indicator breached the critical threshold and (ii) the property price gap was above 11 (the red threshold for the property price gap on its own) within the three years before the breach.
When this happens, we add an asterisk to the relevant EWI.
To interpret these signals correctly from a statistical viewpoint, a few points are worth recalling:
- Over the calibration period, there were naturally many instances in which the indicators breached the thresholds (corresponding to signals denoted by the amber, red and * identifiers) but crises did not materialise within the following three years. The more often this happens, the higher the noise-to-signal ratio.
- This may happen because crises do not materialise at all: the indicator subsequently switches off and imbalances correct themselves. Alternatively, it may happen because the signals may occur "too early" (eg five or six years before a crisis), with the indicator correctly continuing to signal risks until the crisis breaks out.
In general, even when the indicators identify the risk of crises correctly, it is unrealistic to expect them to identify the timing with any precision. - Noisy signals also mean that the statement "66% of crises were preceded by a breach of the EWI threshold" is not equivalent to "the crisis probability is 66% once the threshold is breached". Or putting it differently, the former statement says that "given that a crisis has occurred, the threshold was breached in 66% of the cases"; the latter means "given that the threshold is breached, a crisis occurs in 66% of the cases". The reason the two statements are not equivalent is that some breaches do not herald crises, ie the noise-to-signal ratio is higher than zero. In fact, in our sample and as a rule of thumb, the likelihood of a crisis emerging once the threshold for an indicator is breached is around 50%.
More generally, certain caveats need to be borne in mind:
- EWIs have only two settings: "on" or "off". They do not reflect the gradual intensification of a financial boom. (The use of two thresholds is designed to capture this to some extent.)
- The exact thresholds should not be overemphasised. We have run a battery of checks and drawn on other research to make sure our economic insights are as robust as possible. But the exact optimal thresholds identified can vary by a few percentage points across specifications. Given these uncertainties, whether an indicator is just above or below a threshold is not a first-order issue for monitoring purposes.
- EWIs are based on historical relationships. Thus, structural breaks may reduce their predictive power, eg as a result of increased use of macroprudential measures or changes in prudential regulation more broadly. This is only partly mitigated by evidence indicating that similar variables have displayed consistent predictive power going back to at least the 1870s (eg Schularick and Taylor (2012)).
- EWI thresholds are common across countries. Thus, they cannot take into account country-specific features. This is inevitable: as crises are rare events, it is not possible to calibrate the indicators with any statistical confidence based on the experience of any individual country.
- The EWIs displayed in the table are specifically designed to capture only vulnerabilities linked to the financial cycle. Other vulnerabilities that could lead to banking crises are not considered (eg sovereign crises owing to unsustainable fiscal positions).
Taken together, these caveats suggest that EWIs cannot be analysed in isolation. They are best seen as a useful starting point for a more granular assessment of vulnerabilities.
Formally, we test whether the AUC is statistically significantly different from 0.5. We use backward-looking information for residential property prices, as the associated gaps tend to close ahead of crises (Graph 1). For instance, this is the case for the credit-to-GDP gap (Drehmann et al (2011)). The derivation of how likely a crisis is given an EWI signal is much more sample-dependent than the thresholds shown in Table 2 because of small sample issues.