Managing Ex-Ante Tracking Error
MANAGING EX-ANTE TRACKING ERROR IN A DYNAMIC MARKET ENVIRONMENT
Jean Paul van Straalen - Vice President
Quantitative Strategy Group - ABN AMRO Asset Management
Introduction
Tracking errors are calculated to measure active portfolio risks, to set risk limits and for risk budgeting purposes within the investment process. Practitioners have recently become concerned about the deviation between the realised (ex-post) tracking errors of their portfolios and their predicted (ex-ante) tracking error levels. This is not surprising, given the volatility swings in the last couple of years. The investment industry at large has criticised the accuracy of the risk forecasts that risk factor models in general provided. The general opinion is that the risk models underestimate the realised portfolio risks. We therefore believe it is important to investigate how well the ex-ante tracking error predicts the ex-post tracking error and to make suggestions on how we should manage ex-ante tracking errors going forwards. In our analysis, we use the BARRA Global Equity Model (GEM) to calculate the ex ante tracking error.
Definition Of Tracking Error
The relative risk profile of a portfolio versus a specific benchmark is measured by the tracking error (or active risk). There are two different ways to measure the tracking error, on an ex-ante and ex-post basis.The ex-ante tracking error is a statistical measure that is defined as the forecast annualised standard deviation of the active returns of a portfolio relative to a pre-defined benchmark. An ex-ante tracking error of 2% indicates that there is a two-thirds probability that the portfolio returns will fall within +/- 2% of the benchmark return over the next year (see graph 1).
Source: BARRA, Global Equity Handbook
A higher ex-ante tracking error means there is a higher probability that the portfolio return will deviate from the benchmark return. It reflects the riskiness of the portfolio versus the benchmark. In general, the ex-ante tracking error is a function of the portfolio weights, benchmark weights, the volatility of the stocks and the correlation across stocks.
Source: ABN AMRO Asset Management
The ex-post tracking error measures the time series standard deviation of the realised active returns..
The deviation between ex-ante and ex-post tracking error therefore depends on the accuracy of the estimation of the volatility of stocks and the stability of the correlation across these stocks within the risk model. In the literature, there are several other explanations for the underestimation of the TE of investors’ portfolios. Pope and Yadav (1994) indicate that autocorrelation (momentum) of excess returns is a reason for the underestimation. Hwang and Satchell (2001) demonstrate that ex-ante and ex-post tracking errors differ because portfolio weights are stochastic in nature. Brooks et al (2000) and Gardner et al (2002) add model specification (noise) and model dynamics (eg. new trends) as explanations for the underestimation of the predicted risks. Other explanations commonly found in practice are changes in market volatility and specific risk.
Data And Methodology
BARRA’s GEM risk model uses data from 1988 onwards to calculate stock volatility and correlations. However, the model has an exponential weighting scheme that gives greater importance to recent data by using a half-life of 48 months. Data from 48 months ago thus receives half the weight of the most recent figures. This modelling approach reflects the importance of most recent data, while also including older data in the forecasts.
To test the accuracy of the BARRA risk model, we create random portfolios from a specified universe and calculate ex-ante and ex-post tracking errors for these portfolios.
We restrict our universe to the 877 companies that were included in the MSCI World Index for all our sample period, which starts on 30 June 1998 and ends on 30 June 2003. From this universe, we simulate 50 portfolios consisting of 75 randomly selected companies. The constituents in the simulated portfolios are equally weighted and we keep their weights constant through time. The same methodology applies to the benchmark, which comprise the same 877 companies. For each portfolio, we calculate the month-end ex-ante tracking error and the realised excess performance of the portfolio that is generated in the following month. We repeate this exercise for each month in our sample period (61 observations).
The accuracy of the risk model is analysed using two tests. The first test compares average ex-ante tracking errors with ex-post tracking errors, and the relationship between the market volatility and the ex-post tracking error.
1. For all companies, we use the monthly price returns from FactSet.
The second test is the so-called bias test, which is a statistical test used by BARRA to determine the under- and overestimation of the predicted versus the realised tracking errors.
Analysis 1: Ex-Post Versus Ex-Ante Tracking Errors
Graph 2 presents a frequency distribution of the ex-ante and ex-post tracking errors of the 50 simulated portfolios. Meanwhile, graph 3 features a regression analysis on the ex-ante versus the ex-post tracking errors. The ex-ante tracking error is calculated as the average of the 61 monthly annualised ex-ante tracking errors. The ex-post tracking error is calculated as the standard deviation of the 61 monthly active returns. Multiplying this standard deviation by Ö12 gives us the annualised ex-post tracking error.
Results
It is clear from graph 2 that the ex-post tracking errors are more dispersed than their ex-ante counterparts. Thirty six of the 50 ex-ante tracking errors fall within the 3.50–4.25% interval. The difference with the ex-post is large, as only 9 out 50 fall within the same interval. For the remaining portfolios, BARRA seems to underestimate and overestimate the active risks. At tracking errors lower than 3.5%, realised risks are overestimated, as 13 ex-post portfolio tracking errors fall within this range compared with only four ex-ante. Meanwhile, the underestimation of risk is evident from the 28 portfolios that have ex-post tracking errors higher than 4% against only ten portfolios that had this predicted tracking error.
To evaluate the ex-ante versus ex-post relationship in statistical terms, we regressed the ex-ante numbers on the ex-post tracking errors (graph 3). The slope of the regression is 1.28 and suggests that risk forecasts are lower than realised risks. However, the slope has a t-value of 1.05 and is therefore not significantly different from one. The constant is not significantly different from zero. This analysis reveals that the risk forecasts for our simulated portfolios in our sample period have not underestimated the realised risks to a significant degree.
From the frequency distribution and the regression analysis, we found that the model underestimates the active risk for some portfolios and overestimates risks for others, but has no significant bias over the total sample period. The next step was to investigate the relationship between ex-ante and ex-post numbers through time and to see if there is a relationship between market volatility and the accuracy of our risk forecasts.
To investigate the relationship between ex-ante and ex-post numbers through time, we calculated the rolling 12-month ex-post tracking error for each portfolio. For each month, we then took the average of the 50 simulated portfolios. We compared this average with the average ex-ante tracking error predicted at the start of the rolling 12-month window . The results are presented in graph 4.
First, there is a gradual increase over time of the ex-ante tracking error. This may be due to the slow adaptation of the risk forecast to the increasing market volatility . Second—and more important—there are periods when actual risks are underestimated and periods when these risks are overestimated. The average 12-month rolling ex-post tracking error had its highest level at about 4.5% in June 2000, while in June 1999 the tracking error forecast was only 3.5%. In September 2002, however, the average ex-post tracking error was 3.5%, while the average ex-ante forecast was around 4.25%. The model thus overestimated risk.
To find out how market volatility affects the accuracy of our risk forecasts, we plot a dispersion measure as a proxy for market volatility. For each month, the dispersion was calculated as the standard deviation of the returns of the 877 equally weighted companies. We then average this standard deviation over 12 months to arrive at a 12-month rolling standard deviation (the dotted line in graph 4) .
The correlation between the volatility of the benchmark returns (dispersion) and the ex-post tracking error of the portfolios is—as expected—high. When the realised benchmark volatility is high (low), the ex-post tracking errors are at high (low) levels and risks are under (over) estimated. This clearly shows how the realised volatility of the benchmark affects the accuracy of the risk forecasts in our sample period. It also indicates that the risk model is good at predicting long-term risks but is not so successful in predicting short-term risks due to changing levels of dispersion.
Analysis 2: The Bias Test
Connor (2000) describes that the bias test, developed by BARRA, plays an important role in testing the robustness of risk forecasts. The bias test evaluates the ability of the model to forecast the active risk for a portfolio over a pre-specified period .
2. At June 1999, the ex-post tracking error is the annualised standard deviation of the active monthly returns over the 12-month period starting 1 July 1998. We compare this with the ex-ante tracking error estimate made at the end of June 1998.
3. Half life of 48 months.
4. The dispersion is then a rough estimate of stock specific risk.
5. The assumption is made that the constituents of the portfolios and the benchmark are constant through time. We recognise that frequent rebalancing involves trading costs, but we abstract from this, as it is not a factor captured in our risk model.
For each simulated portfolio, the active risk forecast and the realised active return were combined into a standardised outcome. We had a sample of T (=61) realised active returns for each portfolio: rt, where t=1,…,T, and T standard deviations: σt , where t=1,…,T. The active risk forecast at time t, σt, was calculated by dividing the annual ex-ante tracking error at time t by √12. The standardised outcome at time t was defined as the ratio rt/σt.
The bias statistic is the standard deviation of the T standardised outcomes:
where m is the sample mean of rt/σt.
In this bias test, the null hypothesis is that the BARRA active risk forecasts are unbiased estimates of the deviation of active returns for the simulated equity portfolios. The expected value of the bias statistic is one when the null hypothesis holds true. If the bias statistic is greater than one, this indicates that the active risk has been underestimated, while a number less than one signals that the active risk has been overestimated. In order to test the null hypothesis, we tested whether the bias statistic was significantly different from its expected value of one. For this analysis, we used the 95% confidence interval. The boundaries of this confidence interval were calculated by:
In our analysis, T = 61, which translates into a confidence interval of [0.82, 1.18]. When the bias statistic falls within this interval, the realised active risk is in line with the active risk forecast.
Results
We calculated 50 bias statistics for our simulated portfolios over the total sample period. In graph 5, we present the distribution of our bias statistic analysis.
Within our sample period, 39 portfolios have a bias statistic that falls within the confidence interval [0.82, 1.18]: for these portfolios, the forecast active risk from BARRA is an unbiased prediction of the realised active risk. For nine portfolios, the bias statistic is significantly higher than one, so the model underestimated the risks. In two cases, the BARRA risk model overestimated the actual risk. The nine cases where risks are underestimated leads to some concern about the accuracy of the risk forecasts made. However, we believe this finding needs to be evaluated in the context of high and changing levels of volatility, as discussed earlier on. Because the volatility level is a common factor that affects our total sample, we suspect this to be the main reason for the cases where TE underpredictions are found.
Interpretations And Considerations
We can draw some general conclusions from our analysis. First, in our sample period, ex-post tracking errors were more volatile than ex-ante tracking errors. Over time, however, ex-post tracking errors were sometimes higher and sometimes lower than predicted. More importantly, we found that the realised volatility of the benchmark (dispersion) is related to the ex-post tracking error and therefore to the accuracy of the risk forecast. Second, the outcome of the bias statistic confirms that for the large majority of portfolios (39), the model estimates risk correctly. For nine portfolios, risks were underestimated, while for two they were overestimated. In general, we were satisfied with the accuracy of the risk model forecasts for the 50 simulated portfolios, if we take into account that the sample period was relatively volatile (given the build-up and collapse of the IT bubble) compared with historical standards.
We discussed some explanations for the general finding that TE’s are underestimated in practice. We believe there are two other possible explanations for portfolio managers having had ex-post tracking errors higher than predicted by the risk model. The first explanation is the concentration in systematic risk factors in a portfolio. Equity portfolios are not only exposed to stock specific risks, but also to systematic risk factors such as country, sector or style risks. The extent to which these systematic risk factors influence performance may not have been fully captured by the BARRA risk model. Portfolios with a concentration in active exposures to these risk factors are especially subject to more uncertainty about their risk forecasts. This may explain why ex-post tracking errors have been higher for some portfolios than predicted. In our analysis, the process of randomly generating portfolios ensures that, on average, the portfolios had no active exposures towards these systematic risk factors. Our risk estimates are therefore unbiased and show that the model does a fairly accurate job in predicting active risks.
The second explanation confirmed by our analysis, is changes in volatility. One of the reasons for a deviation between the ex-ante and ex-post tracking errors is the estimation of volatility in the BARRA risk model. Although recent data has more weight in the model, an increase in current volatility will not immediately impact the correlation matrix and the forecast volatility, as the estimation procedure incorporates a long data history. We therefore expect ex-post tracking errors to exceed their ex-ante counterparts. On the other hand, a decline in volatility is also not reflected instantaneously. In a period of declining volatility, we therefore expect ex-post numbers to be lower than their forecasts. Our simulation results have confirmed this result.
Graph 6 shows that the forecast volatility of the BARRA risk model (exponentially weighted) lags the changes in volatility of the MSCI World. We may now be heading into a period where the realised volatility is lower than the predicted one. This is important for the interpretation of ex-ante tracking errors going forwards.
Conclusion
From our simulation results, we found that the BARRA risk model does a fairly accurate job in forecasting long-term portfolio risks, when we consider the relatively high volatility in our sample period. However, the model seems less accurate for short-term predictions. This is because the BARRA model adapts slowly to changing levels of volatility. We believe this has the following consequences:
- The BARRA risk model is a reliable and indispensable tool for setting risk limits and conducting risk budgeting exercises within a long-term orientated investment process.
- The impact of trending or volatile risk factor returns on the ex-post performance of portfolios with a concentration of risk in one or more factors can be substantial. For these cases, managing the ex-ante tracking error more closely is prudent.
- In the short term, the ex-ante tracking error can deviate significantly from the ex-post tracking error. We therefore have to include the current market dynamics and the market’s underlying implied volatility in our evaluation of the ex-ante tracking error. In the past few months, volatility has dropped significantly, almost to the point where it is below the BARRA estimates. We therefore expect that some ex-ante tracking errors may actually be overestimating current risks. If the risk tolerance is stable, this means portfolio managers can increase ex-ante tracking errors and use more of their assigned risk budgets to add value.
LITERATURE
Salomon Smith Barney (1999). “A guide to tracking error analysis”, April 1999.
BARRA, The Global Equity Handbook.
Tierens I and Kierspel A. (2003). “How much “error” in tracking error?” Goldman Sachs, Index and Derivatives Perspective, April 2003.
Tierens I and Kierspel A. (2003). “How much “error” in tracking error? The link with changes in stock volatility and cross–stock dispersion”, Goldman Sachs, Index and Derivatives Perspective, July 2003.
Hwang S and Satchell S. (2001). “Tracking Error: ex ante versus ex post measures”. Journal of Asset Management, volume 2 number 3.
Gardner, Bowie, Brooks and Cumberworth (2000). “Predicted Tracking Errors: Fact or Fantasy?”, Portfolio Risk and Performance Working Party, Faculty and Institute of Actuaries Investment Conference June 2000.
Brooks, Beukes, Gardner and Hibbert (2002). “Predicted Tracking Errors – The Search Continues”. Investment Risk Working Party, Faculty and Institute of Actuaries Finance and Investment Conference June 2002.
Connor G. (2000). “Robust Confidence Intervals for Barra’s Bias Test of Risk Forecasts”. Research Article February 2000.
Pope Y and Yadav P K (1994). “Discovering errors in tracking error”. Journal of Portfolio Management, Winter.
Jean Paul van Straalen - Vice President
Quantitative Strategy Group - ABN AMRO Asset Management
Entry Filed under: Asset Management