Forecast Evaluation in the Presence of Unobserved Volatility

A number of volatility forecasting studies have led to the perception that the ARCH- and Stochastic Volatility-type models provide poor out-of-sample forecasts of volatility. This is primarily based on the use of traditional forecast evaluation criteria concerning the accuracy and the unbiasedness of forecasts. In this paper we provide an analytical assessment of volatility forecasting performance. We use the volatility and log volatility framework to prove how the inherent noise in the approximation of the true- and unobservable-volatility by the squared return, results in a misleading forecast evaluation, inflating the observed mean squared forecast error and invalidating the Diebold-Mariano statistic. We analytically characterize this noise and explicitly quantify its effects assuming normal errors. We extend our results using more general error structures such as the Compound Normal and the Gram-Charlier classes of distributions. We argue that evaluation problems are likely to be exacerbated by non-normality of the shocks and that non-linear and utility-based criteria can be more suitable for the evaluation of volatility forecasts.     

Forecast Evaluation in the Presence of Unobserved Volatility

Abstract

A number of volatility forecasting studies have led to the perception that the ARCH- and Stochastic Volatility-type models provide poor out-of-sample forecasts of volatility. This is primarily based on the use of traditional forecast evaluation criteria concerning the accuracy and the unbiasedness of forecasts. In this paper we provide an analytical assessment of volatility forecasting performance. We use the volatility and log volatility framework to prove how the inherent noise in the approximation of the true- and unobservable-volatility by the squared return, results in a misleading forecast evaluation, inflating the observed mean squared forecast error and invalidating the Diebold–Mariano statistic. We analytically characterize this noise and explicitly quantify its effects assuming normal errors. We extend our results using more general error structures such as the Compound Normal and the Gram–Charlier classes of distributions. We argue that evaluation problems are likely to be exacerbated by non-normality of the shocks and that non-linear and utility-based criteria can be more suitable for the evaluation of volatility forecasts.     

FORECASTING VOLATILITY IN THE PRESENCE OF MODEL INSTABILITY

Recent advances in financial econometrics have allowed for the construction of efficient ex post measures of daily volatility. This paper investigates the importance of instability in models of realised volatility and their corresponding forecasts. Testing for model instability is conducted with a subsampling method. We show that removing structurally unstable data of a short duration has a negligible impact on the accuracy of conditional mean forecasts of volatility. In contrast, it does provide a substantial improvement in a model's forecast density of volatility. In addition, the forecasting performance improves, often dramatically, when we evaluate models on structurally stable data.     

Uncertain population forecasting

"Errors in population forecasts arise from errors in the jump-off population and errors in the predictions of future vital rates. The propagation of these errors through the linear (Leslie) growth model is studied, and prediction intervals for future population are developed. For U.S. national forecasts, the prediction intervals are compared with the U.S. Census Bureau's high-low intervals." In order to assess the accuracy of the predictions of vital rates, the authors "derive the predictions from a parametric statistical model and estimate the extent of model misspecification and errors in parameter estimates. Subjective, expert opinion, so important in real forecasting, is incorporated with the technique of mixed estimation. A robust regression model is used to assess the effects of model misspecification."     

Some Problems in Forecasting Population for Government Planning Purposes

Some governments rely on centralized, official sets of population forecasts for planning capital facilities. But the nature of population forecasting, as well as the milieu of government forecasting in general, can lead to the creation of extrapolative forecasts not well suited to long-range planning. This report discusses these matters, and suggests that custom-made forecasts and the use of forecast guidelines and a review process stressing forecast assumption justification may be a more realistic basis for planning individual facilities than general-purpose, official forecasts.     

Combination forecasting for directional accuracy: An application to survey interest rate forecasts

Using published interest rates forecasts issued by professional economists, two combination forecasts designed to improve the directional accuracy of interest rate forecasting are constructed. The first combination forecast takes a weighted average of the individual forecasters' predictions. The more successful the forecaster was in past forecasts at predicting the direction of change in interest rates, the greater is the weight given to his/her current forecast. The second combination forecast is simply the forecast issued by the forecaster who had the greatest success rate at predicting the direction of change in interest rates in previous forecasts. In cases where two or more forecasters tie for best historic directional accuracy track record, the arithmetic mean of these forecasters is used. The study finds that neither combination forecasting method performs better than coin-flipping at predicting the direction of change in interest rates. Nor does either method beat the simple arithmetic mean of the predictions of all the forecasters surveyed at predicting the direction of change in interest rates.     

Multivariate time series projections of parameterized age-specific fertility rates

1 solution to the dimensionality problem raised by projection of individual age-specific fertility rates is the use of parametric curves to approximate the annual age-specific rates and a multivariate time series model to forecast the curve parameters. Such a method reduces the number of time series to be modeled for women 14-45 years of age from 32 to 40 (the number of curve parameters). In addition, the curves force even longterm fertility projections to exhibit the same smooth distribution across age as historical data. The data base used to illustrate this approach was age-specific fertility rates for US white women in 1921-84. An important advantage of this model is that it permits investigation of the interactions among the total fertility rate, the mean age of childbearing, and the standard deviation of age at childbearing. In the analysis of this particular data base, the contemporaneous relationship between the mean and standard deviation of age at childbearing was the only significant relationship. The addition of bias forecasts to the forecast gamma curve improves forecast accuracy, especially 1-2 years ahead. The most recent US Census Bureau projections have combined a time series model with longterm projections based on demographic judgment. These official projections yielded a slightly higher ultimate mean age and slightly lower standard deviation than those resulting from the model described in this paper.     

Forecasting multivariate longitudinal binary data with marginal and marginally specified models

Forecasting with longitudinal data has been rarely studied. Most of the available studies are for continuous response and all of them are for univariate response. In this study, we consider forecasting multivariate longitudinal binary data. Five different models including simple ones, univariate and multivariate marginal models, and complex ones, marginally specified models, are studied to forecast such data. Model forecasting abilities are illustrated via a real-life data set and a simulation study. The simulation study includes a model independent data generation to provide a fair environment for model competitions. Independent variables are forecast as well as the dependent ones to mimic the real-life cases best. Several accuracy measures are considered to compare model forecasting abilities. Results show that complex models yield better forecasts.     

The Promise of Prediction Contests

This article examines the prediction contest as a vehicle for aggregating the opinions of a crowd of experts. After proposing a general definition distinguishing prediction contests from other mechanisms for harnessing the wisdom of crowds, we focus on point-forecasting contests—contests in which forecasters submit point forecasts with a prize going to the entry closest to the quantity of interest. We first illustrate the incentive for forecasters to submit reports that exaggerate in the direction of their private information. Whereas this exaggeration raises a forecaster's mean squared error, it increases his or her chances of winning the contest. And in contrast to conventional wisdom, this nontruthful reporting usually improves the accuracy of the resulting crowd forecast. The source of this improvement is that exaggeration shifts weight away from public information (information known to all forecasters) and by so doing helps alleviate public knowledge bias. In the context of a simple theoretical model of overlapping information and forecaster behaviors, we present closed-form expressions for the mean squared error of the crowd forecasts which will help identify the situations in which point forecasting contests will be most useful.     

Kalman filter-based modelling and forecasting of stochastic volatility with threshold

We propose a parametric nonlinear time-series model, namely the Autoregressive-Stochastic volatility with threshold (AR-SVT) model with mean equation for forecasting level and volatility. Methodology for estimation of parameters of this model is developed by first obtaining recursive Kalman filter time-update equation and then employing the unrestricted quasi-maximum likelihood method. Furthermore, optimal one-step and two-step-ahead out-of-sample forecasts formulae along with forecast error variances are derived analytically by recursive use of conditional expectation and variance. As an illustration, volatile all-India monthly spices export during the period January 2006 to January 2012 is considered. Entire data analysis is carried out using EViews and matrix laboratory (MATLAB) software packages. The AR-SVT model is fitted and interval forecasts for 10 hold-out data points are obtained. Superiority of this model for describing and forecasting over other competing models for volatility, namely AR-Generalized autoregressive conditional heteroscedastic, AR-Exponential GARCH, AR-Threshold GARCH, and AR-Stochastic volatility models is shown for the data under consideration. Finally, for the AR-SVT model, optimal out-of-sample forecasts along with forecasts of one-step-ahead variances are obtained.     

Scenarios, Uncertainty and Conditional Forecasts of the World Population

Current official population forecasts differ little from those that Whelpton made 50 years ago either in the cohort–component methodology used or in the arguments used to motivate the assumptions. However, Whelpton produced some of the most erroneous forecasts of this century. This suggests that current forecasters should ensure that they give users an assessment of the uncertainty of their forecasts. We show how simple statistical methods can be combined with expert judgment to arrive at an overall predictive distribution for the future population. We apply the methods to a world population forecast that was made in 1994. Accepting that point forecast, we find that the probability is only about 2% that the world population in the year 2030 will be less than the low scenario of 8317 million. The probability that the world population will exceed the high scenario of 10 736 million is about 13%. Similarly, the probability is only about 51% that the high–low interval of a recent United Nations (UN) forecast will contain the true population in the year 2025. Even if we consider the UN high–low intervals as conditional on the possible future policies of its member states, they appear to have a relatively small probability of encompassing the future population.     

Functional time series approach for forecasting very short-term electricity demand

This empirical paper presents a number of functional modelling and forecasting methods for predicting very short-term (such as minute-by-minute) electricity demand. The proposed functional methods slice a seasonal univariate time series (TS) into a TS of curves; reduce the dimensionality of curves by applying functional principal component analysis before using a univariate TS forecasting method and regression techniques. As data points in the daily electricity demand are sequentially observed, a forecast updating method can greatly improve the accuracy of point forecasts. Moreover, we present a non-parametric bootstrap approach to construct and update prediction intervals, and compare the point and interval forecast accuracy with some naive benchmark methods. The proposed methods are illustrated by the half-hourly electricity demand from Monday to Sunday in South Australia.     

Term structure forecasting in affine framework with time-varying volatility

This study extends the affine Nelson–Siegel model by introducing the time-varying volatility component in the observation equation of yield curve, modeled as a standard EGARCH process. The model is illustrated in state-space framework and empirically compared to the standard affine and dynamic Nelson–Siegel model in terms of in-sample fit and out-of-sample forecast accuracy. The affine based extended model that accounts for time-varying volatility outpaces the other models for fitting the yield curve and produces relatively more accurate 6- and 12-month ahead forecasts, while the standard affine model comes with more precise forecasts for the very short forecast horizons. The study concludes that the standard and affine Nelson–Siegel models have higher forecasting capability than their counterpart EGARCH based models for the short forecast horizons, i.e., 1 month. The EGARCH based extended models have excellent performance for the medium and longer forecast horizons.     

Improved autoregressive forecasts in the presence of non-normal errors

This paper is concerned with obtaining more accurate point forecasts in the presence of non-normal errors. Specifically, we apply the residual augmented least-squares (RALS) estimator to autoregressive models to utilize the additional moment restrictions embodied in non-normal errors. Monte Carlo experiments are performed to compare our RALS forecasts to forecasts based on the ordinary least-squares estimator and the least absolute deviations (LAD) estimator. We find that the RALS approach provides superior forecasts when the data are skewed. Compared to the LAD forecast, the RALS forecast has smaller mean squared prediction errors in the baseline case with normal errors.     

Cointegration and Long-Horizon Forecasting

We consider the forecasting of cointegrated variables, and we show that at long horizons nothing is lost by ignoring cointegration when forecasts are evaluated using standard multivariate forecast accuracy measures. In fact, simple univariate Box–Jenkins forecasts are just as accurate. Our results highlight a potentially important deficiency of standard forecast accuracy measures—they fail to value the maintenance of cointegrating relationships among variables—and we suggest alternatives that explicitly do so.     

Predictive Accuracy Gain From Disaggregate Sampling in ARIMA Models

We compare the forecast accuracy of autoregressive integrated moving average (ARIMA) models based on data observed with high and low frequency, respectively. We discuss how, for instance, a quarterly model can be used to predict one quarter ahead even if only annual data are available, and we compare the variance of the prediction error in this case with the variance if quarterly observations were indeed available. Results on the expected information gain are presented for a number of ARIMA models including models that describe the seasonally adjusted gross national product (GNP) series in the Netherlands. Disaggregation from annual to quarterly GNP data has reduced the variance of short-run forecast errors considerably, but further disaggregation from quarterly to monthly data is found to hardly improve the accuracy of monthly forecasts.     

Birth forecasting based on birth order probabilities,with application to U.S. data

"A model for birth forecasting based on prediction of the so-called 'birth order probabilities' is constructed. The relation between this model and recent models of fertility prediction is derived. Birth forecasts with approximate probability limits for the U.S. for the period 1983-1997 are generated. The performance of the proposed model in predicting future fertility is tested by fitting time series models to part of the available series (1917-1982) and ultimately generating birth forecasts for the remainder of the period, then comparing these forecasts with the actual data." The accuracy of the fertility forecasts made are compared with those made by other methods.     

Univariate ARIMA Forecasts of Defined Variables

Many important variables in business and economics are neither measured nor measurable but are simply defined in terms of other measured variables. For instance, the real interest rate is defined as the difference between the nominal interest rate and the inflation rate. There are two ways to forecast a defined variable: one can directly forecast the variable itself, or one can derive the forecast of the defined variable indirectly from the forecasts of the constituent variables. Using Box-Jenkins univariate time series analysis for four defined variables—real interest rate, money multiplier, real GNP, and money velocity—the forecasting accuracy of the two methods is compared. The results show that indirect forecasts tend to outperform direct methods for these defined variables.     

Optimal Forecasts from Markov Switching Models

We derive forecasts for Markov switching models that are optimal in the mean square forecast error (MSFE) sense by means of weighting observations. We provide analytic expressions of the weights conditional on the Markov states and conditional on state probabilities. This allows us to study the effect of uncertainty around states on forecasts. It emerges that, even in large samples, forecasting performance increases substantially when the construction of optimal weights takes uncertainty around states into account. Performance of the optimal weights is shown through simulations and an application to U.S. GNP, where using optimal weights leads to significant reductions in MSFE. Supplementary materials for this article are available online.     

Forecasting Contemporaneously Aggregated Vector ARMA Processes

Given a multiple time series that is generated by a multivariate ARMA process and assuming the objective is to forecast a weighted sum of the individual variables, then under a mean squared error measure of forecasting precision, it is preferable to forecast the disaggregated multiple time series and aggregate the forecasts, rather than forecast the aggregated series directly, if the involved processes are known. This result fails to hold if the processes used for forecasting are estimated from a given set of time series data. The implications of these results for empirical research are investigated using different sets of economic data.