Precision,bias, and uncertainty for state population forecasts: an exploratory analysis of time series models

Many researchers have used time series models to construct population forecasts and prediction intervals at the national level, but few have evaluated the accuracy of their forecasts or the out-of-sample validity of their prediction intervals. Fewer still have developed models for subnational areas. In this study, we develop and evaluate six ARIMA time series models for states in the United States. Using annual population estimates from 1900 to 2000 and a variety of launch years, base periods, and forecast horizons, we construct population forecasts for four states chosen to reflect a range of population size and growth rate characteristics. We compare these forecasts with population counts for the corresponding years and find precision, bias, and the width of prediction intervals to vary by state, launch year, model specification, base period, and forecast horizon. Furthermore, we find that prediction intervals based on some ARIMA models provide relatively accurate forecasts of the distribution of future population counts but prediction intervals based on other models do not. We conclude that there is some basis for optimism regarding the possibility that ARIMA models might be able to produce realistic prediction intervals to accompany population forecasts, but a great deal of work remains to be done before we can draw any firm conclusions.     

Population forecasts and confidence intervals for sweden: a comparison of model-based and empirical approaches

This paper compares several methods of generating confidence intervals for forecasts of population size. Two rest on a demographic model for age-structured populations with stochastic fluctuations in vital rates. Two rest on empirical analyses of past forecasts of population sizes of Sweden at five-year intervals from 1780 to 1980 inclusive. Confidence intervals produced by the different methods vary substantially. The relative sizes differ in the various historical periods. The narrowest intervals offer a lower bound on uncertainty about the future. Procedures for estimating a range of confidence intervals are tentatively recommended. A major lesson is that finitely many observations of the past and incomplete theoretical understanding of the present and future can justify at best a range of confidence intervals for population projections. Uncertainty attaches not only to the point forecasts of future population, but also to the estimates of those forecasts' uncertainty.     

Stability over time in the distribution of population forecast errors

A number of studies in recent years have investigated empirical approaches to the production of confidence intervals for population projections. The critical assumption underlying these approaches is that the distribution of forecast errors remains stable over time. In this article, we evaluate this assumption by making population projections for states for a number of time periods during the 20th century, comparing these projections with census enumerations to determine forecast errors, and analyzing the stability of the resulting error distributions over time. These data are then used to construct and test empirical confidence limits. We find that in this sample the distribution of absolute percentage errors remained relatively stable over time and data on past forecast errors provided very useful predictions of future forecast errors.     

Insights from the Evaluation of Past Local Area Population Forecasts

Local area population forecasts have a wide variety of uses in the public and private sectors. But not enough is known about the errors of such forecasts, particularly over the longer term (20 years or more). Understanding past errors is valuable for both forecast producers and users. This paper (i) evaluates the forecast accuracy of past local area population forecasts published by Australian State and Territory Governments over the last 30 years and (ii) illustrates the ways in which past error distributions can be employed to quantify the uncertainty of current forecasts. Population forecasts from the past 30 years were sourced from State and Territory Governments. Estimated resident populations to which the projections were compared were created for the geographical regions of the past projections. The key features of past forecast error patterns are described. Forecast errors mostly confirm earlier findings with regard to the relationship between error and length of projection horizon and population size. The paper then introduces the concept of a forecast ‘shelf life’, which indicates how far into the future a forecast is likely to remain reliable. It also illustrates how past error distributions can be used to create empirical prediction intervals for current forecasts. These two complementary measures provide a simple way of communicating the likely magnitude of error that can be expected with current local area population forecasts.     

Can Knowledge Improve Population Forecasts at Subcounty Levels?

Recent developments in urban and regional planning require more accurate population forecasts at subcounty levels, as well as a consideration of interactions among population growth, traffic flow, land use, and environmental impacts. However, the extrapolation methods, currently the most often used demographic forecasting techniques for subcounty areas, cannot meet the demand. This study tests a knowledge-based regression approach, which has been successfully used for forecasts at the national level, for subcounty population forecasting. In particular, this study applies four regression models that incorporate demographic characteristics, socioeconomic conditions, transportation accessibility, natural amenities, and land development to examine the population change since 1970 and to prepare the 1990-based forecast of year 2000 population at the minor civil division level in Wisconsin. The findings indicate that this approach does not outperform the extrapolation projections. Although the regression methods produce more precise projections, the least biased projections are often generated by one of the extrapolation techniques. The performance of the knowledge-based regression methods is discounted at subcounty levels by temporal instability and the scale effect. The regression coefficients exhibit a statistically significant level of temporal instability across the estimation and projection periods and tend to change more rapidly at finer geographic scales.     

Interpolation and forecasting of population census data

Since population censuses are not annually implemented, population estimates are needed for the intercensal period. This paper describes simultaneous implementations of the temporal interpolation and forecasting of the population census data, aggregated by age and period. Since age equals period minus cohort, age-period-cohort decomposition suffers from the identification problem. In order to overcome this problem, the Bayesian cohort (BC) model is applied. The efficacy of the BC model for temporal interpolation is examined in comparison with official Japanese population estimates. Empirical results suggest that the BC model is expected to work well in temporal interpolation. Regarding the age-period-cohort decomposition of the Japanese census data, it is shown that the cohort effect is the largest while the other two effects are very small but not negligible. With regard to the forecasting of the Japanese population, the official population forecast considerably outperforms the BC forecast in most forecast horizons. However, the pace of increase in root mean square error for longer-term forecasting is larger in the official population forecast than in the BC forecasts. As a result, a variant of the BC forecast is best for 10-year forecast.     

Confidence intervals for postcensal state population estimates

This paper develops a methodology for constructing confidence intervals around postcensal state population estimates. Using regression equations, forecast intervals are derived around the average age-specific death rates over the postcensal estimation period. These results, combined with the number of postcensal deaths and the most current census counts, are translated into confidence intervals for the age structure. Two approaches are offered for constructing total population confidence intervals. One examines a simulated distribution while the other focuses on the mathematical derivation of population means and variances. The methodology is illustrated by deriving statistically defensible confidence intervals around the July 1, 1975 population of Florida.     

MAPE-R: a rescaled measure of accuracy for cross-sectional subnational population forecasts

Accurately measuring a population and its attributes at past, present, and future points in time has been of great interest to demographers. Within discussions of forecast accuracy, demographers have often been criticized for their inaccurate prognostications of the future. Discussions of methods and data are usually at the centre of these criticisms, along with suggestions for providing an idea of forecast uncertainty. The measures used to evaluate the accuracy of forecasts also have received attention and while accuracy is not the only criterion advocated for evaluating demographic forecasts, it is generally acknowledged to be the most important. In this paper, we continue the discussion of measures of forecast accuracy by concentrating on a rescaled version of a measure that is arguably the one used most often in evaluating cross-sectional, subnational forecasts, Mean Absolute Percent Error (MAPE). The rescaled version, MAPE-R, has not had the benefit of a major empirical test, which is the central focus of this paper. We do this by comparing 10-year population forecasts for U.S. counties to 2000 census counts. We find that the MAPE-R offers a significantly more meaningful representation of average error than MAPE in the presence of substantial outlying errors, and we provide guidelines for its implementation.     

Predictability,complexity, and catastrophe in a collapsible model of population,development, and environmental interactions

"More and more population forecasts are being produced with associated 95 percent confidence intervals. How confident are we of those confidence intervals? In this paper, we produce a simulated dataset in which we know both past and future population sizes, and the true 95 percent confidence intervals at various future dates. We use the past data to produce population forecasts and estimated 95 percent confidence intervals using various functional forms. We, then, compare the true 95 percent confidence intervals with the estimated ones. This comparison shows that we are not at all confident of the estimated 95 percent confidence intervals." (SUMMARY IN FRE)     

The forecast accuracy of Australian Bureau of Statistics national population projections

This paper examines the forecast accuracy of Australian Bureau of Statistics national population projections produced from the 1960s to the early 2000s. As well as total populations, the accuracy of the following is assessed: age-sex-specific populations, the Total Fertility Rate, life expectancy at birth and net international migration. It is shown that forecasts of the 1960s and 70s were the most inaccurate; forecasts of the 1980s and later proved to be much more reliable. The paper goes on to take an alternative perspective on population forecast error through the use of an adapted percentage error measure which accounts for offsetting errors in births, deaths, net migration and the jump-off population. This measure also permits an assessment of the relative contributions of the components of demographic change to overall inaccuracy. It is shown that errors in forecasting net international migration have generally contributed most to inaccuracy followed by births and then deaths and jump-off error. ABS projections of total population are also compared to those produced using a simple naïve model. The paper concludes by arguing that the new error measure could prove valuable in other studies of population forecast accuracy.     

Empirical Prediction Intervals for County Population Forecasts

Population forecasts entail a significant amount of uncertainty, especially for long-range horizons and for places with small or rapidly changing populations. This uncertainty can be dealt with by presenting a range of projections or by developing statistical prediction intervals. The latter can be based on models that incorporate the stochastic nature of the forecasting process, on empirical analyses of past forecast errors, or on a combination of the two. In this article, we develop and test prediction intervals based on empirical analyses of past forecast errors for counties in the United States. Using decennial census data from 1900 to 2000, we apply trend extrapolation techniques to develop a set of county population forecasts; calculate forecast errors by comparing forecasts to subsequent census counts; and use the distribution of errors to construct empirical prediction intervals. We find that empirically-based prediction intervals provide reasonably accurate predictions of the precision of population forecasts, but provide little guidance regarding their tendency to be too high or too low. We believe the construction of empirically-based prediction intervals will help users of small-area population forecasts measure and evaluate the uncertainty inherent in population forecasts and plan more effectively for the future.     

On the validity of MAPE as a measure of population forecast accuracy

The mean absolute percent error (MAPE) is the summary measure most often used for evaluating the accuracy of population forecasts. While MAPE has many desirable criteria, we argue from both normative and relative standpoints that the widespread practice of exclusively using it for evaluating population forecasts should be changed. Normatively, we argue that MAPE does not meet the criterion of validity because as a summary measure it overstates the error found in a population forecast. We base this argument on logical grounds and support it empirically, using a sample of population forecasts for counties. From a relative standpoint, we examine two alternatives to MAPE, both sharing with it, the important conceptual feature of using most of the information about error. These alternatives are symmetrical MAPE (SMAPE) and a class of measures known as M-estimators. The empirical evaluation suggests M-estimators do not overstate forecast error as much as either MAPE or SMAPE and are, therefore, more valid measures of accuracy. We consequently recommend incorporating M-estimators into the evaluation toolkit. Because M-estimators do not meet the desired criterion of interpretative ease as well as MAPE, we also suggest another approach that focuses on nonlinear transformations of the error distribution.     

Forecast accuracy of australian subnational population projections

Despite the considerable resources devoted to making demographic projections in Australia over the past two decades, there have been few attempts to evaluate the performance of these projections in terms of forecast accuracy. This paper first considers the role of accuracy amongst other objectives of projection activity. Accepting accuracy as a legitimate goal, we then assess the performance of 48 sets of population projections and forecasts for states and territories of Australia prepared since 1970. Projection accuracy is assessed by reference to length of forecast horizon, population size and rate of growth. We also examine the main sources of forecast error in selected projections for each state and compare the performance of past projections with alternatives based on simple extrapolation of contemporary population trends.     

An Evaluation of Population Estimates in Florida: April 1, 2000

The housing unit (HU) method is the most commonly used method for making small-area population estimates in the United States. These estimates are used for a wide variety of budgeting, planning, and analytical purposes. Given their importance, periodic evaluations of their accuracy are essential. In this article, we evaluate the accuracy of a set of HU population estimates for counties and subcounty areas in Florida, as of April 1, 2000. We investigate the influence of differences in population size and growth rate on estimation errors; compare the accuracy of several alternative techniques for estimating each of the major components of the HU method; compare the accuracy of 2000 estimates with that of estimates produced in 1980 and 1990; compare the accuracy of HU population estimates with that of estimates derived from other estimation methods; consider the role of professional judgment and the use of averaging in the construction of population estimates; and explore the impact of controlling one set of estimates to another. Our results confirm a number of findings that have been reported before and provide empirical evidence on several issues that have received little attention in the literature. We conclude with several observations regarding future directions in population estimation research.     

Measuring Residential Segregation With the ACS: How the Margin of Error Affects the Dissimilarity Index

The American Community Survey (ACS) provides valuable, timely population estimates but with increased levels of sampling error. Although the margin of error is included with aggregate estimates, it has not been incorporated into segregation indexes. With the increasing levels of diversity in small and large places throughout the United States comes a need to track accurately and study changes in racial and ethnic segregation between censuses. The 2005–2009 ACS is used to calculate three dissimilarity indexes (D) for all core-based statistical areas (CBSAs) in the United States. We introduce a simulation method for computing segregation indexes and examine them with particular regard to the size of the CBSAs. Additionally, a subset of CBSAs is used to explore how ACS indexes differ from those computed using the 2000 and 2010 censuses. Findings suggest that the precision and accuracy of D from the ACS is influenced by a number of factors, including the number of tracts and minority population size. For smaller areas, point estimates systematically overstate actual levels of segregation, and large confidence intervals lead to limited statistical power.     

An Empirical Analysis of the Effect of Length of Forecast Horizon on Population Forecast Errors

Many studies have found that population forecast errors generally increase with the length of the forecast horizon, but none have examined this relationship in detail. Do errors grow linearly, exponentially, or in some other manner as the forecast horizon becomes longer? Does the error-horizon relationship differ by forecasting technique, launch year, size of place, or rate of growth? Do alternative measures of error make a difference? In this article we address these questions using two simple forecasting techniques and population data from 1900 to 1980 for states in the United States. We find that in most instances there is a linear or nearly linear relationship between forecast accuracy and the length of the forecast horizon, but no consistent relationship between bias and the length of the horizon. We believe that these results provide useful information regarding the nature of population forecast errors.     

Spatial weighting improves accuracy in small-area demographic forecasts of urban census tract populations

Existing research in small-area demographic forecasting suffers from two important limitations: (1) a paucity of studies that quantify patterns of error in either total or age/sex-specific estimates and (2) limited methodological innovation aimed specifically at improving the accuracy of such forecasts. This paper attempts to fill, in part, these gaps in existing research by presenting a comparative evaluation of the accuracy of standard and spatially-weighted Hamilton–Perry forecasts for urbanized census tracts within incorporated New Mexico municipalities. These comparative forecasts are constructed for a 10-year horizon (base 1 April 2000 and target 1 April 2010), then compared to the results of the 2010 Census in an ex post facto evaluation. Results are presented for the standard Hamilton–Perry forecasts as well as two sets that incorporate two common variants of spatial weights to improve forecast accuracy. Findings are discussed in the context of what is currently known about error in small-area demographic forecasts and with an eye toward continued innovations.     

On the utility of population forecasts

Many customers demand population forecasts, particularly for small areas. Although the forecast evaluation literature is extensive, it is dominated by a focus on accuracy. We go beyond accuracy by examining the concept of forecast utility in an evaluation of a sample of 2,709 counties and census tracts. Wefind that forecasters provide “value-added” knowledge for areas experiencing rapid change or areas with relatively large populations. For other areas, reduced value is more common than added value. Our results suggest that new forecasting strategies and methods such as composite modeling may substantially improve forecast utility.     

Understanding the demographic implications of climate change: estimates of localized population predictions under future scenarios of sea-level rise

Significant advances have been made to understand the interrelationship between humans and the environment in recent years, yet research has not produced useful localized estimates that link population forecasts to environmental change. Coarse, static population estimates that have little information on projected growth or spatial variability mask substantial impacts of environmental change on especially vulnerable populations. We estimate that 20 million people in the United States will be affected by sea-level rise by 2030 in selected regions that represent a range of sociodemographic characteristics and corresponding risks of vulnerability. Our results show that the impact of sea-level rise extends beyond the directly impacted counties due to migration networks that link inland and coastal areas and their populations. Substantial rates of population growth and migration are serious considerations for developing mitigation, adaptation, and planning strategies, and for future research on the social, demographic, and political dimensions of climate change.     

Modeling and Forecasting Health Expectancy: Theoretical Framework and Application

Life expectancy continues to grow in most Western countries; however, a major remaining question is whether longer life expectancy will be associated with more or fewer life years spent with poor health. Therefore, complementing forecasts of life expectancy with forecasts of health expectancies is useful. To forecast health expectancy, an extension of the stochastic extrapolative models developed for forecasting total life expectancy could be applied, but instead of projecting total mortality and using regular life tables, one could project transition probabilities between health states simultaneously and use multistate life table methods. In this article, we present a theoretical framework for a multistate life table model in which the transition probabilities depend on age and calendar time. The goal of our study is to describe a model that projects transition probabilities by the Lee-Carter method, and to illustrate how it can be used to forecast future health expectancy with prediction intervals around the estimates. We applied the method to data on the Dutch population aged 55 and older, and projected transition probabilities until 2030 to obtain forecasts of life expectancy, disability-free life expectancy, and probability of compression of disability.