Introduction Prediction Mean Square Error Prediction Intervals Empirically Based P.I.s Summary

Interval Forecasting

Based on Chapter 7 of the Time Series Forecasting by Chatfield

Econometric Forecasting, January 2008

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Prediction Mean Square Error Prediction Intervals Empirically Based P.I.s Summary Outline

1 Introduction

2 Prediction Mean Square Error

3 Prediction Intervals

4 Empirically Based P.I.s

5 Summary

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Introduction Prediction Mean Square Error Terminology Prediction Intervals Interval Forecasts Empirically Based P.I.s Density Forecast Summary Fan Chart Introduction Introduction

Most forecasters realize the importance of providing interval forecasts and point forecasts in order to: asses future uncertainty, enable different strategies to be planned for the range of possible outcomes indicated by the interval forecast, compare forecasts from different methods more thoroughly.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Introduction Prediction Mean Square Error Terminology Prediction Intervals Interval Forecasts Empirically Based P.I.s Density Forecast Summary Fan Chart Introduction Terminology

An interval forecast usually consists of an upper and lower limit. The limits are called forecast limits, prediction bounds. The interval is called a confidence interval, forecast region, prediction interval.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Introduction Prediction Mean Square Error Terminology Prediction Intervals Interval Forecasts Empirically Based P.I.s Density Forecast Summary Fan Chart Introduction Predictions

Predictions: are not often a prediction intervals, most often are given as a single value.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Introduction Prediction Mean Square Error Terminology Prediction Intervals Interval Forecasts Empirically Based P.I.s Density Forecast Summary Fan Chart Introduction Interval Forecasts

Reasons for not using interval forecasts: rather neglected in statistical literature, no generally accepted method for calculating prediction intervals (with some exception), theoretical prediction intervals are difficult or impossible to evaluate for many econometric models containing many equations or which depend on non-linear relationship.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Introduction Prediction Mean Square Error Terminology Prediction Intervals Interval Forecasts Empirically Based P.I.s Density Forecast Summary Fan Chart Introduction Density Forecast

By density forecast we mean finding the entire probability distribution of a future value of interest. Linear models with normally distributed innovations: the density forecast is typically normal with mean equal to the point forecast and variance equal to that used in computing a predicion interval. Linear model without normally distributed innovations: seems to be more prevalent and used e.g. in forecasting volatility, different percentiles or quantiles of the conditional probability distribution of future values are estimated.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Introduction Prediction Mean Square Error Terminology Prediction Intervals Interval Forecasts Empirically Based P.I.s Density Forecast Summary Fan Chart Introduction Fan Chart

Fan charts

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Prediction Mean Square Error

Preditcion Mean Square Error (PMSE):

2 E [en(h) ]. Forecast is unbiased that is, when xˆN (h) is the mean of the predictive distriburion, then

2 E [eN (h)] = 0 and E [eN (h) ] = Var[eN (h)].

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Prediction Mean Square Error Potential Pitfall

Assesing forecast uncertainty, remember:

the var of the forecast 6= the var of the forecast error.

Given data up to time N and a particular method or model:

the forecast xˆN (h) is not a random variable, it has variance of zero,

XN+h and eN (h) are random variables, condtioned by the observed data.

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Prediction Mean Square Error

How to evaluate:

2 E [eN (h) ] or Var[eN (h)] and what assumptions should be made?

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Prediction Mean Square Error Example

Consider the zero-mean AR(1):

2 Xt = αXt−1 + εt , {εt } ∼ N(0, σε ).

2 Assume: complete knowledge of the model (α and σε ). The point forecasts xˆN (h) will be

h h αˆ xN rather than α xN .

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Prediction Mean Square Error Bias of PMSE

Even assuming that the true model is known a priori, there will still be biases in the usual estimate obtained by substituting sample estimates of the model parameters and the residual variance into the true-model PMSE formula.

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Known vs. Unknown Parameters

Consider the case of h = 1 and conditioning on Xn = xn,

eN (1) = XN+1 − xˆN (1). If the model parameters were known, then

xˆN (1) = αxN ⇒ eN (1) = εN+1 If parameters are not known:

eN (1) = XN+1 − xˆN (1) = αxN + εN+1 − αˆxN = (α − αˆ)xN + εN+1 Assume: parameters estimates are obtained by a procedure that is asymptotically equivalent to maximum likelihood.

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Prediction Mean Square Error Conditional and Unconditional Errors

Looking once more at equation

eN (1) = (α − αˆ)xN + εN+1. Consider:

conditional on xN forecast error: xN is fixed αˆ - biased estimator of α, then the expected value of eN (1) need not be zero. unconditional forecast error: If, however, we average over all possible values of xN , as well as over εN , then it can be shown that te expectation will indeed be zero giving an unbiased forecast.

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Prediction Mean Square Error Computing PMSE

Important when computing PMSE. To have unconditional PMSE: average over the distribution of future innovations (e.g. eN+1), average over the distribution of current observed values (e.g. xN ).

eN (1) = (α − αˆ)xN + eN+1

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Prediction Mean Square Error Unconditional PMSE

Unconditional PMSE can be usefull to assess the ’success’ of a forecasting method on average. This apporach if used to compute P.I.s, it effectively assumes that te observatoins used to estimate the model parameters are independent of those used to construct the forecasts. This assumption can be justified asymptotically. Box assesed that the correction terms 1 would generally be of order N (effect of parameter uncertainty).

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Parameters Uncertainty Correction

Assume: K-variable vector AR(p) process. True model PMSE at lead time one has to be multiplied by the correction factor

1 1 [1 + K N ] + o( N ) to give the corresponding unconditional PMSE allowing for parameters uncertainty. What does it mean? the more parameters (K ), the shorter the series (N), ⇒ the greater will be the correction term.

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Parameters Uncertainty Example

N = 50, K = 1 and p = 2

the correction for the square root of PMSE is only 2%. for

N = 30, K = 3 and p = 2

the correction for the square root of PMSE rises to 6%.

Pekalski, Swierczyna, Zalewski interval forecasting Formula Introduction Potential Pitfall Prediction Mean Square Error Example Prediction Intervals Known vs. Unknown Parameters Empirically Based P.I.s Conditioning Forecast Error Summary Example Parameters Uncertainty Example

The effect on probabilities. For normal distribution

95% lie in the interval +/ − 1.96.

Suppose the s.d. is 6% larger (no correction factor used). Now,

96.2% lie in the interval +/ − (1.96 × 1.06) = 2.0776.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary Calculating P.I.

In general P.I.s are of the form:

100(1 − α)%.

P.I. for XN+h is given by: p xˆN (h) + / − zα/2 Var[eN (h)].

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary Formula for P.I.

Properties and assumptions of the formula for P.I.:

symmetric about xˆN (h), assumes the forecast is unbiased with PMSE 2 E [eN (h) ] = Var[eN (h)], forecast errors are assumed to be normally distributed.

Note: some authors state that the zα/2 should be replaced by the precentage point of a t-distribution, with appropriate number of degrees of freedom (worth making for less than 20 obs).

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary Formula for P.I.

The formula p xˆN (h) + / − zα/2 Var[eN (h)]. is generally used for P.I.s. preferably after checking the assumptions (e.g. forecast errors are approximately normally distributed) are at least reasonably satisfied. For any given forecasting method the main problem will then lie with evaluating Var[eN (h)].

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary P.I.s derived from a fitted probability model Formulas for PMSE

PMSE can be derived for: ARMA models (also seasonal and integrated), structural state-space, various regression models (typically allow for parameter uncertainty and are conditional in the sense that they depend on the particular values of the explanatory varialbes from where a prediction is being made). Cannot be derived: some complicated simultaneous equation, non-linear.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary P.I.s without model identification

What to do when a forecasting method is selected without any formal model identification procedure ? assume that the method is optimal (in some sense) apply some empirical procedure

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary P.I.s when assumed that method is optimal Example

Exponential smoothing: no obvious trend or seasonality, no attempt to identify the underlying model (i.e. ARIMA(0,1,1)). PMSE formula:

2 2 Var[eN (h)] = [1 + (h − 1)α ]σε .

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary P.I.s when assumed that method is optimal When it is reasonable ?

When it is reasonable? Observed one-step-ahead forecast errors show no obvious autocorrelation. No other obvious features of the data (e.g. trend) which need to be modeled.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary Forecasting methods not based on a probability model

Assume that the method is optimal in the sense that the one-step ahead errors are uncorrelated. Easy to check by looking at the correlogram of the one-step-ahead errors: if there is correlation we have more structure in the data which should improve the forecast.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary Forecasting methods not based on a probability model Example

Holt-Winters method with

additive and multiplicative seasonality

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary The additive case

Properties: results equivalent to SARIMA model for which additive Holt-Winters is optimal, so complicated that it would never be identified in practice.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Calculating P.I.s Prediction Mean Square Error Probability Model Prediction Intervals Non Formal Model Empirically Based P.I.s Example Summary The multiplicative case

No ARIMA model for which the method is optimal. Assume: one-step-ahead forecast errors are uncorrelated. Results:

Var[eN (h)] not monotonic increase with h, P.I.s are wider near a seasonal peak as would intuitively be expected. Remark: Wider P.I.s near a seasonal peak - not captured by most alternative approaches (except variance-stabilizing transformation).

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Empirically based P.I.s

When to use empirically based P.I.s? theoretical formulae not available, doubts about validity of the true model. Remark: computationally intensive based on: observed distribution of errors, simulation or resample methods.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Empirically based P.I.s 1st method

Apply forecasting method to all the past data. Find within-sample ’forecasts’ at 1, 2, 3,... steps ahead (from all available time origins). Find the variance of these errors (at each lead time over the periods of fit). Assume normality.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Empirically based P.I.s 1st method

Result: approximate empirical 100(1 − α)% P.I. for XN+h is given by p xˆN (h) + / − zα/2 Var[eN (h)]. Problems: if N small - assume t-distribution,

long series is needed to get reliable values for se,h, smooth values to make them increase monotonically with h p values of Var[eN (h)] based on in-sample residuals not on out-of-sample forecast errors, results comparable to theoretical formulae (if available).

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Empirically based P.I.s 2nd method

Split data into two parts. Fit on 1st part. Forecast on 2nd part. Get prediction errors. Refit model - move the fitting window.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Simulation and resampling methods

More computationally intensive approach. Increasingly used for the construction of P.I.s.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Simulation and resampling methods Simulation (Monte Carlo approach)

Assumption: Probability time-series model is known (and identified correctly) Generate random innovations Generate possible past and future values Repeat many times Find the interval within which the required percentage of future values lie

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Simulation and resampling methods Resampling (bootstrapping)

Sample from the empirical not theoretical distribution

⇒ distribution-free approach

The idea (the same as for simulation): use the knowledge about the primary structure of the model generate a sequence of possible future values find a P.I. containing the appropriate percentage of future values

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Bootstrapping Brief description

N independent observations, take random sample of size N with replacement. Result: some values occur twice (or more) some not occur at all In time-series: makes no sense - correlation over time

Bootstrap by resampling the fitted errors - depend on model.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Bootstrapping Properties

Properties of bootstrap: Bootstrap P.I.s are useful non-parametric alternative to the usual Box-Jenkins intervals. It is difficult to resample correlated data.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction When to use? Prediction Mean Square Error 1st method Prediction Intervals 2nd method Empirically Based P.I.s Simulation and Resampling Summary Uncertainty in Forecasts

Sources of uncertainty in forecasts from econometric models: the model innovations, having estimates of model parameters rather than true values, having forecasts of exogenous variables rather than true values, misspecification of the model.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Prediction Mean Square Error Prediction Intervals Empirically Based P.I.s Summary Summary Findings and Recomendations

Summarized main findings and recommendations: Formulate a model, that provides a reasonable apporx for the process generating a given series, derive PMSE, and use the formula. Distinction between a forecasting method and a forecasting model should be borne in mind. The former may, or may not, depend (explicitly or implicitly) on the latter. Use not model but method based approach (e.g. the Holt-Winters method).

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Prediction Mean Square Error Prediction Intervals Empirically Based P.I.s Summary Summary Findings and Recomendations

No theoretical formulae, or there are doubts about model assumptions, use the empirically based approach. The reason why out-of-sample forecasting ability is worse than within-sample fit is that the wrong model may have been identified or may change through time. p The formula xˆN (h) + / − zα/2 Var[eN (h)] assumes: model has been correctly identified, innovations are normally distributed, the future will be like the past. Rather than compute P.I.s based on a single ’best’ model, use Bayesian model averaging, or not model-based approach.

Pekalski, Swierczyna, Zalewski interval forecasting Introduction Prediction Mean Square Error Prediction Intervals Empirically Based P.I.s Summary The End

Thank you for your attention.

Pekalski, Swierczyna, Zalewski interval forecasting