On the use of mean square error and directional forecast accuracy for model selection: a Monte Carlo investigation

  • Kunst, R. (Speaker)
  • Mauro Costantini (Contributor)

Activity: Talks and presentationsTalk or oral contributionScience to Science


We propose a new procedure for model selection based on simultaneously targeting the mean square error and directional forecast accuracy criteria. The procedure combines the two types of accuracy measures using a weighting scheme for the selection of the forecasting models. Monte Carlo analysis under different scenarios serves as a tool that assesses the strength of the procedure. To this end, we consider various time series models as generation mechanisms, in particular time-homogeneous univariate and vector autoregressions but also
generating laws that involve thresholds and structural breaks. We evaluate two aspects of forecast model specification: forecast model selection chooses one out of two rival models that are both evaluated over a training sample, whereas forecast model combination determines weights on an average of the two rival models from the training performance. For the evaluation of the training samples, we study rolling and expanding forecast windows and a combination of the two. Although results are quite heterogeneous across designs, we generally find that finding reliable tools for improving directional accuracy is difficult
and that a price must be paid by deteriorating performance such as measured by the mean square error.
Period13 Jul 2022
Event title42nd International Symposium on Forecasting
Event typeConference
LocationOxford, United KingdomShow on map
Degree of RecognitionInternational


  • Forecasting
  • model selection
  • Monte Carlo simulation