SR/SSD 2004-09

Technical Memorandum

MAV/FWC Model Output and Forecaster Temperature Verification For NWFO Shreveport’s Forecast Area.

Michael Berry (Senior Forecaster)
Bill Murrell (Forecaster)
Shreveport, Louisiana

1. Introduction

Forecasters have at their discretion an array of tools and techniques available for composing a forecast. The most important of these tools include the National Weather Service prediction models which are made available to forecast offices through the National Centers for Environmental Prediction (NCEP). Some of these models, in addition to providing an array of mesoscale and synoptic scale parameters, offer model output statistics (MOS). This guidance provides maximum and minimum forecast temperatures for the next 48 hrs for a large number of U.S. cities. The most common of the numerical guidance products, the NGM based MOS guidance, or FWC, has been available for quite some time and its limitations are well known. AVN/GFS based MOS guidance, or MAV, has been made available to forecasters only since May, 2000. This new guidance was developed due to an upgrade of the IBM Class 8 supercomputer at NCEP and it’s limitations are still to be determined.

Brooks and Doswell (1996) point out that verification of weather forecasts is an essential part of any forecasting system. They also indicate that producing forecasts without verifying them is an admission that the quality of the forecasts is a low priority. In order to help improve forecasters’ skill in forecasting maximum and minimum temperatures versus model guidance temperatures, forecasters need to become familiar with any model bias or trend that a model may exhibit over a particular time of the year. This paper will attempt to determine how well the newer MAV temperature guidance performs against not only FWC, but also against the forecasters maximum and minimum temperature for four cities in the NWFO Shreveport’s forecast area of responsibility.

Methodology

Forecasters at NWFO Shreveport verify maximum and minimum temperature forecasts for four cities in their area of responsibility. These cities include Lufkin, Texas, Texarkana, Arkansas and Shreveport and Monroe, Louisiana. The maximum and minimum forecast temperatures produced by the forecasters in the coded cities forecast (CCF) were collected for each of these cities over a 12 month period, beginning July 1st 2001 and ending June 30th 2002. MAV MOS temperature equations change on April 1st and on October 1st, so the data set was split into warm and cold seasons. (Meteorological Development Laboratory, (n.d.)., MOS: Frequently Asked Questions) The data set for the warm season began April 1st and ended September 30th while the cold season data set began October 1st and ended March 31st. Throughout the data set, CCF temperatures were compared to the 0000 UTC and 1200 UTC MAV and FWC MOS model temperature output. It was then determined if the guidance and CCF temperatures were above or below the actual observed temperature for each city. Once this determination was made, the results were then entered into a table. Because the degree of accuracy needed to be known to determine how well the forecaster and the guidance performed, the tables for each city consisted of thirteen different temperature range bins (Table 1).

Table 1: Forecast Temperature Verification Table

The temperature range bins or error rates were set at two degree ranges until a miss of six degrees was reached. These smaller range bins from one to six degrees were necessary to denote a majority of the forecast misses. For example, a one degree miss can be denoted from a three degree miss and likewise a three degree miss from a five degree miss. Above or below a six degree departure from the exact temperature, the range bin increases to three degrees from a departure of seven to ten degrees and a range of four degrees from a departure of eleven to fifteen degrees. A forecast temperature miss of sixteen degrees or greater constituted an extreme forecast temperature miss or the extreme outer range bins while a mark in the zero degree range bin resulted in the MOS or forecaster (CCF) forecasting the maximum or minimum temperature for a given location and period exactly correct. Which bin received a mark above or below the zero bin was dependent upon the degree of forecast accuracy. For example, if a forecasted temperature was 6 degrees above the observed value for the forecaster, yet 3 degrees above for both the MAV MOS and FWC MOS, a mark was made in the 6-5 degree range bin under the CCF column, with marks made in each of the 3-4 degree range bins under the MAV and FWC columns.

Forecasters often weigh their ability to forecast maximum and minimum temperatures verses the guidance available to them. Therefore, this study used percent improvement to determine how forecasters fared against FWC and MAV MOS. (% IMP) is defined as :

where Er represents the absolute error generated by the reference forecast system and Ef is the absolute error from the forecaster (Brooks and Doswell, 1996). A positive %IMP means the forecaster improved on guidance while a negative %IMP means the guidance was better than the forecaster.

This study also looked at how many “accurate” temperature forecasts were made by the models and forecasters. This was studied because a bias either warm or cold is not necessarily good or bad unless it is significant. A two-degree or less temperature disparity was chosen to be the threshold for an “accurate” forecast. This corresponded to the ASOS temperature sensor accuracy of within 1.8 degrees (ASOS Site Technical Manual, chapter 1).

3. Verification Results

A. Cool Season Minimum Temperatures...(October - March)

Forecasters outperformed MAV and FWC MOS guidance in the first period, but MAV MOS outperformed both the forecasters and FWC MOS for periods two through four (Fig. 1). The average percent improvement over FWC MOS was 13.5%, while the average percent improvement over MAV MOS was -0.8%.

All three performers showed a warm bias over the first two periods (Fig 3). FWC MOS continued this trend into periods three and four, while the forecasters and MAV MOS showed a cold bias for the same periods. MAV MOS however had a much more significant cold bias in forecasting minimum temperature for periods three and four.

Cool season minimum temperature accuracy for the forecasters, FWC and MAV MOS, was highest in the first period (Fig. 7). Both the forecasters and MAV MOS made an “accurate” forecast 52% of the time vs. FWC MOS’s 48%. By the fourth period, “accurate” forecasts had dropped to 43% for the forecasters with a 1 percent improvement for MAV MOS with 44% and FWC MOS faring the worst with 38%.

B. Cool Season Maximum Temperatures...(October - March)

The forecasters outperformed the MAV and FWC MOS maximum temperatures during the cool season (Oct - Mar) through all four periods (Fig. 1). The average percent improvement over FWC MOS was 15.2% while the average percent improvement over MAV MOS was 4.6%. The highest forecaster improvement of 19.4% was found in the first period versus FWC MOS.

After examining the cool season maximum temperature distribution (Fig. 4), the forecasters and FWC MOS showed a warm bias during the first two periods. The MAV MOS showed a cool bias through all four periods, particularly the third and fourth periods where MAV MOS was under forecasting maximum temperatures by almost a 2 to1 margin.

The accuracy rate for the forecasters, FWC and MAV MOS was found to be highest in the first period (Fig. 7). The forecasters made an “accurate” forecast 66% of the time, compared to 63% for MAV MOS and 55% for FWC MOS. Less “accurate” forecasts were expected beyond the first period and by the fourth period, the accuracy rate dropped to 46% for the forecasters, 41% for MAV MOS and 40% for FWC MOS.

C. Warm Season Minimum Temperatures...(April - September)

Forecasters outperformed FWC MOS for all four periods but only outperformed MAV MOS over the first two periods (Fig. 2). The average percent improvement over the FWC was 10.7%, while the average percent improvement with the MAV was actually -2.0%.

All performers showed a cool bias forecasting warm season minimum temperatures for all four periods (Fig. 5).

The forecasters and MAV MOS showed nearly the same number of “accurate” forecasts (Fig. 8). In the first period, forecasters had 72% “accurate” forecasts while MAV MOS came in a close second with 71% and FWC MOS with 64%. By the fourth period, the MAV MOS overtook the forecasters with 65% “accurate” forecasts with the forecasters two percent lower at 63%, and the FWC MOS with 59%.

D. Warm Season Maximum Temperatures...(April - September)

Forecasters outperformed MAV and FWC MOS in the first period, but MAV MOS performed
slightly better during the third and fourth periods (Fig. 2). The average percent improvement over FWC MOS was 16.0%, while the average percent improvement over MAV MOS was much lower at 3.5%. This average percent improvement over MAV MOS was a bit misleading since the percent improvement went from 16.86% for the first period, to only 0.2% in the second period. MAV MOS actually performed better in the third period, -2.0% and the fourth period, -1.2% respectively.

All three performers experienced a warm bias for the first three periods (Fig. 6). No bias could be determined by either the forecasters or MAV MOS for the fourth period while FWC MOS continued its warm bias experienced during the first three periods.

Once again, the performers showed more “accurate” forecasts during the first period with forecasters outperforming both MAV and FWC MOS by almost 10 percent (Fig. 8). Forecasters had an “accurate” forecast 76% of the time while the MAV MOS and FWC MOS had 67% and 66% respectively. The forecasters continued to do better even into the fourth period with 60% “accurate” forecasts with MAV MOS at 58% and FWC MOS last with 53%.

E. Forecast Implications...

GFS MOS outperformed FWC MOS for both the cool and warm season. CCF was markedly better than FWC MOS through all four periods in both seasons. CCF showed an improvement over GFS MOS for both seasons in the first period and in the second period for cool season maximum temperatures and warm season minimum temperatures (Fig. 1, 2). GFS MOS verification performance showed a slight improvement over CCF verification in the third and fourth period for both seasons (Fig. 1, 2). This improvement may imply that forecasters have a tendency to place more emphasis on the first 24 hours of the forecast. The results show that more emphasis needs to be placed on the 24 to 48 hour time period of the forecasts in order for forecaster accuracy vs GFS MOS to improve.

Forecaster accuracy vs MOS was highest during the warm season when forecasting first period maximum temperatures (Fig. 8). Forecaster accuracy showed a 17 percent improvement in this period. Forecaster accuracy vs GFS MOS was negligible for the remaining periods in both the warm and cool seasons. This paper did not attempt to research model biases with respect to airmass changes in either the warm season or cool season. Additional research into model biases or trends could serve to improve forecaster accuracy.

4. Conclusion

Over a 12 month period, beginning July 1st, 2001 and ending June 30th, 2002, a temperature data set from four cities was collected. The data set consisted of forecast maximum and minimum temperatures from NWS Shreveport forecasters (CCF), FWC MOS and for the newly developed GFS MOS. Because of a change in the temperature equation that GFS MOS undergoes twice a year, this data set was divided into a warm and cool season.

During the warm season, guidance as well as forecasters predicted too much of a diurnal swing with a cool bias for morning lows and a warm bias for afternoon highs. All three performers had more “accurate” forecasts during the warm season compared to the cool season. This is what is to be expected as the warm season has fewer temperature extremes (more persistence) than the cool season. It should be noted that these results were concluded with the combining of all four cities data sets. Individual city verification could show different results in respect to MOS and CCF accuracy as well as model biases.

Even though MAV MOS did much better than FWC MOS overall, this study cannot always recommend MAV MOS over FWC MOS as each model performed differently under various synoptic and mesoscale weather patterns. Perhaps a more accurate assessment of FWC MOS and MAV MOS biases could have been made if maximum and minimum temperature verification was correlated with airmass changes.

5. References

Automated Surface Observing System Site Technical Manual S100, March 1997: System Overview. Silver Springs, Maryland.

Brooks, Harold, E, and Doswell, Charles, A, 1996: A Comparison of Measures-Oriented and
Distributions-Oriented Approaches to Forecast Verification. Weather and Forecasting,
11, 288-303.

Meteorological Development Laboratory, (n.d.). MOS: Frequently Asked Questions. Retrieved March 8, 2003, from http://www.nws.noaa.gov/mdl/synop/faq.htm.