The NMCGPH94Q (94Q) is the man-made AFOS graphic product from the Hydrometeorological Prediction Center of the National Center for Environmental Protection (NCEP). It is a graphical prediction of the 24-hour mean areal precipitation (MAP) for the contiguous 48 United States.
For years many operational personnel in the National Weather Service have been looking at the 94Q graphic product, often on a daily basis. Several questions often come to mind, such as: How accurate is this product on the local level? Does it work well for all seasons or just certain seasons? Does the product work well for heavy rains, or just light rainfall situations? Is the product only accurate for synoptic-scale, meteorological scenarios?
Now that most of the Weather Forecast Offices (WFOs) in the Southern Region are making daily, local Quantitative Precipitation Forecasts (QPFs), the following questions may also be asked: How dependable is the 94Q product which is available to forecasters for the development of local QPFs? Do local forecasters need to improve upon, and can they improve upon, the 94Q product? Does the 94Q exhibit a "dry" or "wet" bias? Do the predictions exhibit more skill in certain rainfall ranges or categories?
For over two-and-one-half years, the HAS Forecasters at the West Gulf River Forecast Center (WGRFC) have made daily hard copies of the 94Q product. Also, the HAS Forecasters have collected over two-and-one-half years of mean areal precipitation data for the Dallas-Fort Worth Metroplex. An objective study has been performed on the collected data to determine the accuracy of the 94Q product on a local level rather than the national level.
This paper will describe the results of this study and answer the aforementioned questions.
The data consist of the 94Q product and local MAPs collected for the period April 1, 1994, through September 30, 1996. A total of 892 94Qs have been collected and evaluated. MAPs were calculated from the Brainmaker Rainfall Network (Hall, 1996). The MAPs are derived from 36 rain gauge 24-hr reports (1200 UTC to 1200 UTC) located in Dallas and Tarrant Counties of North Central Texas. The MAP generated by the Rainfall Network is simply the algebraic average of all the rainfall reports. The 36 rain gauges are located over 1720 sq mi as shown in Fig. 1.
Although a bit subjective, each of the 94Qs had to be evaluated manually. Tabulations were made for each rainfall category from 0 to 3.0 in. There were no 4.0 or 5.0-in QPFs made during the study period for the verification domain.
For the purpose of this study, accuracy equals skill (S), which was defined as S= R/T, where R is the number of correct forecasts and T is the total number of forecasts. The results of all the 94Qs were then compared to the MAP that occurred on a daily basis, and later on a monthly basis. It should be noted that the 94Q product is a prediction of MAP, not a point rainfall, or point maxima. Also, a "0" forecast (no isohyets drawn) does not mean "no" precipitation but implies no MAP of 0.25 in or greater in the verification domain.
Two comparisons were made for each rainfall category using a "hit or miss" scheme. A "hit" was defined as a forecast that fell within the assigned rainfall increment. As an example, for a forecast of 0.25 in to verify, the observed MAP would have to be in the range of 0.25 to 0.49 in, which is the standard range that is used by the HPC. Likewise, for a 0.50-in forecast, an MAP of 0.50 to 0.99 in would have to be observed. Observed MAP values not occurring within the assigned range resulted in a predictive "miss." For example, if the 94Q product forecasts 0.25 in of MAP, and only 0.10 in was observed, then the forecasts were a miss. Percent of correct forecasts was then calculated for all categories.
Verifications were first computed for each category using an absolute correct approach. This meant that no tolerance range was allowed. Using this approach, an MAP of 0.24 in would have to be considered a miss for a QPF of 0.25 in. It is obvious to most forecasters that a QPF that misses by only a few hundredths of an inch is still a good QPF. Therefore, for the second approach, a tolerance of 10 percent was used for all the categories, and the percent correct recalculated. It may be true that a larger tolerance, say 20 percent, would be more reasonable.
The percent of correct forecasts was computed for all categories independently, for all categories by month, and for all categories combined. Bias was determined by calculating the percent of overforecast and underforecast. One final question was also answered: What percent of the time does it rain (measurable), when the 94Q forecasts precipitation of any value in the verification domain.
The study period (April 1994-September 1996) could be described as one wet year (1994), followed by almost two years of near drought conditions. The year of 1995 began somewhat "wet;" but by the middle of 1995 and through most of 1996, conditions were quite dry with occasional "wet" periods. This is reflected in Fig. 2, the distribution of NCEP rainfall forecasts, with 678 forecasts of "0," or forecasts of less than 0.25 in MAP and 214 forecasts of 0.25 in or more.
It has been noted that the HPC forecasters have higher verification scores in wet years and lower verification scores in dry years (Olson et al., 1995). With this fact in mind, it therefore would not be surprising if the verification results found in this study were of a lower score than results for a more "normal precipitation" study period.
The NCEP QPF accuracy is shown by the following contingency table summary:
Category #of Fcsts Correct (no tolerance) Correct (+10%)These results are depicted in Fig. 3.
"0" 678 97% 97%
0.25 in 107 18% 25%
0.50 in 74 27% 39%
1.00 in 27 26% 33%
2.00 in 5 40% 40%
3.00 in 1 0% 0%
The total accuracy for all groups, excluding "0" forecasts, is as follows:
#of Fcsts Correct (no tolerance) Correct (+10%)
214 22% 32%
A graphical depiction for the 0.25-in category is shown in Fig. 4. The numbers under each month indicate the amount of correct forecasts versus the total number of forecasts made during the month. It is of interest to note the increasing predictive skill during the months of the cool season and then a decrease in skill during the warm season. Graphs of the other categories (not shown) also reflect this pattern.
Fig. 5 depicts the monthly variability of forecast skill for all categories. As in Fig. 4, we see the same general trend of increasing skill during the cooler months and a decrease in skill during the warmer months. A large portion of the total QPF forecasts were made during the six-month period of April through September (73 percent), while only 27 percent was made during the months of October through March. It is of interest to note that while nearly three times as many forecasts are made during the warm season months, they are 33 percent less accurate than those made during the cool season (28 percent compared to 42 percent).
The best skill seems to be in the 0.50-in category, with 39 percent of the forecasts being correct. The graph of this category, (not shown), indicates a little less accuracy during the months of January and February (than the other forecast categories), but slightly more "hits" during the summer months. It should be noted that while the 2.0-in forecast shows a 40 percent accuracy rate, there were only five forecasts made in this group, which is not a large enough sample size to determine any comparative skill.
More 1.0-in forecasts were needed in the study to determine any seasonal trends. However, the 1.0-in accuracy graph (not shown) still depicts the general trend of increasing accuracy during the cool season months.
As can be seen in Fig. 6, a strong bias exists to overforecast the QPF. Of all the QPFs made, 60 percent were too high, when compared to the observed MAP, by at least one category. Only 8 percent were underforecast. However, of all the 2.0 and 3.0-in observed MAPs, 66 percent, or two-thirds, were underforecast by at least one category.
Calculations were also made to determine the answer to the question: How often does it rain, any measurable amount when the HPC forecasters draw an isohyet of any value in the study area? The answer is 73 percent of the time, or nearly 3 times out of 4, it rains at least 0.01 in.
Summary and Conclusions
The character and scale of precipitation events change from summer to winter over North Central Texas. Most rainfall events in this area are highly convective in nature. This is especially true during the warm season, which is generally defined as the months of April through October. During the traditional summer months of June, July, and August, most of the events are mesoscale convective events, dominated by small scale convective processes. During the cool season, November through March, the rainfall events are more synoptic-scale in nature. Still, many of the observed MAPS are usually much lighter than those of convective events.
The seasonal variation of predictive skill is consistent with the monthly average Threat Scores determined by Olson et al. at the HPC. The data in this study set clearly indicates that more skill exists in the winter months than the summer months. This does not come as too much of a surprise, considering that many more events in the winter are synoptic-scale rather than mesoscale. Therefore, the best skill seems to be in the forecasting of synoptic-scale events, rather than mesoscale events. The least skill is indicated in the smallest QPF category of 0.25 in, while the most skill is indicated in the 0.50-in category. However, the 1.0 and 2.0-in categories are not too far from the 0.50 in skill level.
Overall, the QPFs from the HPC forecasters are 32 percent accurate, with a tolerance of 10 percent, or nearly one out of three forecasts are correct. While this may not seem to be a high success rate, an analogy could be used to baseball. If a batter consistently had an average of .320 to .330, he or she would probably end up in the hall of fame.
A close study of the 0.50 to 2.0-in forecasts reveals that most errors were due to location or placement of the QPFs. To a lesser extent, errors were due to timing. When the heavier QPFs were a "miss," the heavier rains usually fell to the east or northeast of the Dallas-Fort Worth Metroplex. When timing was in error, the forecasts were usually too fast, as the data reveals that often the events occurred the next day.
An example of timing, spatial, and quantity errors is illustrated in Fig. 7. The observed MAP for the Dallas-Fort Worth Metroplex for this forecast period was 0.73 in, while the forecast was for an MAP of 2.0 in. From approximately 0000 UTC on June 1 through 0000 UTC on June 2, several locations over Northeast and East Texas received from 2 to 3 in of rain. While MAPs were not computed for Northeast or East Texas, the 2.0-in isohyet on the 94Q product looks very reasonable for that area. The 2.0-in isohyet was too far west, resulting in an overforecast for the verification domain, the forecast was approximately 12 hr fast (or out of phase), and the heavier rains fell to the east and northeast of the Dallas-Fort Worth area.
With these results in mind, could one use the 94Q product to develop local refined 24-hr QPFs, or also as a guide to locally developed incremental 6-hr QPF values? The answer to this is no, not directly. However, it is a good starting point. Much fine tuning of the 94Q forecast would be required to address such items as local terrain effects, timing of the event, and other factors.
Local QPF studies such as this one, should be done at most (if not all) WFOs and RFCs. Forecasters at the WFOs and the HAS forecasters would then know most of the strengths and limitations of the 94Q product and base their QPF forecasts accordingly. Without the local verification studies, one really has no idea how skillful or reliable the 94Q product is in his or her verification domain.
Local studies could also be done on the QPFs that are generated by the models. This could also enhance one's ability to develop local QPFs. Likewise, climatological studies would also help. Other tools, such as Neural Networks, may also greatly help in the production of locally generated QPFs. There is a solid starting point or foundation in the 94Q product. We should strive to improve upon it by increasing our local skill level. This study, for the area in question, indicates the opportunity and possibility of 24-hr QPF improvements in excess of 60 percent.
Many thanks to Wes Junker, Bruce Terry, and Sondra Young, Senior Branch Forecasters at the HPC, for their motivation and support in this study. Special thanks to Chris Bovitz, Hydrologic Forecaster at the WGRFC, who provided valuable graphics assistance.
Hall, T., 1996: Brainmaker, A New Approach to Quantitative and Probability of Precipitation Forecasting. Southern Region Topics, SR/HSD 96-2.
Olson, David A., Junker, Norman W., Korty, B., 1995: Evaluation of 33 Years of Quantitative Precipitation Forecasting at the NMC. Weather and Forecasting, 10, No.3, 498-511.