River Forecast Verification at the ABRFC
Renee Wasko, Bill Lawrence, and Billy Olsen
The Arkansas-Red Basin River Forecast Center (ABRFC) began routinely issuing river forecasts with and without Quantitative precipitation forecasts (QPF) (Fig. 1) in February 1997, supported by conclusions from NOAA Technical Memorandum SR-187 (Reed, Olsen, and Schmidt 1997). One conclusion from SR-187 stated that forecasts with and without QPF would "support basin-wide, river-forecast QPF verification at the RFC." In February 1997, the program "Fcstver" was developed as a tool to compare/verify QPF and non-QPF river forecasts. Since then, four years of river verification statistics have been compiled at the ABRFC. Overall, these statistics have shown that QPF improves river forecasts over non-QPF forecasts. Concerns are raised, however, in the use of QPF during the convective season due to relatively larger river forecast errors when using QPF in the "warm season" versus the "cool season."
Quantitative precipitation forecasts (QPF) are used as input into the ABRFC's hydrologic model. As used at ABRFC, the QPFs are for six-hour increments for the future 24-hr period. From February 1994 to March 2000, individual weather forecast offices provided the ABRFC with QPF files which were mosaicked by ABRFC forecasters into a composite QPF. Since April 2000, the ABRFC has created their composite QPF using the NCEP Hydrometeorological Prediction Center's (HPC) QPF as guidance.
The Fcstver program compares river forecasts based on QPF, non-QPF, and persistence. The persistence forecast assumes no skill in forecasting (i.e., the current stage is forecast for all periods). The program produces both numerical (Fig. 2) and graphical (Fig. 3) output in root-mean-square-error (RMSE), as described by Panofsky and Brier (1965), and bias.
At the time the Fcstver program was written, the ABRFC river forecasts were issued with six-hourly time series, four days into the future. Thus, there are normally 16 time ordinates in each forecast. The Fcstver program determines statistics for the first 15 time ordinates. Forecasts are retrieved from the ABRFC archive database. Recent efforts have restored forecasts back to 1994 into the ABRFC archive database; however, time series for both QPF and non-QPF river forecasts exist only since February 1997. For statistical purposes, the ABRFC uses the overall RMSE numbers, which are averages of the RMSE over the 15 time ordinates. Statistics are run monthly for all ABRFC daily forecast points. The number of daily points increased from 20 in February 1997 to 27 in June 2000 (Fig. 4).
All daily forecasts and any flood forecasts issued for these points are included in the monthly statistics. Monthly statistics for each individual daily forecast point are calculated, but will not be discussed in this paper. The individual statistics are available at http://www.srh.noaa.gov/abrfc/qpfver2/2001data.html
Overall, the average statistics over four years show that river forecasts using QPF are approximately 4% more accurate than the non-QPF river forecasts, as shown in Fig. 5. Breaking these numbers down into average monthly statistics (Fig. 6) shows the QPF river forecasts performed the best during the cool season (October-April), with non-QPF river forecasts performing better in the warm season (May-September). This correlates with the seasonal variation of QPF accuracy. An evaluation of threat scores - the ratio of the correctly predicted area to the threat area - on NMC's QPF from 1984-1993 (Fig. 7) showed the highest scores occur during the cool season and the lowest during the warm season (Olson et al. 1995). Numerical model precipitation forecasts also showed decreased accuracy in the warm season. The character and scale of precipitation events change from summer to winter. Warm season precipitation is dominated by small-scale convective processes, while during the cool season synoptic-scale systems marked by pronounced low-level warm advection tend to predominate. Also, warm season precipitation events in the ABRFC area occur with greater frequency at night (Maddox et al. 1979), while cool season precipitation is more evenly distributed throughout the 24 hours.
Figure 8 shows that during winter months, the river forecasts using QPF improved on the non-QPF river forecasts by 9%. However, during the summer months of this study the non-QPF river forecasts were 5% better than the QPF river forecasts. This study has resulted in the ABRFC re-thinking the amount of QPF used during the warm season.
Looking more closely at the cyclical nature of the RMSE throughout the year, it is shown that RMSE corresponds closely with the volume of discharge in the subject rivers and streams. Comparing the mean monthly discharge for 21 selected ABRFC daily forecast points with the QPF RMSE since February 1997 revealed a 0.86 correlation coefficient (Fig. 9).
When these monthly values are further normalized by month, the correlation is 0.91 (Fig. 10). In other words, lower RMSE corresponds with drier periods across the ABRFC basin and vice- versa. Using this relationship and looking at the average monthly statistics shown in Fig. 6, it appears the ABRFC basin is wetter in the spring and winter months, with drier weather in the summer and fall months. The high correlation between RMSE and river discharge also questions the value of using RMSE as a verification tool for tracking and managing river forecasting performances. Using RMSE to measure forecast accuracy may not be a reflection of forecaster skill, but rather dry and wet periods. In the future, categorical verification statistics may be a better performance tracking measure.
The verification study shows significant forecast errors can occur when using QPF in forecasts. Many times, one "bad" event can skew overall monthly totals drastically, especially if the overall monthly precipitation total is low. An example of this occurred in February 1999 at Blackwell (BLKO2) in north-central Oklahoma. Over 1.50 in of rain was forecast to fall in the 24-hr period ending at 12UTC on February 11, (Fig. 11), but less than 0.10 in actually fell (Fig. 12). Figure 13 shows the resulting hydrographs with and without QPF. Since the total monthly precipitation in this area was only 1.0-1.5 in for February, this one bad event negatively affected the overall February numbers dramatically.
On the other hand, this study shows that during heavy rain events, QPF river forecasts improve forecast accuracy significantly. As an example (Fig. 14), 1.0-2.5 in of rain were forecast for southeastern Oklahoma into western Arkansas for the 24-hr period ending at 12UTC on December 21, 1997. Actual amounts of 1.0-3.0 in were observed (Fig. 15). The resulting hydrograph for Glover (GLOO2) in southeastern Oklahoma with QPF was much higher than the non-QPF forecast hydrograph (Fig. 16). This example demonstrates that during heavy rain events when QPF verifies, QPF river forecasts are much more accurate than non-QPF river forecasts.
Four years of river forecast verification statistics at the ABRFC show an overall improvement of nearly 4% in forecast accuracy with river forecasts based on QPF. River forecasts with QPF are more accurate than non-QPF river forecasts during the cooler months, mainly due to the more synoptic-scale precipitation systems. On the other hand, non-QPF river forecasts outperform QPF river forecasts during the warmer months due to small-scale convective precipitation systems. This has resulted in the ABRFC re-thinking the amount of QPF used during the warm season. QPF is a powerful tool, but due to the probability of error the farther away from the current time, deterministic forecasts using QPF may be implying more confidence than we have. Therefore, the ABRFC is considering using six-hour QPF during the warm convective season and 12-hr QPF during the cool synoptic season. Longer duration QPFs should be used operationally through the issuance of River Flood Watches and Outlooks.
The study revealed some significant QPF errors which negatively affected our overall statistics, especially during dry periods. However, QPF river forecasts did improve forecasts in heavy rain events. RMSE relates directly to the wetness of the basin, thus raising questions regarding the value of RMSE in river forecast verification. In the future, categorical verification may be a better performance tracking measure.
Maddox, R. A., L. R. Hoxit, and C. F. Chappell, 1979: Synoptic and Meso-Scale Aspects of Flash Flood Events. Bull. Amer. Meteor. Soc., 60, 115-123.
Olson, D. A., N. W. Junker, and B. Korty, 1995: Evaluation of 33 Years of Quantitative Precipitation Forecasting at the NMC. Wea. and Forecasting, 10, 498-511.
Panofsky, H. A. and G. W. Brier, 1958: Some Applications of Statistics to Meteorology. Pennsylvania State University, University Park, PA, 224 pp.
Reed, W. B., B. G. Olsen, and J. A. Schmidt, 1997: An Evaluation of River Forecast Model Output for Simulations With and Without Quantitative Precipitation Forecasts (QPF). NOAA Technical Memorandum NWS SR-187. Fort Worth, TX, 16 pp.
Figure 3. Graphical output of the Fcstver program.
Figure 4. ABRFC's 27 daily forecast points.
Figure 5. Overall average RMSE statistics for QPF, non-QPF, and
persistence forecasts over the past four years.
Figure 6. Average monthly RMSE statistics for QPF, non-QPF, and
persistence forecasts over the past four years.
Figure 7. NMC forecasters' mean, minimum, and maximum monthly threat scores for 1.00-inch QPFs for 1984-1993. The minimum and maximum curves are the extreme monthly threat scores during the 10-year period.
Figure 8. Average seasonal RMSE statistics for QPF, non-QPF, and persistence forecasts over the past four years.
Figure 9. Comparison of the mean monthly discharge from 21 selected ABRFC daily forecast points with QPF RMSE since February 1997.
Figure 10. Normalized monthly comparison of the mean monthly discharge for 21 selected ABRFC daily forecast points with QPF RMSE.
Figure 11. QPF for the 24-hr period ending at 1200UTC February 11, 1999.
Figure 12. Actual precipitation totals for the 24-hr period ending at 1200 UTC February 11, 1999.
Figure 13. River forecast hydrograph for Blackwell, OK (BLK02) with QPF forecast (2), non-QPF forecasts (N), and actual stages (o) for February 11-14, 1999.
Figure 14. QPF for the 24-hr period ending at 1200UTC December 21, 1997.
Figure 15. Actual precipitation totals for the 24-hr period ending at 1200UTC December 21, 1997.
Figure 16. River forecast hydrograph for Blackwell, OK (BLK02) with QPF forecast (1), non-QPF forecasts (N), and actual stages (o) for December 20-24, 1997.