Comparison of Arkansas Basin River Forecast Center Quantitative Precipitation Forecasts with Quantitative Precipitation Estimates
William E. Lawrence
Quantitative precipitation forecasts (QPF) are an important input data set for river forecast models that are currently used to produce river forecasts by the Arkansas-Red Basin River Forecast Center (ABRFC). Improving the accuracy of river forecasts is an ongoing goal for the ABRFC. Operational experience at the ABRFC has resulted in a perception that forecast rainfall amounts are sometimes extremely excessive, which could lead to increased error in river forecasts. The primary purpose of this study is to replace perceptions of forecast QPF bias with actual computed bias. This study compares QPF issued by the ABRFC against quantitative precipitation estimates (QPE) produced by the ABRFC to determine if there is any QPF bias, both in amount and location. ABRFC QPEs are gauge-adjusted WSR-88D radar-derived precipitation estimates with human quality control that are used as "observed" precipitation input to the river forecast model. Schmidt, et al. (2000) have shown that ABRFC QPEs produced since mid-1996 have no discernable bias as compared to traditional gauge-only QPEs, and thus form a reasonable basis for comparison with QPF.
The ABRFC maintains an archive of most of its products since 1993. QPF files stored in the network Common Data Format (netCDF) were used for this study. The files were converted into XMRG format, a binary gridded format developed by the National Weather Service (NWS) Office of Hydrology(OH). The grids from each file were then summed in monthly increments,. starting with October 1996. This month was selected as this is generally the time the ABRFC started using P1, an ABRFC designed application, to produce gridded precipitation estimates. Again, Schmidt et al. (2000) have shown P1 to be a reliable algorithm for producing bias-free QPEs. The period of study ends with December 2001.
Several months during the study period had missing data and could not be used for the basis of evaluation. However, during the 63 month period of October 1996 to December 2001, only four months were not usable. This included three months in early 2000, when the ABRFC transformed their QPF production method from using NWS Weather Forecast Office (WFO)-produced QPFs, to an NWS Hydrometeorological Prediction Center (HPC)-based method. In the WFO method, the ABRFC received four six-hourly QPFs from 12 different WFOs within its area. The ABRFC forecaster would then smooth out differences among adjacent WFO forecasts and mosaic them into an ABRFC-produced QPF. Any large differences between adjoining WFOs would be reconciled by coordination and/or consensus. Starting in March 2000, the ABRFC began receiving six-hourly QPF files from HPC. The ABRFC forecaster has the option to adjust any six-hourly period, but emphasis is placed on the first six-hour period. Normally, the HPC QPF will be used with little or no modification. The ABRFC currently processes four six-hourly periods for a total of 24 hours of QPF.
For the entire study period, two distinct methods of calculating monthly QPFs were compared. The first method, the Standard Method, uses 24-hour QPFs that forecast rainfall from 1200 UTC to 1200 UTC each day. The second method, named the Update Method, also uses 24 hours worth of QPF, but sums the first 12 hours of QPF from the 1200 UTC cycle each day with the first 12 hours of QPF from the 0000 UTC cycle. Thus, the Update Method covers the same time period as the Standard Method, but is updated twice as often.
Both QPF and QPE totals were summed on the Hydrologic Rainfall Analysis Project (HRAP) grid. The HRAP grid is based on a polar stereographic projection and was developed by OH for radar-derived gridded precipitation estimates. At the latitude of the ABRFC basin, each grid is approximately 4km x 4km in size. Within the ABRFC area, there are 33016 grid bins. To determine a mean bias for the entire ABRFC area, the average of all QPF grids for a particular month was divided by the average of all of the QPE grids for the same month.
To determine a weighted biases for seasonal and yearly periods, the number of days in each month was multiplied by the average bias for the month so as to give different weights for months of varying days. The winter bias includes the months of December, January, and February. The spring bias includes the months of March, April and May. The summer bias includes the months of June, July and August, and the fall bias includes the months of September, October and November.
a. QPF Bias.
Bias of the QPFs is defined as simply the ratio of QPF to QPE. A ratio greater than 1.0 is a positive bias (over-forecasting), while a ratio (bias) less than 1.0 indicates under-forecasting. In the ABRFC area of responsibility, a persistent and noticeable positive QPF bias exists. As shown in Fig 1, a positive bias exists for every month except December using both the Update Method of QPF, and the Standard Method of QPF. Both Methods show a peak of the bias in late Winter with a secondary max from late spring through most of the summer. The Update Method has greater positive biases 10 out of 12 months. Figure 2 shows that all of the four seasons also have weighted biases that are higher for the Update Method.
Using a weighted average bias for each month, an annual QPF bias for each grid bin can be determined. Using the Standard Method, an annual bias of 1.150 exists. The Update Method produces an annual bias of 1.194. If one solely takes into account recent precipitation forecasting using the HPC guidance method, positive annual biases still exist, but they are reduced to 1.043 for the Standard Method and 1.127 using the Update Method. This represents over-forecasting by 4.3% and 12.7%, respectively.
Figure 3 shows weighted yearly biases for the period of study. All years except 1996 exhibited a higher bias for the Update method of QPF. Only the last three months of the year were examined for 1996, and therefore the numbers derived for 1996 likely do not represent what the entire year bias would be. Since 1997, yearly biases have generally been decreasing.
b. WFO QPF vs. HPC QPF
As previously mentioned, two different QPF sources were used during the study period. For the period from October 1996 until February 2000, the ABRFC utilized WFO guidance in preparing QPF. From June 2000 thru December 2001, the ABRFC used HPC based guidance solely.
Average monthly biases from the two different periods:
Standard Method Update Method
WFO Based QPF 1.196 1.219
HPC Based QPF 1.043 1.127
The HPC based method of producing QPF has a significantly lower bias for both the Standard and Update method as compared to the WFO based QPF. Both methods, however, showed decreasing biases with time, as indicated in the Best Fit line on Fig 4 and 5. This likely indicates increasing skill with time.
It is interesting to note that the HPC based QPFs had a higher Update bias 15 out of 19 months, or 79% of the time, while the WFO based QPF had a higher Update bias only 22 out of the 40 months studied, or 55% of the time.
The most pronounced and prolonged differences of the Standard Method vs the Update Method tended to occur from May until Sept (Fig 1), where the Update method is consistently higher in bias. This is normally the period of the year when convection is most prevalent in the ABRFC region. 0000 UTC forecasts of QPF take into account any late afternoon convection that forms. It appears that these forecasts may have overestimated the temporal and/or spatial extent of this late afternoon convection.
c. Spatial Bias
For the 59 month period studied, spatial biases varied tremendously when compared from month to month. However, Fig 6 shows the total error of standard method QPF in inches (QPF minus QPE) from the entire 59 month study geographically. Darker shades indicate where precipitation was over-forecast, while lighter shades indicate where precipitation was under-forecast. Note that the two extremes, white and black, are areas where QPF was either over-forecasted, or under-forecasted by at least 50 inches.
(Note: Due to some irregularities with the HAS-QPF program used by the ABRFC to compute gridded QPFs before Feb 2000, there exists some missing data bin areas or "spikes" on Fig 6. The author felt it best to leave these grid bins missing for this study. The net result, if these missing grids did not exist, would be even higher biases than shown.)
Although QPF was too high over most of the ABRFC area, there are three broad areas where there was a very significant overforecast of precipitation during the study period. These included the mountainous terrain west of Pueblo, Colorado, the Oklahoma Panhandle into northeast New Mexico, and finally along the Red River from near Childress, Texas to near Wichita Falls, Texas.
All radars tend to perform poorly in heavily mountainous terrain, and this is likely the case west of Pueblo. Radar beam blockage prevents quality estimates of rainfall, especially during the summer convective season when a large percentage of annual rainfall occurs in this region. This, along with a lack of ground truth, may lead to an underestimation of ABRFC's QPEs in this particular area.
The area along the Red River between Childress and Wichita Falls has been locked in a multi-year drought, and has missed significant rainfall events frequently during the past several years. The summer months have been particularly dry and hot in this area during our study period as high pressure has prevailed. This area has simply had bad luck, as many events have produced much less rainfall than expected, with surrounding areas receiving more precipitation.
Radar coverage is again an issue in possibly explaining the excessive over-forecast of precipitation in the Oklahoma Panhandle region into Northeast New Mexico. Although radars do indeed cover the area, the height of the radar beam above ground level is much higher than in most of the remainder of the ABRFC region. This along with a lack of numerous co-op reporting stations may lead to a underestimation of QPEs in this region.
The one large area where precipitation was significantly under-forecast during the study period occurred over southeast Oklahoma into southwest Arkansas. The Ouachita mountains are located roughly in the same area as the deficiency of the QPF, and it is likely that topographic features enhance rainfall in this area. Other smaller areas where QPF was deficient and elevation may play a role include the Palmer Divide area in east central Colorado, as well as just north of the Arkansas River Valley in west central Arkansas where the Ozarks begin.
Overall, there is a positive bias in QPF amounts over the ABRFC basin during the study period. Both sources of guidance for QPF (WFOs and HPC) tended to show decreases in biases as time progressed. The ABRFC is currently using HPC guidance, which shows a positive bias of 4.3% using 24 hour QPF from 1200 UTC, and a positive bias of 12.7% using 12 hours of QPF from 1200 UTC and 12 hours from 0000 UTC.
It should be noted that QPE is just as the name states, an estimate. QPE is not always correct and a reasonable estimate of error is +/- 5%. Therefore, the author concludes that recent HPC QPF guidance using 24 hour QPF from 1200 UTC shows no appreciable bias.
Using updated QPF at 0000 UTC each day led to higher positive biases, especially during the convective season, a finding that the author found surprising. A number of theories have been suggested explaining this occurrence. Forecasters tend to forecast higher precipitation amounts in early periods, when perceived confidence is higher. Afternoon convection in the ABRFC region frequently is not always guaranteed, especially when capping inversions exist. 1200 UTC QPF forecasts may tend to predict these events more conservatively than the 0000 UTC forecast when convection will have likely already developed. There also may be a systematic bias with meteorological models and cycle times. Finally, the study period may be too short to make this conclusion definitive.
Seasonally, QPF bias is lowest during the autumn months for both methods examined. Peaks of bias occur during late winter and early spring as well as during the warmer convective months.
The QPF positive bias from both sources (WFO and HPC) showed a decreasing trend, falling to just
above unity (no positive bias) at the end of each study period. This may or may not show an increase in skill of QPF forecasting. It is possible to obtain forecasts without bias without any accuracy or skill. More likely, achieving a bias value of near unity may indicate more reliable QPF forecasts. (Reynolds, 2002)
Spatial biases are evident in QPF production across the ABRFC basin. Although most of the basin exhibited a clear positive bias in QPF over the 59 month study, there were several areas where QPF was low. These areas were mostly in topographically enhanced areas. This study will hopefully help forecasters acknowledge the existence of these areas, and possibly consider adjusting their forecasts of precipitation in the future, while at the same time taking into account the overall positive bias that exists with QPF in general over the ABRFC basin. Perhaps models and forecasters alike could be made more sensitive to topographic effects as noted in the PRISM climate data sets.
The author would like to thank Billy Olsen, Hydrologist in Charge, ABRFC, James Paul, Senior HAS forecaster, ABRFC, John Schmidt, Senior Hydrologic Forecaster, ABRFC, Dave Reynolds, Branch Chief, Forecast Operations Branch, HPC and Brett McDonald, SOO, Riverton WFO, for their assistance in this study including suggestions and critical review of this paper.
Schmidt, J., Lawrence, B., Olsen, B. 2000: "A Comparison of National Weather Service River Forecast Center Operational Precipitation Processing Methodologies," NOAA Technical Memorandum SR-205. Fort Worth TX (available online at
Reynolds, D., 2002: per personal conversation
Figure 1. Average monthly biases of QPF across ABRFC basin. (Oct 1996 - Dec 2001)
Figure 2. Seasonal biases of QPF across ABRFC basin. (Oct 1996 - Dec 2001)
Figure 3. Yearly biases of QPF across the ABRFC basin.
Figure 4. Monthly biases Oct 1996 - Feb 2000 based on WFO guidance
Figure 6. Total actual difference of 59 month study. Standard QPF total less actual QPE.