Department of Commerce $ National Oceanic & Atmospheric Administration $ National Weather

Stage 1 Project Plan NDFD Uncertainty Products OSIP # 07-040

1. Project Summary Project Lead: Lee Anderson and Douglas Hilderbrand Last Gate: NA Gate Disposition: NA Current Stage Participation (Total FTE hours): 4020

2. Solution Description The IWT has not determined a “preferred” solution during stage 1.

1) Climatological statistics of errors in NDFD forecasts and/or numerical guidance—needs an adequately large sample of past forecasts and verifying observations. Pros: Easy to calculate Cons: (i) Forecast uncertainty is known to vary from case to case. Error statistics from a single forecast will not capture such variations.; (ii) No alternative forecast scenarios are offered

2) Statistics from an ensemble of numerical forecasts--Ensemble can be a combination of forecasts from various sources, including high resolution control forecasts and lower resolution ensemble forecasts from different centers (e.g., NAEFS). Ensemble guidance can be statistically calibrated to improve its reliability (just as described under (1) for a single forecast) Pros: (i) Potentially it can capture case dependent variations in forecast uncertainty; (ii) Beyond uncertainty statistics, it can provide plausible scenarios or trajectories of future weather that is needed as input in sophisticated downstream applications such as streamflow ensemble forecasting, energy demand forecasting, etc. Cons: Requires maintenance of numerical ensemble forecast system(s)

3) Current status: For most variables, the NDFD database contains no information related to forecast uncertainty. This amounts to a serious limitation in terms of the utility of the forecasts

Proposed solution: Expand NDFD database to include forecast uncertainty related information. Necessary steps: A) Agree on format of uncertainty information. The format: i) Should be applicable to all NDFD parameters if possible (ie, no different formats for different parameters) ii) Preferably should have intuitive meaning to forecasters and users iii) Should allow for derivation of uncertainty information in other formats (with certain assumptions) iv) Should be compact (for efficient storage / transmission)

Template Version 3.8 1 Document Version Date: [9/2617/2007] B) Develop numerical guidance (NDGD) for uncertainty information. Potentially, forecast uncertainty information can be derived in different ways. For example, a forecast probability distribution for a parameter can be based on: 1) Statistics of errors in NDFD forecasts and/or numerical guidance. a) Specifics: Needs an adequately large sample of past forecasts and verifying observations b) Pros: Easy to calculate Con: (i) Forecast uncertainty is known to vary from case to case. Error statistics from a single forecast will not capture such variations.; (ii) No alternative forecast scenarios are offered 2) Statistics from an ensemble of numerical forecasts a) Specifics: (i) Ensemble can be a combination of forecasts from various sources, including high resolution control forecasts and lower resolution ensemble forecasts from different centers (e.g., NAEFS); (ii) Ensemble guidance can be statistically calibrated to improve its reliability (just as described under (1) for a single forecast) b) Pros: (i) Potentially it can capture case dependent variations in forecast uncertainty; (ii) Beyond uncertainty statistics, it can provide plausible scenarios or trajectories of future weather that is needed as input in sophisticated downstream applications such as streamflow ensemble forecasting, energy demand forecasting, etc. c) Con: Requires maintenance of numerical ensemble forecast system(s)

Arrange for transmission of numerical guidance to WFOs Develop software tools to display and manipulate (manual adjustment) numerical guidance at WFOs Develop software to derive uncertainty information in other formats (eg,, for web access by external users) Train forecasters in use, manipulation, and interpretation of new forecast format

Status QuoAlternative to proposed solution: Status Quo Pros: No additional resources needed. Cons: Does not address NRC recommendation or requirements for uncertainty products

3. Planned Activities During Next Stage:  Develop CONOPS and update project plan-- Agree on format of uncertainty information.  Define set of NDFD parameters.n.  Explore other alternative solutions (i.e., RTMA capabilities).  Define operational requirements in coordination with NURO tTeam, NFUSE Team, and AMS Ad Hoc Committee on Uncertainty..

Beyond Next Stage:  Develop Business Case and NDFD technical requirements

2   Arrange for transmission of numerical guidance to WFOs  Develop software tools to display and manipulate (manual adjustment) numerical guidance at WFOs  Develop software to derive uncertainty information in other formats (eg,, for web access by external users)  Train forecasters in use, manipulation, and interpretation of new forecast format  Experimental period for comments/feedback.

4. Performance Measures (Reference “Performance Measures Guidance” included in the Project Plan Author’s Guide). Performance Measure Current Value (Baseline) Planned Resultant Value At End of NA Next Stage

Uncertainty of all variables expressed in operational NDFD (ex. Max No uncertainty information TBD (possible example: Range (+/-) in and Min Temperatures) forecast). At Project Completion

5. Organizational Roles and Responsibilities: During Organization Lead Role Responsibility Next NWS ER David Novak IWT member Regional input.Lead development of OSIP Stage documentation, project coordination, product assessment, and scientific leadership. NWS OST Lee Anderson IWT/Project co-lead Co-lead development of OSIP documentation, (Douglas Hilderbrand/Andrea project coordination, product assessment, and Bleistein) scientific leadership. NWS CR/ ISST Greg Mann IWT member ISST representativeParticipate in development of OSIP documentation, project coordination, and product assessments NWS OST/MDL David Ruth IWT member NDFD operationsParticipate in development of OSIP documentation, project coordination, and product assessments NWS OCWWS TBD (S.uzanne Lenihan/A. IWT member RequirementsParticipate in development of OSIP Horvitz/C. Alex) documentation, project coordination, and product assessments NWS OS Climate Bob Livezey IWT member Climate dataParticipate in development of OSIP documentation, project coordination, and product assessments, and scientific leadership NWS NCEP/EMC Zoltan Toth/Geoff Dimego IWT member Participate in development of OSIP 3 documentation, project coordination, and product assessments, and scientific leadershipNCEP modeling efforts (ensembles) NWS OST/SEC Tim Hopkins/Jason Tuell IWT Member AWIPS II implications NWS Strategic Planning Wendy Levine IWT Member Experimental product phase OAR/GSD TBD (C.arl Bullock/D. Davis) IWT Member Product development NWS WR Dave Myrrick IWT Member Regional Input. Beyond NWS ER David Novak IWT member Lead development of OSIP documentation, Next project coordination, product assessment, and Stage scientific leadershipRegional Input. NWS OST Lee Anderson IWT/Project co-lead Co-lead development of OSIP documentation, (Douglas Hilderbrand/Andrea project coordination, product assessment, and Bleistein) scientific leadership. NWS CR/ ISSTNWS CR/ ISST Greg MannGreg Mann IWT member ISST representative Participate in development of OSIP documentation, project coordination, and product assessments NWS OST/MDLNWS MDL David RuthDavid Ruth IWT member NDFD operationsParticipate in development of OSIP documentation, project coordination, and product assessments NWS OCWWSNWS OCWWS Suzanne LenihanSuzanne Lenihan IWT member RequirementsParticipate in development of OSIP documentation, project coordination, and product assessments NWS OSNWS OS Bob LivezeyBob Livezey IWT member Climate dataParticipate in development of OSIP documentation, project coordination, and product assessments, and scientific leadership NWS NCEP/EMCNWS Zoltan Toth/Geoff DimegoZoltan IWT member NCEP modeling efforts (ensembles)Participate in NCEP/EMC Toth development of OSIP documentation, project coordination, and product assessments, and scientific leadership NWS OST/SEC Tim Hopkins/Jason Tuell IWT member AWIPS II implications NWS Strategic Planning Wendy Levine IWT member Experimental product phase OAR/GSD Carl Bullock IWT member Product development

4 6. Milestones and Schedule [Double click table to enable excel worksheet.]

Milestone Lead FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 Org. Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Indicate when the milestone is planned to occur with an "X" in the appropriate time period. Insert or delete rows as necessary. Prepare CONOPS/ORD X Gate 2 X During Next Stage

Gate 3 TBD

Beyond Next Stage

7. Resource Requirements [Double click table to enable excel worksheet. Cells with zeros will automatically sum as entries are made in blank cells.]

5 Required Government FTE Labor (Hours) (= # people x hours/week x weeks) Organization FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 NWS ER 40 OST 80 NWS CR / ISST 20 MDL 30 OCWWS 40 NWS SP 8 NCEP-EMC 40 During OST/SEC 16 Next OAR 8 Stage NWS WR 8 Beyond Next Stage Required 290 0 0 0 0 0 0 0 Per Year Total Total to Approved 290 After FTE Complete Delta 0 0 0 0 0 0 0 0 Completion Non-Labor including Contractors ($K) 290 Item (non-Recurring) Units Org Stage FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 Hardware 0 -290 0 Per Year Software 0 Total to After Complete Communications 0 Completion Contractors 0 Travel 0 Other (specify) 0 Add Rows as Needed Subtotal Non-Recurring 0 0 0 0 0 0 0 0 O&M Total Amount Required ($K) 0 0 0 0 0 0 0 0 Amount Allocated ($K) 0 0 Funded Portion Total Non-Labor NOAA Program Amount Needed ($K) 0 0 Unfunded Portion NOAA Program 8. Acquisition Strategy: 6 During Item Strategy Next None expected Stage Beyond TBD Next Stage

9. Assumptions and Constraints: Assumption/Constraint Description During Requirements will be focused on end-users. Next Task fully staffed Sufficient IWT participation Stage

Beyond Redirected to SREC after Gate 3 approval Next Stage

10. Risk Assessment and Mitigation: Type Description Probability Impact Mitigation Strategy Business NWS field offices will not use the information. Low Medium Modify the products according to feedback from users. Technical Development problems Low High Use alternate methods or correct methods for other solutions. During Schedule TBD Next Limited AWIPS development resources Medium High Tools prioritized in order of need. Stage Resources available Cost TBD Changing requirements Medium High Build-by-build reassessment and prioritization. Clear Business coordination between DSPAC, ISST, and STSIT. There may need to be small enhancement requests for timely changes. Business NWS field offices will not use the information. Low Medium Modify the products according to feedback from users. Changing software (AWIPS2) Medium High Current plans are to keep backward compatibility between Beyond AWIPS and AWIPS2, such that current tools will work on Next Technical the AWIPS2 infrastructure. However, should the technical Stage details of the AWIPS 2 infrastructure change, the Continuing Phase of the project will adapt, if necessary. Schedule Limited AWIPS development resources Medium High Tools prioritized in order of need. Resources available Cost

7 Stage 3 Checklist

[For projects at Gate 2, place an “X” in the “Required” column to indicate items required in Stage 3 and define the “Lead Office/Division” and “Point of Contact”. For projects at Gate 3, complete the “Status” and “Remarks” columns for those items checked as required in Stage 3.] C R o e m q p u Lead Office/ Point of l i e

No. Checklist Item r Status Remarks e t Division Contact e d d

1 Acquisition Plan (Ref: NWSI 1-1101) Applied Research and Development Results 2 X (NWSI 10-103) 3 Business Case Analysis (NWSI 10-103) X Operational Development Plan 4 (NWSI 10-103) 5 Technical Requirements (NWSI 10-103) X NWS Union review for impacts to working 6 conditions (NWSI xxx) Resources for next/future stage allocated 7 through the FIRC or NOAA Program Manager? (NWSI xxx) 8 Training Plan (NWSI xxx) X

8 Stage 4 Checklist

[For projects at Gate 3, place an “X” in the “Required” column to indicate items required in Stage 4 and define the “Lead Office/Division” and “Point of Contact”. For projects at Gate 4, complete the “Status” and “Remarks” columns for those items checked as required in Stage 4.] C R o e m q p u Lead Office/ Point of l i e

Deployment Decision Document 1 (NWSI xxx) Deployment, Assessment, and Lifecycle 2 Support Plan (NWSI xxx) Operational Test & Evaluation (New System) 3 (Ref: NWSI 30-302 Field Test Process)

Operational Acceptance Test (Existing 4 System) (NWSI 30-302) Facility or Site Preparation and Installation 5 Plan (NWSI 30-4104 Operations and Maintenance) Facilities-related factors (e.g., leases, space, 6 power) identified and considered (NWSI 30- 4104) Program Management Responsibility Transfer 7 Plan (NWSI 80-103 Program Management Responsibility Transfer Plan) ILS Plan (NWSM 30-3102 Integrated Support 8 Logistics Planning) Commissioning/Decommissioning Plan (NWSI 80-201 System Commissioning 9 Process NWSI 80-202 System Decommissioning Process) Commissioning Report, Approval and 10 Checklist (NWSI 80-201,NWSI 80-202) Physical Configuration Audit and Functional Configuration Audit 11 (NWSI 30-1203 Configuration Management for Operational Systems)

9 C R o e m q p u Lead Office/ Point of l i e

Funding identified for system deployment and life cycle support (NWSI 80-101 Program 12 Plan, NWSI 10-103 Operational Development Project Plan) Human Factors requirements including personnel skill set and workload requirements identified and funded (NWSM 20-102 13 National Strategic Training and Education Process (NSTEP) and Annual Implementation Plans for Training and Education, Professional Development Series (PDS) IT Security Certification and Accreditation (C&A) Plan developed and approved (NWSI 60-701 Assignment of Responsibilities, NWSI 14 60-702 Management Controls, NWSI 60-703 Operational Controls, NWSI 60-704 Technical Controls) Occupational Safety and Health Policy compliance-related factors identified and 15 considered (NWSI 50-1115 Occupational Safety and Health, NWSI 30-4104 Operations and Maintenance) Environmental Policy compliance-related factors identified and considered (NWSI 50- 16 5116 Environmental Management, NWSI 30- 4104 Operations and Maintenance)

NWS Union review for impacts to working 17 conditions