10 years of the ECMWF Python-based Operational Verification System

10 years of the ECMWF Python-based Operational Verification System
Martin Janousek, ECMWF

Verification of forecasts and all types of NWP products is a fundamental part of the ECMWF research and production procedures. A reliable, comprehensive, and flexible software system providing a wide range of statistics and skill measures and applicable on a wide range of forecast products is instrumental in the continuous monitoring of the production quality, assists the development of new algorithms, and supports the decision process when implementing new model versions.

Since 2008, ECMWF has been operating the verification system Verify. The system has undergone several modifications and in its current version is written entirely in Python, interfacing ECMWF meteorological data format decoding libraries, NumPy and SciPy, as well as in-house developed statistical libraries. It runs a set of SQL databases holding verification scores of all recent and historic ECMWF forecasts.

Verify has been designed with the idea of a user freed from knowledge of complexities of data formats and storage, pre- and post-processing requirements of specific atmospheric parameters and scores, as well as quirks of various datasets. All specific features of both verification methods and datasets are encoded following the object-oriented approach in the system of Python classes of scores, parameters, data accessors etc. Verify accesses data in their native locations and raw formats contributing to the monitoring of the ECMWF information system. A user controls Verify via a rather simple interface, the Verify Language, specifying scores computation and/or generation of plots and diagrams in a rather generic manner. The implemented computational and presentation algorithms are highly flexible and their default behavior can be modified by the Language optional keywords. Advantages and disadvantages of the Verify Language will be discussed in the talk.

Verify currently serves as a primary verification tool for the near-real-time monitoring of the quality of operational NWP products of ECMWF as well as for monitoring of long-term trends in a wide range of forecast skill metrics. Verify databases hold verification data of more than 30 years of deterministic and ensemble forecasts; also scores of special datasets like re-analyses and, whenever available, scores of forecasts of other global NWP centres which serve as benchmarks for assessing ECMWF products quality.

During its lifetime, Verify has been significantly upgraded several times and its features have been extended and enhanced. Demands for adding new scores and extending the list of included datasets have been coming frequently and always required swift response. In this process, Verify has proven its flexible design but it now approaches the point of need of a significant system overhaul.

The talk will be given at a time when ECMWF is in the process of reviewing its verification procedures and systems. As part of the review we are taking a closer look at externally developed verification systems, to see if they could meet our needs and whether there is a potential for mutually beneficial collaborative developments.

Thursday, December 7, 2017
9:30-10:30
FL2-1001, Small Auditorium

Building:

Room Number: 
1001

Type of event:

Will this event be webcast to the public by NCAR|UCAR?: 
Announcement Timing: 
November 30, 2017 to December 7, 2017
Calendar Timing: 
Thursday, December 7, 2017 - 9:30am to 10:30am