JCSDA Seminar - ECMWF's Next Generation Software Stack for the IFS Model and Product Generation: Future Workflow Adaptations

Joint Center for Satellite Data Assimilation Seminar
Dr. Tiago Quintino, Team Leader for Scalability, ECMWF
ECMWF's Next Generation Software Stack for the IFS Model and Product Generation: Future Workflow Adaptations

Starting 2014, ECMWF has embarked on a research programme on HPC Scalability, aiming to achieve Exascale numerical weather prediction systems by 2025.

ECMWF operational forecast generates massive amounts of I/O in short bursts, accumulating to tens of TB in hourly windows. From this output, millions of user-defined daily products are generated by a complex chain of transformations and regridding operators and finally disseminated to member states and commercial clients.

These products are processed from the raw output of the IFS model, within the time critical path and under strict delivery schedule. Upcoming resolution increases and growing popularity will increase both the size and number of these products. Based on expected model resolution increases, by 2020 we estimate the operational model will output over 100 TB/day and need to archive over 400 TB/day. Given that the I/O workload is already one of the strongest bottlenecks in ECMWF's workflow, this is one of the main challenges to reach Exascale NWP.

We present a new software stack that ECMWF is developing to tackle these future challenges in the scalability of model I/O and product generation, and reworking its operational workflows to adapt to forthcoming I/O technologies.

In particular, we will present the adaptation of IFS I/O server to the use of NVRAM technologies as a way to buffer large amounts of forecast outputs en route to the product generation and archival systems, thus minimising file-system I/O within the operational critical path and collocating post-processing with model computation.

November 20, 2017
11:00am - 12:00pm
Room 2A305
NOAA David Skaggs Research Center 

Room Number: 
2A305

Type of event:

Will this event be webcast to the public by NCAR|UCAR?: 
No
Announcement Timing: 
November 13, 2017 to November 20, 2017
Calendar Timing: 
Monday, November 20, 2017 - 11:00am to 12:00pm