Slide1 : The ECMWF ERA-40 Re-analysis Project
Data Management at the BADC A talk by Ag Stephens
Data Scientist, BADC.
4th March 2002.
Slide2 : Outline of this session Introduction to the ERA-40 Project
The model used for ERA-40
What DATA will be available?
BADC management of ERA-40 data
Future plans
Further information: useful documentation ECMWF ERA-40 Re-analysis
Slide3 : Introduction to the ERA-40 Project (1) is used throughout Europe for weather prediction, research and commercial purposes.
concentrates on medium-range forecasts, defined from 3 to 10 days.
is produced by the ECMWF’s own model and assimilation system (the Integrated Forecasting System (IFS)). The European Centre for Medium-Range Weather Forecasts (ECMWF) produces daily operational weather forecasts. This data: ECMWF ERA-40 Re-analysis
Slide4 : Introduction to the ERA-40 Project (2): Objectives To produce and promote use of a comprehensive set of global analyses describing the state of the atmosphere and land and ocean-wave conditions from mid-1957 to 2001.
Foster European and international research by making the observational archive, the analyses and the study reports widely available. The model is now being used by the ERA-40 Project to produce a 40-year Re-analysis dataset. The objectives of the ERA-40 Project are: ECMWF ERA-40 Re-analysis
Slide5 : Introduction to the ERA-40 Project (3): Project Schedule ECMWF ERA-40 Re-analysis Will stretch into 2003
Slide6 : Introduction to the ERA-40 Project (4): Project statistics Data volume: ~30 TB (30,000,000 MB).
Temporal coverage: mid-1957 to 2001.
Spatial coverage: Global, up to 0.1 hPa in the vertical.
Start date: mid-2001.
Estimated end date: October 2003 (stream 3).
ECMWF ERA-40 Re-analysis
Slide7 : Outline of this session Introduction to the ERA-40 Project
The model used for ERA-40
What DATA will be available?
BADC management of ERA-40 data
Future plans
Further information: useful documentation ECMWF ERA-40 Re-analysis
Slide8 : The model used for ERA-40 (1): Model characteristics Spectral model resolution: TL159 (159 waves around the globe) – approx. equivalent to around 1.125 grid (125 km).
Vertical resolution: L60 (60 model levels from the surface to the mesosphere at 0.1 hPa or 65 km).
Model formulation: semi-Lagrangian.
Number of output variables: 154. ECMWF ERA-40 Re-analysis The version of the ECMWF IFS model used to generate ERA-40 has the following characteristics:
Slide9 : The model used for ERA-40 (2): Data assimilation ECMWF ERA-40 Re-analysis The data assimilation scheme for the ERA-40 model is using:
3D-Var – a 3-dimensional variational technique. This takes the previous forecast and interpolates the model data to the position (in 3D space) of observations.
Observations from a range of sources, including:
- NCAR (early observations) - UK Met Office and NCEP (SST and ice limit data) - numerous satellites.
Slide10 : The model used for ERA-40 (3): Observations – satellite data ECMWF ERA-40 Re-analysis
Slide11 : Outline of this session Introduction to the ERA-40 Project
The model used for ERA-40
What DATA will be available?
BADC management of ERA-40 data
Future plans
Further information: useful documentation ECMWF ERA-40 Re-analysis
Slide12 : What data will be available? (1) ERA-40 will produce an enormous range of products including those already available in operational and ERA-15 data. New products include:
fields on isentropic and PV=2 surfaces;
clear-sky radiation fields;
vertical integrals for energy, mass, water and ozone budgets;
additional fields derived during short-range forecasts for support of chemical-transport modelling and other trajectory studies. ECMWF ERA-40 Re-analysis
Slide13 : What data will be available? (2) ECMWF ERA-40 Re-analysis Archive of observations
Analysis and forecast fields on:
Upper air parameters on model and pressure levels;
Upper air parameters on isentropic and PV=±2 surfaces;
Surface and single-level parameters;
Additional vertical integrals for energy, mass, water and ozone budgets The ERA-40 archives will be:
Slide14 : What data will be available? (3) ECMWF ERA-40 Re-analysis Ocean-wave data
Additional fields accumulated from the physical parametrizations
Monthly-means, variances and covariances
Other selected datasets including: time series of selected fields, observation feedback, zonal-mean and grid-point diagnostics, ECMWF/ISLSCP-II dataset. And:
Slide15 : Outline of this session Introduction to the ERA-40 Project
The model used for ERA-40
What DATA will be available?
BADC management of ERA-40 data
Future plans
Further information: useful documentation ECMWF ERA-40 Re-analysis
Slide16 : BADC management of ERA-40 data (1): User survey General response: Users would like all products in all formats but mainly surface and pressure levels on a 1 by 1 grid in NetCDF. The ECMWF user survey results: ECMWF ERA-40 Re-analysis
Slide17 : BADC management of ERA-40 data(2): Why two archives? ECMWF ERA-40 Re-analysis As a user of the data it may become confusing to hear that there are two BADC archives of ECMWF data. They are:
the standard BADC archive populated and managed as a typical BADC dataset.
a “BADC” archive on Fermat at the Manchester High-Performance Computing Centre. This archive is populated by Paul Berrisford at the University of Reading. Users of the data at Manchester must still register as ECMWF/ERA-40 users with us!
Slide18 : BADC management of ERA-40 data (3): Storage plan ECMWF ERA-40 Re-analysis The BADC ERA-40 Storage Plan was written last year to detail what we should do with the data.
It highlights:
how the data should be stored (i.e. grids, levels, formats etc).
a file-naming convention for the data (created in collaboration with Paul Berrisford).
suggestions on how the data can be delivered to users.
Slide19 : BADC management of ERA-40 data (4): Getting the data ECMWF ERA-40 Re-analysis The data is extracted from the ECMWF’s Meteorological Archive and Retrieval System (MARS). It arrives at its destination via:
automated scripts on MARS checking the job queue and submitting jobs 24 hours a day.
automated file ingestion at the BADC to rename files and move to the appropriate directories.
automated scripts on Tornado that check for gaps in the data and send new job listings to be queued on MARS.
Slide20 : BADC management of ERA-40 data (5): More about MARS ECMWF ERA-40 Re-analysis The MARS system also has a sophisticated web-interface:
Slide21 : BADC management of ERA-40 data (6): Tape storage ECMWF ERA-40 Re-analysis Since the volumes of data are so large all ERA-40 data is currently being written to tape (on the ATLAS tapestore).
Once the new BADC AIT-3 Tape Library is working:
conversions of GRIB data to NetCDF can begin.
interpolations to a 1 x 1 grid from the spectral and gaussian grids can begin.
user access to the data will be greatly simplified.
Slide22 : BADC management of ERA-40 data (7): The current situation ECMWF ERA-40 Re-analysis The ECMWF have currently produced the following years of data (that are validated):
1958, 1959, 1973, April 1991-1994.
The BADC archive currently archives:
April 1991- September 1991, 1993,
1994 (except end of December)
Why are we behind production?
problem with automatic extraction script – now sorted!
some data has been produced then problems found.
Slide23 : BADC management of ERA-40 data (8): Where is the data? ECMWF ERA-40 Re-analysis The ERA-40 data is being kept in the following directories:
/badc/ecmwf-e40/data/AA/BC/YYYY/MM/DD/
In files: AABCYYYYMMDDTTP[PPPP].EXT
Where: AA is the Grid type (gg, gp or sp) B denotes whether analysis or forecast (a/f) C is the Level type (m, p, s, t or v) YYYY is the year, MM the month, DD the day TT is the time of the analysis or start of forecast P[PPPP] is the parameter code (1-5 characters) .EXT is the file extension (.grb or .nc)
NOTE: currently most directories will only display a “files_on_atlas” file indicating the files are on the ATLAS tapestore.
Slide24 : BADC management of ERA-40 data (9): What data? ECMWF ERA-40 Re-analysis
N80 Reduced Gaussian Gridded data:
Analyses on Model Levels (ggamYYYYDDHHTTPPP.grb)
Analyses on Pressure Levels (ggapYYYYDDHHTTPPP.grb)
Analyses at the Surface (ggasYYYYDDHHTTPPP.grb)
Analyses on Potential Temperature Levels (ggatYYYYDDHHTTPPP.grb)
Analyses on Potential Vorticity Levels (ggavYYYYDDHHTTPPP.grb)
Forecasts on Model Levels (ggfmYYYYDDHHTTPPP.grb)
Forecasts at the Surface (ggfsYYYYDDHHTTPPP.grb)
Spectral T159 data:
Analyses on Model Levels (spamYYYYDDHHTTPPP.grb)
Analyses on Pressure Levels (spapYYYYDDHHTTPPP.grb)
Analyses on Potential Temperature Levels (spatYYYYDDHHTTPPP.grb)
Slide25 : BADC management of ERA-40 data (10): What parameters? ECMWF ERA-40 Re-analysis
Model Level parameters include:
Specific Humidity, Ozone Mass Mixing Ratio, Temperature, Vorticity and Divergence.
Pressure Level parameters include:
Relative Humidity, Temperature, U- and V-velocity.
Surface/single level parameters include:
2 Meter Temperatures, Soil Temperatures and Wetness, Snow Albedo, Total Cloud Cover, Total Column Water, Sea Ice Temperature and Low Vegetation Cover.
Slide26 : Potential Temperature (Isentropic) Level parameters include:
Specific Humidity, Potential Vorticity, Montgomery Potential.
Potential Vorticity (PV=2)* Level parameters include:
U- and V-velocity, Ozone Mass Mixing Ratio.
*Potential vorticity takes the value 2PVU in the northern hemisphere and -2PVU in the southern hemisphere (1PVU=10-6 m2 s-1 K kg-1). BADC management of ERA-40 data (11): What other parameters? ECMWF ERA-40 Re-analysis
Slide27 : We have a number of tools that can read GRIB data. However, I suggest the grib2brief utility on Tornado and the XCONV/CONVSH package as the most user friendly. BADC management of ERA-40 data (12): Reading the data ECMWF ERA-40 Re-analysis Using grib2brief for a quick look Using XCONV for conversions/plotting etc
Slide28 : Outline of this session Introduction to the ERA-40 Project
The model used for ERA-40
What DATA will be available?
BADC management of ERA-40 data
Future plans
Further information: useful documentation ECMWF ERA-40 Re-analysis
Slide29 : Future plans for ERA-40 (1): Data issues On the Data front, we aim to: ECMWF ERA-40 Re-analysis
Create an archive of NetCDF data and of 1 x 1 gridded data (or tools to convert instantaneously) when disk/tape space available.
Catch up with the production of data from all streams.
Extract time-series data, vertical integral data, means, variances and co-variances.
Slide30 : Future plans for ERA-40 (2): Software developments
On the Software front, we will:
ECMWF ERA-40 Re-analysis
create some makeshift Perl scripts to allow users to manipulate ERA-40 files at the command line on Tornado (probably interfacing with CONVSH software).
connect the Live Access Server (LAS) to the GRIB data, allowing users to locate, subset and plot the data through a web-interface.
The NERC DataGrid proposal (e-Science) will, if successful, develop/utilise a range of tools to allow data manipulation, plotting, subsetting etc,.
Slide31 : Future plans for ERA-40 (3): Access rules The ECMWF workshop on ERA-40 in November included a session on the rules governing dissemination of the data. You may recall the objective:
To foster European and international research by making the observational archive, the analyses and the study reports widely available. ECMWF ERA-40 Re-analysis The BADC has been suggested as a central archive for European (or possibly Global) access to the data, extending the current national distribution rules. This issue is currently undecided. The ERA-40 Observations may become available for distribution as well as the processed data – such work would also dovetail with elements of the PARADISE (e-Science) proposal submitted in January. This is still a speculative statement.
Slide32 : Further information (1): External web pages http://owms.ecmwf.int/research/era/index.html General ERA-40 documentation: ECMWF ERA-40 Re-analysis http://owms.ecmwf.int/research/era/Project_plan_1.html ERA-40 Project Plan: http://owms.ecmwf.int/research/era/Products.html ERA-40 Archive Plan:
Slide33 : Further information (2): Internal web pages http://www.badc.rl.ac.uk/data/ecmwf-e40/index.html
http://www.badc.rl.ac.uk/data/ecmwf-e40/e40_background.html
http://www.badc.rl.ac.uk/data/ecmwf-e40/status.html
http://www.badc.rl.ac.uk/data/ecmwf-e40/params.html
http://www.badc.rl.ac.uk/data/ecmwf-e40/file_naming.html
Please have a read through the ERA-40 documentation and give me any feedback. ECMWF ERA-40 Re-analysis This powerpoint presentation will be available on the web at:
http://www.badc.rl.ac.uk*******