European Centre for Medium Range Weather Forecasts Diagnostic Analyses (ECMWFDIAG)

Information updated on July 3, 2013, 6:19 pm GMT


General Data Description

Diagnostic Analyses data are derived from ECMWF model runs in simulated vertical profiles of diagnostic variables at specific locations or for area averages. The Physical Parametrization Group at ECMWF generates these data as a courtesy to ARM for all the ACRF sites including the changing location of the mobile facility.

These data can only be distributed to ARM scientists. ARM scientists who obtain these data must agree to acknowledge use of the data in their publications and not to share the data with others who are not on the ARM Science Team.

Derived data are defined as data from which the ECMWF fields cannot be re-created.

Such data can be freely distributed by ARM scientists.

The distribution of raw ECMWF fields to third parties is not allowed without the explicit permission of ECMWF. This is the general rule for all data supplied by ECMWF.

Permission can be requested to supply data to groups outside the ARM project for specific projects. In such cases, if permission was given, a conditions form would be produced by ECMWF to be signed by the principal investigator of such a group.

As of 07/25/2012, all ecmwf data covering TWP sites were reprocessed using the new facility ID. Please see the note in the section of reprocessing history for the details.

Expand All Collapse All

Data Stream Names

  • ecmwfflx : ECMWF: radiative fluxes at altitude, 1-hr avg, entire coverage
  • ecmwfsfc : ECMWF: surface variables and fluxes, entire coverage, 1-hr avg
  • ecmwfsfc1l : ECMWF: surface variables and fluxes, single layer, 1-hr avg
  • ecmwfsfce : ECMWF: near-surface and surface variables (extra), entire coverage, 1-hr avg
  • ecmwfsfcml : ECMWF: model multilevel surface fields at 4 levels, entire coverage
  • ecmwften : ECMWF: total and physical tendencies for met and cloud vars, entire coverage, 1-hr avg
  • ecmwfvar : ECMWF: model met. and cloud variables at altitude, entire coverage, 1-hr avg

Measurement Description

ecmwfflx
These files contain the radiative fluxes throughout the atmosphere. The fluxes are averages over the last hour.
They are archived on the so called flux levels of the model (often referred to as half-levels). The full levels on which the variables are stored are placed between two flux levels. Hence, the largest level number represents the surface and level 1 the top of the atmosphere. Full level k is between half level k and k+1. The information of level can be found in ecmwfvar data.
  • net shortwave flux (W/m^2)
  • net longwave flux (W/m^2)
    In addition, new measurements were added on 20101207:
  • Sensible heat flux ( (W/m^2)
  • Turbulent momentum flux - u component (Pa)
  • Turbulent momentum flux - v component (Pa)
ecmwfsfc (through 2000-09-12)
These files contain surface variables and fluxes. The level number is set to 1.
The measurements are:
  • soil temperature in layer 1 (7 cm deep) (K) **
  • shortwave radiation (W/m^2)
  • longwave radiation (W/m^2)
  • latent heat flux over water (W/m^2) **
  • latent heat flux over ice/snow (W/m^2) **
  • sensible heat flux (W/m^2)
  • ground heat flux between layer 1 and layers 2/3/4 (W/m^2) **
  • soil moisture in layer 1 (m^3/m^3) **
  • large-scale rain (mm/s)
  • convective rain (mm/s)
  • large-scale snow (mm/s)
  • convective snow (mm/s)
  • surface pressure (Pa)
  • turbulent moisture flux kg/kg * kg/(m^2*s)) ++
** measurements in nsa and sgp files; no longer contained in new data stream immediately following.
++ measurement in twp files; no longer contained in new data stream immediately following.
Note the following two data streams replace ecmwfsfc beginning 2000-09-13)
The change in datastreams and measurements are a direct result of changes in the land-surface parametrization which became operational for the forecast starting 2000-09-12. The surface, "sfc", data was split into two types of files - sfc1l and sfcml. "1l" stands for 1 layer and is available for all sites. "ml" stands for multi-level and are only available for sites which are considered land in the model (SGP and NSA and Darwin (DAR) for TWP also the mobile facilities PYE, NIM, FKB, HFE).
ecmwfsfc1l (beginning 2000-09-13)
This data stream contains the same measurements as ecmwfsfc which it replaces with the exceptions noted above. The new data stream also contains consistent measurement names across all the sites. The surface level for the TWP sites is now 1; in the older version of these files, it was set to 61. The measurements that are carried forward are:
  • shortwave radiation (W/m^2)
  • longwave radiation (W/m^2)
  • sensible heat flux (W/m^2)
  • large-scale rain (mm/s)
  • convective rain (mm/s)
  • large-scale snow (mm/s)
  • convective snow (mm/s)
  • surface pressure (Pa)
In addition, a new measurement has been added:
  • surface latent heat flux (W/m^2)
ecmwfsfcml (beginning 2000-09-13)
These measurements are all multilevel at 4 levels as represented in the ECMWF land-surface parametrization. The soil model layers have the following depths starting from the top layer (layer 1 in the files):
  1. 7 cm
  2. 21 cm
  3. 72 cm
  4. 189 cm
The measurements are:
  • soil temperature (K)
  • heat flux at layer bottom (W/m^2)
  • moisture flux at layer bottom (kg/(m^2*s))
  • soil wetness (kg/m^2)
ecmwften
These files contain the total and physical tendencies for u,v,T and q on the model levels. The units are as in the ecmwfvar files but per second.
The tendencies are:
  • u total (m/s/s)
  • u physics (m/s/s)
  • v total (m/s/s)
  • v physics (m/s/s)
  • T total (K/s)
  • T physics (K/s)
  • q total (kg/kg/s)
  • q physics (kg/kg/s)
ecmwfvar
These files contain the model variables on each model level. Level 1 is the top full level of the model (10 hPa). The level with the highest number is the lowest altitude full level.
The measurements include:
  • p - pressure (Pa)
  • u - zonal wind component (m/s)
  • v - meridional wind component (m/s)
  • T - temperature (K)
  • q - specific humidity (kg/kg)
  • l - specific cloud liquid water content (kg/kg)
  • i - specific cloud ice content (kg/kg)
  • a - cloud fraction (percent/100.)
  • R - relative humidity (percent/100.)
  • w - omega=vertical velocity in pressure coordinates (Pa/s)
ecmwfsfce (beginning January 1998)
These files contain extra near surface and surface variables including:
  • 2m Temperature (K)
  • 2m specific humidity (kg/kg)
  • 10m wind u-component (m/s)
  • 10m wind v-component (m/s)
  • skin temperature (K)
  • albedo
  • downward surface solar radiation (W/m^2)
  • downward surface longwave radiation (W/m^2)
    In addition, a new measurement was added on 20101207:
  • Surface evaporation (Kg/m^2)

Temporal Coverage

The number of distinct locations that ECMWF can configure in their diagnostic analysis product is limited. Therefore, and since the ECMWF Physical Parametrization group is producing these products as a courtesy to ARM, a prior location often needs to be terminated before a new one can be brought online. The data are therefore not necessarily continuous for all ARM locations. Here's a summary as of 2009-07-29:
SiteBegin Date/End Date
NSA1997-09
SGP IOPs1995-04-17/05-07,
1995-07-17/08-14,
1995-09-22/11-01,
1996-04-01/04-30,
1996-05-01/05-10,
1996-07-15/08-05
SGP1996-08
TWP Darwin2002-03-20
TWP Manus1997-06
TWP Nauru1999-01
NIM2005-05-05/2007-03-27
PYE2005-04-20/2005-10-21
FKB2007-04-01/2008-02-29
HFE2008-05-01/2009-04-30
GRW2009-05-01- present
PGH2011-10-01-present
GAN2011-10-01-present
SMT2011-11-15-present
PVC2012-07-01-present
Note: sgpecmwfsfcX1.c1 (netcdf files) begin 1996-08-01; prior to that only ascii files are available for the surface data.

These diagnostic files are available at the ARM Archive with an approximately one month delay.

Area Covered

The single column data in the diagnostic files are either individual ECMWF model grid points or averages over several grid points in a rectangular domain defined by their upper left (NW) and lower right (SE) corners. The coordinates of the single grid points in the file are the closest model grid point coordinates to the desired location. When the resolution of the model changes (as it did on 2000-11-22) that can result in a change in those coordinates, see Data User Notes below.

SGP

There are three SGP domains over which all the quantities are averaged.
They are numbered 27,28,29. The areas for those domains are:
point numberLongitude upper leftLatitude upper leftLongitude lower rightLatitude lower rightcomment
27: 260.338.8264.634.8
28: 261.734.8263.34.2 (eliminated 2002-03-20)
29: 261.737.263.36.
SGP
ECMWF areas

NSA

The NSA diagnostic data are available for two single points (numbered 13 and 19) located at:

point numberLongitudeLatitudeComment
13: 202.00 70.41 (beginning 2002-03-20)
19: 203.20 71.05 (before 2000-11-22)
19: 204.00 71.12 (beginning 2000-11-22)
NSA
ECMWF areas

TWP DAR

There are three single points representing TWP Darwin. These data begin on 2002-03-20. Other domains in SGP, TWP Manus and TWP Nauru were eliminated to make room for these new points and for the new NSA point.
The Darwin points are number 6, 28 and 50. The points are:

point numberLongitudeLatitude
6: 130.43 -12.82
28: 129.73 -12.47
50: 131.13 -13.52

TWP MAN

There are three TWP Manus domains over which all the quantities are averaged.
They are numbered 48,49,50. The areas for those domains are:

point numberLongitudeLatitudeComment
48: 147.30 -1.93 (before 2000-11-22)
48: 147.37 -1.97 (beginning 2000-11-22)
point numberLongitude upper leftLatitude upper leftLongitude lower rightLatitude lower rightcomment
49: 146.60 -1.10 148.10 -2.80
50: 146.00 -0.60 148.80 -3.30 (eliminated 2002-03-20)
TWP Manus ECMWF areas

TWP NAU

There are three domains for the TWP Nauru diagnostic data. They are numbered 4, 5, 6.
The areas for those domains are:

point numberLongitudeLatitudeComment
4: 167.06 -0.28 (before 2000-11-22)
4: 167.99 -0.53 (beginning 2000-11-22)
point numberLongitude upper leftLatitude upper leftLongitude lower rightLatitude lower rightcomment
5: 166.10 0.30 167.70 -1.30
6: 165.50 0.90 168.30 -1.90 (eliminated 2002-03-20)
TWP Nauru ECMWF areas

The Manus and Nauru areas in the ECMWF model are all model sea points.

AMF PYE 2005

The Point Reyes diagnostic domain consists of two single points:
point numberLongitudeLatitude
18 237.08 38.11
23 236.67 38.11

AMF NIM 2006

The Niamey diagnostic data domain consists of a single point
point numberLongitudeLatitude
76 2.46 13.52

AMF FKB 2007

The Black Forest diagnostic data domain consists of 3 COPS super-site locations:
point numberLongitudeLatitudeNearest Supersite
115 8.32 48.46Supersite M, ARM AMF Murgtal, Heselbach: 8.405E, 48.545N, ca. 500 m
116 8.67 48.68Supersite S, Gliders Airport Deckenpfronn, 8.813E, 48.635N, ca. 600 m
117 7.67 48.68Location POLDIRAD, Waltenheim sur Zorn 7.610E, 48.739N, ca. 120 m

AMF HFE 2008

The China diagnostic data domain consists of 4 COPS super-site locations:
point numberLongitudeLatitudeNearest Supersite
14 120.25 31.37ARM AMF Taihu: 120.22E, 31.42N, ca. 7 m
15 117.00 39.69ARM AMF Xianghe: 116.96E, 39.75N, ca. 36 m
115 116.75 32.49ARM AMF Shouxian: 116.78E, 32.56N, ca. 23 m
116 100.69 39.01ARM AMF Zhanghe, 100.58E, 38.93N, ca. 1483 m

AMF GRW 2009

The Graciosa Island, Azores diagnostic data domain consists of a single point:
point numberLongitudeLatitudeNearest Supersite
14 331.88 39.01Graciosa Island, Azores: 39 5' 28" N, 28 1' 45" W, ca. 15.24 m

AMF PGH 2011

The Ganges Valley, India diagnostic data domain consists of two points:
point numberLongitudeLatitudeNearest Supersite
11979.53 29.31Ganges Valley, India: 29 21' 33.84" N, 79 27' 29.27" E, ca 1943m
12080.8526.78Lucknow, India: 26 50? 24? N, 80 57? 36? E

AMF GAN 2011

The Gan Island, Maldives diagnostic data domain consists of one point:
point numberLongitudeLatitudeNearest Supersite
12173.12 -0.63Gan Island, Maldives: 73.14990 East, 0.69060 South, with ca. 1m.

AMF SMT 2011

Summit Station, Greenland diagnostic data domain consists of one point:
point numberLongitudeLatitudeNearest Supersite
116321.75-72.60Summit Station, Greenland:72 34? 46.5? N, 38 27? 33.07? W with ca. 3216m

AMF PVC 2012

Highlands, Cape Cod, MA diagnostic data domain consists of eleven points:
point numberLongitudeLatitudeNearest Supersite
124 290.00 41.96 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
125 289.81 41.68 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
126 292.31 43.51 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
127 292.50 43.08 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
128 292.88 42.66 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
129 292.59 41.96 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
130 292.68 41.26 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
131 292.14 40.55 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
132 291.78 40.13 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
133 290.16 39.85 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m
134 287.28 40.70 Highlands, Cape Cod, MA:42.03048 N, 70.04932 W with ca. 47.9m

Data Stream Inputs

The diagnostic analysis files are received in ASCII format (1 file per point per type [var,ten,sfc1l,sfcml,sfce]) from the ECMWF and archived in daily tar bundles in the following .00 level data-streams:
pvcecmwfX1.00
pghecmwfX1.00
ganecmwfX1.00
smtecmwfX1.00
fkbecmwfX1.00
grwecmwfX1.00
hfeecmwfX1.00
nimecmwfX1.00
nsaecmwfX1.00
pyeecmwfX1.00
sgpecmwfX1.00
twpecmwfX10.00
twpecmwfX20.00
twpecmwfX30.00

Contacts

Richard Wagener

(631) 344-5886
wagener@bnl.gov

Shaocheng Xie

Translator
(925) 422-6023
xie2@llnl.gov

Lynn Ma

associate
(631) 344-3813
malynn@bnl.gov

Data Source

  • Institution
  • Physical Parametrization Group at ECMWF

Reprocessing History

  • Reprocessing Date
  • Manus(X10): 199706-201206
    Nauru(X20): 199901-201206
    Darwin(X30): 200203-201206

  • Reason Reprocessing
  • 
    All twpecmwf data were reprocessed with the new facility ID (X10 -man, X20- 
    nau and X30-dar). Below is the comparative chart of the change.
    OLD NEW

    twpecmwfX1.00 -----> twpecmwfX10.00
    twpecmwfmanflxX1.c1 ----> twpecmwfflxX10.c1
    twpecmwfmansfcX1.c1 ----> twpecmwfsfcX10.c1
    twpecmwfmansfc1lX1.c1 ----> twpecmwfsfc1lX10.c1
    twpecmwfmansfceX1.c1 ----> twpecmwfsfceX10.c1
    twpecmwfmantenX1.c1 ----> twpecmwftenX10.c1
    twpecmwfmanvarX1.c1 ----> twpecmwfvarX10.c1

    twpecmwfnauX1.00 -----> twpecmwfX20.00
    twpecmwfnauflxX1.c1 ----> twpecmwfflxX20.c1
    twpecmwfnausfcX1.c1 ----> twpecmwfsfcX20.c1
    twpecmwfnausfc1lX1.c1 ----> twpecmwfsfc1lX20.c1
    twpecmwfnausfceX1.c1 ----> twpecmwfsfceX20.c1
    twpecmwfnautenX1.c1 ----> twpecmwftenX20.c1
    twpecmwfnauvarX1.c1 ----> twpecmwfvarX20.c1

    twpecmwfdarX1.00 -----> twpecmwfX30.00
    twpecmwfdarflxX1.c1 ----> twpecmwfflxX30.c1
    twpecmwfdarsfc1lX1.c1 ----> twpecmwfsfc1lX30.c1
    twpecmwfdarsfcmlX1.c1 ----> twpecmwfsfcmlX30.c1
    twpecmwfdarsfceX1.c1 ----> twpecmwfsfceX30.c1
    twpecmwfdartenX1.c1 ----> twpecmwftenX30.c1
    twpecmwfdarvarX1.c1 ----> twpecmwfvarX30.c1

FAQ

Q: The old twpecmwf[xxx]sfcX1.c1 (pre 2000-09-13) files had "turbulent moisture flux". Does this match to anything in the new files (or to anything in SGP/NSA)?
A: (from Christian Jacob, ECMWF): "Yes and no. Due to the backing out procedure we had to apply for the TWP sites, we did not have the surface latent heat flux at those sites and replaced it by the moisture flux (which can be converted into the former). Now that we cleaned up the mess, we do have surface latent heat flux instead, which is cleaner. Again, it comes in different units and is therefore not a direct replacement, although it contains the same information."
Q: Often soil wetness in a generic sense is expressed in terms of (volumetric) soil moisture content, which is unitless or has units of m^3/m^3. For example, a corresponding old term was "soil moisture in layer 1 (m^3/m^3) (soil_moisture in sgp and nsa)." The new "soil wetness" seems to represent the amount of water integrated with depth for each layer. Can someone confirm that this interpretation is correct?
A: The interpretation of soil wetness in the question is correct.
Q: Is the model elevation at Barrow 8 m?
A: No, it is 5 m.
Q: Is there any information on model ozone available?
A: No, there is no information on ozone in any of these products.
Q: Are the variables in the diagnostic files to be interpreted as hourly averages for the hour prior to the time-stamp?
A: From Martin Khler at ECMWF: Yes and No! The answer depends on the type of variable:
for fluxes and tendencies: average over previous hour before time stamp (e.g. latent heat flux)
other variables: instantaneous value at time stamp (e.g. T2m or cloud cover)

Data User Notes

Acknowledgment:
Publications should refer to: "The ECMWF operational analysis and forecasting system: The full evolution http://www.ecmwf.int/products/data/operational_system/index.html"
Why single data points moved in diagnostic ECMWF files
To generate point data in the diagnostic files, ECMWF specifies to the model the lat-lon of the observation point, and get as an output the nearest model point. When the model grid changes (such as on 2000-11-22, where the model resolution increased from ~60 km to ~40 km), the location of the nearest model point changed.
Discussion from Christian Jacob on changes to diagnostic files on 2000-09-13.
In June 2000 we introduced a new land surface parametrization scheme into our model. We immediately realised that the diagnostic tool that we are using (amongst many other applications) to get out the data for the ARM sites was totally inadequate for the new scheme. This, together with some things we did not like even for the old parametrization scheme, led us to invest some time to completely rewrite the surface part of this diagnostic tool. The changes that you see are repercussions of this rewrite. The rewritten diagnostics became operational on 2000-09-12, hence affecting ARM data from 2000-09-13 onwards.

To give you an example of the sort of changes: The old diagnostic tool did not provide any direct information at the surface over sea points in the model. It so happens that Manus and Nauru are way too small islands to be land points even in our highest of model resolutions. Therefore, we had to play all sorts of tricks to back out things like surface fluxes from other parts of the archive. That is the reason why you had different files for the TWP sites in the past. Obviously, since we rewrote the software anyway, we wanted to get rid of that problem, and we did. Hence, now all the files are exactly the same. The only difference is that you have extra files at the locations that are really land in the model (SGP and NSA), that deal with real land parameters, such as soil temperature and soil moisture, which obviously don't exist at sea points.

The difference in measurement names between old and new are also due to the cleanup.

Vertical resolution change
The operational ECMWF model changed vertical resolution from 91 to 137 levels at 12Z 25 June 2013. it affected the data from 26 June onwards.

Acronyms

ECMWF	European Centre for Medium Range Forecasts
GRIB	GRIdded Binary

Citable References

"The ECMWF operational analysis and forecasting system: The full evolution http://www.ecmwf.int/products/data/operational_system/index.html"