Automatic Cell Measures

If you have worked with CMIP data before, you know that cell measure information like areacella is needed to take proper area-weighted means/summations. Yet many times, model centers have not uploaded this information uniformly in all submissions. This can be frustrating for the user.

In intake-esgf, when you call to_dataset_dict(), we perform a search for each dataset being placed in the dataset dictionary, progressively dropping facets to find, if possible, the cell measures that are closest to the dataset being downloaded. Sometimes they are simply in another variant_label, but other times they could be in a different activity_id. No matter where they are, we find them for you and add them to your dataset by default (disable with add_measures=False).

Consider the following search for data with UKESM1-0-LL. We are looking for a land variable gpp, the gross primary productivity.

from intake_esgf import ESGFCatalog
cat = ESGFCatalog().search(
    variable_id="gpp",
    source_id="UKESM1-0-LL",
    variant_label="r2i1p1f2",
    frequency="mon",
    experiment_id="historical",
)
dsd = cat.to_dataset_dict()

The progress bar will let you know that we are searching for cell measure information. We determine which measures need downloaded by looking in the dataset attributes. Since gpp is a land variable, we see that its cell_measures ='area: areacella' which indicates that this data should be also downloaded. However you will also find where land in the cell_methods meaning that we also need sftlf, the land fractions. If you look at the resulting dataset, you will find that both have been associated.

dsd["gpp"]

<xarray.Dataset> Size: 230MB
Dimensions:    (time: 1980, bnds: 2, lat: 144, lon: 192)
Coordinates:
  * time       (time) object 16kB 1850-01-16 00:00:00 ... 2014-12-16 00:00:00
  * lat        (lat) float64 1kB -89.38 -88.12 -86.88 ... 86.88 88.12 89.38
  * lon        (lon) float64 2kB 0.9375 2.812 4.688 6.562 ... 355.3 357.2 359.1
    type       |S4 4B ...
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) object 32kB dask.array<chunksize=(1, 2), meta=np.ndarray>
    lat_bnds   (time, lat, bnds) float64 5MB dask.array<chunksize=(1200, 144, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 6MB dask.array<chunksize=(1200, 192, 2), meta=np.ndarray>
    gpp        (time, lat, lon) float32 219MB dask.array<chunksize=(1, 144, 192), meta=np.ndarray>
    sftlf      (lat, lon) float32 111kB ...
    areacella  (lat, lon) float32 111kB ...
Attributes: (12/46)
    Conventions:            CF-1.7 CMIP-6.2
    activity_id:            CMIP
    branch_method:          standard
    branch_time_in_child:   0.0
    branch_time_in_parent:  113400.0
    creation_date:          2019-07-04T10:57:56Z
    ...                     ...
    title:                  UKESM1-0-LL output prepared for CMIP6
    variable_id:            gpp
    variant_label:          r2i1p1f2
    license:                CMIP6 model data produced by the Met Office Hadle...
    cmor_version:           3.4.0
    tracking_id:            hdl:21.14100/8a19464e-4fff-4ccd-b45c-6c0c79f7e70a

xarray.Dataset

• Dimensions:
  • time: 1980
  • bnds: 2
  • lat: 144
  • lon: 192
• Coordinates: (4)
  • time
    (time)
    object
    1850-01-16 00:00:00 ... 2014-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array([cftime.Datetime360Day(1850, 1, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(1850, 2, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(1850, 3, 16, 0, 0, 0, 0, has_year_zero=True), ...,
           cftime.Datetime360Day(2014, 10, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2014, 11, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.Datetime360Day(2014, 12, 16, 0, 0, 0, 0, has_year_zero=True)],
          dtype=object)

  • lat
    (lat)
    float64
    -89.38 -88.12 ... 88.12 89.38

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    array([-89.375, -88.125, -86.875, -85.625, -84.375, -83.125, -81.875, -80.625,
           -79.375, -78.125, -76.875, -75.625, -74.375, -73.125, -71.875, -70.625,
           -69.375, -68.125, -66.875, -65.625, -64.375, -63.125, -61.875, -60.625,
           -59.375, -58.125, -56.875, -55.625, -54.375, -53.125, -51.875, -50.625,
           -49.375, -48.125, -46.875, -45.625, -44.375, -43.125, -41.875, -40.625,
           -39.375, -38.125, -36.875, -35.625, -34.375, -33.125, -31.875, -30.625,
           -29.375, -28.125, -26.875, -25.625, -24.375, -23.125, -21.875, -20.625,
           -19.375, -18.125, -16.875, -15.625, -14.375, -13.125, -11.875, -10.625,
            -9.375,  -8.125,  -6.875,  -5.625,  -4.375,  -3.125,  -1.875,  -0.625,
             0.625,   1.875,   3.125,   4.375,   5.625,   6.875,   8.125,   9.375,
            10.625,  11.875,  13.125,  14.375,  15.625,  16.875,  18.125,  19.375,
            20.625,  21.875,  23.125,  24.375,  25.625,  26.875,  28.125,  29.375,
            30.625,  31.875,  33.125,  34.375,  35.625,  36.875,  38.125,  39.375,
            40.625,  41.875,  43.125,  44.375,  45.625,  46.875,  48.125,  49.375,
            50.625,  51.875,  53.125,  54.375,  55.625,  56.875,  58.125,  59.375,
            60.625,  61.875,  63.125,  64.375,  65.625,  66.875,  68.125,  69.375,
            70.625,  71.875,  73.125,  74.375,  75.625,  76.875,  78.125,  79.375,
            80.625,  81.875,  83.125,  84.375,  85.625,  86.875,  88.125,  89.375])

  • lon
    (lon)
    float64
    0.9375 2.812 4.688 ... 357.2 359.1

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    array([  0.9375,   2.8125,   4.6875,   6.5625,   8.4375,  10.3125,  12.1875,
            14.0625,  15.9375,  17.8125,  19.6875,  21.5625,  23.4375,  25.3125,
            27.1875,  29.0625,  30.9375,  32.8125,  34.6875,  36.5625,  38.4375,
            40.3125,  42.1875,  44.0625,  45.9375,  47.8125,  49.6875,  51.5625,
            53.4375,  55.3125,  57.1875,  59.0625,  60.9375,  62.8125,  64.6875,
            66.5625,  68.4375,  70.3125,  72.1875,  74.0625,  75.9375,  77.8125,
            79.6875,  81.5625,  83.4375,  85.3125,  87.1875,  89.0625,  90.9375,
            92.8125,  94.6875,  96.5625,  98.4375, 100.3125, 102.1875, 104.0625,
           105.9375, 107.8125, 109.6875, 111.5625, 113.4375, 115.3125, 117.1875,
           119.0625, 120.9375, 122.8125, 124.6875, 126.5625, 128.4375, 130.3125,
           132.1875, 134.0625, 135.9375, 137.8125, 139.6875, 141.5625, 143.4375,
           145.3125, 147.1875, 149.0625, 150.9375, 152.8125, 154.6875, 156.5625,
           158.4375, 160.3125, 162.1875, 164.0625, 165.9375, 167.8125, 169.6875,
           171.5625, 173.4375, 175.3125, 177.1875, 179.0625, 180.9375, 182.8125,
           184.6875, 186.5625, 188.4375, 190.3125, 192.1875, 194.0625, 195.9375,
           197.8125, 199.6875, 201.5625, 203.4375, 205.3125, 207.1875, 209.0625,
           210.9375, 212.8125, 214.6875, 216.5625, 218.4375, 220.3125, 222.1875,
           224.0625, 225.9375, 227.8125, 229.6875, 231.5625, 233.4375, 235.3125,
           237.1875, 239.0625, 240.9375, 242.8125, 244.6875, 246.5625, 248.4375,
           250.3125, 252.1875, 254.0625, 255.9375, 257.8125, 259.6875, 261.5625,
           263.4375, 265.3125, 267.1875, 269.0625, 270.9375, 272.8125, 274.6875,
           276.5625, 278.4375, 280.3125, 282.1875, 284.0625, 285.9375, 287.8125,
           289.6875, 291.5625, 293.4375, 295.3125, 297.1875, 299.0625, 300.9375,
           302.8125, 304.6875, 306.5625, 308.4375, 310.3125, 312.1875, 314.0625,
           315.9375, 317.8125, 319.6875, 321.5625, 323.4375, 325.3125, 327.1875,
           329.0625, 330.9375, 332.8125, 334.6875, 336.5625, 338.4375, 340.3125,
           342.1875, 344.0625, 345.9375, 347.8125, 349.6875, 351.5625, 353.4375,
           355.3125, 357.1875, 359.0625])

  • type
    ()
    |S4
    ...

    long_name :

    Land area type

    standard_name :

    area_type

    [1 values with dtype=|S4]

• Data variables: (6)
  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(1, 2), meta=np.ndarray>

    Array Chunk Bytes 30.94 kiB 16 B Shape (1980, 2) (1, 2) Dask graph 1980 chunks in 5 graph layers Data type object numpy.ndarray

  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(1200, 144, 2), meta=np.ndarray>

    Array Chunk Bytes 4.35 MiB 2.64 MiB Shape (1980, 144, 2) (1200, 144, 2) Dask graph 2 chunks in 7 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(1200, 192, 2), meta=np.ndarray>

    Array Chunk Bytes 5.80 MiB 3.52 MiB Shape (1980, 192, 2) (1200, 192, 2) Dask graph 2 chunks in 7 graph layers Data type float64 numpy.ndarray

  • gpp
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 144, 192), meta=np.ndarray>

    standard_name :

    gross_primary_productivity_of_biomass_expressed_as_carbon

    long_name :

    Carbon Mass Flux out of Atmosphere Due to Gross Primary Production on Land

    comment :

    The rate of synthesis of biomass from inorganic precursors by autotrophs ('producers') expressed as the mass of carbon which it contains. For example, photosynthesis in plants or phytoplankton. The producers also respire some of this biomass and the difference is referred to as the net primary production.

    units :

    kg m-2 s-1

    original_name :

    mo: (stash: m01s19i183, lbproc: 128) / ((SECONDS_IN_DAY: 86400.) * (DAYS_IN_YEAR: 360.))

    cell_methods :

    area: mean where land time: mean

    cell_measures :

    area: areacella

    history :

    2019-07-04T10:57:56Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20). 2019-07-04T10:57:56Z altered by CMOR: Converted type from 'd' to 'f'.

    Array Chunk Bytes 208.83 MiB 108.00 kiB Shape (1980, 144, 192) (1, 144, 192) Dask graph 1980 chunks in 5 graph layers Data type float32 numpy.ndarray

  • sftlf
    (lat, lon)
    float32
    ...

    standard_name :

    land_area_fraction

    long_name :

    Percentage of the grid cell occupied by land (including lakes)

    comment :

    Please express 'X_area_fraction' as the percentage of horizontal area occupied by X.

    units :

    %

    original_name :

    mo: (stash: m01s00i505)

    original_units :

    1

    history :

    2019-07-04T21:31:24Z altered by CMOR: Converted units from '1' to '%'. 2019-07-04T21:31:24Z altered by CMOR: Treated scalar dimension: 'type'. 2019-07-04T21:31:24Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).

    cell_methods :

    area: mean

    cell_measures :

    area: areacella

    [27648 values with dtype=float32]

  • areacella
    (lat, lon)
    float32
    ...

    standard_name :

    cell_area

    long_name :

    Grid-Cell Area for Atmospheric Grid Variables

    comment :

    For atmospheres with more than 1 mesh (e.g., staggered grids), report areas that apply to surface vertical fluxes of energy.

    units :

    m2

    original_name :

    mo: areacella((stash: m01s00i505))

    cell_methods :

    area: sum

    history :

    2019-07-04T21:31:24Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20). 2019-07-04T21:31:24Z altered by CMOR: Converted type from 'd' to 'f'.

    [27648 values with dtype=float32]

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(CFTimeIndex([1850-01-16 00:00:00, 1850-02-16 00:00:00, 1850-03-16 00:00:00,
                 1850-04-16 00:00:00, 1850-05-16 00:00:00, 1850-06-16 00:00:00,
                 1850-07-16 00:00:00, 1850-08-16 00:00:00, 1850-09-16 00:00:00,
                 1850-10-16 00:00:00,
                 ...
                 2014-03-16 00:00:00, 2014-04-16 00:00:00, 2014-05-16 00:00:00,
                 2014-06-16 00:00:00, 2014-07-16 00:00:00, 2014-08-16 00:00:00,
                 2014-09-16 00:00:00, 2014-10-16 00:00:00, 2014-11-16 00:00:00,
                 2014-12-16 00:00:00],
                dtype='object', length=1980, calendar='360_day', freq='30D'))

  • lat
    PandasIndex

    PandasIndex(Index([-89.375, -88.125, -86.875, -85.625, -84.375, -83.125, -81.875, -80.625,
           -79.375, -78.125,
           ...
            78.125,  79.375,  80.625,  81.875,  83.125,  84.375,  85.625,  86.875,
            88.125,  89.375],
          dtype='float64', name='lat', length=144))

  • lon
    PandasIndex

    PandasIndex(Index([  0.9375,   2.8125,   4.6875,   6.5625,   8.4375,  10.3125,  12.1875,
            14.0625,  15.9375,  17.8125,
           ...
           342.1875, 344.0625, 345.9375, 347.8125, 349.6875, 351.5625, 353.4375,
           355.3125, 357.1875, 359.0625],
          dtype='float64', name='lon', length=192))

• Attributes: (46)

  Conventions :

  CF-1.7 CMIP-6.2

  activity_id :

  CMIP

  branch_method :

  standard

  branch_time_in_child :

  0.0

  branch_time_in_parent :

  113400.0

  creation_date :

  2019-07-04T10:57:56Z

  cv_version :

  6.2.20.1

  data_specs_version :

  01.00.29

  experiment :

  all-forcing simulation of the recent past

  experiment_id :

  historical

  external_variables :

  areacella

  forcing_index :

  2

  frequency :

  mon

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.MOHC.UKESM1-0-LL.historical.none.r2i1p1f2

  grid :

  Native N96 grid; 192 x 144 longitude/latitude

  grid_label :

  gn

  history :

  2019-07-04T10:46:49Z ; CMOR rewrote data to be consistent with CMIP6, CF-1.7 CMIP-6.2 and CF standards.; 2019-07-04T10:46:36Z MIP Convert v1.1.2, Python v2.7.12, Iris v1.13.0, Numpy v1.13.3, netcdftime v1.4.1.

  initialization_index :

  1

  institution :

  Met Office Hadley Centre, Fitzroy Road, Exeter, Devon, EX1 3PB, UK

  institution_id :

  MOHC

  mip_era :

  CMIP6

  mo_runid :

  u-bc292

  nominal_resolution :

  250 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  piControl

  parent_mip_era :

  CMIP6

  parent_source_id :

  UKESM1-0-LL

  parent_time_units :

  days since 1850-01-01-00-00-00

  parent_variant_label :

  r1i1p1f2

  physics_index :

  1

  product :

  model-output

  realization_index :

  2

  realm :

  land

  source :

  UKESM1.0-LL (2018): aerosol: UKCA-GLOMAP-mode atmos: MetUM-HadGEM3-GA7.1 (N96; 192 x 144 longitude/latitude; 85 levels; top level 85 km) atmosChem: UKCA-StratTrop land: JULES-ES-1.0 landIce: none ocean: NEMO-HadGEM3-GO6.0 (eORCA1 tripolar primarily 1 deg with meridional refinement down to 1/3 degree in the tropics; 360 x 330 longitude/latitude; 75 levels; top grid cell 0-1 m) ocnBgchem: MEDUSA2 seaIce: CICE-HadGEM3-GSI8 (eORCA1 tripolar primarily 1 deg; 360 x 330 longitude/latitude)

  source_id :

  UKESM1-0-LL

  source_type :

  AOGCM AER BGC CHEM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  Lmon

  table_info :

  Creation Date:(13 December 2018) MD5:2b12b5db6db112aa8b8b0d6c1645b121

  title :

  UKESM1-0-LL output prepared for CMIP6

  variable_id :

  gpp

  variant_label :

  r2i1p1f2

  license :

  CMIP6 model data produced by the Met Office Hadley Centre is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https://ukesm.ac.uk/cmip6. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  cmor_version :

  3.4.0

  tracking_id :

  hdl:21.14100/8a19464e-4fff-4ccd-b45c-6c0c79f7e70a

What makes this particular example difficult is that the cell measures for this model are only found in the piControl experiment, for the r1i1p1f2 variant. Our methods finds the right measures, which you can see by printing out the session log and looking for which areacella files are downloaded / accessed.

print(cat.session_log())

2024-05-02 17:59:43 search begin variable_id=['gpp'], source_id=['UKESM1-0-LL'], variant_label=['r2i1p1f2'], frequency=['mon'], experiment_id=['historical'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]
2024-05-02 17:59:44 combine_time=0.01
2024-05-02 17:59:44 search end total_time=0.94
2024-05-02 17:59:44 file info begin
2024-05-02 17:59:45 file info end total_time=1.26
2024-05-02 17:59:45 begin move_data
2024-05-02 17:59:49 transfer_time=3.53 [s] at 6.71 [Mb s-1] http://esgf-node.ornl.gov/thredds/fileServer/css03_data/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r2i1p1f2/Lmon/gpp/gn/v20190708/gpp_Lmon_UKESM1-0-LL_historical_r2i1p1f2_gn_195001-201412.nc
2024-05-02 17:59:53 transfer_time=7.80 [s] at 5.22 [Mb s-1] http://esgf-node.ornl.gov/thredds/fileServer/css03_data/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r2i1p1f2/Lmon/gpp/gn/v20190708/gpp_Lmon_UKESM1-0-LL_historical_r2i1p1f2_gn_185001-194912.nc
2024-05-02 17:59:53 end move_data
2024-05-02 17:59:53 search begin variant_label=['r2i1p1f2'], source_id=['UKESM1-0-LL'], mip_era=['CMIP6'], activity_id=['CMIP'], experiment_id=['historical'], grid_label=['gn'], table_id=['fx', 'Ofx'], variable_id=['sftlf'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]
2024-05-02 17:59:54 search end no results
2024-05-02 17:59:54 search begin source_id=['UKESM1-0-LL'], mip_era=['CMIP6'], activity_id=['CMIP'], experiment_id=['historical'], grid_label=['gn'], table_id=['fx', 'Ofx'], variable_id=['sftlf'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]
2024-05-02 17:59:56 search end no results
2024-05-02 17:59:56 search begin source_id=['UKESM1-0-LL'], mip_era=['CMIP6'], activity_id=['CMIP'], grid_label=['gn'], table_id=['fx', 'Ofx'], variable_id=['sftlf'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]
2024-05-02 17:59:57 combine_time=0.00
2024-05-02 17:59:57 search end total_time=1.23
2024-05-02 17:59:57 file info begin
2024-05-02 17:59:58 file info end total_time=1.03
2024-05-02 17:59:58 begin move_data
2024-05-02 17:59:58 transfer_time=0.07 [s] at 1.23 [Mb s-1] http://esgf-node.ornl.gov/thredds/fileServer/css03_data/CMIP6/CMIP/MOHC/UKESM1-0-LL/piControl/r1i1p1f2/fx/sftlf/gn/v20190705/sftlf_fx_UKESM1-0-LL_piControl_r1i1p1f2_gn.nc
2024-05-02 17:59:58 end move_data
2024-05-02 17:59:58 search begin variant_label=['r2i1p1f2'], source_id=['UKESM1-0-LL'], mip_era=['CMIP6'], activity_id=['CMIP'], experiment_id=['historical'], grid_label=['gn'], table_id=['fx', 'Ofx'], variable_id=['areacella'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]
2024-05-02 18:00:00 search end no results
2024-05-02 18:00:00 search begin source_id=['UKESM1-0-LL'], mip_era=['CMIP6'], activity_id=['CMIP'], experiment_id=['historical'], grid_label=['gn'], table_id=['fx', 'Ofx'], variable_id=['areacella'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]
2024-05-02 18:00:02 search end no results
2024-05-02 18:00:02 search begin source_id=['UKESM1-0-LL'], mip_era=['CMIP6'], activity_id=['CMIP'], grid_label=['gn'], table_id=['fx', 'Ofx'], variable_id=['areacella'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]
2024-05-02 18:00:05 combine_time=0.00
2024-05-02 18:00:05 search end total_time=2.73
2024-05-02 18:00:05 file info begin
2024-05-02 18:00:06 file info end total_time=1.12
2024-05-02 18:00:06 begin move_data
2024-05-02 18:00:06 transfer_time=0.06 [s] at 1.01 [Mb s-1] http://esgf-node.ornl.gov/thredds/fileServer/css03_data/CMIP6/CMIP/MOHC/UKESM1-0-LL/piControl/r1i1p1f2/fx/areacella/gn/v20190705/areacella_fx_UKESM1-0-LL_piControl_r1i1p1f2_gn.nc
2024-05-02 18:00:06 end move_data