Simplifying Search with Model Groups¶
At a simple level, you can think of intake-esgf as analagous to the ESGF web interface but where results are presented to you as a pandas dataframe in place of pages of web results. However, we believe that the user does not want to wade through either of these. Many times you want to see model results organized by unique combinations of source_id, member_id, and grid_label. That is to say, when you are going to perform an analysis, you would like your model outputs to be self-consistent and from the same run and grid, even across experiments. To assist you in honing in on what sets of results may be useful to your analysis, we introduce the notion of model groups.
Consider the following search, motivated by a desire to study controls (temperature, precipitation, soil moisture) on the carbon cycle (gross primary productivty) across a number of historical and future scenarios.
from intake_esgf import ESGFCatalog
cat = ESGFCatalog().search(
experiment_id=["historical", "ssp585", "ssp370", "ssp245", "ssp126"],
variable_id=["gpp", "tas", "pr", "mrso"],
table_id=["Amon", "Lmon"],
)
print(cat)
Summary information for 10662 results:
variable_id [tas, pr, gpp, mrso]
institution_id [DKRZ, CSIRO-ARCCSS, CSIRO, CNRM-CERFACS, CCCm...
table_id [Amon, Lmon]
grid_label [gn, gr, gr1]
datetime_start [2015-01-16T12:00:00Z, 2015-01-16T00:00:00Z, n...
member_id [r5i1p1f1, r3i1p1f1, r9i1p1f1, r2i1p1f1, r1i1p...
activity_drs [ScenarioMIP, CMIP, AerChemMIP]
experiment_id [ssp370, ssp245, ssp126, ssp585, historical]
mip_era [CMIP6, nan]
datetime_stop [2100-12-16T12:00:00Z, nan, 2099-12-16T12:00:0...
source_id [MPI-ESM1-2-HR, ACCESS-CM2, ACCESS-ESM1-5, CNR...
project [CMIP6]
dtype: object
Even if this exact application does not resonate with you, the situation is a familiar one. We have several thousand results with many different models and variants to sort through. To help guide you to which groups of models might be useful to you, we provide the following function.
cat.model_groups()
source_id member_id grid_label
ACCESS-CM2 r1i1p1f1 gn 20
r2i1p1f1 gn 17
r3i1p1f1 gn 17
r4i1p1f1 gn 20
r5i1p1f1 gn 18
..
UKESM1-0-LL r16i1p1f2 gn 15
r17i1p1f2 gn 12
r18i1p1f2 gn 13
r19i1p1f2 gn 15
UKESM1-1-LL r1i1p1f2 gn 12
Name: variable_id, Length: 874, dtype: int64
This returns a pandas series where the results have been grouped and sorted by source_id, member_id, and grid_label and the counts of datasets returned. Pandas will probably truncate this series. If you want to see the whole series, you can call print(cat.model_groups().to_string()) instead. However, as there are still several hundred possibile model groups, we will not show that here.
Removing Incomplete Groups¶
If you glance through the model groups, you will see that, relative to our search, many will be incomplete. By this we mean, that there are many model groups that will not have all the variables in all the experiments that we wish to include in our analysis. Since we are looking for 5 experiments and 4 variables, we need the model groups with 20 dataset results. We can check which groups satisfy this condition by operating on the model group pandas series.
mgs = cat.model_groups()
print(mgs[mgs==20])
source_id member_id grid_label
ACCESS-CM2 r1i1p1f1 gn 20
r4i1p1f1 gn 20
ACCESS-ESM1-5 r11i1p1f1 gn 20
r12i1p1f1 gn 20
r13i1p1f1 gn 20
..
MRI-ESM2-0 r3i1p1f1 gn 20
UKESM1-0-LL r1i1p1f2 gn 20
r2i1p1f2 gn 20
r3i1p1f2 gn 20
r8i1p1f2 gn 20
Name: variable_id, Length: 143, dtype: int64
The rest are incomplete and we would like a fast way to remove them from the search results. But the reality is that many times our completeness criteria is more complicated than just a number. In the above example, we may want all the variables for all the experiments, but if a model does not have a submission for, say, ssp126, that is acceptable.
intake-esgf provides an interface which uses a user-provided function to remove incomplete entries. Internally, we will loop over all model groups in the results and pass your function the portion of the dataframe that corresponds to the current model group. Your function then needs to return a boolean based on the contents of that sub-dataframe.
def should_i_keep_it(sub_df):
# this model group has all experiments/variables
if len(sub_df) == 20:
return True
# if any of these experiments is missing a variable, remove this
for exp in ["historical", "ssp585", "ssp370", "ssp245"]:
if len(sub_df[sub_df["experiment_id"] == exp]) != 4:
return False
# if the check makes it here, keep it
return True
Then we pass this function to the catalog by the remove_incomplete() function and observe how it has impacted the search results.
cat.remove_incomplete(should_i_keep_it)
print(cat.model_groups())
source_id member_id grid_label
ACCESS-CM2 r1i1p1f1 gn 20
r4i1p1f1 gn 20
ACCESS-ESM1-5 r11i1p1f1 gn 20
r12i1p1f1 gn 20
r13i1p1f1 gn 20
..
MRI-ESM2-0 r3i1p1f1 gn 20
UKESM1-0-LL r1i1p1f2 gn 20
r2i1p1f2 gn 20
r3i1p1f2 gn 20
r8i1p1f2 gn 20
Name: variable_id, Length: 145, dtype: int64
Removing Ensembles¶
Depending on the goals and scope of your analysis, you may want to use only a single variant per model. This can be challenging to locate as not all variants have all the experiments and models. However, now that we have removed the incomplete results, we can now call the remove_ensembles() function which will only keep the smallest member_id for each model group. By smallest, we mean that first entry after a hierarchical sort using the integer index values of each label in the member_id.
cat.remove_ensembles()
print(cat.model_groups())
source_id member_id grid_label
ACCESS-CM2 r1i1p1f1 gn 20
ACCESS-ESM1-5 r11i1p1f1 gn 20
BCC-CSM2-MR r1i1p1f1 gn 20
CanESM5 r1i1p1f1 gn 20
CanESM5-1 r7i1p1f1 gn 20
CanESM5-CanOE r1i1p2f1 gn 20
CESM2-WACCM r2i1p1f1 gn 16
CNRM-CM6-1 r2i1p1f2 gr 20
CNRM-CM6-1-HR r1i1p1f2 gr 20
CNRM-ESM2-1 r1i1p1f2 gr 20
EC-Earth3 r6i1p1f1 gr 20
EC-Earth3-Veg r12i1p1f1 gr 20
INM-CM4-8 r1i1p1f1 gr1 20
IPSL-CM6A-LR r1i1p1f1 gr 20
MIROC-ES2H r1i1p4f2 gn 20
MIROC6 r14i1p1f1 gn 20
MPI-ESM1-2-HR r1i1p1f1 gn 20
MPI-ESM1-2-LR r1i1p1f1 gn 20
MRI-ESM2-0 r3i1p1f1 gn 20
UKESM1-0-LL r1i1p1f2 gn 20
Name: variable_id, dtype: int64
Now the results are much more manageable and ready to be downloaded for use in your analysis.
Feedback¶
What do you think of this interface? We have found that it saves our students days of work, but are interested in critical feedback. Can you think of simpler interface? Are there other analysis tasks that are painful and time consuming that we could automate?