Logging

If you would like details about what intake-esgf is doing, look in the local cache directory (the default location is ${HOME}/.esgf/) for a esgf.log file. This is a full history of everything we have searched, downloaded, or accessed.

You can also look at just this session (since you instantiated the catalog) by calling session_log() and printing it. Consider the following search.

import intake_esgf
from intake_esgf import ESGFCatalog

with intake_esgf.conf.set(all_indices=True):
    cat = ESGFCatalog().search(
        source_id="IPSL-CM6A-LR",
        experiment_id="piControl",
        variable_id="areacella",
        variant_label="r1i1p1f1",
        frequency="fx",
    )
    ds = cat.to_dataset_dict(add_measures=False)
    print(cat.session_log())

2024-05-02 17:59:38 search begin source_id=['IPSL-CM6A-LR'], experiment_id=['piControl'], variable_id=['areacella'], variant_label=['r1i1p1f1'], frequency=['fx'], type=['Dataset'], project=['CMIP6'], latest=[True], retracted=[False]
2024-05-02 17:59:39 combine_time=0.01
2024-05-02 17:59:39 search end total_time=1.29
2024-05-02 17:59:39 file info begin
2024-05-02 17:59:41 file info end total_time=1.61
2024-05-02 17:59:41 begin move_data
2024-05-02 17:59:41 transfer_time=0.04 [s] at 0.79 [Mb s-1] https://esgf-node.ornl.gov/thredds/fileServer/css03_data/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/piControl/r1i1p1f1/fx/areacella/gr/v20200326/areacella_fx_IPSL-CM6A-LR_piControl_r1i1p1f1_gr.nc
2024-05-02 17:59:41 end move_data

In this case you will see how long each index took to return a response and if any failed as well as from where the file was downloaded if not already on your system. Initially we randomize download locations from all available, but as you use intake-esgf we will remember the hosts which provide you the fastest download times. You can see where your data has come from by:

cat.download_summary()

	transfer_time [s]	transfer_size [Mb]	rate [Mb s-1]
host
esgf-node.ornl.gov	2.121207	110.225730	51.963678
crd-esgf-drc.ec.gc.ca	1.517908	71.707932	47.241302
esgf-data.ucar.edu	5.804983	243.145565	41.885664

We use this database to prioritize download locations internally to get you data as fast as we can.