Enhanced Intake-ESM Catalog Demo¶

Overview¶

This notebook compares one Intake-ESM catalog with an enhanced version that includes additional attributes. Both catalogs are an inventory of the NCAR Community Earth System Model (CESM) Large Ensemble (LENS) data hosted on AWS S3.

Prerequisites¶

Concepts	Importance	Notes
Intro to Pandas	Necessary

Time to learn: 10 minutes

Imports¶

import intake
import pandas as pd

Original Intake-ESM Catalog¶

At import time, the intake-esm plugin is available in intake’s registry as esm_datastore and can be accessed with intake.open_esm_datastore() function.

cat_url_orig = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.json'
coll_orig = intake.open_esm_datastore(cat_url_orig)
coll_orig

Here’s a summary representation:

print(coll_orig)

In an Intake-ESM catalog object, the esmcat class provides many useful attributes and functions. For example, we can get the collection’s description:

coll_orig.esmcat.description

We can also get the URL pointing to the catalog’s underlying tabular representation:

coll_orig.esmcat.catalog_file

That’s a CSV file ... let’s take a peek.

df_orig = pd.read_csv(coll_orig.esmcat.catalog_file)
df_orig

However, we can save a step since an ESM catalog object provides a df instance which returns a dataframe too:

df_orig = coll_orig.df
df_orig

Print out a sorted list of the unique values of selected columns

for col in ['component', 'frequency', 'experiment', 'variable']:
    unique_vals = coll_orig.unique()[col]
    unique_vals.sort()
    count = len(unique_vals)
    print (col + ': ' ,unique_vals, " count: ", count, '\n')

Finding Data¶

If you happen to know the meaning of the variable names, you can find what data are available for that variable. For example:

df = coll_orig.search(variable='FLNS').df
df

We can narrow the filter to specific frequency and experiment:

df = coll_orig.search(variable='FLNS', frequency='daily', experiment='RCP85').df
df

The problem: Do all potential users know that `FLNS` is a CESM-specific abbreviation for "net longwave flux at surface”? How would a novice user find out, other than by finding separate documentation, or by opening a Zarr store in the hopes that the long name might be recorded there? How do we address the fact that every climate model code seems to have a different, non-standard name for all the variables, thus making multi-source research needlessly difficult?

The solution:

Enhanced Intake-ESM Catalog!¶

By adding additional columns to the Intake-ESM catalog, we should be able to improve semantic interoperability and provide potentially useful information to the users. Let’s now open the enhanced collection description file:

Note: The URL for the enhanced catalog differs from the original only in that it has -enhanced appended to aws-cesm1-le

cat_url = 'https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le-enhanced.json'
coll = intake.open_esm_datastore(cat_url)
coll

As we did for the first catalog, let’s obtain the description and catalog_file attributes.

print(coll.esmcat.description) # Description of collection
print("Catalog file:", coll.esmcat.catalog_file)
print(coll) # Summary of collection structure

Long names¶

In the catalog’s representation above, note the addition of additional elements: long_name, start, end, and dim. Here are the first/last few lines of the enhanced catalog:

df_enh = coll.df
df_enh

Warning

The long_names are not CF Standard Names, but rather are those documented at the NCAR LENS website. For interoperability, the long_name column should be replaced by a cf_name column and possibly an attribute column to disambiguate if needed.

List all available variables by long name, sorted alphabetically:

nameList = coll.unique()['long_name']
nameList.sort()
print(*nameList, sep='\n')

Search capabilities¶

We can use an intake-esm catalog object’s search function in several ways:

Show all available data for a specific variable based on long name:

myName = 'Salinity'
df = coll.search(long_name=myName).df
df

Search based on multiple criteria:

df = coll.search(experiment=['20C','RCP85'], dim='3D', variable=['T','Q']).df
df

Substring matches¶

In some cases, you may not know the exact term to look for. For such cases, inkake-esm supports searching for substring matches. With use of wildcards and/or regular expressions, we can find all items with a particular substring in a given column. Let’s search for:

entries from experiment = ‘20C’
all entries whose variable long name contains wind

coll_subset = coll.search(experiment="20C", long_name="Wind*")

coll_subset.df

If we wanted to search for Wind and wind, we can take advantage of regular expression syntax to do so:

coll_subset = coll.search(experiment="20C" , long_name="[Ww]ind*")

coll_subset.df

Other attributes¶

Other columns in the enhanced catalog may be useful. For example, the dimensionality column enables us to list all data from the ocean component that is 3D.

df = coll.search(dim="3D",component="ocn").df
df

Spatiotemporal filtering¶

If there were both regional and global data available (e.g., LENS and NA-CORDEX data for the same variable, both listed in same catalog), some type of coverage indicator (or columns for bounding box edges) could be listed.

Temporal extent in LENS is conveyed by the experiment (HIST, 20C, etc) but this is imprecise and requires external documentation. We have added start/end columns to the catalog, but Intake-ESM currently does not have built-in functionality to filter based on time.

We can do a simple search that exactly matches a temporal value:

df = coll.search(dim="3D",component="ocn", end='2100-12').df
df

Summary¶

In this notebook, we used Intake-ESM to explore a catalog of CESM LENS data. We then worked through some helpful features of the enhanced catalog.

What’s next?¶

We will use this data to recreate some figures from a paper published in BAMS that describes the CESM LENS project: Kay et al. (2015)

References¶

de la Beaujardière, J., Banihirwe, A., Shi, C.-F., Paul, K., & Hamman, J. (2019). NCAR CESM LENS Cloud-Optimized Subset. UCAR/NCAR - Computational. 10.26024/WT24-5J82
Kay, J. E., Deser, C., Phillips, A., Mai, A., Hannay, C., Strand, G., Arblaster, J. M., Bates, S. C., Danabasoglu, G., Edwards, J., Holland, M., Kushner, P., Lamarque, J.-F., Lawrence, D., Lindsday, K., Middleton, A., Munoz, E., Neale, R., Oleson, K., … Vertenstein, M. (2015). The Community Earth System Model (CESM) Large Ensemble Project. Bull. Amer. Meteor. Soc. 10.1175/BAMS-D-13-00255.1

Preamble

How to Cite This Cookbook

Example workflows

Reproducing Key Figures from Kay et al. (2015)