Reanalysis data library

Problem

Analysts and models needed ERA5 and MERRA-2 time series for any location, but the data sat in three places (a cloud data warehouse, cloud object storage and on-premise NetCDF archives), each with its own access pattern and naming.

What I did

I wrote and maintain the team’s reanalysis library: a Python package, command-line tool and packaged Windows app behind one interface, with interchangeable backends (Zarr on cloud storage with regional mirrors, BigQuery, and on-premise NetCDF). I added a completeness rule: a request that would return a series with missing timestamps fails loudly instead of handing a silent gap to the models downstream. I also designed a canonical variable registry that extended the library from wind to temperature and surface pressure with no breaking change for existing users.

Result

One interface for everyone, from analysts’ notebooks to non-engineers using the desktop app; gaps surface where they start instead of propagating into model inputs. Over a dozen releases so far, as sole maintainer.

Tools

Python · xarray · Zarr · NetCDF · BigQuery · Google Cloud Storage