Automated reanalysis acquisition pipelines

Problem

Bringing in new reanalysis data was manual: hand-made requests and a legacy container of shell scripts. And the upstream feeds misbehave without raising errors: ERA5’s download service truncates silently at its availability edge, and MERRA-2 publishes on an irregular lag.

What I did

I replaced the manual steps with pipelines that run on a schedule with no one watching. Each run checks its own output with integrity gates, verifies what it publishes by checksum, writes a status record, and emails when it fails. The upstream quirks became design inputs: the ERA5 pipeline takes the last timestamp actually returned as the frontier, and the MERRA-2 pipeline reads what’s published from the archive listing instead of assuming a fixed lag. One command-line entry point with no default source, so a scheduled job never guesses which dataset to fetch.

Result

The last manual link in the reanalysis chain is gone. A job that dies at 2 a.m. is visible at 9 a.m., and partial upstream data stops at the gate instead of reaching the models.

Tools

Python · Copernicus CDS · NASA GES DISC · NetCDF · Zarr