ERA5 doesn't tell you when it stops

If you automate ERA5 downloads, sooner or later you’ll ask for data that doesn’t exist yet. You might expect an error, or at least a block of fill values. The Copernicus Climate Data Store gives you neither.

What actually happens

ERA5 reaches the present with a lag of a few days. When I first wired up an unattended acquisition job, I asked for 288 hours that ran past that edge. The request succeeded. The file came back well-formed. It held 78 hours.

No error, no warning, no missing-value padding for the other 210. The archive simply returned what it had and stopped. Nothing in the response says “this is less than you asked for.”

For a person downloading one file by hand, that’s harmless: you look at it and see where it ends. For a scheduled job writing into a store that models read from, it’s a trap. Code that assumes “I asked for this range, so I have this range” will happily publish a series with a hole at the end, and nothing downstream will notice until a forecast looks odd.

There is no frontier to look up

My first instinct was to find out where the edge is before asking, and only request up to it. That doesn’t work, because nothing tells you where the edge is. The availability lag isn’t a fixed number of hours, and the service doesn’t expose it.

So I turned the problem around. The pipeline asks for everything up to the current month, and the last timestamp that actually comes back is the frontier. Instead of predicting where the data ends, it measures it.

The pipeline then compares what it asked for with what it got, and treats the difference as information rather than as an error. Hours missing before the frontier mean something went wrong, so the run stops and says so. Hours after the frontier are just not published yet: the run records them and the next scheduled run picks them up.

That split is the whole trick.

Not all of the newest data is final

There’s a second subtlety at the edge. The most recent ERA5 data is a preliminary product, ERA5T, which ECMWF releases quickly and may revise later. The two are told apart by expver, a coordinate that varies per timestamp: 0005 marks preliminary ERA5T data and 0001 marks the final product.

A downloaded file can contain both. If you flatten expver away without looking, you lose the fact that the last few days may change. My pipeline records, alongside each run, where the provisional data starts and how many hours it covers, so anyone using the series knows which part is still preliminary, and a later run can replace it.

Its sibling has the opposite problem

MERRA-2, NASA’s reanalysis, fails differently. Its files are published on a lag that isn’t fixed either, and not always in order. On one day I checked, March to July were complete, but August held a single day’s file. A pipeline that asks for “1 January to today” would fail every run.

There, the fix was the same idea in a different form: read what is actually published from the archive’s directory listing, and treat that as the frontier, instead of assuming a lag.

The general lesson

Both feeds are excellent data. Neither tells you, in a way a machine notices, that you got less than you asked for. The pipelines I trust are the ones that:

• measure the edge of the data instead of assuming it,
• treat a gap inside the range and a gap at the end as different things,
• write down what they found, in a status record a person can read the next morning,
• and fail loudly when something is actually wrong, instead of passing a quiet hole downstream.

Silent truncation isn’t a bug you can file and wait out. It’s how the service behaves, so it belongs in the design.