Migrate the bugs first

We had an older R implementation for extreme-wind statistics: typhoon wind fields built from historical storm tracks, and return-period wind speeds from them. It worked, people relied on its numbers, and it had known bugs.

We were moving it into a new Python toolkit, and the obvious plan was to fix the bugs on the way. We didn’t, at least not at first. The first version of the new code reproduced the old output, bugs included.

The problem with fixing as you go

Picture the obvious approach. You port the code, correct the physics where the old version was wrong, run both on the same storm, and the numbers differ.

Now what? Some of the difference is your fixes. Some might be a new bug you introduced in the port: an index off by one, a unit converted twice, a sign flipped. From the output alone, you can’t tell which is which. Every difference looks the same, and “the new numbers are probably better” is not a test.

With scientific code this is worse than usual, because there’s rarely an exact right answer to compare against. The old output is the reference. If you change the reference and the implementation at the same time, you’ve lost your only fixed point.

Two pipelines, on purpose

So the toolkit has two pipelines side by side.

The legacy pipeline reproduces the old implementation’s behaviour exactly, including the bugs we knew about. Its only job is to match the old output, number for number. When it does, that proves the port is faithful: the data handling, the units, the indexing and the plumbing are all right.

The corrected pipeline shares that verified plumbing and changes only the physics we meant to change. Now every difference between the two outputs has a known cause, because we put it there.

The order of work became:

1. Build the legacy pipeline and make it match the R output line by line.
2. Only then switch on the corrected physics.
3. Check that each difference in output traces back to a fix we made, and to nothing else.

Step 1 is slow and slightly absurd. You spend real effort carefully reproducing behaviour you know is wrong. But it turns step 3 from guesswork into a check you can actually run.

What it bought us

• Confidence in the port. When the legacy pipeline matched, we knew the new code read storms, built the wind field and computed return periods the same way the old code did.
• Explainable differences. Every change in the final numbers traces back to a named correction, which is what you need when someone asks why this year’s design wind speed differs from last year’s.

The general lesson

When you replace a system that people trust, separate two questions that are easy to blur:

1. Does the new code do what the old code did?
2. Should it do something different?

Answer the first one completely before you touch the second. Reproducing the bugs is how you answer it, and it’s cheaper than it sounds compared with shipping a “fix” nobody can explain.