Prediction Methodology

How NYC Water Check estimates bacteria levels after rain

This page documents the current model, the historical backtests behind it, and the rainfall-source validation work used to decide how repaired local gauge-derived fields should drive the prediction pipeline.

The goal here is not marketing copy. This is a working methods note for a public-facing prediction system built on real historical shoreline bacteria and rainfall data.

Dataset snapshot

Samples with 7-day rain
13,810
Sites in model set
159
Best validation RMSE
0.7188
Validation threshold accuracy
71.2%

Overview

Current prediction logic

The production estimate starts with each site's own typical bacteria level, then adjusts that baseline using rainfall features learned from the historical record. Missing rain is excluded rather than treated as zero, and every lag follows calendar dates rather than the order in which samples were collected. Shared responses, regularized site-specific responses, seasonal patterns, and a simple site median compete on the same validation measurements. A more complex model needs a meaningful improvement before it replaces a simpler one.

Outcome being modeled

Enterococcus concentration on the log scale. The site's median historical log level acts as the baseline, and the rain model predicts the residual above or below that baseline.

Core rain windows

The model separates rain 1–2, 3–4, and 5–7 days before the estimate date. It excludes sample-day totals and today’s forecast rain. Today’s live rainfall is shown separately; it enters the model tomorrow. Where continuous daily rain is available, it also builds exact calendar-week totals for the preceding four to eight weeks so a gap in sampling can never masquerade as a one-week lag. Additional candidates test rain after a dry spell, rain after wet weeks, and dry weather after a wet period. These effects can increase or decrease an estimate; their direction is learned from the data.

Model family

Ridge regression with log-transformed rain features. This reduces instability from large storm days and keeps the coefficients from swinging too far on sparse patterns. Candidate models are compared on the same forward validation rows, then refit on all eligible history. Local coefficients are pulled toward the shared response to reduce overfitting.

Backtest

Model selection validation

Three non-overlapping forward validation blocks use one calendar cutoff across all sites and training-only site baselines. Every candidate is scored on identical rows with a complete nine-week rain history. Lower RMSE and MAE are better. These measurements guide model selection; they are not an independent accuracy estimate of the final selected system. The separate audit script tests that complete selection procedure on later annual holdouts, including missed high bacteria readings at the site’s 35 MPN cutoff.

CandidateFeaturesValidation rowsValidation RMSE (log)Validation MAE (log)Threshold accuracy
extended-memoryrecentRain, midRain, lateRain, priorWeeksRainMemory, dryWeeks, seasonSin, seasonCos2,2570.71880.541970.0%
rain-sequencerecentRain, midRain, lateRain, priorWeeksRainMemory, dryWeeks, seasonSin, seasonCos, daysSinceRain, firstFlush, rainAfterWet, dryAfterWet2,2570.71880.545170.0%
48hrecentRain2,2570.72290.544670.9%
seasonalseasonSin, seasonCos2,2570.72340.553868.1%
7d-distributedrecentRain, midRain, lateRain2,2570.72340.544971.2%
long-memoryrecentRain, midRain, lateRain, lagWeek1, lagWeek2, lagWeek3, lagWeek4, lagWeek5, lagWeek6, lagWeek7, lagWeek8, dryWeeks, seasonSin, seasonCos2,2570.72980.555070.4%
baselinesite baseline only2,2570.73260.540070.1%
Best numeric fit

extended-memory had the lowest held-out RMSE at 0.7188, before site-specific model selection.

Best threshold result

7d-distributed remained the best shared candidate for validation threshold accuracy at 71.2%.

Main modeling lesson

Extra rainfall features do not improve every site. A site can retain its historical median or seasonal pattern when a rain response does not validate well enough. Error scales are shown with site estimates; a point estimate alone is not a probability of low contamination.

Decision Metric

What happens if we optimize for the swim threshold directly?

We also tested simpler strategies that predict only the safe / unsafe threshold decision. These are useful comparisons, but they optimize a different outcome from the production regression, which is selected for numeric bacteria error and then translated into condition bands.

Older threshold-rule percentages used a different split and rain window and are not comparable with the validation above. The current audit compares the site median, shared seven-day and sequence models, and the complete site-selection procedure on identical annual test rows.

Rainfall Inputs

Which rainfall source appears more trustworthy?

We compared the stored site rain fields against the assigned NOAA daily station records and did the same for Open-Meteo historical forecast data. The validation window covers March 23, 2021 onward. DOHMH beaches, whose source dataset contains no rain, use a single consistent ERA5 reanalysis series backfilled at each beach location through Open-Meteo.

SourceCompared rowsCorrelation to NOAAMAERMSEWithin 0.10 in
Raw stored gauge-derived rain fields30,4690.5050.138 in0.418 in76.7%
Open-Meteo historical forecast30,4690.3210.167 in0.502 in75.0%

Result: the raw stored gauge-derived fields are materially closer to NOAA station observations overall, so the evidence does not support replacing the entire rainfall source with Open-Meteo. Across sites, the raw stored fields beat Open-Meteo at 69 sites versus 37 for Open-Meteo. The model metrics above use the repaired version of those stored fields and drop incomplete seven-day windows instead of silently converting missing values to dry days.

Outlier Review

Where the rain fields were repaired or still look suspect

The main risk is not that the entire stored rainfall source is bad. The current model now repairs the clearest bulk-filled or misaligned sample-date clusters before building rain features, while less certain cases stay on the review list.

September 2, 2021Repaired in model input

`precipitationDayOf = 0.10 in` and `precipitationPreviousDay = 7.13 in` were swapped before feature generation.

August 26, 2021Repaired in model input

`precipitationPreviousSat = 4.45 in` was removed from the prior-week rain window.

June 26, 2025Repaired in model input

`precipitationPreviousMon = 2.54 in` was removed after NOAA and Open-Meteo both showed a near-dry day.

July 27, 2023Still under review

Raritan sites shared `precipitationPreviousTue = 0.28 in`; this stays on the watchlist until it can be confirmed site by site.

Operational conclusion: keep the stored gauge-derived rainfall as the primary source, apply targeted repairs for confirmed sample-date + field problems, and leave ambiguous clusters out of the override set until they can be checked against station-level evidence.

Next Steps

What would make the prediction system stronger?

The current rainfall-and-season model is useful but not definitive. With the clearest rain outliers repaired, the most promising next gains are broader QA automation and richer predictors, not just more rain windows.

Expand rain QA coverage

Turn the manual outlier checks into a recurring station-level validation pass so future imports can be flagged before they affect predictions.

Train directly for the threshold

If the product goal is a safe / unsafe call, the next model should optimize that decision directly rather than optimizing only numeric bacteria error.

Add more physical context

Tide timing, recent prior bacteria readings, and station-level weather context are the likeliest variables to improve the model beyond the current rainfall-and-season model.