Outcome being modeled
Enterococcus concentration on the log scale. The site's median historical log level acts as the baseline, and the rain model predicts the residual above or below that baseline.
Prediction Methodology
This page documents the current model, the historical backtests behind it, and the rainfall-source validation work used to decide how repaired local gauge-derived fields should drive the prediction pipeline.
The goal here is not marketing copy. This is a working methods note for a public-facing prediction system built on real historical shoreline bacteria and rainfall data.
Dataset snapshot
Overview
The production estimate starts with each site's own typical bacteria level, then adjusts that baseline using rainfall features learned from the historical record. Missing rain is excluded rather than treated as zero, and every lag follows calendar dates rather than the order in which samples were collected. Shared responses, regularized site-specific responses, seasonal patterns, and a simple site median compete on the same validation measurements. A more complex model needs a meaningful improvement before it replaces a simpler one.
Enterococcus concentration on the log scale. The site's median historical log level acts as the baseline, and the rain model predicts the residual above or below that baseline.
The model separates rain 1–2, 3–4, and 5–7 days before the estimate date. It excludes sample-day totals and today’s forecast rain. Today’s live rainfall is shown separately; it enters the model tomorrow. Where continuous daily rain is available, it also builds exact calendar-week totals for the preceding four to eight weeks so a gap in sampling can never masquerade as a one-week lag. Additional candidates test rain after a dry spell, rain after wet weeks, and dry weather after a wet period. These effects can increase or decrease an estimate; their direction is learned from the data.
Ridge regression with log-transformed rain features. This reduces instability from large storm days and keeps the coefficients from swinging too far on sparse patterns. Candidate models are compared on the same forward validation rows, then refit on all eligible history. Local coefficients are pulled toward the shared response to reduce overfitting.
Backtest
Three non-overlapping forward validation blocks use one calendar cutoff across all sites and training-only site baselines. Every candidate is scored on identical rows with a complete nine-week rain history. Lower RMSE and MAE are better. These measurements guide model selection; they are not an independent accuracy estimate of the final selected system. The separate audit script tests that complete selection procedure on later annual holdouts, including missed high bacteria readings at the site’s 35 MPN cutoff.
| Candidate | Features | Validation rows | Validation RMSE (log) | Validation MAE (log) | Threshold accuracy |
|---|---|---|---|---|---|
| extended-memory | recentRain, midRain, lateRain, priorWeeksRainMemory, dryWeeks, seasonSin, seasonCos | 2,257 | 0.7188 | 0.5419 | 70.0% |
| rain-sequence | recentRain, midRain, lateRain, priorWeeksRainMemory, dryWeeks, seasonSin, seasonCos, daysSinceRain, firstFlush, rainAfterWet, dryAfterWet | 2,257 | 0.7188 | 0.5451 | 70.0% |
| 48h | recentRain | 2,257 | 0.7229 | 0.5446 | 70.9% |
| seasonal | seasonSin, seasonCos | 2,257 | 0.7234 | 0.5538 | 68.1% |
| 7d-distributed | recentRain, midRain, lateRain | 2,257 | 0.7234 | 0.5449 | 71.2% |
| long-memory | recentRain, midRain, lateRain, lagWeek1, lagWeek2, lagWeek3, lagWeek4, lagWeek5, lagWeek6, lagWeek7, lagWeek8, dryWeeks, seasonSin, seasonCos | 2,257 | 0.7298 | 0.5550 | 70.4% |
| baseline | site baseline only | 2,257 | 0.7326 | 0.5400 | 70.1% |
extended-memory had the lowest held-out RMSE at 0.7188, before site-specific model selection.
7d-distributed remained the best shared candidate for validation threshold accuracy at 71.2%.
Extra rainfall features do not improve every site. A site can retain its historical median or seasonal pattern when a rain response does not validate well enough. Error scales are shown with site estimates; a point estimate alone is not a probability of low contamination.
Decision Metric
We also tested simpler strategies that predict only the safe / unsafe threshold decision. These are useful comparisons, but they optimize a different outcome from the production regression, which is selected for numeric bacteria error and then translated into condition bands.
Older threshold-rule percentages used a different split and rain window and are not comparable with the validation above. The current audit compares the site median, shared seven-day and sequence models, and the complete site-selection procedure on identical annual test rows.
Rainfall Inputs
We compared the stored site rain fields against the assigned NOAA daily station records and did the same for Open-Meteo historical forecast data. The validation window covers March 23, 2021 onward. DOHMH beaches, whose source dataset contains no rain, use a single consistent ERA5 reanalysis series backfilled at each beach location through Open-Meteo.
| Source | Compared rows | Correlation to NOAA | MAE | RMSE | Within 0.10 in |
|---|---|---|---|---|---|
| Raw stored gauge-derived rain fields | 30,469 | 0.505 | 0.138 in | 0.418 in | 76.7% |
| Open-Meteo historical forecast | 30,469 | 0.321 | 0.167 in | 0.502 in | 75.0% |
Result: the raw stored gauge-derived fields are materially closer to NOAA station observations overall, so the evidence does not support replacing the entire rainfall source with Open-Meteo. Across sites, the raw stored fields beat Open-Meteo at 69 sites versus 37 for Open-Meteo. The model metrics above use the repaired version of those stored fields and drop incomplete seven-day windows instead of silently converting missing values to dry days.
Outlier Review
The main risk is not that the entire stored rainfall source is bad. The current model now repairs the clearest bulk-filled or misaligned sample-date clusters before building rain features, while less certain cases stay on the review list.
`precipitationDayOf = 0.10 in` and `precipitationPreviousDay = 7.13 in` were swapped before feature generation.
`precipitationPreviousSat = 4.45 in` was removed from the prior-week rain window.
`precipitationPreviousMon = 2.54 in` was removed after NOAA and Open-Meteo both showed a near-dry day.
Raritan sites shared `precipitationPreviousTue = 0.28 in`; this stays on the watchlist until it can be confirmed site by site.
Operational conclusion: keep the stored gauge-derived rainfall as the primary source, apply targeted repairs for confirmed sample-date + field problems, and leave ambiguous clusters out of the override set until they can be checked against station-level evidence.
Next Steps
The current rainfall-and-season model is useful but not definitive. With the clearest rain outliers repaired, the most promising next gains are broader QA automation and richer predictors, not just more rain windows.
Turn the manual outlier checks into a recurring station-level validation pass so future imports can be flagged before they affect predictions.
If the product goal is a safe / unsafe call, the next model should optimize that decision directly rather than optimizing only numeric bacteria error.
Tide timing, recent prior bacteria readings, and station-level weather context are the likeliest variables to improve the model beyond the current rainfall-and-season model.