A forecast can help a buyer think about the future. It can also give an unjustified impression of precision. The engineering task is to communicate both the estimate and the limits of the evidence behind it.
A central price estimate, a direction probability and a prediction interval answer different questions. They need different evaluations. Improvement in one does not demonstrate improvement in the others.
Start with the comparator
A forecast should be compared with a named alternative that could have been used at the time. Persistence and a trailing average are different baselines; a percentage improvement against one cannot be presented as an improvement against the other.
The evaluation also needs an explicit horizon, target population and date window. Market-wide averages can hide weak performance for individual species or grades. Pooled error and species-balanced error provide different views and should be labelled accordingly.
Treat a range as a testable claim
A nominal 80% interval expresses an intended coverage level. Observed coverage asks how often the realised outcome actually fell inside it on a defined sample. Width matters too: a very wide interval may cover more outcomes while giving little practical guidance.
No method name replaces that check. Historical coverage can change when market conditions, data quality or the forecast population change. A range needs evaluation at the level and horizon where a buyer will use it.
What the FishFacts audit found
A published Peterhead diagnostic examined 60,361 resolved outcomes from 7 January to 25 September 2026. Nominal 80% intervals covered 63.14% of those outcomes; coverage balanced by date was 62.85%.
The ranges were too narrow on that sample. A retrospective candidate brought coverage closer to its target, but the audit contained no prospective results for that candidate. That distinction matters: correcting a historical replay is a development result, not proof of performance on new observations.
Read the Peterhead diagnostic and its scope.
Historical and prospective evidence
A historical evaluation can establish whether a specified model improves on a specified baseline under a defined replay. Development data, frozen confirmation data and a prospective forecast ledger have different evidential roles.
Once results have been inspected, changing a model and testing it on the same outcomes does not create a fresh independent confirmation. The development record should preserve what changed and which data informed the change.
From modelling to an operating system
Useful forecasting requires more than a model. Inputs need provenance and freshness checks. Issued forecasts need to retain their identity. Failures need to become visible. Resolved outcomes need to be compared with what was actually issued.
Those are data-platform and observability problems as well as statistical ones. They are central to how we build.
Explore the FishFacts engineering note or inspect the market-by-market evidence.