A variable with a correlation of 0.82 with your target should improve your model. That is the reasonable prior. We tested it, and it made the model worse. The explanation is not a statistical artefact — it is a design distinction that, once made, changes how you think about feature engineering in retrieval systems.
Background: retrieval-based forecasting
The methodology underlying 42Analytics products works by finding historical market states that are geometrically similar to the current state, then transporting their subsequent trajectories through the current price distribution. The quality of the forecast depends on two things: the quality of the retrieved analogues, and the quality of the transport.
The geometry — the high-dimensional space in which "similarity" is defined — is built from the features that encode market state. Choosing those features well is the central engineering problem.
In conventional machine learning, feature selection is largely about correlation with the target variable: features that co-move with what you are trying to predict are useful. In retrieval-based forecasting, the question is more specific, and the answer can be different.
The experiment
We were evaluating the beef market as a potential application for the methodology. Early results with supply-side state variables (slaughter volumes, price series, grain costs) showed modest but real improvement over a naive baseline: +3.25pp mean absolute error improvement at H4–H12.
UK farmgate milk prices had a measured correlation of ρ = +0.82 with steer prices at a lag of six months — one of the strongest relationships we had found in any market. The causal mechanism is established agricultural economics: when milk prices are high, dairy herds are retained for milk production; when milk prices fall, surplus dairy cows are culled and enter the beef supply chain, pressuring beef prices six months later.
The correlation is real. The mechanism is real. We ran four ablation studies adding milk to the geometry in various forms — as a level, as a six-month return, with and without exchange rate signals — and every version degraded retrieval performance compared to the baseline without milk.
| Experiment | Additions vs baseline | Result | |---|---|---| | v1 (baseline) | Supply costs only | +3.25pp | | v2 | + EUR/GBP FX | +2.80pp | | v3 | + EUR/GBP + milk level | +2.71pp | | v4 | + milk level | +2.47pp | | v5 | + milk 6m return | +2.30pp |
The degradation is consistent and monotonic. Adding a variable with ρ = 0.82 made the model worse by 0.78pp.
The explanation: where the variable is in time
The distinction that resolves this is between two categories of variables:
State variables describe where the market is now. They are observable at the anchor date and capture the current configuration of supply, demand, and market structure. Slaughter volumes, current price levels, recent spreads — these describe the state the market is in.
Leading indicators describe where the market is going. They are observable now but their information content is about the future, not the present. Milk prices at time t do not encode the current beef market state — they encode a prediction about where beef prices will be at time t+6.
This distinction is critical for retrieval. When the geometry encodes milk prices, you retrieve historical market dates where milk prices were similar to today's. But those historical dates had similar future beef trajectories only if the lag structure held consistently, if the culling mechanism operated at the same tempo, and if the intervening six months were not disrupted by other regime factors. The geometry has no mechanism to ensure this. You are matching on a leading indicator, which means you are effectively asking the analogue pool to do your forecasting for you — and getting confused analogues as a result.
State variables encode the current condition. Analogues that share a similar current condition tend to have similar subsequent trajectories. Leading indicators encode future conditions. Analogues that share a similar leading indicator may have had very different current conditions, and very different subsequent paths.
What this implies architecturally
The correct architecture for markets with strong leading indicator structure is two-stage:
-
Retrieve on state: use the supply-side variables, current prices, and volume signals to find historical market states that are structurally similar to today. These analogues give you the distributional prior — a probabilistic forecast grounded in genuine historical similarity.
-
Correct with leading indicators: apply a post-retrieval adjustment based on the current value of leading indicators. If milk prices are high, shift the distribution in the direction the lag relationship predicts. This keeps the leading indicator's signal without corrupting the retrieval geometry.
We tested the correction step: residual ~ β × MilkUK_price(t−6) on the v1 walk-forward output. The coefficient was statistically significant (p = 0.013) but negative — the direction was wrong. On the test set, the correction made things worse.
The result is not a refutation of the two-stage architecture. It is a data sparsity problem. With 95 monthly anchors, it is not possible to reliably separate the milk signal from regime variation. The lag relationship changes sign across periods — high milk prices in 2020 predicted different beef dynamics than high milk prices in 2023, because the dairy sector responded differently. A correction layer requires enough anchors to estimate a reliable coefficient, and monthly livestock data does not provide them. The architecture is correct; the dataset is too small to execute it.
The broader principle
This experiment produced a rule we now apply consistently across all new datasets:
State variables belong in the geometry. Leading indicators belong in a post-retrieval correction layer.
To classify any candidate feature:
- Is it observable at the anchor date? If not, it cannot be a state variable.
- Does it describe the current condition of the market, or a future condition? If it primarily encodes future dynamics through a lag relationship, it is a leading indicator.
- Does removing it leave a gap in the description of current market structure? If yes, it is probably a state variable. If no, it is probably a leading indicator.
Variables that are ambiguous — exchange rates, for example, which partially encode current import competition and partially encode future trade dynamics — require empirical testing. We found that EUR/GBP degraded retrieval in the beef market (likely because it added dimensions that diluted the tighter supply-state analogues). FX signals may be useful in correction layers; the evidence suggests they are not useful in the retrieval geometry for this market.
Where this applies
The state/leading indicator distinction applies to any retrieval-based system where the geometry encodes multiple variables jointly. It is less relevant to standard regression models, where each variable contributes independently and leading indicators can be included directly as predictors.
For retrieval systems, the geometry is holistic: every variable contributes to the definition of "similarity," and a variable that is superficially correlated with the target can still degrade retrieval by diluting the dimensions that actually discriminate between good and poor analogues. High correlation with the target is necessary but not sufficient. The variable also needs to encode the right thing — the present state, not a prediction about the future.
The geometry must describe where the market is. Everything else belongs downstream.