Your maintenance system indicates that the pump will fail in 18 days. It never says how reliable this prediction is. This single missing figure explains why nobody takes action. A prediction without a prediction interval is not a prediction. It is a bet.
An output no one can act on
I have spent the last ten years measuring uncertainty in real-world signals, and I always encounter the same problem: a model produces a clean figure, with no margin. This newsletter opens up the field that we designed with TrustalAI Predictive : the reliability of each individual prediction for predictive models, and not just for computer vision.
Look at where these models operate in a factory. An MES generates an alert. An ERP returns a demand volume. A predictive maintenance model returns a time-to-failure. Each output is a single value, and none of them indicates how much it could be wrong.
A planner therefore reads "failure in 18 days" without being able to size their decision. Should they order the part now? Wait a week? Stop the line? The figure seems precise, but it hides everything the decision needs. It also appears more certain than it is. A bias that forecasters call overprecision: an isolated figure suggests a much lower risk than the one actually carried by the model. The decimal lies for you.
Why a good average RMSE is not enough
The usual defense is a benchmark: "the model scores well on the test set". Fine. But a low average error on historical data says nothing about the prediction in front of you, on this specific machine, under these specific conditions. Three findings help make this concrete.
Aggregated metrics hide the failures that matter. An accuracy indicator is dominated by the largest subgroups: a model can therefore look excellent while systematically failing on a small, critical category. The clinical machine learning community has given a name to this phenomenon: hidden stratification (hidden stratification).
The test score also does not determine behavior in production. A study conducted by Google among others on underspecification showed that models with identical scores on a test set can behave very differently once deployed.
And the average error can remain stable while individual predictions degrade. Across 32 datasets from four industries, one study identified a stable average error even as the unpredictability of individual predictions increased.
Reread that sentence. Your dashboard remains green while the specific predictions you rely on progressively degrade.
Underlying all of this is a more fundamental problem. Modern neural networks are poorly calibrated by default: their raw confidence level is not a probability of being right. ReLU networks can even produce confidence levels close to certainty on inputs that look nothing like anything seen in training. Maximum confidence in a situation the model has never encountered: that is exactly what a bet disguised by a decimal looks like.
The breakthrough: a 95% interval for each prediction, with a drift signal
TrustalAI Predictive adds a calibrated 95% interval to each prediction at the time of inference, and distinguishes between two types of uncertainty. Aleatoric uncertainty (aleatoric uncertainty) corresponds to the irreducible noise present in the data. This is what the prediction interval aims to cover. Epistemic uncertainty, on the other hand, corresponds to the model's own ignorance. When it increases, the model has left the regime in which its interval remains valid.
This is your out-of-distribution (out-of-distribution, OOD) data signal, provided as a first-tier output rather than an afterthought. Industrial time series specifically defeat static validation because they drift over time.
The decision variable is the boundary of the interval, not the point estimate. Take the example from our introduction: "failure in 18 days ± 6" versus "failure in 18 days ± 40". Same announced date, opposite decisions. In the first case, you can plan with confidence. In the second, the model tells you it doesn't know: inspect before committing to parts and a production shutdown.
This is not a heuristic. In predictive maintenance research, the cost-optimal time to replace a part is near the lower bound of the confidence interval on the remaining useful life, not on the point estimate. The interval is the actionable figure. The point estimate never has been.
Three domains, the same missing figure
The same gap appears in three domains where we operate.
Predictive maintenance
Most predictive maintenance models use IIoT sensors: vibration, electrical current, and wear signals feed a deep learning model that returns a time-to-failure. On paper, they outperform preventive maintenance. In practice, unplanned downtime is still very expensive, costing up to $2.3 million per hour on automotive assembly lines.
The gap between a premature replacement and a breakdown in production is precisely where a calibrated interval creates value. A point estimate of RUL cannot quantify this asymmetry. Its confidence interval can. Add a confidence score to each alert and you distinguish a true alarm from a false alarm. That is the difference between preserving availability and wasting maintenance costs on parts that still had useful life. That is OEE you preserve instead of losing it to a false alert.
Power grid
Electric load forecasting is where the missing margin becomes visible to everyone. In the US, the five-year peak demand forecast jumped from 38 GW to 128 GW in two years, about a +456% increase, and is openly challenged by analysts working on the exact same data (Grid Strategies, National Load Growth Report, 2025). A single figure is useless when deciding whether or not to build new capacity. The interval is the whole story, because the reserve margin is what keeps the lights on during demand peaks.
Supply chain
Forecasting demand as a single figure has a global cost. Inventory distortions, including both out-of-stocks and overstocks, reached approximately $1.77 trillion in 2025. A well-known and dated case: Nike’s inventory surplus reached $9.7 billion, forcing markdowns that weighed on the bottom line (Modern Retail, 2023). This is what happens when a point forecast meets the volatility of the real world without a margin of uncertainty.
The proof
TrustalAI Predictive is a TRL 9 product, validated in real operational conditions. Its official figures are as follows: −81% errors, −84% false positives, 20 ms latency for the reliability layer added to predictive models.
These results must be distinguished from our VEDECOM PoC, which corresponds to another use case: multi-sensor fusion for autonomous driving. In that project, we reduced critical false positives by −83% and perception errors by −80%. The underlying method is the same, but the domain is different. I separate them intentionally, as mixing figures validated in different contexts is precisely how trust is eroded.
What this changes for a CTO
You stop treating every prediction the same way. Today, most pipelines treat all outputs as equal. With a calibrated interval for each prediction, you can sort by reliability: high-confidence forecasts are executed automatically, while uncertain forecasts are escalated to a human. This mechanism is known in the literature as selective prediction, or reject option (selective prediction / reject option) (arXiv, 2025). It becomes even more important as decisions become agentic: when a recommendation acts without a human in the loop, the interval becomes the last available filter.
It also addresses the reason why no one acts on alerts. In clinical monitoring, 85% to 99% of alarm signals require no intervention, and clinicians rationally learn to ignore them all (Joint Commission / AHRQ). Alarm fatigue is not a human problem. It is what happens when every alert seems just as or just as little certain. Rank alerts based on what the model actually knows, and the processing queue becomes manageable again.
Finally, when a predictive model feeds a high-risk decision under the EU AI Act, having a calibrated interval for each prediction provides the piece of evidence you can present: proof that the decision was made with a known confidence level, and not from an isolated figure. This is a capability you either have or do not have, long before a regulatory deadline turns it into a compliance requirement.
Reliability per prediction or aggregated monitoring
This is the distinction on which this entire approach is built. Aggregated monitoring observes the average behavior of a model over a period, after the fact. It answers the question: "is the model healthy this week?" It triggers when a distribution has already drifted, meaning after decisions have been made based on degraded predictions. It is a rearview mirror. And its failures are silent: nothing crashes, outputs keep coming, dashboards stay green while quality deteriorates.
Reliability per prediction answers another question, before the decision: "can I act on this specific forecast, now?" It associates each inference with a calibrated 95% interval and a drift signal, in real time, even on a black-box model. Aggregated monitoring tells you that the model drifted last Tuesday. Reliability per prediction tells you that this prediction is unreliable, this second, while you can still act. Both approaches have their place, but only one reaches the exact moment of decision.
Test it on your data
If you run predictive models in production, the fastest way to see this is to launch a two-week PoC on your existing production data. No model changes, no process changes. We add the reliability layer, and you visualize the intervals per prediction as well as the drift signals on your own forecasts. You decide next.
A forecast without an interval is not a prediction. It is a bet with a decimal point.
FAQ
What is a prediction interval, and why does a point forecast need one?
A prediction interval is the range within which a forecast is expected to fall for a given confidence level, such as 95%. A point forecast provides a single figure with no margin: the decision-maker therefore cannot know how wrong it might be. The interval transforms an isolated figure into information that allows a decision to be sized; this is what makes it actionable.
Why is a low average RMSE not enough to trust a forecast?
A low average error rate describes past performance across a test set. It says nothing about the individual forecast in front of you. A study across four industries showed that average error can remain stable while the reliability of individual predictions decreases (Vela et al., 2022). Aggregated metrics can also hide failures concentrated in small, critical subgroups.
How does reliability per prediction differ from model monitoring?
Aggregated monitoring looks at average behavior over a period, after the fact, and detects drift once decisions have already been made. Reliability per prediction associates each inference with a calibrated confidence interval and a drift signal, in real time, before the decision. One is a post-mortem view; the other is available at the very moment you act.
Does adding a reliability layer involve retraining our models?
No. TrustalAI Predictive is plug-and-play on a black-box model. The solution works alongside your existing pipeline, reads the context, and returns reliability metrics for each prediction, without retraining or modification of your core algorithm. At the edge, it adds about 20 ms of latency, making it compatible with real-time decisions.
Share
Related articles
