Your maintenance tool predicts the failure of pump P-204 in 18 days. It does not state how reliable this number is. Without prediction intervals, no one knows whether to shut down the line or keep it running.
This number arrives alone. "18 days ± 6 days" and "18 days ± 40 days" display the same date and dictate two opposite decisions: in one case you plan the shutdown, in the other you instrument and re-evaluate. A forecast without its interval is not a prediction. It is a bet with a decimal point.
A point output cannot be arbitrated
A MES displays a batch end date. An ERP displays a volume to be produced. A predictive maintenance model displays a time to failure. All three output a clean number, directly consumable by the downstream system. None of the three displays its uncertainty.
This is what makes them unusable when making a decision. An industrial decision is made based on a gap: between what the model predicts and what it could predict if it were wrong. Without this gap, the forecast cannot be arbitrated. The maintenance manager can neither follow nor reject it with an argument. They follow it out of habit, or ignore it out of prudence.
The sensors are not to blame. A network of IIoT (industrial IoT sensors) feeding vibration, temperature, and production data gives the model abundant material. The model outputs a single point, and what it knows about its own uncertainty stops at the input of the inference. A good aggregate accuracy (RMSE, MAPE) actually masks the forecasts on which it is most heavily mistaken.
An average error says nothing about the current forecast
The RMSE is a batch statistic, not a property of the current forecast
The RMSE is to the model what the MTBF is to the fleet: a retrospective average that says nothing about the machine in front of you. Classic maintenance indicators (MTBF, MTTR, availability, OEE) describe an overall behavior over a past period. No one uses them to decide whether to dismantle a bearing this Tuesday.
The RMSE works the same way. Calculated after the fact, on a test set, over hundreds of series. It does not transfer to the forecast displayed on the screen. A model with an excellent RMSE still produces very poor local forecasts, and it produces them with the same typographical confidence as the good ones. This is the fundamental limitation of any global accuracy metric.
What the M4 and M5 competitions showed
The academic forecasting community settled this debate several years ago. The M competitions rely on aggregated error metrics, sMAPE and MASE, measured over tens of thousands of series.
However, the M4 had asked for something else: a 95% interval in addition to the point. Only 20 submissions provided one, even though 25% of the prize was dedicated to it. Most methods produced intervals that were too narrow. The organizers had warned that 95% intervals would vastly underestimate reality, and that the gap would worsen with the forecasting horizon. They were right.
The M5 drew the consequence of this, with a dedicated "Uncertainty" track: 42,840 sales series, nine requested quantiles from 0.005 to 0.995. The quality criterion is no longer the average error but the calibration, i.e., the gap between the observed coverage and the announced nominal level (Makridakis et al., 2022).
The community organizing the forecasting competitions has therefore stopped settling for the single point. The industry, on the other hand, has stuck to the single number.
Confidence interval or prediction interval? The distinction that changes everything
The two terms circulate as synonyms. The confusion is costly.
A confidence interval is a range that relates to a population parameter, typically the average of the observed phenomenon. A prediction interval is a range that relates to a future observation not yet realized, the one on which the decision will be made. The second is always wider than the first: it adds the uncertainty on the estimated parameter and the inherent variability of the upcoming observation.
Rob Hyndman, editor of the International Journal of Forecasting, documented a case where a confidence interval had been used as a prediction interval. Instead of covering 95% of the probability space of the future observation, it covered about 20% (Hyndman, 2013). The displayed figure said 95. The actual coverage was 20. Nothing in the output allowed anyone to notice it.
TrustalAI Predictive documents its output as a 95% confidence interval (95% CI). This is the official notation of the product, and the one used by industrial data science teams. What matters here is the bounded quantity: the upcoming forecast—failure, demand peak, or volume to produce—and not an aggregated parameter of the training set. The boundary is recalculated for each forecast, at the moment it is produced.
An interval per forecast, plus drift and out-of-domain
18 days ± 6 days, or 18 days ± 40 days
Let's go back to pump P-204. The model predicts a failure in 18 days, and TrustalAI Predictive attaches its 95% interval to this forecast.
First case, 18 days ± 6 days. The window fits within a production slot. We plan the shutdown, order the part, and notify operations.
Second case, 18 days ± 40 days. The displayed date is identical. The lower boundary falls in a week, the upper boundary in two months. We do not plan anything. We tighten the sensor sampling, add a vibration measurement, and re-evaluate in ten days.
Same date, two opposite decisions. What has changed is what we know about the model's reliability in this specific case.
Why the interval alone is not enough
Attaching a calibrated interval to each prediction is a well-equipped research field. Conformal prediction provides coverage guarantees without assumptions on the data distribution and valid for finite samples: for an error level α, the true value falls within the interval with a probability of at least 1−α, regardless of the underlying model (Kaiser & Herzog, 2025). The French ecosystem contributes to this, with the open-source library MAPIE, born from a collaboration between Capgemini, Quantmetry, Michelin, and ENS Paris-Saclay under the Confiance.ai program.
This guarantee has one condition: data exchangeability. Industrial time series violate this by design, between strong temporal dependencies and distribution drift over the course of operations (IEEE, 2025). An interval set once and never monitored again ends up lying too, with the same authority as the point it was supposed to correct.
Hence the architecture of TrustalAI Predictive, which combines three signals instead of one:
the 95% confidence interval attached to each forecast;
model drift detection, which monitors the gradual disconnect of the model from the actual process;
out-of-distribution detection, which flags situations that fall outside the validity domain on which the model was built.
Drift is the most dangerous of the three because it is silent. No error, no alert: the model continues to predict with the same confidence on a process that has changed. Teams discover it when operations complain, and by that time the model has been wrong for weeks. The same slow-wear mechanism applies to PLM, MES, and ERP chains whose embedded predictions drive multi-million euro decisions.
The three domains: downstream already thinks in ranges
A common thread connects the three major uses of predictive models in industry, and it is rarely pointed out. Downstream decision-making already thinks in ranges. It is upstream that sends a single point.
Domain | What decision-making already uses | What the model sends | Range quantity |
|---|---|---|---|
Predictive maintenance | A downtime window | A time to failure | Bounded RUL, measured by PICP |
Energy grid | A sizing margin in MW | A peak demand | Load quantiles |
Supply chain | A safety stock in units | A demand volume | Quantile at service level |
Predictive maintenance: RUL and its interval
Remaining useful life (RUL) is the core focus of predictive maintenance. Recent research produces it with bounds. Recent work published in Applied Sciences evaluates predictive uncertainty on NASA C-MAPSS datasets and reports that conformal recalibration improves the coverage of prediction intervals.
The reference metric is no longer RMSE alone but PICP (Prediction Interval Coverage Probability), which is the proportion of actual values effectively contained within the predicted interval. The literature is explicit on the starting point: a point estimate without an interval is misleading in real operations, where noise and variability of conditions dominate (ScienceDirect, 2023).
The "18 days ± 6 days" on the ground corresponds literally to what research produces. MRO teams that arbitrate between calendar-based preventive maintenance and condition-based intervention already work with windows. They receive dates.
Energy grid: peak demand at ± MW
The electricity sector shifted about ten years ago. The GEFCom competitions, organized by the IEEE Working Group on Energy Forecasting, have a track dedicated to probabilistic load forecasting: they no longer evaluate the point alone, but the predictive intervals and quantiles at fixed nominal levels (Hong et al., 2016).
The reason is economic, and RTE states it in its Forecast Balance. The adopted approach is deliberately cautious because under-sizing the system is far more expensive than over-sizing it. The grid operator sizes for contingencies, not for a central scenario. A critical infrastructure decision made based on a distribution, then. And the AI model powering it often outputs a single number.
Supply chain: safety stock is already an interval
A supply chain planner works with a distribution every day, often without framing it that way. Safety stock is a prediction interval expressed in units: its size depends on the degree of demand uncertainty and forecast error, and it is calculated from the quantiles of error over the replenishment lead time, at the targeted service level.
Quantile forecasting is therefore necessary to calculate safety stock, yet it remains widely neglected in practice. The planner manually reconstructs, using assumptions inherited from the 1960s, the range that their demand model does not provide. The same logic applies to financial markets, where an order is not executed the same way depending on the confidence attached to the price forecast: this is the focus of reliability through prediction applied to trading.
Reliability by prediction or aggregate monitoring: a difference of timing
Monitoring tools detect drift after the fact, on aggregated history. By the time the alert is triggered, the decision has already been made and the incident has already incurred a cost. Between aggregate monitoring and per-prediction reliability, the difference lies less in what is measured and more in when it is measured.
Aggregate monitoring / post-mortem | Per-prediction reliability | |
|---|---|---|
When | After execution, over a historical window | Before the decision, at the moment of inference |
On what | A population of predictions | Each forecast, one by one |
What we learn | The model drifted last month | This specific forecast is reliable, or it is not |
Possible decision | Retrain, correct after the fact | Execute, escalate, or suspend now |
Latency | Hours to weeks | 20 ms |
Impact on the model | Often retraining | None: external layer, black-box |
Monitoring remains useful. It manages the model's lifecycle, supports traceability, and triggers retraining. It simply cannot qualify the forecast sent to the MES in the twenty milliseconds that follow. The reliability layer sits in parallel with the existing model, as plug-and-play, without accessing the code or weights, hence the black-box designation.
It delivers a confidence score in real-time. This is the logic already applied to vision models by TrustalAI Vision, transposed to time series.
What this changes for a CTO
The operational benefit boils down to one thing: being able to handle predictions differently depending on their reliability.
Narrow interval: automatic execution, the forecast goes directly into the MES, ERP, or production schedule.
Wide interval: human review, we apply more instrumentation, cross-reference with other sensors, and delay the arbitration.
Out-of-distribution or drift detected: automation suspension, the case returns to human supervision, and the model is flagged to the data team.
Beyond a dozen alarms per hour, the operator is considered overloaded, and their response capacity drops sharply (Chemical Engineering). An alert that arrives without a confidence level cannot be prioritized: it enters the queue at the same rank as the others. This is how false alerts end up causing operators to miss real ones.
Sorting by reliability reduces the volume of alarms processed with high priority without losing any signals. Nothing is discarded; what is uncertain is routed to a different queue. And the cost of non-reliability becomes quantifiable: engineering hours spent verifying anomalies that are not real, one more unplanned downtime, forced downtime despite the model, and maintenance costs incurred for nothing.
What the regulations require
The EU AI Act (Regulation EU 2024/1689) stipulates in Article 15 that high-risk AI systems must achieve an appropriate level of accuracy and robustness, and that "the levels of accuracy and the relevant accuracy metrics" must be declared in the instructions for use. An average RMSE measured on a historical test set is a weak declaration. A reliability metric produced forecast by forecast, timestamped and logged, is a documented and traceable declaration. TrustalAI contributes to this demonstration of compliance, but does not guarantee it on its own. This topic is covered in detail in our EU AI Act White Paper.
What TrustalAI Predictive brings
TrustalAI Predictive is the reliability layer of TrustalAI for predictive models, time series, and scoring. It attaches a 95% confidence interval to each forecast, detects model drift, and flags situations outside the validity domain. This includes embedded systems, where leaving the design domain is a safety issue in its own right, as in ADAS.
Official product metrics, measured in real operational environments (TRL9): −81% errors, −84% false positives, 20 ms latency.
For a technical team, three properties matter as much as these figures. No retraining: the layer connects in parallel with the existing model on its output stream. No access to the model: black-box compatible, without exposing intellectual property. No process changes: the inference pipeline remains in place, whether the model relies on deep learning or classic statistical methods. This is what makes the layer applicable as-is to predictions embedded in PLM, MES, and supply chain systems, where an unqualified forecast propagates to all downstream planning.
Putting it into practice
The test is simple to set up, precisely because it does not touch the model. We replay your historical data through TrustalAI Predictive, separate the forecasts that would have been executed automatically from those that would have been escalated, and then look at how many past incidents fell into the second category.
Test the compatibility of TrustalAI Predictive with your existing predictive models: request a 2-week PoC on your historical data, without retraining and without access to your model.
FAQ
What is the difference between a confidence interval and a prediction interval?
A confidence interval bounds a population parameter, most often an average. A prediction interval bounds a future observation not yet realized, the one on which the decision is made. The second is always wider, as it combines the uncertainty of the estimated parameter and the inherent variability of the upcoming observation. Confusing the two gives a false sense of accuracy.
Why is RMSE not enough to evaluate a predictive model in production?
RMSE is a batch statistic, calculated after the fact on a test set and across multiple series. It describes average performance, never the forecast displayed at any given moment. A model with an excellent RMSE can still produce very poor local forecasts, without any distinguishing signal. Deciding based on RMSE is like judging a machine based on the MTBF of its fleet.
How do you know if an AI forecast is reliable at the moment it is produced?
You need a metric calculated per forecast, and not over an aggregated history. A 95% confidence interval attached to the output indicates the plausible range of error in that specific case. It should be completed by model drift detection and out-of-distribution detection, which signal the gradual decline of the model and situations that fall outside its validity domain.
Do you need to retrain your model to get a prediction interval?
No. TrustalAI Predictive works as an external layer: it plugs in parallel with the existing model, plug-and-play, without retraining, without additional annotation, and without accessing the code or weights. The model remains unchanged, as does the inference pipeline. The confidence interval and drift signals are added to the output in real-time, with a latency of 20 ms.
What is PICP in predictive maintenance?
PICP, or Prediction Interval Coverage Probability, measures the proportion of actual values effectively contained within the predicted interval. A 95% interval with an observed PICP of 70% is poorly calibrated: it promises coverage that it does not deliver. This is the metric that allows verifying whether a remaining useful life range is honest, beyond just the average error.
Share
Related articles
