A multi-sensor fusion most often processes its sources with equal weight. Including the one that, at this exact moment, sees nothing. Weighting each detection by its uncertainty changes the result, and it is measurable.
Your sensors are never wrong at the same time. An average, however, is.
A robotaxi enters a cloud of smoke
The facts are public and documented by a regulator. On June 20, 2026, an unoccupied robotaxi drives into thick smoke that was masking an active, unmarked fire scene. It brakes hard, attempts to pull over, and comes to a stop. No injuries.
The system knew how to recognize conventional emergency scene signals: cones, flashing lights, closed lanes. It had no internal representation of the following concept: "my camera is looking through a wall of particles, and what it returns to me is no longer worth much". The failure is not about obstacle detection, but about the awareness of the reliability of the source.
The fix adds a rule: detect smoke. There remain all the other degraded conditions that no one will ever enumerate.
What blind fusion actually does
Multi-sensor fusion (sensor fusion) is the operation that combines detections from multiple sources into a single representation of the scene. This is where the decision is made to believe what we see or not. In most deployed stacks, it takes one of the five forms below. None of them asks the source if it is in good shape.
The average: a vote where everyone has a voice
Including the one that saw nothing. If three out of four sources place a pedestrian at 6 m and the fourth, blinded, places them at 9 m, the fused output is only slightly wrong. That is precisely the problem: it looks clean, and no one raises their hand.
The confidence score: the source most sure of itself, not the most accurate
This is the most widespread trick, and it relies on a misunderstanding. A detector's score relates to the class, not the position. A high score says that there is probably something there; not that this thing is there. The model calibration literature adds a second problem: pre-trained detectors are over-confident, and NMS accentuates this bias on the surviving scores. Weighting by the score therefore amounts to listening to the most assertive source.
Inter-source NMS: eliminating instead of fusing
It therefore fuses nothing. It designates a winner by score and discards the rest. If the winner is the degraded source, which is often the noisiest, the system has just chosen exactly the wrong one.
Averaging or weighting: the difference is arithmetic, not philosophical
This is the turning point. The five approaches above share an assumption: the weight of a source is decided before looking at the scene. Weighting by uncertainty per detection means deciding this weight during the scene, detection by detection.
Fusion scheme | What it assumes | What breaks in real-world conditions |
|---|---|---|
Simple average | All sources are equal | The faulty source pulls the output toward error, with no signal |
Weighting by confidence score | The score reflects localization quality | The score relates to the class, and detectors are over-confident |
Inter-source NMS | The best score is the best measurement | Redundant information is eliminated instead of combined |
Majority vote | Sources are independent | Two correlated sources count the same evidence twice |
Static R matrix per sensor | Sensor noise is constant | Noise depends on the scene, distance, weather, time of day |
Weighting by uncertainty per detection | Each detection carries its own expected error | This is the assumption that we can measure and verify |
Uncertainty per detection is an estimate of the expected error produced for each detection, and not for the sensor that produced it. The same camera can be excellent on the object in the center of the field and very poor on the one coming out of backlighting, in the same image. A fusion that cannot make this distinction treats both detections equally.
The principle has already been measured elsewhere. A German industrial and academic team used the uncertainties of their detectors as weights in a multimodal fusion: average accuracy gains up to 7%, and up to 20% when the sensors are temporally misaligned (Feng et al., IEEE IV 2020). The gain grows as conditions degrade, which is exactly what the thesis predicts.
"It's a Kalman filter, that has existed since 1960"
This is the most common objection, and it is legitimate. Let's address it in three steps.
Yes, Kalman already weights by uncertainty
A measurement noise covariance matrix is the statistical description of the expected error of a source. The mechanism is documented in black and white by an automotive consortium: "one of the "side-results" of a Kalman filter is a covariance matrix. This covariance matrix is used as the core of accuracy information provided" (CAR 2 CAR Communication Consortium, 2021). The question is therefore not if we weight. It is with which R.
No, this R does not describe the scene
In this same document, R is constructed from standard deviations "as configured for the simulated sensors", then updated by a simple geometric rotation based on the relative position of the object. It is therefore fixed at configuration time, from laboratory characterization. Nothing in this loop says: this particular detection, in this particular scene, is less reliable than usual.
This is not an implementation flaw. An entire branch of research, adaptive Kalman filtering, exists because a constant Q and R are insufficient as soon as conditions change.
And a neural detector, for its part, outputs no covariance
This is where the objection fails. When the "measurement" is no longer a sensor but a neural network, there is no longer a metrological characterization to put into R. The observation is made by the object detection literature for autonomous driving: "bbox coordinates are output as deterministic coordinate values instead of a score, and thus the confidence of the detected bbox is unknown. Moreover, the objectness score does not reflect the reliability of the bbox well" (Choi et al., ICCV 2019).
The practitioner then fills the box with whatever is at hand: a constant sigma per sensor, or the score. Both are wrong.
Kalman has never been the problem. It is actually exactly the right equation. The problem is that ever since we replaced sensors with neural networks, the R box has been empty.
The invisible trap: two correlated sources make the system over-confident
There is a second way to go wrong, which is more discreet and does not show up in the logs.
When two estimates share common information, treating them as independent is like counting the same evidence twice. Two cameras looking at the same scene, or a vehicle and an infrastructure camera observing the same pedestrian: the case is commonplace. The result is inconsistent in the estimation sense. The reported covariance becomes smaller than the actual error. The system believes it is more certain than it actually is, without triggering any alerts. For a critical safety block, this is the worst possible failure mode.
Covariance Intersection, introduced by Julier and Uhlmann in 1997, fuses in a guaranteed-conservative manner when correlation is unknown, at the cost of "too conservative" results (Noack et al., Automatica 2017). It is a bound, not an optimum. Between naively averaging, which makes the system over-confident, and pessimistically bounding, which under-utilizes sensors, there remains a gap. This gap is closed with a true measurement of uncertainty per detection.
Why a fixed weight is wrong as soon as the scene goes out of nominal
This is the core of the multi-sensor topic, and it is physical before being algorithmic.
Modalities do not degrade under the same conditions
An automotive safety research team measured camera, lidar, and radar under reproducible conditions: calibrated rain from 16 mm/h to 98 mm/h, fog with visibility of approximately 8 m, by day and by night (Sezgin et al., 2023).
Modality | What degrades it | Nature of the degradation |
|---|---|---|
Radar | Heavy rain, water film on the radome | Moderate loss of points, due to hardware and not optical reasons |
Lidar | Dense fog, suspended droplets | The sensor detects the fog and mistakes it for objects: the point cloud fills with false positives |
Camera | Night, much more than rain | Collapse of scores, vehicle often not detected, even with night scenarios included in the training |
The lidar in dense fog does not "see less". It sees the fog. This is not a loss of range; it is an active production of perception errors.
The consequence for fusion
Fog does not fall on the radar. Night does not fall on the lidar. Sensors do not fail together; that is the very reason for having several of them on board. So why give them a weight that never changes?
This same team draws an unambiguous conclusion: we must detect the degradation of a sensor and estimate its severity so that downstream processes can take it into account. This is the argument against fixed weight, formulated from the sensor level.
The box is already in the standard, since 2023
This is the point that few teams keep in mind, and it is the one that changes the conversation.
What ETSI TS 103 324 imposes
The European standard governing the collective perception service in V2X is explicit and normative:
"For every component provided in the kinematic state and attitude space of an object in the CPM the corresponding confidence value shall be provided to a confidence level of 95 %." (ETSI TS 103 324, 2023 version)
The "shall" is not a recommendation. Every perceived object broadcast in a CPM message must carry its confidence value, at a 95% confidence level.
It goes further. It provides for up to four lower triangular correlation matrices, from which the receiver reconstructs the complete covariance, Σ = D·C·D. The probabilistic fusion pipe is standardized from end to end. The standard even distinguishes classification confidence (which object?) from the confidence value on the kinematic state (where exactly?).
And here is where the standard stops
The perception quality indicator relies on three ingredients: the age of the object, a detection confidence, and a detection success indication. The normative formula is an exponential moving average on the persistence of the detection. Have I been seeing this object for a long time, and without interruption?
For the only ingredient that relates to the actual confidence of the detector, the text points elsewhere: "the detection confidence and the detection success indication are system specific assessments of the ITS-S's object detection system".
The standard therefore mandates the field, specifies the format, sets the level at 95%, and leaves the measurement to the transmitter. But the transmitter, if running a neural detector, has nothing to put there.
The standard provided the box. It did not specify who would fill it.
An equivalent standard exists on the SAE side for sharing sensor data between vehicles, with the same logic: the container is specified, the calculation remains the responsibility of the transmitter.
Why this plays out in late fusion
Cooperative perception schemes fall into three families. Early fusion exchanges raw data and assumes a shared encoder. Intermediate fusion exchanges latent features and assumes compatible spaces between agents. Late fusion (late fusion) lets each agent produce its own detections, and only transmits the outputs.
A point of honesty: intermediate fusion currently dominates research, on benchmarks such as OPV2V (ICRA 2022) or DAIR-V2X (CVPR 2022). It dominates because, in a laboratory, all models belong to the same team. As soon as you step outside, no one exchanges features and no one opens up their model.
Late fusion is therefore not the optimal theoretical choice. It is the only deployable choice between actors who will never share their code, and that is why weighting quality becomes critical there: it is the only lever left. This is also what makes the black-box approach viable. A reliability layer (reliability layer) that plugs in as plug-and-play on the client model's outputs needs neither to retrain it, nor to access its IP.
Redundancy is not repetition
Regulation (EU) 2024/1689 addresses the accuracy and robustness of AI systems in its Article 15. Two paragraphs matter here.
The first targets errors and inconsistencies that occur "within the system or the environment in which the system operates". Fog, smoke, and night are therefore within scope. The degraded case is not an excuse; it is a use case.
The second is even more interesting: "the robustness of high-risk AI systems may be achieved through technical redundancy solutions, which may include backup or fail-safe plans". The text names technical redundancy as a path to robustness for high-risk systems, without saying how to make it work. And this is where the observation becomes useful: stacking four sensors and averaging them is not redundancy, it is repetition. Redundancy only produces robustness if the system knows which of its sources to believe, at the moment it must decide.
A third point is under-utilized. Article 15 requires that accuracy levels be declared in the instructions for use. But declaring accuracy assumes knowing how to measure it, hence characterizing uncertainty. This is a direct bridge to traceability, the audit trail (audit trail), and the type-approval file.
A reliability layer contributes to this demonstration. It does not guarantee regulatory compliance, which remains a system responsibility. See our article on what the law requires of industrial AI systems.
TrustalAI Vision, and its multi-source version V-Fusion
TrustalAI Vision is a reliability layer that estimates in real time the reliability of each prediction of a vision model, without retraining, as a black-box, with no access to the client's IP. It is reliability per prediction applied to perception: per-prediction reliability, and not average performance on a test set. TrustalAI V-Fusion is its multi-source version, applied to the combination of several streams.
What the engine produces, concretely
For each detection and for each source, the engine calculates a set of standard deviations (σx, σy, σw, σh) that serve as fusion weights. This is, literally, the missing R from the filter, built per detection instead of being configured per sensor.
Two objects that the model made indistinguishable become separable: 7 m ± 0.6 m on one side, 6 m ± 1.3 m on the other.
A degraded source sees its weight drop for the detections concerned, not for the entire frame.
The downstream system has an actionable confidence metric before the decision, rather than an a posteriori finding, after execution, once action has been initiated.
The latency fits within the constraints of embedded computing: less than 100 ms, 20 ms on CPU and 8 ms on NVIDIA Jetson Orin. This is compatible with a real-time perception loop and with edge computing architectures, as we detail in edge AI reliability at 20 ms.
What it achieved in the VEDECOM PoC
The PoC conducted with the VEDECOM Institute focused on a probabilistic fusion using TrustalAI uncertainties as weights. The most relevant point here is not the raw gain, but the comparison against 7 fusion methods.
Indicator | Baseline | Uncertainty-weighted fusion | Gain |
|---|---|---|---|
Position errors | 1.44 m | 0.51 m | -65 % |
Orientation errors | 6.28 ° | 2.35 ° | -63 % |
False positives | — | — | -83 % |
(VEDECOM PoC, benchmark vs 7 fusion methods)
What it changes for validation
Fewer false positives, thus fewer unjustified emergency stops.
A documented perception confidence, allowing for a gradual reduction in human supervision.
Heterogeneous sensors finally reconcilable, even when they do not have the same quality.
Reduced validation cost: a measured uncertainty is an uncertainty that can be declared.
This last point is the most structural on a Level 4 (Level 4) or Level 5 (Level 5) approval program, where the burden of proof weighs heavier than nominal performance.
And what about multi-agent in all this?
The same arithmetic applies when the sources are no longer the sensors of a single vehicle, but distinct agents: an autonomous shuttle and an instrumented intersection, for example. The economic stakes of this case are different, and we address them separately in cooperative perception: making offboard sensors profitable. The underlying logic, however, does not change: an agent whose reliability we do not know how to measure is an agent we can only integrate at a fixed weight.
FAQ
What is uncertainty-weighted multi-sensor fusion?
It is a fusion where the weight of each source is calculated from the expected error of each detection, rather than fixed in advance per sensor. Concretely, an uncertainty engine produces a standard deviation per detection, which serves as the measurement noise matrix in the combination. A temporarily degraded source then carries less weight, without manual intervention or hardcoded rules.
How is this different from a classic Kalman filter?
The Kalman filter already weights by uncertainty: that is the role of the R matrix. But R historically describes the sensor, calibrated once, then applied to all detections. A neural detector, however, produces no covariance. The equation remains correct; it is its input that is empty. A measurement of uncertainty per detection fills this box.
Why not simply weight by the detector's confidence score?
Because the score relates to the class, not the position. A high score indicates that there is probably an object, not that it is in the right place. Furthermore, calibration literature shows that pre-trained detectors are over-confident. Weighting by the score therefore means prioritizing the most assertive source, not the most accurate one.
Do V2X standards already provide for this uncertainty field?
Yes. The ETSI TS 103 324 standard requires that each component of the kinematic state of a perceived object carry its confidence value at a 95% level, and provides the matrices to reconstruct the complete covariance. However, it leaves the calculation of this confidence to an assessment specific to the transmitter's detection system. The field exists; the measurement remains to be provided.
Do we need to retrain perception models to use it?
No. The approach is black-box and plug-and-play: it applies to the outputs of existing models, without access to the code or weights, and without retraining. This is also what makes it usable between different actors, in late fusion, where no party would agree to share its internal features. The qualification cycle fits within a 2-week PoC.
Share
Related articles
