The Hidden Assumption in Infrared-Visible Fusion
Many infrared-visible image fusion methods begin from a convenient assumption: the infrared and visible images are already well registered, and both modalities are locally reliable.
That assumption is useful for building clean benchmarks and controlled comparisons. It is also fragile.
In real sensing systems, dual sensors may have spatial misalignment. The two modalities may have temporal delay. Visible images may suffer from low light, motion blur, glare, rain, haze, or overexposure. Infrared images may contain thermal noise, low contrast, saturation, or weak target boundaries.
If a fusion method assumes both modalities are clean and aligned, it may produce pleasing results on standard examples while becoming unstable under realistic conditions.
This is the starting point for RobustIRFusion.
Why Misalignment Matters
Misalignment is not just an inconvenience for visualization. It changes what fusion means.
If a visible edge is shifted away from the corresponding infrared target, blindly injecting visible texture can create ghosting or false structure. If the system forces two inconsistent signals into the same representation, the fused image may become less useful for downstream perception, even if it looks sharper in some regions.
For practical multimodal perception, the question is not:
How can we always combine both modalities as much as possible?
The better question is:
How should the system behave when one modality is locally unreliable or spatially inconsistent with the other?
That question points toward reliability-aware fusion rather than purely symmetric feature mixing.
Why Reliability Matters
Visible light is often rich in texture, but texture is not always trustworthy. It can be damaged by low illumination, overexposure, blur, sensor noise, or weather-like degradation. Infrared is often more stable for thermal targets, but it is not perfect either.
A robust fusion system should not treat every feature as equally useful. It should have a way to reduce the influence of unreliable information, especially when that information is visually detailed but physically misleading.
This is why the paper direction is not simply “a new Mamba fusion network” or “a better fusion backbone.” The main idea is to make the fusion process more conservative and controlled under imperfect sensing conditions.
Infrared as an Anchor, Visible Light as Conditional Detail
The current project hypothesis is:
Infrared should act as the primary anchor for robust sensing, while visible information should be injected as conditional detail when it is reliable.
In a simplified form, the idea looks like this:
R = E_ir(I_ir)
X = E_vi(I_vi)
Delta = f_delta(R)
g = sigmoid(f_gate(R))
q = f_quality(R, X, mismatch)
U = P_ir(R) + g * q * P_vi(X)
Y = SSM(U; Delta)
F = R + YHere, R is an infrared-centered representation, X is a visible representation, g controls visible texture injection, and q estimates local reliability or misalignment confidence.
The important part is not the exact notation. The important part is the behavior: visible features should not be injected equally everywhere. Their influence should depend on infrared priors and local confidence.
What Evidence Would Convince Me
For this project to be convincing, clean benchmark scores are not enough.
The evidence should include:
- aligned fusion results on datasets such as MSRS, M3FD, and TNO;
- controlled synthetic misalignment tests, including shifts, rotations, and local perturbations;
- visible degradation tests, such as low light, blur, noise, glare, and overexposure;
- ablations comparing symmetric fusion, infrared-control only, and infrared-control with reliability confidence;
- visualizations of
gandqmaps; - examples where the method avoids ghosting or unreliable visible texture injection.
The key story should be:
Infrared-centered control provides a stable anchor, and reliability/misalignment confidence prevents unreliable visible texture from corrupting the fused result.
What This Project Should Not Become
The scope has to stay narrow.
This project should not expand into full registration, segmentation-assisted fusion, diffusion fusion, video fusion, event-camera fusion, or a large downstream task suite before the core proof is complete.
The first version only needs to prove one thing well:
Reliability-aware infrared-centered fusion is more robust than symmetric fusion or clean-only infrared-controlled fusion under misalignment and modality degradation.
If the method cannot prove that, adding more components will not fix the story. If it can prove that, the project already has a clear contribution.
Current Status
This is the active paper line. The earlier EMFusion idea remains useful as a branch or base model: infrared-controlled selective state-space fusion where infrared features generate control variables such as Delta and g.
RobustIRFusion extends that direction by making reliability and misalignment central rather than treating clean alignment as the default world.
The next work is practical and evidence-driven: dataset setup, synthetic misalignment and degradation generation, baseline evaluation, a minimal architecture, and tightly scoped ablations.