Infrared–Visible Multimodal Object Detection
红外–可见光多模态目标检测
Adaptive RGB–IR Fusion under Changing Visual Conditions
变化视觉条件下的自适应 RGB–红外融合
SAT301 Final Year Project, Xi’an Jiaotong-Liverpool University · May 20262026 年 5 月 · Supervised by Dr. Xiaohui Zhu· 导师：朱晓辉博士
An RGB–infrared object detection project investigating whether lightweight adaptive modality weighting can dynamically balance visual and thermal information under changing scene conditions.
一项 RGB–红外目标检测研究：轻量自适应模态加权能否在变化场景中动态平衡可见光与热红外信息。
Should RGB and infrared always contribute equally?
RGB 和红外是否应该始终等权贡献？
The technical question on this page is narrower than “does fusion help?”. It asks whether a multimodal detector can learn when to trust visible or infrared information.
本页的技术问题比“融合有没有用”更窄：多模态检测器能否学会何时信任可见光或红外信息。
Texture, colour, boundaries
纹理、颜色、边界
Useful in well-lit scenes, but the signal can degrade under low light, glare, rain, or haze.
光照充足时很有用，但在低光、眩光、雨天或雾霾下信号会变差。
Thermal contrast
热对比
Often stronger in low light, but it carries less texture, less colour, and weaker fine detail.
低光下通常更稳，但纹理、颜色和细结构更弱。
Learn a modality preference instead of fixing one
学习模态偏好，而不是手工固定权重
If the two sensors are not equally reliable in every scene, a detector that always mixes them 1:1 may waste a trustworthy channel or over-trust a degraded one. The project therefore tests whether a lightweight weighting rule can move with the input.
如果两个传感器并非在每个场景都同样可靠，始终 1:1 混合可能会浪费可信通道，或过度信任已经退化的通道。因此本项目检验轻量加权规则能否随输入变化。
Can the model learn when to trust RGB or IR?
模型能否学会何时信任 RGB 或红外？
The page is organised around this one question, not around platform reproduction or dataset migration.
整页围绕这一问展开，而不是平台复现或数据集迁移。
Reliability is scene-dependent
可靠性随场景变化
Visible cues should matter more when texture is intact; thermal cues should matter more when visible contrast collapses.
纹理完整时可见光应更重要；可见光对比崩溃时热红外应更重要。
Fix the detector, change the mix
固定检测器，只改混合方式
Static weights, a dynamic gate, and a conservative residual gate are compared against default multimodal fusion.
将静态权重、动态门控和保守残差门控与默认多模态融合对比。
Not a guaranteed win
并不保证全面更好
If default fusion is already strong, adaptive weighting may only appear when modality quality becomes uneven.
若默认融合已经很强，自适应加权可能只在模态质量更不均匀时才显现。
A registered RGB–infrared pair
一对已配准的 RGB–红外图像
Each sample is a spatially aligned visible image and infrared image. The pair below is a night scene from the project materials: pedestrians that fade in RGB remain clear in IR.
每个样本都是空间对齐的可见光图与红外图。下面是项目材料中的夜景：RGB 里几乎看不见的行人，在红外中仍然清楚。
Paired input成对输入 Same camera geometry, two physical measurements. No extra metadata is required at inference beyond the image pair.同一几何关系，两种物理测量。推理时除图像对外不需要额外元数据。
Two streams, one DiffusionDet head
双流特征，一个 DiffusionDet 检测头
The experimental framework is E2E-MFD-HOD on Detectron2 with a DiffusionDet detection head. The diagram stays conceptual: only the verified pipeline is shown.
实验框架是 Detectron2 上的 E2E-MFD-HOD，检测头为 DiffusionDet。图保持概念层：只画出已核实的流程。
Adaptive modality weighting
自适应模态加权
Instead of assuming equal RGB / IR importance, the fusion step learns a visible weight \(\alpha\) from features. \(\alpha\) may vary by feature level and by input.
融合步不再假设 RGB / 红外同等重要，而是从特征中学习可见光权重 \(\alpha\)。\(\alpha\) 可以随特征层和输入变化。
Fixed static weighting
固定静态加权
Manually set visible ratios to 0.25, 0.50, and 0.75.
把可见光比例手工设为 0.25、0.50 和 0.75。
Lightweight dynamic gating
轻量动态门控
Predict \(\alpha\) from the two feature streams.
由两路特征预测 \(\alpha\)。
Conservative residual gating
保守残差门控
Keep the default fusion path and learn only a small correction.
保留默认融合路径，只学习一个小修正。
Preference, not a hand-set mix
学习偏好，而不是手调混合比
Ask the model to allocate trust instead of freezing it.
让模型分配信任，而不是把比例冻死。
Multiclass boxes, labels, and scores
多类检测框、类别与分数
Object bounding boxes
目标边界框
Horizontal detections on the aligned scene.
对齐场景上的水平检测框。
Category names
类别名称
A multiclass vocabulary, including people and common vehicles.
多类词表，包括行人和常见车辆。
Confidence
置信度
Each box carries a detector confidence used by AP.
每个框带有检测置信度，用于计算 AP。
Multiclass object detection
多类目标检测
No extra heads beyond detection are claimed here.
这里不声称检测以外的额外任务头。
AP, and only AP
只用 AP
The report uses standard COCO-style detection metrics. Hover or focus a card for the short definition used on this page.
报告使用标准 COCO 风格检测指标。把指针放在卡片上，或聚焦卡片，即可看到本页使用的简短定义。
Four readings of the same question
同一问题的四种读法
Result 1 — Multimodal advantage
结果 1 — 多模态优势
On the candidate-init official route, combining RGB and infrared produces the strongest overall detector. Two multimodal seeds stay within 0.33 AP of each other.
在 candidate-init 官方路线上，RGB 与红外结合得到最强整体检测器。两个多模态随机种子相差不超过 0.33 AP。
Key message要点 Combining RGB and infrared produces the strongest overall detector.结合 RGB 与红外得到最强的整体检测器。
Result 2 — Full-benchmark fusion ablation
结果 2 — 全基准融合消融
Once the default multimodal detector is strong, simple adaptive reweighting does not beat it on the full benchmark. The negative result is part of the answer.
默认多模态检测器已经较强时，简单自适应再加权在全基准上没有超过它。这一负结果也是答案的一部分。
Exact full-benchmark AP values全基准 AP 精确值
| Method方法 | AP | AP50 |
|---|---|---|
| Default multimodal默认多模态 | 53.1935 | 82.3669 |
| Static 0.25静态 0.25 | 52.1932 | — |
| Static 0.50静态 0.50 | 52.8243 | — |
| Static 0.75静态 0.75 | 52.8200 | — |
| Dynamic gate动态门控 | 52.9163 | — |
| Residual gate残差门控 | 53.1922 | — |
Key message要点 Simple adaptive reweighting does not outperform the strong default fusion on the full benchmark.简单自适应再加权在全基准上没有超过较强的默认融合。
Result 3 — Learned modality preference
结果 3 — 学到的模态偏好
The dynamic gate still changes its preference across the feature pyramid. That is a behavioural observation, not a causal proof that the gate “understands” weather or lighting.
动态门控在特征金字塔各层上的偏好仍会变化。这是行为观察，不是它“理解”天气或光照的因果证明。
Key interpretation解读 Modality preference changes across feature scales. Causality is not claimed.模态偏好随特征尺度变化。这里不声称因果关系。
Result 4 — Supplementary challenge-scene evaluation
结果 4 — 补充性挑战场景评估
This split is automatically constructed from image statistics. It is not an official benchmark. It only asks whether adaptive fusion looks more useful when modality quality is more uneven.
该划分由图像统计自动构造，不是官方基准。它只用来问：当模态质量更不均匀时，自适应融合是否显得更有用。
Visible, infrared, equal fusion, dynamic fusion
可见光、红外、等权融合、动态融合
Switch condition. Only plates that exist in the source materials are shown. Glare and failure-case images were not available, so those tabs stay empty rather than being filled with substitutes.
切换条件查看。只展示源材料里已有的图。眩光和失败例当时没有图，因此对应页签留空，不用替代图填上。
Evidence boundary证据边界 Detection overlays and TP/FN counts are shown only when they appear on the source qualitative plates. Missing panes are left empty on purpose.检测框与 TP/FN 仅在原始质性图里已经标注时才显示。缺图的格子会留空，不会用未核实图片填上。