Abysstree渊林ovc
Back to notes返回笔记

Draft草稿 / 2026-07-04

From Optoelectronics to Computational Imaging and Multimodal AI从光电信息到计算成像与多模态 AI

A draft research-identity essay about moving from optoelectronics toward AI systems for multimodal visual perception, computational imaging, and deployable visual restoration.一篇研究身份草稿:记录我如何从光电信息走向面向多模态视觉感知、计算成像和可部署视觉恢复的 AI 系统。

The Starting Point: Optoelectronics, Not Pure Software

My technical background started from optoelectronic information science and engineering. That background does not make me a traditional optical designer by default, but it gives me a useful way of looking at visual AI systems: images are not just arrays of pixels. They are produced by optics, sensors, sampling, noise, exposure, reconstruction, compression, and deployment constraints.

This is why I am interested in the region between imaging hardware and learning-based perception. A model that works on a clean benchmark can still fail when the sensor pair is misaligned, the visible image is overexposed, the infrared image has low contrast, or the camera pipeline introduces artifacts. I want my work to live closer to those real imaging conditions.

Why I Moved Toward AI-Connected Imaging

I am not trying to build my main identity around traditional optical design, optical materials, or pure medical imaging. Those are important areas, but they are not the direction that best connects my background, interests, and near-term goals.

The direction that feels more durable to me is AI-connected imaging: using learning-based methods while still respecting where visual data comes from. This includes multimodal visual perception, computational imaging, infrared-visible fusion, RGB-T perception, depth cameras, LiDAR or point-cloud perception, ISP-aware restoration, and deployable camera systems.

The common thread is not a single model family. It is a question:

How should AI systems use imperfect visual signals from real sensing pipelines?

That question is broad enough to support future PhD applications in computational imaging, multimodal perception, robotics perception, or computer vision, while also being practical enough for industry roles in ISP, AI camera systems, autonomous sensing, AR/VR, and edge vision.

Multimodal Perception: When One Sensor Is Not Enough

Infrared-visible perception is a natural example. Visible cameras preserve texture, color, and scene detail, but they can be fragile under low light, glare, haze, blur, rain, or overexposure. Infrared sensors can preserve thermal targets and work in darkness, but they may have weaker texture, lower contrast, thermal noise, or less precise boundaries.

The interesting problem is not simply how to merge two images into something visually pleasing. The deeper problem is how to decide which modality should be trusted, where, and under what degradation. In real systems, two modalities are rarely perfectly aligned or equally reliable.

This is the motivation behind my current RobustIRFusion paper line: infrared-centered fusion under misalignment and reliability uncertainty. Instead of treating infrared and visible images as symmetric inputs, I am interested in using infrared as a stable anchor and visible information as conditional detail.

Computational Imaging and AI Camera Restoration

The same thinking also appears in AI camera restoration. Real camera images can contain artifacts caused by optics, sensors, display-camera sampling, demosaicing, denoising, sharpening, compression, or the interaction between these stages.

Photo moire removal is a compact example. When a phone photographs a screen, the display pixel structure interacts with camera sampling and ISP processing. The result is not just generic noise. It is a structured imaging artifact. Removing it well requires both restoration engineering and awareness of how the artifact is formed.

This is why I want to build small portfolio modules under an AI camera restoration line. Each module should be narrow enough to finish, but polished enough to show real engineering taste: reproducible inference, visible before/after results, timing, crop comparisons, failure cases, and clear documentation.

What I Want This Portfolio To Show

This website should not become a heavy third project. It should be a public entry point for the work I am already doing.

The portfolio should show four things:

  1. I can define research problems from real sensing constraints, not only from benchmark trends.
  2. I can build reproducible systems around datasets, baselines, metrics, and visualization.
  3. I can connect optics, ISP, computational imaging, and modern AI methods.
  4. I can communicate projects clearly enough for future advisors, collaborators, and interviewers to understand the direction quickly.

For now, the site will stay lightweight: project pages, research notes, technical logs, and a small number of polished essays. The goal is not to write constantly. The goal is to make each public note support the larger research narrative.

Current Threads

The current main paper line is RobustIRFusion: misalignment- and reliability-aware infrared-centered fusion for robust multimodal perception.

The current portfolio demo line is Photo Moire Removal: a small AI camera restoration project based first on a reproducible UHDM/ESDNet baseline and a before/after result experience.

Together, these projects form the first version of the identity I want to build:

AI systems for multimodal visual perception, computational imaging, and deployable visual restoration.

起点:光电信息,而不是纯软件

我的技术背景从光电信息科学与工程开始。这个背景并不意味着我默认要成为传统光学设计者,但它给了我一种很有用的视觉 AI 视角:图像不是单纯的像素数组。它们由光学、传感器、采样、噪声、曝光、重建、压缩,以及真实部署中的各种约束共同产生。

这也是为什么我会对成像硬件和学习式感知之间的区域感兴趣。一个模型可以在干净的基准测试上表现很好,但当传感器对不准、可见光过曝、红外低对比度,或者相机处理流程引入伪影时,仍然可能失效。我希望自己的工作更靠近这些真实成像条件。

为什么我转向 AI 关联成像

我并不打算把自己的主要身份建立在传统光学设计、光学材料或纯医学影像上。这些方向都很重要,但它们不是最能连接我的背景、兴趣和近期目标的方向。

对我来说,更持久的方向是 AI 关联成像:使用学习式方法,同时仍然尊重视觉数据从哪里来。这包括多模态视觉感知、计算成像、红外-可见光融合、RGB-T 感知、深度相机、LiDAR 或点云感知、ISP 感知恢复,以及可部署相机系统。

共同主线不是某一个模型家族,而是一个问题:

AI 系统应该如何使用来自真实传感流程的不完美视觉信号?

这个问题足够宽,可以支撑未来面向计算成像、多模态感知、机器人感知或计算机视觉的 PhD 申请;同时它也足够实际,可以连接 ISP、AI 相机系统、自动感知、AR/VR 和边缘视觉等工业方向。

多模态感知:当一个传感器不够时

红外-可见光感知是一个自然例子。可见光相机保留纹理、颜色和场景细节,但在低照度、眩光、雾、模糊、雨天或过曝下会很脆弱。红外传感器可以保留热目标,并在黑暗中工作,但它也可能有纹理弱、对比度低、热噪声或边界不够精确的问题。

真正有趣的问题不只是如何把两张图合成得更好看。更深的问题是:在什么位置、什么条件下,系统应该信任哪一个模态。真实系统里,两种模态很少完全对齐,也很少同等可靠。

这就是我当前 RobustIRFusion 论文方向的动机:在错位和可靠性不确定下做红外中心融合。与其把红外和可见光当成完全对称的输入,我更关心如何让红外作为稳定锚点,并把可见光信息作为有条件的细节注入。

计算成像与 AI 相机图像恢复

同样的思路也出现在 AI 相机图像恢复里。真实相机图像中的伪影可能来自光学、传感器、显示屏-相机采样、去马赛克、去噪、锐化、压缩,或者这些阶段之间的相互作用。

拍屏摩尔纹去除是一个紧凑例子。当手机拍摄屏幕时,显示屏像素结构会和相机采样、ISP 处理发生交互,结果不是普通噪声,而是一种结构化成像伪影。要把它去除好,需要图像恢复工程,也需要理解伪影是如何形成的。

这也是为什么我想在 AI 相机图像恢复下面做一些小型作品集模块。每个模块应该足够窄,窄到可以完成;也要足够精致,能体现工程品味:可复现推理、可见的前后对比结果、耗时、裁剪区域对比、失败案例和清晰文档。

我希望这个作品集展示什么

这个网站不应该变成第三条重型项目线。它应该是我已有工作的公开入口。

这个作品集应该展示四件事:

  1. 我能从真实传感约束中定义研究问题,而不只是追随基准测试趋势。
  2. 我能围绕数据集、基线、指标和可视化构建可复现系统。
  3. 我能连接光学、ISP、计算成像和现代 AI 方法。
  4. 我能把项目表达清楚,让未来导师、合作者和面试官快速理解方向。

目前,这个网站会保持轻量:项目页、研究笔记、技术日志和少量打磨过的文章。目标不是持续高频写作,而是让每一篇公开笔记都服务于更大的研究叙事。

当前线索

当前主要论文方向是 RobustIRFusion:面向鲁棒多模态感知的错位感知、可靠性感知红外中心融合。

当前作品集 Demo 方向是 Photo Moire Removal:一个小型 AI 相机图像恢复项目,先基于可复现的 UHDM/ESDNet 基线和前后对比结果体验。

它们共同构成了我想建立的第一版身份:

面向多模态视觉感知、计算成像和可部署视觉恢复的 AI 系统。