Skip to content

Project Timeline — LWIR MFSR Development History

This is the chronological narrative of the entire LWIR Multi-Frame Super-Resolution project, reconstructed from git histories across all 11 repositories. It explains when each tool was started, why approaches changed, where things stalled, and how the project arrived at its current state.

Phase 0: The First Attempt (May--Jun 2018)

Repository: super-resolution/ (27 commits)

The SR research began in May 2018 with a classical (non-neural) approach. The work focused on Lucas-Kanade optical flow registration and SIFT keypoint matching to align LWIR frames, then combining them into a super-resolved output. Over about a month, the codebase progressed through homography estimation, iterative registration, SSIM/MSE computation, and valid-region overlap calculation. The last commit on June 7, 2018 mentions "sped up sift matching, now loading homographies from disk."

This effort produced a working prototype but was shelved for six years. The classical methods could align frames but the super-resolution quality was limited by the registration accuracy achievable with hand-crafted features on low-contrast thermal imagery.

Phase 1: HighRes-net and the ProbaV Dataset (Nov 2024--Apr 2025)

Repository: HighRes-net/ (251 commits, original upstream from Jul 2019)

The project restarted in late 2024 when the HighRes-net architecture (originally published for the ESA ProbaV satellite SR challenge) was adapted for LWIR data. The upstream repo had been dormant since 2019; local work began on November 26, 2024 with Docker/CUDA setup and a ChatGPT-assisted rewrite to get the model training.

The Registration Detour (Dec 2024--Feb 2025)

Almost immediately, the core problem emerged: LWIR frames need sub-pixel alignment before super-resolution can work, and existing methods were inadequate. From December 2024 through February 2025, the majority of HighRes-net work was actually registration research:

  • Dec 11--18, 2024: Built an interactive registration app with multiple metrics (NCC, perceptual loss, SSIM, VGG activations). Spent days debugging wrap-around artifacts and instability in perceptual loss.
  • Jan 21--28, 2025: Intensive sprint on sub-pixel alignment: spiral search around metric centroids, multi-metric voting schemes, recursive grid search. Many late-night commits suggest frustration with convergence.
  • Jan 29--Feb 8, 2025: Integrated Facebook SAM for segmentation-assisted registration, implemented multi-scale search (LR then HR space), built a full model manager, added data augmentation, experimented with L1+L2 combined losses and edge-based losses. Training runs were crashing overnight due to augmentation bugs.

Architecture Experimentation (Mar--Apr 2025)

By March 2025, attention shifted to the network architecture itself:

  • Mar 1--4, 2025: Built Swin Transformer-style hierarchical fusion blocks, residual prediction heads, and multiple custom fusion modules. The goal was to replace HighRes-net's ShiftNet with something more capable.
  • Mar 15--16, 2025: Experimented with explicit sub-pixel shift injection into the fusion model, trying to leverage the registration metadata directly.
  • Apr 1, 2025: Last commits are on model block diagrams and evaluation code. The HighRes-net line of work was effectively paused here.

Why it stalled: The fundamental problem was not the architecture but the data pipeline. Registration quality was inconsistent, training was unstable (see Training Instability), and the model could not consistently beat bicubic upsampling. The project needed better aligned training data before network improvements would matter.

Phase 2: lwir-align — Dedicated Registration Pipeline (Mar--Sep 2025)

Repository: lwir-align/ (55 commits)

Recognizing that registration was the bottleneck, a dedicated alignment pipeline was spun off on March 18, 2025. This tool registers 675 high-altitude (HA) LWIR frames against low-altitude (LA) mosaics using SuperGlue and RAFT.

SuperGlue and RAFT (Mar--Jul 2025)

  • Mar 18, 2025: Intense single-day sprint to build the initial pipeline (7 commits in one day).
  • Mar 24, 2025: Refactored with Cursor (AI-assisted IDE). Integrated SuperGlue for keypoint matching and fixed major optical flow bugs in RAFT.
  • Mar 25, 2025: Also attempted to process the ProbaV satellite dataset for comparison.
  • Jun 2--3, 2025: Returned after a two-month gap. Fixed critical homography direction bug that had wasted 3 hours. Added mosaic preprocessing with contrast stretching and a two-stage keypoint strategy (detect at half-res, match at full-res) to manage GPU memory.
  • Jun 17, 2025: Deep dive into camera undistortion parameters from Metashape XML files.
  • Jul 8--31, 2025: Intensive group-by-group registration campaign. Images were divided into spatial groups and registered sequentially. SuperGlue kept failing on some groups ("uhg. nothing group 8 refuses to register with superglue") until the discovery that pre-warping images with initial guess matrices made SuperGlue work reliably. RAFT was discovered to work where SuperGlue failed (Jul 22: "RAFT is working!!!").

The Birth of Hephaestus (Jul 30, 2025)

On July 30, the lwir-align repo contains a pivotal commit: "started work on this registration tool with openGL support because the transforms I was getting from GIMP were not right." The manual registration workflow using GIMP was producing inaccurate homographies. This frustration directly spawned Hephaestus as a dedicated tool the very next day.

Presentation and Polish (Aug--Sep 2025)

The last phase of lwir-align focused on refining registration matrices and building HTML presentations with Plotly to communicate progress. See LWIR-Align for full details.

Phase 3: Aerofocus — Blur Detection (Apr 2025)

Repository: aerofocus/ (82 commits over 4 days)

A brief but intense detour in early April 2025. Aerofocus was a blur detection and focus quality assessment tool for LWIR imagery, built April 7--10, 2025 (82 commits in 4 days — one of the highest-velocity sprints in the project). The tool used motion data from flight logs to correlate blur with aircraft dynamics.

Notable commit from Apr 10: "working with gemini 2.5 is amazing" — this was one of the first tools built primarily with AI assistance (Gemini 2.5 Pro).

Aerofocus addressed a real concern — some LWIR frames are motion-blurred and should be excluded from SR training — but the tool was not integrated into the main pipeline and saw no further development after April 10.

Phase 4: mfsr_mosaic — Georeferencing and Tiling (May--Jun 2025)

Repository: mfsr_mosaic/ (23 commits)

With registration proving difficult at the frame level, May 2025 brought a new approach: use georeferencing and INS (Inertial Navigation System) data to align frames in geographic coordinates, then extract aligned tile pairs. See MFSR Mosaic for details.

  • May 1, 2025: Project started as "a new approach to building my MFSR training data set."
  • May 6, 2025: Discovered the actual flight altitudes (LA ~800m, HA ~1218m ASL), which determine the SR magnification factor.
  • May 8, 2025: Built tiling utilities, footprint calculations, and GeoTIFF tools.
  • May 27, 2025: Attempted INS-based camera calibration with Claude's help. Multiple rounds of fixes. The yaw offset correction improved angle alignment but the calibration was never fully solved.
  • Jun 2, 2025: Final commits moved the "failed calibration attempt" into its own folder. The commit message reads: "i am not going to continue trying this approach."

Why it stalled: The INS calibration approach could not produce the sub-pixel accuracy needed for SR training data. The camera-to-INS offset had too many degrees of freedom and the ground truth for calibration was insufficient. The project pivoted back to feature-based registration (lwir-align Phase 2).

Phase 5: Hephaestus — Interactive Registration GUI (Jul--Oct 2025)

Repository: hephaestus/ (98 commits)

Born from frustration with GIMP's inadequate homography tools, Hephaestus became one of the most developed tools in the project. See Hephaestus for architecture details.

  • Jul 31, 2025: First commit. Built with PyQt6 and OpenGL for real-time red-cyan anaglyph display. Integrated RAFT and SuperGlue from day one. Immediately went through a massive refactor ("the neverending code refactor" — 7 commits on launch day alone).
  • Aug 17--19, 2025: Ran batch analysis overnight, then worked on batch vector processing and matrix refinement.
  • Sep 10, 2025: Cleanup and getting the main scripts working.
  • Oct 9--10, 2025: Added Parker's ground-truth dataset browser and mosaic-scale cross-modality (LWIR-to-VNIR) registration support. This was the Phase 2 expansion into visible-spectrum alignment.

Phase 6: PIUnet — Neural Network Training (Oct--Nov 2025)

Repository: piunet/ (32 commits, original upstream from May 2021)

With aligned training data finally available from the lwir-align + hephaestus pipeline, the project could attempt neural super-resolution again — this time with PIUnet instead of HighRes-net. See PIUnet Architecture for the technical deep-dive.

  • Oct 16, 2025: Major push — added LWIR dataset support, preprocessing pipeline, temporal sequence analysis, footprint intersection analysis. Restructured to standard Python package layout. Many results visualizations generated.
  • Oct 20, 2025: Fixed PIUnet registration and computed correct 16-bit PSNR. Generated WebP comparison images.
  • Oct 27, 2025: Added residual learning implementation, demo inference scripts, Windows setup for boss evaluation (PowerShell, Miniconda support), and a quick-start guide.
  • Nov 13, 2025: Last commit: "Add interactive 26-tile inference results website" — a presentation for the boss meeting.

The Boss Meeting (Nov 2025): PIUnet scored 0.35--0.99 dB below bicubic on all 3 test sequences. See Bicubic Gap for the full post-mortem. The core problem was that spatially-invariant alignment (TERN module) could not handle the complex warping in aerial thermal data, and there was simply not enough well-aligned training data. The project was paused.

Phase 7: MatchAnything and Cross-Modality (Oct 2025)

Repository: matchanything-standalone/ (13 commits over 2 days)

A quick exploration of MatchAnything for cross-modality thermal-to-visible feature matching, built October 27--28, 2025. This was related to Hephaestus Phase 2 and Parker's ground-truth registration dataset.

The critical discovery was documented in the final commits: MatchAnything "requires manual pre-alignment" — it cannot match thermal to visible images from scratch. This limitation led to offering Hephaestus as the GUI alternative for Parker's workflow.

Phase 8: lwir_tile_validator — Quality Control (Sep--Oct 2025)

Repository: lwir_tile_validator/ (16 commits)

Built September 28--October 29, 2025 to systematically validate the aligned tile pairs produced by the registration pipeline. See LWIR Tile Validator for details.

  • Sep 28--30, 2025: Initial build with ProbaV dataset export, major UI refactor when the code hit 1000 lines.
  • Oct 6, 2025: Added patch swapping and sharpness scoring.
  • Oct 29, 2025: Added 9-tile grid layout, export browse mode, and fixed segfault in grid reinitialization. This was the last active development.

Phase 9: Research Consolidation and New Direction (Apr 2026--Present)

Repository: lwir-mfsr-research/ (Vue.js presentation site, no git history) Repository: mfsr-project-wiki/ (this wiki)

After the Nov 2025 boss meeting and the subsequent project pause through winter, April 2026 marks the consolidation phase:

  • Building this wiki as institutional memory of the entire effort
  • Compiling research papers and reports into actionable articles
  • Documenting struggles and anti-patterns to avoid repeating them
  • Planning the next phase: new 2x data collection (see Data Collection v2), modern architectures, and better evaluation strategies (see Evaluation Strategy)

Summary Statistics

Repository Commits Active Period Status
super-resolution/ 27 May--Jun 2018 Archived
HighRes-net/ 251 Nov 2024--Apr 2025 Paused
lwir-align/ 55 Mar--Sep 2025 Paused
aerofocus/ 82 Apr 7--10, 2025 Shelved
mfsr_mosaic/ 23 May--Jun 2025 Failed, archived
hephaestus/ 98 Jul--Oct 2025 Paused (Phase 2 incomplete)
piunet/ 32 Oct--Nov 2025 Paused (failed to beat bicubic)
matchanything-standalone/ 13 Oct 27--28, 2025 Shelved (requires pre-alignment)
lwir_tile_validator/ 16 Sep--Oct 2025 Paused
probav_data_processed/ 0 -- Data only, no git
lwir-mfsr-research/ 0 Apr 2026 Active (presentation site)

Total commits across all repos: 597

Key Lessons

  1. Registration dominates everything. More than half the total development effort across all repos was spent on image alignment. Super-resolution is impossible without sub-pixel registration, and LWIR thermal imagery is exceptionally difficult to register due to low contrast and lack of sharp features.

  2. The data problem was underestimated. The original flights (21051/21052) were not designed for SR research. The altitude ratio, sparse overlap, and single-site limitation created problems that no amount of network architecture tuning could solve. See Existing Data Limitations.

  3. Tools proliferated because each approach hit a wall. The progression from HighRes-net's built-in ShiftNet to lwir-align to mfsr_mosaic to hephaestus was not planned — each tool was born from the failure of the previous approach. This is why the project has 11 repositories instead of 2.

  4. AI-assisted development accelerated dramatically over the project's life. Early work (2018, late 2024) was manual. By April 2025, Gemini 2.5 was writing large chunks of aerofocus. By mid-2025, Claude and Cursor were integral to lwir-align and hephaestus development. The wiki itself is being compiled by Claude agents.

  5. The bicubic gap is the defining failure. After a year of work and 597 commits across 11 repos, PIUnet still could not beat bicubic interpolation. See Bicubic Gap for why, and Evaluation Strategy and Data Collection v2 for the path forward.