LWIR-Align

Registration pipeline for aligning High-Altitude (HA) LWIR frames to Low-Altitude (LA) mosaic imagery. This is the geometric backbone of the MFSR project -- without accurate frame-to-mosaic homographies, no training data can be generated.

What It Does

LWIR-Align takes 675 HA thermal frames (1024x768, 16-bit, captured at ~1218m ASL from flight 21052) and registers each one to two LA mosaics (flight 21051, ~800m ASL) via 3x3 homography matrices. The output is a per-frame H_<timestamp>_to_mosaic{A|B}.json file that can warp the HA frame into the mosaic's coordinate space.

The pipeline handles the full registration lifecycle:

1. Undistortion -- Correct Jenoptik lens distortion using camera calibration parameters extracted from Metashape XML (scripts/batch_undistort_filtered_ha.py, scripts/undistort_image.py)
2. Group organization -- 675 frames divided into 8 temporal groups (group1: 85 frames, group2: 144, group3: 103, group4: 96, group5: 65, group6: 4, group7: 11, group8: 22)
3. Initial guess generation -- Compose pairwise HA-to-HA homographies with a manually established anchor to get coarse frame-to-mosaic transforms
4. Refinement -- Use SuperGlue or RAFT to refine the coarse homographies into sub-pixel accurate alignments
5. Dual-mosaic registration -- Register each frame to both Mosaic A and Mosaic B where coverage overlaps, linked by mosaicB_to_mosaicA_homography.json
6. Tile metadata generation -- Produce comprehensive metadata for downstream tools like LWIR Tile Validator

Source: lwir-align/README.md, lwir-align/CLAUDE.md

Registration Methods

All methods implement AbstractRegistrationMethod (src/methods/base.py) with a common interface: register(reference, template) -> H returning a 3x3 homography.

SuperGlue (src/methods/superglue.py)

Primary method. Uses SuperPoint keypoint detection + SuperGlue learned matching. The refinement script (scripts/refine_group_to_mosaic.py) projects HA frame corners into mosaic space, creates a bounding box (expanded 10%), and filters SuperGlue matches to lie within that region. This spatial filtering is critical because the mosaic is far larger than any single HA frame.

Key challenge: SuperGlue fails on group 3 due to 180-degree rotation and LWIR's low contrast. The solution was pre-warping HA images with the initial guess before running SuperGlue, which gave the matcher a fighting chance (commit 380cb35: "superglue is no longer failing now that we are prewarping the images with the initial guess").

RAFT (src/methods/raft.py)

Optical flow method used as both a fallback and a refinement stage. Two variants:

• Downsampled RAFT -- Resize both images and compute dense flow
• Tiled RAFT (scripts/refine_with_raft_tiled.py) -- Divide the overlap region into tiles, compute flow per tile, fit a homography from the tile displacements. This was found to be slightly better than downsampled RAFT (commit 6884a4c: "a tiled raft approach seems to work slightly better than a downsampled raft approach").

RAFT was the breakthrough for group 3 (commit 6bad62c: "RAFT is working!!!").

Deep Homography (src/methods/deep_homography.py)

Neural homography regression. Implemented but less prominent in the actual processing pipeline than SuperGlue and RAFT.

Other Methods

Also implemented: LoFTR (src/methods/loftr.py), UDHN (src/methods/udhn.py), DGC-Net (src/methods/dgc.py), CoTR (src/methods/cotr.py). These appear to be comparison/evaluation implementations rather than production methods.

Source: lwir-align/src/methods/, git log

Metrics

Defined in src/metrics/alignment_metrics.py:

Metric	Implementation	Purpose
NMI (Normalized Mutual Information)	Joint histogram with 32 bins, entropy-based	Robust to intensity differences between HA and LA
NCC (Normalized Cross Correlation)	Zero-mean, normalized dot product	Measures linear relationship between pixel intensities
SSIM (Structural Similarity Index)	Via skimage.metrics.structural_similarity	Perceptual quality of alignment
L1 (Normalized L1 Loss)	Mean absolute difference, normalized	Simple pixel-level error
Combined	Weighted combination	Composite score for ranking

All metrics support optional validity masks to exclude black/invalid regions from the computation. The tiled RAFT refinement improved SSIM by over 1.2% compared to standard SuperGlue refinement alone.

Source: lwir-align/src/metrics/alignment_metrics.py

Group-Based Processing Structure

The 675 HA frames are organized into 8 temporal groups based on flight path segments. Each group has its own subdirectory under flight_21052/filtered_enhanced_undistorted/:

group{N}/
  initial_guess_to_mosaic/    # Coarse homographies from anchor composition
  refined_to_mosaic/          # SuperGlue-refined homographies
  refined_to_mosaic_raft_tiled/  # RAFT-refined (best quality)

The master orchestration script (scripts/master_process_all_groups.py) automates the multi-stage pipeline:

1. Detect missing files/directories
2. Generate initial guess homographies where needed
3. Pass 1: Refine to best mosaic (SuperGlue)
4. Pass 2: Cross-mosaic registration (dual coverage)
5. Generate analysis and visualization

Group 3 required special handling -- it had a 180-degree rotation relative to the other groups, causing SuperGlue to fail entirely. The solution involved manual corrections (group3/*_corrected.json) and RAFT-based recovery (new_registration_plan/group3_tiled_raft_recovery_plan.md).

Source: lwir-align/scripts/master_process_all_groups.py, lwir-align/CLAUDE.md

Results Across Groups 1-8

Group	Frames	Registration Method	Notes
1	85	SuperGlue + RAFT tiled	Standard processing
2	144	SuperGlue + RAFT tiled	Largest group
3	103	Manual anchor + RAFT	180-degree rotation; SuperGlue failed; required manual corrections
4	96	SuperGlue + RAFT tiled	Standard processing
5	65	SuperGlue + RAFT tiled	Standard processing
6	4	SuperGlue	Very small group
7	11	SuperGlue	Small group
8	22	SuperGlue	Group 8 also had SuperGlue difficulties (commit b37bffd: "group 8 refuses to register with superglue")

Coverage analysis (scripts/analyze_coverage.py) found that with a minimum of 8 overlapping frames per patch, 294 viable training tiles could be generated.

Source: lwir-align/new_registration_plan/coverage_analysis_and_next_steps.md, git log

Tile Metadata Generation

The script scripts/generate_comprehensive_tile_metadata.py is the critical bridge to downstream tools. It:

1. Generates a regular tile grid over each mosaic (384x384 or 192x192 pixels, with 32-pixel overlap)
2. Uses Shapely spatial queries (STRtree) to find which HA frame footprints overlap each tile
3. Produces 4 metadata JSON files: tile_metadata_mosaic{A|B}_{384x384|192x192}.json
4. Each tile record includes: tile ID, position, coverage ratios, list of contributing HA images with their homography paths

These metadata files are consumed directly by LWIR Tile Validator for visual QC and by the ProbaV dataset exporter for training data generation.

Source: lwir-align/scripts/generate_comprehensive_tile_metadata.py

Relationship to Downstream Tools

To LWIR Tile Validator

The tile validator reads lwir-align's output as read-only data: - Mosaic GeoTIFFs from flight_21051/ - HA PNGs from flight_21052/filtered_enhanced_undistorted/group{1-8}/ - Comprehensive tile metadata from comprehensive_tile_metadata/ - Homography matrices from group{N}/refined_to_mosaic/

To PIUnet Training Data

The MFSR dataset creation plan (new_registration_plan/mfsr_dataset_creation_plan.md) describes a three-phase workflow: 1. Geometric indexing -- Generate footprints.json from all refined homographies 2. Tiling strategy -- Determine optimal tile placement based on coverage density 3. Patch extraction -- Extract ProbaV-format training data (HR mosaic patch + multiple LR HA patches per tile)

The recommended refinement chain for best quality: Initial Guess -> SuperGlue Refinement -> Tiled RAFT Refinement.

Interactive Registration Tool

When automated methods failed (especially group 3), a custom OpenGL-based registration tool was built (registration_tool/) to replace GIMP for manual matrix adjustment (commit 6b5d0b8: "started work on this registration tool with openGL support because the transforms I was getting from gimp were not right"). This tool later evolved into the Hephaestus project.

Struggle Log

SuperGlue Fails on 180-Degree Rotation (Group 3)

• Hypothesis: SuperGlue's learned matching would handle arbitrary orientations in LWIR
• Failure Mode: Zero or near-zero valid matches for all group 3 frames
• Root Cause: LWIR imagery lacks the rich texture that SuperGlue was trained on; combined with 180-degree rotation, the matcher had no signal
• Resolution: Pre-warp with initial guess + fall back to RAFT for dense flow
• Anti-Pattern: Do not assume learned feature matchers will generalize to extreme rotations in low-contrast thermal imagery

GIMP Transforms Were Inaccurate

• Hypothesis: GIMP's interactive warping could provide usable homography matrices
• Failure Mode: The transforms from GIMP were not geometrically correct for the pipeline
• Root Cause: GIMP's internal transform representation doesn't map cleanly to a standard 3x3 homography
• Resolution: Built custom OpenGL registration tool with direct matrix control
• Anti-Pattern: Do not use general-purpose image editors for precision geometric registration

Key Files

Path	Purpose
src/methods/base.py	Abstract registration interface
src/methods/superglue.py	SuperGlue implementation
src/methods/raft.py	RAFT optical flow implementation
src/metrics/alignment_metrics.py	NMI, NCC, SSIM, L1 metrics
scripts/master_process_all_groups.py	Master orchestration script
scripts/refine_group_to_mosaic.py	SuperGlue refinement with spatial filtering
scripts/refine_with_raft_tiled.py	Tiled RAFT refinement (best quality)
scripts/generate_comprehensive_tile_metadata.py	Tile metadata for downstream tools
scripts/generate_footprints.py	Frame footprint index
scripts/analyze_coverage.py	Coverage density analysis
project_paths.py	Centralized path configuration
new_registration_plan/mfsr_dataset_creation_plan.md	Dataset generation plan

Environment

• Conda environment: superglue-env (primary), geo_env (geospatial), lwir-registration (GUI tool)
• Last active: October 2025
• Location: /home/geoff/projects/ceres/superrez/lwir-align/