MFSR Mosaic¶
MFSR Mosaic is the georeferencing, tiling, and HA/LA patch pair extraction pipeline. Its goal is to produce precisely aligned High-Altitude (HA) to Low-Altitude (LA) image patch pairs for training and evaluating PIUnet and other MFSR models.
Repository: /home/geoff/projects/ceres/superrez/mfsr_mosaic/
Active: May 2025 -- Jun 2025 (23 commits)
Conda env: geo_env (Python 3.10, GDAL, rasterio, geopandas)
What It Does¶
- Georeferencing: Convert raw HA PNG images (flight 21052) into georeferenced GeoTIFFs using INS telemetry or manual KML ground overlays
- Tiling: Tile LA mosaics (flight 21051) into patches at optimal rotation angles for maximum valid-pixel coverage
- Patch Pair Extraction: (planned) Match HA images to LA mosaic tiles via SuperGlue, extract aligned HA/LA patch pairs at 384x384 (LA) and 128x128 (HA, Proba-V scale)
The Calibration Journey¶
Getting HA images to land in the right geographic location turned out to be a multi-stage calibration problem. Each approach improved accuracy but revealed the next limitation.
Stage 1: Raw INS Georeferencing¶
Script: claude_create_ha_geotiffs.py
Used INS telemetry (roll, pitch, yaw, lat, lon, altitude) from captures.log plus camera FOV parameters to project image footprints onto the ground and create GeoTIFFs with GCPs.
Problem: Images were rotated far from their true orientation. The INS yaw reading did not correspond to the camera's actual heading due to an unknown boresight offset between the IMU and camera.
Stage 2: Yaw Offset Calibration¶
Scripts: claude_create_ha_geotiffs.py, analyze_yaw_offsets.py
Iterative process comparing INS-based footprints against manually determined orientations. Converged to a constant yaw offset:
This dramatically improved rotational alignment -- images now pointed in approximately the correct direction.
(Source: mfsr_mosaic/handoff.md, lines 46--49)
Stage 3: Manual KML Registration in Google Earth¶
10 HA images were manually aligned in Google Earth Pro by dragging their corners to match visible ground features, producing *_aligned.kml files. These served as ground truth for further calibration attempts.
Stage 4: Full 6-DOF Boresight Calibration (failed)¶
Script: claude_full-calibration-solver.py
Attempted to solve for the full 6-DOF boresight calibration (residual rotation and position offset) between IMU and camera using the 10 manually aligned KMLs as ground truth.
- Hypothesis: With 10 ground-truth KML alignments, a least-squares solver could recover the full IMU-to-camera transform.
- Failure Mode: The script ran and produced parameters (saved to
calibration_output/imu_camera_calibration.json), but the transformation was never conclusively validated as correct. - Root Cause: The core transformation logic -- comparing initial KML guesses to manually fixed versions to derive residuals -- did not reach a satisfactory working state. The problem may have been underdetermined or sensitive to the accuracy of the manual KML alignments.
- Anti-Pattern: Do not attempt a full 6-DOF boresight solve from a small number of manually placed ground truth points without rigorous validation infrastructure (e.g., leave-one-out cross-validation, reprojection error metrics).
- Resolution: Abandoned this approach; moved failed scripts to their own folder. The working approach became KML-based georeferencing via
convert_kml_to_geotiff.py.
(Source: mfsr_mosaic/handoff.md, lines 36--41, 53--58)
Stage 5: KML-to-GeoTIFF Conversion (working solution)¶
Script: convert_kml_to_geotiff.py
Parses KML files with <GroundOverlay> / gx:LatLonQuad elements. Uses GDAL to:
1. Create an in-memory VRT with 4 corner GCPs
2. Warp with tps=True (thin plate spline) and dstAlpha=True for transparency
3. Output a perspectively warped, georeferenced GeoTIFF
This was a key breakthrough. Output GeoTIFFs align correctly with KML outlines in QGIS/Google Earth. The main challenge was getting GDAL to do perspective warping instead of affine warping.
(Source: mfsr_mosaic/handoff.md, lines 59--64; mfsr_mosaic/convert_kml_to_geotiff.py, lines 11--27)
Stage 6: SuperGlue Alignment (planned, not started)¶
The next planned step is to use SuperGlue for precise HA-to-LA mosaic alignment: 1. Use georeferenced HA footprints (from KML or INS+calibrated yaw) to define an ROI in the LA mosaic 2. Extract the corresponding LA mosaic region 3. Run SuperPoint+SuperGlue matching between the HA source PNG and the LA ROI 4. Compute homography for each HA image relative to the LA mosaic 5. Extract aligned patch pairs
(Source: mfsr_mosaic/handoff.md, lines 78--93; mfsr_mosaic/monday_pickup.md, lines 37--57)
Key Findings¶
LA-HA Mosaic Misalignment¶
The LA mosaics (...Ranch A Jenoptik.tif and ...Ranch B Jenoptik.tif from flight 21051) are not well-aligned with the HA mosaic (...texturedv3_005.tif from flight 21052) despite covering the same ground. This was observed visually in QGIS and is critical for any feature matching or data fusion between mosaic products.
(Source: mfsr_mosaic/dataset_learnings.md, lines 35--37)
Altitude Confirmation from Telemetry¶
Analysis of captures.log telemetry filtered to the Carinalli Ranch area confirmed:
| Area | Flight 21051 (LA) | Flight 21052 (HA) | Difference |
|---|---|---|---|
| Ranch A | 796.4m ASL median | 1217.3m ASL median | ~421m |
| Ranch B | 798.7m ASL median | 1218.8m ASL median | ~420m |
This validated the HA/LA distinction from actual telemetry rather than flight plan altitudes (which were counterintuitively close: planned 2600 vs 2500 in unspecified units).
(Source: mfsr_mosaic/dataset_learnings.md, lines 39--52)
Tiling Parameter Confusion¶
Tiling exploration runs produced inconsistent results (438 vs 154 patches) due to confusion about how --patch_size interacts with downscaled mosaic inputs. The effective full-resolution patch size (EFR_PS) depends on the downscaling factor, which was not clearly documented in the scripts.
Anti-Pattern: When tiling scripts accept a patch size parameter and the input image may be downscaled, the script must either (a) accept the desired full-resolution patch size and the downscale factor as separate explicit parameters, or (b) prominently output the calculated effective full-resolution patch size.
(Source: mfsr_mosaic/monday_pickup.md, lines 91--98)
Dataset Details¶
Flight 21052 (High Altitude)¶
- Raw images: 16-bit PNGs at
/home/geoff/projects/flight_21052/Raw/ - Captures log:
/home/geoff/projects/flight_21052/Raw/captures.log(ground truth for attitude/position) - Filtered captures: 4701 records in
filtered_ha_captures_by_geojson.csv - Dev subset: 20 images near HA mosaic center in
dev_subset_ha_captures.csv - Quality filter:
flight_21052/21052_better_images.csv(human-curated, blurry images excluded)
Flight 21051 (Low Altitude)¶
- Mosaic A: 5446x5454 pixels,
...Carinalli_Ranch A Jenoptik.tif - Mosaic B: 4832x4979 pixels,
...Carinalli_Ranch B Jenoptik.tif - Both in EPSG:4326 (WGS84)
Coverage Requirements (for MFSR)¶
- Each LA tile needs >= 8 qualifying HA images
- Each HA footprint must cover >= 80% of the LA tile area
- Target LA tile size: 384x384 pixels
- Target HA patch size: 128x128 pixels (Proba-V scale, 3x downsampling)
(Source: mfsr_mosaic/monday_pickup.md, lines 26--32)
Script Inventory¶
| Script | Purpose | Status |
|---|---|---|
convert_kml_to_geotiff.py |
KML GroundOverlay to perspective-warped GeoTIFF | Working, key tool |
claude_create_ha_geotiffs.py |
INS-based georeferencing with yaw offset | Working (GCPs only, no pixel warp) |
analyze_yaw_offsets.py |
Compare INS yaw vs manual target orientations | Working, calibration tool |
analyze_farm_area_altitudes.py |
Altitude analysis filtered to farm area | Working, analysis tool |
filter_captures_by_geojson.py |
Filter captures.log by GeoJSON footprints | Working |
create_dev_subset.py |
Select 20 captures nearest mosaic center | Working |
generate_calibration_kmls.py |
Generate KMLs from full calibration | Has linter errors |
claude_full-calibration-solver.py |
6-DOF boresight calibration | Failed, abandoned |
scripts/run_tiling_exploration.py |
LA mosaic tiling parameter sweep | Working but confusing parameters |
Current State¶
What works:
- convert_kml_to_geotiff.py reliably produces perspective-warped GeoTIFFs from KML ground overlays
- Yaw offset calibration (272.4830 deg) gets INS-based footprints rotationally close
- Dataset filtering, subsetting, and altitude analysis scripts are functional
- Tiling exploration can sweep rotation angles and anchor corners
What is next:
1. Set up SuperGlue environment (superglue_env with PyTorch + CUDA)
2. Implement HA-to-LA mosaic alignment using georeferenced footprints for ROI selection
3. Extract aligned patch pairs (384x384 LA, 128x128 HA) for MFSR training
4. Integrate with Metashape integration plans for improved ground truth
What was abandoned: - Full 6-DOF boresight calibration from manual KML ground truth (inconclusive results)
Relationship to Other Pipeline Tools¶
- Produces georeferenced HA images and tiled LA patches that feed into Hephaestus for interactive alignment verification
- Aligned patch pairs are the training data for PIUnet MFSR network
- SuperGlue alignment step connects to the same SuperGlue infrastructure used in Hephaestus Phase 1
- LA-HA mosaic misalignment finding motivates the Metashape integration approach for co-registered mosaics
- Quality-filtered image list (
21052_better_images.csv) informed by Aerofocus blur detection