Overview
The Semi-Field Image Processing Pipeline efficiently transforms raw images from our semi-field setups into structured outputs, facilitating plant species identification and morphological analysis. This pipeline automates the data transformation process, enabling high-throughput analysis for plant phenotyping research.
Here's a detailed table that summarizes the Semi-Field Image Processing Pipeline, including the processes, inputs, outputs, and additional columns for tools/software used and key objectives:
Stage | Key Objectives | Processes | Inputs | Outputs | Tools/Software Used |
|---|---|---|---|---|---|
Image Preprocessing | Ensure color consistency and prepare images for analysis | Color calibration, Image conversion | Raw images, Color Calibration Card | Calibrated images (jpg) | RawTherapee |
Potting Area Reconstruction (SfM) | Create a global coordinate reference system for spatial analysis | Orthomosaic creation | Preprocessed images | Metashape project files, camera reference information | Agisoft/Metashape |
Plant Detection | Accurately detect plant locations | individual image detection | Images, YOLOv8 detection model | single class (“plant”) detection results | YOLOv8 |
Remap Plant Detection | Transform bounding box coordinates from pixel representations to real-world global coordinates. | Bounding box coordinate transformation | Images, “plant” detection results | projected bounding box coordinates, external camera parameters | Agisoft/Metashape |
Species Assignment | assign species labels | Species assignment | Shapefiles | QGIS and other GIS tools | |
Vegetation Segmentation and Cutouts | Extract individual plant regions and classify species | Digital image processing, Semantic & Instance segmentation, Cutout extraction | Annotated images | Semantic masks, Instance masks, Plant cutouts | Image Processing Tools |
Review and Data Compilation | Ensure accuracy and integrity of processed data for analysis | Quality assurance, Error correction, Data aggregation | Processed images, Metadata | Verified image batches, Compiled datasets | Manual Review, Data Management Tools |
Local plant detection results are remapped to global potting area coordinates.
Pipeline Stages
1. Image Preprocessing
Inputs:
Raw images from the BenchBot imaging system.
61 MP, Sony Alpha 7R IV, 55mm lens (9504 x 6336)
~8.3 pixels/mm at ~2 meters above ground
Processes:
Color Calibration: Using RawTherapee software and a color checker card to ensure color consistency across images. Raw images are corrected and converted to high-resolution JPEGs.
Image Conversion: Converts raw image formats to standard formats suitable for analysis.
Mask Generation: Creates binary masks to delineate areas of interest and exclude background noise.
Outputs:
Calibrated JPEG images.
Preprocessed images ready for further analysis.
Binary masks for segmentation.
Detection results in normalized xyxy format (CSV).
2. Potting Area Reconstruction (SfM)
Inputs:
Preprocessed images.
Camera reference information (CSV).
Processes:
3D Coordinate Mapping: Generates a global coordinate reference system (CRS) using developed images and ground control point metadata.
Orthomosaic Creation: Stitches images to create a detailed orthomosaic and camera reference data.
Outputs:
Metashape project files (.psx) for projecting image coordinates to 3D coordinates.
Detailed metadata with camera information and detection results (JSON).
3. Plant Localization and Species Assignment
Inputs:
Preprocessed images and binary masks.
Shapefiles delineating species potting groups.
Processes:
General Plant Detection: Uses a general “plant” detection model, such as YOLOv5, to localize target plants in each image and extract bounding boxes.
Coordinate Transformation: Remaps local plant detection results to global potting area coordinates.
Species Assignment: Assigns species labels to bounding boxes based on overlaps with species potting group boundaries in shapefiles.
Outputs:
Detection files with coordinates of each plant (CSV format).
Annotated images with bounding boxes.
Updated metadata with assigned species labels for each bounding box.
4. Vegetation Segmentation and Cutout Generation
Inputs:
Annotated images with bounding boxes.
Processes:
Digital Image Processing: Uses techniques like index thresholding, unsupervised classification, and morphological operations to separate vegetation from the background within bounding boxes.
Semantic Segmentation: Classifies each pixel to generate semantic masks, identifying plant species and other features.
Instance Segmentation: Differentiates individual plants within clusters to create instance masks.
Cutout Extraction: Crops individual plants from the full image based on instance masks.
Metadata Annotation: Attaches metadata to each cutout, including species information and morphological properties.
Outputs:
Semantic masks for species classification.
Instance masks for individual plant identification.
Individual plant cutouts with black backgrounds.
Metadata files containing species and morphological information for each cutout.
5. Review and Data Compilation
Inputs:
Processed batches of images and metadata.
Processes:
Quality Assurance: Experts visually examine a random set of 50 images per batch to verify species detection and segmentation accuracy.
Error Correction: Log files are reviewed for discrepancies, and necessary corrections are made.
Data Aggregation: Compiles metadata into structured formats for analysis and storage.
Quality Control: Verifies the accuracy of segmentations and annotations.
Outputs:
Verified image batches ready for storage and further analysis.
Compiled datasets ready for research and analysis.
Quality reports highlighting any anomalies or issues.
Backed-up data products on NCSU storage facilities.
Summary
The Semi-Field Image Processing Pipeline is a comprehensive system that automates the transformation of raw imaging data into detailed, structured outputs. By integrating advanced detection and segmentation technologies, the pipeline supports high-throughput analysis, enabling researchers to gain insights into plant growth and development efficiently.
0 Comments