| Table of Contents | |||
|---|---|---|---|
|
...
Upload, running the pipeline, and storage
Batch Upload
Benchbot operators manually upload batches to “upload” blob container.
Upload batches include images and metadata from a single location and capture period. Batches are named after location ID and date, for example, NC_2022-03-22.
Benchbot operators will manually upload (by dragging and dropping) batch folder into the “Uploads” blob container using Azure Storage Explorer.
Metadata includes:
Ground control point location (csv) measured in meters.
Species Map (csv) that lists species for each row.
Running the pipeline and storage
Blob containers are mounted to VM.
A cron job is scheduled to start the pipeline every N hours.
To avoid processing the same batch multiple times, new batches are selected using a batch log file.
The pipeline runs and stores data to temporary locations until complete.
While blob container do not technically use directories or “folders”, this project uses a pseudo-directory structure for organizing the multiple data products that are being created and updated. The temporary storage locations replicate the pseudo-directory structure used in blob containers.
After the pipeline finishes, the processed batch is logged to the batch log file.
Lastly, temporary batch data is moved permanently to the appropriate blob containers.
AutoSfM runs in a docker container
All code is written in Python 3.9
...
Pipeline
The pipeline includes seven main processes and places data in five temporary storage locations
...
Preprocessing
Raw images are preprocessed using color calibration card.
Mapping and Detection
AutoSfM
...
|
Data Flow
Preprocessing
AutoSfM
The AutoSfM process takes in developed images and ground control point metadata to create a global coordinate reference system (CRS). An orthomosaic, or collage of stitched images, and detailed camera reference information is generated, the latter being used to convert local image coordinates into global potting area locations.
For example, an image 2000 pixels high and 4000 pixels wide has a local center point at (1000, 2000), half its height and half its width, measured in pixels. Camera reference information allows us to project this local center point to a geographical potting area location in meters, (1.23m, 4.56m) for example. Knowing the general location of species pot groups, we assign species labels to each general “vegetation” detection results. We relate the global potting area locations to local image bounding boxes coordinates allowing to us to fill in the missing species label.
Inputs:
Developed images
Masks
Ground control point information (.csv)
Outputs:
camera matrix and orientation information for each image (.csv)
fov for each image (.csv)
and processing accuracy assessment (.csv)
orthomosaic of potting area (.tif)
Detection
Object detection is performed to identify plant locations and create local bounding box coordinates. More detailed pixel wise segmentation is performed with the bounding box areas.
Model - YOLOv5
Single class - “plant”
Trained and tested on 753 images captured and labeled for the 2021 OpenCV AI competition.
mAP_0.5 = 0.93, mAP_0.5:0.95 = 0.67, recall = 0.9, precision = 0.93
Inputs:
Developed images
Trained model
Outputs:
Local detection results for each image. Detection results are in normalized xyxy format (.csv).
| Gallery | ||||||
|---|---|---|---|---|---|---|
|
Detection results from 2022-03-11
Metashape (psx) project for projecting image coordinates to real-world 3D coordinates
Remap
Local plant detection results are remapped to global potting area coordinates.
Inputs:
Images
camera reference information (.csv)
Outputs:
detailed metadata with camera information and detection results (.json)
...
WHY?
Image annotations need specie-level information, but the detection model only provides the location of “plants”. We also have many overlapping images resulting in duplicate segments which can lead to an imbalanced or homogeneous dataset.
Species mapping: We can infer the species of each detection result with a user-defined species map and geospatial data. If we know what row or general geographic area each species is located, we can label each bounding box appropriately.
...
Unique detection result: The benchbot is taking 6 images along a single row of 4 pots. These images overlap considerably and the same plant is often detected, and thus segmented, multiple times at different angles. While multiple angles are good, its important to identify the unique, or primary detection result (when the camera is directly over the plants). Doing so allows us to:
maximize synthetic image diversity and avoid using the same plant segment (albeit at slightly different angles) multiple times, which could lead to homogeneous data and thus poor model performance.
Identify unique plant/pot position throughout their growth stages leading to detailed phenotypic profiles
Monitoring: Monitor for inconsistencies and error in image capture across sites using detailed reporting of camera reference information
Segment Vegetation and Cutout Data
...
Assign Species
At the start of each "season," shapefiles are generated to delineate the boundaries for different species potting groups. These shapefiles are then used to assign specific species labels to individual bounding boxes, based on their intersection with the shapefile. If there is an overlap between a bounding box global coordinates and the designated shapefile features—illustrated as rectangular features below—the label of the overlapping shapefile feature is then attributed to the respective bounding box. This assignment process follows the "Remapping" phase, where bounding box coordinates were transformed from pixel representations to real-world global coordinates.
Vegetation Segmentation
Plant Cutout Generation
Digital image processing techniques like index thresholding, unsupervised classification, and morphological operations are used to separate vegetation from the background
...
Crop image to bbox
Vegetation index
Excess green VI
...
within pre-identified bounding boxes. The specific segmentation approach might vary depending on the plant species and the size of the bounding box. These extracted plant regions, or "cutouts," serve as building blocks for creating synthetic data.
Sub-Image Data (Cutouts): Details and Metadata
Cutouts are cropped sections from full-sized images, each containing a single plant instance. Each cutout's metadata includes:
Parent image ID
Unique cutout ID and number
Species classification
Primary status
Exceeding cropped image border status
Camera position
Bounding box from which the cutout originated within the full-sized image
Specific cutout properties like area, perimeter, and color statistics, which are valuable for in-depth analysis.
Furthermore, the metadata inherits EXIF data from the parent image and incorporates additional details:
Review
For each batch, a weed scientist or agronomic expert selects a random set of 50 images for visual examination. During this process, they review the species detection outcomes, along with the semantic and instance masks. Should any discrepancies arise, the batch's log files are scrutinized for any errors or anomalies. A batch that successfully clears the inspection stage, free of mislabeling or significant errors, proceeds to the next step. Upon approval, the full-sized images and their corresponding sub-images are transferred to the semif-developed and semif-cutouts blob containers in Azure. Additionally, all data products are securely backed up to the NCSU storage facilities, ensuring data integrity and availability.