Software User Guide
- Wesley Moss
Instructions of how to use the GUI python scripts to control the system.
The main script is designed to run on NVIDIA Jetson and performs realtime plant detection and segmentation.
The colour detection script uses an OpenCV colour mask to target objects based on colour. This can be used for testing and calibration of the system’s accuracy onto printed squares. This script can be run on a PC as it does not require much processing power.
Main - YOLO plant detection
This script uses YOLO to identify plants for targeting. Currently it differentiates based on monocot and dicot, which can be toggled in the GUI to choose to target either monocots or dicots.
System overview
Plants are detected and classified as either dicot or monocot. If the system is targeting dicots (as in the example above) they will be marked as green and will be assigned for firing. When the system is in segmentation mode it will create a grid of possible firing positions (based on the horizontal positions of the valves and the “Firing Pixel Grid Height Value”) and assign firing positions if they overlap where the segmentation model has identified plants. Each green dot will be fired at by the system.
NOTE: The targeting system relies on accurate calibration of “vehicle velocity”, “row size” and “spray centre offset”. If the velocity of physical position of the camera/valves changes, these must be updated.
GUI controls explanation
Sliders
Vehicle Velocity - Defines velocity of vehicle in m/s. This must be accurately set or the system will spray too early or too late.
Vehicle Offset- Can be used to fine tune vehicle velocity value used by system.
Horizontal line offset - Moves the horizontal line up and down in the frame. This frame is the centre of the buffer area. Only objects in the frame’s buffer zone are targeted for spraying.
Buffer - Adjusts the size of the buffer area. Only objects in the frame’s buffer zone are targeted for spraying. The buffer zone should be set according the FPS and velocity to make sure there are no dead zones as the vehicle moves forward.
Row Size - Size of the rows corresponding to where each valve will spray in the frame.
Spray centre offset - Adjusts where the centre of the rows are in the frame. Used to compensate for errors between the centre of the frame and the centre of the spray head.
Fire Delay - Delay in ms of when to fire the valve compared to calculated time.
Pix2meter Offset - Adjusts the Pix2meetr value, which is the physical size each vertical pixel corresponds to in the frame. Pix2meter is hardcoded in calibration and this slider is used to make small adjustments as needed.
Frame Rate - Maximum frame rate of the system in FPS. Note this is the maximum FPS and it may actually be lower than the slider as the FPS is limited by the camera and the system.
Firing Pixel Grid Height - Sets the vertical spacing of the firing pixel grid when in segmentation mode. Note: “Rebuild Pix Array” must be selected if this value is changed
Square size, Square Spacing and Square Pattern - Controls the size of a square fired instead of a single dot. This feature is still in development for the YOLO model targeting. Do not use.
Buttons
Hide live cam feed / show live cam feed - Displays or hides the camera feed. Note: performance is slightly faster with the feed hidden. Fire - Turns on/off whether the system will fire the valves if it sees an object. This must be turned on to spray.
PLC Check - Checks if the PLC is connected. Output is in the python terminal, not the GUI
Set Valve Time - Changes the valve open time in the PLC. Cycles through values: 5ms, 10ms, 20ms, 30ms. Current value is displayed in top right of GUI. PLC must be connected for this to work.
Test Shot - Fires all valves
Toggle Grid - Displays/hides the grid lines in the video image
Record on/off - Saves the frame of the video window everytime there is a firing event and “Fire” is on. This saves in the /Recorder folder. The video feed does not need to be displayed for this to work.
BBox/Segment - Toggles whether the system is targeting plants based on segmentation or the centre of the bounding box.
Class Label - Toggles display of the class label of each identified object (must be in BBox mode) and prints the YOLO verbose in the terminal.
Rebuild Pix Array - Rebuilds the grid of pixels targeted. Must be selected if “Firing Pixel Grid Height” is changed.
Quit - Closes the program. Note: if the camera is not connected it may not completely end the process even if the GUI closes.
Non-GUI Controls
There are several settings relating to the camera (e.g. flash on/off, exposure time, gamma) and phsycial setup (e..g distance between camera and valves) that are not changed in the GUI. These must be changed in the code itself before running the script.
Colour Detection Program
System overview
This script uses openCV to identify objects based on their colour. Click on an object in the image to set its colour to be targeted. Use sliders to adjust colour mask as needed. Centre of the detected objected will be targeted.
GUI controls explanation
Sliders
H, S, V -Controls the threshold for the HSV values used in the colour mask above/below the value set by clicking on a colour.
Square size - Number of rows and columns to fire at detected objected (only if “square pattern” is selected)
Square Spacing- Vertical spacing in pixels of square (only if “square pattern” is selected)
Vehicle Velocity - Defines velocity of vehicle in m/s. This must be accurately set or the system will spray too early or too late.
Vehicle Offset- Can be used to fine tune vehicle velocity value used by system.
Horizontal line offset - Moves the horizontal line up and down in the frame. This frame is the centre of the buffer area. Only objects in the frame’s buffer zone are targeted for spraying.
Buffer - Adjusts the size of the buffer area. Only objects in the frame’s buffer zone are targeted for spraying. The buffer zone should be set according the FPS and velocity to make sure there are no dead zones as the vehicle moves forward.
Detected Area Upper/Lower Thres - Defines and lower and upper threshold for the size of the objects that are targeted, Can be used to filter out smaller or larger objects mistakenly detected by the colour mask.
Row Size - Size of the rows corresponding to where each valve will spray in the frame.
Spray centre offset - Adjusts where the centre of the rows are in the frame. Used to compensate for errors between the centre of the frame and the centre of the spray head.
Fire Delay - Delay in ms of when to fire the valve compared to calculated time.
Pix2meter Offset - Adjusts the Pix2meetr value, which is the physical size each vertical pixel corresponds to in the frame. Pix2meter is hardcoded in calibration and this slider is used to make small adjustments as needed.
Frame Rate - Maximum frame rate of the system in FPS. Note this is the maximum FPS and it may actually be lower than the slider as the FPS is limited by the camera and the system.
Buttons
Hide live cam feed / show live cam feed - Displays or hides the camera feed. Note: performance is slightly faster with the feed hidden.
Square Pattern/Single Shot - Toggles between shooting a single shot at each target and a square pattern
Fire - Turns on/off whether the system will fire the valves if it sees an object. This must be turned on to spray.
PLC Check - Checks if the PLC is connected. Output is in the python terminal, not the GUI
Set Valve Time - Changes the valve open time in the PLC. Cycles through values: 5ms, 10ms, 20ms, 30ms. Current value is displayed in top right of GUI. PLC must be connected for this to work.
Show Colour Filter - Displays binary colour mask used for object detection. This window can be used to fine tune the colour mask settings.
Test Shot - Fires all valves
Toggle Grid - Displays/hides the grid lines in the video image
Record on/off - Saves the frame of the video window everytime there is a firing event and “Fire” is on. This saves in the /Recorder folder. The video feed does not need to be displayed for this to work.
Quit - Closes the program. Note: if the camera is not connected it may not completely end the process even if the GUI closes.
Non-GUI Controls
There are several settings relating to the camera (e.g. flash on/off, exposure time, gamma) and phsycial setup (e..g distance between camera and valves) that are not changed in the GUI. These must be changed in the code itself before running the script.