/
Working and Annotating in CVAT

Working and Annotating in CVAT

Owned by Leafia Sheraden Cox
Last updated: Mar 04, 2025
10 min read

This page assumes you have already uploaded data to CVAT and have created a project

A physical mouse is recommended for annotating in CVAT.

There is no known way to operate the annotation interface with a keyboard alone and trackpads have been found to be inefficient and difficult to use with the interface.

The Annotation Interface

Screenshot 2025-02-28 133112-20250228-183806.png
The CVAT Annotation Interface. A more detailed breakdown of the annotation interface can be found in the CVAT manual.

Navigation Bar & Menu

The navigation panel and drop-down menu, allow you to switch between frames, change the annotation mode, save your work, and more.

 

The CVAT interface arranges the different images in a dataset as different “frames” with each frame corresponding to a single image.

 

You will need to annotate each frame individually.

 

For a more detailed explanation of the navigation bar and menu, see the official CVAT documentation

image-20250228-192859.png
The Navigation bar and Menu
image-20250228-192308.png
An overview of how to navigate through frames within the interface. Image taken from CVAT manual

Controls Sidebar

Offers tools for navigating within the image, annotation tools, and additional options to merge, split, and group labels.

 

Note that some options are hidden in the controls sidebar and can be accessed using the bottom button (.....)

 

The primary tools that you’ll be using in the controls sidebar are the Polygon tool (⬠) and the AI tools (🪄).

 

For a more detailed explanation of the controls sidebar, see the official CVAT documentation.

image-20250228-193718.png
The controls sidebar.

Objects Sidebar

Displays annotated objects and includes a label filter, lists of objects (current frame) and labels (objects on the frame), and appearance settings.

 

The primary thing that we are concerned with are objects and their labels.

 

For a more detailed explanation of the objects sidebar, see the official CVAT documentation.

image-20250228-195710.png
The objects sidebar.

Workspace

The main annotation area where images and videos are displayed for annotation.

 

Some workspace settings can also be found in the general account settings. See the Settings section in CVAT Basics .

 

For a more detailed explanation of the workspace, see the official CVAT documentation.

image-20250228-200315.png
The workspace.

Labels

In order to properly annotate, you must ensure that you have the correct labels set up. These labels differ from dataset to dataset and are stored in the label_colors.txt file within dataset’s zip file. When you create a project, the labels are automatically read in from the file.

In our case, we typically have two labels: background, and <species name>

The <species name> label is what we will use to identify and annotate the particular cover crop species present in an image.

If there are multiple cover crop species present in a single image, each should have its own label titled <species name>. For example, if both hairy vetch and winter pea are both present in a single image there should be two different labels, once called hairy_vetch and another called winter_pea.

The background label is automatically created by preprocessing and it represents everything that is not our plant of interest.

image-20250303-155337.png
An object with the <species name> label. In this case, winter_pea
image-20250303-155431.png
An object with the background label

Dealing with the Background class

For each frame, you’ll need to deal with the label class background that is created by default.

We are only concerned with labeling the plants in the image and therefore do not need the “background” label. Hence you will need to delete the extraneous label.

There are two methods of removing objects: using the objects sidebar and using keyboard shortcuts.

Using the Objects sidebar

  1. In the objects sidebar, identify the objects labeled background. There may be multiple objects with the same label. You will need to remove each object labeled background.

  2. Click the icon of three vertical dots (⋮) next to the object’s label.

  3. Click on Remove in the dropdown that appears.

  4. Repeat for each object labeled background

  5. If you accidentally remove an object that you intended to keep, click the Undo (↶) icon in the Navigation bar .

  6. Similarly, if you accidentally undid something that you did not intend to, click the Redo (↷) icon in the Navigation bar.

There is a limit to the number of successive actions that you can undo (↶) or redo (↷) using the Navigation bar and keyboard shortcuts.

image-20250303-154914.png
1. Identify the objects labeled background
image-20250303-154926.png
2. Click the ⋮ icon
image-20250303-155035.png
3. Click on Remove
image-20250303-155157.png
3. The background class object is now deleted. Repeat for each object labeled background.
Screen Recording 2025-03-04 113240.mp4
A screen recording illustrating how to remove the background class from an image.

Using Keyboard Shortcuts

  1. In the Workspace hover over the image until you see BACKGROUND appear by the upper right corner of the image. This should also highlight the outline of the object.

  2. Hit the Delete (Del or ⌦) key on your keyboard. Note that the Delete (Del) key is different from the Backspace (⟵ or ⌫) key and the shortcut will not work if you use the wrong key.

  3. Repeat for each object labeled BACKGROUND.

  4. If you accidentally remove an object that you intended to keep, press the Ctrl and Z ( Ctrl + Z ) keys simultaneously to undo (↶) the previous action.

  5. Similarly, if you accidentally undid something that you did not intend to, press the Ctrl and Y ( Ctrl + Y ) keys simultaneously to redo (↷) the previous action.

There is a limit to the number of successive actions that you can undo (↶) or redo (↷) using the Navigation bar and keyboard shortcuts.

image-20250303-155103.png
1. Hover over the image until you see BACKGROUND
image-20250303-161137.png
2. Press Delete (Del or ⌦) on your keyboard
image-20250303-155143.png
3. The background class object is now deleted. Repeat for each object labeled background.

Annotating

To annotate an image for Ag Image Repository, you will be creating and modifying polygons that outline the target plants that are visible in the image.

Preprocessing identifies and labels large, easily distinguishable sections of plant in the image but often misses small parts, stems, edges, and gaps.

The purpose of manually annotating the images is to outline and label those parts that are missing as well as modify existing polygons to account for gaps between plants and incorrectly generated edges.

image-20250304-160846.png
An example of an unlabeled image. Note the imprecise outlines for things like stems, nearby features being grouped in one polygon which ignores gaps between plants, lack of distinction between plants in the foreground and background, and etc.

 

Using the Polygon Tool

Creating a Polygon

  1. Select the Polygon (⬠) tool on the controls sidebar and choose the <species name> label.

  2. Click Shape to enter drawing mode.

image-20250304-192437.png
image-20250304-192518.png
image-20250304-192601.png
image-20250304-192648.png

There are two ways to draw a polygon: either create points by clicking or by dragging the mouse on the screen while holding Shift.

Clicking points

  1. Left-click to create a point

    1. Continue to left click while following the outline of the plant. This will create a polygon that matches the shape of the plant.

    2. You can right click the previously created point to delete it.

    3. You can zoom in and out on the image by scrolling the mouse wheel

Clicking Points
Shift Drag

  1. While holding Shift (🡅) on your keyboard you can drag your mouse along the outline of the plant to create a polygon following the path of your cursor.

Shift Drag

Finishing your Polygon

It is okay for your polygon to overlap with another.

  1. Once you have finished outlining your shape, complete your polygon by clicking Done (✔) in the navigation bar or by pressing the N key on your keyboard

  2. To check your work, select the checkbox next to Show bitmap at the bottom of the objects sidebar

image-20250304-192918.png
image-20250304-192944.png
image-20250304-193016.png
image-20250304-193104.png

Adjusting a Polygon

  1. You can reactivate the polygon tool (⬠) by pressing the N key on your keyboard

  2. To edit a polygon, click on its center while holding Shift. This will open the polygon editor. Note: Clicking on the edge of the Polygon will not work

  3. In the editor you can create new points or delete part of a polygon by closing the line on another point.

  4. You can move the points by right-clicking and dragging the point to a new location.

  5. To finish editing a polygon click Done (✔) in the navigation bar or by press the N key on your keyboard

  6. To cancel editing, press the Escape (Esc) key on your keyboard

image-20250304-193357.png
image-20250304-193429.png

 

Using the Segment Anything Model (SAM)

The Segment Anything Model (SAM) uses masks to identify objects in an image. We use it for creating semi-automated annotations.

  1. In the controls sidebar select the magic wand icon to open AI Tools (🪄).

  2. Select the <species name> label in the dropdown

  3. In the Interactors dropdown select <Segment Anything 2.0>

  4. Toggle the Convert masks to polygons on

  5. Click Interact

  6. Click on the plant in the image to add it to the mask.

  7. Use the left click to add parts to the mask and the right click to remove parts from the mask.

  8. Once you have selected your mask, finish by clicking Done (✔) in the navigation bar or by pressing the N key on your keyboard

More points improve outline accuracy, but make shape editing harder. Fewer points make shape editing easier, but reduce outline accuracy.

image-20250303-184509.png
image-20250303-184528.png
image-20250303-184652.png
image-20250303-184907.png
image-20250303-185057.png

Cutting Holes in Polygons

To remove areas from a polygon, use the Slice mask/polygon tool which looks like a pentagon with a diagonal line through it and is located in the hidden section of the controls sidebar (.....)

  1. Click the bottom icon of the controls sidebar (.....)

  2. Click the icon of a bisected pentagon to open the tool

  3. Click on the polygon you want to cut

  4. Click the edge of the polygon you’re cutting

  5. Outline the area you want to cut by using left click to add points or right click to remove points from your outline

  6. When you are done, left click just outside of the image where you want the cut to end

  7. The new polygon will be the same as your previous but will exclude the area you just cut out

  8. You may now delete the previous polygon that does not include the cut-out

image-20250303-185214.png
image-20250303-185322.png
image-20250304-140501.png
image-20250304-140315.png
image-20250304-141318.png
image-20250304-141752.png
image-20250304-135406.png
image-20250304-135514.png
image-20250304-135420.png
image-20250304-135434.png
image-20250304-135445.png

 

 

 

 

Saving your work

To save your progress, click the Save (🖫) icon in the Navigation Bar or press the Ctrl and S ( Ctrl + S ) keys simultaneously.

Be sure to save your work often!

image-20250303-161846.png
Click the Save (🖫) icon
image-20250303-162010.png
Press Ctrl + S on your keyboard at the same time

Related content