...
Pixel-wise labels must be accurate. The process of labeling by hand can take minutes to hours even when using third-part annotation tools. Labels for semantic segmentation are the most time consuming. The need for high accuracy, complex leaf structures, and amorphous shapes make labeling plants one of the most difficult labeling tasks. Many have reported the high time requirement needed for labeling images of weeds.
...
Company: Precise BPO Solution
1000 images
11726 segments
$0.125 per segment*
*While TAMU was given a discount of $0.095 per segment, $0.125, the non-discounted price, is used here. It is unlikely the company will provide the same discount for images over 1000.
Time to label all images - 2.5 weeks
Number of workers unknown
$1465.75 total cost
Other Labeling Services
Precise BPO Solutions were relatively inexpensive compared to other more known labeling services. However, their total time (2.5 weeks for 11,726 segments) is not scalable. Using more workers may decrease turn-around-time but brings increased costs.
Using third-party labeling services like Google AI Platform and Amazon SageMaker come with high costs when considering scale and time. Less time for labeling requires more workers which increases costs.
Google Cloud (AI Platform)
uses “unit” pricing. For example, 2 segments x 2 workers = 4 units.
Prices start at $870 for 1,000 units
Image segmentation is their most expensive labeling task
Amazon SageMaker (Mechanical Turk)
Pricing for number of reviewable objects, in our case segments, plus tasks
$0.08 per object review + $0.84 per semantic labeling task
reviewing 16272 per week * (730 hours in a month / 168 hours in a week) = 70705.71 per month
semantic segmentation is most expensive labeling task
Amazon SageMaker (vendor: Cogito)
highest rated labeling vendor in amazon marketplace
Company | Workers | Segments | Worker hours | Cost |
|---|---|---|---|---|
1 | 1000 | $870 | ||
1 | 1 | $0.08 + $0.84 | ||
1 | $5.04 | |||
3,600 hrs or 360,000 annotations | $5400 / year | |||
Agricultural Scenes are Diverse
...
Labeling Cost Projections for SemiField
Precise BPO Solutions
We can estimate the expected costs and time for SemiField data collection using the Texas A&M numbers.
here we use a smaller average of 8 segments per image (instead of 11.726 like TAMU)
$0.125 per segment
2.5 weeks (13 working days from 8am - 5pm) = ~104 worker hours
104 hours translates to 32 seconds per segment (104 / 11726)
Images | Segments | Worker Hours estimate |
|---|
BPO Solutions | Google Cloud AI Platform** | Amazon Turk** | |
|---|---|---|---|
1,000* |
11,726 | 6,249.958 | $1,465.750 | $20,403 | $20,638 | |
1,000 | 8,000 |
4,264 | $1,000 |
$13,920 | $14,080 | |
10,000 |
80,000 |
42,640 | $10,000 |
$139,200 | $140,800 | |
25,000 |
200,000 |
106600
$25,000.000
100,000
426400
$100,000.000
250,000
1066000
$250,000.000
*TAMU example
...
106,600 | $25,000 | $348,000 | $352,000 | ||
50,000 | 400,000 | 213,200 | $50,000 | $696,000 | $704,000 |
100,000 | 800,000 | 426,400 | $100,000 | $1,392,000 | $1,408,000 |
250,000 | 2,000,000 | 1,066,000 | $250,000 | $3,480,000 | $3,520,000 |
*TAMU example **Using 2 workers for quicker turn around time