...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
user testing
Plant breeders or crop production
Anna locke
Ben
Vasu
susana
ruben alvarez
matt krakowsky
rachel -
dominic
jeff dunne
nonoy bandillo
jim holland
reza
craig
hamid
carlos
jessica spencer
superintendents
phillip winslow
jeremy martin
john Garner
keith starke
Superusers
jeremy davis -
jason ward
brynna
thiago
current dev tasks
- buttons for map rotation + tooltipAkshat Nitin Savla
- bug fix - img export on map rotation Akshat Nitin Savla
- develop env for app 2 - dronepilotapp-dev.psi.ncsu.edu
- get new VM for app 1 for super users
- gpu usage error (ODM - openmvs) Jinam
- load Rob’s unprocessed data from this year Jinam
- merge research stations branch Jinam
- move sandhills data from transfer/flights to flights
- db updates: cog_path from ortho_url ; add research_station
- update app1 codebase: superusers, sandhills server Jinam app2 - OIT puppet file edit Jinam
- add cc_fields and srs_fields geotiff to app 2 Jinam
October 15th
Dec 9 - 9-12 in-person user testing
setup new VM with OIT for app 1 - on-prem for superusers
OIT - will in the future create new VLAN for research computing
jinam - debugging issues in HPC processing
October 8th
chris submitting ticket to cals_it to install app 1 on the lenovo server for ‘superusers’. Jinam work on setting upload timeout to many hours to allow for the uploads.
the December 9th week is #1 choice for user testing with GameTheory. User testing will be a face to face 2-3 hour workshop like event. Fikret has a graduate student class that week that involves folks like Joe and Jeff. Chris check with Fikret for schedule and possible cancellation due to low registration.
setting up development environment for app 2 on the same VM. Will use containers to separate development and production environments. Jinam working on it.
Continued bugs on sfm code that prevents use of the gpu’s and subsequent timeouts on the cpu’s. Possible use of Vyshnavi given all the competing demands on Jinam. Chris checking with Mikah and Steven about using her time.
GameTheory coming into town for face to face user testing.
...
Amanda - ontology stuff update
Catchup on adding data for 2024 - running through the whole pipeline
Jacob - talk to HPC team about preferred queue
NoSQL db - dashboarding + other insights/saved queries tool
Cloudiness dropdown - code done, need to merge
Jinam - update HPC codebase (add cleaned up code)
Jinam - working on adding research stations to DPP
Additional data product - download one image for the whole grid too along with images for each plot
User testing in October
Plant breeders or crop production
Anna locke
Ben
Vasu
susana
ruben alvarez
matt krakowsky
rachel -
dominic
jeff dunne
nonoy bandillo
jim holland
reza
craig
hamid
carlos
jessica spencer
superintendents
phillip winslow
jeremy martin
john Garner
keith starke
Superusers
jeremy davis -
jason ward
brynna
Involve GameTheory for user testing
UI updates: Tooltips for users - shift + cursor for moving map, etc/ feedback+home button on last page
Eric Sills question for Chris. GitHub actions - service account idea.
...
Continue to collect Sandhills imagery weekly and transferring to sandhills workstation
When weather permits, we collect both RGB and MS/Thermal in high priority areas
Processing manually, about 1 week behind
Processing on workstation can still take 6-8 hours on larger missions. We typically queue up 3 flights and run overnight, and most mornings the processing is finished, but not always (typically ~8 missions collected in a day)
We should look into the cluster based processing capability of metashape
Still waiting on charging stations and Altum-PT sensors (asked Benchmark for an update yesterday, have yet heard back)
Currently charging extra batteries in Raleigh and switching out with Jeremy in Sandhills
Jevon is helping change the 20TB mount point of the sandhills workstation from /mnt/data/raw to /mnt/data
We needed space for ./metashape, ./pdata (processed data), and ./scripts to sit at the ./data level
May need to do some reconfig with Globus after moving mount point
Once setup, we’ll start moving and processing data again
Purchased additional SD and CFExpress cards for sensors. Altum-PT collects A LOT of images, 4 flights is ~180GB of images
Effort to Automate Metashape pipeline (Metashape Hackathon?)
https://github.com/ucdavis/metashape
...
Plan to fly this Thursday - looks like nice weather and hope to collect multispectral data
Setup M300s with new KML flight plans - new flight plans are designed based on priority (will discuss in detail at next meeting)
Developing map that will illustrate missions and priorities
Still waiting for battery charging stations - hopefully next week
Globus setup last week - will transfer raw and processed data this weekend
Jacob Fosso Tandecould you setup a ./pdata/ folder on the sandhills workstation for the processed imagery (as well as give proper permissions for transfer)/sandhills/raw and /sandhills/process/
You should set up the same file structure at source then copy sandhills and everything will map correctly.
I do not have access to your source system but globus script that can do the setup with very little interaction. After the first run, the transfer will happen automatically. However , I do have another script that you can use anytime provided you give it the correct input.
If this needs a separate globus collection, could you set this up as well
No, a collection for a station. Beneath the root folder, other folders can be added as needed
Brynna will post a processing report from one of the larger metashape projects
Jinam is still hung up with his visa. Susan working with him closely
...
| Anchor | ||||
|---|---|---|---|---|
|
/rs1/shares/cals-research-stations/sandhills/raw read only, even for developers
/rs1/shares/cals-research-stations/sandhills/processed read only for users, write access for developers only
Ask Jevon to setup accounts on workstation
| Anchor | ||||
|---|---|---|---|---|
|
...
additional batteries are ordered. All told, each UAV will have 8 available batteries (4 flights)
Cloud City Drones was required to cancel their bid, bid issued to Benchmark (local vendor) May 8th. We were told 2-3 weeks for DJI equipment (platform and P1 sensor), 8-10 weeks for Multispectral sensor. (They have been strung along about delivery dates for the Altum-PT sensor - see Joe Gauge)
Need to
Update from Joe about flying multiple sensors
Prioritizing Flights
‘Non-objective flights
Goal is weekly flights
‘Objective’ Flights
best to identify crops and objectives as clear as possible before last minute (scheduling)
Cotton (A3/A5/D2B)
Soybeans (A4)
Prioritizing Sensors
If conditions are not conducive to useful data, is there any value in flying the sensor?
On partly cloudy days, you cannot ‘easily’ radiometrically correct for shadows within the flights and data - as such, is there any reason to fly the sensor in those conditions?
First Flights Planned for Thursday (May 11th)
Update from Jeremy on Planting
Matrise 300 with 8 batteries
4 flights possible
Need to prioritize areas and sensor (s)
Value of multispectral with little-to-no plant material?
https://phenome-force.github.io/PhenomeForce/
...
- Stitching - we need someone to explore licensing on HPC’s. Charges based on users or cores could be a problem. Jevon Smith do you have time to look at licensing from Metashape vs Pix4D?
- downstream metadata to add post-ingestion (additional columns)
sandhills or other station, field, crop, researcher
- 30TB available in HPC 'storage’ for storage and processing (central storage)
...
Allow researchers to search for all data corresponding to a particular set of coordinates and year combination. They can do this via a polygon selection tool that allows them to identify individual fields.
Allow researchers to upload a field map (with RTK level plot corners) so that plot averages (or percentiles) for any band can be calculated. Plots values should be downloadable. See commercial provider Solvi for indications of intended scope.
User defines plots in at least 10 polygons
Additional metadata will be written back to the database based on use of the web tool. Data such as crop, researcher and likely more can be appended to the metadata.
...
Position Update: UAV Pilot
PSI Drone architectural design session (Thu Mar 16, 2023 3pm – 5pm)
Overview of current pipeline
Research Station Hardware Discussion
Adding compute capabilities to the research station hardware (Javon)
benefits/limitations/cost?
Decided to purchase additional workstation dedicated to this project and housed at research station
Rob’s recommendation ~4TB NVME for processing, then perhaps 4-8TB SSD for short-to mid term storage and/or overflow.
GeForce 3000 RTK series
University/PSI/commercial post-processing compute resources available for post-processing imagery (did not discuss)
Not a lot of GPU’s
HPC is poorly administered, but should get better, cover all our basses for now with workstation (plan B)
Flights (targeted versus broad farm-level)
...
Position Update: Developer Position KIRTAN DESAI
Position Update: UAV Pilot
General Expectations
# of flights, how many programs/fields?
First year ~1/week at Sandhills
Turf, Cotton, and Peanuts?
Training and training material development
Lower priority at this point, but may be able to work in with some extension efforts
Timing-first flights?
How to incorporate prior work done on ‘leveling data‘ - Rob
Bring in design group - leaning toward ‘unstructured’ uploads
UAV Equipment updates?
Do we know the expected delivery for sensors and platform
Things to check - with dual mount both sensors RTK capable?
Vision for first-year use (1 w/ UAV Position, 1 w/ Jeremy @ Sandhills)
Hardware Discussion
Before any hardware is ordered, need to consider if image post-processing will occur ‘off-site’, at a remote HCP Center, or ‘in-the-cloud’
Need to ask Javon about added costs if done remotely (GPU, etc)
Perhaps spec out two options
Arch-Design - need to decide when to bring in Chris’s ‘Geospatial Guy’ (Milad)
Design Group Meeting (two weeks)
To Do
Javon - additional costs for hardware at remote site if off-site post-processing images
Email susan about Equipment status
Follow-up with potential UAV Pilot for year 1
2 weeks - do the demo (lead off - then talk with design team afterward, 2 hours)
Setup meeting to discuss Sebastiano Busato, Daniel Perondi involvement
Demo complete workflow with data (could be a long meeting)
Include Milad
Call Amanda about Developer position and if she is okay with the Kirtan (Chris)
...