TODOs
...
Adding steps about the tools and installing those tools ( rawtherapee, mobaxterm)
Steps in rawtherapee
Command line instructions
Pics or screenshots will be very helpful
Provide video overview
Requirements
How to get requirements
Additional requirements and instructions
Azure Matt
SUNNY Matt
NFS Locker Access Chris?
(updated 12/10/2024)
Flow chart of data transfer and processing for both Field and SemiF (Matthew Kutugata) could be a general flowchart placed in the main preprocessing page.
Courtney Belcher Courtney Belcher could give it a shot
Backlog Spreadsheet:
| Lref gdrive file | ||
|---|---|---|
|
...
When downloading a Semifield batch, we need to copy it It copies over to the local user from the NFS Lockers.
| Code Block |
|---|
cd ..
cd psa_images/semifield_tools/
screen -R <batch_name>
python3 copy_from_lockers_test.py <batch_name> |
Semifield batches can take anywhere from a few minutes to half an hour to copy over, depending on the batch size. This It is why it is important to open a new screen for copying. Be sure only Only to copy one or two batches at a time within their respective screens to avoid disk space problems.
...
If the batch has either been completed copying or there has been an error copying, there will be an input line. If there is no input line, the process has not completed.
...
It is important to check if the copied batch was brought to the local user entirely.
We must know the number # of images in the original batch. This can be checked using this command line:
...
| breakoutMode | wide |
|---|
. Use this command:
| Code Block |
|---|
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l |
Sometimes this command may need to be run two or three times before it output the correct value.
Check the # of images in the local directory after download has completed.
(Moving files from longterm storage to the local working directory, before you run the the pipeline).
*The output should be equal to the above command; should be 0 after running the pipeline
Use this command:
| Code Block |
|---|
ls /home/psa_images/temp_data/semifield-upload/<batch>/*.ARW | wc -l |
After running the pipeline, check if all the ARW files have converted to JPGs and have moved to the longterm “developed” storage.
Use this command:
| Code Block |
|---|
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l |
Example: MD_2024-07-02
| Code Block |
|---|
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/ |
...
MD_2024-07-02/*.ARW | wc -l |
output: 560
| Code Block |
|---|
ls /home/psa_images/temp_data/semifield-upload/MD_2024-07-02/*.ARW | wc -l |
output (before running the pipeline): 560
output (after running the pipeline): 0
| Code Block |
|---|
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/MD_2024-07-02/images/*.jpg | wc -l |
output (after running the pipeline): 560
Trouble shooting:
If developed jpgs are not in the longterm storage, check to see if they’re still in the local temp_data outputs folder:
Check for jpgs:
| Code Block |
|---|
ls /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.jpg | wc -l |
check for same number of .pp3 files (how you know they’ve been developed):
| Code Block |
|---|
ls /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.pp3 | wc -l |
If they’re both present and have the same number, we know they’ve been successfully preprpocessed, but may not have been successfully move to the longterm locker storage location.
If there is an “Assertion Error”, try going into RawTherapee, and create & save the profile again.
2. Color Profiling in RawTherapee
...
| Code Block |
|---|
./RawTherapee_5.8.AppImage |
...
RawTherapee will launch in an separate window and open to the most recent profile that has been worked on. Select the appropriate batch and allow it to load. Scroll through and make a mental note if some images are out of the ordinary (i.e. too bright, too dark, out of focus, etc.).
...
2.1 Creating a color profile
Our goal when creating a color profile is to have every image look similarly to the rest of the batch. When RawTherapee launches, it will bring us to the last batch we worked on by default. Be sure to navigate to the intended batch. Note when there is more than one folder within the batch. We must adjust every image in each folder before sending the batch into the pipeline.
It is good practice to scan through all images to get an idea of what they are looking like, if there are any outliers, or anything to make note of.
2.2 Look for a Color Checker
We want to use an image with a color checker, if one is available within the batch. Choose the most average looking picture (lighting and color-wise).
...
In the Semifield, we typically use a more compact Color Checker. The Field color checker may . It is preferred to select an average image with both a color checker and biomass. If there is no biomass present, choose an average image with a color checker.
...
*In Semifield applications, a smaller Color Checker lanyard is typically used. The Field Color Checker might be used instead, but the compact smaller one is preferred for this application.
2.2 Creating a color profile
- Either is acceptable.
2.
...
12 Bring in a
...
-
...
Recent profile & Color Picker
...
To bring in a recent profile, find the folder icon towards the top right of the screen, located to the left of the save button. This will open a window containing a list of saved profiles within the local user. Choose the most recent date in relation to the image being worked on.
Once a previous profile is loaded in and applied, use the color picker tool. The gray we want is two squares away from blackColor Picker tool is found on the tool bar above the working image or use the White Balance tab on the panel to the right. Choose “Pick”.
...
Only one square on the Color Checker is used for in this application. Referring to the image above, the gray residing two squares above the black is the color to use. Use this gray for every profile you make. Choose the hand icon to return to normal cursor mode.
2.
...
13 Adjust exposure & saturation
...
Now, navigate into the Exposure tab. This is the only tab used in preprocessing, aside from the tab to access the Color Picker. There are only a few helpful settings to toggle here.
The goal is to create an image that looks most closely like a real-
2.24 Save new profile and exit RawTherapee
-
3. Running the pipeline
When life view of these plants. The human eye is the most accurate check, but it can be helpful to reference the color graph located on the top left of the scree. It shifts when settings are changed. A good rule of thumb is for the peak of the three curves to land near this region of the graph.
...
*Laptop screens and Monitor settings will impact perception of images. Try to use the same screen when creating color profiles.
Exposure Compensation & Highlight Compression
These two go hand-in-hand. Exposure compensation should always be set to slightly more than Highlight Compression
Lightness
Has a similar effect to Exposure Compensation
Saturation
Toggle keeping in mind the intensity of colors on the color checker.
3. Running the pipeline
...
** Don't run >3 batches in the pipeline at once. This applies to both Semifield and Field combined. **
Profiling can be done and batches can be prepared for the pipeline, but wait to run them until another batch finished.
After the color profile has been saved, we must send it through the pipeline. Be sure to be navigated to the Semifield folder and make a new screen, use uppercase r (-R). If returning to an already existing screen, use lowercase r (-r) with the appropriate screen name.
...
If there is an indication the pipeline is finished running, we want to crosscheck the number # of images within in the original batch with the number # of jpgs JPGs and pp3s PP3s created.
We can use both of these lines respectively:
| Code Block | ||
|---|---|---|
| ||
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l |
| Code Block | ||
|---|---|---|
| ||
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l |
And compare to the number # of arws ARWs using this:
| Code Block | ||
|---|---|---|
| ||
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l |
As long as the output number we get from each of these commands # are equivalent, we know we have successfully applied the color profile has successfully been applied to every image in the batch.
If the number # of jps JPGs and pp3s PP3s are not equivalent to the original number # of images, try running the pipeline again within the same screen. Double check the name of the batch that was typed in, this is where errors are likely to happen. Also, be sure to be navigated to the Semifield folder. If we are within navigated to the Field folder, the pipeline will not be successful.
...
When entire batch is fully completed, the row can be hidden by right clicking on the row number#, then click hide.
4. Blob Analyzer (????)
...
4.1 Installing conda (miniconda)
...
Available batches:
| Code Block | breakoutMode | wide
|---|
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/ -lh |
# of images in batch:
| Code Block | ||
|---|---|---|
| ||
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l |
# of jpgs JPGs processed:
| Code Block | ||
|---|---|---|
| ||
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l |
# of pp3 PP3 files created:
| Code Block | breakoutMode | wide
|---|
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l |
...