TODOs (updated 12/10/2024)

1. Flow chart of data transfer and processing for both Field and SemiF (Matthew Kutugata) could be a general flowchart placed in the main preprocessing page.

   1. Courtney Belcher Courtney Belcher (Unlicensed) could give it a shot

...

Backlog Spreadsheet:

** THIS PAGE IS BASED ON BBV3.0 - SONY CAMERA **

1.

...

Downloading a batch

When downloading a Semifield batch, It copies over to the local user from the NFS Lockers.

...

To exit the screen, use CTRL+A D

...

1.1 Checking if the batch was completely copied

It is important to check Check if the copied batch was brought to the local user entirely. We must know the # of images in the original batch. Use this command:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

...

Check the # of images in the local directory after download has completed:

ls /home/psa_images/temp_data/semifield-upload/<batch>/*.ARW | wc -l

(Moving files from longterm storage to the local working directory, before you run the the pipeline).

*The output should be equal to the above command; should be 0 after running the pipeline

Use this command:

ls /home/psa_images/temp_data/semifield-upload/<batch>/*.ARW | wc -l

After running the pipeline, check if all the ARW files have converted to JPGs and have moved to the longterm “developed” storage. Use this command:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

 ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/MD_2024-07-02/*.ARW | wc -l

ls /home/psa_images/temp_data/semifield-upload/MD_2024-07-02/*.ARW | wc -l

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/MD_2024-07-02/images/*.jpg | wc -l

Trouble shooting:

If developed jpgs are not in the longterm storage, check to see if they’re still in the local temp_data outputs folder:

• Check for jpgs:

ls /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.jpg | wc -l

check for same number of .pp3 files (how you know they’ve been developed):

ls /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.pp3 | wc -l

If they’re both present and have the same number, we know they’ve been successfully preprpocessed, but may not have been successfully move to the longterm locker storage location.

...

Here is an example of checking if batches were copied

2. Color Profiling in RawTherapee

...

RawTherapee will launch in an a separate window and open opens to the most recent profile that has been recently worked on profile. Select the appropriate batch and allow it to load. Scroll through and make a mental note if some images are out of the ordinary (i.e. too bright, too dark, out of focus, Amiga in the frame, etc.).

...

When creating a color profile

...

Our goal when creating a color profile , the goal is to have every image look similarly to the rest of the batch. When RawTherapee launches, it will bring us to the last batch we worked on by default. Be sure to navigate to the intended batch. Note when there is more than one folder within the batch. We must adjust every image in each folder before sending the batch into the pipeline.

It is good practice to scan through all images to get an idea of what they are looking like, if there are any outliers, or anything to make note of.

...

...

2.1 Look for a Color Checker

We want to use Using an image with a color checker, if one is available within the batch. Choose the , choose the most average looking picture (lighting and color-wise). It is preferred to select an average image with both a color checker and biomass. If there is no biomass present, choose an average image with a color checker.

...

*In Semifield applications, a smaller Color Checker lanyard is typically used. The Field Color Checker might be used instead, but the smaller one is preferred for this application. Either is acceptable.

2.

...

2 Bring in a Recent profile & Color Picker

...

To bring in a recent profile , find the folder icon towards for the location being worked on. At the top right of the screenscree, located to the left of the save buttonlocate the folder icon. This will open a window containing a list of saved profiles within the local user. Choose the most recent date in relation to the image being worked on.

Once a the previous profile is loaded in and applied, use the color picker tool. The Color Picker tool is found on the tool bar above the working image or use the White Balance tab on the panel to the right. Choose “Pick”.

...

Only one square on the Color Checker is used for in this application. Referring to the image above, the gray The gray square residing two squares above the black is the color to use. Use this gray for every profile. Choose the hand icon to return to normal cursor mode.

2.

...

3 Adjust exposure & saturation

...

Now, navigate into the Exposure tab. This is the only tab used in preprocessing, aside from the tab to access the Color Picker. There are only a few helpful settings to toggle here.

...

3. Running the pipeline

** Don't Do not run >3 batches in the pipeline at once. **

...

If the # of JPGs and PP3s are not equivalent to the original # of images, try running the pipeline again within the same screen. Double check the name of the batch that was typed in, this is where errors are likely to happen. Also, be sure to be navigated to the Semifield folder. If navigated to the Field folder, the pipeline will not be successful.

...

4. Update Backlog Spreadsheet

Change the progress column to the appropriate term to indicate where that batch is in the process. Make any notes/check box when done, if applicable.

When entire batch is fully completed, the row can be hidden by right clicking on the row #, then click hide.

4. Blob Analyzer (????)

4.1 Installing conda (miniconda) 

...

5. Troubleshooting

If developed jpgs are not in the longterm storage, check to see if they’re still in the local temp_data outputs folder:

• Check for jpgs:

mkdirls -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm  ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash

Close the remote connection, then SSH back in

4.2 Setting up a conda environment

Still in the command line inside of SUNNY:

conda create -n <your env name> python=3.10
conda activate <your env name>
cd /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.jpg | wc -l

check for same number of .pp3 files (how you know they’ve been developed):

ls /home/psa_images/analyze_blobs
pip install -r requirements.txt

4.3 Running the code 

Your conda environment must be activated to run the code.

Run these in the command line:

conda activate <your env name>
cd /home/psa_images/analyze_blobs
python ANALYZE_BLOB.py

1. Creates 2 time stamped files in the “results” folder in the main project directory. 

   1. A text files that provides a general summary of missing batches

   2. A csv file that can be viewed in excel or google docs

...

temp_data/semifield-outputs/MD_2024-07-26/images/*.pp3 | wc -l

If they’re both present and have the same number, we know they’ve been successfully preprpocessed, but may not have been successfully move to the longterm locker storage location.

• If there is an “Assertion Error”, try going into RawTherapee, and create & save the profile again.

6. Common & Useful Commands

Available batches:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/ -lh

...

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l

 ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/MD_2024-07-02/*.ARW | wc -l

ls /home/psa_images/temp_data/semifield-upload/MD_2024-07-02/*.ARW | wc -l

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/MD_2024-07-02/images/*.jpg | wc -l