Page Comparison

TODOs (updated 12/10/2024)

1. Minor cleaning/formatting for both Field and SemiF

2. Flow chart of data transfer and processing for both Field and SemiF (Matthew Kutugata) could be a general flowchart placed in the main preprocessing page.

   1. Courtney Belcher Courtney Belcher could give it a shot

...

cd ..
cd psa_images/semifield_tools/
screen -R <batch_name>
python3 copy_from_lockers_test.py <batch_name>

Semifield batches can take anywhere from a few minutes to half an hour to copy over, depending on the batch size. It is important to open a new screen for copying. Only to copy one or two batches at a time within their respective screens to avoid disk space problems.

...

We must know the # of images in the original batch. Use this command:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

Sometimes this command may need to be run two or three times before it output the correct value.

Check the # of images in the local directory after download has completed.

...

Use this command:

ls /home/psa_images/temp_data/semifield-upload/<batch>/*.ARW | wc -l

...

Use this command:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

...

 ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/MD_2024-07-02/*.ARW | wc -l

ls /home/psa_images/temp_data/semifield-upload/MD_2024-07-02/*.ARW | wc -l

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/MD_2024-07-02/images/*.jpg | wc -l

Trouble shooting:

If developed jpgs are not in the longterm storage, check to see if they’re still in the local temp_data outputs folder:

• Check for jpgs:

ls /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.jpg | wc -l

check for same number of .pp3 files (how you know they’ve been developed):

ls /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.pp3 | wc -l

If they’re both present and have the same number, we know they’ve been successfully preprpocessed, but may not have been successfully move to the longterm locker storage location.

• If there is an “Assertion Error”, try going into RawTherapee, and create & save the profile again.

2. Color Profiling in RawTherapee

...

We want to use an image with a color checker, if one is available within the batch. Choose the most average looking picture (lighting and color-wise). If they all look reasonably similar, we can choose whichever we wantIt is preferred to select an average image with both a color checker and biomass. If there is no biomass present, choose an average image with a color checker.

...

*In Semifield applications, a smaller Color Checker lanyard is typically used. The Field Color Checker might be used instead, but the smaller one is preferred for this application. Either is acceptable.

...

• Exposure Compensation & Highlight Compression

  • These two go hand-in-hand. Exposure compensation should always be set to slightly more than Highlight Compression

• Lightness

  • Has a similar effect to Exposure Compensation

• Saturation

  • Toggle keeping in mind the intensity of colors on the color checker.

3. Running the pipeline

...

** Don't run >3 batches in the pipeline at once. This applies to both Semifield and Field combined. **

Profiling can be done and batches can be prepared for the pipeline, but wait to run them until another batch finished.

After the color profile has been saved, send it through the pipeline. Be sure to be navigated to the Semifield folder and make a new screen, use uppercase r (-R). If returning to an already existing screen, use lowercase r (-r) with the appropriate screen name.

...

We can use both of these lines respectively:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l

And compare to the # of ARWs using this:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

...

Available batches:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/ -lh

# of images in batch:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

# of JPGs processed:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

# of PP3 files created:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l

...