Page Comparison

TODOs (updated 12/10/2024)

1. Minor cleaning/formatting for both Field and SemiF

2. Flow chart of data transfer and processing for both Field and SemiF (Matthew Kutugata) could be a general flowchart placed in the main preprocessing page.

   1. Courtney Belcher Courtney Belcher could give it a shot

...

cd ..
cd psa_images/semifield_tools/
screen -R <batch_name>
python3 copy_from_lockers_test.py <batch_name>

Semifield batches can take anywhere from a few minutes to half an hour to copy over, depending on the batch size. It is important to open a new screen for copying. Only to copy one or two batches at a time within their respective screens to avoid disk space problems.

...

We must know the # of images in the original batch. Use this command:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

Sometimes this command may need to be run two or three times before it output the correct value.

Check the # of images in the local directory after download has completed.

...

Use this command:

ls /home/psa_images/temp_data/semifield-upload/<batch>/*.ARW | wc -l

...

Use this command:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

...

 ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/MD_2024-07-02/*.ARW | wc -l

ls /home/psa_images/temp_data/semifield-upload/MD_2024-07-02/*.ARW | wc -l

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/MD_2024-07-02/images/*.jpg | wc -l

Trouble shooting:

If developed jpgs are not in the longterm storage, check to see if they’re still in the local temp_data outputs folder:

• Check for jpgs:

ls /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.jpg | wc -l

check for same number of .pp3 files (how you know they’ve been developed):

ls /home/psa_images/temp_data/semifield-outputs/MD_2024-07-26/images/*.pp3 | wc -l

If they’re both present and have the same number, we know they’ve been successfully preprpocessed, but may not have been successfully move to the longterm locker storage location.

• If there is an “Assertion Error”, try going into RawTherapee, and create & save the profile again.

2. Color Profiling in RawTherapee

...

./RawTherapee_5.8.AppImage

Be sure to be located within the Semifield_tools folder.

RawTherapee will launch in a an separate window , opening and open to the most recent profile that has been worked on. Select the appropriate batch and allow it to load. Scroll through , making and make a mental note if some images are out of the ordinary (i.e. too bright, too dark, out of focus, etc.).

...

We want to use an image with a color checker, if one is available within the batch. Choose the most average looking picture (lighting and color-wise). If they all look reasonably similar, we can choose whichever we wantIt is preferred to select an average image with both a color checker and biomass. If there is no biomass present, choose an average image with a color checker.

...

*In Semifield applications, a smaller Color Checker lanyard is typically used. The Field Color Checker might be used instead, but the smaller one is preferred for this application. Either is acceptable.

...

• Exposure Compensation & Highlight Compression

  • These two go hand-in-hand. Exposure compensation should always be set to slightly more than Highlight Compression

• Lightness

  • Has a similar effect to Exposure Compensation

• Saturation

  • Toggle keeping in mind the intensity of colors on the color checker.

3. Running the pipeline

...

** Don't run >3 batches in the pipeline at once. This applies to both Semifield and Field combined. **

Profiling can be done and batches can be prepared for the pipeline

...

When , but wait to run them until another batch finished.

After the color profile has been saved, we must send it through the pipeline. Be sure to be navigated to the Semifield folder and make a new screen, use uppercase r (-R). If returning to an already existing screen, use lowercase r (-r) with the appropriate screen name.

...

If there is an indication the pipeline is finished running, we want to crosscheck the # of images in the original batch with the # of JPGs and PP3s created.

We can use both of these lines respectively:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l

And compare to the # of ARWs using this:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

As long as the output # we get from each of these commands are equivalent, we know we have successfully applied the color profile has successfully been applied to every image in the batch.

If the # of JPGs and PP3s are not equivalent to the original # of images, try running the pipeline again within the same screen. Double check the name of the batch that was typed in, this is where errors are likely to happen. Also, be sure to be navigated to the Semifield folder. If we are within navigated to the Field folder, the pipeline will not be successful.

...

Available batches:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/ -lh

# of images in batch:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

# of JPGs processed:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

# of PP3 files created:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l

...