TODOs (updated 12/10/2024)
Steps in rawtherapee for SemiF (vid)
Reformat Command line instructions for SemiF
Minor cleaning/formatting for both Field and SemiF
Flow chart of data transfer and processing for both Field and SemiF (Matt), could be a general flowchart placed in the main preprocessing page
Backlog Spreadsheet:
1. Download Batch
When downloading a Semifield batch, we need to copy it over to the local user from the NFS Lockers.
cd .. cd psa_images/semifield_tools/ screen -R <batch_name> python3 copy_from_lockers.py <batch_name>
Semifield batches can take anywhere from a few minutes to half an hour to copy over, depending on the batch size. This is why it is important to open a new screen for copying. Be sure only to copy one or two batches at a time within their respective screens to avoid disk space problems.
To exit the screen, use CTRL+A D
Checking in to see if the copying of a batch has been completed by using this string of commands:
cd psa_images/semifield_tools/ screen -r <batch_name>
If the batch has either been completed copying or there has been an error copying, there will be an input line. If there is no input line, the process has not completed.
To exit the screen, use CTRL+A D
1.1 Checking if the batch was completely copied
It is important to check if the copied batch was brought to the local user entirely.
We must know the # of images in the original batch. This can be checked using this command line:
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l
for example ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/MD_2024-07-02/*.ARW | wc -l
output: 560
- Check the number of images in the local directory after moving files from longterm storage to the local working directory. (before you run the …PIPLINE.py). This should equal the same as the above command. But should equal 0 after running the pipeline
ls /home/psa_images/temp_data/semifield-upload/<batch>/*.ARW | wc -l
for example: ls /home/psa_images/temp_data/semifield-upload/MD_2024-07-02/*.ARW | wc -l
output (before running the pipeline): 560
output (after running the pipeline): 0
- After running the pipeline (…PIPELINE.py), you want to check if all the ARW files were properly converted to JPGs and moved to the longterm “developed” storage.
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l
for example: ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/MD_2024-07-02/images/*.jpg | wc -l
output (after running the pipeline): 560
- These 3 commands should output the same number.
-
2. Color Profiling in RawTherapee
Launch RawTherapee:
./RawTherapee_5.8.AppImage
Be sure we are located within the Semifield_tools folder.
RawTherapee will launch in an separate window, opening to the most recent profile that has been worked on. Select the appropriate batch and allow to load. Scroll through making a mental note if some images are out of the ordinary (i.e. too bright, too dark, out of focus, etc).
2.1 Creating a color profile
Our goal when creating a color profile is to have every image look similarly to the rest of the batch. It is good practice to scan through all images to get an idea of what they look like, if there are any outliers, or anything else to note.
2.2 Look for a Color Checker
We want to use an image with a color checker, if one is available within the batch. Choose the most average looking picture (lighting and color-wise). If they all look reasonably similar, choose any of the options.
*In Semifield applications, we typically use a smaller, Color Checker lanyard. The Field Color Checker may be used instead, but the larger one is preferred for this application. Either is acceptable.
2.21 Bring in a recent profile
To bring in a recent profile, find the folder icon towards the top right of the screen, located to the left of the save button. This will open a window containing a list of saved profiles within the local user. Choose the most recent date in relation to the image being worked on now. It is helpful to use the name & date modified arrows, making it easier to find most recent profiles.
2.22 Color Picker
Once a previous profile is loaded in and applied, use the color picker tool. The Color Picker tool is found on the tool bar above the working image or using the White Balance tab on the panel to the right. Choose “Pick”.
We are only interested in using a specific gray with the color picker tool. In the image above, the gray we want is two squares above the black. Use this gray for every profile you make.
When finished, be sure to choose the hand icon to return to normal cursor mode.
2.23 Adjust exposure & saturation
Now, we navigate into the Exposure tab. This is the only tab we will work in, aside from the tab to access the Color Picker. There are only a few helpful settings to toggle here.
Our goal is to create an image that looks most closely like a real-life view of these plants. Our eyes are the most accurate check, but it can be helpful to keep an eye on the color graph in the top left that shifts when settings are changed. A good rule of thumb is for the peak of the three curves to land near this region of the graph.
*Laptop screens and Monitor settings will impact what we are working on. Try to always use the same screen when creating color profiles.
Exposure Compensation & Highlight Compression
These two go hand-in-hand. Exposure compensation should always be set to slightly more than Highlight Compression
Lightness
Has a similar effect to Exposure Compensation
Saturation
Toggle keeping in mind the intensity of colors on the color checker.
3. Running the pipeline
When the color profile has been saved, we must send it through the pipeline. Be sure to be navigated to the Semifield folder and make a new screen, use uppercase r (-R). If returning to an already existing screen, use lowercase r (-r) with the appropriate screen name.
cd .. cd psa_images/semifield_tools/ screen -R <batch_name> python3 semifield_CLOUDY_pipeline.py <batch_name>
To exit the screen, use CTRL+A D
Check on a batch sent into the pipeline by using this string of commands:
cd psa_images/semifield_tools/ screen -r <batch_name>
If the pipeline has either been completed running or there has been an error running it, there will be an input line. If there is no input line, the process has not completed.
To exit the screen, use CTRL+A D
3.1 Checking the success of the pipeline
If there is an indication the pipeline is finished running, we want to crosscheck the # of images in the original batch with the # of JPGs and PP3s created.
We can use both of these lines respectively:
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l
And compare to the # of ARWs using this:
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l
As long as the output # we get from each of these commands are equivalent, we know we have successfully applied the color profile to every image in the batch.
If the # of JPGs and PP3s are not equivalent to the original # of images, try running the pipeline again within the screen. Double check the name of the batch that was typed in, this is where errors are likely to happen. Also, be sure to be navigated to the Semifield folder. If we are within the Field folder, the pipeline will not be successful.
5. Update Backlog Spreadsheet
Change the progress column to the appropriate term to indicate where that batch is in the process. Make any notes/check box when done, if applicable.
When entire batch is fully completed, the row can be hidden by right clicking on the row #, then click hide.
4. Blob Analyzer (????)
4.1 Installing conda (miniconda)
Use these commands below from the command line in the SUNNY server:
mkdir -p ~/miniconda3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 rm ~/miniconda3/miniconda.sh ~/miniconda3/bin/conda init bash
Close the remote connection, then SSH back in
4.2 Setting up a conda environment
Still in the command line inside of SUNNY:
conda create -n <your env name> python=3.10 conda activate <your env name> cd /home/psa_images/analyze_blobs pip install -r requirements.txt
4.3 Running the code
Your conda environment must be activated to run the code.
Run these in the command line:
conda activate <your env name> cd /home/psa_images/analyze_blobs python ANALYZE_BLOB.py
Creates 2 time stamped files in the “results” folder in the main project directory.
A text files that provides a general summary of missing batches
A csv file that can be viewed in excel or google docs
5. Common & Useful Commands
Available batches:
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/ -lh
# of images in batch:
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l
# of JPGs processed:
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l
# of PP3 files created:
ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l
0 Comments