SemiField Preprocessing

List of things we want to include

TODOs

1. Adding steps about the tools and installing those tools ( rawtherapee, mobaxterm)

2. Steps in rawtherapee

3. Command line instructions

4. Pics or screenshots will be very helpful

   1. Provide video overview

5. Requirements

   1. How to get requirements

6. Additional requirements and instructions

   1. Azure

   2. SUNNY

   3. NFS Locker Access

1. Download Batch

When downloading a Semifield batch, we need to copy it over to the local user from the NFS Lockers.

cd ..
cd psa_images/semifield_tools/
screen -R <batch_name>
python3 copy_from_lockers.py <batch_name>

Semifield batches can take anywhere from a few minutes to half an hour to copy over, depending on the batch size. This is why it is important to open a new screen for copying. Be sure only to copy one or two batches at a time within their respective screens to avoid disk space problems.

To exit the screen, use CTRL+A D

Checking in to see if the copying of a batch has been completed by using this string of commands:

cd psa_images/semifield_tools/
screen -r <batch_name>

If the batch has either been completed copying or there has been an error copying, there will be an input line. If there is no input line, the process has not completed.

To exit the screen, use CTRL+A D

1.1 Checking if the batch was completely copied

It is important to check if the copied batch was brought to the local user entirely.

We must know the number of images in the original batch. This can be checked using this command line:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

-

-

-

-

2. Color Profiling

When creating the color profile ---------

2.1

3. Running the pipeline

When the color profile has been saved, we must send it through the pipeline. Be sure to be navigated to the Semifield folder and make a new screen, use uppercase r (-R). If returning to an already existing screen, use lowercase r (-r) with the appropriate screen name.

cd ..
cd psa_images/semifield_tools/
screen -R <batch_name>
python3 semifield_CLOUDY_pipeline.py <batch_name>

To exit the screen, use CTRL+A D

Check on a batch sent into the pipeline by using this string of commands:

cd psa_images/semifield_tools/
screen -r <batch_name>

If the pipeline has either been completed running or there has been an error running it, there will be an input line. If there is no input line, the process has not completed.

To exit the screen, use CTRL+A D

If there is an indication the pipeline is finished running, we want to crosscheck the number of images within the original batch with the number of jpgs and pp3s created.

We can use both of these lines respectively:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l

As long as the output number we get from each of these commands are equivalent, we know we have successfully applied the color profile to every image in the batch.

If the number of jps and pp3s are not equivalent to the original number of images, try running the pipeline again within the screen. Double check the name of the batch that was typed in, this is where errors are likely to happen. Also, be sure to be navigated to the Semifield folder. If we are within the Field folder, the pipeline will not be successful.

Backlog Spreadsheet:

Common & Useful Commands

Available batches:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/ -lh

# of images in batch:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-upload/<batch>/*.ARW | wc -l

# of jpgs processed:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.jpg | wc -l

# of pp3 files created:

ls /mnt/research-projects/s/screberg/longterm_images2/semifield-developed-images/<batch>/images/*.pp3 | wc -l

Blob Analyzer

Installing conda (miniconda) 

Use these commands below from the command line in the SUNNY server:

1. mkdir -p ~/miniconda3

2. wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh

3. bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3

4. rm ~/miniconda3/miniconda.sh

5. ~/miniconda3/bin/conda init bash

6. Close the remote connection then ssh back in

Setting up a conda environment

Still in the command line inside of SUNNY:

1. conda create -n <your env name> python=3.10

2. conda activate <your env name>

3. cd /home/psa_images/analyze_blobs

4. pip install -r requirements.txt

Running the code 

Your conda environment must be activated to run the code. Run these in the command line.

1. conda activate <your env name>

2. cd /home/psa_images/analyze_blobs

3. python ANALYZE_BLOB.py

   1. Creates 2 time stamped files in the “results” folder in the main project directory. 

      1. A text files that provides a general summary of missing batches

      2. A csv file that can be viewed in excel or google docs