Setup run hpc

[ghar1821]

Describe your changes

This PR introduces new configuration setup to support running benchmarks on the WEHI HPC environment and some implementation changes.

Config to run on WEHI hpc

The hpc's scratch system is prone to task collisions when multiple jobs access the same shared cache/temp folders. It is not smart enough to keep each job isolated (or maybe it is a nextflow problem, I am not too sure). To resolve this, I introduced env variables (on top of few that are already mandatory) like NUMBA_CACHE_DIR, and APPTAINER_TMPDIR. These are the "parent" directory. Each task will create a sub directory within it with the Task ID as the directory name and use it. This prevents methods that write out temp files (e.g., CytoNorm, BatchAdjust) from overwriting each other's temp files.

This config will only be used to run the job on WEHI hpc and have no impact on the config to run the job on open problems AWS.

Apptainer Config - apart from adding the mandatory cache, I have to use envwhitelist to ensure these the env variables are properly passed into the container. Otherwise, the caches from the containers will default to my home directory or the /tmp folders in the node, which I am not allowed to use.

Before running the jobs, one need to pull and build all the apptainer images first. There is a separate script for this with a README file in the scripts/run_benchmark/wehi_hpc folder.
It is not possible to let the child jobs to pull and build the images by setting ociAutoPull=True. This is because concurrent jobs running the same methods/metrics will overwrite the same image files simultaneously, leading to cache deadlocks. Alternatively, if setting ociAutoPull=False, the head job will be overwhelmed pulling and building the images and just die, even with a 2-day pullTimeout.
Hence the script to pull and build the images are necessary.

I also updated the default retry attempts to 3x. This mitigates random "Bus Errors" that somehow resolves itself after a second or third attempt.

Implementation changes

Changes were made to methods/metrics:

• One control and no control method will only get either samples from one control plus non-control samples or just no control samples.
  They will no longer be given access to other samples to correct or to train the model.
• CytoVI: Now uses a MinMax scaler fitted on Batch 1 post-correction for normalization. Excluded from current run. More work needed to work out the optimal way to run this method.
• Ratio Inconsistent Peaks: Added handling for edge cases where methods return only zero for a given marker/donor/cell type, preventing division by zero when calculating sd.
• HarmonyPy: Removed redundant transpose operation. The latest harmonypy updates no longer require this.
• CytoNorm: Fixed a bug in to mid methods where recompute was incorrectly set to FALSE (now TRUE).
• Perfect Integration: Fixed a bug where string-based batch columns (vs. integers) resulted in only control samples being returned. Note: Most datasets currently use int for batches, which violates our schema. See Issue Batch obs is not always str in datasets #121 for long-term fix.
• BatchAdjust: Fixed a (dumb) requirement where non-control samples need "Batch_" somewhere in the sample name.
• Updated get_obs_var_for_integrated helper to handle type mismatches when overriding string-based batch columns with integer maps for perfect integration.
• Resource Tuning: adjusted time, mem, and cpu requirements:
• Low: Control methods.
• Mid: Most methods/metrics.
• High/Very High: rPCA.
• Update batchadjust, cytonorm to use HPC temp dir if the environment variable is set or else
  default to what is set by viash. See previous section why this is needed.

Checklist before requesting a review

• I have performed a self-review of my code

• Check the correct box. Does this PR contain:

  • Breaking changes
  • New functionality
  • Major changes
  • Minor changes
  • Bug fixes

• Proposed changes are described in the CHANGELOG.md

• CI Tests succeed and look good!

[ghar1821]

Update batchadjust, cytonorm to use HPC temp dir if the environment variable is set or else
default to what is set by viash. See previous section why this is needed.

Is there are reason why only these two methods were adapted?

no i have updated all methods that used temp folders (need FCS files). No longer need the HPC temp dir anymore. See changes i described above.