Skip to content

Troubleshooting

Start with the command and configuration

Run oncotracer check --config /absolute/path/to/run.yml. It lists missing paths and settings without starting analysis. oncotracer doctor --backend conda checks the installed analysis tools.

Message or symptom What to do
setup or check is not recognized Activate the environment created during installation; use command -v oncotracer to check which command your terminal is using
A path does not exist Use the actual absolute path; /data/... and /work/... in examples must be replaced
Updating an existing project Save and check again; setup replaces the configuration by default and retains previous settings under config/backups/
GATK is found but the saved configuration says it is unavailable Save configuration and check detects tools before saving their paths. Explicit paths are preserved; clear an incorrect caller prefix or select a candidate with detect_tools. Multiple complete environments require your choice
Select an existing project without adding a subfolder Choose Use existing project · overwrite settings, then Choose existing project. Save uses that exact folder and backs up its previous settings in config/backups/
Yes is disabled after stopping an existing run Restart setup with the updated version. Dedicated OncoTracer projects can be deleted after Stop and confirmation. Unrelated folders and projects containing original inputs or shared resources remain protected; the dialog explains the reason
auto will not overwrite a file To resume, repeat run; for a different analysis choose a new --config-dir and --outdir
expected CSV header Use the exact header for your command; a four-column Illumina table belongs to setup --samplesheet, not auto --sample-table
Terminal shows > and waits Paste the closing CSV line on its own, or cancel with Ctrl+C; see command basics
More than one R1/R2 match Give each library one unambiguous pair; keep different samples separate
Both POD5 and BAM inputs are configured Keep only the input type you want to use
CPU methylation basecalling is very slow Reuse completed modified-base BAMs with --modbam if available; raw signal can take days on CPU
No FASTQ-selected reads with MM/ML tags Match the FASTQs to BAMs made with modified-base calling, or use raw POD5
no_classifier_probes Check usable human alignment and hg38 probe coordinates. CpG calls elsewhere cannot support a MARLIN prediction

For methylation, install --conda does not install optional Dorado, Modkit, or classifier resources. See the methylation guide.

Start with the exact command, backend, YAML, outdir, and source identity. Avoid deleting reference indexes, .oncotracer-native/state.json, or partial outputs before identifying the failing stage.

1. Verify the installed executable and backend

oncotracer --version
oncotracer provenance --json
oncotracer doctor --backend conda --json

The provenance JSON identifies the executable and source. The doctor JSON adds the backend, all five configured prefixes or image, semantic command/package checks, and whether Nextflow is required for the native CNA runtime (false). The complete Illumina variant branch has a separate Nextflow/resource preflight. A failed check returns a nonzero status.

Use the backend that actually runs the analysis:

oncotracer doctor --backend docker
oncotracer doctor --backend singularity
oncotracer doctor --backend poetry

2. Validate the configuration without computation

CONFIG="$PWD/project/config/illumina.auto.yml"

test -s "$CONFIG"
sed -n '1,220p' "$CONFIG"
oncotracer run \
  --backend conda \
  --config "$CONFIG" \
  --dry-run

The YAML must have one key: value per line. Replace existing values rather than adding duplicate keys. Check absolute paths, the sequencing mode, and a results directory reserved for this analysis.

3. Find the failing native command

Open:

<outdir>/.oncotracer-native/trace.tsv
<outdir>/.oncotracer-native/state.json

The trace records stage, start/end time, exit code, working directory, and a shell-escaped rendering of the original argument array.

OUT="$PWD/project/results"

column -t -s $'\t' "$OUT/.oncotracer-native/trace.tsv" | tail -30
cat "$OUT/06_workflow_summary/workflow_summary.txt" 2>/dev/null || true

Search stage logs and non-empty error files:

OUT="$PWD/project/results"

find "$OUT" -type f \
  \( -name '*.stderr.log' -o -name '*.command.err' -o -name '*error*log' \) \
  -size +0c -print | sort

Many programs write progress to standard error; inspect content and exit status rather than treating any non-empty file as automatic failure.

4. Resume safely

After fixing the reported problem, continue the saved project:

oncotracer run \
  --backend conda \
  --config "$PWD/project/config/illumina.auto.yml"

Completed steps are reused. See resume details.

Advanced: rerun completed steps or resolve an output-ownership error

Use --force only to deliberately invalidate reusable stages:

oncotracer run \
  --backend conda \
  --config "$PWD/project/config/illumina.auto.yml" \
  --force

OncoTracer now rejects a second writer while the first process holds the exact outdir run lock. A new run may claim only an absent or empty output directory. Resume and --force require the existing .oncotracer-native/output-owner.json to match the same exact OncoTracer runtime and canonical location; they never adopt or delete a nonempty unowned directory. Preserve an unowned or mismatched tree and choose a new outdir.

5. Public QuickStart download problems

Repeat the failed curl command from QuickStart 1 or QuickStart 2. --continue-at - resumes an interrupted download. Use the same analysis directory and output filename.

Repeat the guide's md5sum -c block and continue only when every file says OK. If a completed download fails its checksum, preserve that file under a different name and download a fresh copy. Do not analyze a partial or mismatched file. Downloading reads does not create a configuration: follow the separate setup, check, and run steps afterwards.

6. Conda solver or prefix problems

The five environment definitions are independent. Recreate incomplete prefixes:

oncotracer install --conda --force
oncotracer doctor --backend conda

Do not set R_HOME, R_LIBS, R_LIBS_USER, or R_LIBS_SITE to another R installation. Native qDNAseq and ichorCNA stages invoke the exact Rscript in their own prefix with those ambient variables removed. Diagnose the exact prefixes; do not substitute a login-shell command -v result.

An ownership error means the selected --prefix is populated but was not created by this installer, a fixed child has lost or mismatched its marker, a symlink is present in the target path, or an interrupted journal cannot be authenticated. Do not add a marker manually and do not delete the path broadly. Preserve it, inspect the reported path, and choose a new absent or empty dedicated prefix. --force intentionally cannot adopt or erase an unowned directory. If a managed prefix is reported active, allow the named process to finish or select another prefix instead of replacing files beneath it. After an uncatchable interruption, OncoTracer can report an oncotracer-preserved sibling containing an unsealed package-manager tree. It is not automatically deleted: inspect it, confirm it contains no unique or foreign data, and remove only that exact path when appropriate. Rerunning the installer recovers the journal and restores or completes the managed installation.

Check storage and inode availability when environment creation fails:

df -h "$HOME" "$PWD"
df -i "$HOME" "$PWD"

Poetry installation additionally requires Poetry 2.0 or newer and an exact, clean Git checkout matching the executable provenance. Version, dirty-tree, or source-identity failures occur before managed targets are changed.

7. Docker errors

Use the exact image selected for your analysis. These commands match the current installation guide; replace both image values if your project uses another tag or digest. A bare install --docker selects the current default image.

docker info
oncotracer install --docker --image carlosfarkas/oncotracer:terminal-progress-20261022
oncotracer doctor --backend docker --image carlosfarkas/oncotracer:terminal-progress-20261022

If an older launcher accepts intermediate outputs but then reports a different OncoTracer runtime, update the launcher and image together. Open Continue an existing analysis, choose the output project and Use current image, then Load saved settings and check. Do not remove the owner marker or move results to an empty output directory to bypass this check.

Common causes:

  • the invoking user cannot access the Docker daemon;
  • a configured path is outside the roots derived from the YAML;
  • a symlink points outside a mounted root;
  • the filesystem does not permit the invoking UID/GID to write outputs;
  • the requested image tag/digest is unavailable.

Use absolute paths and test project permissions:

PROJECT_DIR="$PWD/project"
mkdir -p "$PROJECT_DIR/results"
touch "$PROJECT_DIR/results/.write-test"
rm "$PROJECT_DIR/results/.write-test"

8. Singularity or Apptainer errors

command -v apptainer || command -v singularity
oncotracer install --singularity
oncotracer doctor --backend singularity

Check that the recorded SIF exists and all YAML paths are bind-visible. On managed HPC systems, ask the administrator about allowed bind roots and cache locations.

The installed SIF must have its adjacent strict .oncotracer.json sidecar. A missing, malformed, path-mismatched, source-mismatched, or checksum-mismatched pair is preserved and rejected; --force does not make an unowned file safe to replace. Choose a new absent --sif destination. For an intact owned pair, --force pulls and validates a same-directory candidate before an atomic swap, so a pull, doctor, provenance, or publication failure leaves the prior image available. Never pre-delete a shared or active SIF to work around an ownership error.

9. Illumina input errors

Automatic Setup supports one consistent layout per run:

  • one single-end FASTQ per sample; or
  • exactly one R1/R2 pair per sample.

Check names and gzip integrity:

READS="$PWD/project/input/fastq"

find "$READS" -maxdepth 1 -type f -print | sort
gzip -t "$READS/Patient_A_R1.fastq.gz"
gzip -t "$READS/Patient_A_R2.fastq.gz"

Common failures include duplicate sample IDs, missing mates, mixed single/paired layouts.

10. Confirm independent Illumina normal outputs

The generated/manual samplesheet preserves each submitted normal status, but normal rows are not reference inputs. Confirm that each expected normal has its own qDNAseq status and result files. This path is for browser-generated setup; auto writes config/illumina.samplesheet.csv, and explicit input sheets can remain elsewhere. Use the illumina_samplesheet path recorded in your YAML:

SHEET="$PWD/project/config/samplesheet.csv"
QDNA="$PWD/project/results/01_samurai_illumina/qdnaseq"

sed -n '1,40p' "$SHEET"
cat "$QDNA/qdnaseq_sample_status.json"
find "$QDNA/bins" "$QDNA/segments" "$QDNA/plots" -maxdepth 1 -type f -print | sort

If a normal row is absent from completed_samples, inspect its entry in the same status JSON. A mathematically invalid sample may fail post-normalization QC without stopping later viable samples; OncoTracer must not silently turn the remaining normals into a panel.

11. ONT library or barcode errors

FASTQ_PASS="$PWD/project/input/fastq_pass"

find "$FASTQ_PASS" -maxdepth 2 -type d -print | sort
find "$FASTQ_PASS" -maxdepth 2 -type f -print | sort | head -40

Confirm that:

  • for named libraries, ont_named_samples: true is saved and ont_folder contains the selected named directories;
  • each named library contains exactly one fastq_pass (directly or under one run directory), with at most one nonempty barcode;
  • for legacy layouts, ont_folder is the barcode parent or one MinKNOW run;
  • every selected library/barcode exists and its sample-name list matches in length and order;
  • completed FASTQs are in the selected pass directories, not failed siblings;
  • ont_min_age_minutes is not excluding completed data unexpectedly.

Review used/skipped/warning logs beneath 01_samurai_ont/logs/.

12. Reference or indexing failures

New setup projects download prepared hg38 indexes by default; they do not build BWA indexes. A deliberately selected local build (--build_reference) can need at least 80 GiB of addressable memory. See reference choices.

Inspect the lpwgs_root recorded in your YAML. For default browser/terminal setup it is PROJECT/reference; auto or custom configurations may use another root. Replace REFERENCE_ROOT below with that exact directory:

REFERENCE_ROOT="$PWD/project/reference"
free -h
df -h "$PWD/project"
find "$REFERENCE_ROOT/references" -maxdepth 3 -type f -ls 2>/dev/null | head -50
find "$REFERENCE_ROOT/.oncotracer/reference-cache" -maxdepth 3 -type f -ls 2>/dev/null | head -50

An existing REFERENCE_ROOT/references/samurai_hg38 or REFERENCE_ROOT/references/samurai_ichorcna_hg38_500kb is external and read-only. A checksum, manifest, physical-lock, tool-identity, layout, or symlink error there stops the run without repair. This includes an otherwise usable FASTA/index directory created by another tool but lacking OncoTracer's exact .oncotracer manifests and locks. Do not manufacture or copy those records. Coordinate shared-reference maintenance outside OncoTracer, or use a new project root so OncoTracer can create its own content-addressed cache. Never delete a valid shared reference during another active run.

qDNAseq annotations are published under REFERENCE_ROOT/.oncotracer/reference-cache/qdnaseq-hg38-<binsize>kb-*/generations/. A failed build leaves no current-generation pointer and is safe to rerun. A changed published generation fails closed rather than being repaired in place. OncoTracer deliberately ignores and preserves the older REFERENCE_ROOT/.oncotracer/qdnaseq-bin-data location.

13. Classifier or GISTIC2 failures

First confirm the CNA-only stages completed:

OUT="$PWD/project/results"

test -s "$OUT/03_cna_codification/cna_events.tsv"
test -s "$OUT/03_cna_codification/cna_cytogenomic_notation.tsv"

Then inspect:

find "$OUT/05_cna_classifier" -maxdepth 3 -type f -print | sort | head -100
cat "$OUT/05_cna_classifier/native_classifier_summary.json"

Keep gistic_required: false unless the study requires a fatal GISTIC2 branch. Start with deterministic offline settings before enabling model/network enrichment.

A completed GISTIC2 run can produce an all_lesions.conf_90.txt containing only the header: no significant focal lesions passed the selected thresholds. This is a valid result. Check diagnostics/gistic/gistic2_status.tsv and diagnostics/gistic_parsed/gistic_parse_metrics.json under the classifier folder; the parser retains the analyzed sample count even when n_lesions is zero.

GISTIC2 is off by default. Browser setup disables it for fewer than two assigned samples, and terminal setup saves run_gistic: false for a single sample. Native YAML runs also require explicit opt-in and skip cohorts below gistic_min_samples (default: 2) before looking for GISTIC2 or downloading its reference.

For a failed variant branch, run oncotracer check --variant-tools --config /path/to/run.yml. The browser and setup --run perform this executable check automatically. Ordinary check still validates configuration on machines without caller tools. Managed Illumina runs use the verified variant environment saved by oncotracer install --conda (or --poetry), including bcftools 1.23.1 and HTSlib 1.24. Re-run installation to repair a missing or incompatible managed variant environment; unrelated Conda environments are not selected as replacements. An unmanaged host installation can explicitly select variant_tool_prefix. Set variant_strelka_prefix for a separate Strelka2 environment. An explicit variant prefix must contain all requested shared tools and callers. See the variant setup guide for installation and resource selection.

14. Confirm stable release identity

oncotracer provenance --json
sha256sum oncotracer 2>/dev/null || true

For a stable asset, source_commit, source_sha256, and binary checksum must agree with release-provenance.json and SHA256SUMS from the same release.

15. Standalone payload-cache safety

The copied executable normally manages its own content-addressed directory below the XDG cache. If ONCOTRACER_PAYLOAD_CACHE is set accidentally, return to the safe default before retrying:

unset ONCOTRACER_PAYLOAD_CACHE
oncotracer --version

The override is an advanced integration option. It must name an absent or empty dedicated path, or a complete cache already owned by that exact executable archive. Never point it at a home directory, XDG root, shared reference, analysis, or scientific-data directory. OncoTracer rejects symlinked, unowned, mismatched, or unexpectedly populated paths and preserves them rather than recursively deleting them. Inspect an ownership or integrity error; do not respond with a broad cache deletion.

The default cache layout is:

${XDG_CACHE_HOME:-$HOME/.cache}/oncotracer/2.1.0/<executable-sha256>/payload

Every --dry-run uses a separate temporary payload and removes it on success, error, or interruption. A dry-run must not create persistent XDG state, outputs, environments, images, or SIF files.

16. Report a reproducible problem

Include only de-identified information:

  • exact OncoTracer version and provenance JSON;
  • backend and doctor JSON;
  • YAML with sensitive paths/sample names redacted consistently;
  • failing trace row and relevant stage log;
  • operating system, available memory/storage, and container/runtime version;
  • whether the problem reproduces in QuickStart 1 or 2.

Never attach identifiable patient data or private pathology text to a public issue.