Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mgatk bcall has trouble running #65

Closed
mrland99 opened this issue Jan 23, 2023 · 6 comments
Closed

mgatk bcall has trouble running #65

mrland99 opened this issue Jan 23, 2023 · 6 comments
Assignees
@caleblareau
Labels
bug

Comments

@mrland99
Copy link

Describe the bug
I was trying to run mgatk bcall -i ../YfDogmaSeqMtdna-atac-mito/RC-1652_T4i5_D6_multiome/cr-arc-results/atac_possorted_bam.bam -o regular_mgatk_call -n regular_mgatk_call -c 4 -g hg38 -qc -bt CB -b ./short_barcodes.tsv --snake-stdout but getting errors similar to issues #30, #61 .

I spent a bit of time trying to debug based on other issues but nothing seemed to work. At first I thought it was a snakemake issue based on the other issues above, but I was able to successfully run mgatk bcall -i barcode/test_barcode.bam -n bc2 -o bc2d -bt CB -mb 200 -z on the test data in the same environment.

Any thoughts? Any tips for things I can check to see about my data if it's a data specific issue vs environment/code?

A summary of .log files

Here is output of cat logs/*:

Mon Jan 23 16:54:51 EST 2023: Starting analysis with mgatk
Mon Jan 23 16:54:51 EST 2023: mgatk will process 14 samples
Mon Jan 23 16:54:52 EST 2023: Processing samples with 4 threads
Mon Jan 23 16:55:12 EST 2023: mgatk successfully processed the supplied .bam files
Mon Jan 23 16:55:28 EST 2023: Successfully created final output files
Mon Jan 23 16:55:28 EST 2023: Final bams retained since --keep-qc-bams was specified.
Mon Jan 23 17:03:07 EST 2023: Starting analysis with mgatk
Mon Jan 23 17:03:07 EST 2023: mgatk will process 14 samples
Mon Jan 23 17:03:07 EST 2023: Processing samples with 4 threads
Mon Jan 23 17:03:27 EST 2023: mgatk successfully processed the supplied .bam files
Mon Jan 23 17:03:43 EST 2023: Successfully created final output files
Mon Jan 23 17:03:43 EST 2023: Final bams retained since --keep-qc-bams was specified.
Mon Jan 23 17:04:24 EST 2023: Starting analysis with mgatk
Mon Jan 23 17:04:24 EST 2023: mgatk will process 14 samples
Mon Jan 23 17:04:24 EST 2023: Processing samples with 4 threads
Mon Jan 23 17:04:42 EST 2023: mgatk successfully processed the supplied .bam files
Mon Jan 23 17:04:56 EST 2023: Successfully created final output files
Mon Jan 23 17:04:56 EST 2023: Final bams retained since --keep-qc-bams was specified.
cat: logs/filterlogs: Is a directory
input_directory: 'regular_mgatk_call/temp/barcoded_bams'
output_directory: 'regular_mgatk_call'
script_dir: '/lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk'
fasta_file: '/lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/anno/fasta/hg38.fasta'
mito_chr: 'chrM'
mito_length: '16569'
name: 'regular_mgatk_call'
base_qual: '0'
remove_duplicates: 'True'
handle_overlap: 'False'
low_coverage_threshold: '10'
barcode_tag: 'CB'
umi_barcode: ''
alignment_quality: '0'
emit_base_qualities: 'False'
proper_paired: 'False'
NHmax: '1'
NMmax: '4'
max_javamem: '8000m'
Config file regular_mgatk_call/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job                           count    min threads    max threads
--------------------------  -------  -------------  -------------
all                               1              1              1
make_depth_table                  1              1              1
make_final_sparse_matrices        1              1              1
total                             3              1              1

Select jobs to execute...

[Mon Jan 23 16:55:13 2023]
rule make_depth_table:
    output: regular_mgatk_call/final/regular_mgatk_call.depthTable.txt
    jobid: 1
    reason: Missing output files: regular_mgatk_call/final/regular_mgatk_call.depthTable.txt
    resources: tmpdir=/tmp

Config file regular_mgatk_call/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
[Mon Jan 23 16:55:14 2023]
Finished job 1.
1 of 3 steps (33%) done
Select jobs to execute...

[Mon Jan 23 16:55:14 2023]
rule make_final_sparse_matrices:
    output: regular_mgatk_call/final/regular_mgatk_call.A.txt.gz, regular_mgatk_call/final/regular_mgatk_call.C.txt.gz, regular_mgatk_call/final/regular_mgatk_call.G.txt.gz, regular_mgatk_call/final/regular_mgatk_call.T.txt.gz, regular_mgatk_call/final/regular_mgatk_call.coverage.txt.gz
    jobid: 2
    reason: Missing output files: regular_mgatk_call/final/regular_mgatk_call.C.txt.gz, regular_mgatk_call/final/regular_mgatk_call.coverage.txt.gz, regular_mgatk_call/final/regular_mgatk_call.T.txt.gz, regular_mgatk_call/final/regular_mgatk_call.G.txt.gz, regular_mgatk_call/final/regular_mgatk_call.A.txt.gz
    resources: tmpdir=/tmp

Config file regular_mgatk_call/.internal/parseltongue/snake.gather.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
gzip: regular_mgatk_call/final/regular_mgatk_call.A.txt: No such file or directory
gzip: regular_mgatk_call/final/regular_mgatk_call.C.txt: No such file or directory
gzip: regular_mgatk_call/final/regular_mgatk_call.G.txt: No such file or directory
gzip: regular_mgatk_call/final/regular_mgatk_call.T.txt: No such file or directory
gzip: regular_mgatk_call/final/regular_mgatk_call.coverage.txt: No such file or directory
Waiting at most 5 seconds for missing files.
MissingOutputException in rule make_final_sparse_matrices in file /lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/snake/Snakefile.Gather, line 33:
Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
regular_mgatk_call/final/regular_mgatk_call.A.txt.gz
regular_mgatk_call/final/regular_mgatk_call.C.txt.gz
regular_mgatk_call/final/regular_mgatk_call.G.txt.gz
regular_mgatk_call/final/regular_mgatk_call.T.txt.gz
regular_mgatk_call/final/regular_mgatk_call.coverage.txt.gz
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-01-23T165513.011598.snakemake.log
Config file regular_mgatk_call/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Job stats:
job                   count    min threads    max threads
------------------  -------  -------------  -------------
all                       1              1              1
make_sample_list          1              1              1
process_one_sample       14              1              1
total                    16              1              1

Select jobs to execute...

[Mon Jan 23 16:54:56 2023]
rule process_one_sample:
    input: regular_mgatk_call/.internal/samples/GGGTTTGTCATGGCCA-1.bam.txt
    output: regular_mgatk_call/temp/ready_bam/GGGTTTGTCATGGCCA-1.qc.bam, regular_mgatk_call/temp/ready_bam/GGGTTTGTCATGGCCA-1.qc.bam.bai, regular_mgatk_call/qc/depth/GGGTTTGTCATGGCCA-1.depth.txt, regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.A.txt, regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.C.txt, regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.G.txt, regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.T.txt, regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.coverage.txt
    jobid: 10
    reason: Missing output files: regular_mgatk_call/qc/depth/GGGTTTGTCATGGCCA-1.depth.txt
    wildcards: sample=GGGTTTGTCATGGCCA-1
    resources: tmpdir=/tmp

[Mon Jan 23 16:54:56 2023]
rule process_one_sample:
    input: regular_mgatk_call/.internal/samples/AAACAGCCAATGAGGT-1.bam.txt
    output: regular_mgatk_call/temp/ready_bam/AAACAGCCAATGAGGT-1.qc.bam, regular_mgatk_call/temp/ready_bam/AAACAGCCAATGAGGT-1.qc.bam.bai, regular_mgatk_call/qc/depth/AAACAGCCAATGAGGT-1.depth.txt, regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.A.txt, regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.C.txt, regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.G.txt, regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.T.txt, regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.coverage.txt
    jobid: 15
    reason: Missing output files: regular_mgatk_call/qc/depth/AAACAGCCAATGAGGT-1.depth.txt
    wildcards: sample=AAACAGCCAATGAGGT-1
    resources: tmpdir=/tmp

[Mon Jan 23 16:54:56 2023]
rule process_one_sample:
    input: regular_mgatk_call/.internal/samples/GGTTAGCGTAGCCTAA-1.bam.txt
    output: regular_mgatk_call/temp/ready_bam/GGTTAGCGTAGCCTAA-1.qc.bam, regular_mgatk_call/temp/ready_bam/GGTTAGCGTAGCCTAA-1.qc.bam.bai, regular_mgatk_call/qc/depth/GGTTAGCGTAGCCTAA-1.depth.txt, regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.A.txt, regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.C.txt, regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.G.txt, regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.T.txt, regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.coverage.txt
    jobid: 8
    reason: Missing output files: regular_mgatk_call/qc/depth/GGTTAGCGTAGCCTAA-1.depth.txt
    wildcards: sample=GGTTAGCGTAGCCTAA-1
    resources: tmpdir=/tmp

[Mon Jan 23 16:54:56 2023]
rule process_one_sample:
    input: regular_mgatk_call/.internal/samples/AAACATGCAGGATGGC-1.bam.txt
    output: regular_mgatk_call/temp/ready_bam/AAACATGCAGGATGGC-1.qc.bam, regular_mgatk_call/temp/ready_bam/AAACATGCAGGATGGC-1.qc.bam.bai, regular_mgatk_call/qc/depth/AAACATGCAGGATGGC-1.depth.txt, regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.A.txt, regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.C.txt, regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.G.txt, regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.T.txt, regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.coverage.txt
    jobid: 3
    reason: Missing output files: regular_mgatk_call/qc/depth/AAACATGCAGGATGGC-1.depth.txt
    wildcards: sample=AAACATGCAGGATGGC-1
    resources: tmpdir=/tmp

Config file regular_mgatk_call/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Select jobs to execute...
Config file regular_mgatk_call/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Config file regular_mgatk_call/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Config file regular_mgatk_call/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Using shell: /usr/bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Building DAG of jobs...
Select jobs to execute...
Building DAG of jobs...
Using shell: /usr/bin/bash
Using shell: /usr/bin/bash
Provided cores: 4
Provided cores: 4
Rules claiming more threads will be scaled down.
Rules claiming more threads will be scaled down.
Select jobs to execute...
Select jobs to execute...
Traceback (most recent call last):
  File "/lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/python/oneSample.py", line 82, in <module>
    pysam.index(outputbam)
  File "/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/pysam/utils.py", line 61, in __call__
    save_stdout=kwargs.get("save_stdout", None))
  File "pysam/libcutils.pyx", line 323, in pysam.libcutils._pysam_dispatch
OSError: No such file or directory: 'regular_mgatk_call/temp/ready_bam/GGGTTTGTCATGGCCA-1.qc.bam'
Waiting at most 5 seconds for missing files.
Traceback (most recent call last):
  File "/lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/python/oneSample.py", line 82, in <module>
    pysam.index(outputbam)
  File "/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/pysam/utils.py", line 61, in __call__
    save_stdout=kwargs.get("save_stdout", None))
  File "pysam/libcutils.pyx", line 323, in pysam.libcutils._pysam_dispatch
OSError: No such file or directory: 'regular_mgatk_call/temp/ready_bam/AAACAGCCAATGAGGT-1.qc.bam'
Traceback (most recent call last):
  File "/lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/python/oneSample.py", line 82, in <module>
    pysam.index(outputbam)
  File "/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/pysam/utils.py", line 61, in __call__
    save_stdout=kwargs.get("save_stdout", None))
  File "pysam/libcutils.pyx", line 323, in pysam.libcutils._pysam_dispatch
OSError: No such file or directory: 'regular_mgatk_call/temp/ready_bam/GGTTAGCGTAGCCTAA-1.qc.bam'
Waiting at most 5 seconds for missing files.
Waiting at most 5 seconds for missing files.
Traceback (most recent call last):
  File "/lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/python/oneSample.py", line 82, in <module>
    pysam.index(outputbam)
  File "/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/pysam/utils.py", line 61, in __call__
    save_stdout=kwargs.get("save_stdout", None))
  File "pysam/libcutils.pyx", line 323, in pysam.libcutils._pysam_dispatch
OSError: No such file or directory: 'regular_mgatk_call/temp/ready_bam/AAACATGCAGGATGGC-1.qc.bam'
Waiting at most 5 seconds for missing files.
MissingOutputException in rule process_one_sample in file /lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/snake/Snakefile.Scatter, line 21:
Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
regular_mgatk_call/temp/ready_bam/AAACAGCCAATGAGGT-1.qc.bam
regular_mgatk_call/temp/ready_bam/AAACAGCCAATGAGGT-1.qc.bam.bai
regular_mgatk_call/qc/depth/AAACAGCCAATGAGGT-1.depth.txt
regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.A.txt
regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.C.txt
regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.G.txt
regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.T.txt
regular_mgatk_call/temp/sparse_matrices/AAACAGCCAATGAGGT-1.coverage.txt
MissingOutputException in rule process_one_sample in file /lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/snake/Snakefile.Scatter, line 21:
Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
regular_mgatk_call/temp/ready_bam/GGTTAGCGTAGCCTAA-1.qc.bam
regular_mgatk_call/temp/ready_bam/GGTTAGCGTAGCCTAA-1.qc.bam.bai
regular_mgatk_call/qc/depth/GGTTAGCGTAGCCTAA-1.depth.txt
regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.A.txt
regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.C.txt
regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.G.txt
regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.T.txt
regular_mgatk_call/temp/sparse_matrices/GGTTAGCGTAGCCTAA-1.coverage.txt
MissingOutputException in rule process_one_sample in file /lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/snake/Snakefile.Scatter, line 21:
Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
regular_mgatk_call/temp/ready_bam/GGGTTTGTCATGGCCA-1.qc.bam
regular_mgatk_call/temp/ready_bam/GGGTTTGTCATGGCCA-1.qc.bam.bai
regular_mgatk_call/qc/depth/GGGTTTGTCATGGCCA-1.depth.txt
regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.A.txt
regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.C.txt
regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.G.txt
regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.T.txt
regular_mgatk_call/temp/sparse_matrices/GGGTTTGTCATGGCCA-1.coverage.txt
Shutting down, this might take some time.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Exiting because a job execution failed. Look above for error message
MissingOutputException in rule process_one_sample in file /lila/home/landm/anaconda3/envs/crc_multiome_r/lib/python3.7/site-packages/mgatk/bin/snake/Snakefile.Scatter, line 21:
Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
regular_mgatk_call/temp/ready_bam/AAACATGCAGGATGGC-1.qc.bam
regular_mgatk_call/temp/ready_bam/AAACATGCAGGATGGC-1.qc.bam.bai
regular_mgatk_call/qc/depth/AAACATGCAGGATGGC-1.depth.txt
regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.A.txt
regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.C.txt
regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.G.txt
regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.T.txt
regular_mgatk_call/temp/sparse_matrices/AAACATGCAGGATGGC-1.coverage.txt
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-01-23T165454.927262.snakemake.log
cat: logs/rmdupslogs: Is a directory

Post an ls -lRh of mgatk_output_folder

regular_mgatk_call/:
total 5.0K
drwxr-sr-x 2 landm peer 4.0K Jan 23 16:54 fasta
drwxr-sr-x 2 landm peer 4.0K Jan 23 16:55 final
drwxr-sr-x 4 landm peer 4.0K Jan 23 17:14 logs
drwxr-sr-x 3 landm peer 4.0K Jan 23 17:04 qc
drwxr-sr-x 5 landm peer 4.0K Jan 23 17:04 temp

regular_mgatk_call/fasta:
total 33K
-rw-r--r-- 1 landm peer 17K Jan 23 16:54 chrM.fasta
-rw-r--r-- 1 landm peer  19 Jan 23 16:54 chrM.fasta.fai

regular_mgatk_call/final:
total 128K
-rw-r--r-- 1 landm peer 119K Jan 23 16:54 chrM_refAllele.txt
-rw-r--r-- 1 landm peer    0 Jan 23 16:55 regular_mgatk_call.depthTable.txt

regular_mgatk_call/logs:
total 20K
-rw-r--r-- 1 landm peer 1.3K Jan 23 17:04 base.mgatk.log
drwxr-sr-x 2 landm peer 4.0K Jan 23 16:54 filterlogs
-rw-r--r-- 1 landm peer  623 Jan 23 17:04 regular_mgatk_call.parameters.txt
-rw-r--r-- 1 landm peer 3.8K Jan 23 16:55 regular_mgatk_call.snakemake_gather.log
-rw-r--r-- 1 landm peer  12K Jan 23 16:55 regular_mgatk_call.snakemake_scatter.log
drwxr-sr-x 2 landm peer 4.0K Jan 23 16:54 rmdupslogs

regular_mgatk_call/logs/filterlogs:
total 2.0K
-rw-r--r-- 1 landm peer 18 Jan 23 17:04 AAACAGCCAATGAGGT-1.filter.log
-rw-r--r-- 1 landm peer 18 Jan 23 17:04 AAACATGCAGGATGGC-1.filter.log
-rw-r--r-- 1 landm peer 19 Jan 23 17:04 GGGTTTGTCATGGCCA-1.filter.log
-rw-r--r-- 1 landm peer 18 Jan 23 17:04 GGTTAGCGTAGCCTAA-1.filter.log

regular_mgatk_call/logs/rmdupslogs:
total 0

regular_mgatk_call/qc:
total 1.0K
drwxr-sr-x 2 landm peer 4.0K Jan 23 16:54 quality

regular_mgatk_call/qc/quality:
total 0

regular_mgatk_call/temp:
total 3.0K
drwxr-sr-x 2 landm peer 4.0K Jan 23 16:54 barcoded_bams
drwxr-sr-x 2 landm peer 4.0K Jan 23 16:54 quality
drwxr-sr-x 2 landm peer 4.0K Jan 23 16:54 temp_bam

regular_mgatk_call/temp/barcoded_bams:
total 202K
-rw-r--r-- 1 landm peer  28K Jan 23 17:04 AAACAGCCAACACCTA-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACAGCCAACACCTA-1.bam.bai
-rw-r--r-- 1 landm peer 3.9K Jan 23 17:04 AAACAGCCAATGAGGT-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACAGCCAATGAGGT-1.bam.bai
-rw-r--r-- 1 landm peer  17K Jan 23 17:04 AAACAGCCACAATGTT-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACAGCCACAATGTT-1.bam.bai
-rw-r--r-- 1 landm peer 4.0K Jan 23 17:04 AAACAGCCAGATAGAC-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACAGCCAGATAGAC-1.bam.bai
-rw-r--r-- 1 landm peer 3.4K Jan 23 17:04 AAACAGCCAGCAATAA-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACAGCCAGCAATAA-1.bam.bai
-rw-r--r-- 1 landm peer 3.7K Jan 23 17:04 AAACAGCCAGGTTCAC-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACAGCCAGGTTCAC-1.bam.bai
-rw-r--r-- 1 landm peer 6.1K Jan 23 17:04 AAACATGCAATAATGG-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACATGCAATAATGG-1.bam.bai
-rw-r--r-- 1 landm peer 3.3K Jan 23 17:04 AAACATGCACAATTAC-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACATGCACAATTAC-1.bam.bai
-rw-r--r-- 1 landm peer 4.1K Jan 23 17:04 AAACATGCAGGATGGC-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACATGCAGGATGGC-1.bam.bai
-rw-r--r-- 1 landm peer  16K Jan 23 17:04 AAACATGCAGTTATGT-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 AAACATGCAGTTATGT-1.bam.bai
-rw-r--r-- 1 landm peer 3.5K Jan 23 17:04 CCCTGGACATTGTGAT-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 CCCTGGACATTGTGAT-1.bam.bai
-rw-r--r-- 1 landm peer  41K Jan 23 17:04 GGGTTTGTCATGGCCA-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 GGGTTTGTCATGGCCA-1.bam.bai
-rw-r--r-- 1 landm peer 3.5K Jan 23 17:04 GGTTAGCGTAGCCTAA-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 GGTTAGCGTAGCCTAA-1.bam.bai
-rw-r--r-- 1 landm peer 4.3K Jan 23 17:04 TCATTGCGTACAAAGA-1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 16:54 TCATTGCGTACAAAGA-1.bam.bai

regular_mgatk_call/temp/quality:
total 0

regular_mgatk_call/temp/temp_bam:
total 132K
-rw-r--r-- 1 landm peer 3.8K Jan 23 17:04 AAACAGCCAATGAGGT-1.temp0.bam
-rw-r--r-- 1 landm peer 3.9K Jan 23 17:04 AAACAGCCAATGAGGT-1.temp1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 17:04 AAACAGCCAATGAGGT-1.temp1.bam.bai
-rw-r--r-- 1 landm peer 4.0K Jan 23 17:04 AAACATGCAGGATGGC-1.temp0.bam
-rw-r--r-- 1 landm peer 4.1K Jan 23 17:04 AAACATGCAGGATGGC-1.temp1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 17:04 AAACATGCAGGATGGC-1.temp1.bam.bai
-rw-r--r-- 1 landm peer  40K Jan 23 17:04 GGGTTTGTCATGGCCA-1.temp0.bam
-rw-r--r-- 1 landm peer  41K Jan 23 17:04 GGGTTTGTCATGGCCA-1.temp1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 17:04 GGGTTTGTCATGGCCA-1.temp1.bam.bai
-rw-r--r-- 1 landm peer 3.5K Jan 23 17:04 GGTTAGCGTAGCCTAA-1.temp0.bam
-rw-r--r-- 1 landm peer 3.6K Jan 23 17:04 GGTTAGCGTAGCCTAA-1.temp1.bam
-rw-r--r-- 1 landm peer 1.7K Jan 23 17:04 GGTTAGCGTAGCCTAA-1.temp1.bam.bai

Describe the sequencing assay being analyzed

Dogmaseq data, scRNA and scATAC processed by cellranger.

Clarify if the execution successful on the test data provided in the repository

Test data ran.

Additional Info
Python = 3.7.12
mgatk=0.6.7
snakemake=7.19.1
R = 4.1.1
Java=1.8.0_31
@mrland99 mrland99 added the bug label Jan 23, 2023
@mrland99
Copy link
Author

On second thought I wonder if this is just due to these reads not having enough mito. I'll look into that in the meantime.

@caleblareau
Copy link
Owner

caleblareau commented Jan 24, 2023 via email

@mrland99
Copy link
Author

Yes I am, I believe this is the second step here.

@caleblareau
Copy link
Owner

ok cool-- depending on where you are at, I'd potentially recommend using the coverage based estimation (discussed in the wiki) if you have cells with low coverage-- it tends to perform a bit better once you have the deletions identified and is less computationally intensive (you can do mgatk tenx -> heteroplasmy estimation)

@caleblareau
Copy link
Owner

Closed due to inactivity but LMK if there's more that needs to be done here...

@rah4927
Copy link

rah4927 commented Aug 24, 2023

@caleblareau I am facing the same issue, could we reopen this issue and try to figure out why.

Select jobs to execute...
gzip: mgatk_out/final/mgatk.A.txt: No such file or directory
gzip: mgatk_out/final/mgatk.C.txt: No such file or directory
gzip: mgatk_out/final/mgatk.G.txt: No such file or directory
gzip: mgatk_out/final/mgatk.T.txt: No such file or directory
gzip: mgatk_out/final/mgatk.coverage.txt: No such file or directory
Waiting at most 5 seconds for missing files.
MissingOutputException in rule make_final_sparse_matrices in file /lila/home/sahar1/.conda/envs/mgatk_conda/lib/python3.8/site-packages/mgatk/bin/snake/Snakefile.Gather, line 33:
Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
mgatk_out/final/mgatk.A.txt.gz
mgatk_out/final/mgatk.C.txt.gz
mgatk_out/final/mgatk.G.txt.gz
mgatk_out/final/mgatk.T.txt.gz
mgatk_out/final/mgatk.coverage.txt.gz
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-08-07T115155.393284.snakemake.log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@caleblareau caleblareau

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@caleblareau @mrland99 @rah4927