Improper folder specification; file missing / extra file present. See documentation Calls: importMito -> checkGrep. Execution halted

[Srishtyaku19]

### Dear Authors @caleblareau , @vincent6liu @ruochiz @AntonJMLarsson
When I'm trying to run the mgatk tenx command on my data, it collapses with the error code submitted below, and also, no snakemake.log.txt file is being populated with results, it's an empty file, with nothing.
Is there any help that I could retrieve on this issue.

**Please can you suggest how to resolve this ISSUE. It would be a great help. As I'm stuck with this since 1.5 months.

I tried the dependencies version compatibilities, like downgrading the pulp version, checking the Picard.jar installation, checking the JAVA compatibility, and also keeping the flag --keep duplicates, so my bam files are generated. but it stucks with the error above, and it crashes in the middle of the pipeline, so my snakelike .log file is empty.**

(mgatk) srishty@collin-r-server:/mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked$ ./run_mgatk.sh
Thu Jan 30 17:23:09 GMT 2025: mgatk v0.7.0
Thu Jan 30 17:23:10 GMT 2025: Found bam file: /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/outs/possorted_bam.bam for genotyping.
Thu Jan 30 17:23:10 GMT 2025: Found file of barcodes to be parsed: /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/outs/filtered_peak_bc_matrix/barcodes.tsv
Thu Jan 30 17:23:10 GMT 2025: User specified mitochondrial genome matches .bam file
Thu Jan 30 17:30:34 GMT 2025: Finished determining/splitting barcodes for genotyping.
Thu Jan 30 17:30:34 GMT 2025: Genotyping samples with 40 threads
Config file /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/.internal/parseltongue/snake.scatter.yaml is extended by additional config specified via the command line.
Assuming unrestricted shared filesystem usage.
host: collin-r-server
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 40
Rules claiming more threads will be scaled down.
Job stats:
job count

---

all 1
call_variants 1
make_depth_table 1
make_final_sparse_matrices 1
process_one_slice 40
total 44

Select jobs to execute...
Execute 40 jobs...

host: collin-r-server
Error in checkGrep(grep(".A.txt", files)) :
Improper folder specification; file missing / extra file present. See documentation
Calls: importMito -> checkGrep
Execution halted
(mgatk) srishty@collin-r-server:/mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked$ [E::hts_open_format] Failed to open file "/mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/temp_bam/barcodes.40.temp0.bam" : No such file or directory
RuleException:
SamtoolsError in file /home/srishty/miniconda3/envs/mgatk/lib/python3.12/site-packages/mgatk/bin/snake/Snakefile.tenx, line 97:
'samtools returned with error 1: stdout=, stderr=samtools sort: can't open "/mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/temp_bam/barcodes.40.temp0.bam": No such file or directory\n'
File "/home/srishty/miniconda3/envs/mgatk/lib/python3.12/site-packages/mgatk/bin/snake/Snakefile.tenx", line 97, in __rule_process_one_slice
File "/home/srishty/miniconda3/envs/mgatk/lib/python3.12/site-packages/pysam/utils.py", line 83, in call
[Thu Jan 30 17:31:14 2025]
Error in rule process_one_slice:
jobid: 0
input: /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/.internal/samples/barcodes.40.bam.txt
output: /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/qc/depth/barcodes.40.depth.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.A.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.C.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.G.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.T.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.coverage.txt

Exiting because a job execution failed. Look above for error message
WorkflowError:
At least one job did not complete successfully.
[Thu Jan 30 17:31:14 2025]
Error in rule process_one_slice:
jobid: 25
input: /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/.internal/samples/barcodes.40.bam.txt
output: /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/qc/depth/barcodes.40.depth.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.A.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.C.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.G.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.T.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.40.coverage.txt

[E::hts_open_format] Failed to open file "/mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/temp_bam/barcodes.39.temp0.bam" : No such file or directory
RuleException:
SamtoolsError in file /home/srishty/miniconda3/envs/mgatk/lib/python3.12/site-packages/mgatk/bin/snake/Snakefile.tenx, line 97:
'samtools returned with error 1: stdout=, stderr=samtools sort: can't open "/mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/temp_bam/barcodes.39.temp0.bam": No such file or directory\n'
File "/home/srishty/miniconda3/envs/mgatk/lib/python3.12/site-packages/mgatk/bin/snake/Snakefile.tenx", line 97, in __rule_process_one_slice
File "/home/srishty/miniconda3/envs/mgatk/lib/python3.12/site-packages/pysam/utils.py", line 83, in call
[Thu Jan 30 17:31:21 2025]
Error in rule process_one_slice:
jobid: 0
input: /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/.internal/samples/barcodes.39.bam.txt
output: /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/qc/depth/barcodes.39.depth.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.39.A.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.39.C.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.39.G.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.39.T.txt, /mnt/claw-raid/Srishty/mtscATAC/results/masked/241121-PBMC_results_masked/mgatk_output/temp/sparse_matrices/barcodes.39.coverage.txt

Exiting because a job execution failed. Look above for error message
WorkflowError:
At least one job did not complete successfully.

[AbhinavSoni20191995]

Getting the same error, hopefully some solution will be cool, also stuck for last few days.

[caleblareau]

Can you clarify if you are able to run mgatk tenx on the test data?

[AbhinavSoni20191995]

This is the yml file for the conda environment which works for me at the moment, and does not show any error:

name: /home/h4/abso493b/$/data/horse/ws/abso493b-conda_mgatk/mgatk_kernel
channels:

• bioconda
• conda-forge
  dependencies:
• _libgcc_mutex=0.1=conda_forge
• _openmp_mutex=4.5=2_gnu
• _r-mutex=1.0.1=anacondar_1
• alsa-lib=1.2.13=hb9d3cd8_0
• argcomplete=3.5.2=pyhd8ed1ab_0
• binutils_impl_linux-64=2.43=h4bf12b8_2
• bioconductor-biobase=2.66.0=r44h3df3fcb_0
• bioconductor-biocgenerics=0.52.0=r44hdfd78af_3
• bioconductor-data-packages=20250105.1=hdfd78af_0
• bioconductor-delayedarray=0.32.0=r44h3df3fcb_1
• bioconductor-genomeinfodb=1.42.0=r44hdfd78af_2
• bioconductor-genomeinfodbdata=1.2.13=r44hdfd78af_0
• bioconductor-genomicranges=1.58.0=r44h3df3fcb_2
• bioconductor-iranges=2.40.0=r44h3df3fcb_2
• bioconductor-matrixgenerics=1.18.0=r44hdfd78af_0
• bioconductor-s4arrays=1.6.0=r44h3df3fcb_0
• bioconductor-s4vectors=0.44.0=r44h3df3fcb_2
• bioconductor-sparsearray=1.6.0=r44h3df3fcb_0
• bioconductor-summarizedexperiment=1.36.0=r44hdfd78af_0
• bioconductor-ucsc.utils=1.2.0=r44h9ee0642_1
• bioconductor-xvector=0.46.0=r44h15a9599_2
• bioconductor-zlibbioc=1.52.0=r44h3df3fcb_2
• brotli=1.1.0=hb9d3cd8_2
• brotli-bin=1.1.0=hb9d3cd8_2
• bwidget=1.10.1=ha770c72_0
• bzip2=1.0.8=h4bc722e_7
• c-ares=1.34.4=hb9d3cd8_0
• ca-certificates=2025.1.31=hbcca054_0
• cairo=1.18.2=h3394656_1
• contourpy=1.3.1=py311hd18a35c_0
• curl=8.11.1=h332b0f4_0
• cycler=0.12.1=pyhd8ed1ab_1
• cyrus-sasl=2.1.27=h54b06d7_7
• dbus=1.13.6=h5008d03_3
• double-conversion=3.3.0=h59595ed_0
• expat=2.6.4=h5888daf_0
• font-ttf-dejavu-sans-mono=2.37=hab24e00_0
• font-ttf-inconsolata=3.000=h77eed37_0
• font-ttf-source-code-pro=2.038=h77eed37_0
• font-ttf-ubuntu=0.83=h77eed37_3
• fontconfig=2.15.0=h7e30c49_1
• fonts-conda-ecosystem=1=0
• fonts-conda-forge=1=0
• fonttools=4.56.0=py311h2dc5d0c_0
• freetype=2.12.1=h267a509_2
• fribidi=1.0.10=h36c2ea0_0
• gcc_impl_linux-64=14.2.0=h6b349bd_1
• gfortran_impl_linux-64=14.2.0=hc73f493_1
• graphite2=1.3.13=h59595ed_1003
• gsl=2.7=he838d99_0
• gxx_impl_linux-64=14.2.0=h2c03514_1
• harfbuzz=10.2.0=h4bba637_0
• icu=75.1=he02047a_0
• jq=1.7.1=hd590300_0
• kernel-headers_linux-64=3.10.0=he073ed8_18
• keyutils=1.6.1=h166bdaf_0
• krb5=1.21.3=h659f571_0
• lcms2=2.17=h717163a_0
• ld_impl_linux-64=2.43=h712a8e2_2
• lerc=4.0.0=h27087fc_0
• libblas=3.9.0=28_h59b9bed_openblas
• libbrotlicommon=1.1.0=hb9d3cd8_2
• libbrotlidec=1.1.0=hb9d3cd8_2
• libbrotlienc=1.1.0=hb9d3cd8_2
• libcblas=3.9.0=28_he106b2a_openblas
• libclang-cpp19.1=19.1.7=default_hb5137d0_1
• libclang13=19.1.7=default_h9c6a7e4_1
• libcups=2.3.3=h4637d8d_4
• libcurl=8.11.1=h332b0f4_0
• libdeflate=1.23=h4ddbbb0_0
• libdrm=2.4.124=hb9d3cd8_0
• libedit=3.1.20250104=pl5321h7949ede_0
• libegl=1.7.0=ha4b6fd6_2
• libev=4.33=hd590300_2
• libexpat=2.6.4=h5888daf_0
• libffi=3.4.2=h7f98852_5
• libgcc=14.2.0=h77fa898_1
• libgcc-devel_linux-64=14.2.0=h41c2201_101
• libgcc-ng=14.2.0=h69a702a_1
• libgfortran=14.2.0=h69a702a_1
• libgfortran-ng=14.2.0=h69a702a_1
• libgfortran5=14.2.0=hd5240d6_1
• libgl=1.7.0=ha4b6fd6_2
• libglib=2.82.2=h2ff4ddf_1
• libglvnd=1.7.0=ha4b6fd6_2
• libglx=1.7.0=ha4b6fd6_2
• libgomp=14.2.0=h77fa898_1
• libiconv=1.17=hd590300_2
• libjpeg-turbo=3.0.0=hd590300_1
• liblapack=3.9.0=28_h7ac8fdf_openblas
• libllvm19=19.1.7=ha7bfdaf_1
• liblzma=5.6.4=hb9d3cd8_0
• libnghttp2=1.64.0=h161d5f1_0
• libnsl=2.0.1=hd590300_0
• libntlm=1.8=hb9d3cd8_0
• libopenblas=0.3.28=pthreads_h94d23a6_1
• libopengl=1.7.0=ha4b6fd6_2
• libpciaccess=0.18=hd590300_0
• libpng=1.6.46=h943b412_0
• libpq=17.2=h3b95a9b_1
• libsanitizer=14.2.0=h2a3dede_1
• libsqlite=3.48.0=hee588c1_1
• libssh2=1.11.1=hf672d98_0
• libstdcxx=14.2.0=hc0a3c3a_1
• libstdcxx-devel_linux-64=14.2.0=h41c2201_101
• libstdcxx-ng=14.2.0=h4852527_1
• libtiff=4.7.0=hd9ff511_3
• libuuid=2.38.1=h0b41bf4_0
• libwebp-base=1.5.0=h851e524_0
• libxcb=1.17.0=h8a09558_0
• libxcrypt=4.4.36=hd590300_1
• libxkbcommon=1.8.0=hc4a0caf_0
• libxml2=2.13.5=h8d12d68_1
• libxslt=1.1.39=h76b75d6_0
• libzlib=1.3.1=hb9d3cd8_2
• make=4.4.1=hb9d3cd8_2
• matplotlib=3.10.0=py311h38be061_0
• matplotlib-base=3.10.0=py311h2b939e6_0
• munkres=1.1.4=pyh9f0ad1d_0
• mysql-common=9.0.1=h266115a_4
• mysql-libs=9.0.1=he0572af_4
• ncurses=6.5=h2d0b736_3
• numpy=2.2.2=py311h5d046bc_0
• oniguruma=6.9.10=hb9d3cd8_0
• openjpeg=2.5.3=h5fbd93e_0
• openldap=2.6.9=he970967_0
• openssl=3.4.0=h7b32b05_1
• packaging=24.2=pyhd8ed1ab_2
• pango=1.56.1=h861ebed_0
• pcre2=10.44=hba22ea6_2
• pillow=11.1.0=py311h1322bbf_0
• pip=25.0=pyh8b19718_0
• pixman=0.44.2=h29eaf8c_0
• pthread-stubs=0.4=hb9d3cd8_1002
• pyparsing=3.2.1=pyhd8ed1ab_0
• pyside6=6.8.2=py311h9053184_0
• python=3.11.11=h9e4cc4f_1_cpython
• python-dateutil=2.9.0.post0=pyhff2d567_1
• python_abi=3.11=5_cp311
• pyyaml=6.0.2=py311h2dc5d0c_2
• qhull=2020.2=h434a139_5
• qt6-main=6.8.2=h588cce1_0
• r-abind=1.4_5=r44hc72bb7e_1006
• r-askpass=1.2.1=r44h2b5f3a1_0
• r-base=4.4.2=hc737e89_2
• r-crayon=1.5.3=r44hc72bb7e_1
• r-curl=6.0.1=r44h2700575_0
• r-httr=1.4.7=r44hc72bb7e_1
• r-jsonlite=1.8.9=r44h2b5f3a1_0
• r-lattice=0.22_6=r44hb1dbf0f_1
• r-matrix=1.7_2=r44h2ae2be5_0
• r-matrixstats=1.5.0=r44h2b5f3a1_0
• r-mime=0.12=r44hb1dbf0f_3
• r-openssl=2.2.2=r44he8289e2_0
• r-r6=2.5.1=r44hc72bb7e_3
• r-sys=3.4.3=r44h2b5f3a1_0
• readline=8.2=h8228510_1
• sed=4.8=he412f7d_0
• setuptools=75.8.0=pyhff2d567_0
• six=1.17.0=pyhd8ed1ab_0
• sysroot_linux-64=2.17=h0157908_18
• tk=8.6.13=noxft_h4845f30_101
• tktable=2.10=h8bc8fbc_6
• toml=0.10.2=pyhd8ed1ab_1
• tomlkit=0.13.2=pyha770c72_1
• tornado=6.4.2=py311h9ecbd09_0
• unicodedata2=16.0.0=py311h9ecbd09_0
• wayland=1.23.1=h3e06ad9_0
• wheel=0.45.1=pyhd8ed1ab_1
• xcb-util=0.4.1=hb711507_2
• xcb-util-cursor=0.1.5=hb9d3cd8_0
• xcb-util-image=0.4.0=hb711507_2
• xcb-util-keysyms=0.4.1=hb711507_0
• xcb-util-renderutil=0.3.10=hb711507_0
• xcb-util-wm=0.4.2=hb711507_0
• xkeyboard-config=2.43=hb9d3cd8_0
• xmltodict=0.14.2=pyhd8ed1ab_1
• xorg-libice=1.1.2=hb9d3cd8_0
• xorg-libsm=1.2.5=he73a12e_0
• xorg-libx11=1.8.11=h4f16b4b_0
• xorg-libxau=1.0.12=hb9d3cd8_0
• xorg-libxcomposite=0.4.6=hb9d3cd8_2
• xorg-libxcursor=1.2.3=hb9d3cd8_0
• xorg-libxdamage=1.1.6=hb9d3cd8_0
• xorg-libxdmcp=1.1.5=hb9d3cd8_0
• xorg-libxext=1.3.6=hb9d3cd8_0
• xorg-libxfixes=6.0.1=hb9d3cd8_0
• xorg-libxi=1.8.2=hb9d3cd8_0
• xorg-libxrandr=1.5.4=hb9d3cd8_0
• xorg-libxrender=0.9.12=hb9d3cd8_0
• xorg-libxt=1.3.1=hb9d3cd8_0
• xorg-libxtst=1.2.5=hb9d3cd8_3
• xorg-libxxf86vm=1.1.6=hb9d3cd8_0
• yaml=0.2.5=h7f98852_2
• yq=3.4.3=pyhd8ed1ab_1
• zstd=1.5.6=ha6fb4c9_0
• pip:
  • appdirs==1.4.4
  • argparse-dataclass==2.0.0
  • attrs==25.1.0
  • biopython==1.85
  • certifi==2025.1.31
  • charset-normalizer==3.4.1
  • click==8.1.8
  • conda-inject==1.3.2
  • configargparse==1.7
  • connection-pool==0.0.3
  • datrie==0.8.2
  • docutils==0.21.2
  • dpath==2.2.0
  • fastjsonschema==2.21.1
  • gitdb==4.0.12
  • gitpython==3.1.44
  • humanfriendly==10.0
  • idna==3.10
  • immutables==0.21
  • iniconfig==2.0.0
  • jinja2==3.1.5
  • jsonschema==4.23.0
  • jsonschema-specifications==2024.10.1
  • jupyter-core==5.7.2
  • kiwisolver==1.4.8
  • markupsafe==3.0.2
  • mgatk==0.7.0
  • nbformat==5.10.4
  • optparse-pretty==0.1.1
  • pandas==2.2.3
  • plac==1.4.3
  • platformdirs==4.3.6
  • pluggy==1.5.0
  • psutil==6.1.1
  • pulp==2.9.0
  • pysam==0.23.0
  • pytest==8.3.4
  • pytz==2025.1
  • referencing==0.36.2
  • regex==2024.11.6
  • requests==2.32.3
  • reretry==0.11.8
  • rpds-py==0.22.3
  • ruamel-yaml==0.18.10
  • ruamel-yaml-clib==0.2.12
  • smart-open==7.1.0
  • smmap==5.0.2
  • snakemake==8.27.1
  • snakemake-interface-common==1.17.4
  • snakemake-interface-executor-plugins==9.3.3
  • snakemake-interface-report-plugins==1.1.0
  • snakemake-interface-storage-plugins==3.3.0
  • tabulate==0.9.0
  • throttler==1.2.2
  • traitlets==5.14.3
  • typing-extensions==4.12.2
  • tzdata==2025.1
  • urllib3==2.3.0
  • wrapt==1.17.2
  • yte==1.5.7
    prefix: /home/h4/abso493b/$/data/horse/ws/abso493b-conda_mgatk/mgatk_kernel

[biobenkj]

Hijacking the thread - @caleblareau I was able to run the test data (test_barcode.bam) and think it may be related to installing mgatk as part of anaconda/mamba (calling python and snakemake from within conda). What has worked for me is cloning the repo and building/installing from source or installing mgatk using pip from a python installation outside of anaconda/mamba. The R components can absolutely still be in your PATH through anaconda/mamba. Not super clear on the issue yet, but will update more after working through this error with a grad student in the lab.