'Pipestat compatible': False, 'Commands submitted': '0 of 2', 'Jobs submitted': 0

[Melody-cell]

Thanks for the great tool!
When i run this:
looper run /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/examples/tutorial/tutorial_refgenie.yaml

it appears this error:
/home/wangmengj/software/anaconda3/envs/pepatac/lib/python3.9/site-packages/pydantic/_migration.py:283: UserWarning: pydantic.error_wrappers:ValidationError has been moved to pydantic:ValidationError.
warnings.warn(f'{import_path} has been moved to {new_location}.')
Looper version: 1.6.0
Command: run
This PEP configures looper through the project config. This approach is deprecated and will be removed in future versions. Please use a looper config file. For more information see looper.databio.org/en/latest/looper-config
Pipestat compatible: False

[1 of 2] sample: tutorial1; pipeline: PEPATAC

Found existing status: /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline/tutorial1/PEPATAC_tutorial1_failed.flag. Skipping sample. Use rerun to ignore failed status.
Calling pre-submit function: refgenconf.looper_refgenie_populate
Refgenie asset (hg38/bowtie2_index) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/ensembl_gtf) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/ensembl_rb) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/fasta) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/feat_annotation) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/refgene_anno) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (rCRSd/bowtie2_index) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (rCRSd/fasta) tag not specified in refgenie.tag_overrides section. Using the default tag: default
/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/pipelines/pepatac.py --output-parent /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline --cores 4 --mem 8000 --sample-name tutorial1 --input /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/examples/data/tutorial1_r1.fastq.gz --input2 /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/examples/data/tutorial1_r2.fastq.gz --single-or-paired paired --genome hg38 --chrom-sizes /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/fasta/default/hg38.chrom.sizes --TSS-name /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/refgene_anno/default/hg38_TSS.bed --anno-name /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/feat_annotation/default/hg38_annotations.bed.gz --trimmer skewer --aligner "bowtie2" --genome-index /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/bowtie2_index/default/. --prealignment-index rCRSd=/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/rCRSd/bowtie2_index/default/. --deduplicator samblaster --peak-caller "macs3" --peak-type fixed --extend 250 --genome-size "2.7e9" --frip-ref-peaks None --fasta /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/fasta/default/hg38.fa
Writing script to /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/submission/PEPATAC_tutorial1.sub

[2 of 2] sample: tutorial2; pipeline: PEPATAC

Found existing status: /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline/tutorial2/PEPATAC_tutorial2_failed.flag. Skipping sample. Use rerun to ignore failed status.
Calling pre-submit function: refgenconf.looper_refgenie_populate
Refgenie asset (hg38/bowtie2_index) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/ensembl_gtf) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/ensembl_rb) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/fasta) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/feat_annotation) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (hg38/refgene_anno) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (rCRSd/bowtie2_index) tag not specified in refgenie.tag_overrides section. Using the default tag: default
Refgenie asset (rCRSd/fasta) tag not specified in refgenie.tag_overrides section. Using the default tag: default
/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/pipelines/pepatac.py --output-parent /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline --cores 4 --mem 8000 --sample-name tutorial2 --input /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/examples/data/tutorial2_r1.fastq.gz --input2 /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/examples/data/tutorial2_r2.fastq.gz --single-or-paired paired --genome hg38 --chrom-sizes /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/fasta/default/hg38.chrom.sizes --TSS-name /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/refgene_anno/default/hg38_TSS.bed --anno-name /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/feat_annotation/default/hg38_annotations.bed.gz --trimmer skewer --aligner "bowtie2" --genome-index /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/bowtie2_index/default/. --prealignment-index rCRSd=/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/rCRSd/bowtie2_index/default/. --deduplicator samblaster --peak-caller "macs3" --peak-type fixed --extend 250 --genome-size "2.7e9" --frip-ref-peaks None --fasta /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/04_genome_folder/alias/hg38/fasta/default/hg38.fa
Writing script to /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/submission/PEPATAC_tutorial2.sub

Looper finished
Samples valid for job generation: 2 of 2
Commands submitted: 0 of 2
Jobs submitted: 0
{'Pipestat compatible': False, 'Commands submitted': '0 of 2', 'Jobs submitted': 0}

Did anyone know how to solve it?

Best,
Melody

[donaldcampbelljr]

This may offer a clue:

Found existing status: /home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline/tutorial1/PEPATAC_tutorial1_failed.flag. 
Skipping sample. 
Use rerun to ignore failed status.

looper will not resubmit if it finds a failed flag unless you use the -i argument to ignore flags. You can also clear the output folder (if the pipeline did not get very far) and attempt to re-run it.

Alternatively you can use the rerun command. More info on commands here: Looper commands

[Melody-cell]

@donaldcampbelljr ,
Thank you for you reply, it appeared a new error:

Cutting parallel chroms in half to accommodate two tracks.
Registering input file: '/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline/tutorial2Ctrl/aligned_danR11/tutorial2Ctrl_sort_dedup.bam'
Temporary files will be stored in: '/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline/tutorial2Ctrl/aligned_danR11_exact/tmp_tutorial2Ctrl_sort_dedup_cuttrace_j6k__5gb'
Processing with 30 cores...
Reduce step (merge files)...
Merging 1880 files into output file: '/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline/tutorial2Ctrl/aligned_danR11_exact/tutorial2Ctrl_exact.bw'
Merging 1880 files into output file: '/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/processed/results_pipeline/tutorial2Ctrl/aligned_danR11/tutorial2Ctrl_smooth.bw'
needLargeMem: trying to allocate 22949658888 bytes (limit: 17179869184)
Traceback (most recent call last):
File "/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/tools/bamSitesToWig.py", line 397, in
ct.combine(good_chromosomes)
File "/home/wangmengj/workspace/project06_bam_ATAC_qc/01_example_pepatac/pepatac_tutorial/tools/pepatac/tools/bamSitesToWig.py", line 320, in combine
p = subprocess.call(cmd, shell=True)
File "/home/wangmengj/software/anaconda3/envs/pepatac/lib/python3.9/subprocess.py", line 349, in call
with Popen(*popenargs, **kwargs) as p:
File "/home/wangmengj/software/anaconda3/envs/pepatac/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/home/wangmengj/software/anaconda3/envs/pepatac/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: '/bin/sh'

What can I do to solve it?

[donaldcampbelljr]

Based on the error given (OSError: [Errno 7] ), I suspect you may be attempting to process too many files at once?

Processing with 30 cores...
Reduce step (merge files)...
Merging 1880 files into output file:

I'm not a 100% certain. Can you paste the entire log from the PEPATAC_log.md?