Errno 24: Too many open files (ubuntu)

[ElinVidevall]

Was asked by wasade to leave my error message here.

deblur workflow --seqs-fp /media/elin/Elins_hdd_3/output/gutMicrobiome/analyses_run1-3_20170308/output_multiple_split_libraries/seqs.fna --output-dir /media/elin/Elins_hdd_3/output/gutMicrobiome/analyses_run1-3_20170308/deblur_output/220bp -m 2 -t 220 -O 6
Traceback (most recent call last):
File "/home/elin/Program/miniconda2/envs/deblurenv/bin/deblur", line 4, in
import('pkg_resources').run_script('deblur==1.0.0', 'deblur')
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/setuptools-27.2.0-py3.5.egg/pkg_resources/init.py", line 744, in run_script
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/setuptools-27.2.0-py3.5.egg/pkg_resources/init.py", line 1499, in run_script
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur-1.0.0-py3.5.egg-info/scripts/deblur", line 678, in
deblur_cmds()
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 716, in call
return self.main(*args, **kwargs)
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur-1.0.0-py3.5.egg-info/scripts/deblur", line 593, in workflow
out_dir_split)
File "/home/elin/Program/miniconda2/envs/deblurenv/lib/python3.5/site-packages/deblur/workflow.py", line 613, in split_sequence_file_on_sample_ids_to_files
outputs[sample] = open(join(outdir, sample + '.fasta'), 'w')
OSError: [Errno 24] Too many open files: '/media/elin/Elins_hdd_3/output/gutMicrobiome/analyses_run1-3_20170308/deblur_output/220bp/split/E.run2.Fa.na.1109.791.S.fasta'

A couple things to note: I specified 6 threads according to the -O 6 parameter but deblur seems to be using only 1. I have 1158 samples in this run (3 MiSeq runs).
OS: Ubuntu 14.04
CPUs: 12
RAM: 48 GB
deblur, version 1.0.0

[wasade]

Thanks @ElinVidevall. I believe this is related, or a duplicate, of #77.

@ElinVidevall, this step is not necessary when running via q2-deblur, which is a possible work around. Another option for the time being may be to partition your data and do multiple runs with Deblur.

The -O 6 use will not kick in during this stage as the splitting step, when coming from QIIME1, is serial right now.

[bassio]

ElinVidevall, could you please send the first few sequences of your fastq files?

I just had this error (on ubuntu), I fixed it by making sure that deblur workflow runs on files generated by split_libraries_fastq.py from qiime1 .. i.e the sample headers consist of sample ids _ underscore then the rest of the identifier.

I suspect the issue is in mishandling of the sequence headers because I ran deblur workflow while keeping intermediate temporary files and the output directory was filled with thousands of files, each named to an identifier of one of the sequences in the fastq.

Perhaps the culprit is in the function split_sequence_file_on_sample_ids_to_files ? And that deblur is always expecting an underscore in the sequence headers?

wasade, what do you think?

[wasade]

Closing as this appears likely to be a duplicate of #77. Easiest work around is q2-deblur.

@bassio, that is most likely the culprit. Note that "_" was a reserved character in QIIME1.