Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check broken links #89

Closed
wants to merge 6 commits into from
Closed

Check broken links #89

wants to merge 6 commits into from

Conversation

danilodileo
Copy link
Collaborator

@danilodileo danilodileo commented Oct 21, 2024 •
edited
Loading

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
    • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
    • If necessary, also make a PR on the nf-core/magmap branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

I tried to address the #90 bug that stops the pipeline when it finds a broken link from outside source (like ncbi).
Now there is a module "check_broken_links.nf" that checks whether the links are broken or not. if they aren't, the links are then used to stage the genome, otherwise it will be discarded.

erikrikarddaniel
erikrikarddaniel reviewed Oct 21, 2024
View reviewed changes
Copy link
Member

@erikrikarddaniel erikrikarddaniel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little puzzled, and unsure what the best approach is. Your idea is to first check the presence of files using your curl-based module, the let them go to staging? If you'd do it the other way around, first stage, then check, you'd loose the idea with this I suppose?

You're calling the new module for every file. Some sort of loop over a collected channel would be much more effective. You could perhaps have a module that just returns a channel of correct urls after looping over all? (One might end up with long commands, but that could be dealt with by splitting perhaps? I can't see a way of splitting a channel now though; only based on reading files.)

Even better would be if one could do this directly with nextflow/groovy and avoid the module altogether.

Let's discuss tomorrow.

subworkflows/local/sourmash.nf Outdated Show resolved Hide resolved
modules/local/check_broken_links.nf Outdated Show resolved Hide resolved
modules/local/check_broken_links.nf
Comment on lines +19 to +22
# Use curl to check if the URL returns 404
if curl -Is "${genome_fna}" | grep -q "404 Not Found"; then
echo "Broken link: ${genome_fna}"
exit 0 # Exit successfully but don't emit anything
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only works on remote files, right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it does.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 20, 2025 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 6392a85

+| ✅ 214 tests passed       |+
!| ❗   4 tests had warnings |!

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in ro-crate-metadata.json: "description": "

    \n \n <source media="(prefers-color-scheme: dark)" srcset="docs/images/nf-core-magmap_logo_dark.png">\n <img alt="nf-core/magmap" src="docs/images/nf-core-magmap_logo_light.png">\n \n

    GitHub Actions CI Status\nGitHub Actions Linting StatusAWS CICite with Zenodo\nnf-test\n\nNextflow\nrun with conda\nrun with docker\nrun with singularity\nLaunch on Seqera Platform\n\nGet help on SlackFollow on TwitterFollow on MastodonWatch on YouTube\n\n## Introduction\n\nnf-core/magmap is a bioinformatics pipeline that ...\n\n TODO nf-core:\n Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the\n major pipeline sections and the types of output it produces. You're giving an overview to someone new\n to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction\n\n\n Include a figure that guides the user through the major workflow steps. Many nf-core\n workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples. \n Fill in short bullet-pointed list of the default steps in the pipeline 1. Read QC (FastQC)2. Present QC for raw reads (MultiQC)\n\n## Usage\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow.Make sure to test your setup with -profile test before running the workflow on actual data.\n\n Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.\n Explain what rows and columns represent. For instance (please edit as appropriate):\n\nFirst, prepare a samplesheet with your input data that looks as follows:\n\nsamplesheet.csv:\n\ncsv\nsample,fastq_1,fastq_2\nCONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz\n\n\nEach row represents a fastq file (single-end) or a pair of fastq files (paired end).\n\n\n\nNow, you can run the pipeline using:\n\n update the following command to include all required parameters for a minimal example \n\nbash\nnextflow run nf-core/magmap \\\n -profile <docker/singularity/.../institute> \\\n --input samplesheet.csv \\\n --outdir <OUTDIR>\n\n\n> [!WARNING]\n> Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.\n\nFor more details and further functionality, please refer to the usage documentation and the parameter documentation.\n\n## Pipeline output\n\nTo see the results of an example test run with a full size dataset refer to the results tab on the nf-core website pipeline page.\nFor more details about the output files and reports, please refer to the\noutput documentation.\n\n## Credits\n\nnf-core/magmap was originally written by Danilo Di Leo, Emelie Nilsson and Daniel Lundin.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n If applicable, make list of people who have also contributed \n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the contributing guidelines.\n\nFor further information or help, don't hesitate to get in touch on the Slack #magmap channel (you can join with this invite).\n\n## Citations\n\n Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. \n If you use nf-core/magmap for your analysis, please cite it using the following doi: 10.5281/zenodo.XXXXXX Add bibliography of tools and data used in your pipeline \n\nAn extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.\n\nYou can cite the nf-core publication as follows:\n\n> The nf-core framework for community-curated bioinformatics pipelines.\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.\n",
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: If you use nf-core/magmap for your analysis, please cite it using the following doi: 10.5281/zenodo.XXXXXX Add bibliography of tools and data used in your pipeline

✅ Tests passed:

Run details

  • nf-core/tools version 3.1.1
  • Run at 2025-01-20 14:05:41

@danilodileo danilodileo deleted the check-broken-links branch January 29, 2025 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erikrikarddaniel erikrikarddaniel erikrikarddaniel left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@danilodileo @erikrikarddaniel