Implementation of nf-gpt into crisprseq pipeline. #193

LeonHornich · 2024-09-02T11:24:14Z

This is a very early stage implementation of the nf-gpt plugin into the pipeline. It can currently handles a list of genes extracted from the drugZ, bagel2 or mle module output and parses it to the plugin.

You can run the changes using:
export OPENAI_API_KEY=your-api-key
nextflow run . -profile test_screening,docker --outdir test -dump-channels --gpt_interpretation drugz,mle,bagel2,rra -resume

DISCLAIMER! You will need a functioning Open AI api key. for this to work.

Since this is very early version and only comes with the bare minimum of functionality there are a lot of adjustment that have to me made:

add option to generate multiple different responses per model and query.

Keep in mind this implementation does not aim to achieve 100% correct results from the gpt model, but instead is to build a foundational implementation for (future) LLMs into a nf-core pipeline.

If you have any suggestions or requested changes please let me know and i will try to adjust the code.

github-actions · 2024-09-02T11:24:28Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.0.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

github-actions · 2024-09-02T11:41:50Z

`nf-core lint` overall result: Failed ❌

Posted for pipeline commit 05c4d56

+| ✅ 235 tests passed       |+
#| ❔   3 tests were ignored |#
!| ❗   5 tests had warnings |!
-| ❌   1 tests failed       |-

❌ Test failures:

actions_ci - Minimum pipeline NF version '23.04.0' is not tested in '.github/workflows/ci.yml'

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 2.3.0
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!

❔ Tests ignored:

files_exist - File is ignored: conf/test.config
files_exist - File is ignored: conf/test_full.config
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-crisprseq_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: docs/images/nf-core-crisprseq_logo_light.png
files_exist - File found: docs/images/nf-core-crisprseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-crisprseq_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowCrisprseq.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.umi_bin_size= 1
nextflow_config - Config default value correct: params.medaka_model= r941_min_high_g303
nextflow_config - Config default value correct: params.aligner= minimap2
nextflow_config - Config default value correct: params.vsearch_minseqlength= 55
nextflow_config - Config default value correct: params.vsearch_maxseqlength= 57
nextflow_config - Config default value correct: params.vsearch_id= 0.99
nextflow_config - Config default value correct: params.min_reads= 30.0
nextflow_config - Config default value correct: params.min_targeted_genes= 3.0
nextflow_config - Config default value correct: params.bagel_reference_essentials= https://raw.githubusercontent.com/hart-lab/bagel/master/CEGv2.txt
nextflow_config - Config default value correct: params.bagel_reference_nonessentials= https://raw.githubusercontent.com/hart-lab/bagel/master/NEGv1.txt
nextflow_config - Config default value correct: params.hit_selection_iteration_nb= 1000.0
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.validationSchemaIgnoreParams= genomes,igenomes_base
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
nextflow_config - Config default value correct: params.gpt_drugz_gene_amount= 100
nextflow_config - Config default value correct: params.gpt_drugz_question= Which of the following genes enhance or supress drug activity?
nextflow_config - Config default value correct: params.gpt_mle_gene_amount= 100
nextflow_config - Config default value correct: params.gpt_mle_question= What genes are known to have pan-effects on cancer?
nextflow_config - Config default value correct: params.gpt_bagel2_gene_amount= 100
nextflow_config - Config default value correct: params.gpt_bagel_question= What can you tell me about these genes in the context of functional genomics?
nextflow_config - Config default value correct: params.gpt_rra_gene_amount= 100
nextflow_config - Config default value correct: params.gpt_rra_question= What genes are known to have pan-effects on cancer?
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-crisprseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-crisprseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-crisprseq_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (289 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awsfulltest_screening.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - ORIENT_REFERENCE found in conf/modules.config and Nextflow scripts.
modules_config - CAT_FASTQ found in conf/modules.config and Nextflow scripts.
modules_config - PEAR found in conf/modules.config and Nextflow scripts.
modules_config - BAGEL2_BF found in conf/modules.config and Nextflow scripts.
modules_config - BAGEL2_PR found in conf/modules.config and Nextflow scripts.
modules_config - BAGEL2_FC found in conf/modules.config and Nextflow scripts.
modules_config - DRUGZ found in conf/modules.config and Nextflow scripts.
modules_config - BAGEL2_GRAPH found in conf/modules.config and Nextflow scripts.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - FIND_ADAPTERS found in conf/modules.config and Nextflow scripts.
modules_config - MAGECK_COUNT found in conf/modules.config and Nextflow scripts.
modules_config - MAGECK_MLE found in conf/modules.config and Nextflow scripts.
modules_config - MAGECK_TEST found in conf/modules.config and Nextflow scripts.
modules_config - MAGECK_GRAPHRRA found in conf/modules.config and Nextflow scripts.
modules_config - MAGECK_FLUTEMLE found in conf/modules.config and Nextflow scripts.
modules_config - HITSELECTION found in conf/modules.config and Nextflow scripts.
modules_config - HITSELECTION_MLE found in conf/modules.config and Nextflow scripts.
modules_config - HITSELECTION_BAGEL2 found in conf/modules.config and Nextflow scripts.
modules_config - HITSELECTION_RRA found in conf/modules.config and Nextflow scripts.
modules_config - VENNDIAGRAM found in conf/modules.config and Nextflow scripts.
modules_config - MAGECK_MLE_DAY0 found in conf/modules.config and Nextflow scripts.
modules_config - CRISPRCLEANR_NORMALIZE found in conf/modules.config and Nextflow scripts.
modules_config - CUTADAPT found in conf/modules.config and Nextflow scripts.
modules_config - CUTADAPT_FIVE_PRIME found in conf/modules.config and Nextflow scripts.
modules_config - CUTADAPT_THREE_PRIME found in conf/modules.config and Nextflow scripts.
modules_config - SEQTK_SEQ_MASK found in conf/modules.config and Nextflow scripts.
modules_config - EXTRACT_UMIS found in conf/modules.config and Nextflow scripts.
modules_config - VSEARCH_CLUSTER found in conf/modules.config and Nextflow scripts.
modules_config - VSEARCH_SORT found in conf/modules.config and Nextflow scripts.
modules_config - PREPROCESSING_SUMMARY found in conf/modules.config and Nextflow scripts.
modules_config - MATRICESCREATION found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ALIGN_UMI_1 found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ALIGN_UMI_2 found in conf/modules.config and Nextflow scripts.
modules_config - RACON_1 found in conf/modules.config and Nextflow scripts.
modules_config - RACON_2 found in conf/modules.config and Nextflow scripts.
modules_config - MEDAKA found in conf/modules.config and Nextflow scripts.
modules_config - SEQTK_SEQ_FATOFQ found in conf/modules.config and Nextflow scripts.
modules_config - CLUSTERING_SUMMARY found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ALIGN_ORIGINAL found in conf/modules.config and Nextflow scripts.
modules_config - ALIGNMENT_SUMMARY found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - TEMPLATE_REFERENCE found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ALIGN_TEMPLATE found in conf/modules.config and Nextflow scripts.
modules_config - CIGAR_PARSER found in conf/modules.config and Nextflow scripts.
modules_config - CRISPRSEQ_PLOTTER found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-09-19 11:53:17

…gpt functionality calling

…and mageck mle.

…modules.

…e is run in the first place, otherwise error is thrown.

nextflow_schema.json

… generated gpt tockens to 15k. Adjusted gpt questions.

pulling remote changes

…ference.

nextflow.config

LeonHornich · 2025-03-02T10:29:48Z

Hi @LaurenceKuhl do you know if someone can help me out with this? I still think that the template update messed up some things and I have been unable to get the pr back up and running. The gpt implementation seems to not be the cause and the fork contains some error which I am having trouble fixing as it points to files all over the pipeline. If you don't have the time that's also no problem 😄

mirpedrol · 2025-03-05T15:45:09Z

Hello @LeonHornich, the changes in the dev branch replaced nf-validation by nf-schema, when resolving the conflicts while merging dev to this PR some duplications were added (see https://github.com/nf-core/crisprseq/pull/193/files#diff-61ea3cf4e947c6ba28f51817a80d81e89b8f806db0de65d874149cc075ac3c6dR47 for example).
I haven't checked if there are more of this errors, but you can go through the code changes and make sure that we are only using nf-schema now.

LaurenceKuhl changed the base branch from master to dev September 2, 2024 11:38

Leon Hornich and others added 10 commits September 2, 2024 15:43

added nf-gpt plugin to nextflow.config

1c830ed

introduced new boolean parameter gpt_interpretation that controls nf-…

96074ea

…gpt functionality calling

changed min nxf version to 24.03.0-edge

c34bcc9

added nf-gpt parameters to nextflow.config

e30179b

gpt plugin can now be parsed with individual data from drugZ, bagel2 …

6211a12

…and mageck mle.

gpt questions can now be specified in custom config.

fbc2de6

Data parser refactoring. Now 1 parser can handle data from different …

da32fa1

…modules.

merge gpt changes to dev branch

2d20407

adjusted some formating

a936cfb

Added rra support. Also, gpt interpretation now only works when modul…

78c2c74

…e is run in the first place, otherwise error is thrown.

mirpedrol reviewed Sep 12, 2024

View reviewed changes

nextflow_schema.json Outdated Show resolved Hide resolved

Leon Hornich and others added 11 commits September 12, 2024 20:00

grouped nf-gpt related parameters in nextflow_schema.json.

615618f

added more checks before nf-gpt calling

295045c

Merge branch 'dev' into dev

e8c2b56

Merge branch 'dev' into dev

05c4d56

Parsing process now selects column by index not by name. Adjusted max…

0aee7db

… generated gpt tockens to 15k. Adjusted gpt questions.

Merge branch 'dev' of github.com:LeonHornich/crisprseq into dev

cb0442b

pulling remote changes

Updated schema json. Ran code formater.

f4ce2ec

updated default gpt question for each module, removing request for re…

eecb6f4

…ference.

updated schema file with new default values.

71b745e

Merge branch 'dev' into dev

c55bec1

Merge branch 'dev' into dev

e37c4d1

LaurenceKuhl reviewed Nov 11, 2024

View reviewed changes

nextflow.config Outdated Show resolved Hide resolved

LaurenceKuhl and others added 3 commits November 11, 2024 15:19

Update nextflow.config

4e2b551

Merge branch 'nf-core:dev' into dev

ef44d1d

Merge branch 'nf-core:dev' into dev

f3fd5f3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of nf-gpt into crisprseq pipeline. #193

Implementation of nf-gpt into crisprseq pipeline. #193

LeonHornich commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited by nf-core-bot

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

❌ Test failures:

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

LeonHornich commented Mar 2, 2025

mirpedrol commented Mar 5, 2025

Implementation of nf-gpt into crisprseq pipeline. #193

Are you sure you want to change the base?

Implementation of nf-gpt into crisprseq pipeline. #193

Conversation

LeonHornich commented Sep 2, 2024 • edited Loading

github-actions bot commented Sep 2, 2024 • edited by nf-core-bot Loading

github-actions bot commented Sep 2, 2024 • edited Loading

nf-core lint overall result: Failed ❌

❌ Test failures:

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

LeonHornich commented Mar 2, 2025

mirpedrol commented Mar 5, 2025

LeonHornich commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited by nf-core-bot

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

`nf-core lint` overall result: Failed ❌