Dev ratio #288

bjlang · 2024-09-13T13:55:43Z

PR checklist

…_schema_coda

Coda workflow. Files that are unique to the coda workflow have been added to the corresponding folders (conf, modules, subworkflows...). Files that are present in the pipeline have been added as ***_coda to avoid overwriting the original ones (modules_coda.config, nextflow_coda.config, main_coda.nf... etc).

…tion

…mability

… type

Update dev-ratio with changes from dev branch

…ndance into dev-ratio

Include CoDa workflow in main workflow

- use contrast file information for differential analysis with PropD

add GPROFILER2 module to ENRICHMENT subworkflow

Co-authored-by: WackerO <[email protected]>

322 add module deseq2

move pathway related logic out of differential and correlation subworkflows

pinin4fjords

I think we may have a difference of opinion here, but I really think components should be kept simple, and complexity around channel manipulations should be pushed up to workflows, it's just the nf-core way.

I think I'm going to have a go at pushing the differential workflow up into nf-core/modules (in simplified form).

pinin4fjords · 2024-11-05T09:38:37Z

subworkflows/local/differential/main.nf

+    take:
+    ch_counts             // [ meta_exp, counts ] with meta keys: method, args_diff
+    ch_samplesheet        // [ meta_exp, samplesheet ]
+    ch_contrasts          // [ meta_contrast, contrast_variable, reference, target ]


Suggested change

ch_contrasts // [ meta_contrast, contrast_variable, reference, target ]

ch_contrasts // [ meta_contrast, contrast_variable, reference, target ]

method // limma, deseq2 ...

pinin4fjords · 2024-11-05T09:43:26Z

subworkflows/local/differential/main.nf

+    ch_counts
+        .branch {
+            propd:  it[0]["method"] == "propd"
+            deseq2: it[0]["method"] == "deseq2"
+            limma:  it[0]["method"] == "limma"
+        }
+        .set { ch_counts }


Suggested change

ch_counts

.branch {

propd: it[0]["method"] == "propd"

deseq2: it[0]["method"] == "deseq2"

limma: it[0]["method"] == "limma"

}

.set { ch_counts }

Sorry, I still don't like this. We should keep the interface simple rather than forcing people to stuff things into meta maps.

I don't see your problem. The meta is existing in every nf-core module and subworkflow and from what I see it is also not uncommon to use the meta information for downstream analysis decisions. Take for example the single_end flag in RNAseq, ChIPseq, ATACseq,... Both modules (e.g. trimgalore ) and subworkflows are using it, and thus forcing the flag to exist. This is even also used for branching.

The single-end thing is a special case, we try and discourage the propagation of assumptions about fields in the meta.

The branching you point at is necessitated by the need to direct outputs form the first processes to the right place. It's not engineered in to the inputs. If I were writing that subworkflow without that first process I'd be saying the same thing- the single end status would be a straighforward subworkflow input.

The point here is that the tool status has nothing to do with the count file. You're forcing people to build it into the meta to suit the use case, but it would just make life harder for other users of the subworkflow.

https://nf-co.re/docs/guidelines/components/modules#types-of-meta-fields

pinin4fjords · 2024-11-05T09:46:17Z

subworkflows/local/differential/main.nf

+
+workflow DIFFERENTIAL {
+    take:
+    ch_counts             // [ meta_exp, counts ] with meta keys: method, args_diff


Suggested change

ch_counts // [ meta_exp, counts ] with meta keys: method, args_diff

ch_abundance // [ meta_exp, counts ] with meta keys: method, args_diff

This won't always be counts- e.g. for arrays it will be 'intensities', for PROTEUS it will be whatever that techology uses.

pinin4fjords · 2024-11-05T09:47:52Z

subworkflows/local/differential/main.nf

+    // Perform differential analysis with propd
+    // ----------------------------------------------------
+
+    // TODO propd currently don't support blocking, so we should not run propd with same contrast_variable, reference and target,


You can assume that will be resolved at the workflow level (filtering the contrasts channel)

pinin4fjords · 2024-11-05T09:53:09Z

subworkflows/local/differential/main.nf

+include { DESEQ2_DIFFERENTIAL } from '../../../modules/nf-core/deseq2/differential/main'
+include { DESEQ2_DIFFERENTIAL as DESEQ2_NORM } from "../../../modules/nf-core/deseq2/differential/main"
+include { LIMMA_DIFFERENTIAL } from '../../../modules/nf-core/limma/differential/main'
+include { FILTER_DIFFTABLE as FILTER_DIFFTABLE_LIMMA } from '../../../modules/local/filter_difftable'


This subworkflow will never run BOTH Limma AND DESeq2, so you don't need to import and call it twice. You should output DESeq2 + Limma to the same channels, which get filtered by the same process.

but that is the idea with the multiply running options no?
If the user wants to run --pathway deseq2_gsea,limma_gsea
Both deseq2 and limma would be run, but the output would be channeled into different places.
How would you deal with that instead?

Ahh yes, I think maybe this is down to our difference in thoughts on architecture.

In my view the subworkflow would run twice to make that happen.

ahhhhh so you would have import various subworkflows (eg DIFFERENTIAL_DESEQ2, DIFFERENTIAL_LIMMA) inside the workflow and run them.... actually that is nice :)

Yep. We keep all components as simple as possible, and manage complexity via channel operations in the calling workflow.

subworkflows/local/differential/main.nf

pinin4fjords · 2024-11-05T16:37:12Z

modules/local/propr/propd/main.nf

+        'community.wave.seqera.io/library/bioconductor-limma_r-ggplot2_r-propr:17abd3f137436739' }"
+
+    input:
+    tuple val(meta), path(count), path(samplesheet), val(contrast_variable), val(reference), val(target)


This is very un-nf-core, and doesn't match the analagous differential modules. I think this will need to be split for nf-core

okay that is not a problem

even though why this is not recommended by nf-core?

Ahh- has there been a change? Could you point at that guideline?

not sure if there a change/guideline for that...that is why i asked because i was not sure why you say this is un-nf-core.

I mean, the convention in modules is not to combine everything into a single input channel, but only to combine related things. You could copy the interface of the other differential modules, which would have better consistency and would prevent us having to do special operations to fit this module.

pinin4fjords · 2024-11-05T16:39:00Z

modules/local/propr/propd/main.nf

+
+    // conda "${moduleDir}/environment.yml"
+    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
+        'oras://community.wave.seqera.io/library/bioconductor-limma_r-ggplot2_r-propr:209490acb0e524e3' :


We need to use the https versions of the singularity URIs for Seqera Containers. It's a bit of a pain, but here's how: https://nfcore.slack.com/archives/CJRH30T6V/p1729023924708159?thread_ts=1729023915.530259&cid=CJRH30T6V

…nd not arrayLists

…annels and not arrayLists" This reverts commit 655ad61.

Fix tests; add version channels; remove params access in subworkflow

311 add module gsea

Cristina Araiz and others added 30 commits July 18, 2024 10:38

add propr, mygene and fiter_var module

eff2e81

add functional subworkflows into subworkflows/local

708c4db

add schema_tools and tools_samplesheet to assets

4a76cef

add crg.config and modules.config (as modules_coda) to conf

93ae92d

add main.nf as main_coda.nf

da30859

add nextflow.config and nextflow schema as nextflow_coda and nextflow…

7487789

…_schema_coda

add YMC counts and samplesheet to help for testing

a91626b

upload mygene module with nf-core module install

6f8d5b0

fix linting errors

969d15a

solve remaining lint erros

a2d3824

update propr/grea with nf-core modules install

b59f668

modify propr main.nf to match nf-core repository

45b776e

install propd with nf-core modules install

d60f031

fix pre-commit errors in .json files

8b25351

install propr/propr with nf-core modules install

fc123ec

small fixes to allow main_coda.nf to run

e3fde5b

Allow caching for -resume

d45c4aa

Use recommended format for optional parameters

20c6cc6

Skip copying matrix if not necessary to allow resuming pipeline execu…

5a3727b

…tion

Keep original matrix file name in the annotation copy to improve resu…

0a3927a

…mability

Move main_coda logic into differentialabundance as experimental study…

05cfb3a

… type

Merge pull request #284 from nf-core/dev

5e5c6e4

Update dev-ratio with changes from dev branch

Merge branch 'nf-core:dev-ratio' into dev-ratio

718075b

Do not put not not in the not wrong place!

b218940

Merge branch 'dev-ratio' of https://github.com/bjlang/differentialabu…

3a69aae

…ndance into dev-ratio

Merge pull request #283 from bjlang/dev-ratio

6e6a740

Include CoDa workflow in main workflow

- use validated input for experimental workflow

04abe25

- use contrast file information for differential analysis with PropD

Merge branch 'nf-core:dev-ratio' into dev-ratio

2fe78de

small fix

3254d25

Breeshey Roskams-Hieter and others added 17 commits October 29, 2024 16:32

remove trailing whitespace

47bb659

add new line at end of file

90001a2

Adapt Limma call

f7c5c77

Merge pull request #333 from nf-core/323_add_module_gprofiler2

4889998

add GPROFILER2 module to ENRICHMENT subworkflow

Merge remote-tracking branch 'upstream/dev-ratio' into dev-ratio

1198761

Implement review comments

953eb51

add deseq2 pathway name

4a5aa60

add DESEQ2 block to this workflow

3546cb0

set nsub to lower numebr given nature of test data

ba1ea24

resolve merge conflicts

d6da7ad

fix linting

a2ddd8a

update channel name to samples_and_matrix

a282a47

Co-authored-by: WackerO <[email protected]>

update channel name to samples_and_matrix

0e4a84d

Co-authored-by: WackerO <[email protected]>

update channel name to samples_and_matrix

a885116

Co-authored-by: WackerO <[email protected]>

Merge pull request #335 from nf-core/322_add_module_deseq2

0f9c9fc

322 add module deseq2

Merge pull request #318 from bjlang/dev-ratio

6d0d3b8

move pathway related logic out of differential and correlation subworkflows

Merge branch 'dev-ratio' into 311-add-module-gsea

69f5b36

pinin4fjords requested changes Nov 5, 2024

View reviewed changes

pinin4fjords reviewed Nov 5, 2024

View reviewed changes

pinin4fjords mentioned this pull request Nov 7, 2024

Create 'differential' subworkflow #341

Open

bjlang and others added 10 commits November 13, 2024 14:07

Fix test runs to finish except for GSEA

ad3173e

fix linting

440e40f

Fix tests; add version channels; remove params access in subworkflow

846ca55

Forward version channel from subworkflows

95a9b25

Ensure ch_transcript_lengths and ch_control_features being channels a…

655ad61

…nd not arrayLists

Revert "Ensure ch_transcript_lengths and ch_control_features being ch…

047099e

…annels and not arrayLists" This reverts commit 655ad61.

Fix DEseq2 call

13bc1b7

Merge pull request #357 from bjlang/dev-ratio

8059211

Fix tests; add version channels; remove params access in subworkflow

Merge branch 'dev-ratio' into 311-add-module-gsea

128ffe8

Merge pull request #329 from nf-core/311-add-module-gsea

9e4a105

311 add module gsea

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dev ratio #288

Dev ratio #288

bjlang commented Sep 13, 2024

pinin4fjords left a comment

pinin4fjords Nov 5, 2024

pinin4fjords Nov 5, 2024

bjlang Nov 13, 2024

pinin4fjords Nov 13, 2024 •

edited

Loading

pinin4fjords Nov 13, 2024

pinin4fjords Nov 13, 2024

pinin4fjords Nov 5, 2024

pinin4fjords Nov 5, 2024

pinin4fjords Nov 5, 2024

suzannejin Nov 6, 2024 •

edited

Loading

pinin4fjords Nov 6, 2024

suzannejin Nov 6, 2024

pinin4fjords Nov 8, 2024

pinin4fjords Nov 5, 2024

suzannejin Nov 6, 2024

suzannejin Nov 6, 2024

pinin4fjords Nov 6, 2024

suzannejin Nov 6, 2024

pinin4fjords Nov 6, 2024

pinin4fjords Nov 5, 2024

	ch_contrasts // [ meta_contrast, contrast_variable, reference, target ]
	ch_contrasts // [ meta_contrast, contrast_variable, reference, target ]
	method // limma, deseq2 ...

	ch_counts // [ meta_exp, counts ] with meta keys: method, args_diff
	ch_abundance // [ meta_exp, counts ] with meta keys: method, args_diff

Dev ratio #288

Are you sure you want to change the base?

Dev ratio #288

Conversation

bjlang commented Sep 13, 2024

PR checklist

pinin4fjords left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pinin4fjords Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suzannejin Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pinin4fjords Nov 13, 2024 •

edited

Loading

suzannejin Nov 6, 2024 •

edited

Loading