Updates

This PR updates and integrates the ManualReview workflow originally authored by @xuefzhao and previously reviewed by @cwhelan into the main GATK-SV pipeline.

• Add RefineComplexVariants workflow (after CleanVcf) to refine complex variants and perform filtering of complex variants and translocations based on PE and RD evidence reassessment
  • Updates were made to parametrize the required PE count, reduce lines of code and number of files, correct handling of certain complex subtypes, review only PASS CPX variants, remove extraneous functionalities, and require only inputs produced by previous steps of GATK-SV
• Rescue mobile element deletions (LINE1 and HERVK) that were converted to BND for lack of depth evidence (5-10kb) at the end of CleanVcf
  • Updates were made to cap the size of potential ME DELs at 10kb and to add SVLEN to converted records
• Set FILTER to PASS if empty in FilterGenotypes
• Recalculate QUAL after genotype filtering
• Sanitize header at end of FilterGenotypes
• Add RefineComplexVariants to documentation
• Add downstream steps to dockstore.yml

Questions

• Rescued ME DELs still have UNRESOLVED_TYPE - do we want to remove? [Yes - removed]
• After recalculating QUAL, QUAL is . for mCNVs - is this ok? [Yes - left as-is]
• Are we ready to update the pipeline diagram for v1.0? I can add it to this PR if so [Yes - added]
  

For the future

• Rewrite RefineComplexVariants as a GATK tool for better coding practices, efficiency, and test coverage
• Integrate PE and depth evidence assessment for CPX and CTX into GenotypeComplexVariants and eliminate this downstream step
• Check for LINE1 and HERVK overlap when requiring depth evidence for large CNVs rather than correcting post-hoc
• Revisit conversion of some INS variants to CPX
• Add more thorough assessment of PE evidence for CTX and CPX including dispersion, interruptions, etc

Testing

• Validate all JSONs with womtool & Terra validation script, and verify that JSONs are present for FilterGenotypes, CleanVcf, and RefineComplexVariants (FilterGenotypes JSON was not produced after rebase)
• Tested RefineComplexVariants on 1KG Dragen callset in Terra with only inputs produced by previous steps of GATK-SV. It succeeded and the outputs were as expected: site counts changed only for INS (a subset converted to CPX) and CPX (some gained from INS conversions, some filtered as UNRESOLVED), and per-sample counts of CPX and CTX decreased as shown below

• Tested CleanVcf on the reference panel on cromwell and verified that DEL:ME records were produced
• Tested FilterGenotypes on the reference panel on cromwell and verified that QUAL was now in [0,99] and empty filter statuses had been converted to PASS

Ongoing testing [Complete]

• Re-testing RefineComplexVariants with updated docker image (previously tested with script overrides) [Complete]
• Re-testing RefineComplexVariants with no-call caching for cost/runtime estimate and resource monitoring [Complete, cost $9.94 and ran for about 7 hours for ~3k samples]
• Re-testing RefineComplexVariants after linting changes [Complete]
• Testing FilterGenotypes on outputs of RefineComplexVariants [Complete]