Add check for duplicate records to MainVcfQc

[kjaisingh]

This PR addresses Issue #701.

Description

• Includes sript identify_duplicates.py that takes a VCF files as input, and produces separate TSVs for 1. duplicate counts aggregated by category and 2. duplicate records tagged to a category.
• Includes script merge_duplicates.py that aggregates multiple count and record TSVs produced by identify_duplicates.py into aggregated TSVs.
• Outputs new files agg_duplicate_records.tsv and agg_duplicate_counts.tsv that represent the aggregated TSVs.
• Invokes both scripts through two separate tasks in MainVcfQc.wdl:
  • IdentifyDuplicates is a distributed task that runs in parallel across all VCFs, and is executed once SubsetVcfBySamplesList completes. Currently allows for an optional parameter identify_duplicates_custom to be passed, which is a path to a script that can be run in place of identify_duplicates.py.
  • MergeDuplicates is a distributed task that runs in parallel across all VCFs, and is executed once all IdentifyDuplicates tasks complete. Currently allows for an optional parameter merge_duplicates_custom to be passed, which is a path to a script that can be run in place of merge_duplicates.py.

Testing

• This Terra job shows an example run of the pipeline for the 1KGP cohort with this change.
• The TSV files 1kgp_2batch_test_cohort.agg_duplicate_counts.tsv and 1kgp_2batch_test_cohort.agg_duplicate_records.tsv in the linked GCS folder contain example outputs from the 1KGP cohort.
• The Python scripts themselves can also be tested locally through the following CLI commands, where ex1.vcf and ex2.vcf are sample VCF files:
  • python src/sv-pipeline/scripts/identify_duplicates.py --vcf ex1.vcf --fout ex1
  • python src/sv-pipeline/scripts/identify_duplicates.py --vcf ex2.vcf --fout ex2

  • python src/sv-pipeline/scripts/merge_duplicates.py \
    --records ex1_duplicate_records.tsv ex2_duplicate_records.tsv \
    --counts ex1_duplicate_counts.tsv ex2_duplicate_counts.tsv \
    --fout agg
    

• Validated all WDLs with womtool.

Pre-Merge Changes Required

• Remove automated Dockstore image sync for development branch.
• Remove custom script parameters for IdentifyDuplicates and MergeDuplicates.

[mwalker174]

This looks great, I have just a few minor suggestions.