forked from wtsi-npg/npg_qc
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathChanges
2092 lines (1783 loc) · 98.3 KB
/
Changes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
LIST OF CHANGES FOR NPG-QC PACKAGE
release 74.0.2 (2025-02-20)
- Replace mark docker image with mark golang executable from github,
on ISG's docker image, for documentation publishing
release 74.0.1 (2024-12-17)
- Fix perlbrew installation by installing libdevel-patchperl-perl in runner
- Fixes/updates for lane-level RoboQC lane-level assessment:
- Excluded study-specific lane-lane assessment in npg_qc::autoqc::check::review
This helps to avoid uncertainty in evaluating lanes where samples belong to
different studies at a price of not being able to perform study-specific
evaluation of one-library lanes.
- Updated the expression evaluator to allow for multiple types of
result objects to be used in a single expression.
release 74.0.0 (2024-12-02)
- npg_qc::autoqc::qc_store - changed loading of ORM classes into memory
to on demand at run time.
- Added lane-level assessment to RoboQC:
- Extended npg_qc::Schema::Result::Review class to enable saving of
sequencing (lane-level) QC outcomes to database
- Extended npg_qc::autoqc::check::review to enable lane-level assessment
and documented the new feature.
- Made redundant 'qc_type' key of the 'robo' configuration.
- Fixed a bug in creating a list of class names in
npg_qc::autoqc::check::review.
A list of names for result classes to be loaded was inferred
from all criteria in the 'robo' section under consideration.
Until now this did not pesent a problem since even when a number
of different criteria sections were present in the study RoboQC
section of the configuration file, it was possible to load all of
the AutoQC results for any given entity.
The bug is revealed when lane-level assessment is added to
the 'robo' section under the 'default' configuration. This
type of assessment might require presence of lane-level autoqc
results for checks that are never performed on sample level,
'tag_metrics' check being the most obvious example.
The fix is to infer the list of class names from the criteria,
which are going to be evaluated for the entity, rather than
from a collection of criteria from the whole 'robo' section.
- Added instructions for running a standard dev server
- Move from Miniconda to Miniforge
release 73.0.0 (2024-10-24)
- Removed from the repository unused test data.
- npg_qc::autoqc::check::review:
1. Previously the code allowed for an empty applicability_criteria hash,
which resulted in a particular set of QC criteria being applied to every
and any product. Very early on this was an intended behaviour for UKB
data. The main filter was the study id. The default section of the product
configuration file does not have an external filter, so there is a real
danger of the review check being run indiscriminately for any product.
While this will never be an intention, small errors in the YML file might
have this effect.
2. Ability to set UQC outcomes was introduced for the Heron project. This
functionality was never used and is unlikely to be used in future, it is
now removed.
release 72.2.1 (2024-10-04)
- Added .github/dependabot.yml file to auto-update GitHub actions
- GitHub CI - updated deprecated v2 runner action to v3
- docs
- automated creation/update of confluence pages for docs
- Following a release on 07/09/2024, see https://metacpan.org/dist/App-perlbrew/changes
the checksum of the script served by https://install.perlbrew.pl had changed.
https://install.perlbrew.pl is a redirect to raw
https://github.com/gugod/App-perlbrew/blob/master/perlbrew-install, so
the change originates from GitHub and can be trusted. Our CI flow compares
the checksum of the downloaded script to the expected value. We now store
an updated expected checksum value, which corresponds to the latest release.
- npg_qc::autoqc::check::review:
The forthcoming lane-level evaluations are impossible if the code continues
to error when library_type attribute is not defined for the entity under
consideration. The library_type attribute is now set when possible, no error
if it is undefined.
release 72.2.0 (2024-08-30)
- npg_qc::autoqc::check::review:
1. To enable access to information about a sequencing run (from RunInfo.xml,
{r,R}unParameters.xml), added an optional 'runfolder_path' attribute.
The attribute will have to be set by the caller.
2. Added a new applicability criteria type - 'sequencing_run'
3. To avoid running the review check unnecessarily in the pipeline context,
moved ROBO criteria validation to the can_run method.
4. Introduced an option of defining the ROBO section within the default
section of the product configuration file.
- Explained QC outcome change for complex products (merged)
release 72.1.2 (2024-08-05)
- handle cases where the same tag sequence is seen in both index reads
release 72.1.1 (2024-05-24)
- Removing Tidyp dependency from CI
release 72.1.0
- Fixed the bug that was introduced while updating the SeqQC summary display
for pulldown_metrics incomplete results. The bug manifestation is a display
appearence of an unexpected raw mean depth value just before the coverage
value. All values that should be displayed were unaffected by the bug.
- Removed an unused part of the bin/call_gtck_composite_rpt.pl script.
This part contained the code that failed the compilation following the
deletion of the short_info role in wtsi-npg/npg_tracking#812
release 72.0.1
- Ensured that retrieving values from a partially defined pulldown_metrics
result does not trigger an error. Added a test to demonstrate that such
result can be loaded to the database without problems.
- Updated the SeqQC summary display for pulldown_metrics incomplete results
so that just 'n/a' is displayed.
- autoqc rna seqqc max memory hardcoded limit increased to 8000MB
release 72.0.0
- QC View sorting order to put merged lanes before unmerged.
- Simplify schema exclusions now the tables are gone
- Fix QC change doc to use transaction as intended
- Update tag sniff pod and deps
- Force one-page API response for tag sets
- Update README
release 71.1.0
- Updated DBIC ORM following the production MySQL database server upgrade
to MySQL v8.+
- Fixed a problem with archival of incomplete genotyping autoqc results
- Removed unused CI scripts
- Removed listing of non-existing files fro MANIFEST
- Fixed a warning when running the t/60-autoqc-results-result.t test
release 71.0.1
- Change to Perl versions from Perlbrew for CI
release 71.0.0
- Removed the plugin that enable to run the SeqQC server under the
daemon utility.
- Updated SOP for changing QC outcomes
release 70.3.0
- tag_sniff get list of known tags via a URL rather than the warehouse DB
release 70.2.1
- update version of github actions
release 70.2.0
- extend documentation for setting manual QC outcomes programmatically
- change CI runner from Ubuntu 18.04 to ubuntu-latest
release 70.1.0
- add haplotag metrics
release 70.0.2
- fix missing apt pkgs for github actions
release 70.0.1
- use bcftools mpileup instead of samtools for genotype QC check
- npg_substitution_metrics.pl avoid negative values and made metrics calculations more consistent
release 70.0.0
- Add substitution metrics (CtoA) QC check
- Added SS tag_sequence to reported matches
release 69.12.0
- Removed an unused script - genotype_call_results_reporter.
- Switched the QC outcomes reporter to retrieve LIMS server URL from
the configuration file that is used by other applications.
- Update tableExport JavaScript package to version 1.26.0
release 69.11.1
- Normalisation of negative controls with PhiX:
1. added a script to compute linear regression stats,
2. added a document that captures an interim projetc review.
- bower dependencies are expressed using https urls as unauth git endpoints
are not supported in github anymore
- update bower to 1.8.13
release 69.11.0
- PDL CPAN Perl package had a flurry of releases recently, the latest
at this time is 2.074 (2022-02-08). This verson has an undentified
bug, which results in a failure of the reverse_average_percent_error
method in the npg_qc::autoqc::role::sequence_error class. The failure
was observed in the GitHub CI runner and on a local VM.
Pinned PDL version to 2.068 or earlier since 2.068 is the vrsion we
have in the latest production software stack and it is known to work.
- Added a scipt that generates a dataset which is used in the
computation of linear regression coefficients for normalisation of
negative controls with PhiX.
release 69.10.1
- updated documentation
release 69.10.0
- switch to [email protected]
- Use samplesheet test data instead of XML files.
Remove all cached LIMS and NPG XML feeds from t/data/autoqc and
t/data/autoqc/insert_size. Change the tests to use newly generated
samplesheets in t/data/autoqc/samplesheets. The samplesheets are
manually edited to reproduce previously available test cases.
- Make all defined values in artic generic result available for display.
- pulldown_metrics autoqc results - backward-compatible correct calculation
of the number of purity filtered bases when using the latest Picard/GATK.
release 69.9.0
- Change database column type from unsigned int to insigned bigint
for columns where we store number of reads and similar values and
where we do not already set type to bigint. Some NovaSeq runs produce
more data than we envisaged when these tables were set up.
- Update pulldown_metrics autoqc check to use the up-to-date Picard
metrics via GATK, use CRAM files instead of BAM files as input.
- Drop redundant tag_index column from the tag_metrics QC database
table. The results of this QC check are always per lane. Only two
distinct values in this culumn - NULL and -1, the latter from pre-
composition times.
release 69.8.1
- bug fix npg_substitution_metrics.pl to not bail on single read data
release 69.8.0
- mqc skipper - a more generic way to disregard spiked-in controls when
counting the number of studies the samples in the run belong to. The change
is driven by the fact that spiked-in controls can now belong to a number of
different studies.
- replace fastx_toolkit use with seqtk
- npg_substitution_metrics.pl script clean-up
release 69.7.0
- added new npg_substitution_metrics.pl script
- move from Travis CI to GitHub Actions
- switch to node 14 for CI, using npm from default conda pkg
- remove dependency on npg_common::sequence::reference::base_count, which
has been moved to an internal repository and is no longer available for
the CI builds
- make labels for number of reads in SeqQC generic artic display
consistent with display for bam flagstats data, ie use 'reads'
instead of 'fragments'
- do not generate autoqc generic artic results for spiked-in control
release 69.6.1
- mqc skipper bug fix - inspect qc outcomes for review autoqc results
only
release 69.6.0
- GbS Minor_v1.0 plex included in genotype data extracted from iRODS
- change the type of the stats column in the 'samtools_stats' table from
'mediumglob' to 'longglob' to accommodate the ampliconstats data for
384 samples
- mqc skipper - no fast tracking if any of mqc_outcome values are
undefined in the review autoqc objects for a lane
release 69.5.0
- measures to speed-up generation of SeqQC pages:
- add a prefetch for compositions and components when retrieving autoqc
results from a database;
- use in-line custom sort function instead of a closure for ref_match
result objects
- disable a check for circular references in serialization of autoqc
result objects
release 69.4.0
- mqc skipper not to consider runs tagged 'no_auto' or 'no_mqc_skipper'
release 69.3.0
- mqc skipper - following a change in RoboQC creteria for real samples
(artic QC pass requirement is dropped), add a threshold for the number
of artic fails
- add option to restrict to a single gbs_plex to genotype_call_results_reporter
release 69.2.0
- the mqc skipper script is changed to always take into account the outcome
of RoboQC assessment
- improvements for SeqQC display of the generic results for artic data
release 69.1.0
- an extension for the npg_mqc_skipper script to make it work for the
Heron study, ie to take into consideration provisional manual QC
outcomes from the review autoqc resuls and finalise QC outcomes
for sequencing and libraries for runs that are expedited to archival
release 69.0.1
- npg_qc::autoqc::checks::generic::artic - bug fix in a parser for
artic QC summary for cases when the summary is empty
release 69.0.0
- npg_qc::autoqc::checks::generic::artic - a proper autoqc check class,
which repleces bin/npg_autoqc_generic4artic script
- make pp_name attribute required for the generic check object
- custom scripts for robo qc and autoqc for outputs of the artic pipeline
(npg_autoqc_generic4artic and npg_simple_robo4artic) are removed
- npg_qc::autoqc::checks::review
- breaking change of the RoboQC section of the product configuration file
- introduction of the applicability criteria to allow for conditions that
are based on different LIMS properties
release 68.9
- DBIx class for the generic table: add missing standard custom relationship
- SeqQC template simplification and database access optimisation for rna
autoqc results
- extension of the SeqQC template for the generic check - a link to
appliconstats index page for plots is added
- deployment of the latest version of Catalyst, 5.90128, does not seem
to include the Starman; the dependency was probably not explicit;
starman is added to the list of required packages in npg_qc_viewer/Build.PL
release 68.8.0
- specialised generic check for ampliconstats
release 68.7.1
- bug fix for the order of assignment of keys for the doc->{'meta'}
part of the generic result object for the ncov2019_artic_nf pp
release 68.7.0
- ability to invoke, via the qc script, specialised versions of
autoqc checks
- reusable methods for autoqc generic check
release 68.6.0
- autoqc generic result object:
rename 'desc' attribute to 'pp_name' since 'desc' is a reserved
word in some SQL flavours; hence qouting the column name, which
DBI(x) does not do, might be required;
to be able to produce distinct visual representation in SeqQC for
different external pipelines check_name method to return the
name of the class concatenated with the name of the pipeline
- npg_autoqc_generic4artic:
uses sample control flags instead of sample
supplier names to distinguish between real samples and different
sample control types
- SeqQC view for the autoqc generic result
release 68.5.0
- npg_simple_robo4artic:
uses sample control flags instead of sample
supplier names to decide on the robo evaluation criteria and the
proposed manual QC outcome;
evaluates positive controls in the same way as real samples
release 68.4.0
- npg_qc::autoqc::checks::generic and npg_qc::autoqc::results::generic -
check and result objects for generic autoqc
- DBIx class for the generic result
- a new script, npg_autoqc_generic4artic, to generate generic autoqc
results for artic pp
release 68.3.0
- npg_simple_robo4artic:
extended to consider negative and positive controls;
no action (skip) for positive control, a separate set of evaluation
criteria for negative controls;
all outcomes are recorded as preliminary manual QC outcomes;
QC summary is is recorded in the 'info' attribute of the review
autoqc check
- type of the 'info' attribute for file-based autoqc results is relaxed,
changed from a hash of strings to a hash of arbitrary entities to allow
for a nested data structure
release 68.2.1
- allow the dot character in QC outcomes column values
release 68.2.0
- tweak for genome coverage figure in qc summary - fall back to target
if no default autosomes only number available.
- a new script - npg_simple_robo4artic - to generate autoqc review
results from third party QC data
- skip ref_match autoqc checks only for true GBS samples
- SeqQC viewer: make visual cues for utility (user) QC outcomes always
visible regardless of whether they concur with the manual QC outcome
or not
release 68.1.0
- criteria_md5 attribute is added to the review autoqc result object;
previously it was computed at a point of loading to the database
and was available only in objects retrieved from the database;
availability of this value at check execution time will be useful
for planned generalisation of the review check
- ability to save autoqc review outcomes to uqc is added
- autoqc review check: enable public 'lims' attribute for the
object's constructor - needed to speed up pipeline's setup
- remove old InterOp file parsers and database loaders
release 68.0.0
- interop parser - extra tests and a fix for some result hash keys
- provisions for having an array of result objects in the result
attribute of the autoqc check object
- ability to pass multiple qc_out directories to the check
- using moniker role to generate file name root in a standard way
from a composition - needed if the result attribute is an array
since passing the file name root to the check will not be the right
thing to do (multiple file names are needed in this case)
- redundant result role methods filename_root_from_filename and
is_old_style_result are removed
- pass, info and comments attributes moved from the result to the
its parent base class (both in npg_qc::autoqc::result namespace)
to make these attributes available for results objects that are
derived directly from the base class.
- new interop autoqc check
- retrieval of interop autoqc check results by SeqQC web app is
currently supressed
- division by zero bug fixed in npg_qc::autoqc::role::bam_flagstats
for cases when nothing mappes in target regions
release 67.5.0
- npg_qc::illumina::interop::parser - a stand-alone custom parser
for Illumina InterOp files; unlike Illumina's own parser, this
parser does not round the results of the calculation
release 67.4.0
- bug fix for mqc skipper script which, because of filtering on
study name always gave count of unique studies on a run as one
- bam_flagstats autoqc check - detect and parse samtools markdup metrics
release 67.3.0
- change default file type to cram for genotype check
- script for skipping manual QC step: an additional filter
to ensure that the all lanes for a run that bypasses
manual QC deplexed properly
release 67.2.0
- script to move NovaSeq runs for a particular study
- change default file type to cram for bam_flagstats
release 67.1.0
- if DO_NOT_USE reference selected don't run VerifyBamID but provide
explanatory comment in json output file
- in qc store, capture error inferring a path to the archive dir,
which we need to retrieve autoqc data, rather from inferring
some other path
release 67.0.0
- fix regular expression used in collapser after change in template
- adapter check switched to use cram files; fixed an old bug in
parsing blat output which resulted in incorrect adapter counters
per cycle
- record absolute path of the verifyBamID executable
- remove provisions for retrieval of autoqc results from old style
run folders
release 66.6.0
- review autoqc check: do not exit on criteria evaluation error
- table and DBIx class for the review autoqc check
- SeqQC display for review autoqc results
- qc_store retrieval from a database reimplemented to ensure that
merged data are not retrieved when run data are required
- SeqQC templates updated to conform with Template::Toolkit v2.29
release 66.5.1
- add placeholder results file for bcfstats where no ref genotypes exist
- ensure immutability of a private attribute in autoqc checks review
release 66.5
- update spatial_filter results parsing
- add target autosome stats to bam_flagstats qc check
- allow 'last_modified' date in mqc tables to be pre-set
- new autoqc check 'review' to evaluate results of other
checks against configurable criteria
- SeqQC: add a visual que for failed autoqc review results
release 66.4
- allow to finalise undecided library manual qc outcomes for
all entities
- remove library type restriction from genotype_call can_run (retained as double check in seq_alignment)
release 66.3
- sequence_summary table - extend storage for the header column
- remove fastqcheck database loader
- autoqc database deletion script update
release 66.2
- SeqQC: heatmaps from samtools stats files as a source
- relax JSON parser (allow non-utf8 characters) when loading sequence_summary
results to the db
- ref match results sort order - make deterministic (important for consistency
of data we load to the warehouse), also add a method to return top two results
- fixes for composite genotype script following previous release
release 66.1.1
- composite genotype check script (call_gtck_composite_rpt.pl) search
for run folders disabled - only use iRODS cram files
release 66.1
- autoqc results loader to check for potential composition digest clashes
- amend upstream_tags check to find previous runs which have reached
status "analysis complete" (instead of "run complete")
release 66.0
- GUI for utility QC (inactive)
- add new bcfstats qc check + viewer change
- skip incomplete tag zero results in genotype_call POSTing to LIMs
- minor changes to genotype qc check to support checking against externally
supplied genotypes instead if they exist
- handle target samtools stats file - parsing, archiving and viewing.
- queries to qc_store should always have support for tracking db look-up
- methods for loading autoqc results moved from npg_qc::autoqc::collection
to npg_qc::autoqc::qc_store
- little-used method load_run of npg_qc::autoqc::qc_store class, which did
not support propagation of tracking db handle, is removed
- unused public methods in npg_qc::autoqc namespace removed
- mqc outcomes API - a new method, get_library_outcomes, for retrieving
library qc outcomes as boolean values
- deciding whether the entity is subject to manual QC:
remove special provisions for GCLP;
enforce that libraries and lanes cannot be qc-ed on the same page
- enable saving qc outcomes for multi-component compositions
- SeqQC GUI changes for multi-component compositions:
enable display of autoqc results;
enable manual QC
- changes to allow for SeqQC display of results corresponding to multi-
component compositions
- Illumina QC data loader restricted to loading cluster density data, its
data pre-loading functionality dropped
- amend spatial_filter check to take a directory name to search instead
of an explicit list of input files (that approach became unworkable
with highly plexed lanes)
- add result_file_path attribute to file-based result objects
- samtools stats file parser
- qX_yield and gc_fraction autoqc checks to use samtools stats files
instead of fastqcheck file
- SeqQC:
links for latest ivc and analysis removed since the target files are
no longer produced;
link for full rna seqc analysis follows info in the result object
- remove can_run restriction of verify_bam_id check for (cD|R)NA libraries
release 65.4.1
- store value of output_dir from the RNA-SeQC check into the database
release 65.4
- qX_yield autoqc check extension: capture and save yields for q30 and q40
release 65.3
- fixed bug in cluster_density calculation where no TileMetrics interop
file exists
release 65.2
- add new metric mitochondrial genes content as mt_pct_tpm to RNA-SeQC check
- can_run returns false for pulldown_metrics check for tag 0
- genotype check does not require the existence of irods: input_files
release 65.1.1
- explicitly use pre v6 bwa for rna_seqc rRNA alignment as input not name sorted
(tolerated in old version but not bwa0_6)
release 65.1
- bwa to bwa0_6 for rna_seqc rRNA alignment
release 65.0
- adapter autoqc check - discard code for fastq input
- add genotype_call_results_reporter script to report genotype_call
results to the LIMs
- tag_metics matches_pf_percent only stored to three decimals places now
so trim in viewer
- modified spatial filter qc check to read all run_lane filter.stats files
- changed naming of spatial_filter stats files to <run>_<lane>*.spatial_filter.stats
- upstream_tags qc check - replaced bamindexdecoder with bambi decode
- dropped ability for qc checks to sub-sample input fastq
instead run the checks directly off the cached fastq files
- NovaSeq changes, new InterOp file format and per lane read counts now 64-bit unsigned ints
- make travis happy as running apt-get update by default was disabled.
- NovaSeq run info file size is around 95KB, too large for the text column type.
Column type changed to mediumtext.
- upstream_tags autoqc check: standardise access to the tracking database
- 'execute' method of the parent autoqc check to error if input files do not exist
- dependency on tracking XML feeds removed
- bwa0_6 (newer bwa) expalicitly requested where bwa aligner is used
- all C source code moved to https://github.com/wtsi-npg/npg_qc_utils, executables are
built by Conda https://github.com/wtsi-npg/npg_conda/tree/devel/recipes/npg_qc_utils/65.0;
Build.PL changed accordingly
release 64.6.3
- update parsing of spatial filter stats in spatial_filter QC check
release 64.6.2
- QC viewer changes for genotype call
release 64.6.1
- tweak to GbS library type check in genotype call as arrived as GBS (now case-insensitive).
- Exit rna seqc gracefully if bam has no reads
- more stringent ref check now required for Pf3D7_v3.fa/hs37d5__Pf3D7_v3.fa
in genotype check
release 64.6
- add new metric glogin_pct_tpm to RNA-SeQC check
- DBIx schema loader script: alter table to add new globin metric column
- add functionality to pass some of the files necessary to run RNA-SeQC
as CLI options for program qc for more flexibility for stand alone run
- add new role for rna_seqc to implement ability to extract values from
column other_metrics to be stored in ml warehouse database.
- add tag_hops_power and tag_hops_percent columns to tag metrics
- remove run_timeline component on the Illumina QC loader
- stop generating DBIx class for run_timeline db table
- remove unused methods of the run_graph Clearpress model
- add new role for rna_seqc to implement ability to extract values from
column other_metrics to be stored in ml warehouse database
- changes for GbS genotype calling check
- tag_sniff
handle separators in BC tags for dual-index runs
allow dual-index runs to be revcomped independantly
when tag has a separator -t option operates on subtags
allowed to ignore an index when the BC tag contains a separator
release 64.5
- qc outcomes model: code generalisation to simplify addition of
further qc types
- uqc outcomes - saving and updating enabled
- switch to node 6.12.2 in travis
- stop generating DBIx classes for unused tables
- tiny tweaks to rna and verify bam id help links
- ensure, where practical, that saving and retrieving qc outcomes
works for multi-component compositions
- add functionality to allow more columns to be added to table dynamically
- add data for new extra column for sample_name in summary table
- treat all dynamically added methods in the same way
release 64.4.1
- fix bam_flagstats check library_size bug
release 64.4
- utility QC feature: db tables, DBIx classes, retrieval
- Add STAR aligner bams as a valid RNA alignments
- update bam_flagstats to be able run when no markdups metrics file exists
release 64.3.1
- error in a statement for data update is fixed
release 64.3
- DBIx schema loader script: automatically add npg_qc::Schema::Composition
- drop unique consctraint for (id_run, position and tag_index) columns;
- make a foreign key to seq_composition table not nullable and set
a unique constraint on the foreign key column in entity tables;
- do not use id_run, position and tag index column values from
entity tables, compute rpt lists from compositions;
- do not list explisitly id_run, position and tag_index in return
datastructes for both get and save outcomes methods
- a method to find a composition without creating one
release 64.2
- Cannot have default Moose constructor in derived DBIx classes - fix for
custom MqcOutcomeEnt resultset
- DBIx achema loader script: automatically add npg_qc::Schema::Composition
role when generating result classes for autoqc tables
- DBIx classes for autoqc db tables:
remove 'create_component' factory method which was only needed
for back-filling composition foreign key;
inherit 'composition' attribute from a newly created
npg_qc::Schema::Composition role;
document 'composition' attribute
- composition-enabled manual qc:
db tables linked to the seq_composition table;
search proceeds via releated composition tables;
when mqc outcomes are saved, a composition is created if it does not exist;
'composition' attribute is added to mqc entity result classes;
- upgrade bower to 1.8.2 in travis
- stop building on Travis under Perl 5.16,
see https://rt.cpan.org/Public/Bug/Display.html?id=123310
release 64.1
- SeqQC viewer - function to draw readonly representation of the
utility flag
- autoqc checks parect object - refactore to use a new factory for
rpt list to composition transformation
- upstream_tags modified so it can work with the rpt_list argument
- constraints in autoqc db tables are reset to allow for saving
results for multi-component compositions
- tag_sniff modified to load the whole tag table upfront
revcomp matches now output in reverse video
release 64.0
- composition-enabled autoqc objects storage
schema changes
DBIx binding updates
script for back-filling foreign keys to composition table
- bug fix on search_autoqc method
- db composition creation via a factory method
- db autoqc loader - unused 'update' option removed
- db autoqc loader - will retry a transaction that failed because
a db lock could not be aquired
- saving a component with subset attribute value 'all' gives an error
- npg_qc::autoqc::results::collection
remove unused functionality;
ensure existing methods work with objects that do not have id_run,
position, etc
- remove superfluous method call from a template
- db query for tag_index NULL or NOT NULL: undef as a string
might not give problems now, but will when all db records are
linked to compositions
- remove unused qc_chema property from TransferObjectFactory
- set explicit dist to precise in travis yml
- simplify and optmise pool detection in the viewer
- remove links to Clarity LIMs, generalise code for linking to LIMs
- deleted VC (view-controller) part of the Clearpress npg_qc web server
- index unq_seq_compos_rp for seq_component becomes unique
release 63.1
- label fix for rna_seqc in SeqQC viewer
- MySQL does not save correctly integers to float columns;
change column definitions for rna_seqc
release 63.0
- update script for deleting autoqc database results so that it
can handle composition-based tables
- remove attribute qc_report_dir from check object: the output directory
is created by default using the sample's filename root
- npg_qc::autoqc::qc_store - load rna_seqc results into new rna_seqc
table in npg_qc database.
- SeqQC:
- added template for rna_seqc check with selected metrics shown
in summary
- include a link to original RNA-SeQC report in check's template
- Build will fail if no node, npm or bower are available in path
- stop using multiple versions of samtools
- do not install NPGQC web app
- increase HTTP timeout when reporting manual qc outcomes to LIMs
release 62.9
- fix for GD image generation test in response to a change in constructor
behaviour in GD.pm
- compute expected path in tests in a way that takes into account
substitutions
- genotype and verify_bam_id checks:
speed up pipeline job submission for deeply plexed lanes by
moving calls that involde access to reference repository out of the
can_run method
- composition-based autoqc results db search via DBIx: allow for
a search query that contains fields from the main table
release 62.8
- tag_sniff modified clip and revcomp options to handle split tags
- Sort results for merged genotypes as well - to avoid out of order failure.
- specify an appropriate "accept" header when sending manual qc
outcomes to LIMs (header was not set resulting in 406 responce
code)
release 62.7
- current staging directories do not have autoqc results in
bustard_path; stop looking there
- ClearPress Perl library is pinned to v. 473.0.5
- upstream_tags autoqc check: stop falling back on files in IRODs
release 62.6
- make building the iRODS handle of npg_qc::autoqc::checks::genotype
lazy to avoid calling baton until required at runtime
- translation from a database composition representation to the
npg_tracking::glossary::composition type object
- db query for compisition-based tables should include a condition
on tag_index if only lanes or plexes are being retrieved
- Fix in npg_qc::utils::bam_genotype for rare out of order mpileup failures
- Travis CI build: when building dependencies, us the same
branch as the one the pull request is made to.
- back to testing with mysql 5.7.13 in travis
- drop node 0.12 from node travis matrix
- add perl 5.22-shrplib to travis matrix
- to reduce uncertainty in ranking close results, increase precision
used for ranking contamination results
- call_gtck_composite_rpt.pl - supply rpt list to genotype check
- Code and tests changes to reduce number of warnings under Perl 5.22.2
- sequence mismatch - do not return PDL special values, treat NaN as
a bad value and return undefined instead
release 62.5
- a very concise replacement for NPG_QC pages
- no templates rna_seqc results - we should not try rendering
release 62.4
- autoqc loader does not try loading a result for which there is
no coresponding DBIx binding
release 62.3
- make bam_flagstats check runnable under the qc script
- remove autoqc loader backwards compatibility with analysis code
that does not generate related results for the bam_flagstats results
- Arguments given to the qc script are passed through to the relevant
qc check object.
- QC script does not try to infer the location of input files.
- QC script accepts the rpt_list argument.
- tag_index attribute of the check object cannot be assigned to undef.
- added Broad Institute's RNA-SeQC to autoqc QC checks
- SeqQC - genotype view fixed (escaping)
release 62.1
- Bug fix in build action for cgi files.
release 62.0
- Towards composition-based autoqc:
- npg_qc::autoqc::results::result - Composition-aware,
id_run, position and path attributes become optional.
- npg_qc::autoqc::qc_store - Use DBIx syntax for quering on tag_index
(future compatibility with planned table schema changes), db load
compositions of size 1 only.
- npg_qc::autoqc::db_loader - Result classes support composition,
but most tables have not been converted yet. Do not generate
composition if no support in the result table.
- npg_qc::Schema::ResultSet - Enchancement for a search for tables that
do not yet support compositions: convert undefined values
for columns under a unique constraint to database defaults.
- SeqQC - outdated template for split_stats removed
- genotype role refactored to remove hardcoded uri encoding
- remove contamination check that is not run any longer,
result object remains
- to keep similar code together and to avoid listing explicitly
an increasing number of non-runnable checks, move checks_list()
method to the collection object
- genotype check, croak if error closing samtools pileup command
- norm_fit
- modify peak finding to allow for peaks in first/last bin
- added monotonity condition when estimating stdev
- drop stdev when outputting filtered modes
release 61.2
- SeqQC:
- Reintroduce collapser
- using grunt/npm to automate js testing
- VerifyBAMId changed condition MIN_AVG_DEPTH from > 4 to > 2 for fail
release 61.1
- SeqQC
- Using --force-latest for bower install deps during Build
- improve test robustness (larger page sample for mech to check)
release 61.0
- using test matrix with node 4.4.2 and 0.12.9 for travis
- help page for manual qc
- redesign of both client and server side for generic qc outcome updates
- updates to already stored outcomes, including final outcomes,
are not an error, will be skipped
- require that all library outcomes are marked explicitly before the
final sequencing outcome is saved
- custom class for authentication in tests - works with JSON POST requests
- mqc widgets not shown if no lims data available
- title for pages with data for one id run show run status
- SeqQC
- bugfix only build request for qc_outcomes if rptkeys in page
- upgrading jquery to 2.2.3 from 2.1.4
- upgrading requirejs to 2.2.0 from 2.1.20
- upgrading bcviz to 1.3.2 from 1.3.1
- upgrading table-export to 1.5.0 from 1.2.2
- upgrading qunit to 1.23.0 from 1.20.0
- add unique db constraint for short descriptions in qc outcome
dictionaries; the constraints should have been set when the tables
were created
release 60.0
- JSON service (retrieval) for qc outcomes
- update link to picard in POD
- SeqQC:
- remove use of autocrud plugin - it's never been used
- simplify the authorisation and it make more secure (do not fetch
user credentials from the url), move the code to a new User model
- in tests extend the User model to allow for getting user credentials
from the url
- avoid warnings in tests by setting the default for the username to
an empty string
- if the user is logged in and is authorised to do manual qc, display
a visual cue on the top of the page
- tag metrics column moved next to the column with tag index and tag sequence
- show deplexing percent for individual tags in tag metrics column
- retrieve qc outcomes from npg_qc_viewer qcoutcomes JSON service for
initial page rendering
release 59.8
- give LIMs time to process posting qc outcomes as individual events
- SeqQC
- upgrading bcviz to 1.3.1 from 1.3.0 (individual minified files)
release 59.7
- SeqQC
- bower configuration files in npg_qc/npg_qc_viewer
- moving client test files to npg_qc/npg_qc_viewer/t/client
- fix regression, missing require for table-export
release 59.6
- SeqQC
- generating bcviz plots only when they will become visible and remove
them when they will not be visible
- set max number of plexes for manual QC as 400
release 59.5
- autoqc results db loader: an option to force a re-build of related objects
- SeqQC
- fix warning about undef value in the adapter template
- fix closing div tag in verify bam id template
- upgrading jquery to 2.1.4 from 2.0.3
- upgrading requirejs to 2.1.20 from 2.1.8
- upgrading qunit to 1.20.0 from 1.15.0
- upgrading bcviz to 1.3.0 from 1.2.1
- travis ci testing perl modules for npg_qc and npg_qc_viewer
- require DBD::[email protected] to run tests
release 59.4
- saving manual qc values for libraries - allow for an undefined or empty
list of tag indices
- tags in lanes from GCLP runs are not subject to library qc
- composition-based autoqc results - disable version checking between the version
of the module that serialized the object and the version of the same module that
is performing de-serialization
- script to import manual qc values from the old warehouse RT#493757
- update_outcome method of mqc entities renamed to update_nonfinal_outcome
- a new update_outcome method allows for update of final outcomes
- method to toggle stored final manual qc outcomes
- mqc reporter - reporting might cause a loss of link between the product and
flowcell data in the ml warehouse, so prepare all data, then report
- SeqQC
- New functionality to allow export summary table to CSV
release 59.3
- transparent search query for autoqc objects irrespectively of the details of
object's database implementation (in-table identifies or composition)
- custom parent class for resultset objects
- calling ->new() on resultset object to create a new result replaced with
calling more intuitive ->new_result()
- upgrade genotype check to use samtools1 and bcftools1
- SeqQC
- New dictionary table for plex level library MQC. (Issue 238)
release 59.2
- library-level manual QC for individual plexes.
release 59.1
- 'write2file' method of autoqc result objects removed since it duplicated
the 'store' method
- alternative base class npg_qc::autoqc::results::base for autoqc results objects,
able to represent results that are derived from merged files and described by
a composition of multiple components
- autoqc result objects for samtools stats file, sam/bam headers and digests
derived from the new base class
- bam_flagstats result extension to produce samtools_stats and sequence_summary
results (related objects) either at run time or later
- database tables for storing component and composition objects
- autoqc db loader extension
- to load data represented by classes derived from the new base class
- to build related objects in memory and to load them to the database
bypassing serialization to a file - a way to save cram headers and stats
file where the analysis pipeline did not run the current code that
produces samtools_stats and sequence_summary objects as individual files
- ability to mark records as current
- reporter for manual QC results retrieves LIMs info from ml warehouse;
it skips GCLP runs RT#480489.
- tag_sniff new option to truncate tag sequences
- tag_sniff checks common tag sequences against tag sets in the warehouse
- SeqQC
- add the total yield in library view RT#488973
- summary table: display supplier sample name along the library barcode RT#488964,
remove separate sample column
- fix missing composite genotype check visualisation broken when cleaning
template for ml warehouse
- fix missing space between forward and reverse when reporting percent gc
fraction
- genotype check data extraction scripts - updated scripts to fail more readily
and informatively when an iRODS error occurs
release 59.0
- SeqQC
- remove ajax proxy controller
- stop 00 tests accessing live db credentials and databases
- using mlwarehouse for dwh information displayed in SeqQC viewer
- removing tests related to template compilation
- providing links to lims for pool/library/sample
- autoqc check and autoqc objects: remove unused 'sequence_type' attribute
- autoqc check and result objects - use existing roles for id_run and
position definitions
- result object - to remove dependency on id run, position and tag index,
allow individual results to overwrite the root of the output file name
- refactor autoqc check to use file_type attribute name instead of
input_file_ext
- tags_reporters check - remove hardcoded solexa path
- genotype and verify_bam_id_checks - use standard way of reporting run-time
problems
- bam_flagstats
- one call to compute all metrics
- method for finding stats files
- if id_run not given, derive the root of the output file name from
the input sequence file name
release 58.9
- move method for file name creation from SeqQC to autoqc so that it's
more accessible
- SeqQC
- upstream_tags template - do not report potentially confusing tag index value when
there is no id_run in the high-scoring tags table
- remove misleading bam_flagstats check info
release 58.8
- SeqQC
- display insert size normal fit confidence and number of modes in the
summary table
- do not show adaptor plots for low contamination
- provenance in lane title not in every check
- prepend all page titles with application name and version
- different page title colour if running in non-live environment
- use updated LIMs server url
- link to NPG tracking web server running on the same host as this app
- showing reference species and version below lane titles
- Using container configuration for travis
release 58.7
- bam_flagstats autoqc result object:
new field, subset, added to be used instead of human_split
human_split is retained for now to be compatible with existing data and code
- autoqc loader will only load fields that correspond to database columns
release 58.6
- Using "jquery-unveil" plugin for heatmaps
- Removing data from DOM after passing them to local variables