-
Notifications
You must be signed in to change notification settings - Fork 19
/
Copy pathChanges
2295 lines (2010 loc) · 112 KB
/
Changes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
LIST OF CHANGES
release 102.1.0 (2025-01-16)
- Fixed perlbrew installation by installing libdevel-patchperl-perl in
GitHub runner.
- Fixed CI for ubuntu 24.04 by manually installing libmodule-runtime-perl
- Extend the lims api to consider the sample_uuids attribute (in its plural
form) while using the ml_warehouse driver mainly for tag0. Also, it
extends tests for novaseq multiruns to use sample_uuid (in its single
form).
release 102.0.0 (2024-11-28)
- Removed 'bin/npg_deletable_dr_runs' since dr storage is decommissioned
release 101.6.0 (2024-10-24)
- Added sample_uuid and sample_lims methods to st::api::lims
- Patched st::api::lims::samplesheet to allow for scalar attributes
ending with 's'. Previously the values of attributes with names
ending with 's' were automatically converted to lists. In fact,
this was necessary only for comma-separated strings of email addresses.
The part of code that deals with email addresses was changed to
work on attributes with names starting with 'email' rather than
ending with 's'. Prior to applying this patch the value of the
new 'sample_lims' attribute was returned as a list when a samplesheet
was used as a source of LIMS data.
- Documented handling of multiple values in st::api::lims.
- Added is_lane method and composition_object attribute to st::api::lims.
release 101.5.1 (2024-10-04)
- Added .github/dependabot.yml file to auto-update GitHub actions
- Following a release on 07/09/2024, see https://metacpan.org/dist/App-perlbrew/changes,
the checksum of the script served by https://install.perlbrew.pl had changed.
https://install.perlbrew.pl is a redirect to raw
https://github.com/gugod/App-perlbrew/blob/master/perlbrew-install, so
the change originates from GitHub and can be trusted. Our CI flow compares
the checksum of the downloaded script to the expected value. We now store
an updated expected checksum value, which corresponds to the latest release.
- GitHub CI - updated deprecated v2 runner action to v3
release 101.5.0 (2024-09-04)
- Moved the parser for the LIMS reference genome notation from
npg_tracking::data::reference::find to a new package
npg_tracking::data::reference::util, thus providing a stand-alone
implementation of the parser.
Removed dependency of st::api::lims on npg_tracking::data::reference::find
since the latter brings in the dependency on npg_tracking::data::reference::list,
which requires that the reference repository root is defined and exists.
Instead used the reference genome notation parser from
npg_tracking::data::reference::util.
- fixes issues with tests for npg_seq_pipeline 68.5.0 and higher
release 101.4.0 (2024-08-30)
- Added species_from_reference_genome method to st::api::lims. If the
reference_genome value is defined, this new method returns the name of the
species. parse_reference_genome from npg_tracking::data::reference::find
is used for parsing the reference genome string.
release 101.3.0 (2024-08-05)
- Provided a direct access to the default section of the study configuration
file.
- Enhanced a textual listing of instruments
Added a column for recently used staging servers so that the loaders can
easily spot instruments, which write to the same server.
- Update documentation for tags
- docs
- Automatic confluence update for 'docs/api_usage.md',
'docs/instruments.md', 'docs/run_states.md', 'docs/user_management.md',
'docs/tag_semantics.md'
release 101.2.0 (2024-05-24)
- remove dependency on HTML::Tidy
- update NovaSeqX icon
- Added a footer to the tracking run web page.
release 101.1.0
- Deleted npg_tracking::illumina::run::short_info and
npg_tracking::illumina::run::locations, their functionality is moved to
npg_tracking::illumina::run::folder.
- Moved a build method for the run_folder attribute from
npg_tracking::illumina::run_folder to npg_tracking::illumina::run::folder.
- Preparation for the deletion of unused instrument_utilisation table from
the tracking db schema:
- dropped instrument_utilisation table from the db schema, which is used
for testing the tracking web application,
- suppressed generation of the DBIx ORM class for this table and deleted
the class.
- Added an option to exclude lanes from a merge (see st::api::lims
aggregate_libraries method).
- Extended the basic search functionality of the tracking web application by
enabling a search for lane annotations.
release 101.0.0
- npg_tracking::illumina::run::short_info
- Stopped parsing out run details from the run folder name. Deleted
attributes, which were populated this way: 'name', 'instrument_string',
'slot', 'flowcell_id'.
- Deleted 'short_reference' method.
- npg_tracking::illumina::run::folder
- Removed dependency on 'short_reference' method, which used to be
provided by npg_tracking::illumina::run::short_info.
- Introduce a check to ensure that the run folder name on staging and
in the database, when both are available, are the same.
release 100.0.1
- For aggregation of lims objects by library in st::api::lims, do not
consider the same library with different tag index in the same lane
as a error. Chromium single cell ATAC libraries have four copies of
each sample, each with a different tag.
release 100.0.0
- To help monitor shadow run folders, include the run folder path into the
logging messages in the staging monitor classes code.
- To explain the discovery and handling of shadow run folders, added an
extensive comment in 'bin/staging_area_monitor'.
- Deleted npg_tracking::illumina::run::folder::validation role. Most of
the functionality there depended on parsing run information from the run
folder name. This parsing will be dropped in the next release.
Added a simple check for mismatching run folder names to the find_live
method in Monitor::Staging to replace a similar check in the deleted role.
- Updated unit tests to ensure that they do not rely on retrieving the
run ID and other run details from the run folder name.
- Removed unused test file and run folders.
- Deleted 'aggregate_xlanes' method from 'st::api::lims'. A more generic
'aggregate_libraries' method in the same class can be used instead. The
pipeline has been amended to cope with this breaking change, see
https://github.com/wtsi-npg/npg_seq_pipeline/pull/815.
release 99.0.0
- st::api::lims
- Removed batch_id constructor argument. It was used by the xml driver,
it is no longer required.
- Fix a bug in propagating driver details to children to ensure that
(1) a switch of the driver type does not happen;
(2) the database handle and the flowcell ID are passed to children
consistently.
- Removed provisions for the xml st::api::lims driver from the old samplesheet
generator npg::samplesheet.
- To avoid code repetitions, created a parent class for different samplesheet
generators and moved common code there.
- Extended the code for the samplesheet daemon to generate samplesheets for
NovaSeqX instruments.
- Removed remaining unused test data from the era of xml st::api::lims driver.
- Updated signature and iRODS documentation link in template for emails
- Updated NPG tracking url to https protocol and port for links in emails
- Removed link to illumina/runs qc page in emails reporting status change event
to subscribers
release 98.1.0
- Added a new method, latest_revision_for_modification, to
npg_tracking::Schema::Result::Instrument to retrieve the latest modification
revision for the instrument.
- Added 'dragen_software_version' script argument
for bin/npg_samplesheet_generator_NovaSeqXSeries. If not set and
id_run is defined, defaults to the record for the latest 'Dragen'
modification for the NovaSeqX instrument on which the flowcell will be
sequenced.
release 98.0.0
- Removed the use of 'xml' lims driver from all tests.
- Deleted 'xml' lims driver class and all classes which supported this
functionality, together with all related tests and test data.
- In release 97.0.0, the default LIMS driver type in samplesheet generation
was changed to 'ml_warehouse'. Following this change, the code for
auto-generation of MiSeq default samplesheets is now simplified to exclude
an explicit creation of LIMS objects.
- Changes to st::api::lims, which are not related to the 'xml' lims driver
removal:
- deleted all previously deprecated methods ('seq_qc_state',
'associated_lims', 'associated_child_lims', 'associated_child_lims_ia');
- deleted methods for which we do not have data in ml warehouse, our
primary source of LIMS data ('request_id', 'project_id', 'project_name');
- deleted all code for computing inline tag indexes and inferring
tag sequences and library types from sample description since
definitions like this have not been in use for the last six years;
- reimplemented the 'reference_genome' method to exclude a fallback to
a study reference for tag zero and lane-level objects when samples
have different references;
- ensured 'required_insert_size' method works correctly for objects that
are initialised with the 'rpt_list' attribute, ie have the 'position'
attribute undefined;
- added 'aggregate_libraries' method, which is similar to the existing
'aggregate_xlanes' method, but instead of getting an instruction to
merge libraries, discovers the libraries which can be merged and returns
a list of merged entities and, separately, a list of singletons.
- npg_tracking::illumina::run::long_info role:
- restricted the 'all_lanes_mergeable' flag to NovaSeq Standard workflow,
ie dropped an additional case of HiSeq Rapid Run since HiSeq instruments
are no longer in use;
- ensured the detection of NovaSeqX and NovaSeq instrument types is
unambiguous.
- Updated DBIx classes from the prod database. Changes are due to the MySQL
database server upgrade to v8.+
- Finding the runfolder path - stopped errors being raised when a runfolder
is found in 'analysis' or 'outgoing' in the presence of a duplicate runfolder
in 'incoming' (the duplicates in 'incoming' are sometimes created by
instruments well after the run was mirrored and the runfolder moved to
'analysis'). The duplicate runfolder in 'incoming' is disregarded'.
- DRAGEN samplesheet generation - limit analysis to human samples only.
- Fixed incorrect string comparison in the npg_move_runfolder script.
- Ensure that pull requests to the master branch source dependencies from
master branches (devel branch was hardcoded).
release 97.0.2
- pin DBD::mysql to 4.050 to keep temp support for old mysql clients
release 97.0.1
- update httpd.conf with new values for mpm event
release 97.0.0
- Change CI workflow to standard file name
- Change to Perl versions from Perlbrew
- Add Perl 5.34.1 to the test matrix
- Switched the default LIMS driver in samplesheet generation from 'xml'
to 'ml_warehouse'.
- Added 'lims_driver_type' attribute to npg::samplesheet.
- Updated the tests for samplesheet generation, which are using the xml
LIMS driver, to set the driver type explicitly.
- In samplesheet generation, dropped a special case of MiSeq instruments
without a batch; last use in production was over 5 years ago.
release 96.0.0
- Fixed a regression in the npg_move_runfolder script,
which made it unusable.
- Rejig run folder tracking function into method
- Stop using XML LIMS data in a test
- Import samplesheet daemon
- Ensure we have a loader to generate ORM classes
- Flagged planned onboard analysis
- locate_runfolder: simplify, cope with complex globs, link page for multiple
locations
- Allow NovaSeqX samplesheets to embed instrument and slot
release 95.0.0
- Remove singularity recipe and related files
- Import mlwh drivers
- Reimplemented waiting for the onboard analysis
- Added a samplesheet generator for NovaSeqX
release 94.0.1
- Fixed bugs in the staging area daemon script:
- an incorrect method name
- use of previously deleted method
release 94.0.0
- Added checks for on-board analysis in progress
- Improved logging for a long wait case
- Simplified run complete and mirrored detection
- Simplified the staging area daemon.
Bring the code in accordance with the current practice:
- consider only one staging area,
- narrow down the glob pattern for areas below the staging
area to 'IL*'.
- Removed roles which defined attributes that were used
in one class only.
- Changed the way the connection to the database is established
by the staging monitor. Set up the connection explicitly in
the script rather than inside one of the classess.
release 93.0.0
- Introduced a new instrument format database record for NovaSeqX,
updated test data and one of the tests.
- Fixed visual representation for 'cBot 2' model:
the instrument now appears in the visual listing,
the instrument does not appear in the list of sequencing instruments
when a new run is created.
- Added basic provisions for NovaSeqX run tracking:
dragen_analysis_path attribute, platform_NovaSeqX method,
correct identification of run completion.
- Added basic provisions for analysis of NovaSeqX data, ie extracting
relevant information from RunInfo.xml.
- Added visual representation for NovaSeqX.
- Removed the GAII model from the 'Instrument Types' filter on the runs
listing page.
release 92.8.1
- update version of github actions
release 92.8.0
- change CI runner from Ubuntu 18.04 to ubuntu-latest
- add ad-hoc api usage example for run annotation and tagging
release 92.7.0
- add primer bed path to gbs_plex find.
release 92.6.0
- modified is_i5opposite for NovaSeq v1.5 single-end runs
release 92.5.0
- Fixed instrument name check.
- Fixed the problem in the script that generates the Illumina instruments
uptime report. A few month ago the lab stopped recording the washing
cycle. As a consequence, statuses of some instruments do not get updated
for many months. The instruments with no status within the report period
were excluded from the report.
- Simplify directory view for Tileviz data.
release 92.4.0
- Staging area daemon - when the folder name is not in the database,
prior to updating the database check that for the given run id:
1. the current run status is "run pending",
2. the instrument name is consistent between the database record
and the RunInfo.xml file.
- Switched the default st::api::lims driver from xml to samplesheet.
Fixed tests, which implicitly relied on the xml driver being the default.
Some tests used the default driver type. For those tests either test
samplesheets were generated and used as a source of LIMS data or the
driver type was explicitly set as 'xml'.
- In samplesheet generation (npg::samplesheet) explicitly set the default
st::api::lims driver to xml. This driver was previously used implicitly.
- Apache https config:
redirected all non-SSL requests to a port with enabled SSL
removed provisions for self-compiled httpd server.
release 92.3.1
- A WHILE loop in run_list.tt2 iterates over a number of pages (40 runs
per page) in a visual listing of runs. The number of pages is now
(January 2022) sufficiently high to trigger an error in Template Toolkit,
which, by default, allows 1000 iterations only. The value of
$Template::Directive::WHILE_MAX variable is, therefore, set to 2000.
release 92.3.0
- New link in sigularity image to pass npg_tracking configuration file.
- Removed ability of the tracking server to provide directory views of
visible to the server /nfs partitions since nfs staging servers are
no longer used.
- Removed co-location functionality in npg_tracking views. Co-location checks
will not make sense once npg_tracking runs in containers in head nodes.
Staging areas are not local to head node/reachable from within the
container.
- Removed old team names from run add template.
- Removed unused daemon runners for staging daemons and the Jenkins server.
- Made NovaSeq instruments skip wash statuses after completing a run, this
does not affect statuses set manually through the UI.
- Added lab location to instrument pages. Instruments are also now
colour-coded on the graphical instrument list page.
release 92.2.0
- Added singularity definition for the tracking web server.
- Tracking web pages:
added folder name column to run listing pages,
added external instrument name to the instrument page.
- Patched and refined a script for adding roles to users.
release 92.1.0
- Change LDAP URL to read-only URL
- Added a SOP for creating/decommitioning instruments.
- Removed the samplesheet daemon definition npg_tracking::daemon::samplesheet
and the auto-generator for samplesheets npg::daemon::samplesheet::auto.
Both are no longer in use, the functionality was moved to
https://github.com/wtsi-npg/npg_ml_warehouse, see
https://github.com/wtsi-npg/npg_ml_warehouse/commit/0b98fc4defb8338914f8d11448df4b30cd75528c
release 92.0.0
- Create 'primer_panel' alias for the 'gbs_plex_name' methods in
st::api::lims. The new accessor will be used in contexts that are outside
of processing GBS runs.
- Remove the API wrapping around tracking XML API.
This API has not been used in production for a long time. The part of it
that links run id and batch id (npg::api::run) was used in tests. Tests
are amended either to use samplesheets is a source of LIMS data or, where
they are still using XML LIMS driver for st::api::lims, batch id is used
explicitly.
st::api::lims::xml previously had dependency on npg::api::run to allow
for retrieval of batch_id from run XML API in cases where it was not
supplied explicitly. This dependency is removed without changing the
public interface of the class. Formally speaking, the batch_id
attribute remains optional for creating an instance of the class.
However, failure to supply it will result in an error when trying to
access batch XML API of the LIMS system.
- Remove code for sending run status events to LIMS, this functionality is
no longer required.
- Remove st::api::event class, which is no longer used.
- Stop generating XML output from the tracking server. Remove all templates
for generating XML output, all associated code in views, tests and test
data.
release 91.15.0
- Remove status daemon. This daemon is no longer run since the pipeline
updates run and lane statuses by writing directly to the database.
Remove the script, npg_status_watcher, for detecting run status JSON
files in run folders and updating database run and lane statuses,
which was run by the status daemon.
- Removed live tests for XML and warehouse drivers. Live data can change
at any time without notice, causing tests to fail.
release 91.14.0
- Remove live tests for st::api::lims XML driver and other XML-based APIs.
The CI tests are mostly run on hosts with no access to either production
or development hosts.
- NPG_TEST_TRACKING_CONFIGPATH environment variable can be used to set a
custom location of the Clearpress configuration file for unit tests.
The default location of the file, data/config.ini, is used if the
variable is unset.
- Move README file to README.md, improve Markdown format. Add sections
describing environment variables and tags.
- Move db schema dump, which is only used for tests, from the data to t/data
directory.
- Block access to tsv files via httpd_sfweb.conf
release 91.13.0
- Extend npg_status2file script to optionally save statuses to the tracking
database and renamed the script to npg_status_save.
- Move from Travis CI to GitHub Actions.
- metaref_repository attribute is added to the reference repository API
release 91.12.0
- Change instrument and instrument format pages to only show current
instruments and instrument formats
- Remove superfluous columns in display tables for lists of instruments
and instrument formats
- Remove unused views (templates) for an instrument manufacturer
- Remove manufacturer model and view classes, create manufacturer_name
method in the instrument and instrument_format model classes
release 91.11.0
- a partial reimplementation of the serialization framework for components
and compositions in order to speed-up frequently used methods
- add geno files to disabled access list in http_sfweb.conf
- cookie decryption - use an up-to-date Crypt::CBC API and algorithm
- reduce the sleep type of the status daemon from 60 to 1 sec in order to
reduce the time it takes to loop over the run folders on lustre
release 91.10.0
- script npg_samplesheet4MiSeq is removed this utility has been moved
to a different git package; the samplesheet daemon stays in this
package till all MiSeq instruments are redirected off old staging servers
release 91.9.0
- Change label 'New Instrument Mod' to 'New Instrument Upgrade' on the Admin page
- Remove 'New Instrument Status' and 'New Run Status' options from Admin page
- Sort the dropdown menu of instruments on the 'New Run' page
- Staging servers usage page retired
release 91.8.0
- BotSeq renamed Duplex-Seq
- added new functions to long_info for BotSeq analysis
- added support for NovaSeq v1.5 reagents, i5 index sequenced in opposite direction
release 91.7.0
- remove the template for an unused tracking server url - run/summary -
and all underlying code to remove obsolete SQL queries used by that code
release 91.6.0
- enforce custom type NpgTrackingReadableFile to check that the
given path is not only readable, but is actually a file
- discontinue 'npgdonotmove' and 'npg_do_not_move' file and
directory names being used as flags to stop moving the run folder
from analysis to outgoing
release 91.5.0
- ignore new revision field optionally added to primer panel name in LIMs.
release 91.4.0
- disable downloading *.fa (consensus) files from tracking server
views of staging folders
- add executable for npg samplesheet4MiSeq wrapper
- st::api::lims - two new accessors, 'sample_is_control'
and 'sample_control_type'
release 91.3.0
- pp_archive_path runfolder accessor - a path to an archive directory
for third party portable pipelines
release 91.2.0
- add support for finding primer_panel bed files
release 91.1.0
- staging monitor changes for SP flowcells
- Update URL for production Sequencescape LIMS
- method for creating a lane-level st::api::lims object from
any other st::api::lims object where the resulting object has
the same driver as the original one
- add support for finding primer_panel bed files
release 91.0.0
- modified is_i5opposite when there is no reverse read
- removed scripts for monitoring instruments via ftp; related
modules and methods, their tests and test data are removed
- code for staging monitor reorganised so that the Monitor::Runfolder
class has methods for updating run, run_lane records and run tags
and the Monitor::Runfolder::Staging class mainly deals with staging
folder inspection and changes; some methods renamed to remove
ambiguity about their functionality
- some of the functionality of the instrument monitor is incorporated
into the staging monitor script code, namely, (1) setting 'multiplex',
'paired_read' and 'single_read' tags and (2) deletion of superfluous
lanes; this functionality is now available for NovaSeq instruments,
while previously it was missing for this instrument type
- redundant 'rta' run tag is not longer set
- redundant Mirror.completed file is no longer created in the run
folder
- functionality for checking instruments' wash status and setting
'was required' instrument tag is lost; this shoud be done outside
of staging monitor
- npg_tracking::util::pipeline_config:
move bqsr and haplotype caller to a tertiary section which can be
set specifically by reference or by using a default study config;
restrict access to attributes which should not be set by the caller
(local_bin, product_config);
enable setting product release configuration file path via the
new product_conf_file_path attribute
- genome reference hierarchy in the current samplesheet is not compatible with
the latest version of MiSeq Illumina software; setting Workflow value to
GenerateFastQ prevents the instrument's software from using this value,
hence we can stop setting genome reference path altogether
- make impossible to download cbcl file via tracking web pages
- runs_on_batch method of npg::run::model is refactored not to cache
the return value; proper input checking is added
- error on run creation if the batch id associated with the run is already
associated with one of active runs, i.e. a run that has no current status
or its current status is not either 'run cancelled' or 'run stopped early
release 90.3.0
- npg_tracking::glossary::moniker -
added a method for file name parsing to retrieve id_run,
position, tag_index, suffix and extension;
private attributes are prevented from being serialized under
MooseX::Storage framework
- tracking server instrument visualization - remove visual cue
for connection delay since ftp-ing to instruments is discontinued
release 90.2.1
- staging area daemon to save the flowcell barcode to the
database if it does not have this value already available
release 90.2.0
- reference finder for compositions not to return
a PhiX reference when at least one other reference
is found
- staging area daemon will validate database flowcell barcode
against the value in files in the run folder; in case of mismatch
the run will not track
release 90.1.0
- no_archive attribute for run folder API
release 90.0.0
- cleanup and simplification of runfolder subpaths logic
- add uses_patterned_flowcell attribute to long_info
release 89.1.0
- Where semantic names are not possible, standard file and directory
names are based on sha256 digest of the composition JSON
(previously the md5 digest was used). This makes possible
conversion from the name back to the composition using the ml
warehouse records, which might be useful in case of clients'
queries.
- Unused run folder path accessors removed (reports_path,
lane_archive_path, lane_archive_paths, lane_qc_path,
lane_qc_paths).
release 89.0.1
- live test patch to reflect a change in spiked PhiX tag index
release 89.0.0
- the samplesheet driver's parser extended to cope with data from
multiple runs
- samplesheet generator extended to cache data for multiple runs
- status monitor will now work only with RFC 3339 compliant
timestamps defined in WTSI::DNAP::Utilities::Timestamp
- pipeline configuration module is moved to this package from
npg_seq_pipeline package in order for the product configuration
be accessible from other packages
- staging area daemon modified so processing for each folder is wrapped as
try/catch block
release 88.2.1
- patch to accommodate different URL info passed to the CGI
module by different Apache httpd versions
release 88.2
- scripts to list run folders that are candidates for deletion in
the Data Recovery area
- status monitor: improved logging, which now shows timestamps
- alleviate differences in cloud hosts time zone configuration by
storing UTC offset as a part of the timestamp in serialized
run and lane status objects
release 88.1.1
- test data fixes following ml warehouse schema changes, see
https://github.com/wtsi-npg/ml_warehouse/pull/104 for details
release 88.1
- status monitor running with inotify disabled should be able to
recognise when a cached file is changed
- 2 sec sleep introduced in the beginning of the job that saves
statuses to a file to ensure no two statuses of the same run/lane
have the same timestamp so that the iscurrent db flag could be
assigned inambigiously when the statuses are saved to a database
release 88.0
- tracking database:
add column `incompatible_tag` to the tag table;
populate `incompatible_tag` column with values previously
hardcoded in a DBIx class;
add tags for instrument workflow;
simplify code in DBIx classes for setting and unsetting tags;
drop requirement for a valid user id when unsetting a tag;
drop a check for tag existence when unsetting a tag
- staging monitor to set/fix tracking db record about instrument side and
workflow type by setting/resetting appropriate run tags
- Change staging area monitor so for NovaSeq runs, both CopyComplete + RTAComplete
or RTAComplete + max waiting time must be true before a run can be considered
copy complete.
release 87.4
- changes to support geno_refset repository
- moved method for creating full file names from npg_tracking::product
to the moniker role
- extension of the composition/component serialization role to ensure that the
serialised objects can be deserialized
- tracking server run page links to run folder locations:
'Latest Analysis' and 'Latest IVC' links removed since the target files
are no longer generated;
'Tileviz' link is updated to point to a location in the new-style runfolder
- 'locate_runfolder' cgi script:
code to handle redundant links and paths is removed;
if the target tileviz directory is not available, fall back to the one
expected for the old-style run folder
- redundant filename method of the illumina component object removed
- status monitor:
two modes - with and without inotify;
option to set the prefix from command line so that multiple monitors
could potentially run on the same host, each monitor dealing with
its own staging area;
avoid repeatedly saving the same statuses
- when searching snv_path use bait_name=Exome for (cD|R)NA library types
- tracking database:
add a unique db constraint for all types of status descriptions;
make temporal_index column in run_status_dict table non-nullable and
add a unique constraint on it;
simplify DBIx binding for statuses;
add a method for temporal comparison of run statuses
release 87.3
- for a run on ESA, redirect to an ESA url
release 87.2
- added method to find file with MT genes inside transcriptome repository
- change attributes in transcriptome-find to use lazy and builder rather than
lazy_build to conform with Moose's best practices
- sequencing instrument checker:
treat error reading a recipe file as non-fatal;
adjust warning message generation to use methods supported by the poller
object instance
- a new factory method, create_tag_zero_object, to create st::api::lims object
for tag zero
- a new factory method, aggregate_xlanes, to create a list of st::api::lims
objects for merges across lanes
release 87.1
- bug fix in JavaScript for run creation - the reflector cannot
be used any longer, allow client side JS to request XML feeds
directly from LIMs server
release 87.0
- new role - file and directory naming factory
- added bin/npg_move_runfolder script
- make NovaSeq instruments represented on web pages
- removing reflector allow client side JS to request XML feeds directly from LIMs server
- use lims_url config file entry to set LIMs server location
- gseq cluster hoop jumping for test 10-gbs_plex.t
- use 'fasta' as default aligner when searching for references
- long_info role:
when parsing RunInfo.xml look for FlowcellLayout rather than version,
remove unused methods and attributes,
use run parameters file to retrieve additional information
release 86.10.1
- following changes to npg_tracking::illumina::run::short_info role,
patch staging monitor, which started to fail if run id could
not be retrieved
- other changes to staging monitor:
improve reporting
change the order when validating run folders, will try
to retrieve run id first, then validate the name
propagate db schema handle
use db schema accessor that is used by npg_tracking::illumina::runfolder
and its roles
remove custom code where existing APIs can be used
release 86.10
- the bug in paged run views where a particular instrument was selected
is fixed
- the frame height for displaying runs on an instrument page is increased
- displayed instrument statuses in the instrument view are limited
to most recent (created within a year from the current date)
- npg_tracking::illumina::runfolder is simplified at a cost of losing
ability to pass to the constructor an arbitrary path
- npg_tracking::illumina::run::short_info role - added database
look-up aided ability to deal with arbitrary run folder names
- use icons from the latest set of brand images
- display institute's logo and favicon only if pages are served
from institute's servers
- removed unnecessary aliasing of urls
- deleted unused (dif_files_path and pb_cal_path) runfolder accessors
- method to find file with globin genes inside transcriptome repo added
- changes to support gbs_plex repository
release 86.9
- to account for ephemeral staging areas on OpenStack, extend staging
directory path match expression in Apache httpd config file
- use bash shell when executing system calls from Perl
- copied srpipe::runfolder module (from SVN data_handling package)
as npg_tracking::illumina::runfolder to this package, removed tests
that depended on test folder hierarchy in the data_handling package
release 86.8
- remove redundant script for reporting sequencing failures to RT
- remove uptime info from instrument page - function no longer available
- correct handling of transition from 'down for service' to 'down for repair'
release 86.7
- stop building on Travis under Perl 5.16,
see https://rt.cpan.org/Public/Bug/Display.html?id=123310
- reference find role: when parsing a genome reference string
return the analysis, along with the organism, strain
and transcriptome version, if present
- refactor transcriptome::find to improve code reuse and add
new find file/path/index attributes and functionalities
- when inflating rpt key, cast id_run, position and tag index
(if any) values to integers and raise an error if conversion
gives a warning
- unused tracking server features removed:
simultaneous status update for multiple runs;
simultaneous status update for multiple instruments;
simultaneous mod update for multiple instruments;
XML feeds for individual run statuses with listing of
all runs at this status;
flowcell verification;
complex computation of width of padding id_run in run name
for display purposes
release 86.6
- unused db tables, which were renamed in the previous release,
are dropped; exclusions in teh DBIx class loader script removed
- Jenkins daemon - a bug in locating prod ssl server certificates fixed
- added a convenience factory class to transform rpt list strings
into a composition
release 86.5
- small fix in locate_runfolder for rna seqc links
- remove unused classes for image generation
- be compatible with ClearPress version 475.3.3 and above
- set explicit dist to precise in travis yml
- add dual configuration for apache mpm worker/event
- Jenkins' daemon with HTTPS as default configuration
- drop or rename orphaned tables from db
- drop
cost_code
cost_group
sensor_data_instrument
sensor_data
sensor_instrument
sensor
mail_run_project_followers
- rename
event_type_service to todrop_event_type_service
ext_service to todrop_ext_service
event_type_subscriber to todrop_event_type_subscriber
release 86.4
- reference find role: remove unused 'subset' attribute and
previously deprecated 'single_ref_found' method
- increate http tmeout for event notification
- run tracking server on two ports - one accepting http traffic, another
one https only; the same set of URLs on both ports
- tracking web GUI link to request LIMs page now changed to point to
a more useful lane page; request id will not be available if XML
LIMs feeds are discontinued
- link to auth script residing in the same virtual host
- sensor-related code removed
- mkfastq option added to make a samplesheet suitable for 10x mkfastq
release 86.3
- event notification - do LIMs notification first; if it fails and is run
multiple times as a consequence, users are not spammed
release 86.2
- event notification:
use faster ml warehouse st::api::lims driver (ml_warehouse_fc_cache)
find templates directory relative to the running script bin
fix npg tracking server url in reports' text
filter out undefined email addresses
bug fixes
- remove models for db tables that are due to be dropped
- update apache conf so server run with 1 thread/child w/mpm_worker_module
Config::IniFiles used by ClearPress::util does not seem to be thread safe
- replace image generation/manipulation routines by a stub
returning a simple static image (these routines are used
in npg_qc in web pages that are no longer used).
- st::api::lims helper over boolean values for a pool fixed to work
correctly for compositions
release 86.1
- update models, droping cost code and cost group
- updating templates to remove broken R&D watermark based on cost_code
- deleting fixtures and tests for cost code and cost group
- under modperl information about request url is not available in
env. variables; now getting it directy from the CGI module
- when possible, any subsequent requests should go to the server
from which the original page was served (makes sense in general
and helps to set up proper dev environment)
- web pages - drop buttons linking to non-existing pages
release 86.0
- httpd config files for httpd v.2.4.23 and mod-perl v.2
- simple uptime report for Illumina sequencing instruments
- reinmplementation of events reporting
- timing of emails to followers is changed from statuses
'run complete' and 'run archived' to 'qc review pending' and 'qc complete'.
- ClearPress library is pinned to v. 473.0.5
release 85.6
- code and tests changes to reduce number of warnings under Perl 5.22.2
- use mysql 5.7.13 in travis (installing from deb)
- Run web page
- removed image browser link
- Add EBI submissions link
- Build.PL fix to ensure that images under htdocs dir are copied to blib
when building.
- updated links to NPG QC since the pages moved to SeqQC
release 85.5
- removed all code that relates to instrument utilisation
- removed an option to create database entries via XML POST.
- removed custom error view
- deleted special privisions for the 'pipeline' user
- removed unused xml views for run lists
- Install all files, including cgi-related, within the install action.
- Remove instrument utilisation views for web pages.
- Deprecate 'request approval' instrument status, enable status change
from either of down statuses directly to 'wash required'.
- Remove code and tests for deprecated instrument statuses.
- Create flags to mark users and usergroups as deprecated,
deprecate 'approvers' group.
- User and usergroup lists are alphabetically sorted and include only
current entries.
- Fixed instrument status menu for batch updates to display only
current statuses.
- Composition-enabled LIMs object and reference finder.
- Subset support added to the rpt key based composition factory.
- Objects's attributes based composition factory sets only those components
that are either defined or required. Previously an undef value might
have been used for a component attribute even if the value did not have
to be set, i.e. the code was not future compatible with planned removal
of 'Maybe' qualifies from the tag_index and subset definition.
release 85.4
- Run/instrument tracking; added full support for HiSeq 4000 and limited support
for NextSeq and MiniSeq
- Add HiSeq 4000 to graphical instrument list
- Limit menu for changing run status, when not analyst, to those on instrument
- Composition framework - the composition object becomes immutable as long
as its components are immutable.
- Functionality for an incremental assembly of a composition moved to
a generic composition factory.
- Empty compositions are not allowed.
- Composition constructor returns a sorted composition.
- Public sort function is removed.
- Composition - find is implemented as a binary search.
release 85.3
- Add deployment to GitHub Releases
- In order to have correctly versioned files in a distribution,
changed the builder base class.
- Travis conf update to pre-load perl-dnap-utilities from github.
- Samplesheet daemon:
fixed logger;
changed reference repository root so that it can be run on a staging host.
- Changes to handle a different number of tiles in each lane
- Increased TAG_INDEX_MAX to 9999 to handle pools with >999 samples
release 85.2
- Added 'sample_accession_numbers' to lims shim
- Deprecated old accessors for qc outcomes.
release 85.1
- Added 'purpose' to lims shim
- utf8 option for test sqlite db
release 85.0
- lims shim
- endeavours to ensure consistency of primary info (e.g. id_run, position)
with drivers
- will pass through arguments to driver constructor if it takes them
release 84.12
- composition and component serialization to rpt
release 84.11
- Operations (inflation and deflation) on rpt strings and their hash
representation and lists of rpt values (based on npg_qc::autoqc::role::rpt_key)
- Factory to create composition from a string representation of an rpt list
- New filename method for the illumina component
- Alter "run complete" email alert text to make clear analysis and qc time
release 84.10
- create multi-attribute methods automatically
- added method st::api::lims::any_sample_consent_withdrawn
- removed potential warnings in event notification RT#495946
release 84.9
- transcriptome find updated for RNA-SeQC directory and its gtf file
- generic serializable sequence composition framework and illumina
sequencing specific component
- lims shim now works out driver type from a given driver
- sequencing failure RT emails cope with GCLP (no batch_id)
- add predicates for glossary roles
release 84.8
- jenkins daemon definition: remove hardcoded path to prod user home directory
- refactor snv finder to stop warnings when ref genome is undefined RT#481506
- make function for parsing reference gemone public since it is used outside
of the module where it is defined
- use RTAComplete file in fallback local (instead of ftp) determination of run
progress (which is consistent with ftp process and required for HiSeqX)
release 84.7
- accommodate LIMs-defined dual indexes
- Illumina samplesheet generation: do not truncate single index when shorter first
part of dual index is present; this was written with spiked Phix in mind,
however, spikes are not reflected in LIMs for MiSeq runs
release 84.6
- when changing ownership of run folder, set gid bit on every level
- reporting croaks without full stacktraces for folder and folder/location
- daemon control:
do not use deprecated load_class method
read a list of production users from a configuration file
release 84.5
- staging monitor extension:
move qc complete runfolders from analysis to outgoing
propagate group ownership inside the runfolder
- staging/instrument monitor:
remove superfluous run lanes
assume all runs are RTA
- perform runfolder name validation against the tracking db run record
rather than against xml feeds; compare to expected runfolder name
if this name is not in the database yet
- add sample_supplier_name, sample_cohort and sample_donor_id to
st::api::lims interface
release 84.4
- to avoid the need to create deep directory hierarchy containing www-live
for a live server, set dev env variable value for the Apache server
explicitly
release 81.3
- accessor methods for frequently used configuration file entries
- status monitor script and staging area daemon definition - allow for
farm-specific variations of path prefixes for staging areas
- staging area and status daemons use common log directory /nfs/{gs|sf}XX/log
release 81.2
- web server configuration generalisation to allow for running on
different clusters
- cross-cluster staging links for both tracking and seqqc links
- make sample name compalsory in samplesheet generation
- provide path substitution mechanism to cope with running code on a file
server where the abs_path may differ from the cannonical network path.
release 81.1
- extended samplesheet contains only the Data section, Illumina
on-instrument reference paths removed
- factor out parsing database configuration files into a separate
module
- when using MySQL database in testing, do not use database configuration
files implicitly from the user's home directory; force the caller to
supply the location of the file
- remove default repository root from npg::samplesheet
- no longer stop runs without batch id or with 13-digit entries in the
batch id field from moving from run mirrored to analysis pending (
allows automatic analysis of GCLP and QC runs reespectively).
- optionally change group ownership of certain analysis directories on
moving runfolder form incoming to analysis
release 81.0
- remove default config file
- allow environment variable NPG_REPOSITORY_ROOT to set repository root
- move config of staging areas and repository into a config file
- remove vestigial code for dealing with manual qc state
- remove good_bad column in run_lane table
- remove manual_qc_status table
- npg daemon 'local' host option avoids ssh
release 80.11
- removed staging area sf48
release 80.10
- default to in-memory option for SQLite in tests
- added a method to transparently return live or dev LIMs base URL
- reference genome values - trim white space
release 80.9
- do not try to infer phix reference from library/sample name
- correct regular expression for modern MiSeq reagent kits - previously
part of the id starting from teh dash was not returned by the flowcel_id accessor