Code

DM-43722 EFD Transform Implementation#18

Code

Open

rcboufleur wants to merge 119 commits intomainlsst-dm/consdb:mainfrom tickets/DM-43722lsst-dm/consdb:tickets/DM-43722Copy head branch name to clipboard

+80,500-3

Commits

Commits on Oct 15, 2024

• Implement first version of EFD transformations

  glaubervila

  committed
  57e1b52

  Copy full SHA for 57e1b52

• Update misssing values in the config file

  

  rcboufleur

  authored and

  glaubervila

  committed
  83eadd3

  Copy full SHA for 83eadd3

• Fix exception message for unimplemented dialect

  

  rcboufleur

  authored and

  glaubervila

  committed
  07ea890

  Copy full SHA for 07ea890

• Refactor Aggregate class to handle column-wise operations

  

  rcboufleur

  authored and

  glaubervila

  committed
  b7c30cc

  Copy full SHA for b7c30cc

• Refactor Aggregate class to Summary for better clarity and consistency

  

  rcboufleur

  authored and

  glaubervila

  committed
  5e587ab

  Copy full SHA for 5e587ab

• Refactor ATAOS_correctionOffsets_w function to use mean

  

  rcboufleur

  authored and

  glaubervila

  committed
  7134953

  Copy full SHA for 7134953

• the ExposureEfd table schema was changed to be compatible with config_LATISS.yml

  glaubervila

  committed
  5cc2231

  Copy full SHA for 5cc2231

• Added VisitEFD table and fixed empty values

  glaubervila

  committed
  8732968

  Copy full SHA for 8732968

• Upsert transactions now commit every 100 rows. Docstrings were added to dao classes.

  glaubervila

  committed
  ead6f10

  Copy full SHA for ead6f10

• Refactor configuration file loading and validation

  glaubervila

  committed
  3816d47

  Copy full SHA for 3816d47

• Changed lsst_efd_client -> InfluxDB API for topic queries

  glaubervila

  committed
  e1a94ee

  Copy full SHA for e1a94ee

• fixed lint

  glaubervila

  committed
  d65a16f

  Copy full SHA for d65a16f

• Lint fixes

  

  rcboufleur

  authored and

  glaubervila

  committed
  f512a97

  Copy full SHA for f512a97

• Additional parameters inserted in config yml

  

  rcboufleur

  authored and

  glaubervila

  committed
  7225480

  Copy full SHA for 7225480

• Lint fixes

  

  rcboufleur

  authored and

  glaubervila

  committed
  11eba2d

  Copy full SHA for 11eba2d

• Added Dockerfile for Efd Transform

  glaubervila

  committed
  6fd11b4

  Copy full SHA for 6fd11b4

• The main command was added to the dockerfile

  glaubervila

  committed
  70be00e

  Copy full SHA for 70be00e

• Implement retrieval of packed time series from InfluxDB using API queries

  

  rcboufleur

  authored and

  glaubervila

  committed
  f663262

  Copy full SHA for f663262

• Lint fixes

  

  rcboufleur

  authored and

  glaubervila

  committed
  4366644

  Copy full SHA for 4366644

• Fixed lint

  glaubervila

  committed
  92462c8

  Copy full SHA for 92462c8

• Fixed lint

  glaubervila

  committed
  45559a5

  Copy full SHA for 45559a5

• Implements new config file formats and validation

  

  rcboufleur

  authored and

  glaubervila

  committed
  21dc88e

  Copy full SHA for 21dc88e

• Implements unique topic querying to avoid duplicate queries

  

  rcboufleur

  authored and

  glaubervila

  committed
  fb66a51

  Copy full SHA for fb66a51

• Changed base image to w_2024_33

  glaubervila

  committed
  7c73a43

  Copy full SHA for 7c73a43

• Connection to usdf_efd api now uses environment variables

  glaubervila

  committed
  880e782

  Copy full SHA for 880e782

• Added copy of sqlite test database

  glaubervila

  committed
  4c7c5ad

  Copy full SHA for 4c7c5ad

• Config changes refactored into transformations

  

  rcboufleur

  authored and

  glaubervila

  committed
  0643536

  Copy full SHA for 0643536

• Config changes refactored into transformations

  

  rcboufleur

  authored and

  glaubervila

  committed
  613cdce

  Copy full SHA for 613cdce

• Removed template envs

  glaubervila

  committed
  9d65166

  Copy full SHA for 9d65166

• minor changes in dockerfile

  glaubervila

  committed
  c437b71

  Copy full SHA for c437b71

• All files related to trasnform_efd have been moved to the same folder. Temporary files have been moved to the tests folder.

  glaubervila

  committed
  1ce8920

  Copy full SHA for 1ce8920

• The Dockerfile has been changed to the new file structure.

  glaubervila

  committed
  95e9d04

  Copy full SHA for 95e9d04

• Processing warnings due to computation errors were accounted for

  

  rcboufleur

  authored and

  glaubervila

  committed
  bb93200

  Copy full SHA for bb93200

• Lint error fix

  

  rcboufleur

  authored and

  glaubervila

  committed
  4abc0ba

  Copy full SHA for 4abc0ba

• ESS accelerometer fields were added

  

  rcboufleur

  authored and

  glaubervila

  committed
  ef65871

  Copy full SHA for ef65871

• Segmentation of queries with large number of fields is implemented

  

  rcboufleur

  authored and

  glaubervila

  committed
  058855c

  Copy full SHA for 058855c

• Typo fix in the configuration file

  

  rcboufleur

  authored and

  glaubervila

  committed
  e86bc75

  Copy full SHA for e86bc75

• Start date and end date parameters are now optional with default value based on now - deltatime.

  glaubervila

  committed
  4db0f53

  Copy full SHA for 4db0f53

• Updated summary functions

  

  rcboufleur

  authored and

  glaubervila

  committed
  7ef80f5

  Copy full SHA for 7ef80f5

• Fix in time format definitions

  

  rcboufleur

  authored and

  glaubervila

  committed
  4aead0b

  Copy full SHA for 4aead0b

• Changed config file path

  glaubervila

  committed
  7f43ea4

  Copy full SHA for 7f43ea4

• Test postgresql consdb connection

  glaubervila

  committed
  635a8b5

  Copy full SHA for 635a8b5

Commits on Oct 16, 2024

• Fixed missing schema name in transform efd insert queries

  glaubervila

  committed
  e30acba

  Copy full SHA for e30acba

• Added more logs and test access to consdb tables

  glaubervila

  committed
  294d816

  Copy full SHA for 294d816

• Fix sqlalchemy get table from metadata

  glaubervila

  committed
  f1049fc

  Copy full SHA for f1049fc

• Fix comparisons of time aware indexes

  rcboufleur

  committed
  90005f5

  Copy full SHA for 90005f5

• Fixed sqlalchemy table from metadata with sqlite

  glaubervila

  committed
  0bd52d1

  Copy full SHA for 0bd52d1

Commits on Oct 17, 2024

• Column and topic mapping refactored

  rcboufleur

  committed
  284b0df

  Copy full SHA for 284b0df

Commits on Oct 18, 2024

• Fixed lint with pre-commit

  glaubervila

  committed
  6745196

  Copy full SHA for 6745196

• Fixed lint

  glaubervila

  committed
  33fc5b7

  Copy full SHA for 33fc5b7

Commits on Oct 24, 2024

• Conficts resolved and merged.

  rcboufleur

  committed
  6c2b725

  Copy full SHA for 6c2b725

• Schema generator based on config files created

  rcboufleur

  committed
  2615fdb

  Copy full SHA for 2615fdb

• Temporary files update

  rcboufleur

  committed
  8ea745d

  Copy full SHA for 8ea745d

• Column added to generate_schema.py

  rcboufleur

  committed
  c05ff0f

  Copy full SHA for c05ff0f

• Column added to generate_schema

  rcboufleur

  committed
  f46ce25

  Copy full SHA for f46ce25

• A queue manager has been added to control execution by periods.

  glaubervila

  committed
  fd30ae8

  Copy full SHA for fd30ae8

• minor fixes

  glaubervila

  committed
  9acbdab

  Copy full SHA for 9acbdab

• Last minute value transformation included

  rcboufleur

  committed
  a4846da

  Copy full SHA for a4846da

• New config files generated

  rcboufleur

  committed
  7f80094

  Copy full SHA for 7f80094

Commits on Oct 25, 2024

• Updated instruments in testing files

  rcboufleur

  committed
  e4ce479

  Copy full SHA for e4ce479

• Updated schema structures

  rcboufleur

  committed
  8928f5c

  Copy full SHA for 8928f5c

• Create new task after run all tasks

  glaubervila

  committed
  c445d6c

  Copy full SHA for c445d6c

• Minor fix in create new tasks

  glaubervila

  committed
  ea09f56

  Copy full SHA for ea09f56

Commits on Nov 6, 2024

• Minor changes in run sh

  glaubervila

  committed
  b66af91

  Copy full SHA for b66af91

• Added permision

  glaubervila

  committed
  7a7f99d

  Copy full SHA for 7a7f99d

• fixed dockerfile

  glaubervila

  committed
  df31088

  Copy full SHA for df31088

Commits on Nov 7, 2024

• diferent schemas by instruments

  glaubervila

  committed
  9c38464

  Copy full SHA for 9c38464

• Update to properly handle schemas by instrument

  rcboufleur

  committed
  e114cce

  Copy full SHA for e114cce

• yaml extensions fixed

  rcboufleur

  committed
  f4b416d

  Copy full SHA for f4b416d

• Fixed hardcoded schema for Queue manager

  glaubervila

  committed
  626f148

  Copy full SHA for 626f148

Commits on Nov 8, 2024

• Added timewindow to queue manager table

  glaubervila

  committed
  927b852

  Copy full SHA for 927b852

• Timewindow column added to the yaml files

  rcboufleur

  committed
  1d7556a

  Copy full SHA for 1d7556a

• Schema generator yaml files updated

  rcboufleur

  committed
  160d265

  Copy full SHA for 160d265

• Test files updated

  rcboufleur

  committed
  38a8589

  Copy full SHA for 38a8589

Commits on Nov 22, 2024

• Update Dockerfile.efdtransform

  rcboufleur

  authored
  0e53e5c

  Copy full SHA for 0e53e5c

• Update Dockerfile.efdtransform

  rcboufleur

  authored
  4fd17e3

  Copy full SHA for 4fd17e3

Commits on Dec 6, 2024

• Merge remote-tracking branch 'origin/main' into tickets/DM-43722

  rcboufleur

  committed
  b43bc79

  Copy full SHA for b43bc79

Commits on Dec 8, 2024

• Updates configuration files

  rcboufleur

  committed
  92a2a4b

  Copy full SHA for 92a2a4b

• Updates schemas yaml files

  rcboufleur

  committed
  0cd3d38

  Copy full SHA for 0cd3d38

• Implements unpivoted array support for array like results

  rcboufleur

  committed
  8ff1413

  Copy full SHA for 8ff1413

• Updates test files

  rcboufleur

  committed
  6bc51b8

  Copy full SHA for 6bc51b8

• Implements new unpivoted tables

  rcboufleur

  committed
  8ecd421

  Copy full SHA for 8ecd421

• Updates column name in unpivoted tables

  rcboufleur

  committed
  7cc9885

  Copy full SHA for 7cc9885

• Rerun pre-commit

  rcboufleur

  committed
  c889ec8

  Copy full SHA for c889ec8

• Fixes error on github actions

  rcboufleur

  committed
  9de2cc6

  Copy full SHA for 9de2cc6

• Update summary.py

  rcboufleur

  authored
  28a900c

  Copy full SHA for 28a900c

• Fix column name bug unpivoted data

  rcboufleur

  committed
  59d2e2a

  Copy full SHA for 59d2e2a

• Realocates testing files

  rcboufleur

  committed
  36fae9c

  Copy full SHA for 36fae9c

• Fixes pytest for config_model

  rcboufleur

  committed
  38c12fd

  Copy full SHA for 38c12fd

• Fixes bugs in processing unpivoted results

  rcboufleur

  committed
  8acfdd0

  Copy full SHA for 8acfdd0

Commits on Dec 9, 2024

• Updates temporary files

  rcboufleur

  committed
  a5c1e66

  Copy full SHA for a5c1e66

• Fix trailing spaces

  rcboufleur

  committed
  84f9295

  Copy full SHA for 84f9295

• Fixes configuration files and schemas

  rcboufleur

  committed
  cbf7755

  Copy full SHA for cbf7755

• Update Dockerfile.efdtransform

  rcboufleur

  committed
  c15a890

  Copy full SHA for c15a890

Commits on Jan 16, 2025

• Update import path for future compatibility and proper pytest funcionality

  rcboufleur

  committed
  e5f3d69

  Copy full SHA for e5f3d69

• Update import path for future compatibility and proper pytest funcionality

  rcboufleur

  committed
  844c765

  Copy full SHA for 844c765

• Update import path and fixes last minute method computation.

  rcboufleur

  committed
  55b748a

  Copy full SHA for 55b748a

• Update import path for future compatibility and proper pytest funcionality

  rcboufleur

  committed
  2e25430

  Copy full SHA for 2e25430

• Update import path and fixes changes in summary methods and attributes.

  rcboufleur

  committed
  5aa6cdb

  Copy full SHA for 5aa6cdb

• Sets __init__ for dao module

  rcboufleur

  committed
  db35f7e

  Copy full SHA for db35f7e

• Update import path for future compatibility and proper pytest funcionality

  rcboufleur

  committed
  c17fb15

  Copy full SHA for c17fb15

• Update import path for future compatibility and proper pytest funcionality

  rcboufleur

  committed
  b514760

  Copy full SHA for b514760

• Update import path and updates changes to maintain compatibility with summary methods and attributes

  rcboufleur

  committed
  b539b10

  Copy full SHA for b539b10

• Update import path for future compatibility and proper pytest funcionality

  rcboufleur

  committed
  686e228

  Copy full SHA for 686e228

• Update import path for future compatibility and proper pytest funcionality

  rcboufleur

  committed
  5a7cccf

  Copy full SHA for 5a7cccf

• Fixes typo in column name

  rcboufleur

  committed
  13d1a42

  Copy full SHA for 13d1a42

• Fixes typo in column names

  rcboufleur

  committed
  ef1fbec

  Copy full SHA for ef1fbec

• Implements pytest for config_model

  rcboufleur

  committed
  cb38275

  Copy full SHA for cb38275

• Implements pytest for generate_schema

  rcboufleur

  committed
  84bed6a

  Copy full SHA for 84bed6a

• Implements pytest for queue_manager

  rcboufleur

  committed
  fb05010

  Copy full SHA for fb05010

• Implements pytest for summary methods

  rcboufleur

  committed
  35c3455

  Copy full SHA for 35c3455

• Updates temporary files used in local testing

  rcboufleur

  committed
  abd378b

  Copy full SHA for abd378b

• Updates Dockerfile configurations

  rcboufleur

  committed
  fc262f4

  Copy full SHA for fc262f4

• Update PYTHONPATH in Dockerfile

  rcboufleur

  committed
  f877ff3

  Copy full SHA for f877ff3

• Fix LATISS schema tables

  rcboufleur

  committed
  21b08c4

  Copy full SHA for 21b08c4

• Fix lint errors

  rcboufleur

  committed
  1346e2a

  Copy full SHA for 1346e2a

• Updates temporary files (local run)

  rcboufleur

  committed
  13f1aa4

  Copy full SHA for 13f1aa4

Commits on Jan 17, 2025

• Fix all Ruff linting issues across the codebase

  rcboufleur

  committed
  c4115fd

  Copy full SHA for c4115fd

• Fix pre-commit issues across the transform_efd codebase

  rcboufleur

  committed
  cc46130

  Copy full SHA for cc46130