add rule to build dsp pars for SiPM data

[gipert]

Needs:

• https://github.com/legend-exp/legend-dataflow-overrides/pull/30
• https://github.com/legend-exp/legend-dataflow-overrides/pull/29
• https://github.com/legend-exp/legend-dataflow-config/pull/93
• https://github.com/legend-exp/legend-datasets/pull/37

@rosannadeckert

[codecov]

Codecov Report

Attention: Patch coverage is 14.28571% with 30 lines in your changes missing coverage. Please review.

Project coverage is 43.65%. Comparing base (8a76eea) to head (e4d6230).
Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
...w/src/legenddataflow/scripts/create_chankeylist.py	0.00%	17 Missing ⚠️
workflow/src/legenddataflow/execenv.py	0.00%	8 Missing ⚠️
workflow/src/legenddataflow/cfgtools.py	0.00%	3 Missing ⚠️
workflow/src/legenddataflow/patterns.py	71.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #98      +/-   ##
==========================================
+ Coverage   43.63%   43.65%   +0.02%     
==========================================
  Files          17       18       +1     
  Lines        1155     1159       +4     
==========================================
+ Hits          504      506       +2     
- Misses        651      653       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gipert]

why is snakefmt so stupid?

[gipert]

> snakemake --workflow-profile workflow/profiles/lngs /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5
Using workflow specific profile workflow/profiles/lngs for setting default command line arguments.
/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/workflow/src/legenddataflow/create_pars_keylist.py:115: UserWarning: No Keys found
  warnings.warn(msg, stacklevel=0)
Multiple includes of /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/workflow/rules/channel_merge.smk ignored
Multiple includes of /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/workflow/rules/channel_merge.smk ignored
Multiple includes of /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/workflow/rules/channel_merge.smk ignored
Assuming unrestricted shared filesystem usage.
host: legend-data
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 125
Rules claiming more threads will be scaled down.
Provided resources: mem_swap=3500
Job stats:
job          count
---------  -------
build_dsp        1
total            1

INFO: starting workflow
Select jobs to execute...
Execute 1 jobs...

[Thu Feb 13 14:05:43 2025]
localrule build_dsp:
    input: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/../ref-raw/generated/tier/raw/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_raw.lh5, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p08/r000/l200-p08-r000-cal-T%-par_dsp-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp_dplms-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p10/r000/l200-p10-r000-cal-20240226T104834Z-par_dsp_svm_hyperpars.yaml
    output: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/par/l200-p13-r001-ath-20241210T230220Z-par_dsp_db.yaml
    log: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/log/20250213T140532Z/tier_dsp/l200-p13-r001-ath-20241210T230220Z-tier_dsp.log
    jobid: 0
    reason: Missing output files: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5
    wildcards: datatype=ath, period=p13, run=r001, experiment=l200, timestamp=20241210T230220Z
    resources: tmpdir=/tmp, runtime=300, mem_swap=25

Traceback (most recent call last):
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/.snakemake/legend-dataflow/venv/bin/build-tier-dsp", line 10, in <module>
    sys.exit(build_tier_dsp())
             ^^^^^^^^^^^^^^^^
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/.snakemake/legend-dataflow/venv/lib/python3.12/site-packages/legenddataflow/scripts/tier/dsp.py", line 76, in build_tier_dsp
    f"ch{chan_map[chan].daq.rawid:07}/raw": Props.read_from(file)
         ~~~~~~~~^^^^^^
KeyError: 'B00000A'
[Thu Feb 13 14:06:04 2025]
Error in rule build_dsp:
    jobid: 0
    input: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/../ref-raw/generated/tier/raw/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_raw.lh5, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p08/r000/l200-p08-r000-cal-T%-par_dsp-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp_dplms-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p10/r000/l200-p10-r000-cal-20240226T104834Z-par_dsp_svm_hyperpars.yaml
    output: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/par/l200-p13-r001-ath-20241210T230220Z-par_dsp_db.yaml
    log: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/log/20250213T140532Z/tier_dsp/l200-p13-r001-ath-20241210T230220Z-tier_dsp.log (check log file(s) for error details)
    shell:
        PRODENV=/data1/shared/l200-p13/prodenv LGDO_BOUNDSCHECK=false DSPEED_BOUNDSCHECK=false PYGAMA_PARALLEL=false PYGAMA_FASTMATH=false DISABLE_TQDM=true apptainer exec --env=PRODENV=/data1/shared/l200-p13/prodenv --env=LGDO_BOUNDSCHECK=false --env=DSPEED_BOUNDSCHECK=false --env=PYGAMA_PARALLEL=false --env=PYGAMA_FASTMATH=false --env=DISABLE_TQDM=true --bind=/run/user/2074 /data2/public/prodenv/containers/legendexp_legend-base_latest_20241110203225.sif /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/.snakemake/legend-dataflow/venv/bin/build-tier-dsp --log /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/log/20250213T140532Z/tier_dsp/l200-p13-r001-ath-20241210T230220Z-tier_dsp.log --tier dsp --configs /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/config --metadata /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs --datatype ath --timestamp 20241210T230220Z --input /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/../ref-raw/generated/tier/raw/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_raw.lh5 --output /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5 --db-file /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/par/l200-p13-r001-ath-20241210T230220Z-par_dsp_db.yaml --pars-file /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p08/r000/l200-p08-r000-cal-T%-par_dsp-overwrite.yaml /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp-overwrite.yaml /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp_dplms-overwrite.yaml /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p10/r000/l200-p10-r000-cal-20240226T104834Z-par_dsp_svm_hyperpars.yaml 
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Trying to restart job 0.
Select jobs to execute...
Execute 1 jobs...

[Thu Feb 13 14:06:04 2025]
localrule build_dsp:
    input: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/../ref-raw/generated/tier/raw/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_raw.lh5, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p08/r000/l200-p08-r000-cal-T%-par_dsp-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp_dplms-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p10/r000/l200-p10-r000-cal-20240226T104834Z-par_dsp_svm_hyperpars.yaml
    output: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/par/l200-p13-r001-ath-20241210T230220Z-par_dsp_db.yaml
    log: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/log/20250213T140532Z/tier_dsp/l200-p13-r001-ath-20241210T230220Z-tier_dsp.log
    jobid: 0
    reason: Missing output files: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5
    wildcards: datatype=ath, period=p13, run=r001, experiment=l200, timestamp=20241210T230220Z
    resources: tmpdir=/tmp, runtime=300, mem_swap=25
Traceback (most recent call last):
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/.snakemake/legend-dataflow/venv/bin/build-tier-dsp", line 10, in <module>
    sys.exit(build_tier_dsp())
             ^^^^^^^^^^^^^^^^
  File "/data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/.snakemake/legend-dataflow/venv/lib/python3.12/site-packages/legenddataflow/scripts/tier/dsp.py", line 76, in build_tier_dsp
    f"ch{chan_map[chan].daq.rawid:07}/raw": Props.read_from(file)
         ~~~~~~~~^^^^^^
KeyError: 'B00000A'
[Thu Feb 13 14:06:44 2025]
Error in rule build_dsp:
    jobid: 0
    input: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/../ref-raw/generated/tier/raw/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_raw.lh5, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p08/r000/l200-p08-r000-cal-T%-par_dsp-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp_dplms-overwrite.yaml, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p10/r000/l200-p10-r000-cal-20240226T104834Z-par_dsp_svm_hyperpars.yaml
    output: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5, /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/par/l200-p13-r001-ath-20241210T230220Z-par_dsp_db.yaml
    log: /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/log/20250213T140532Z/tier_dsp/l200-p13-r001-ath-20241210T230220Z-tier_dsp.log (check log file(s) for error details)
    shell:
        PRODENV=/data1/shared/l200-p13/prodenv LGDO_BOUNDSCHECK=false DSPEED_BOUNDSCHECK=false PYGAMA_PARALLEL=false PYGAMA_FASTMATH=false DISABLE_TQDM=true apptainer exec --env=PRODENV=/data1/shared/l200-p13/prodenv --env=LGDO_BOUNDSCHECK=false --env=DSPEED_BOUNDSCHECK=false --env=PYGAMA_PARALLEL=false --env=PYGAMA_FASTMATH=false --env=DISABLE_TQDM=true --bind=/run/user/2074 /data2/public/prodenv/containers/legendexp_legend-base_latest_20241110203225.sif /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/.snakemake/legend-dataflow/venv/bin/build-tier-dsp --log /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/log/20250213T140532Z/tier_dsp/l200-p13-r001-ath-20241210T230220Z-tier_dsp.log --tier dsp --configs /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/config --metadata /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs --datatype ath --timestamp 20241210T230220Z --input /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/../ref-raw/generated/tier/raw/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_raw.lh5 --output /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tier/dsp/ath/p13/r001/l200-p13-r001-ath-20241210T230220Z-tier_dsp.lh5 --db-file /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/generated/tmp/par/l200-p13-r001-ath-20241210T230220Z-par_dsp_db.yaml --pars-file /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p08/r000/l200-p08-r000-cal-T%-par_dsp-overwrite.yaml /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp-overwrite.yaml /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/lar/p03/r000/l200-p03-r000-lar-T%-par_dsp_dplms-overwrite.yaml /data1/shared/l200-p13/prodenv/prod-orig/dev-v0.0.1/inputs/dataprod/overrides/dsp/cal/p10/r000/l200-p10-r000-cal-20240226T104834Z-par_dsp_svm_hyperpars.yaml 
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2025-02-13T140533.117908.snakemake.log
WorkflowError:
At least one job did not complete successfully.

[ggmarshall]

Ok so reason for this is that its grabbing from the overrides probably a p11 file and trying to apply but obviously the geds don't exist we just need to reset the overrides to not have files, will fix in my override update shortly

[gipert]

Tests should hopefully pass now, as I've merged all metadata PRs.

[gipert]

wow. finally