v0.4.1

What's Changed

• add timestamp to workflow by @rsriya in #192
• Transfer data to local after processing by @dachengx in #194
• Download log of jobs by @dachengx in #196
• Untar outpus if transferred to local by @dachengx in #197
• Check if there is no crucial user installed package by @dachengx in #198
• Upload miscellaneous information to DB by @dachengx in #199
• Ignore processed data while processing by @dachengx in #200
• Download raw_records* from only raw_records_rse by @dachengx in #201
• Use dependency tree for instruction of processing by @dachengx in #202
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #204
• Limit the number CPU usage of JAX and TensorFlow by @dachengx in #209
• Optimize the order of data_types in processing by @dachengx in #210
• Clearly list the data_type to process when submitting job by @dachengx in #211
• If self-installed package, do not upload to rucio by @dachengx in #212
• Set environmental variable of CPUs usage for all jobs by @dachengx in #213
• Still need local sites even if do not transfer to local by @dachengx in #214
• Set defualt dagman_maxjobs to be 5000 by @dachengx in #215
• Fix $lte for single run by @shenyangshi in #217
• Ask reporter to include the Outsource section of XENON_CONFIG by @dachengx in #222
• Forbid using development container by @dachengx in #221
• No per-chunk processing of MV by @dachengx in #223
• Force renew .dbtoken before workflow submission by @dachengx in #224
• Limit the filename in 255 characters by @dachengx in #226
• Allow staging-out lower data_types and keeping downloaded raw_records* by @dachengx in #228
• Decide whether the dependency exists based on specified RSE by @dachengx in #229
• Add script to test the resources usage: memory, time, storage by @dachengx in #230
• Automatically assign resources by @dachengx in #231
• Fix bug when multiple json files exists by @dachengx in #232
• Keep the folder being tarred by @dachengx in #233
• Stage out lower, combine or upper level outputs by @dachengx in #234
• List a directory in DAV by @dachengx in #235
• FIx bug that non-LED data_types is not filtered out in LED mode by @dachengx in #236
• Assign the resources usage of LED calibration by @dachengx in #237
• Allow multiple raw_records* RSEs by @dachengx in #238
• Set same requirements for lower level and combine jobs by @dachengx in #239
• Only extract input and output folders by @dachengx in #240
• Prioritize the first assign RSE by @dachengx in #241
• Allow multiple per-chunk processing by @dachengx in #242
• Small updates and fixes by @cfuselli in #244
• Change logic for EU sites by @cfuselli in #245
• Print locations of raw_records* by @dachengx in #243
• Lower memory usage of lower level processing by @dachengx in #246
• Revert part of #244 and #245 by @dachengx in #247
• Set special memory usage for LED calibration runs by @dachengx in #249
• Add constraint for SURFSARA2_USERDISK by @dachengx in #250
• Set lowest disk requirement as 200MB by @dachengx in #251
• Update memory usage of upper level jobs by @dachengx in #252
• Save lineage of data_types in summary.json by @dachengx in #253
• Copy .xenon_config to avoid its change after the job submission by @dachengx in #254
• Remove "download" jobs because they are not used by @dachengx in #255
• Add option to first stage the rucio files then download by @dachengx in #256
• Piecewise function for NV disk usage by @dachengx in #257
• Add suffix of combine and untar jobs by @dachengx in #258
• Piecewise function for NV disk usage again by @dachengx in #259
• Update memory usage of NV data by @dachengx in #260
• Copy local files to workflow folder to avoid overwritten by @dachengx in #261
• First dagman_static_retry retries will use the same resources by @dachengx in #262
• Bump version: 0.4.0 → 0.4.1 by @dachengx in #263

New Contributors

• @pre-commit-ci made their first contribution in #204
• @shenyangshi made their first contribution in #217
• @cfuselli made their first contribution in #244

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

• Testing single thread processor by @FaroutYLq in #163
• Remove CNAF from glidein site specificiation by @FaroutYLq in #164
• Reduce transfer thread and one more core in event level jobs by @FaroutYLq in #165
• fixed typo by @rsriya in #167
• No longer need production if you don't even try to upload to rucio by @FaroutYLq in #168
• Proposal to use pre-commit for continuous integration by @dachengx in #170
• Use environmental variable to pass the proxy by @dachengx in #171
• First restructure, make it more compatible with strax(en) by @dachengx in #172
• Move Shell to utilix, move all RSE related codes to config.py by @dachengx in #173
• Configuration of disk should be in MB by @dachengx in #174
• Tarball strax(en) and cutax for later user installation by @dachengx in #175
• Move around pegasus.conf by @dachengx in #176
• Remove unused python files by @dachengx in #177
• Apply per-chunk storage by @dachengx in #178
• Add more options for job submission by @dachengx in #179
• Move extraction of run_ids to utils by @dachengx in #180
• Use pyproject.toml to install outsource by @dachengx in #181
• Bump version of utilix by @dachengx in #182
• Upload outousrce to pypi by @dachengx in #190
• Bump to v0.4.0 by @dachengx in #191

New Contributors

• @rsriya made their first contribution in #167
• @dachengx made their first contribution in #170

Full Changelog: v0.3.1...v0.4.0

v0.3.1

Final status for SR1 processing.

What's Changed

• Given up afterpulses computation but doesn't hurt to merge by @FaroutYLq in #135
• Removed attach process by @FaroutYLq in #136
• Enable ref_mon_nv computation by @FaroutYLq in #138
• Debug when cuts_basic fail by @FaroutYLq in #139
• Try requiring AVX2 only by @FaroutYLq in #140
• echo Rundb api accessibility by @FaroutYLq in #141
• Avoid regeneration of .dbtoken on job node by @FaroutYLq in #144
• Debug combine wrapper by @FaroutYLq in #146
• Changed default to development by @FaroutYLq in #149
• Remove event_info re-registering by @FaroutYLq in #153
• Removed outdated cuts removal by @FaroutYLq in #154
• Retire grid proxy info by @FaroutYLq in #155
• Removed outdated plugin unregister by @FaroutYLq in #156
• Put back x509 proxy check by @FaroutYLq in #157
• Improved logging when using gfal2 by @rynge in #159
• Temporary bypassing integrity check in pegasus by @FaroutYLq in #158
• Adding more types for tar balling outputs by @FaroutYLq in #160
• No more local output by @FaroutYLq in #161

Full Changelog: v0.3.0...v0.3.1

v0.3.0

Stable version for SR1 reprocessing for v14 correction.

What's Changed

• Update README.md by @FaroutYLq in #116
• Update README.md by @FaroutYLq in #117
• Debug info for mismatch between chunks and metadata by @FaroutYLq in #115
• Not to updatedb if upload failed by @FaroutYLq in #119
• Added peak_basics into to-check in RunDB by @FaroutYLq in #120
• Debug missing comma in RunConfig by @FaroutYLq in #122
• Trying to add cuts_basic into to-process list by @FaroutYLq in #123
• Revisit check chunk n by @FaroutYLq in #124
• No records or records_nv to upload in rucio by @FaroutYLq in #125
• Save events and changing order by @FaroutYLq in #127
• Added output for events jobs by @FaroutYLq in #129
• Preupload to avoid rucio orphan files by @FaroutYLq in #133

Full Changelog: v0.2.2...v0.3.0

v0.2.2

Stable version for SR1 reprocessing for v13 correction.

What's Changed

• first commit of log_digest executable by @cecilia-ferrari in #91
• Improved usage of raw_records_rses by @FaroutYLq in #94
• Debug check_chunk_n by @FaroutYLq in #95
• Moveing to davs for staging by @rynge in #96
• More debug info before rucio attach by @FaroutYLq in #97
• Bug fix for staging by @FaroutYLq in #98
• Added event_ambience and event_shadow by @FaroutYLq in #99
• Debug dids_to_attach by @FaroutYLq in #101
• Adjustments: set output site to enable proper cleanup, exclude the Da… by @rynge in #100
• Removed hardcode Dakota by @FaroutYLq in #102
• removed s1_cnn temporarily by @FaroutYLq in #103
• Shadow and ambience as a new type by @FaroutYLq in #104
• put s1_cnn back by @FaroutYLq in #105
• imposed check_n_chunk also in runstrax by @FaroutYLq in #107
• More maxidle by @FaroutYLq in #108
• Added requirements for open firewall to the ports we need by @rynge in #109
• Timing info for admix.upload by @FaroutYLq in #111
• Prioritize heavy low level types in runstrax by @FaroutYLq in #113
• Forbid multi-threading by @FaroutYLq in #114

New Contributors

• @cecilia-ferrari made their first contribution in #91

Full Changelog: v0.2.0...v0.2.2

v0.1.0

This release is the first working version of outsource. It converts the xenon1t workflow into a Pegasus framework, apart from a few minor things related to the runDB control.