Db's needed locally to run without external condition data access

[katilp]

@caredg for the record:

When condition data are read from /cvmfs/cms-opendata.cern.ch the full database is read before the job starts. It can be several GBs. The analysis jobs typically need only a fraction of these condition data.

Downloading all these files could be avoided by having the needed database files locally.
For example, TriggerInfoTool/GeneralInfoAnalyzer can be run by copying nice databases in the local container and modifying the config file to access them locally instead of the full condition database.

The database files needed to the GeneralInfoAnalyzer job are:

~/CMSSW_5_3_32/src/TriggerInfoTool $ ls -lh *db
-rw-r--r-- 1 cmsusr cmsusr  84K Jun 20 14:54 AlCaRecoHLTpaths8e29_1e31_v13_offline.db
-rw-r--r-- 1    123    130 584K Jan 21  2016 L1GtPrescaleFactorsAlgoTrig_CRAFT09v2_hlt.db
-rw-r--r-- 1    123    130 260K Jan 21  2016 L1GtPrescaleFactorsTechTrig_CRAFT09v2_hlt.db
-rw-r--r-- 1 cmsusr cmsusr  38K Jun 20 18:27 L1GtStableParameters_CRAFT09_hlt.db
-rw-r--r-- 1    123    130 231K Jan 21  2016 L1GtTriggerMaskAlgoTrig_CRAFT09v2_hlt.db
-rw-r--r-- 1    123    130 214K Jan 21  2016 L1GtTriggerMaskTechTrig_CRAFT09v2_hlt.db
-rw-r--r-- 1    123    130  34K Jan 21  2016 L1GtTriggerMaskVetoAlgoTrig_CRAFT09_hlt.db
-rw-r--r-- 1    123    130 125K Jan 21  2016 L1GtTriggerMaskVetoTechTrig_CRAFT09v2_hlt.db
-rw-r--r-- 1    123    130 410K Jan 21  2016 L1GtTriggerMenu_CRAFT09_hlt.db

and they can be copied to the container from a host with /cvmfs/cms-opendata-conddb.cern.ch mounted with

$ docker cp /cvmfs/cms-opendata-conddb.cern.ch/FT_53_LV5_AN1/L1GtPrescaleFactorsAlgoTrig_CRAFT09v2_hlt.db my_od:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool
$ docker cp /cvmfs/cms-opendata-conddb.cern.ch/FT_53_LV5_AN1/L1GtPrescaleFactorsTechTrig_CRAFT09v2_hlt.db my_od:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool
[...]

The modifications needed for the config file are:

#needed to get the actual prescale values used from the global tag
#process.load('Configuration.StandardSequences.FrontierConditions_GlobalTag_cff')
#process.GlobalTag.connect = cms.string('sqlite_file:/cvmfs/cms-opendata-conddb.cern.ch/FT_53_LV5_AN1_RUNA.db')
#process.GlobalTag.globaltag = 'FT_53_LV5_AN1'

process.load("CondCore.DBCommon.CondDBSetup_cfi")

# Define CondDB tags
from CondTools.L1Trigger.L1CondEnum_cfi import L1CondEnum
from CondTools.L1Trigger.L1O2OTags_cfi import initL1O2OTags
initL1O2OTags()

process.db0 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/AlCaRecoHLTpaths8e29_1e31_v13_offline.db')
)

process.db1 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      cms.PSet(
        record = cms.string('L1GtStableParametersRcd'),
        tag = cms.string( "L1GtStableParameters_" + initL1O2OTags.tagBaseVec[ L1CondEnum.L1GtStableParameters ] )
      )
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/L1GtStableParameters_CRAFT09_hlt.db')
)

process.db2 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      cms.PSet(
        record = cms.string('L1GtPrescaleFactorsAlgoTrigRcd'),
        tag = cms.string( 'L1GtPrescaleFactorsAlgoTrig_' + initL1O2OTags.tagBaseVec[ L1CondEnum.L1GtPrescaleFactorsAlgoTrig ] )
      )
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/L1GtPrescaleFactorsAlgoTrig_CRAFT09v2_hlt.db')
)

process.db3 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      cms.PSet(
        record = cms.string('L1GtPrescaleFactorsTechTrigRcd'),
        tag = cms.string( 'L1GtPrescaleFactorsTechTrig_' + initL1O2OTags.tagBaseVec[ L1CondEnum.L1GtPrescaleFactorsTechTrig ] )
      )
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/L1GtPrescaleFactorsTechTrig_CRAFT09v2_hlt.db')
)

process.db4 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      cms.PSet(
        record = cms.string('L1GtTriggerMaskAlgoTrigRcd'),
        tag = cms.string( 'L1GtTriggerMaskAlgoTrig_' + initL1O2OTags.tagBaseVec[ L1CondEnum.L1GtTriggerMaskAlgoTrig ] )
      )
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/L1GtTriggerMaskAlgoTrig_CRAFT09v2_hlt.db')
)

process.db5 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      cms.PSet(
        record = cms.string('L1GtTriggerMaskTechTrigRcd'),
        tag = cms.string( 'L1GtTriggerMaskTechTrig_' + initL1O2OTags.tagBaseVec[ L1CondEnum.L1GtTriggerMaskTechTrig ] )
      )
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/L1GtTriggerMaskTechTrig_CRAFT09v2_hlt.db')
)


process.db6 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      cms.PSet(
        record = cms.string('L1GtTriggerMaskVetoAlgoTrigRcd'),
        tag = cms.string( 'L1GtTriggerMaskVetoAlgoTrig_' + initL1O2OTags.tagBaseVec[ L1CondEnum.L1GtTriggerMaskVetoAlgoTrig ] )
      )
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/L1GtTriggerMaskVetoAlgoTrig_CRAFT09_hlt.db')
)

process.db7 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      cms.PSet(
        record = cms.string('L1GtTriggerMaskVetoTechTrigRcd'),
        tag = cms.string( 'L1GtTriggerMaskVetoTechTrig_' + initL1O2OTags.tagBaseVec[ L1CondEnum.L1GtTriggerMaskVetoTechTrig ] )
      )
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/L1GtTriggerMaskVetoTechTrig_CRAFT09v2_hlt.db')
)

process.db8 = cms.ESSource("PoolDBESSource",
    process.CondDBSetup,
    toGet = cms.VPSet(
      cms.PSet(
        record = cms.string('L1GtTriggerMenuRcd'),
        tag = cms.string( 'L1GtTriggerMenu_' + initL1O2OTags.tagBaseVec[ L1CondEnum.L1GtTriggerMenu ] )
      )
      ),
    timetype = cms.string('runnumber'),
    connect = cms.string('sqlite_file:/home/cmsusr/CMSSW_5_3_32/src/TriggerInfoTool/L1GtTriggerMenu_CRAFT09_hlt.db')
)

and the output is as:

 ~/CMSSW_5_3_32/src/TriggerInfoTool $ cmsRun GeneralInfoAnalyzer/python/triggerinfoanalyzer_cfg.py
210620 21:24:21 1220 Xrd: XrdClientConn: Error resolving this host's domain name.
210620 21:24:21 1220 secgsi_InitProxy: cannot access private key file: /home/cmsusr/.globus/userkey.pem
20-Jun-2021 21:24:22 CEST  Initiating request to open file root://eospublic.cern.ch//eos/opendata/cms/Run2011A/ElectronHad/AOD/12Oct2013-v1/20001/001F9231-F141-E311-8F76-003048F00942.root
20-Jun-2021 21:24:26 CEST  Successfully opened file root://eospublic.cern.ch//eos/opendata/cms/Run2011A/ElectronHad/AOD/12Oct2013-v1/20001/001F9231-F141-E311-8F76-003048F00942.root
Here is the information passed to the constructor:
Configuration:
   ProcessName = HLT
   TriggerName = @
   DataSetName = SingleMu
   TriggerResultsTag = TriggerResults::HLT
   TriggerEventTag = hltTriggerSummaryAOD::HLT
Begin processing the 1st record. Run 166782, Event 340184599, LumiSection 309 at 20-Jun-2021 21:25:34.094 CEST
Currently analyzing trigger HLT_IsoMu12_v5
TriggerInfoAnalyzer::analyzeTrigger: path HLT_IsoMu12_v5 [120] prescales L1T,HLT: 1,100
 Trigger path status: WasRun=1 Accept=0 Error =0
 Last active module - label/type: hltL1sL1SingleMu7/HLTLevel1GTSeed [9 out of 0-62 on this path]
Currently analyzing trigger HLT_IsoMu15_v9
TriggerInfoAnalyzer::analyzeTrigger: path HLT_IsoMu15_v9 [121] prescales L1T,HLT: 1,40
 Trigger path status: WasRun=1 Accept=0 Error =0
 Last active module - label/type: hltL1sL1SingleMu10/HLTLevel1GTSeed [9 out of 0-62 on this path]

[...]

Currently analyzing trigger HLT_Mu8_v3
TriggerInfoAnalyzer::analyzeTrigger: path HLT_Mu8_v3 [112] prescales L1T,HLT: 60,40
 Trigger path status: WasRun=1 Accept=0 Error =0
 Last active module - label/type: hltL1sL1SingleMu3/HLTLevel1GTSeed [9 out of 0-49 on this path]
20-Jun-2021 21:25:34 CEST  Closed file root://eospublic.cern.ch//eos/opendata/cms/Run2011A/ElectronHad/AOD/12Oct2013-v1/20001/001F9231-F141-E311-8F76-003048F00942.root

=============================================

MessageLogger Summary

 type     category        sev    module        subroutine        count    total
 ---- -------------------- -- ---------------- ----------------  -----    -----
    1 fileAction           -s file_close                             1        1
    2 fileAction           -s file_open                              2        2

 type    category    Examples: run/evt        run/evt          run/evt
 ---- -------------------- ---------------- ---------------- ----------------
    1 fileAction           PostEndRun
    2 fileAction           pre-events       pre-events

Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
System                  3                   3

[katilp]

An update: @clelange prepared a new main db file that can be used to avoid the separate process blocks for each database file in the config file.

It is a stripped version of /cvmfs/cms-opendata-conddb.cern.ch/FT_53_LV5_AN1.db and points to a local FT_53_LV5_AN1 directory instead of /cvmfs/cms-opendata-conddb.cern.ch/FT_53_LV5_AN1 for the database files

The usage would then be:

• have that file in the directory where the jobs is run
• have the local db files in a local FT_53_LV5_AN1 directory
• in the config file:

  process.GlobalTag.connect = cms.string('sqlite_file:FT_53_LV5_AN1_stripped.db')
  process.GlobalTag.globaltag = 'FT_53_LV5_AN1::All'