release.notes

[v6r5-pre3]

NEW: Executor framework

*Core
NEW: MySQL.py - added Test case for Time.dateTime time stamps
NEW: MySQL.py - insertFields and updateFields can get values via Lists or Dicts
NEW: DataIntegrityDB - use the new methods from MySQL and add test cases
NEW: DataIntegrityHandler - check connection to DB and create tables (or update their schema)
NEW: DataLoggingDB - use the new methods from MySQL and add test cases
NEW: DataLoggingHandler - check connection to DB and create tables (or update their schema)
FIX: ProcessPool - killing stuck workers after timeout
CHANGE: DB will throw a RuntimeException instead of a sys.exit in case it can't contact the DB
CHANGE: Several improvements on DISET
CHANGE: Fixed all DOS endings to UNIX
CHANGE: Agents, Services and Executors know how to react to CSSection/Module and react accordingly
NEW: install tools are updated to deal with executors
FIX: dirac-install - add -T/--Timeout option to define timeout for distribution downloads
BUGFIX: avoid PathFinder.getServiceURL and use Client class ( DataLoggingClient,LfcFileCayalogProxyClient ) 

*RSS
CHANGE: removed code execution from __init__
CHANGE: removed unused methods
NEW: Log all policy results 

*Resources
NEW: updated SSHComputingElement which allows multiple job submission

*WMS
CHANGE: WMS Optimizers are now executors
CHANGE: SandboxStoreClient can directly access the DB if available
CHANGE: Moved JobDescription and improved into JobManifest
FIX: typo in JobLoggingDB
NEW: JobState/CachedJobState allow access to the Job via DB/JobStateSync Service automatically
BUGFIX: DownloadInputData - when not enough disk space, message was using "buffer" while it should be using "data"
FIX: the sandboxmetadataDB explosion when using the sandboxclient without direct access to the DB
NEW: Added support for reset/reschedule in the OptimizationMind
CHANGE: Whenever a DB is not properly initialized it will raise a catchable RuntimeError exception 
        instead of silently returning
FIX: InputDataResolution - just quick mod for easier extensibility, plus removed some LHCb specific stuff
NEW: allow jobids in a file in dirac-wms-job-get-output
NEW: JobManager - zfill in %n parameter substitution to allow alphabetical sorting


*Transformation
FIX: TransformationAgent - a small improvement: now can pick the prods status to handle from the CS, 
     plus few minor corrections (e.g. logger messages)

*Accounting
NEW: AccountingDB - added retrieving RAW records for internal stuff
FIX: AccountingDB - fixed some logic for readonly cases

[v6r4p7]

*WMS
FIX: SandboxStoreClient - catch exception when SandboxMetadataDB can not be instantiated

*DMS
BUGFIX: StorageElement - added Lost and Unavailable metadata to the output of getFileMetadata()

[v6r4p6]

*Core
FIX: proper generation of the release notes

*DMS
FIX: more protection in StrategyHandler 

[v6r4p5]

*Configuration
NEW: Added function to generate Operations CS paths

*Core
FIX: Added proper ProcessPool checks and finalisation

*DataManagement
FIX: don't set Files.Status to Failed for non-existign files, failover transfers won't go
FIX: remove @classmethods here and there to unblock requestHolder
CHANGE: RAB, TA: change task timeout: 180 and 600 (was 600 and 900 respectively)
FIX: sorting replication tree by Ancestor, not hopAncestorgit add DataManagementSystem/Agent/TransferAgent.py
NEW: TA: add finalize
CHANGE: TransferAgent: add AcceptableFailedFiles to StrategyHandler to ban FTS channel from scheduling
FIX: if there is no failed files, put an empty dict


*RSS
FIX: RSS is setting Allowed but the StorageElement checks for Active

*Workflows
FIX: Part of WorfklowTask rewritten to fix some issues and allow 'ANY' as site

*Transformation
FIX: Wrong calls to TCA::cleanMetadataCatalogFiles

[v6r4p4]

*Core
FIX: Platform.py - check if Popen.terminate is available (only from 2.6)

[v6r4p3]

*Core
FIX: ProcessPool with watchdog and timeouts - applied in v6r3 first

[v6r4p2]

*StorageManagement
BUGFIX: StorageElement - staging is a Read operation and should be allowed as such

*WMS
BUGFIX: InProcessComputingElement, JobAgent - proper return status code from the job wrapper

*Core
FIX: Platform - manage properly the case of exception in the ldconfig execution

[v6r4p1]

*DMS
FIX: TransferDB.getChannelObservedThroughput - the channelDict was created in a wrong way

*RSS
FIX: ResourceStatus was not returning Allowed by default

[v6r4]

*Core
FIX: dirac-install-db.py: addDatabaseOptionsToCS has added a new keyed argument
NEW: SGETimeLeft.py: Support for SGE backend
FIX: If several extensions are installed, merge ConfigTemplate.cfg
NEW: Service framework - added monitoring of file descriptors open
NEW: Service framework - Reduced handshake timeout to prevent stuck threads
NEW: MySQL class with new high level methods - buildCondition,insertFields,updateFields
     deleteEntries, getFields, getCounters, getDistinctAttributeValues
FIX: ProcessPool - fixes in the locking mechanism with LockRing, stopping workers when the
     parent process is finished     
FIX: Added more locks to the LockRing
NEW: The installation tools are updated to install components by name with the components module specified as an option

*DMS
FIX: TransferDB.py - speed up the Throughput determination
NEW: dirac-dms-add-files: script similar to dirac-dms-remove-files, 
     allows for 1 file specification on the command line, using the usual dirac-dms-add-file options, 
     but also can take a text file in input to upload a bunch of files. Exit code is 0 only if all 
     was fine and is different for every error found. 
NEW: StorageElementProxy- support for data downloading with http protocol from arbitrary storage, 
     needed for the web data download
BUGFIX: FileCatalogCLI - replicate operation does a proper replica registration ( closes #5 )     
FIX: ReplicaManager - __cleanDirectory now working and thus dirac-dms-clean-directory

*WMS
NEW: CPU normalization script to run a quick test in the pilot, used by the JobWrapper
     to report the CPU consumption to the accounting
FIX: StalledJobAgent - StalledTimeHours and FailedTimeHours are read each cycle, refer to the 
     Watchdog heartBeat period (should be renamed); add NormCPUTime to Accounting record
NEW: SiteDirector - support for the operation per VO in multi-VO installations
FIX: StalledJobAgent - get ProcessingType from JDL if defined
BUGFIX: dirac-wms-job-peek - missing printout in the command
NEW: SiteDirector - take into account the number of already waiting pilots when evaluating the number of pilots to submit
FIX: properly report CPU usage when the Watchdog kill the payload.

*RSS
BUGFIX: Result in ClientCache table is a varchar, but the method was getting a datetime
NEW: CacheFeederAgent - VOBOX and SpaceTokenOccupancy commands added (ported from LHCbDIRAC)
CHANGE: RSS components get operational parameters from the Operations handler

*DataManagement
FIX: if there is no failed files, put an empty dict

*Transformation
FIX: Wrong calls to TCA::cleanMetadataCatalogFiles

[v6r3p14]

* Core

BUGFIX: ProcessPool.py: clean processing and finalisation
BUGFIX: Pfn.py: don't check for 'FileName' in pfnDict

* DMS

NEW: dirac-dms-show-fts-status.py: script showing last hour history for FTS channels
NEW: TransferDBMonitoringHandler.py: new function exporting FST channel queues
BUGFIX: TransferAgent.py,RemovalAgent.py,RegistrationAgent.py - unlinking of temp proxy files, corection of values sent to gMonitor
BUGFIX: StrategyHandler - new config option 'AcceptableFailedFiles' to unblock scheduling for channels if problematic transfers occured for few files
NEW: TransferAgent,RemovalAgent,RegistrationAgent - new confing options for setting timeouts for tasks and ProcessPool finalisation
BUGFIX: ReplicaManager.py - reverse sort of LFNs when deleting files and directories to avoid blocks
NEW: moved StrategyHandler class def to separate file under DMS/private

* TMS

FIX: TransformationCleaningAgent.py: some refactoring, new way of disabling/enabline execution by 'EnableFlag' config option

*Transformation
NEW: Handle TargetSE in TaskManager for WorkflowTasks. It will make and AND of sites associated 
     to each SE in TargetSE and those given in Site, if any. This is needed until scheduling to SEs 
     is supported on the WMS
 
*Resources
NEW: SSHLSFComputingElement added 

[v6r3p16]

*Core
FIX: proper generation of the release notes

*DMS
FIX: more protection in StrategyHandler 

[v6r3p15]

*DataManagement
FIX: if there is no failed files, put an empty dict

*Transformation
FIX: Wrong calls to TCA::cleanMetadataCatalogFiles


[v6r3p14]

* Core

BUGFIX: ProcessPool.py: clean processing and finalisation
BUGFIX: Pfn.py: don't check for 'FileName' in pfnDict

* DMS

NEW: dirac-dms-show-fts-status.py: script showing last hour history for FTS channels
NEW: TransferDBMonitoringHandler.py: new function exporting FST channel queues
BUGFIX: TransferAgent.py,RemovalAgent.py,RegistrationAgent.py - unlinking of temp proxy files, corection of values sent to gMonitor
BUGFIX: StrategyHandler - new config option 'AcceptableFailedFiles' to unblock scheduling for channels if problematic transfers occured for few files
NEW: TransferAgent,RemovalAgent,RegistrationAgent - new confing options for setting timeouts for tasks and ProcessPool finalisation
BUGFIX: ReplicaManager.py - reverse sort of LFNs when deleting files and directories to avoid blocks
NEW: moved StrategyHandler class def to separate file under DMS/private

* TMS

FIX: TransformationCleaningAgent.py: some refactoring, new way of disabling/enabline execution by 'EnableFlag' config option

[v6r3p13]

*Core
FIX: Added proper ProcessPool checks and finalisation

*DataManagement
FIX: don't set Files.Status to Failed for non-existign files, failover transfers won't go
FIX: remove @classmethods here and there to unblock requestHolder
CHANGE: RAB, TA: change task timeout: 180 and 600 (was 600 and 900 respectively)
FIX: sorting replication tree by Ancestor, not hopAncestorgit add DataManagementSystem/Agent/TransferAgent.py
NEW: TA: add finalize
CHANGE: TransferAgent: add AcceptableFailedFiles to StrategyHandler to ban FTS channel from scheduling

[v6r3p12]

*Core
FIX: Platform.py - check if Popen.terminate is available (only from 2.6)

[v6r3p11]

*Core
FIX: ProcessPool with watchdog and timeouts

[v6r3p10]

*StorageManagement
BUGFIX: StorageElement - staging is a Read operation and should be allowed as such

*WMS
BUGFIX: InProcessComputingElement, JobAgent - proper return status code from the job wrapper

*Core
FIX: Platform - manage properly the case of exception in the ldconfig execution

[v6r3p9]

*DMS
FIX: TransferDB.getChannelObservedThroughput - the channelDict was created in a wrong way

[v6r3p8]

*Web
CHANGE: return back to the release web2012041601

[v6r3p7]

*Transformation
FIX: TransformationCleaningAgent - protection from deleting requests with jobID 0 
Ä
[v6r3p6]

*Core
FIX: dirac-install-db - proper key argument (follow change in InstallTools)
FIX: ProcessPool - release all locks every time WorkignProcess.run is executed, more fixes to come
FIX: dirac-configure - for Multi-Community installations, all vomsdir/vomses files are now created

*WMS
NEW: SiteDirector - add pilot option with CE name to allow matching of SAM jobs.
BUGFIX: dirac-pilot - SGE batch ID was overwriting the CREAM ID
FIX: PilotDirector - protect the CS master if there are at least 3 slaves
NEW: Watchdog - set LocalJobID in the SGE case

[v6r3p5]

*Core:
BUGFIX: ProcessPool - bug making TaskAgents hang after max cycles
BUGFIX: Graphs - proper handling plots with data containing empty string labels
FIX: GateWay - transfers were using an old API
FIX: GateWay - properly calculate the gateway URL
BUGFIX: Utilities/Pfn.py - bug in pfnunparse() when concatenating Path and FileName

*Accounting
NEW: ReportGenerator - make AccountingDB readonly
FIX: DataCache - set daemon the datacache thread
BUGFIX: BasePlotter - proper handling of the Petabyte scale data

*DMS:
BUGFIX: TransferAgent, RegistrationTask - typos 

[v6r3p4]

*DMS:
BUGFIX: TransferAgent - wrong value for failback in TA:execute

[v6r3p3]

*Configuration
BUGFIX: Operations helper - typo

*DMS:
FIX: TransferAgent - change the way of redirecting request to task

[v6r3p2]

*DMS
FIX: FTSRequest - updating metadata for accouting when finalizing FTS requests

*Core
FIX: DIRAC/__init__.py - default version is set to v6r3

[v6r3p1]

*WMS
CHANGE: Use ResourcesStatus and Resources helpers in the InputDataAgent logic

*Configuration
NEW: added getStorageElementOptions in Resources helper

*DMS
FIX: resourceStatus object created in TransferAgent instead of StrategyHandler

[v6r3]

*Core
NEW: Added protections due to the process pool usage in the locking logic

*Resources
FIX: LcgFileCatalogClient - reduce the number of retries: LFC_CONRETRY = 5 to 
     avoid combined catalog to be stuck on a faulty LFC server
     
*RSS
BUGFIX: ResourceStatus - reworked helper to keep DB connections     

*DMS
BUGFIX: ReplicaManager::CatalogBase::_callFileCatalogFcnSingleFile() - wrong argument

*RequestManagement
FIX: TaskAgents - set timeOut for task to 10 min (15 min)
NEW: TaskAgents - fill in Error fields in case of failing operations

*Interfaces
BUGFIX: dirac-wms-select-jobs - wrong use of the Dirac API

[v6r2p9]

*Core
FIX: dirac-configure - make use of getSEsForSite() method to determine LocalSEs

*WMS
NEW: DownloadInputData,InputDataByProtocol - check Files on Tape SEs are on Disk cache 
     before Download or getturl calls from Wrapper
CHANGE: Matcher - add Stalled to "Running" Jobs when JobLimits are applied   
CHANGE: JobDB - allow to specify required platform as Platform JDL parameter,
        the specified platform is taken into account even without /Resources/Computing/OSCompatibility section

*DMS
CHANGE: dirac-admin-allow(ban)-se - removed lhcb-grid email account by default, 
        and added switch to avoid sending email
FIX: TaskAgents - fix for non-existing files
FIX: change verbosity in failoverReplication 
FIX: FileCatalog - remove properly metadata indices 
BUGFIX: FileManagerBase - bugfix in the descendants evaluation logic  
FIX: TransferAgent and TransferTask - update Files.Status to Failed when ReplicaManager.replicateAndRegister 
     will fail completely; when no replica is available at all.

*Core
FIX: dirac-pilot - default lcg bindings version set to 2012-02-20

[v6r2p8]

*DMS:
CHANGE: TransferAgent - fallback to task execution if replication tree is not found

[v6r2p7]

*WMS
BUGFIX: SiteDirector - wrong CS option use: BundleProxy -> HttpProxy
FIX: SiteDirector - use short lines in compressed/encoded files in the executable
     python script

[v6r2p6]

*DataManagement
FIX: Bad logic in StrategyHandler:MinimiseTotalWait

*Core
CHANGE: updated GGUS web portal URL

*RSS
BUGFIX: meta key cannot be reused, it is popped from dictionary

*Framework
FIX: The Gateway service does not have a handler
NEW: ConfingTemplate entry for Gateway
FIX: distribution notes allow for word wrap

*WorkloadManagement
FIX: avoid unnecessary call if no LFN is left in one of the SEs
FIX: When Uploading job outputs, try first Local SEs, if any


[v6r2p5]

*RSS
BUGFIX: several minor bug fixes

*RequestManagement
BUGFIX: RequestDBMySQL - removed unnecessary request type check

*DMS
BUGFIX: FileCatalogClienctCLI - wrong evaluation of the operation in the find command
NEW: FileCatalog - added possibility to remove specified metadata for a given path 
BUGFIX: ReplicaManager - wrong operation order causing failure of UploadLogFile module

*Core
NEW: dirac-install - generate cshrc DIRAC environment setting file for the (t)csh 

*Interfaces
CHANGE: Job - added InputData to each element in the ParametricInputData

*WMS
CHANGE: dirac-jobexec - pass ParametericInputData to the workflow as a semicolon separated string

[v6r2p4]

*WMS
BUGFIX: StalledJobAgent - protection against jobs with no PilotReference in their parameters
BUGFIX: WMSAdministratorHandler - wrong argument type specification for getPilotInfo method

*StorageManagement
BUGFIX: RequestFinalizationAgent - no method existence check when calling RPC method

[v6r2p3]

*WMS
CHANGE: Matcher - fixed the credentials check in requestJob() to simplify it

*ConfigurationSystem
CHANGE: Operations helper - fix that allow no VO to be defined for components that do not need it

*Core
BUGFIX: InstallTools - when applying runsvctrl to a list of components make sure that the config server is treated first and the sysadmin service - last
        
[v6r2p2]

*WMS
BUGFIX: Matcher - restored logic for checking private pilot asking for a given DN for belonging to the same group with JOB_SHARING property.

[v6r2p1]

*RequestManagementSystem
BUGFIX: RequestCleaningAgent - missing import of the "second" interval definition 

[v6r2]

*General
FIX: replaced use of exec() python statement in favor of object method execution

*Accounting
CHANGE: Accounting 'byte' units are in powers of 1000 instead of powers of 1024 (closes #457)

*Core
CHANGE: Pfn.py - pfnparse function rewritten for speed up and mem usage, unit test case added
FIX: DISET Clients are now thread-safe. Same clients used twice in different threads was not 
closing the previous connection
NEW: reduce wait times in DISET protocol machinery to improve performance    
NEW: dirac-fix-mysql-script command to fix the mysql start-up script for the given installation
FIX: TransferClient closes connections properly
FIX: DISET Clients are now thread-safe. Same client used twice in different threads will not close the previous connection
CHANGE: Beautification and reduce wait times to improve performance
NEW: ProcessPool - added functionality to kill all children processes properly when destroying ProcessPool objects
NEW: CS Helper for LocalSite section, with gridEnv method
NEW: Grid module will use Local.gridEnv if nothing passed in the arguments
CHANGE: Add deprecated sections in the CS Operations helper to ease the transition
FIX: dirac-install - execute dirac-fix-mysql-script, if available, to fix the mysql.server startup script
FIX: dirac-distribution - Changed obsoleted tar.list file URL
FIX: typo in dirac-admin-add-host in case of error
CHANGE: dirac-admin-allow(ban)-se - use diracAdmin.sendMail() instead of NotificationClient.sendMail()

*Framework
BUGFIX: UserProfileDB - no more use of "type" variable as it is a reserved keyword 

*RequestManagement:
FIX: RequestDBFile - more consistent treatment of requestDB Path
FIX: RequestMySQL - Execution order is evaluated based on not Done state of subrequests
NEW: RequestCleaningAgent - resetting Assigned requests to Waiting after a configurable period of time

*RSS
CHANGE: RSS Action now inherits from a base class, and Actions are more homogeneous, they all take a uniform set of arguments. The name of modules has been changed from PolType to Action as well.
FIX: CacheFeederAgent - too verbose messages moved to debug instead of info level
BUGFIX: fixed a bug preventing RSS clients to connect to the services     
FIX: Proper services synchronization
FIX: Better handling of exceptions due to timeouts in GOCDBClient   
FIX: RSS.Notification emails are sent again
FIX: Commands have been modified to return S_OK, S_ERROR inside the Result dict. This way, policies get a S_ERROR / S_OK object. CacheFeederAgent has been updated accordingly.
FIX: allow clients, if db connection fails, to reconnect ( or at least try ) to the servers.
CHANGE: access control using CS Authentication options. Default is SiteManager, and get methods are all.
BUGFIX: MySQLMonkey - properly escaped all parameters of the SQL queries, other fixes.
NEW: CleanerAgent renamed to CacheCleanerAgent
NEW: Updated RSS scripts, to set element statuses and / or tokens.
NEW: Added a new script, dirac-rss-synch
BUGFIX: Minor bugfixes spotted on the Web development
FIX: Removed useless decorator from RSS handlers
CHANGE: ResourceStatus helper tool moved to RSS/Client directory, no RSS objects created if the system is InActive
CHANGE: Removed ClientFastDec decorator, using a more verbose alternative.
CHANGE: Removed useless usage of **kwargs on helper functions.  
NEW: added getSESitesList method to RSSClient      
FIX: _checkFloat() checks INTEGERS, not datetimes

*DataManagement
CHANGE: refactoring of DMS agents executing requests, allow requests from arbitrary users
NEW: DFC - allow to specify multiple replicas, owner, mode when adding files
CHANGE: DFC - optimization of the directory size evaluation
NEW: Added CREATE TEMPORARY TABLES privilege to FileCatalogDB
CHANGE: DFC - getCatalogCounters() update to show numbers of directories
NEW: lfc_dfc_copy script to migrate data from LFC to DFC
FIX: dirac-dms-user-lfns - fixed the case when the baseDir is specified
FIX: FTS testing scripts were using sys.argv and getting confused if options are passed
NEW: DFC - use DirectoryUsage tables for the storage usage evaluations
NEW: DFC - search by metadata can be limited to a given directory subtree
NEW: DFC - search by both directory and file indexed metadata
BUGFIX: DFC - avoid crash if no directories or files found in metadata query
NEW: DFC FileCatalogHandler - define database location in the configuration
NEW: DFC - new FileCatalogFactory class, possibility to use named DFC services
FIX: FTSMonitor, FTSRequest - fixes in handling replica registration, setting registration requests in FileToCat table for later retry
FIX: Failover registration request in the FTS agents.      
FIX: FTSMonitor - enabled to register new replicas if even the corresponding request were removed from the RequestManagement 
FIX: StorageElement - check if SE has been properly initialized before executing any method     
CHANGE: LFC client getReplica() - make use of the new bulk method lfc.lfc_getreplicasl()
FIX: LFC client - protect against getting None in lfc.lfc_readdirxr( oDirectory, "" )  
FIX: add extra protection in dump method of StorageElement base class
CHANGE: FailoverTransfer - create subrequest per catalog if more than one catalog

*Interface
NEW: Job.py - added method to handle the parametric parameters in the workflow. They are made available to the workflow_commons via the key 'GenericParameters'.
FIX: Dirac.py - fix some type checking things
FIX: Dirac.py - the addFile() method can now register to more than 1 catalog.

*WMS
FIX: removed dependency of the JobSchedulingAgent on RSS. Move the getSiteTier functionality to a new CS Helper.
FIX: WMSAdministratorHandler - Replace StringType by StringTypes in the export methods argument type
FIX: JobAgent - Set explicitly UseServerCertificate to "no" for the job executable
NEW: dirac-pilot - change directory to $OSG_WN_TMP on OSG sites
FIX: SiteDirector passes jobExecDir to pilot, this defaults to "." for CREAM CEs. It can be set in the CS. It will not make use of $TMPDIR in this case.
FIX: Set proper project and release version to the SiteDirector     
NEW: Added "JobDelay" option for the matching, refactored and added CS options to the matcher
FIX: Added installation as an option to the pilots and random MyProxyServer
NEW: Support for parametric jobs with parameters that can be of List type

*Resources
NEW: Added SSH Grid Engine Computing Element
NEW: Added SSH Computing Element
FIX: make sure lfc client will not try to connect for several days

*Transformation
FIX: TransformationDB - in setFileStatusForTransformation() reset ErrorCount to zero if "force" flag and    the new status is "unused"
NEW: TransformationDB - added support for dictionary in metadata for the InputDataQuery mechanism     

[v6r1p13]

*WMS
FIX: JobSchedulingAgent - backported from v6r2 use of Resources helper

[v6r1p12]

*Accounting
FIX: Properly delete cached plots

*Core
FIX: dirac-install - run externals post install after generating the versions dir

[v6r1p11]

*Core
NEW: dirac-install - caches locally the externals and the grid bundle
FIX: dirac-distribution - properly generate releasehistory and releasenotes

[v6r1p10]

*WorloadManagement
FIX: JobAgent - set UseServerCertificate option "no" for the job executable

[v6r1p9]

*Core
FIX: dirac-configure - set the proper /DIRAC/Hostname when defining /LocalInstallation/Host

*DataManagement
FIX: dirac-dms-user-lfns - fixed the case when the baseDir is specified
BUGFIX: dirac-dms-remove-files - fixed crash in case of returned error report in a form of dictionary 

[v6r1p8]

*Web
FIX: restored Run panel in the production monitor

*Resources
FIX: FileCatalog - do not check existence of the catalog client module file

[v6r1p7]

*Web
BUGFIX: fixed scroll bar in the Monitoring plots view

[v6r1p6]

*Core
FIX: TransferClient closes connections properly

[v6r1p5]

*Core
FIX: DISET Clients are now thread-safe. Same clients used twice in different threads was not 
     closing the previous connection
NEW: reduce wait times in DISET protocol machinery to improve performance   

[v6r1p4]

*RequestManagement
BUGFIX: RequestContainer - in isSubRequestDone() treat special case for subrequests with files

*Transformation
BUGFIX: TransformationCleaningAgent - do not clear requests for tasks with no associated jobs

[v6r1p3]

*Framework
NEW: Pass the monitor down to the request RequestHandler
FIX: Define the service location for the monitor
FIX: Close some connections that DISET was leaving open

[v6r1p2]

*WorkloadManagement
BUGFIX: JobSchedulingAgent - use getSiteTiers() with returned direct value and not S_OK

*Transformation
BUGFIX: Uniform use of the TaskManager in the RequestTaskAgent and WorkflowTaskAgent

[v6r1p1]

*RSS
BUGFIX: Alarm_PolType now really send mails instead of crashing silently.

[v6r1]

*RSS
CHANGE: Major refactoring of the RSS system
CHANGE: DB.ResourceStatusDB has been refactored, making it a simple wrapper round ResourceStatusDB.sql with only four methods by table ( insert, update, get & delete )
CHANGE: DB.ResourceStatusDB.sql has been modified to support different statuses per granularity.
CHANGE: DB.ResourceManagementDB has been refactored, making it a simple wrapper round ResourceStatusDB.sql with only four methods by table ( insert, update, get & delete )
CHANGE: Service.ResourceStatusHandler has been refactored, removing all data processing, making it an intermediary between client and DB.
CHANGE: Service.ResourceManagementHandler has been refactored, removing all data processing, making it an intermediary between client and DB.
NEW: Utilities.ResourceStatusBooster makes use of the 'DB primitives' exposed on the client and does some useful data processing, exposing the new functions on the client.
NEW: Utilities.ResourceManagementBooster makes use of the 'DB primitives' exposed on the client and does some useful data processing, exposing the new functions on the client.
CHANGE: Client.ResourceStatusClient has been refactorerd. It connects automatically to DB or to the Service. Exposes DB and booster functions.
CHANGE: Client.ResourceManagementClient has been refactorerd. It connects automatically to DB or to the Service. Exposes DB and booster functions.
CHANGE: Agent.ClientsCacheFeederAgent renamed to CacheFeederAgent. The name was not accurate, as it also feeds Accouting Cache tables.
CHANGE: Agent.InspectorAgent, makes use of automatic API initialization.
CHANGE: Command. refactor and usage of automatic API initialization.
CHANGE: PolicySystem.PEP has reusable client connections, which increase significantly performance.
CHANGE: PolicySystem.PDP has reusable client connections, which increase significantly performance.
NEW: Utilities.Decorators are syntactic sugar for DB, Handler and Clients.
NEW: Utilities.MySQLMonkey is a mixture of laziness and refactoring, in order to generate the SQL statements automatically. Not anymore sqlStatemens hardcoded on the RSS.
NEW: Utilities.Validator are common checks done through RSS modules
CHANGE: Utilities.Synchronizer syncs users and DIRAC sites
CHANGE: cosmetic changes everywhere, added HeadURL and RCSID
CHANGE: Removed all the VOExtension logic on RSS
BUGFIX: ResourceStatusHandler - getStorageElementStatusWeb(), access mode by default is Read
FIX: RSS __init__.py will not crash anymore if no CS info provided
BUGFIX: CS.getSiteTier now behaves correctly when a site is passed as a string

*dirac-setup-site
BUGFIX: fixed typos in the Script class name

*Transformation
FIX: Missing logger in the TaskManager Client (was using agent's one)
NEW: Added UnitTest class for TaskManager Client

*DIRAC API
BUGFIX: Dirac.py. If /LocalSite/FileCatalog is not define the default Catalog was not properly set.
FIX: Dirac.py - fixed __printOutput to properly interpret the first argument: 0:stdout, 1:stderr
NEW: Dirac.py - added getConfigurationValue() method

*Framework
NEW: UsersAndGroups agent to synchronize users from VOMRS server.

*dirac-install
FIX: make Platform.py able to run with python2.3 to be used inside dirac-install
FIX: protection against the old or pro links pointing to non-existent directories
NEW: make use of the HTTP proxies if available
FIX: fixed the logic of creating links to /opt/dirac directories to take into account webRoot subdirs

*WorkloadManagement
FIX: SiteDirector - change getVO() function call to getVOForGroup()

*Core:
FIX: Pfn.py - check the sanity of the pfn and catch the erroneous case

*RequestManagement:
BUGFIX: RequestContainer.isSubrequestDone() - return 0 if Done check fails

*DataManagement
NEW: FileCatalog - possibility to configure multiple FileCatalog services of the same type

[v6r0p4]

*Framework
NEW: Pass the monitor down to the request RequestHandler
FIX: Define the service location for the monitor
FIX: Close some connections that DISET was leaving open

[v6r0p3]

*Framework
FIX: ProxyManager - Registry.groupHasProperties() wasn't returning a result 
CHANGE: Groups without AutoUploadProxy won't receive expiration notifications 
FIX: typo dirac-proxy-info -> dirac-proxy-init in the expiration mail contents
CHANGE: DISET - directly close the connection after a failed handshake

[v6r0p2]

*Framework
FIX: in services logs change ALWAYS log level for query messages to NOTICE

[v6r0p1]

*Core
BUGFIX: List.uniqueElements() preserves the other of the remaining elements

*Framework
CHANGE: By default set authorization rules to authenticated instead of all
FIX: Use all required arguments in read access data for UserProfileDB
FIX: NotificationClient - dropped LHCb-Production setup by default in the __getRPSClient()

[v6r0]

*Framework
NEW: DISET Framework modified client/server protocol, messaging mechanism to be used for optimizers
NEW: move functions in DIRAC.Core.Security.Misc to DIRAC.Core.Security.ProxyInfo
CHANGE: By default log level for agents and services is INFO
CHANGE: Disable the log headers by default before initializing
NEW: dirac-proxy-init modification according to issue #29: 
     -U flag will upload a long lived proxy to the ProxyManager
     If /Registry/DefaultGroup is defined, try to generate a proxy that has that group
     Replaced params.debugMessage by gLogger.verbose. Closes #65
     If AutoUploadProxy = true in the CS, the proxy will automatically be uploaded
CHANGE: Proxy upload by default is one month with dirac-proxy-upload
NEW: Added upload of pilot proxies automatically
NEW: Print info after creating a proxy
NEW: Added setting VOMS extensions automatically
NEW: dirac-proxy-info can also print the information of the uploaded proxies
NEW: dirac-proxy-init will check that the lifetime of the certificate is less than one month and advise to renew it
NEW: dirac-proxy-init will check that the certificate has at least one month of validity
FIX: Never use the host certificate if there is one for dirac-proxy-init
NEW: Proxy manager will send notifications when the uploaded proxies are about to expire (configurable via CS)
NEW: Now the proxyDB also has a knowledge of user names. Queries can use the user name as a query key
FIX: ProxyManager - calculate properly the dates for credentials about to expire
CHANGE: ProxyManager will autoexpire old proxies, also auto purge logs
CHANGE: Rename dirac-proxy-upload to dirac-admin-proxy-upload
NEW: dirac-proxy-init will complain if the user certificate has less than 30 days
CHANGE: SecurityLogging - security log level to verbose
NEW: OracleDB - added Array type 
NEW: MySQL - allow definition of the port number in the configuration
FIX: Utilities/Security - hash VOMS Attributes as string
FIX: Utilities/Security - Generate a chain hash to discover if two chains are equal
NEW: Use chain has to discover if it has already been dumped
FIX: SystemAdministrator - Do not set  a default lcg version
NEW: SystemAdministrator - added Project support for the sysadmin
CHANGE: SysAdmin CLI - will try to connect to the service when setting the host
NEW: SysAdmin CLI - colorization of errors in the cli
NEW: Logger - added showing the thread id in the logger if enabled
     
*Configuration
NEW: added getVOfromProxyGroup() utility
NEW: added getVoForGroup() utility, use it in the code as appropriate
NEW: added Registry and Operations Configuration helpers
NEW: dirac-configuration-shell - a configuration script for CS that behaves like an UNIX shellCHANGE: CSAPI - added more functionality required by updated configuration console
NEW: Added possibility to define LocalSE to any Site using the SiteLocalSEMapping 
     section on the Operations Section     
NEW: introduce Registry/VO section, associate groups to VOs, define SubmitPools per VO
FIX: CE2CSAgent - update the CEType only if there is a relevant info in the BDII  

*ReleaseManagement
NEW: release preparations and installation tools based on installation packages
NEW: dirac-compile-externals will try go get a DIRAC-free environment before compiling
NEW: dirac-disctribution - upload command can be defined via defaults file
NEW: dirac-disctribution - try to find if the version name is a branch or a tag in git and act accordingly
NEW: dirac-disctribution - added keyword substitution when creating a a distribution from git
FIX: Install tools won't write HostDN to the configuration if the Admin username is not set 
FIX: Properly set /DIRAC/Configuration/Servers when installing a CS Master
FIX: install_site.sh - missing option in wget for https download: --no-check-certificate
FIX: dirac-install-agent(service) - If the component being installed already has corresponding 
     CS section, it is not overwritten unless explicitly asked for
NEW: dirac-install functionality enhancement: start using the switches as defined in issue #26;
CHANGE: dirac-install - write the defaults if any under defaults-.cfg so dirac-configure can 
        pick it up
FIX: dirac-install - define DYLD_LIBRARY_PATH ( for Mac installations )     
NEW: dirac-install - put all the goodness under a function so scripts like lhcb-proxy-init can use it easily
FIX: dirac-install - Properly search for the LcgVer
NEW: dirac-install will write down the releases files in -d mode   
CHANGE: use new dirac_install from gothub/integration branch in install_site.sh
NEW: Extensions can request custom external dependencies to be installed via pip when 
     installing DIRAC.
NEW: LCG bundle version can be defined on a per release basis in the releases.cfg 
NEW: dirac-deploy-scripts - when setting the lib path in the deploy scripts. 
     Also search for subpaths of the libdir and include them
NEW: Install tools - plainly separate projects from installations

*Accounting
CHANGE: For the WMSHistory type, send as JobSplitType the JobType
CHANGE: Reduced the size of the max key length to workaround mysql max bytes for index problem
FIX: Modified buckets width of 1week to 1 week + 1 day to fix summer time end week (1 hour more )

*WorkloadManagement
CHANGE: SiteDirector - simplified executable generation
NEW: SiteDirector - few more checks of error conditions   
NEW: SiteDirector - limit the queue max length to the value of MaxQueueLengthOption 
     ( 3 days be default )
BUGFIX: SiteDirector - do not download pilot output if the flag getPilotOutput is not set     
NEW: JobDB will extract the VO when applying DIRAC/VOPolicy from the proper VO
FIX: SSHTorque - retrieve job status by chunks of 100 jobs to avoid too long
NEW: glexecComputingElement - allow glexecComputingElement to "Reschedule" jobs if the Test of
     the glexec fails, instead of defaulting to InProcess. Controlled by
     RescheduleOnError Option of the glexecComputingElement
NEW: SandboxStore - create a different SBPath with the group included     
FIX: JobDB - properly treat Site parameter in the job JDL while rescheduling jobs
NEW: JobSchedulingAgent - set the job Site attribute to the name of a group of sites corresponding 
     to a SE chosen by the data staging procedure 
CHANGE: TimeLeft - call batch system commands with the ( default ) timeout 120 sec
CHANGE: PBSTimeLeft - uses default CPU/WallClock if not present in the output  
FIX: PBSTimeLeft - proper handling of (p)cput parameter in the batch system output, recovery of the
     incomplete batch system output      
NEW: automatically add SubmitPools JDL option of the job owner's VO defines it     
NEW: JobManager - add MaxParametericJobs option to the service configuration
NEW: PilotDirector - each SubmitPool or Middleware can define TargetGrids
NEW: JobAgent - new StopOnApplicationFailure option to make the agent exiting the loop on application failure
NEW: PilotAgentsDB - on demand retrieval of the CREAM pilot output
NEW: Pilot - proper job ID evaluation for the OSG sites
FIX: ComputingElement - fixed proxy renewal logic for generic and private pilots
NEW: JDL - added %j placeholder in the JDL to be replaced by the JobID
BUGFIX: DownloadInputData - bug fixed in the naming of downloaded files
FIX: Matcher - set the group and DN when a request gets to the matcher if the request is not 
     coming from a pilot
FIX: Matcher = take into account JobSharing when checking the owner for the request
CHANGE: PilotDirector, dirac-pilot - interpret -V flag of the pilot as Installation name

*DataManagement
FIX: FileCatalog/DiractoryLevelTree - consistent application of the max directory level using global 
     MAX_LEVELS variable
FIX: FileCatalog - Directory metadata is deleted together with the directory deletion, issue #40    
CHANGE: FileCatalog - the logic of the files query by metadata revisited to increase efficiency 
FIX: LcgFileCatalog - use lfcthr and call lfcthr.init() to allow multithread
     try the import only once and just when LcgFileCatalogClient class is intantiated
NEW: LcgFileCatalogClient - new version of getPathPermissions relying on the lfc_access method to solve the problem
     of multiple user DNs in LFC.     
FIX: StorageElement - get service CS options with getCSOption() method ( closes #97 )
FIX: retrieve FileCatalogs as ordered list, to have a proper default.
CHANGE: FileCatalog - allow up to 15 levels of directories
BUGFIX: FileCatalog - bug fixes in the directory removal methods (closes #98)
BUGFIX: RemovalAgent - TypeError when getting JobID in RemovalAgent
BUGFIX: RemovalAgent - put a limit to be sure the execute method will end after a certain number of iterations
FIX: DownloadInputData - when files have been uploaded with lcg_util, the PFN filename
     might not match the LFN file name
FIX: putting FTSMonitor web page back
NEW: The default file catalog is now determined using /LocalSite/FileCatalog. The old behavior 
     is provided as a fallback solution
NEW: ReplicaManager - can now deal with multiple catalogs. Makes sure the surl used for removal is 
the same as the one used for registration.   
NEW: PoolXMLCatalog - added getTypeByPfn() function to get the type of the given PFN  
NEW: dirac-dms-ban(allow)-se - added possibility to use CheckAccess property of the SE

*StorageManagement
FIX: Stager - updateJobFromStager(): only return S_ERROR if the Status sent is not
recognized or if a state update fails. If the jobs has been removed or
has moved forward to another status, the Stager will get an S_OK and
should forget about the job.
NEW: new option in the StorageElement configuration "CheckAccess"
FIX: Requests older than 1 day, which haven't been staged are retried. Tasks older than "daysOld" 
     number of days are set to Failed. These tasks have already been retried "daysOld" times for staging.
FIX: CacheReplicas and StageRequests records are kept until the pin has expired. This way the 
     StageRequest agent will have proper accounting of the amount of staged data in cache.
NEW: FTSCleaningAgent will allow to fix transient errors in RequestDB. At the moment it's 
     only fixing Requests for which SourceTURL is equal to TargetSURL.
NEW: Stager - added new command dirac-stager-stage-files          
FIX: Update Stager code in v6 to the same point as v5r13p37
FIX: StorageManager - avoid race condition by ensuring that Links=0 in the query while removing replicas

*RequestManagement
FIX: RequestDBFile - get request in chronological order (closes issue #84)
BUGFIX: RequestDBFile - make getRequest return value for getRequest the same as for

*ResourceStatusSystem
NEW: Major code refacoring. First refactoring of RSS's PEP. Actions are now function 
     defined in modules residing in directory "Actions".
NEW: methods to store cached environment on a DB and ge them.
CHANGE: command caller looks on the extension for commands.
CHANGE: RSS use now the CS instead of getting info from Python modules.
BUGFIX: Cleaned RSS scripts, they are still prototypes
CHANGE: PEP actions now reside in separate modules outside PEP module.
NEW: RSS CS module add facilities to extract info from CS.
CHANGE: Updating various RSS tests to make them compatible with
changes in the system.
NEW: CS is used instead of ad-hoc configuration module in most places.
NEW: Adding various helper functions in RSS Utils module. These are
functions used by RSS developers, including mainly myself, and are
totally independant from the rest of DIRAC.
CHANGE: Mostly trivial changes, typos, etc in various files in RSS     
CHANGE: TokenAgent sends e-mails with current status   

*Transformation
CHANGE: allow Target SE specification for jobs, Site parameter is not set in this case
CHANGE: TransformationAgent  - add new file statuses in production monitoring display
CHANGE: TransformationAgent - limit the number of files to be treated in TransformationAgent 
        for replication and removal (default 5000)
BUGFIX: TransformationDB - not removing task when site is not set
BUGFIX: TransformationCleaningAgent - archiving instead of cleaning Removal and Replication 
        transformations 
FIX: TransformationCleaningAgent - kill jobs before deleting them        

*Workflow
NEW: allow modules to define Input and Output parameters that can be
     used instead of the step_commons/workflow_commons (Workflow.py, Step.py, Module.py)

*Various fixes
BUGFIX: Mail.py uses SMTP class rather than inheriting it
FIX: Platform utility will properly discover libc version even for the new Ubuntu
FIX: Removed old sandbox and other obsoleted components