ChangeLog.txt

Release Notes

* 5/25/2022
  * 0.0.6 for containerization

* 2/18/2020
  * using use_only_local_data in AtlasPJB

* 9/25/2019
  * not to delete empty output datasets for prod
  * to use done instead of failed when input has no files

* 7/9/2019
  * added CPU abuse check

* 5/30/2019
  * using json SW check in AtlasAJB

* 5/2/2019
  * added NQUEUED_NUC_CAP

* 4/2/2019
  * max attempts/failures for analysis

* 3/25/2019
  * task.architecture=cmtconfig@image

* 3/22/2019
  * setting task.currentprio=1500 if s or g in gridpref

* 3/18/2019
  * added --priority in panda-client
  * removed high prio from ready tasks in getTasksTBP

* 3/15/2019
  * to automatically fix empty memory requirements

* 3/11/2019
  * using fake in specialHandling for WAITING_CO_JUMBO

* 3/5/2019
  * added osMatching

* 2/28/2019
  * added useZipToPin

* 2/27/2019
  * checking nNewJobs when setting exhausted after scouts

* 2/19/2019
  * improved AtlasAJB for GPU PQs to request architecture matching

* 2/13/2019
  * added toStaging

* 2/5/2019
  * using nClosed in AtlasAJB

* 1/28/2019
  * added eshigh
  * prio boost for tasks with jumbo

* 1/17/2019
  * using nEventsReady for jumbo in AtlasPJB
  * not to count throttled jobs in nAct in AtlasAJB

* 1/15/2019
  * processingType=blah-cont for container based analysys
  * added USER_TASKS_MOVE_INPUT

* 12/21/2018
  * to use only MCORE for ES scouts

* 12/19/2018
  * to use only MCORE for ES with prio>=900

* 12/11/2018
  * disabled data locality check in AtlasAJB for taskPrio>=2000

* 12/3/2018
  * disabled IO intensity check when no candidate available permanently

* 11/29/2018
  * added correction to nPilot for resource shares
  * added release and nFiles checks to JumboWatchDog

* 11/26/2018
  * using 500kB for analysis outDiskCount
  * using 50GB for directIO maxSize

* 11/20/2018
  * using TAPE for IO intensity check in AtlasPJB

* 11/19/2018
  * added the second look in AtlasAJB to ignore data locality
  * set transient for finished+merged+100%

* 11/5/2018
  * using new task params for ES

* 10/24/2018
  * changed rerefine not to reset some attributes

* 10/22/2018
  * added proc_status

* 10/16/2018
  * added decAttOnFailedES

* 10/5/2018
  * added discardEvents in retryTask

* 9/24/2018
  * using resolve_archives=True in list_replicas

* 9/20/2018
  * added maxJumboPerSite

* 9/7/2018
  * added enableJumbo

* 7/23/2018
  * changed getScout to ignore nEvents<10*nCore
  * added HS06sPerEventFixed

* 7/19/2018
  * changed AtlasAJB to check cmtConfig for nightlies

* 7/17/2018
  * added orderByLB

* 7/16/2018
  * fixed maxWalltime and maxCpuCount for multiple consumers

* 5/23/2018
  * changed AtlasAJB for hospital

* 5/7/2018
  * excluding validation from *_AES for es conversion

* 4/25/2018
  * make IO intensity cutoff configurable

* 4/19/2018
  * using notDiscardEvents for ES tasks by default

* 4/16/2018
  * added minCpuEfficiency

* 3/20/2018
  * notDiscardEvents

* 3/14/2018
  * to reset NG

* 3/1/2018
  * added a protection against state change in DB

* 2/28/2018
  * changed exhausted messages to INFO
  * fixed finished tasks with mutable datasets

* 2/14/2018
  * fixed retryTask to go through when input is mutable

* 2/8/2018
  * changed not to retry failed child tasks

* 1/26/2018
  * added useJumbo=pending

* 1/19/2018
  * setting superStatus and T_TASK.status consistently

* 1/17/2018
  * added nStandby

* 12/15/2017
  * changed to exclude workDirSize from NG

* 12/14/2017
  * changing gshare of prod tasks when 95% done

* 12/7/2017
  * checking maxattempt when retryTask_JEDI

* 11/29/2017
  * setting allowAccessLAN to analysis tasks
  * changed InputChunk not to ignore input size for directIO sites if NG is used

* 11/27/2017
  * using resurrectConsumers for ES tasks by default
  * changed InputChunk to ignore input size for directIO sites

* 11/17/2017
  * using NG as tot size of input when pooing input files in child tasks

* 11/15/2017
  * added noInputPooling

* 11/14/2017
  * fixed double counting of input files in AtlasPJB

* 11/13/2017
  * setting prodDBlockToken=local to lib.tgz
  * using 60d of lifetime for merged parent datasets

* 11/10/2017
  * Changed call to getHS06sec to new mode

* 11/6/2017
  * sending analysis tasks with short scouts to exhausted

* 10/25/2017
  * fixed AtlasTS to resurrect ES datasets when tasks are retried
  * getSURLsFromLFC to also check for gsiftp copies
  * change regex to accept se's not ending with /

* 10/16/2017
  * unified PQ for analysis
  * disable timeout for exhausted when auto finish is disabled
  * setting resource_type for buildJob
  * fixed AtlasAJB to unified PQ

* 10/10/2017
  * adding the offset to nGBPerJob

* 10/09/2017
  * added usePrefetcher

* 9/27/2017
  * added resurrectConsumers

* 9/26/2017
  * removed splitrule check in reactivatePending

* 9/15/2017
  * added task.diskIO

* 9/14/2017
  * fixed flip-flop of AES flag in refiner

* 9/11/2017
  * disabled IO check for event service in AtlasPJB

* 9/6/2017
  * added disableAutoFinish
  * deleting event service datasets
  * fixed respectSplutRule to set nFilesTobeUsed properly

* 8/28/2017
  * fixed AtlasAJB/PJB for 3 digtis caches

* 8/23/2017
  * using (type=SCRATCHDISK)\notforextracopy=1 for secondary copy of user output

* 8/18/2017
  * added registerEsFiles

* 8/14/2017
  * added skipShortInput

* 8/11/2017
  * added AES_MAXPENDIG
  * added SCOUT_CPUTIME_RANK, SCOUT_RAMCOUNT_RANK, SCOUT_RAMCOUNT_MARGIN

* 8/10/2017
  * fixed retryTask not to change running files

* 8/9/2017
  * fixed AF when tasks are re-refined

* 7/11/2017
  * disabled second copy when --destSE

* 7/10/2017
  * fixed cloneJobs to set correct ramCount for each PQ

* 7/5/2017
  * getting nTasks and nFiles from config

* 7/3/2017
  * considering maxNumFiles in getAtom
  * added AES_NESCONSUMERS and AES_NESCONSUMERS
  * changed AtlasPJB to skip opportunistic sites for high prio jobs
  * enabled automatic AES task conversion

* 6/27/2017
  * added getNumUnprocessedEvents_JEDI

* 6/26/2017
  * skip scouts for ES
  * ES without ueRealNumEvt or nEventsPerFile
  * WQ-GS alignment

* 5/17/2017
  * changed nRunningCap to ignore highPrioWaiting in AtlasPJT

* 5/12/2017
  * setting nEsConsumers=1 etc for esFraction

* 5/8/2017
  * added nSimEvents

* 5/4/2017
  * fixed AtlasAJB for AtnalysisBase
  * fixed getAvailableFiles

* 5/3/2017
  * added JediDBProxyFiltered
  * added inFilePosEvtNum

* 5/2/2017
  * changed AtlasPJB for unified PQs

* 4/28/2017
  * fixed brokerage for git rel

* 4/25/2017
  * fake co-jumbo

* 4/24/2017
  * resource_type tagging

* 4/14/2017
  * added noExecStrCnv

* 4/11/2017
  * added throttleJobsInPausedTasks

* 4/5/2017
  * added pseudo output dataset

* 3/27/2017
  * changed AtlasAPP to extend lifetime of user output datasets

* 3/2/2017
  * added esFraction

* 2/22/2017
  * fixed generateBuildJob for nightlies
  * changed AtlasTS to make duble copy for users

* 2/20/2017
  * changed AtlasPTB for task.nucleus=satellite
  * added respectSplitRule
  * added indexConsistent

* 2/7/2017
  * added DISK_THRESHOLD_xyz in AtlasPTB

* 1/31/2017
  * removed dq2 dependency
  * changed not to change oldstatus when throttled->paused
  * fixed eventRatio for nSecond % nMaster != 0

* 1/13/2017
  * introduced GROUPBYATTR in getTasksToBeProcessed

* 1/12/2017
  * fixed for incexec with input containers

* 12/20/2016
  * changed setScoutJobData to set exhausted when task_ramCount<4GB and scout_ramCount>4GB

* 12/15/2016
  * added nQueueCap
  * added mergeEsAtOS

* 12/9/2016
  * added nRunningCap to AtlasPJT
  * disabled setDEL in AtlasPP

* 12/5/2016
  * increased scout priority to 901

* 12/1/2016
  * changed AtlasPWD to boost task priorities at 95%
  * fixed AtlasAJB and JG to support cmake-based nightlies

* 11/28/2016
  * fixed setScout to propagate exhausted to DEFT

* 11/24/2016
  * changed AtlasPJB not to use baseramcount for pmerge
  * disabled transfer backlog check when negative T1 weight in AtlasPTB
  * changed thresholds for data availability check configurable in AtlasPTB

* 11/22/2016
  * fixed AtlasAJB to use configurator tables instead of schedconfig.space

* 11/16/2016
  * changed AtlasAW to use PRESTAGE_LIMIT
  * added a hint to setDelFlag
  * using 1 for negative ramCount
  * fixed AtlasPJB for no-input tasks

* 11/14/2016
  * changed outDiskCount not to scale with workDiskCount

* 11/9/2016
  * changed AtlasPTB not to ignore job execution ability
  * introduced cutoff based on ioIntensity and mbToMove in AtlasPJB

* 11/7/2016
  * changed JG not to use compact input params for user's trf jobs

* 10/28/2016
  * changed AtlasDDM to work with DN mapped to multiple rucio accounts

* 10/17/2016
  * removing multi-byte characters from task params
  * added eventRatio

* 10/6/2016
  * added addEsParams
  * support for reassignTaskToNucleus=None

* 9/30/2016
  * changed tobeprocess for ratio in float
  * changed ratio of output to check boundaryID

* 9/29/2016
  * added support ratio for output datasets

* 9/21/2016
  * changed AtlasPTB to use deep in list_dataset_replicas for primary dataset replica lookup

* 9/16/2016
  * setting DEL to event ranges

* 9/13/2016
  * using maxPSS for analy tasks

* 9/2/2016
  * using cloneJobs for ES even with nConsumer=1

* 8/29/2016
  * changed analysis to work with containers of containers

* 8/1/2016
  * using campaign in WQ 

* 7/29/2016
  * fixed AtlasPJB to use score queues for pmerge even if site is specified

* 7/25/2016
  * fixed IC for cpuEfficiency=0
  * tagging of shares

* 7/19/2016
  * changed TRB to check boolean value of noWaitParent
  * changed IC for NG+NF+LB

* 7/14/2016
  * setting old modtime in JG for high prio + running + one or more reset
  * using bulk increment in getOutput
  * using bulk fetch and insert for files in JG

* 7/12/2016
  * changed AtlasTS and AtlasPPP to register and delete ES datasets

* 7/11/2016
  * added jumbo job stuff
  * added requestType
  * added getOldMergeJobPandaIDs

* 7/7/2016
  * added removeTags to AtlasAPP

* 7/5/2016
  * added rescueUnLockedTasksWithPicked

* 7/4/2016
  * changed ToBeProcessed to check done datasets as well

* 6/30/2016
  * added ignoreMissingInDS
  * using bulk fetching for JEDI_DATASET_CONT_FILEID_SEQ

* 6/28/2016
  * resetting transferType for pmerge
  * using lfn sort for pmerge even if reuseSecOnDemand is used
  * fixed AtlasPWD for deleted datasets
  * fixed TC for paused

* 6/24/2016
  * enabled history-aware brokerage for analysis
  * added splitByFields

* 6/23/2016
  * using longer lifetime for unmerged datasets when AOD merge is not very successful
  * added useFileAsSourceLFN
  * added forceAvalanche
  * put logrotate file to etc/panda to avoid automatic installation on test nodes

* 6/20/2016
  * using _tid\d+ as a deliminator for NthField
  * setting ramCount=2000 for anal tasks by default
  * using eventService=2 for tasks with JC
  * enabled space check in AtlasPJB

* 6/13/2016
  * changed not to reset errorDiag until next JG attempt
  * fixed 24h timeout in AtlasPJB to send pmerge to remote sites

* 6/11/2016
  * changed TR not to update the same task simultaneously

* 6/8/2016
  * increased space limit to 100TB in AtlasPTB

* 6/6/2016
  * using rucio in registerDatasetSubscription
  * fixed AtlasPWD to use Production Output for subscriptions

* 6/2/2016
  * removed scope from tmpin

* 5/25/2016
  * changed AtlasPTB to send more log messages
  * added writeInputToFile
  * set nqueuelimit to 10k for opportunistic

* 5/24/2016
  * changes for failedAttempt and hs06sec in job

* 5/18/2016
  * removed maxinputsize from JG

* 5/16/2016
  * fixed ZeroDivision error in AtlasPTB
  * changed AtlasPTB to skip data locality check when no nucleus has data

* 5/12/2016
  * changed t1weight based on IO intensity

* 5/11/2016
  * added getFailureCountsForTask_JEDI
  * changed lifetime of um.log to 365 days

* 5/6/2016
  * changed JG not to lock when nQueued<nQueueLimit
  * set maxNumJobs=nQueueLimit/10 when nQueued<nQueueLimit

* 5/4/2016
  * fixed lockedBy IS NULL in getTaskDatasetsWithID_JEDI 
  * changed lifetime of transient datasets to 14 days

* 5/3/2016
  * changed job brokerages to use mergeRamCount for pmerge

* 5/2/2016
  * updated to second version of network brokerage
  * changed AtlasPJB to ignore t1w for pmerge
  * avoid assigning tasks to nuclei with large transfer backlogs
  * fixed taskRW in AtlasPTB

* 4/28/2016
  * changed retryTask to update t_task.start_time

* 4/27/2016
  * increased nQueueLimit for group to 40k
  * changed JG not to lock counter for pmerge or nQueued<nQueueLimit
  * fixed JG for uniform tasks with ES and non-ES

* 4/26/2016
  * resetting walltimeunit when re-refining tasks
  * updates/protections for new network metrics sources

* 4/25/2016
  * changed AtlasPTB to respect dst: when input is unavailable

* 4/20/2016
  * fixed unlockTasks to release picked files
  * introduced 5% of tolerance in minRamCount
  * introduced MBPerCoreFixed
  * changed JG to set maxWalltime=24h to scouts with cpuTime=0 or None

* 4/19/2016
  * fixed getScoutJobData to use jobs with nEvents<10 but wallTime>6h

* 4/18/2016
  * using 95th percentile for cpuTime
  * fixed dst:XYZ to use nucleus instead of PQ

* 4/14/2016
  * using smaller minTotalWalltime for WQ with site in AtlasPJT

* 4/13/2016
  * changed resume/pauseTask to  activate/throttle jobs
  * fixed JG for pending tasks to release lock
  * fixed dump in ThreadUtils
  * imporved AtlasAJB to work with includedSite=''
  * improved unlockTasks to release tasks locked by dead processes
  * fixed oldPandaIDs in JG for job cloning and multiple consumers

* 4/12/2016
  * added splitRule to mapper attributes

* 4/10/2016
  * fixed sortParallelJobsBySite for oldPandaIDs

* 4/7/2016
  * added eventService to WQ

* 4/5/2016
  * fixed AtlasAJB to works with strings for excluded/includedSites
  * changed AtlasPPP to pause tasks when duplication is found

* 4/4/2016
  * added fractionFreeSpace to AtlasPTB
  * changed AtlasPTB to take fullRW into account for space estimation

* 3/31/2016
  * fixed AtlasAJB for ath releases
  * disabled to send scouts to non-short queues if site is set
  * fixed AtlasAJB for remote access not to ignore resource matching

* 3/30/2016
  * fixed prepareTasksToBeFinished_JEDI for scouting+noWaitParent
  * using assessmentTime for early avalanche

* 3/24/2016
  * fixed scoutJobData for ES
  * fixed retryTask for active exhausted tasks

* 3/23/2016
  * fixed JG to record retry history for multiple consumers
  * fixed JG for ddd:

* 3/21/2016
  * changed JS to take job multiplicity into account
  * fixed JRB to set implicit parameters for job cloning

* 3/17/2016
  * removed quote from error message for upstream
  * changed fatal errors in JG to temporary
  * added skipUnavailableFiles to InputChunk

* 3/16/2016
  * added rerefine to setOnHold
  * added putLogToOS
  * fixed AtlasTR to correctly set/unset transient to datasets

* 3/15/2016
  * set cloudToTask for tasks with no outputs
  * fixed updateByCF for tasks with no outputs
  * added isDistributedDataset
  * changed AtlasPJB to take into account the number of files to move
  * using dict instead of list in AGIS json

* 3/11/2016
  * changed AtlasPJB to take EC into account when walltime estimation
  * changed getTasksToRefine to delete leftover

* 3/10/2016
  * changed AtlasDDM to support registration of multiple locations

* 3/8/2016
  * fixed InputChunk for distributed datasets

* 3/7/2016
  * changed AtlasDDM and AtlasAJB to support distributed datasets
  * fixed AtlasPJB to correctly get sites associated to a nucleus

* 3/4/2016
  * changed CF to lookup for assigning tasks

* 3/3/2016
  * fixed maxEvents for zero event files

* 2/25/2016
  * added allowInputLAN

* 2/23/2016
  * changed AtlasPJB to support allowInputWAN
  * removed some AGIS dependency from JG

* 2/22/2016
  * changed to update superstatus when scouting->exhausted

* 2/18/2016
  * changed hasZeroShare to work with processingType=validation in non-validation WQ
  * added doForPreStaging to AtlasAWD

* 2/16/2016
  * changed getScoutJobData to use jobs with nEvents<10 and wall>6h too

* 2/15/2016
  * added workQueueName
  * aded mcore_repro
  * added altStageOut

* 2/10/2016
  * fixed retryTask to correctly set status of secondary datasets
  * fixed to register final datasets when internal merge is used
  * removed T1 constraint from AtlasPJB for high prio or scouts  

* 2/9/2016
  * added nokill to reassignTask

* 2/8/2016
  * added network weights to brokerage

* 2/4/2016
  * fixed ToBeProcessed to correctly read secondary files when multiple RAM requirements
  * changed finishTask to ignore failGoalUnreached
  * setting nucleus and eventService in jobSpec

* 2/3/2016
  * fixed MIDDLENAME for multi fileds of addNthFieldOfInDSToLFN
  * changed AtlasPTB to use any nuclei where input is available when no nuclei can run jobs

* 2/1/2016
  * added reassignTaskToNucleus
  * fixed insertFilesForDataset_JEDI for ready->defined in scout

* 1/30/2016
  * changed log messages of AtlasPTB to be compatible with bamboo

* 1/27/2016
  * changed AtlasPTB to get messages from AtlasPJB

* 1/25/2016
  * changed addNthFieldToLFN to allow multiple fields

* 1/19/2016
  * changed AtlasPP to set lifetime to failed log datasets 
  * fixed getAnalSitesWithData to ignore STAGING

* 1/18/2016
  * changed getScoutJobData to reduce baseWalltime when calculating cpuTime
  * changed TR to respect the boolean vale for failWhenGoalUnreached

* 1/15/2016
  * changed message type for WORLD taskbrokerage
  * setting maxWalltime to maxCpuCount for HS06
  * changed AtlasPJB to support 3 digits base releases

* 1/14/2016
  * changed setScoutJobData so that tasks go to exhausted when execution time is unexpectedly too long
  * removed TiersOfATLAS from AtlasDDM

* 1/5/2016
  * changed restartTasksForCF to restart ready tasks with defined DSs

* 12/31/2015
  * fixed CF to use nFilesForScout for toupdate in scouting

* 12/16/2015
  * changed restartTasksForCF to restart defined tasks with only ready DSs

* 12/15/2015
  * using roundi up for the brokerage as well
  * added protection against nFilesPerJob=0 and nEventsPerJob=0

* 12/9/2015
  * rounding up job.minramcount with 500MB chunks

* 12/8/2015
  * fixed PP to go finished instead of exhausted if goal is reached
  * changed AtlasPJB to send pmerge to anywhere after 24h

* 12/7/2015
  * removed validationrelease from AtlasPJB
  * enabled duplication check for production

* 11/27/2015
  * fixed for reassignTask not to make subscriptions for distributed datasets 
  * removed offet for zero events in getEffectiveNumEvents

* 11/23/2015
  * added updateJobMetrics
  * fixed nEventsPerMergeJob

* 11/19/2015
  * changed AtlasPP to delete empty datasets
  * fixed AtlasPTB for WORLD + no output
  * changed getScout to ignore jobs with nEvents<10

* 11/18/2015
  * fixed AtlasPTB for WORLD

* 11/16/2015
  * changed getScout to fill IO info

* 11/13/2015
  * fixed AtlasPTB for negative t1Weight in WORLD
  * changed AtlasPTB for all prod jobs to respect fairsharepolicy 

* 11/11/2015
  * fixed AtlasPTB for missing nuclei in RW
  * added protection to RW calculation against cpuTime=NULL
  * changed AtlasAJB to take wansinklimit into account for WAN weight

* 11/10/2015
  * changed prod/anal prokerages to use max/minrss

* 11/9/2015
  * changed JG to avoid passing parallelOutMap to TB

* 11/6/2015
  * removed T1 constraint for ES job brokerage
  * changed AtlasPTB to use confgurator tables
  * using jobseed=std by default
  * changed AtlasPTB to take size of expired secondaries into account

* 11/5/2015
  * changed jobseed=all when switchEStoNormal is not used

* 11/4/2015
  * changed JG to remove job parameters for empty streams

* 11/2/2015
  * giving a smaller weight to tape for analy

* 10/27/2015
  * tweaked freezeDataset to avoid a rucio bug
  * fixed setCloudToTask to respect distributed datasets

* 10/26/2015
  * fixed for internal merge + zero suppression
  * added checkDuplication
  * avoiding duplication for non-zero RAM chunk

* 10/22/2015
  * changed insertFiles to separate FID retrieval
  * changed AtlasPJB to cap based on nActivated+nStarting when data is available

* 10/21/2015
  * changed getScoutJobData to ignore jobs with nEvents<10
  * using bulk insert in insertFiles
  * fixed slow insertFiles due to givenFiles

* 10/20/2015
  * fixed getMaxAtom for pmerge to use getNum*PerMergeJob
  * added allowNoOutput

* 10/19/2015
  * changed AtlasTPP to set lifetime to transient datasets

* 10/16/2015
  * fixed resetUnused to take ramcount into account

* 10/14/2015
  * added dynamicNumEvents stuff
  * added maxNumEventRanges

* 10/12/2015
  * added dynamic nQueueLimit
  * added timeoutForPendingVoLabel

* 10/8/2015
  * ignoring TEST when converting token to RSE

* 10/7/2015
  * added a margin to the multiplier of disk count estimation for pmerge
  * fixed CF for nEventsPerRange

* 10/6/2015
  * changed JG to release the lock as soon as tasks are taken when nQueued<nQueueLimit
  * fixed lackOfJobs

* 10/2/2015
  * fixed JG to reset destinationDBlock when the site doesn't have endpoint for ddd:	
  * more fixed CF for seq_num+skipFilesUsedBy

* 10/1/2015
  * fixed CF for seq_num+skipFilesUsedBy
  * changed JG to lock just before final task update

* 9/30/2015
  * added ddd:
  * changed AtlasPJB to work with cpuTime=0

* 9/29/2015
  * fixed setScoutJobDataToTasks_JEDI to take young tasks into account
  * suppressed files in JediDataset.str 

* 9/28/2015
  * added retrial module

* 9/27/2015
  * fixed AtlasProdWatchDog for reassigning

* 9/25/2015
  * added protection to TC for empty commentStr

* 9/24/2015
  * changed APJB to take priority-based shares into account 

* 9/22/2015
  * removed amiFlag

* 9/21/2015
  * fixed failWhenGoalUnreached for goal=100.0
  * fixed rescuePicked to lock tasks

* 9/11/2015
  * added setScoutJobDataToTasks

* 9/9/2015
  * added WORLD

* 9/1/2015
  * added skipFilesUsedBy

* 8/27/2015
  * changed AtlasPJB to skip share check for test/validation

* 8/25/2015
  * setting activity for registerDatasetLocation
  * changed task.outDiskCount to include size of intermediate files
  * changed task.outDiskCount to scale with events when outDiskUnit has PerEvent
  * changed AtlasPJB so that merge jobs avoid SHORT

* 8/24/2015
  * modified retryChildTask not to reactivate children if they were aborted/broken

* 8/21/2015
  * changed prepareTasksToBeFinished so that scouting can be kicked
  * increasing taskprio by 1 for ready/scout so that scouts don't wait behind the bulk

* 8/20/2015
  * fixed getTasksToBeProcessed to respect priorities of prod tasks
  * changed the limit to avoid inactive sites to prio=800

* 8/18/2015
  * increased the limit on how many times dataset is duplicated in a single cycle

* 8/11/2015
  * removed RRC-KI-T1 from slow site list

* 8/6/2015
  * added a mechanism so that a single task can generate both ES and normal jobs

* 8/5/2015
  * changed PJB for zero-walltime to avoid short queues

* 8/4/2015
  * changed locCheck to cloudLocCheck

* 8/3/2015
  * changed PP so that child tasks immediately go to finished w/o exhausted when all inputs are successful
  * added cloud locality check
  * added failWhenGoalUnreached

* 7/30/2015
  * fixed AtlasPJB to check inactive sites with nActivated>0

* 7/30/2015
  * changed AtlasPJB to avoid inactive sites for job.prio>=900

* 7/16/2015
  * changed AtlasPJB to avoid slow sites for job.prio>=900

* 7/14/2015
  * sending scouts to sites where walltime is more than 24h

* 7/13/2015
  * fixed getSubChunk to correctly take secondary files into account

* 7/10/2015
  * changed cpuTime for scouts to always update the value
  * changed walltime messages more informative

* 7/9/2015
  * added job cloning
  * added message to errorDialog when goal is reached

* 7/8/2015
  * fixed retryTask to reset picked files
  * setting amiFlag
  * changed goal to have the first decimal place
  * changed child tasks not to go done when parents are not done
  * fixed reactivatePendingTasks to timeout child tasks correctly
  * increased upper limit of outdiskcount to 10MB

* 7/7/2015
  * added goal
  * added transient flag to datasets

* 7/6/2015
  * added size check in taskParam copy from DEFT

* 6/26/2015
  * added 50% of safety mergin to cpuTime
  * added more message for input file shortage
  * fixed maxWalltime for CPU efficiency
  * added stayOutputOnSite

* 6/24/2015
  * decomposed INSERT+SELECT to copy task params from DEFT
  * changed tgtNumEventsPerJob to ignore file boundaries

* 6/23/2015
  * added reactivateTasks for high prio tasks
  * reduced delay for reassignTask
  * added traceback to dumpErrorMessage

* 6/22/2015
  * improved HS06sPerEvents

* 6/19/2015
  * fixed resolveGOC in AtlasDDMClient

* 6/18/2015
  * changed to allow to set workDiskUnit
  * added a special role to outDiskCount=-1
  * fixed getTasksToExecCommand_JEDI for concurrent updates

* 6/15/2015
  * extened workQueue to allow site as a param
  * using always closed state for containers

* 6/11/2015
  * added tgtNumEventsPerJob
  * improved performance of file lookup

* 6/10/2015
  * adding offset in walltime estimation with HS06
  * kick ready tasks when some datasets have toupdate
  * set ready to toupdate datasets when nUnprocessed>0

* 6/8/2015
  * using force kill for finish/reassignTasks
  * changed PP to kick child tasks when parents are done
  * improved AtlasPJB to take ES into account
  * improved AtlasPJB to take network connectivity into account
  * changed to propagate exhausted to superStatus
  * added runUntilClosed
  * added support for walltimeUnit=HS06
  * changed JG to keep attemptNr when using a list of LFNs

6/3/2015
  * changed AtlasJobBrokers to take throttled jobs into account

* 5/27/2015
  * changed PP to set failed when nFiles=0 is finished
  * added ipConnectivity
  * set amount in getTaskParamsWithID_JEDI

* 5/20/2015
  * changed to allow a list of sites in task.site
  * added useCompleteOny to getAvailableFiles

* 5/18/2015
  * unset maxSize in InputChunk when nFilesPerJob or nEventsPerJob is set

* 5/13/2015
  * changed getAtom to take nEventsPerJob into account

* 5/12/2015
  * changed getJobsToBeProcessed for merge jobs to ignore maxJobs

* 5/11/2015
  * added useRealNumEvents
  * added exhausted

* 5/6/2015
  * changed JG to refresh job statistics before the innermost loop
  * fixed getTasksTobeProcessed to take maxJobs into account more precisely

* 5/5/2015
  * introduced maxFailure
  * using list_dataset_replicas of rucio API

* 4/27/2015
  * added restartTasksForContentsUpdate

* 4/26/2015
  * diabled to set ramCount using scouts
  * added debug messages to getTasksToBeFinished and prepareTasksToBeFinished

* 4/2/2015
  * introduced negative walltime to disable lopping job detection

* 3/29/2015
  * changed AtlasTS to set metadata to prod datasets
  * modified to prepareTasks
  * fixed insertFiles for nLost

* 3/27/2015
  * changed to propagate number of input events to DEFT

* 3/26/2015
  * changed insertFiles to fill nEvents for input
  * changed AtlasPJB to add criteria=+-tag

* 3/24/2015
  * changed JG to serizlize threads
  * changed latestDBR to use DBR with one or more replicas

* 3/23/2015
  * set the upper limit on wallTime from scout job data

* 3/20/2015
  * changed AtlasAJB for release check
  * changed AtlasTS to always register user datasets

* 3/19/2015
  * changed AtlasTR to set transient metadata

* 3/18/2015
  * changed AtlasPP to set transient metadata

* 3/17/2015
  * fixed getSiteStorageEndpointMap for NL
  * fixed JG to use currentPriority instead of currentPriority

* 3/12/2015
  * changed not to update contents when task is pending

* 3/11/2015
  * added cap in AtlasPJB
  * added invalid character check in dataset name

* 3/10/2015
  * added lockProcess
  * changed queue lengh to 450*4
  * changed chunk size per site to 25

* 3/6/2015
  * introduced manyAssigned in AtlasPJB

* 3/3/2015
  * changed AtlasPJB to send logging info to the logger
  * changed AtlasPJB to assign highPrio/scouts to home+foreign T1

* 2/26/2015
  * changed AtlasPJB to check free disksize with rucio
  * fixed truncation error when throttling tasks
  * fixed prepareTasks for early avalanche

* 2/23/2015
  * added noThrottle
  * added partialFinish
  * using AVG for throttle

* 2/18/2015
  * added disableReassign
  * fixed AtlasPJT when high prio tasks are waiting

* 2/13/2015
  * changed AtlasPJB to send scouts to sites associated to T1 SE
  * changed reacivatePendingTasks to finish tasks when they have some finished files 
  * changed CF to allow 200k files

* 2/11/2015
  * changed AtlasPJB to include nDefined in nQueued
  * added bulkSendMsg to MsgWrapper
  * changed AtlasAJB to send brokerage logs
  * removed setReplicaMetaDataAttribute from AtlasTS
  * added soft finish
  * fixed insertFiles to take largeness of nEventsPerFile into account

* 2/10/2015
  * modified MW to escape <>
  * removed data locality check from AtlasPJB
  * fixed JG to set correct coreCount for pmerge
  * changed AtlasPJB to count jobs in all clouds
  * fixed getTasksToBeProcessed for zero SEQ_NUMBER

* 2/6/2015
  * added submitting messages to JG

* 2/5/2015
  * fixed typo in AtlasPPP 

* 2/4/2015
  * using long to get LB info
  * fixed InputChunk for nFilesPerMergeJob

* 2/3/2015
  * removed slow lookup in insertFiles
  * adding originPandaID to jobName

* 2/2/2015
  * added mergeCoreCount
  * changed AtlasDDM to use Rucio API for dataset registration

* 2/1/2015
  * fixed reuse of libTgz

* 1/30/2015
  * fixed reactivatePending to limit length of errorDialog

* 1/29/2015
  * fixed RW calculation to take effective size into account

* 1/27/2015
  * removed ND-specific handling for release check in AtlasJobBrokerages
  * fixed AtlasJobBrokerages to take nStarting into account

* 1/26/2015
  * changed InputChunk to recalculate weight once the site is chosen

* 1/24/2015
  * fixed CF so that retried child is not done before the parent re-opens output

* 1/22/2015
  * improved SQL in duplicateFilesForReuse 

* 1/20/2015
  * using max when calculating memory requirements based on scouts
  * fixed getTobeProcessed for secondary with nFilesTobeUsed=nFilesUsed

* 1/15/2015
  * added reuseSecOnDemand
  * changed TR to treat JEDITemporaryError as a temporary error

* 1/14/2015
  * changed getTobeProcessed to use accurate estimation when reading secondary files
  * changed insertFiles to release more files for event-split while the parent is running
  * fixed insertFiles to take event-level splitting into account when setting open to mutables

* 1/13/2015
  * fixed getFileList for did
  * fixed insertFiles to use master status for status of mutable secondary
  * removed origin setting from reassignTasks
  * fixed reassignTasks to work during the task is being assigned

* 1/12/2015
  * fixed slow lookup table in insertFiles 

* 1/11/2015
  * introduced cutoff for walltime calculation using scout jobs

* 1/9/2015
  * fixed TaskBroker not to read all files
  * introduced nFilesPerGroup
  * increased the max file limit in datasets to 200k
  * changed CF to automatically set num_records for seq_number

* 1/5/2015
  * fixed sufficient file check in CF for multiple input streams

* 12/29/2014
  * fixed getFiles for UnknownDataset

* 12/24/2014
  * fixed CF to take nEventsPerJob and nFilesPerFile for seq_number

* 12/23/2014
  * fixed CF for noInput+seqNum

* 12/19/2014
  * changed AtlasBrokerage to task BL into account

* 12/18/2014
  * added num_records to dataset attributes
  * removed lifetime setting for user datasets
  * added machinery to throttle tasks

* 12/17/2014
  * added unlockSingleTask_JEDI
  * fixed getAvailableFiles for checkCompleteness

* 12/16/2014
  * changed for originpandaid
  * added more debug messages for reassignTask
  * setting job.attemptNr=max(file.attemptNr)
  * checking mutable before preparing tasks
  * enabled consistency check for chained prod tasks

* 12/15/2014
  * fixed getTasksToExecCommand_JEDI for recovery from pending
  * added reqID to JobSpec
  * setting lifetime to panda/user datasets using rucio api
  * introduced seq_number
  * added support for offset of seq_number

* 12/12/2014
  * added junk file cleaner in AtlasPP
  * improved AtlasTS to open output/log datasets

* 12/10/2014
  * to rescue random_seed
  * fixed ddo check in AtlasTR

* 12/9/2014
  * added some debug messages in insertFilesForDataset_JEDI

* 12/5/2014
  * using t1Weight=0 when undefined
  * increased threshold for fax overflow to 0.5 

* 12/4/2014
  * changed AtlasProdJobBroker to send scouts to T1
  * changed AtlasProdJobBroker to count # of jobs regardless of priorities
  * fixed for ddo:

* 12/3/2014
  * using old=True for replica lookup
  * using nEventsPerFile when rucio gives nevents=None

* 12/2/2014
  * added debug messages to AtlasDDM

* 12/1/2014
  * setting ddmBackEnd for ATLAS tasks

* 11/27/2014
  * set frozenTime for pending in Refiner

* 11/26/2014
  * fixed setSplitRule for -
  * using frozenTime for pending
  * added getOldBuildFileSpec
  * checking attribute lengths before registering tasks

* 11/25/2014
  * added unlockTasks
  * added maxAttemptES
  * using merge scout attributes    

* 11/24/2014
  * added pauseTask and resumeTask
  * unthrottling to generate merge jobs

* 11/19/2014
  * fixed AtlasDDM for analysis+rucio
  * skipping setOwner for rucio
  * limitting nRecords with event-splitting in insertFiles
  * using pid to lock in CF

* 11/18/2014
  * fixed AtlasDDM for DQDatasetExistsException
  * change to avoid contention in insertFilesForDataset_JEDI

* 11/17/2014
  * changed JG to (un)set lockedTime
  * changed AtlasProdJobThr to send logging messages 
  * improved throttling in AtlasProdJobThr

* 11/14/2014
  * changed AtlasAnalJB to check core matching
  * fixed reassignTask to not use pending as oldStatus

* 11/13/2014
  * fixed setCloudToTasks_JEDI to avoid late update
  * skipping file lookup for prod merging

* 11/12/2014
  * added random and reusable to dataset attributes

* 11/11/2014
  * fixed getRW
  * removed patch

* 11/10/2014
  * changed meas_date to xrdcp_last_update

* 11/7/2014
  * fixed reassignTaskToSite 

* 11/6/2014
  * introduced respectLB and nEventsPerMergeJob

* 11/5/2014
  * fixed JG to set destinationDBlock
  * fixed AtlasProdJB to check storageToken

* 11/4/2014
  * fixed nGBPerMergeJob
  * changed for lumiBlockNr

* 10/30/2014
  * added child process check in Interaction
  * added more informative message for refine failure

* 10/29/2014
  * added ProcessWrapper
  * fixed JediDBProxy to correctly set superstatus when clouds are assigned
  * fixed insertFiles for filelist on container

* 10/28/2014
  * added memory monitor in Interaction

* 10/27/2014
  * fixed tape lookup in AtlasAnalJB
  * added timeout to join in JG
  * added meas_date limit to NN
  * changed AtlasAnalJB for AthAnalysis
  * changed to allow tape access to users
  * added t1Weight
  * fixed AtlasProdTaskPP to delete intermediate datasets

* 10/24/2014
  * added safety margin in walltime estimation

* 10/21/2014
  * fixed AtlasProdJobThrottler for composit work queues

* 10/20/2014
  * fixed JG for startEvent

* 10/15/2014
  * improved AtlasProdTR to set destination from dst:

* 10/5/2014
  * fixed AtlasProdJobBroker for pre-assigned high prio tasks  

* 10/3/2014
  * fixed JG to propagate middle-name to merged files 

* 9/26/2014
  * improved getTasksToBeFinished_JEDI to avoid row lock contention
  * changed AtlasAnalPostProcessor to skip freezing trn
  * fixed infinite output/log dataset making

* 9/25/2014
  * setting minRamCount and maxCpuCount to pmerge jobs

* 9/24/2014
  * improved AtlasAnalysisPP for multiple threads processing the same task
  * increased timeout of Interaction to 600

* 9/23/2014
  * changed for prod tasks to avalanche when 5 scout jobs succeeded 
  * introduced scoutSuccessRate
  * fixed AtlasProdJobBroker for noInput
  * capped maxDiskCount until maxwdir is used everywhere
  * added capability to reset resource requirements
  * added split rule for merge step

* 9/22/2014
  * removed the hint from getOutputFiles_JEDI
  * added protection for non exiting cliParams to AtlasAnalPP

* 9/16/2014
  * added umNameAtEnd

* 9/12/2014
  * cleanup for reqID
  * added a hint to getOutputFiles_JEDI
  * reduced fetch size in getOutputFiles_JEDI

* 9/11/2014
  * added protection to getScoutData against widely distributed input sizes
  * fixed AtlasAnalJobBroker to take effective input size into account for walltime

* 9/10/2014
  * fixed undefined variables in JediDBProxy
  * added addNthFieldToLFN

* 9/8/2014
  * fixed file lookup for container
  * introduced trn_unknown
  * added protection against changed comm_parameter

* 9/7/2014
  * removing duplicated panda.um

* 9/3/2014
  * changed AtlasDDM to use listDatasets2
  * changed TR to set proper status/error when registration is failed

* 9/2/2014
  * changed AtlasDDM to work with dq2 and rucio
  * updating startTime at submitssion or retry
  * changed pending killer to take startTime into account

* 8/29/2014
  * fixed getAnalSitesWithData
  * added fixedSandbox
  * setting 0 for SKIPEVENTS when unspecified
  * resetting cloud and site

* 8/27/2014
  * fixed retryTask for running inputs
  * not to overwite useScout when incexec
  * disabled file lookup for merge jobs
  * keeping trn_log

* 8/26/2014
  * fixed JG to bunch of datasets for each task in a single cycle
  * introduced max num of datasets in getTasksTobeProcessed 
  * replacing LS in splitRule
  * fixed retryTask

* 8/25/2014
  * fixed JG for merging with only logs

* 8/24/2014
  * reusing lib.tgz when DDM endpoints are shared
  * resetting nFiles and nEvents when incexec
  * using firstSubmission

* 8/22/2014
  * fixed TC to delete only existing params for incexec

* 8/21/2014
  * adding All to email notification when all input succeeded
  * updating nFiles of output/log/lib datasets when the task is done

* 8/20/2014
  * added protection against large --nGBPerJob
  * fixed JG to use MB in maxDiskCount
  * fixed JG not to include input size in maxDiskCount for remoteIO sites
  * fixed retryTask for task with multiple inputs

* 8/19/2014
  * reduced cutoff to 15 for selection of WAN access sites
  * updating task when incexec

* 8/18/2014
  * introduced fairshare in tobeProcessed
  * allowing retry for maxAttempt<=attemptNr
  * fixed JG to register all output datasets 

* 8/17/2014
  * fixed PP to check only master inputs
  * sending task view in notification
  * setting retry in history
  * added safety mergin in workDiskCoung
  * fixed getScoutData
  * fixed AtlasAnalJB to correctly use scout data

* 8/16/2014
  * locking task in JG
  * fixed for xmlConfig

* 8/15/2014
  * fixed for loadXML
  * fixed for retry
  * fixed incexec to use comm_parameters
  * changed AtlasProdPP to freeze output and log datasets
  * decreased limit for instance renewal in Interaction

* 8/14/2014
  * changed to allow reactivation of aborted tasks 
  * changed some method to avoid row contention
  * fixed AtlasDDM for sites which share the same DDM endpoints 

* 8/13/2014
  * removed delay after scouts
  * fixed AtlasAnalJB for nThrottle
  * changed to lock tasks before prepare or process
  * added more debug info to AtlasAnalJB
  * removing currentPrrio limit when counting jobs in AtlasAnalJB

* 8/12/2014
  * fixed file rescue for pseudo input
  * improved performance of getTasksToBeProcessed
  * fixed TaskRefinerBase for GRL
  * fixed file rescue for orphaned merging
  * added lock against multiple processing
  * using tobroken in JG

* 8/11/2014
  * fixed AtlasAnalJB for remote access sites

* 8/6/2014
  * fixed JG for ES jobs to have temporary log file names

* 8/5/2014
  * added campaign info
  * skipping some checks in AtlasProdJobBrokerage when site is preassigned

* 8/1/2014
  * introduced waitInput for Event Picking
  * fixed scouts to take nEventsPerJob into account

* 7/30/2014
  * added a protection against multiple registration of the same task

* 7/29/2014
  * modified scout data calculation to take coreCount into account

* 7/25/2014
  * added more patterns for hospital queues

* 7/24/2014
  * setting superStatus

* 7/23/2014
  * added retryChildTasks_JEDI
  * added stateChangeTime
  * setting maxAttempt for secondary files as well

* 7/22/2014
  * removed US constraint from network-aware brokerage

* 7/19/2014
  * added a protection to CF to set ready to task when only secondary is checked
  * fixed maxDiskCout

* 7/10/2014
  * fixed retryTask
  * changed status=timeout to aborted
  * fixed calculation for effective walltime
  * removed <> from http logging messages
  * added refresh() to *Base
  * added http logging
  * changed to propagate aborting to DEFT
  * added caching for SiteMapper

* 7/8/2014
  * changed PP to kill child tasks when parent is aborted/failed/broken

* 7/7/2014
  * changed CF to remove lost files

* 7/3/2014
  * added consistency check for input datasets

* 7/2/2014
  * implemented to process dependent tasks before parent is completed

* 6/30/2014
  * added vo info to FQAN list when submtting jobs
  * using basename as LFN for useListPFN
  * added ticketID and ticketSystemType
  * fixed Booster in AtlasProdWatchDog

* 6/27/2014
  * fixed TaskRefiner to properly tasks when parent task failed to complete

* 6/26/2014
  * changed AtlasProdJobBroker to work with old fairsharepolicy format

* 6/25/2014
  * added support for multiple consumers

* 6/20/2014
  * fixed for pre-processing

* 6/11/2014
  * added templates for cron and logrotate
  * modified email notification to use BigPandaMon and JIRA

* 6/1/2014
  * added eventService to JEDI_Tasks

* 5/30/2014
  * added plugins for general purpose
  * fixed job_retry_history

* 5/27/2014
  * added multiple attempts to ctl script

* 5/26/2014
  * changed error level of import failure in FactoryBase

* 5/21/2014
  * changed ATLAS anal brokerage to use CVMFS check for nightlies
  * added addEventServiceTask.py

* 5/13/2014
  * added reassignTask

* 5/8/2014
  * added a build number to version name

* 5/6/2014
  * fixed for t_task.total_events

* 4/29/2014
  * use startEvent and endEvent of files for event service

* 4/23/2014
  * changed CC to allow nEventsPerInputFile

* 4/21/2014
  * initializing nevents, total_req_jobs and metadata

* 4/17/2014
  * introduced toabort
  * fixed pp to set failed to datasets for failed tasks

* 4/14/2014
  * fixed for task broker

* 4/9/2014
  * added AtlasAnalWatchDog
  * improved log in AtlasProdJobBroker
  * added randomSleep to all workers

* 4/7/2014
  * fixed RucioFC lookup
  * fixed to set higher priorities to pmerge
  * fixed to use proper currentPriority in AtlasAnalJobBroker
  * changed to reuse lib.tgz in analysis
  * fixed splitter for merging
  * changed AtlasAnalJobBroker to take throttled jobs into account

* 3/25/2014
  * fixed to avoid duplicated usage of premerged datasets
  * fixed to set status=scouting even if scout jobs run on multiple datasets

* 3/20/2014
  * changed xrd to fax

* 3/19/2014
  * chaned to set t_task.start_time

* 3/15/2014
  * chaned to propagate aborted to DEFT

* 3/14/2014
  * changed to set parent_tid=taskid when no parent
  * changed the initial task status to waiting from submit
  * chaned to propagate more task status to DEFT
  * reset splitRule when waiting for parent

* 3/12/2014
  * fixed notification for merge

* 3/10/2014
  * improved ATLAS job brokerage for --site=A,B,C,AUTO
  * added a mechanism for task chain
  * fixed JEDI email notification to show process and dataset summary

* 3/7/2014
  * fixed wrong errorDiaglog="No build jobs succeeded" for group anal

* 3/4/2014
  * added protection to TR against errors happen before impl or taskSpec is generated
  * changed finished/partial to done/finished

* 3/3/2014
  * changed watchdog to allow vo/label specific acction
  * enabled FAX

* 2/20/2014
  * changed ATLAS job brokerage to allow jobs go to brokeroff when preassigned 
  * removed disk size check for directIO sites from ATLAS job brokerage
  * enabled rucio endpoint

* 2/17/2014
  * fixed disk threshold for anal brokerage
  * fixed for transHome=None

* 2/13/2014
  * changed for new columns in t_task

* 2/11/2014
  * ignoring panda internal datasets
  * added protection for unknown dataset
  * support for wild-card in input dataset/container names

* 2/7/2014
  * fixed getScoutData to calculate data per MB

* 2/6/2014
  * fixed AtlasJB to use LOCALGROUPDISK

* 2/4/2014
  * adding attemptNr to output/log LFN
  * fixed for log+ObjectStore

* 2/3/2014
  * implemented event service functions
  * fixed ATLAS pp to delete transient or empty datasets
  * fixed ATLAS job brokerge for nightlies 

* 1/29/2014
  * added protection against RUCIOTEST
  * making new lib when previous attempt failed at the same site

* 1/22/2014
  * added pfnList
  * fixed disableAutoRetry to disable JEDI retry too
  * enabled recordRetryHistory_JEDI

* 1/19/2014
  * added disableAutoRetry

* 1/17/2014
  * added support for loadXML
  * using Job_Retry_History

* 1/15/2014
  * added safety merging to the output offset
  * fixed to set scope for lib

* 1/14/2014
  * fixed getScoutData to use lib size as work dir size
  * fixed InputChunk to use the output offset

* 1/13/2014
  * not chaning task status by merging
  * closing libDS
  * fixed to send email when no scouts succeeded
  * fixed to use unique name for lib.tgz
 
* 1/9/2014
  * fixed JG for cache

* 12/20/2013
  * fixed for multiple input streams

* 12/19/2013
  * added --noEmail
  * incexec with dsForXYZ

* 12/17/2013
  * changed to work with T_TASK
  
* 12/13/2013
  * added include/exclude in CF
  * set containerName when expand

* 12/12/2013
  * fixed for input container
  * fixed for DBR caching
  * added protection against task duplication
  * fixed for retry

* 12/11/2013
  * fixed AnalRefiner for DBR
  * fixed AtlasAnalJobBroker for data locality
  * fixed makeJobParameters for merging

* 12/10/2013
  * added pandamon URLs to AtlasAnalPostProcessor
  * fixed AtlasAnalJobBrokerage for remote access
  * implemented TaskSetupper for ATLAS analysis

* 12/2/2013 
  * set task status to broken when no scout jobs succeeded
  * fixed retry for scouting

* 12/1/2013 
  * increased the limit on the number of files 

* 11/25/2013 
  * refactored PostProcessor
  * added email notification

* 11/22/2013 
  * removed direct LFC dependence from AtlasDDMClient

* 11/21/2013 
  * setting transferType and sourceSite when remote access is used
  * implemented eventPicking and GRL

* 11/18/2013
  * changed getBestNNetworkSites_JEDI

* 11/15/2013
  * implemented task retry and incexec
  * added a protection to CF for too many input

* 11/9/2013
  * fixed getBestNNetworkSites_JEDI for table merging

* 11/7/2013
  * implemented merging

* 10/29/2013
  * added containerExpansion

* 10/20/2013
  * added nEventsPerRange
  * fixed CF to take offset for input into account

* 10/11/2013
  * added check if DBR is well replicated
  * fixed for /E
  * set jobsetID
  * added LATEST DBR lookup

* 10/10/2013
  * improved JG for pathena

* 10/4/2013
  * implemented getSatelliteSites
 
* 9/26/2013
  * added preprocessing function

* 9/9/2013
  * added analysis functions
  * tagged 0.0.1

* 9/4/2013
  * fixed first event to start from 1
  * fixed random seed

* 8/31/2013
  * fixed prepareTask

* 8/29/2013
  * fixed Watchdoc
  * uploading logs when setupper is failed

* 8/28/2013
  * added support for log merge

* 8/19/2013
  * added timeout for pending

* 8/12/2013
  * fixed CF for nFilesPerJob + scouting 

* 8/9/2013
  * fixed splitter to respect nFilesPerJob even in scouting 

* 8/8/2013
  * added a capability to make build jobs

* 7/31/2013
  * added fullSimulation for job splitter 
  * removed defualt walltime in AtlasProdTaskRefiner
  * improved AtlasJobBroker to upload log snippet
  * changed AtlasJobBroker to take max/minmemory and mintime into account

* 7/29/2013
  * added an error message to CF when files are missing
  * changed FR to take oldAccompanyDatasetNames into account

* 7/22/2013
  * added TaskSetupper

* 7/19/2013
  * added TG for FileRecovery

* 7/16/2013
  * fixed JG for FileRecovery

* 7/11/2013
  * fixed getScoutJobData

* 7/10/2013
  * added FileRecovery

* 7/3/2013
  * alpha version 

* 6/28/2013
  * added missing file remover to AtlasProdTaskBroker 

* 6/25/2013
  * fixed JG to take maxNumJob into account
  * added reniceJEDI

* 6/21/2013
  * fixed JobThrottler for maxNumJob and minPriority

* 6/20/2013
  * added jediTaskID to the WHERE clause for all UPDATE

* 6/19/2013
  * improved CF to directly avalanche when skipScout=True
  * improved JG to reset unused files
  * improved some queries to use AUX table
  * fixed AtlasDDMClient to ignore duplicated files with different attempt number
  * added support for non-input tasks

* 6/14/2013
  * fixed to support secondary datasets which has non-integer ratio to master 

* 6/11/2013
  * fixed WorkQueueMapper

* 6/6/2013
  * added ZombiCleaner

* 6/3/2013
  * changed schema names configurable
  * fixed CF for broken datasets

* 6/1/2013
  * added TaskCommando

* 5/27/2013
  * added TaskBroker

* 5/22/2013
  * renamed taskID to jediTaskID

* 5/10/2013
  * first version

 LocalWords:  serizlize NthField