sw week april06

Event Tag / DDM Integration : 

Event Tag / DDM Integration Caitriana Nicholson Helen McGlone University of Glasgow

Outline: 

Outline Background Integration plan Site services installation Implementation of functionality Testing and results The full chain Conclusions and next steps

Background: 

Background Event tag infrastructure allows efficient event selection see plenary talk by J. Cranshaw http://agenda.cern.ch/askArchive.php?base=agenda&categ=a057207&id=a057207s7t3/transparencies Query to the Tag Database currently returns a list of file GUIDs DDM tools (DQ2) requires transport of files in datasets See https://uimon.cern.ch/twiki/bin/view/Atlas/DistributedDataManagement How can the 2 systems be integrated?

Slide4: 

DQ2 File1, File6, File8 File1, File2, File3 File4, File5, File6 File7, File8, File9 Dataset B Dataset C Local SE “NMuon > 3” File1, File6, File8 ?

Integration Plan: 

Integration Plan Include some dataset info in Tags Set up site services at Glasgow Extend DQ2 functionality Test implementation possibilities “Full chain” with Rome AOD Running from within a job

Site Services at Glasgow: 

Site Services at Glasgow Some initial problems with installation outside CERN For notes and patches, see http://ppewww.ph.gla.ac.uk/~helenm/TAGdatabases Now have DQ2 (0.1.4) running at http://grid02.ph.gla.ac.uk:8000/dq2/ FTS channel CERN -> Glasgow Glasgow’s new role as PPS “Tier-1” should allow more scope for testing

Extension of DQ2 functionality: 

Extension of DQ2 functionality registerNewDataset requires LFNs and GUIDs Don’t want to store LFNs in tags (may not know LFN at tag creation time) New version allows registration by GUID + VUID of dataset in which file already exists Available in next DQ2 release

Implementation Options: 

Implementation Options Suppose results of tag query return files from several datasets, across several sites Can create one new dataset, register as “incomplete” at these sites then subscribe new dataset to local site “Incomplete subscription model” Or create one dataset for each site where source files present, register as “complete” at each site then subscribe each to local site “Complete subscription model”

Slide9: 

DQ2 File1, File6, File8 File1, File2, File3 Local SE “NMuon > 3” File1, File6, File8 File1, File6, File8 File1, File6, File8 File1, File6, File8 Incomplete Subscription

Slide10: 

DQ2 File1, File6, File8 File1, File2, File3 Local SE “NMuon > 3” File1, File6, File8 File1 File6 File8 Complete Subscription

Testing: 

Testing Helen’s results? My results?

Rome Tags: 

Rome Tags Rome AOD stored on CASTOR in 38 datasets (directories) > nsls /castor/cern.ch/grid/atlas/datafiles/rome/recov10/merge rome.003017.merge.A7_w4jet rome.003034.merge.J1_Pt_17_35 rome.003036.merge.J3_Pt_70_140 rome.003038.merge.J5_Pt_280_560 rome.003039.merge.J6_Pt_560_1120 rome.003047.merge.H3_120_gamgam … Also have corresponding event tags

Rome Tags: 

Rome Tags Each AOD directory used to populate a dataset in DQ2 Filenames and guids taken from POOL file catalogue mysqlcatalog_mysql://reader@lxfs6021.cern.ch/FileCatalogRome registerNewDataset -f <xml catalogue> <dataset> Available on production service > ./dq2 listDatasets ‘%rome%merge%’ rome.003017.merge.A7_w4jet rome.004140.merge.GG1_pythia_gamgam rome.004321.aod.AcerMC_Zbb_4l rome.003110.merge.FJ3_fwjets_e1000 rome.004313.aod.H13_150_WW …

The Full Chain: 

The Full Chain Test full chain with Rome tags: Query tag database and get list of GUIDs (and DUIDs) Register as new dataset in DQ2 Find locations of existing datasets Currently all at CERN Subscribe dataset (incomplete) to these locations Dataset copied to local site Works for fake data files; problem with real AOD files Should be resolved with newer DQ2 release

Conclusions and Next Steps: 

Conclusions and Next Steps Integration proceeding as planned DQ2 0.2.0 should allow full chain to run with Rome AOD Awaiting decision on dataset ID to be included in tags Then try to run from within Athena Make utility available for user testing (June?)