Virtual Data Management for Grid Computing : Virtual Data Management for Grid Computing Michael Wilde
Argonne National Laboratory
wilde@mcs.anl.gov
Gaurang Mehta, Karan Vahi
Center for Grid Technologies
USC Information Sciences Institute
gmehta,vahi @isi.edu
Outline : Outline The concept of Virtual Data
The Virtual Data System and Language
Tools and approach for Running on the Grid
Pegasus: Grid Workflow Planning
Virtual Data Applications on the Grid
Research issues
The Virtual Data Concept : The Virtual Data Concept Enhance scientific productivity through:
Discovery and application of datasets and programs at petabyte scale
Enabling use of a worldwide data grid as a scientific workstation
Virtual Data enables this approach by creating datasets from workflow “recipes” and recording their provenance.
Virtual Data Concept : Virtual Data Concept Motivated by next generation data-intensive applications
Enormous quantities of data, petabyte-scale
The need to discover, access, explore, and analyze diverse distributed data sources
The need to organize, archive, schedule, and explain scientific workflows
The need to share data products and the resources needed to produce and store them
GriPhyN – The Grid Physics Network : GriPhyN – The Grid Physics Network NSF-funded IT research project
Supports the concept of Virtual Data, where data is materialized on demand
Data can exist on some data resource and be directly accessible
Data can exist only in a form of a recipe
The GriPhyN Virtual Data System can seamlessly deliver the data to the user or application regardless of the form in which the data exists
GriPhyN targets applications in high-energy physics, gravitational-wave physics and astronomy
Virtual Data System Capabilities : Virtual Data System Capabilities Producing data from transformations with uniform, precise data interface descriptions enables…
Discovery: finding and understanding datasets and transformations
Workflow: structured paradigm for organizing, locating, specifying, & producing scientific datasets
Forming new workflow
Building new workflow from existing patterns
Managing change
Planning: automated to make the Grid transparent
Audit: explanation and validation via provenance
Virtual Data Example: Galaxy Cluster Search : Jim Annis, Steve Kent, Vijay Sehkri, Fermilab, Michael Milligan, Yong Zhao, University of Chicago Galaxy cluster
size distribution DAG Virtual Data Example: Galaxy Cluster Search Sloan Data
Slide9 : mass = 200
decay = WW
stability = 1
LowPt = 20
HighPt = 10000 Virtual Data Application:
High Energy Physics Data Analysis Work and slide by
Rick Cavanaugh and
Dimitri Bourilkov,
University of Florida
Lifecycle of Virtual Data : I have some subject images, what analyses are available? which can be applied to this format? Lifecycle of Virtual Data I want to apply an image registration program to thousands of objects. If the results already exist, I’ll save weeks of computation. I’ve come across some interesting data, but I need to understand the nature of the analyses applied when it was constructed before I can trust it for my purposes. I need to document the devices and methods used to measure the data, and keep track of the analysis steps applied to the data.
A Virtual Data Grid : A Virtual Data Grid
Virtual Data Scenario : Virtual Data Scenario On-demand data generation Update workflow following changes Manage workflow; psearch –t 10 –i file3 file4 file5 –o file8 summarize –t 10 –i file6 –o file7 reformat –f fz –i file2 –o file3 file4 file5 conv –l esd –o aod –i file 2 –o file6 simulate –t 10 –o file1 file2 Explain provenance, e.g. for file8:
VDL and Abstract Workflow : VDL and Abstract Workflow VDL descriptions User request data file “c”
Example Workflow Reduction : Example Workflow Reduction Original abstract workflow
If “b” already exists (as determined by query to the RLS), the workflow can be reduced
Mapping from abstract to concrete : Mapping from abstract to concrete
Query RLS, MDS, and TC, schedule computation and data movement
Workflow management in GriPhyN : Workflow management in GriPhyN Workflow Generation: how do you describe the workflow (at various levels of abstraction)? (Virtual Data Language and catalog “VDC”))
Workflow Mapping/Refinement: how do you map an abstract workflow representation to an executable form? (“Planners”: Pegasus)
Workflow Execution: how to you reliably execute the workflow? (Condor’s DAGMan)
Executable Workflow Construction : Executable Workflow Construction VDL tools used to build an abstract workflow based on VDL descriptions
Planners (e.g. Pegasus) take the abstract workflow and produce an executable workflow for the Grid or other environments
Workflow executors (“enactment engines”) like Condor DAGMan execute the workflow
Terms : Terms Abstract Workflow (DAX)
Expressed in terms of logical entities
Specifies all logical files required to generate the desired data product from scratch
Dependencies between the jobs
Analogous to build style dag
Concrete Workflow
Expressed in terms of physical entities
Specifies the location of the data and executables
Analogous to a make style dag
Outline : Outline The concept of Virtual Data
The Virtual Data System and Language
Tools and Approach for Running on the Grid
Pegasus: Grid Workflow Planning
Virtual Data Applications on the Grid
Research issues
VDL: Virtual Data Language Describes Data Transformations : VDL: Virtual Data Language Describes Data Transformations Transformation - “TR”
Abstract template of program invocation
Similar to "function definition"
Derivation – “DV”
“Function call” to a Transformation
Store past and future:
A record of how data products were generated
A recipe of how data products can be generated
Invocation
Record of a Derivation execution
Example Transformation : Example Transformation TR t1( out a2, in a1, none pa = "500", none env = "100000" ) {
argument = "-p "${pa};
argument = "-f "${a1};
argument = "-x –y";
argument stdout = ${a2};
profile env.MAXMEM = ${env};
} $a1 $a2 t1
Example Derivations : Example Derivations DV d1->t1 ( env="20000", pa="600", a2=@{out:run1.exp15.T1932.summary}, a1=@{in:run1.exp15.T1932.raw},
);
DV d2->t1 ( a1=@{in:run1.exp16.T1918.raw}, a2=@{out.run1.exp16.T1918.summary}
);
Workflow from File Dependencies : Workflow from File Dependencies TR tr1(in a1, out a2) {
argument stdin = ${a1};
argument stdout = ${a2}; }
TR tr2(in a1, out a2) {
argument stdin = ${a1};
argument stdout = ${a2}; }
DV x1->tr1(a1=@{in:file1}, a2=@{out:file2});
DV x2->tr2(a1=@{in:file2}, a2=@{out:file3}); file1 file2 file3 x1 x2
Example Workflow : Example Workflow Complex structure
Fan-in
Fan-out
"left" and "right" can run in parallel
Uses input file
Register with RLS
Complex file dependencies
Glues workflow findrange findrange analyze preprocess
Workflow step "preprocess" : Workflow step "preprocess" TR preprocess turns f.a into f.b1 and f.b2
TR preprocess( output b[], input a ) { argument = "-a top"; argument = " –i "${input:a}; argument = " –o " ${output:b};
}
Makes use of the "list" feature of VDL
Generates 0..N output files.
Number file files depend on the caller.
Workflow step "findrange" : Workflow step "findrange" Turns two inputs into one output
TR findrange( output b, input a1, input a2, none name="findrange", none p="0.0" ) { argument = "-a "${name}; argument = " –i " ${a1} " " ${a2}; argument = " –o " ${b}; argument = " –p " ${p};
}
Uses the default argument feature
Can also use list[] parameters : Can also use list[] parameters
TR findrange( output b, input a[], none name="findrange", none p="0.0" ) { argument = "-a "${name}; argument = " –i " ${" "|a}; argument = " –o " ${b}; argument = " –p " ${p};
}
Workflow step "analyze" : Workflow step "analyze" Combines intermediary results
TR analyze( output b, input a[] ) { argument = "-a bottom"; argument = " –i " ${a}; argument = " –o " ${b};
}
Complete VDL workflow : Complete VDL workflow Generate appropriate derivations
DV top->preprocess( b=[ @{out:"f.b1"}, @{ out:"f.b2"} ], a=@{in:"f.a"} );
DV left->findrange( b=@{out:"f.c1"}, a2=@{in:"f.b2"}, a1=@{in:"f.b1"}, name="left", p="0.5" );
DV right->findrange( b=@{out:"f.c2"}, a2=@{in:"f.b2"}, a1=@{in:"f.b1"}, name="right" );
DV bottom->analyze( b=@{out:"f.d"}, a=[ @{in:"f.c1"}, @{in:"f.c2"} );
Compound Transformations : Compound Transformations Using compound TR
Permits composition of complex TRs from basic ones
Calls are independent
unless linked through LFN
A Call is effectively an anonymous derivation
Late instantiation at workflow generation time
Permits bundling of repetitive workflows
Model: Function calls nested within a function definition
Compound Transformations (cont) : Compound Transformations (cont) TR diamond bundles black-diamonds
TR diamond( out fd, io fc1, io fc2, io fb1, io fb2, in fa, p1, p2 ) {
call preprocess( a=${fa}, b=[ ${out:fb1}, ${out:fb2} ] );
call findrange( a1=${in:fb1}, a2=${in:fb2}, name="LEFT", p=${p1}, b=${out:fc1} );
call findrange( a1=${in:fb1}, a2=${in:fb2}, name="RIGHT", p=${p2}, b=${out:fc2} );
call analyze( a=[ ${in:fc1}, ${in:fc2} ], b=${fd} );
}
Compound Transformations (cont) : Compound Transformations (cont) Multiple DVs allow easy generator scripts:
DV d1->diamond( fd=@{out:"f.00005"}, fc1=@{io:"f.00004"}, fc2=@{io:"f.00003"}, fb1=@{io:"f.00002"}, fb2=@{io:"f.00001"}, fa=@{io:"f.00000"}, p2="100", p1="0" );
DV d2->diamond( fd=@{out:"f.0000B"}, fc1=@{io:"f.0000A"}, fc2=@{io:"f.00009"}, fb1=@{io:"f.00008"}, fb2=@{io:"f.00007"}, fa=@{io:"f.00006"}, p2="141.42135623731", p1="0" );
...
DV d70->diamond( fd=@{out:"f.001A3"}, fc1=@{io:"f.001A2"}, fc2=@{io:"f.001A1"}, fb1=@{io:"f.001A0"}, fb2=@{io:"f.0019F"}, fa=@{io:"f.0019E"}, p2="800", p1="18" );
Functional MRI Analysis : Functional MRI Analysis
fMRI Example: AIR Tools : fMRI Example: AIR Tools TR air::align_warp( in reg_img, in reg_hdr, in sub_img,
in sub_hdr, m, out warp )
{
argument = ${reg_img};
argument = ${sub_img};
argument = ${warp};
argument = "-m " ${m};
argument = "-q";
}
TR air::reslice( in warp, sliced, out sliced_img, out sliced_hdr )
{
argument = ${warp};
argument = ${sliced};
}
fMRI Example: AIR Tools : fMRI Example: AIR Tools TR air::warp_n_slice(
in reg_img, in reg_hdr, in sub_img, in sub_hdr, m = "12",
io warp, sliced, out sliced_img, out sliced_hdr )
{
call air::align_warp( reg_img=${reg_img}, reg_hdr=${reg_hdr},
sub_img=${sub_img}, sub_hdr=${sub_hdr},
m=${m},
warp = ${out:warp} );
call air::reslice( warp=${in:warp}, sliced=${sliced},
sliced_img=${sliced_img}, sliced_hdr=${sliced_hdr} );
}
TR air::softmean( in sliced_img[], in sliced_hdr[], arg1 = "y",
arg2 = "null", atlas, out atlas_img, out atlas_hdr )
{
argument = ${atlas};
argument = ${arg1} " " ${arg2};
argument = ${sliced_img};
}
fMRI Example: AIR Tools : fMRI Example: AIR Tools DV air::i3472_3->air::warp_n_slice(
reg_hdr = @{in:"3472-3_anonymized.hdr"},
reg_img = @{in:"3472-3_anonymized.img"},
sub_hdr = @{in:"3472-3_anonymized.hdr"},
sub_img = @{in:"3472-3_anonymized.img"},
warp = @{io:"3472-3_anonymized.warp"},
sliced = "3472-3_anonymized.sliced",
sliced_hdr = @{out:"3472-3_anonymized.sliced.hdr"},
sliced_img = @{out:"3472-3_anonymized.sliced.img"} );
…
DV air::i3472_6->air::warp_n_slice(
reg_hdr = @{in:"3472-3_anonymized.hdr"},
reg_img = @{in:"3472-3_anonymized.img"},
sub_hdr = @{in:"3472-6_anonymized.hdr"},
sub_img = @{in:"3472-6_anonymized.img"},
warp = @{io:"3472-6_anonymized.warp"},
sliced = "3472-6_anonymized.sliced",
sliced_hdr = @{out:"3472-6_anonymized.sliced.hdr"},
sliced_img = @{out:"3472-6_anonymized.sliced.img"} );
fMRI Example: AIR Tools : fMRI Example: AIR Tools DV air::a3472_3->air::softmean(
sliced_img = [
@{in:"3472-3_anonymized.sliced.img"},
@{in:"3472-4_anonymized.sliced.img"},
@{in:"3472-5_anonymized.sliced.img"},
@{in:"3472-6_anonymized.sliced.img"} ],
sliced_hdr = [
@{in:"3472-3_anonymized.sliced.hdr"},
@{in:"3472-4_anonymized.sliced.hdr"},
@{in:"3472-5_anonymized.sliced.hdr"},
@{in:"3472-6_anonymized.sliced.hdr"} ],
atlas = "atlas",
atlas_img = @{out:"atlas.img"},
atlas_hdr = @{out:"atlas.hdr"}
);
Query Examples : Query Examples Which TRs can process a "subject" image?
Q: xsearchvdc -q tr_meta dataType subject_image input
A: fMRIDC.AIR::align_warp
Which TRs can create an "ATLAS"?
Q: xsearchvdc -q tr_meta dataType atlas_image output
A: fMRIDC.AIR::softmean
Which TRs have output parameter "warp" and a parameter "options"
Q: xsearchvdc -q tr_para warp output options
A: fMRIDC.AIR::align_warp
Which DVs call TR "slicer"?
Q: xsearchvdc -q tr_dv slicer
A: fMRIDC.FSL::s3472_3_x->fMRIDC.FSL::slicer
fMRIDC.FSL::s3472_3_y->fMRIDC.FSL::slicer fMRIDC.FSL::s3472_3_z->fMRIDC.FSL::slicer
Query Examples : Query Examples List anonymized subject-images for young subjects. This query searches for files based on their metadata.
Q: xsearchvdc -q lfn_meta
dataType subject_image
privacy anonymized
subjectType young
A: 3472-4_anonymized.img
For a specific patient image, 3472-3, show all DVs and files that were derived from this image, directly or indirectly.
Q: xsearchvdc -q lfn_tree
3472-3_anonymized.img
A: 3472-3_anonymized.img
3472-3_anonymized.sliced.hdr
atlas.hdr
atlas.img
…
atlas_z.jpg
3472-3_anonymized.sliced.img
Virtual Provenance: list of derivations and files : Virtual Provenance: list of derivations and files
...
…
...
(excerpted for display)
Virtual Provenance in XML: control flow graph : Virtual Provenance in XML: control flow graph
...
...
...
...…
(excerpted for display…)
Invocation Provenance : Invocation Provenance Completion status and resource usage Attributes of executable transformation Attributes of input and output files
Slide43 : Future: Provenance & Workflow for Web Services
Outline : Outline The concept of Virtual Data
The Virtual Data System and Language
Tools and Issues for Running on the Grid
Pegasus: Grid Workflow Planning
Virtual Data Applications on the Grid
Research issues
Outline : Outline Introduction and the GriPhyN project
Chimera
Overview of Grid concepts and tools
Pegasus
Applications using Chimera and Pegasus
Research issues
Exercises
Motivation for Grids: How do we solve problems? : Motivation for Grids: How do we solve problems? Communities committed to common goals
Virtual organizations
Teams with heterogeneous members & capabilities
Distributed geographically and politically
No location/organization possesses all required skills and resources
Adapt as a function of the situation
Adjust membership, reallocate responsibilities, renegotiate resources
The Grid Vision : The Grid Vision “Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”
On-demand, ubiquitous access to computing, data, and services
New capabilities constructed dynamically and transparently from distributed services
The Grid : The Grid Emerging computational, networking, and storage infrastructure
Pervasive, uniform, and reliable access to remote data, computational, sensor, and human resources
Enable new approaches to applications and problem solving
Remote resources the rule, not the exception
Challenges
Heterogeneous components
Component failures common
Different administrative domains
Local policies for security and resource usage
Data Grids for High Energy Physics : Data Grids for High Energy Physics www.griphyn.org www.ppdg.net www.eu-datagrid.org Slide courtesy Harvey Newman, CalTech
Grid Applications : Grid Applications Increasing in the level of complexity
Use of individual application components
Reuse of individual intermediate data products
Description of Data Products using Metadata Attributes
Execution environment is complex and very dynamic
Resources come and go
Data is replicated
Components can be found at various locations or staged in on demand
Separation between
the application description
the actual execution description
Grid3 – The Laboratory : Grid3 – The Laboratory Supported by the National Science Foundation and the Department of Energy.
Globus Monitoring and Discovery Service (MDS) : Globus Monitoring and Discovery Service (MDS) The MDS architecture is a flexible hierarchy. There can be several levels of GIISes; any GRIS can register with any GIIS; and any GIIS can register with another, making this approach modular and extensible.
GridFTP : GridFTP Data-intensive grid applications transfer and replicate large data sets (terabytes, petabytes)
GridFTP Features:
Third party (client mediated) transfer
Parallel transfers
Striped transfers
TCP buffer optimizations
Grid security
Important feature is separation of control and data channel
A Replica Location Service : A Replica Location Service A Replica Location Service (RLS) is a distributed registry service that records the locations of data copies and allows discovery of replicas
Maintains mappings between logical identifiers and target names
Physical targets: Map to exact locations of replicated data
Logical targets: Map to another layer of logical names, allowing storage systems to move data without informing the RLS
RLS was designed and implemented in a collaboration between the Globus project and the DataGrid project
Slide56 : LRCs contain consistent information about logical-to-target mappings on a site
RLIs nodes aggregate information about LRCs
Soft state updates from LRCs to RLIs: relaxed consistency of index information, used to rebuild index after failures
Arbitrary levels of RLI hierarchy
Condor’s DAGMan : Condor’s DAGMan Developed at UW Madison (Livny)
Executes a concrete workflow
Makes sure the dependencies are followed
Executes the jobs specified in the workflow
Execution
Data movement
Catalog updates
Provides a “rescue DAG” in case of failure
Outline : Outline The concept of Virtual Data
The Virtual Data System and Language
Tools and Approach for Running on the Grid
Pegasus: Grid Workflow Planning
Virtual Data Applications on the Grid
Research issues
Pegasus Outline : Pegasus Outline Introduction
Pegasus and How it works
Pegasus Components and Internals
Deferred Planning
Pegasus Portal
Introduction : Introduction Scientific applications in various domains are compute and data intensive.
Not feasible to run on a single resource (limits scalability).
Scientific application can be represented as a workflow.
Use of individual application components.
Reuse data if possible (Virtual Data).
Generating an Abstract Workflow : Generating an Abstract Workflow Available Information
Specification of component capabilities
Ability to generate the desired data products
Select and configure application components to form an abstract workflow
assign input files that exist or that can be generated by other application components.
specify the order in which the components must be executed
components and files are referred to by their logical names
Logical transformation name
Logical file name
Both transformations and data can be replicated
Generating a Concrete Workflow : Generating a Concrete Workflow Information
location of files and component Instances
State of the Grid resources
Select specific
Resources
Files
Add jobs required to form a concrete workflow that can be executed in the Grid environment
Each component in the abstract workflow is turned into an executable job
Slide64 : Concrete Workflow Generator Domain
Knowledge Resource
Information Location
Information Concrete Workflow
Generator Plan submitted to the grid Abstract Workflow
Pegasus : Pegasus Pegasus - Planning for Execution on Grids
Planning framework
Take as input an abstract workflow
Abstract Dag in XML (DAX)
Generates concrete workflow.
Submits the workflow to a workflow executor (e.g. Condor Dagman).
Pegasus Outline : Pegasus Outline Introduction
Pegasus and How it works
Pegasus Components and Internals
Deferred Planning
Pegasus Portal
Slide68 : Abstract Workflow Reduction
Slide69 : Jobs d, e, f have output files that have been found in the Replica Location Service.
Additional jobs are deleted.
All jobs (a, b, c, d, e, f) are removed from the DAG. Job e Job g Job h Job d Job a Job c Job f Job i Job b Optimizing from the point of view of Virtual Data
Slide70 : Job e Job g Job h Job d Job a Job c Job f Job i Job b adding transfer nodes for the input files for the root nodes Planner picks execution
and replica locations
Plans for staging data in
Slide71 : Staging and registering for each job that materializes data (g, h, i ). KEY
The original node
Input transfer node
Registration node
Output transfer node
Node deleted by Reduction algorithm Job e Job g Job h Job d Job a Job c Job f Job i Job b Staging data out and registering
new derived products
Pegasus Outline : Pegasus Outline Introduction
Pegasus and How it works
Pegasus Components and Internals
Deferred Planning
Pegasus Portal
Pegasus Components : Pegasus Components
Replica Location Service : Replica Location Service Pegasus uses the RLS to find input data
LRC LRC LRC RLI Pegasus uses the RLS to register new data products
Pegasus Components : Pegasus Components
Transformation Catalog : Transformation Catalog Transformation Catalog keeps information about domain transformations (executables).
It keeps information like the logical name of the transformation, the resource on the which the transformation is available and the URL for the physical transformation.
The new transformation catalog also allows to define different types of transformation, Architecture, OS, OS-version and glibc version of the transformation.
Profiles which allow a user to attach Environment variables, hints, scheduler related information, job-manager configurations
Transformation Catalog : Transformation Catalog A set of standard api's are available to perform various operations on the transformation Catalog.
Various implementations are available for the transformation catalog like
Database :
Useful for large Catalogs and supports fast adds, deletes and queries.
Standard database security used.
File Format :
Useful for small number of transformations and easy editing.
isi vds::fft:1.0 /opt/bin/fft INSTALLED INTEL32::LINUX ENV::VDS_HOME=/opt/vds
Users can provide their own implementation.
Pegasus Components : Pegasus Components
Resource Information Catalog (Pool Config) : Resource Information Catalog (Pool Config) Pool Config is an XML file which contains information about various pools/sites on which DAGs may execute.
Some of the information contained in the Pool Config file is
Specifies the various job-managers that are available on the pool for the different types of schedulers.
Specifies the GridFtp storage servers associated with each pool.
Specifies the Local Replica Catalogs where data residing in the pool has to be cataloged.
Contains profiles like environment hints which are common site-wide.
Contains the working and storage directories to be used on the pool.
Resource Information Catalog Pool Config : Resource Information Catalog Pool Config Two Ways to construct the Pool Config File.
Monitoring and Discovery Service
Local Pool Config File (Text Based)
Client tool to generate Pool Config File
The tool gen-poolconfig is used to query the MDS and/or the local pool config file/s to generate the XML Pool Config file.
Monitoring and Discovery Service : Monitoring and Discovery Service MDS provides up-to-date Grid state information
Total and idle job queues length on a pool of resources (condor)
Total and available memory on the pool
Disk space on the pools
Number of jobs running on a job manager
Can be used for resource discovery and selection
Developing various task to resource mapping heuristics
Can be used to publish information necessary for replica selection
Publish gridftp transfer statistics
Developing replica selection components
Pegasus Components : Pegasus Components
Site Selector : Site Selector Determines job to site mappings.
Selector Api provides easy pluggable implementations.
Implementations provided
Random
RoundRobin
Group
NonJavaCallout
Research on advance algorithms for site selector in progress
Pegasus Components : Pegasus Components
Transfer Tools / Workflow Executors : Transfer Tools / Workflow Executors Various transfer tools are used for staging data between grid sites.
Define strategies to reliably and efficiently transfer data
Current implementations
Globus url copy
Transfer
Transfer2 (T2)
Stork
Workflow executors take the concrete workflow and act as a metascheduler for the Grid.
Current supported workflow executors
Condor Dagman
Gridlab GRMS
System Configuration - Properties : System Configuration - Properties Properties file define and modify the behavior of Pegasus.
Properties set in the $VDS_HOME/etc/properties can be overridden by defining them either in $HOME/.chimerarc or by giving them on the command line of any executable.
eg. Gendax –Dvds.home=path to vds home……
Some examples follow but for more details please read the sample.properties file in $VDS_HOME/etc directory.
Basic Required Properties
vds.home : This is auto set by the clients from the environment variable $VDS_HOME
vds.properties : Path to the default properties file
Default : ${vds.home}/etc/properties
Pegasus Clients in the Virtual Data System : Pegasus Clients in the Virtual Data System gencdag
Converts DAX to concrete Dag
genpoolconfig
Generates a pool config file from MDS
tc-client
Queries, adds, deletes transformations from the Transformation Catalog
rls-client, rls-query-client
Queries, adds, deletes entry from the RLS
partiondax
Partitions the abstract Dax into smaller dax’s.
Concrete Planner Gencdag : Concrete Planner Gencdag The Concrete planner takes the DAX produced by Chimera and converts into a set of condor dag and submit files.
You can specify more then one execution pools. Execution will take place on the pools on which the executable exists. If the executable exists on more then one pool then the pool on which the executable will run is selected depending on the site selector used.
Output pool is the pool where you want all the output products to be transferred to. If not specified the materialized data stays on the execution pool
Authentication removes sites which are not accessible.
Pegasus Outline : Pegasus Outline Introduction
Pegasus and How it works
Pegasus Components and Internals
Deferred Planning
Pegasus Portal
Abstract To Concrete Steps : Abstract To Concrete Steps
Original Pegasus configuration : Original Pegasus configuration Simple scheduling: random or round robin
using well-defined scheduling interfaces.
Deferred Planning through Partitioning : Deferred Planning through Partitioning A variety of partitioning algorithms can be implemented
Mega DAG is created by Pegasus and then submitted to DAGMan : Mega DAG is created by Pegasus and then submitted to DAGMan
Mega DAG Pegasus : Mega DAG Pegasus
Re-planning capabilities : Re-planning capabilities
Complex Replanning for Free (almost) : Complex Replanning for Free (almost)
Planning & Scheduling Granularity : Planning & Scheduling Granularity Partitioning
Allows to set the granularity of planning ahead
Node aggregation
Allows to combine nodes in the workflow and schedule them as one unit (minimizes the scheduling overheads)
May reduce the overheads of making scheduling and planning decisions
Related but separate concepts
Small jobs
High-level of node aggregation
Large partitions
Very dynamic system
Small partitions
Partitioned Workflow Processing : Partitioned Workflow Processing Create workflow partitions
partition the abstract workflow into smaller workflows.
create the xml partition graph that lists out the dependencies between partitions.
Create the MegaDAG (creates the dagman submit files)
transform the xml partition graph to it’s corresponding condor representation.
Submit the MegaDAG
Each job invokes Pegasus on a partition and then submits the plan generated back to condor.
Future work for Pegasus : Future work for Pegasus Staging in executables on demand
Expanding the scheduling plug-ins
Investigating various partitioning approaches
Investigating reliability across partitions
Pegasus Outline : Pegasus Outline Introduction
Pegasus and How it works
Pegasus Components and Internals
Deferred Planning
Pegasus Portal
Portal : Portal Provides proxy based login
Authentication to the sites
Configuration for files
Metadata based workflow creation
Workflow Submission and Monitoring
User Notification
Portal Demonstration : Portal Demonstration
Outline : Outline The concept of Virtual Data
The Virtual Data System and Language
Tools and Issues for Running on the Grid
Pegasus: Grid Workflow Planning
Virtual Data Applications on the Grid
Research issues
Types of Applications : Types of Applications Gravitational Wave Physics
High Energy Physics
Astronomy
Earthquake Science
Computational Biology
LIGO Scientific Collaboration : LIGO Scientific Collaboration Continuous gravitational waves are expected to be produced by a variety of celestial objects
Only a small fraction of potential sources are known
Need to perform blind searches, scanning the regions of the sky where we have no a priori information of the presence of a source
Wide area, wide frequency searches
Search is performed for potential sources of continuous periodic waves near the Galactic Center and the galactic core
Search for binary inspirals collapsing into black holes.
The search is very compute and data intensive
Slide109 : Additional resources used: Grid3 iVDGL resources
LIGO Acknowledgements : LIGO Acknowledgements
Patrick Brady, Scott Koranda, Duncan Brown, Stephen Fairhurst University of Wisconsin Milwaukee, USA
Stuart Anderson, Kent Blackburn, Albert Lazzarini, Hari Pulapaka, Teviet Creighton Caltech, USA
Gabriela Gonzalez, Louisiana State University
Many Others involved in the Testbed
www.ligo.caltech.edu
www.lsc-group.phys.uwm.edu/lscdatagrid/
LIGO Laboratory operates under NSF cooperative agreement PHY-0107417
Montage : Montage Montage (NASA and NVO)
Deliver science-grade custom mosaics on demand
Produce mosaics from a wide range of data sources (possibly in different spectra)
User-specified parameters of projection, coordinates, size, rotation and spatial sampling.
Mosaic created by Pegasus based Montage from a run of the M101 galaxy images on the Teragrid.
Small Montage Workflow : Small Montage Workflow ~1200 nodes
Montage Acknowledgments : Montage Acknowledgments Bruce Berriman, John Good, Anastasia Laity, Caltech/IPAC
Joseph C. Jacob, Daniel S. Katz, JPL
http://montage.ipac. caltech.edu/
Testbed for Montage: Condor pools at USC/ISI, UW Madison, and Teragrid resources at NCSA, Caltech, and SDSC.
Montage is funded by the National Aeronautics and Space Administration's Earth Science Technology Office, Computational Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology.
Other Applications Southern California Earthquake Center : Other Applications Southern California Earthquake Center Southern California Earthquake Center (SCEC), in collaboration with the USC Information Sciences Institute, San Diego Supercomputer Center, the Incorporated Research Institutions for Seismology, and the U.S. Geological Survey, is developing the Southern California Earthquake Center Community Modeling Environment (SCEC/CME).
Create fully three-dimensional (3D) simulations of fault-system dynamics.
Physics-based simulations can potentially provide enormous practical benefits for assessing and mitigating earthquake risks through Seismic Hazard Analysis (SHA).
The SCEC/CME system is an integrated geophysical simulation modeling framework that automates the process of selecting, configuring, and executing models of earthquake systems.
Acknowledgments :
Philip Maechling and Vipin Gupta
University Of Southern California
Astronomy : Astronomy Galaxy Morphology (National Virtual Observatory)
Investigates the dynamical state of galaxy clusters
Explores galaxy evolution inside the context of large-scale structure.
Uses galaxy morphologies as a probe of the star formation and stellar distribution history of the galaxies inside the clusters.
Data intensive computations involving hundreds of galaxies in a cluster
The x-ray emission is shown in blue, and the optical mission is in red. The colored dots are located at the positions of the galaxies within the cluster; the dot color represents the value of the asymmetry index. Blue dots represent the most asymmetric galaxies and are scattered throughout the image, while orange are the most symmetric, indicative of elliptical galaxies, are concentrated more toward the center. People involved: Gurmeet Singh, Mei-Hui Su, many others
Biology Applications : Biology Applications Tomography (NIH-funded project)
Derivation of 3D structure from a series of 2D electron microscopic projection images,
Reconstruction and detailed structural analysis
complex structures like synapses
large structures like dendritic spines.
Acquisition and generation of huge amounts of data
Large amount of state-of-the-art image processing required to segment structures from extraneous background.
Dendrite structure to be rendered by
Tomography Work performed by Mei-Hui Su with Mark Ellisman, Steve Peltier, Abel Lin, Thomas Molina (SDSC)
BLAST: set of sequence comparison algorithms that are used to search sequence databases for optimal local alignments to a query : BLAST: set of sequence comparison algorithms that are used to search sequence databases for optimal local alignments to a query Lead by Veronika Nefedova (ANL) as part of the
Paci Data Quest Expedition program 2 major runs were performed using Chimera and Pegasus:
60 genomes (4,000 sequences each),
In 24 hours processed Genomes selected from DOE-sponsored sequencing projects
67 CPU-days of processing time delivered
~ 10,000 Grid jobs
>200,000 BLAST executions
50 GB of data generated
2) 450 genomes processed
Speedup of 5-20 times were achieved because the compute nodes we used efficiently by keeping the submission of the jobs to the compute cluster constant.
Virtual Data Example: Galaxy Cluster Search : Jim Annis, Steve Kent, Vijay Sehkri, Fermilab, Michael Milligan, Yong Zhao, University of Chicago Galaxy cluster
size distribution DAG Virtual Data Example: Galaxy Cluster Search Sloan Data
Cluster Search Workflow Graph and Execution Trace : Cluster Search Workflow Graph and Execution Trace Workflow jobs vs time
Capone Grid Interactions : Capone Grid Interactions ATLAS Slides courtesy of Marco Mambelli, U Chicago ATLAS Team
ATLAS “Capone” Production Executor : ATLAS “Capone” Production Executor Reception
Job received from work distributor
Translation
Un-marshalling, ATLAS transformation
DAX generation
VDL tools generate abstract DAG
Input file retrieval from RLS catalog
Check RLS for input LFNs (retrieval of GUID, PFN)
Scheduling: CE and SE are chosen
Concrete DAG generation and submission
Pegasus creates Condor submit files
DAGMan invoked to manage remote steps
A job in Capone (2, execution) : A job in Capone (2, execution) Remote job running / status checking
Stage-in of input files, create POOL FileCatalog
Athena (ATLAS code) execution
Remote Execution Check
Verification of output files and exit codes
Recovery of metadata (GUID, MD5sum, exe attributes)
Stage Out: transfer from CE site to destination SE
Output registration
Registration of the output LFN/PFN and metadata in RLS
Finish
Job completed successfully, communicates to Windmill that jobs is ready for validation
Job status is sent to Windmill during all the execution
Windmill/DQ validate & register output in ProdDB
Ramp up ATLAS DC2 : Ramp up ATLAS DC2
Outline : Outline The concept of Virtual Data
The Virtual Data System and Language
Tools and approach for Running on the Grid
Pegasus: Grid Workflow Planning
Virtual Data Applications on the Grid
Research issues
Research directions in Virtual Data Grid Architecture : Research directions in Virtual Data Grid Architecture
Research issues : Focus on data intensive science
Planning is necessary
Reaction to the environment is a must (things go wrong, resources come up)
Iterative Workflow Execution:
Workflow Planner
Workload Manager
Planning decision points
Workflow Delegation Time (eager)
Activity Scheduling Time (deferred)
Resource Availability Time (just in time)
Decision specification level
Reacting to the changing
environment and recovering from failures
How does the communication takes place? Callbacks, workflow annotations etc… Planner Manager Concrete workflow Tasks Abstract workflow info info Grid Resource Manager Research issues
Provenance Hyperlinks : Provenance Hyperlinks
Goals for a Dataset Model : Goals for a Dataset Model