SCEC/CME Project - How Earthquake Simulations Drive Middleware Requirements : SCEC/CME Project - How Earthquake Simulations Drive Middleware Requirements Philip Maechling
SCEC IT Architect
24 June 2005
Southern California Earthquake Center : Southern California Earthquake Center Consortium of 15 core institutions and 39 other participating organizations, founded as an NSF STC in 1991
Co-funded by NSF and USGS under the National Earthquake Hazards Reduction Program (NEHRP)
Mission:
Gather data on earthquakes in Southern California
Integrate information into a comprehensive, physics-based understanding of earthquake phenomena
Communicate understanding to end-users and the general public to increase earthquake awareness and reduce earthquake risk Core Institutions
University of Southern California (lead)
California Institute of Technology
Columbia University
Harvard University
Massachusetts Institute of Technology
San Diego State University
Stanford University
U.S. Geological Survey (3 offices)
University of California, Los Angeles
University of California, San Diego
University of California, Santa Barbara
University of Nevada, Reno
Participating Institutions
39 national and international universities and research organizations http://www.scec.org
Recent Earthquakes In California : Recent Earthquakes In California
Observed Areas of Strong Ground Motion : Observed Areas of Strong Ground Motion
Simulations Supplement Observed Data : Simulations Supplement Observed Data
SCEC/CME Project : SCEC/CME Project Goal: To develop a cyberinfrastructure that can support system-level earthquake science – the SCEC Community Modeling Environment (CME) Support: 5-yr project funded by the NSF/ITR program under the CISE and Geoscience Directorates
Start date: Oct 1, 2001 SCEC/ITR
Project NSF
CISE GEO SCEC
Institutions IRIS USGS ISI SDSC Information
Science Earth
Science www.scec.org/cme
SCEC/CME Scientific Workflow Construction : SCEC/CME Scientific Workflow Construction A major SCEC/CME objective is the ability to construct and run complex scientific workflow for SHA 9000 Hazard Curve files (9000 x 0.5 Mb = 4.5Gb) Extract
IMR
Value Plot
Hazard
Map Lat/Long/Amp
(xyz file) with 3000
datapoints (100Kb) Calculate
Hazard
Curves Gridded Region
Definition IMR Definition ERF Definition Probability of
Exceedence
and IMR
Definition GMT Map
Configuration
Parameters Define
Scenario
Earthquake Pathway 1 example
SCEC/CME Scientific Workflow System : SCEC/CME Scientific Workflow System
SCEC/CME SRB-based Digital Library : SCEC/CME SRB-based Digital Library
SRB-based Digital Library
More than 100 Terabytes of tape archive
4 Terabytes of on-line disk
5 Terabytes of disk cache for derivations
Slide10 : Component Library
Workflow
Template
Editor
(CAT) Workflow
Template (WT) Query for data given metadata L. Hearn @ UBC K. Olsen @ SDSU Execution requirements I/O data descriptions COMPONENTS J. Zechar @ USC (Teamwork: Geo + CS) Domain
Ontology Workflow
Library Metadata
Catalog Conceptual Data
Query Engine
(DataFinder) Data
Selection D. Okaya @ USC Query for WT Workflow
Instance (WI) Workflow
Mapping
(Pegasus) Executable
Workflow Grid information services Grid Query for components INTEGRATED WORKFLOW ARCHITECTURE Engineer Tools Tools
SCEC/CME HPC Allocations : SCEC/CME HPC Allocations SCEC/CME researchers have need and have access to significant High Performance Computing capabilities
TeraGrid Allocations (April 2005 – March 2006)
TG-MCA03S012 (Olsen) 1,020,000 SUs
TG-BCS050002S (Okaya) 145,000 Sus
USC HPCC Allocations
CME Group Allocations (Maechling) 100,000 SUs
Investigator Allocations (Li, Jordan) 300,000 SUs
SCEC Cluster
Dedicated Pentium 4 16 Processor Cluster (102 GFlops)
SCEC/CME TeraGrid Support : SCEC/CME TeraGrid Support TeraGrid Strategic Application Collaboration (SAC) greatly improved our AWM run-time on TeraGrid
Advanced TeraGrid Support (ATS) for TeraShake 2 and CyberShake simulations
SDSC Visualization Services support for SCEC simulations.
Three Types of Simulations : Three Types of Simulations SCEC/CME supports widely varying types of earthquake simulations
Each Simulation type creates it’s own set of middleware requirements
Will Describe three examples and comment on their middleware implications and on computational system requirements:
Probabilistic Seismic Hazard Maps
3D Waveform Propagation Simulations
3D Waveform-based Intensity Measure Relationship
Slide14 : Probabilistic Seismic Hazard Maps
Example Hazard Curve : Example Hazard Curve Site: USC ERF: Frankel-02
IMR: Field IMT: Peak Velocity
Time Period: 50 Years
Probabilistic Hazard Map Calculations : Probabilistic Hazard Map Calculations
Characteristic of PSHA Simulations : Characteristic of PSHA Simulations 10k Independent hazard curve calculations for each map calculations.
High throughput, not high performance, computing problem.
10k resulting files per map
Metadata saved for each file
Short run times on each calculation
Overhead of starting up job is expensive.
Would like to offer map calculations as service to SCEC users (who may not have an allocation)
Middleware Implications : Middleware Implications High throughput scheduling
Well Suited to Condor Pool
Bundling of short run-time jobs will reduce job startup overhead.
Bundling of jobs useful for clusters execution.
Metadata tracking with a RDBMS-based catalog system (e.g. Metadata Catalog System (MCS) and Replication Location Service (RLS)
Databases present installation and operational problems at ever site we request them
Software support for interpreted language on Computational Clusters
Implemented in an interpreted programming language
On-demand execution by non-allocated user
3D Wave Propagation Simulations : 3D Wave Propagation Simulations
Characteristics of 3D Wave Propagation Simulations : Characteristics of 3D Wave Propagation Simulations More physically realistic than existing PSHA but more computationally expensive.
High Performance Computing, cluster-based codes
4D data calculations (time varying volumetric data)
Output large volumetric data sets
Physics limited by resolution of grid. Higher ground motion frequencies require denser grid. Double of density increases storage by factor of 8.
Example: TeraShake Simulation : Example: TeraShake Simulation
Magnitude 7.7 earthquake on southern San Andreas
Mesh of ~2 billion cubes, dx=200 m
0.011 sec time step, 20,000 time steps: 3 minute simulation
Kinematic source (from Denali) from Cajon Creek to Bombay Beach
60 sec source duration
18,886 point sources, each 6,800 time steps in duration
240 processors at San Diego SuperComputer Center DataStar
~ 20,000 CPU hours, approximately 5 days wall clock
~ 50 Tbytes of output
During execution 'on-the-fly' graphics (…attempt aborted!)
Metadata capture and storage in the SCEC digital library
Domain Decomposition For TeraShake Simulations : Domain Decomposition For TeraShake Simulations
Simulations Supplement Observed Data : Simulations Supplement Observed Data
Peak Velocity : Peak Velocity NW-SE Rupture SE-NW rupture
Slide25 :
Montebello: 337 cm/s
Downtown: 52 cm/s
Long Beach: 48 cm/s
San Diego: 8 cm/s
Palm Springs: 36 cm/s Montebello: 8 cm/s
Downtown: 4 cm/s
Long Beach: 9 cm/s
San Diego: 6 cm/s
Palm Springs: 23 cm/s
SE-NW NW-SE
Break-down of output : Break-down of output
Middleware Implications for 3D Wave Propagation Simulations : Middleware Implications for 3D Wave Propagation Simulations Multi-day high performance runs
Check point restart support needed
Schedule reservations on clusters
Reservations and special queues are often arranged.
Large file and data movement
TeraByte transfers require high reliably, long term, data transfers
Ability to stop and restart
Can we move restart from one system to another
Draining of temporary storage during runs
Storage required for full often exceeds capability of scratch, so output files must be moved during simulation
Middleware Implications for 3D Wave Propagation Simulations : Middleware Implications for 3D Wave Propagation Simulations
On the fly visualization for rapid validation of results
Verify before full simulation is completed
Standard protocols for data transfers, and metadata registration into SRB-based storage
Slide29 : Waveform-based Intensity Measure Relationship (CyberShake)
Slide30 : Intensity-Measure Relationship
List of Supported IMTs
List of Site-Related Ind. Params IMT, IML(s) Site(s) Rupture
Attenuation Relationships
Simulation IMRs
exceed. prob. computed using a suite of synthetic seismograms Vector IMRs
compute joint prob. of exceeding multiple IMTs
(Bazzurro andamp; Cornell, 2002) Multi-Site IMRs
compute joint prob. of exceeding IML(s) at multiple sites
(e.g., Wesson andamp; Perkins, 2002) Various IMR types (subclasses) Gaussian dist. is assumed; mean and std. from various parameters
CyberShake Simulations Push Macro and Micro Computing : CyberShake Simulations Push Macro and Micro Computing CyberShake requires large forward wave propagation simulations, volumetric data storage
CyberShake requires 100k seismogram synthesis computations using multi-Terabyte volumetric data sets. During synthesis processing, this data needs to be disk-based.
100k of data files, and metadata, files to be managed
High throughput requirements are driving implementation toward TeraGrid wide computing approach.
High throughput requirements are driving integration of non-TeraGrid grids with TeraGrid
Example CyberShake Region (200km x 200km) : Example CyberShake Region (200km x 200km) USC: 34.05,-118.24
minLat=31.889,
minLon=-120.60,
maxLat=36.1858,
maxLon=-115.70
CyberShake Strain Green Tensor AWM : CyberShake Strain Green Tensor AWM Large (TeraShake Scale) forward calculations for each site.
SHA typically ignore rupture andgt; 200km from site, so this is used as cutoff distance.
20km buffer distance is used around edges of volume to reduce edge effects
65km depth to support frequencies of interest
Volume is 440km x 440km x 65km at 200m spacing
1.573 Billion mesh pts
Simulation time 240 seconds
Volumetric Data Saved for 2 horizontal simulations
Estimated Storage per site is: 7 TB (4.5 data 2.5 checkpoint files)
Ruptures in ERF within 200KM of USC : Ruptures in ERF within 200KM of USC 43227 Ruptures in Frankel02 ERF with M 5.0 or larger within 200km of USC
CyberShake Computational Elements : CyberShake Computational Elements
CyberShake Seismogram Synthesis : CyberShake Seismogram Synthesis Requires calculation of 100,000+ seismogram for each site.
Estimate Rupture Variations scale by magnitude:
Mw 5.0 x 1 = 20,450
Mw 6.0 x 10 = 216,990
Mw 7.0 x 100 = 106,900
Mw 8.0 x 1000 = 9,000
------------------
353,340 Ruptures
x 2 components
Current estimated number of seismogram files per site is 43,000 (due to combining components and variations into single file per rupture).
CyberShake Seismogram Synthesis : CyberShake Seismogram Synthesis Seismogram synthesis stage requires disk-based data storage of large volumetric data sets so tape based archive of volumetric data sets does not work.
To distribute seismogram synthesis across TeraGrid, we need to either duplicate TB of data, or have global visibility on disks systems
Example Hazard Curve : Example Hazard Curve Site: USC ERF: Frankel-02
IMR: Field IMT: Peak Velocity
Time Period: 50 Years
Workflows run Using Grid VDS Workflow Tools : Workflows run Using Grid VDS Workflow Tools
Examples Hazard Map Region (50km x 50km at 2km grid spacing = 625 sites) : Examples Hazard Map Region (50km x 50km at 2km grid spacing = 625 sites) OpenSHA SA 1.0 Frankel 2002 ERF and Sadigh with 10% POE in 50 years.
Summary of SCEC Experiences : Summary of SCEC Experiences As soon as we develop a computational capability, the geophysicists develop application that push the technology.
Compute technology, data management technology, resource sharing technology all are applied.
In many ways, IT capabilities required for geophysical problems exceed what is currently possible and limit the state of knowledge in geophysics and public safety.
For example, higher frequency simulations, are of significant interest, but exceed computational and storage capabilities currently available.
Major Middleware related issues for SCEC/CMESecurity and Allocation Management : Major Middleware related issues for SCEC/CME Security and Allocation Management No widely accepted CA makes adding organizations to SCEC grid problematic.
Ability to run under group allocations for 'on demand' requests. (Community Allocation ?)
Major Middleware related issues for SCEC/CMESoftware Installation and Maintenance : Major Middleware related issues for SCEC/CME Software Installation and Maintenance
Middleware software stack, even at supercomputer systems, support should include micro jobs support such as Java.
Database management support for database-oriented tools such as Metadata Catalogs are important (backup, recovery, cleanup, performance, modifications)
Guidelines for tools in middleware software stack, should describe when local installations are required and when remote installations are acceptable for tools such as RLS and MCS
Major Middleware related issues for SCEC/CMESupercomputing and Storage : Major Middleware related issues for SCEC/CME Supercomputing and Storage Globally (TeraGrid – wide) visible disk storage
Well supported, reliable file transfers with monitoring and restart of jobs with problems are essential.
Interoperability between grid tools and data management tools such as SRB must include data and metadata and metadata search.
Major Middleware related issues for SCEC/CMEScheduling Issues : Major Middleware related issues for SCEC/CME Scheduling Issues Support for Reservation-based scheduling
Partial run and restart capability
Failure detection and alerting
Major Middleware related issues for SCEC/CMEUsability Related and Monitoring : Major Middleware related issues for SCEC/CME Usability Related and Monitoring Monitoring tools that include status of available storage resources.
On-the-fly visualizations for run-time validation of results
Interfaces to workflow systems are complex, developer oriented interfaces. Easier to user interfaces needed