GT 3 Base Services Inside- Information Services: GT 3 Base Services Inside - Information Services KISTI Supercomputing Center Grid Technology Research Department Eunsung Kim eskim74@kisti.re.kr


Contents: Contents Overview Components Service Data Provider Components Service Data Aggregation Components Registry Components Applications GT3 Index Service GRAM RIPS MMJFS ContainerRegistry and VORegistry


Overview: Overview


GT3 Information Services: GT3 Information Services What is GT3 Information Services? Grid service which provides information about Grid resources modular Java component framework for OGSA service developers can use to implement various information management solutions for GT3-compatible OGSA Services and Service Data


GT2 vs. GT3 Information Services: GT2 vs. GT3 Information Services Components MDS  Index Service, Service Data Providers and Aggregators, Query and Notification Framework GRAM Reporter  Resource Information Provider Service Data Format LDIF  XML Data Source GLUE providers  GLUE providers, Service Instance Query Mechanisms LDAP  XPath, XQuery


Service Data: Service Data A Grid service instance maintains a set of service data elements (SDE) Declared via an extended XSD element declaration, placed in a WSDL portType Values carried in a portType or in an instance Includes basic introspection information, interface-specific data, and application state Pull and push models for information query GridService::FindServiceData operation Pull: queries this information via extensible query language NotificationSource::Subscribe Push: Subscribe to notification of changes to information


Why Service Data?: Why Service Data? Discovery often requires instance-specific, perhaps dynamic information Service data offers a general solution Every service must support some common service data, and may support any additional service data desired Not just meta-data, but also instance state Part of GT MDS-2 model contained in OGSI Defines standard data model, and query op Complements soft-state registration and notification


OGSI ServiceData Model: OGSI ServiceData Model ServiceData for self-description Model service with properties! Fine-grained view of resource functionality Data scoped by service instance Domain-dependent state Service discovery/monitoring information Stateful properties of service, e.g. cpu-load Infrastructure state Introspection for client 'tooling', e.g. handles Parameters of generic features ServiceDataNames introspects available data


Components: Components


Basic Components: Basic Components Service Data Provider Components Service Data Aggregation Components Registry Components


Service Data Provider Components: Service Data Provider Components Service Data Provider components provide a standard mechanism for dynamic generation of service data via external programs External provider programs can be the core providers that are part of GT3 or can be user-created, custom providers


In Detail: Service Data Providers: In Detail: Service Data Providers Service Data Provider interfaces are designed to support execution in either synchronous ('pull') mode asynchronous ('push') mode A valid provider is composed of any Java class which implements at least one of three predefined Java Interfaces generates a compatible form of XML output as the result of its execution


In Detail: Service Data Providers: In Detail: Service Data Providers Provider Interfaces SimpleDataProvider synchronous provider which produces XML output in the form of a Java OutputStream DOMDataProvider synchronous extension of SimpleDataProvider which can also produce XML output in the form of a Java org.w3c.dom.Document AsyncDataProvider asynchronous version of SimpleDataProvider sending the output to the specified callback Object, which is assumed that the provider implementer and the provider caller have both agreed on the callback interface at compile-time


In Detail: Provider Interfaces: In Detail: Provider Interfaces SimpleDataProvider Basic interface which all service data providers must implement


In Detail: Provider Interfaces: In Detail: Provider Interfaces DOMDataProvider Generic interface for XML service data providers that are capable of emitting a org.w3c.dom.Document object at runtime


In Detail: Provider Interfaces: In Detail: Provider Interfaces AsyncDataProvider Asynchronous version of provider interface


In Detail: GT3 Providers: In Detail: GT3 Providers AsyncDocumentProvider An asynchronous version of a generic XML document provider ScriptExecutionProvider ServiceDataProvider that provides a generic way to execute scripts which produce XML documents HostScriptProvider Constructs Host service data from the output of multiple scripts


In Detail: GT3 Providers: In Detail: GT3 Providers ForkInfoProvider ServiceDataProvider which monitors local system PIDs PBSInfoProvider ServiceDataProvider which queries PBS for queue information SimpleSystemInformationProvider Basic MDS GRIS-sytle sensor which emits system information in XML, with state managed directly as an XML Document using JDOM - JDK 1.3 compatible


In Detail: Provider Manager: In Detail: Provider Manager Provider execution is handled by the ServiceDataProviderManager class, which schedules and manages provider execution as Java TimerTasks ServiceDataProviderManager uses an XML-based configuration file to load and link installed Service Data Providers during runtime through standard Java reflection methods Configuration file $GLOBUS_LOCATION/etc/indexservice.providers $GLOBUS_LOCATION/etc/rips.providers


In Detail: Provider Manager: In Detail: Provider Manager Configuration entry for the provider in a configuration file enables your provider for execution by the Provider Manager publishes the existence of your provider to clients Required attribute in the configuration entry the 'class' attribute, which is simply the fully qualified Java class name


In Detail: Custom Data Handlers: In Detail: Custom Data Handlers The default data processing behavior of the Provider Manager take the logical XML document result of a provider’s execution wrap it in a new SDE and then add it to the Service’s ServiceDataSet We can override the default data processing logic in the Provider Manager by specifying the 'handler' attribute in the Provider’s configuration file


In Detail: Custom Data Handlers: In Detail: Custom Data Handlers Custom callback routine provides a mechanism for facilitating sorting, merging or other data preprocessing and transformation allows the developer complete control for output data Use case: GRAM RIPS uses a custom callback handler for the Fork and PBS providers to search, sort and atomize child 'Job' elements from the logical result document into new SDEs


In Detail: ProviderExecutionPortType: In Detail: ProviderExecutionPortType enumProviders


In Detail: ProviderExecutionPortType: In Detail: ProviderExecutionPortType executeProvider


In Detail: Mechanisms: In Detail: Mechanisms Provider Manager SimpleDataProvider DomDataProvider AsyncDataProvider Information Providers Custom Data Handler enumProvider executeProvider Service or User


Basic Components: Basic Components Service Data Provider Components Service Data Aggregation Components Registry Components


Service Data Aggregation Components: Service Data Aggregation Components ServiceDataAggregator components provide a reusable mechanism for handling subscription, notification, and updating of locally stored copies of service data which is generated by other services By using the ServiceDataAggregator class in your service code service Data from both locally executing information providers and other OGSA service instances can be aggregated into any given service


In Detail: Service Data Aggregator: In Detail: Service Data Aggregator ServiceDataAggregator component is used to perform server-side notification subscription management similar to the SinkListener sample Key additional feature of ServiceDataAggregator notification data that is processed by the deliverNotification() function is actually copied and stored locally as a SDE, which includes creation timestamp, TTL and source metadata


In Detail: Service Data Aggregator: In Detail: Service Data Aggregator Aggregated SDEs are organized by SDE QName stored in a array which is returned as a org.gridforum.ogsa.ServiceDataSetType to FindServiceData name queries Originator field (GSH type) of the OGSA ServiceDataType is used as the 'primary key' to differentiate like-named entries from each other to identify the source of the data itself


In Detail: AggregatorPortType: In Detail: AggregatorPortType addSubscription and removeSubscription


In Detail: Mechanisms: In Detail: Mechanisms Aggregator Grid Services addSubscription removeSubscription Service or User … deliverNotification


Basic Components: Basic Components Service Data Provider Components Service Data Aggregation Components Registry Components


Registry Components: Registry Components Registry components maintain a set of available peer Grid Service Handles provides soft-state cataloging of a set of Grid Services i.e., the registry of services is periodically updated with existence notification messages and any existing entries which fail to refresh within the timeout period are eventually expired Registries can be used to support query or other operations that may apply to one or more services in a set


In Detail: RegistrationPortType: In Detail: RegistrationPortType registerService and unregisterService


In Detail: Mechanisms: In Detail: Mechanisms Registry Grid Services registerService unregisterService …


Applications: Applications


Applications: Putting it All Together: Applications: Putting it All Together Various GT3 Information Services components provide a convenient way to address common and recurring tasks with Service Data in the GT3 OGSA environment One or all of the components can be used to manage data depending on application needs


Applications: Putting it All Together: Applications: Putting it All Together Current GT3 OGSA services using Information Services components GT3 Index Service ServiceDataProviderManager, ServiceDataAggregator, and Registry components GRAM Resource Information Provider Service (RIPS) ServiceDataProviderManager components GRAM Master Managed Job Factory service (MMJFS) ServiceDataAggregator components ContainerRegistry and VORegistry Registry components


Use Case: GT3 Index Service : Use Case: GT3 Index Service GT3 Index Service combines the ServiceDataProviderManager components along with the ServiceDataAggregator and Registry components creates a dynamic data-generating and indexing node, similar in concept to GT2 MDS GRIS/GIIS is implemented as a ServiceSkeleton so that developers can easily customize and extend its functionality Registry and ServiceDataAggregator components can interconnect multiple GT3 Index Services together into arbitrarily complex topologies to form both Virtual Organizations and Virtual ServiceDataSets


Use Case: GT3 Index Service: Use Case: GT3 Index Service Key capabilities An interface for connecting external information provider programs to service instances A generic framework for aggregation of service data A Registry of Grid services Java package: org.globus.ogsa.impl.base.index


Use Case: GT3 Index Service: Use Case: GT3 Index Service Aggregator Mechanism (caching here) Provider Mechanism Java Provider Grid Service Grid Service Grid Service Index Service User Service Data Service Data Service Data Notification Message Existence Notification Message Service Data Service Data Collective Layer Resource Layer Registry Mechanism GSH


Use Case: GRAM RIPS: Use Case: GRAM RIPS GRAM Resource Information Provider service uses the ServiceDataProviderManager components, along with custom providers and the OGSA core component NotificationServiceSkeleton monitors forked processes, PBS scheduler queues, and local host system statistics polls the host system periodically for this information since changes to the local hosting environment are inherently non-deterministic Java package: org.globus.ogsa.impl.base.gram.rips


Use Case: GRAM RIPS: Use Case: GRAM RIPS Execution frequency of this polling is specified on a per-provider-instance basis is specified by the client when the provider is executed the first time via the ProviderExecutionPortType executeProvider() method Sweeper thread periodically traverses the service’s internal SDE list and invalidates any entries which have not received updates to the timestamp metadata


Use Case: GRAM RIPS: Use Case: GRAM RIPS Special processing in its custom 'jobDataHandler' function breaking out specific child ‘Job’ elements from the logical result document creating individual SDEs for each, assigning appropriate TTL metadata to new entries and updating the TTL data on existing SDEs In order to detect whether a notification is warranted based on a state change examines the state of each Job element during the callback


Use Case: GRAM RIPS: Use Case: GRAM RIPS RIPS tuning global configuration parameters that can be set in the RIPS deployment descriptor in server-config.wsdd


Use Case: GRAM RIPS: Use Case: GRAM RIPS RIPS tuning (cont.) The 'sweeperDelay' value is the amount of time in milliseconds for the Sweeper thread to wait between execution cycles should generally be at least as long as the average amount of time spent in the 'jobDataHandler' callback The 'gracePeriod' is the amount of time in milliseconds to add to the entries’ GoodUntil TTL value to generate its AvailableUntil value If the sweeper encounters an entry whose AvailableUntil value is expired, it deletes the entry The GoodUntil TTL value is set by default to be the provider execution frequency in milliseconds


Use Case: MMJFS: Use Case: MMJFS GRAM Master Managed Job Factory service exposes aggregated service data to GRAM clients by using the ServiceDataAggregator to subscribe to one or more RIPS installations for information on running jobs and queue status Java package: org.globus.ogsa.impl.base.gram.mmjfs


Use Case:ContainerRegistry and VORegistry: Use Case: ContainerRegistry and VORegistry ContainerRegistryImpl This class implements a registry for all services in a container Java package: org.globus.ogsa.impl.base.registry VORegistryImpl This registry implements a repository of services within a virtual orginization It supports local as well as remote service publication and lookup Java package: org.globus.ogsa.impl.samples.registry


Q & A: Q andamp; A Thanks for Coming!