Bentley 2004

Registries and metadata-driven searches in a solar-terrestrial grid context: 

Registries and metadata-driven searches in a solar-terrestrial grid context Rob Bentley (UCL/MSSL) and the EGSO Team 27-29 October 2004, Greenbelt MD VOs in Space and Solar Physics Workshop

Outline: 

Outline Overview of EGSO Outline of the query requirements How metadata is used within the system The EGSO catalogues and Registries Example query

EGSO – European Grid of Solar Observations: 

EGSO – European Grid of Solar Observations EGSO is a Grid test-bed related to a particular application Designed to improve access to solar data for the solar physics and other communities Addresses the generic problem of a distributed heterogeneous data set and a scattered user community Funded under the Information Society Technologies (IST) thematic priority of the EC’s Fifth Framework Program (FP5) Started March 2002; duration of 36 months (or so) Involves 12+ groups in Europe and the US, led by UCL-MSSL 4 in UK, 3 in France, 2+ in Italy, 1 in Switzerland, 2 in US Several associate partners, mainly in the US Objectives include: Building enhanced search capability for solar data Support of user community scattered around the world Provide access to data centres & observatories around the world Where possible, provide ability to process data at source

The Solar Virtual Observatory family: 

The Solar Virtual Observatory family Partners and collaborators provide expertise in solar physics and IT UK UCL-MSSL & UCL-CS, RAL, University of Bradford France IAS (Orsay), Obs. de Paris-Meudon, International Space Univ. (Strasbourg) Italy Istituto Nazionale di Astrofisica, Politecnico di Torino INAf includes Obs. of Turin, Trieste, Florence and Naples Switzerland University of Applied Sciences (Aargau) US SDAC at NASA-GSFC, National Solar Observatory SDAC and NSO are also part of the US VSO Belgium Royal Observatory of Belgium Netherlands ESA-ESTEC – Solar Group US VSO: Stanford University, Montana State University CoSEC: Lockheed-Martin VSPO: LEP at NASA-GSFC (Lab. Extraterrestrial Physics)

Accessible Providers: 

Accessible Providers Extracted TRACE data available to EGSO through CoSEC; need to improve links to VSO “sourced” datasets (e.g for SHA, HAO/MLSO, etc.) Want to add access to space plasma data through VHO, VSPO, etc. Planning addition of BASS2000 (France), SolarNet (Italy), plus other optical & radio ground-based observatories in Europe and US

Use of solar observations: 

Use of solar observations The appearance of the Sun changes dramatically with wavelength Emissions originate from different layers in the atmosphere and different physical phenomena For a complete picture we need to use as wide a range of observations as possible Mixture of multi-wavelength observations from space- and ground-based platforms Identifying observations that match some User search criteria and then retrieving them are major problems

Linking into the wider context: 

Linking into the wider context Increasing desire to use solar data in study problems that span communities Space weather heliosphere, magnetosphere, ionosphere… Climate physics Planetary physics Astrophysics Need to find ways of tying these data together Single data model covering all solar system not practical Intersecting data models in a general pool should be possible EGSO trying to achieve interoperability (at some level) with the space plasma community Main purpose of bringing the VOs together…

Linking into the wider context: 

Linking into the wider context Increasing desire to use solar data in study problems that span communities Space weather heliosphere, magnetosphere, ionosphere… Climate physics Planetary physics Astrophysics Need to find ways of tying these data together Single data model covering all solar system not practical Intersecting data models in a general pool should be possible EGSO trying to achieve interoperability (at some level) with the space plasma community Main purpose of bringing the VOs together…

Generic Query: 

Generic Query Identify suitable observations (many serendipitous) Want to access as many different types of data as are available Identification should be possible without accessing the data Existing cataloguing differs in quality, contents, and dependencies Data volumes are increasing rapidly - SDO will produce 2 TB/day User only wants to know if data addressing a problem exists Locate the data Data scattered, with differing means of access (some proprietary) Large and small data providers, with varying resources Process the data Involves extraction and calibration of a subset of raw data Often only need a subset of each data set Uses code defined by instrument teams (SolarSoft, C…) Return results to the User Compare results from different instruments SolarSoft (IDL) provides a standard platform for analysis Note exchange in order of 3rd and 4th bullets in the Grid expression of the problem

The EGSO Search Engine: 

The EGSO Search Engine Enhanced cataloguing describes the data more fully Standardized versions of observing catalogues (UOC) tie together the heterogeneous data sets Search Registry is an abstraction of entries in the UOC and allows narrowing of the search in initial stages New types of catalogue allow searches on events, features and phenomena, not just date & time, pointing, etc… Solar Event Catalogue (SEC) - derived from published lists Solar Feature Catalogue (SFC) - generated by feature recognition Ancillary data used to provide additional search criteria QLK Server provides Phone book access to images, time-series, derived products, etc.; can also do limited processing DSO Server gives Yellow Page information on instruments, etc. Similar hierarchical cataloguing required in other data Grid projects EGSO is improving the quality and availability of metadata

Catalogue relationships: 

Catalogue relationships Objective of the improved metadata, etc. is to be able to pose questions like: Identify events when a filament eruption occurred within 30° of the north-west limb and there were good observations in H, EUV and soft X-rays

Simplified Architecture: 

Simplified Architecture SEC, SFC, UOC, DSO, QLK, CoSEC Archive access can be by FTP, HTTP, Web Services, cgi-bin… through adaptor modules Brokers manage the metadata and decides and allocates resources Consumer supports GUI and API access

The UOC and Search Registry: 

The UOC and Search Registry Unified Observing Catalogues (UOC) Unified (metadata) form of observing catalogues used to tie together the heterogeneous data, leaving the data unchanged Increase interoperability by expressing coordinates in “standard” formats Self describing, quantized by time and instrument, with no dependencies on ancillary data or proprietary software (and with any errors corrected) Standards defined for future data sets (e.g. STEREO, ILWS, Solar-B) Search Registry is an abstraction of entries in the UOC Registry allows the Broker to identify instruments that: have data properties matching the search - Static SR probably have observations during search time interval - Dynamic SR Reduces need to interact with Data Providers that are unlikely to have data matching the search Static Search Registry (sSR) is able to support access to different types of data from solar and heliospheric observations Tries to describe instrument capabilities & observing objectives in common way Location of observation platform more important as include space plasma data, and for some future solar missions First step in search - later steps can be dealt with by other VOs EGSO sSR includes instruments on Ulysses, ACE, Cassini, SDO, STEREO… The reasoning behind the UOC is universal, allows observations to be described in more interoperable way Significant similarities in the way we use data means that the static Search Registry can be used to tie solar and heliospheric data together

Slide14: 

Part of the EGSO Data Model

Contents of the Static Search Registry: 

Contents of the Static Search Registry Instrument and Observatory Includes space and ground based, solar and heliospheric Observing Domain What trying to observe Solar disk, interior, corona, heliosphere, magnetosphere Observable entity Photons, particles, fields with sub-divisions Common terms Imager, spectrometer, polarimeter, coronagraph Oscillations, waves, H-alpha, composition, irradiance… Information related to location of observatories and operating interval of the observatories & instruments in separate table

Finding the data: 

Finding the data Data could be located anywhere in the world User only needs to know observations exist, not where located System should isolate the user from the intricacies of access System should be able to optimize use of sources Handling of replicated data and aggregated sources Choice of source - most capable, closest, least used, etc. Must respect any data use policies (proprietary data) and ensure integrity of data providers Burden on data providers minimized to encourage participation In EGSO, data sources are interfaced by the Provider Role Information about the data sources is held in the Data Registry - this is managed by the Broker role Which instrument data sets are hosted by each data archive Which data archives interfaced to each Provider Role Provider Role uses adaptor modules used to handle different access protocols Standardizes way data source interface appears to the system and simplifies addition of new data sources Also allows access to data “hosted” by other VOs

EGSO GUI: 

EGSO GUI GUI supports Event Drive and Date Driven queries Others being added Series of portlets allow user to tailor their search depending on the thrust of their query Wavelengths, Instruments… Advanced static Search Registry currently being deployed API will access similar capabilities

EGSO GUI: 

EGSO GUI GUI supports Event Drive and Date Driven queries Others being added Event Drive search returns list of events the used can examine and then select A list of instruments that made observations is returned After instruments are selected, list of files returned

EGSO GUI: 

EGSO GUI GUI supports Event Drive and Date Driven queries Others being added Event Drive search returns list of events the used can examine and then select A list of instruments that made observations is returned After instruments are selected, list of files returned

EGSO GUI: 

EGSO GUI GUI supports Event Drive and Date Driven queries Others being added Event Drive search returns list of events the used can examine and then select A list of instruments that made observations is returned After instruments are selected, list of files returned

Query Work Flow: 

Query Work Flow User specifies query through the GUI or API Static Search Registry narrows the search, based on the criteria specified in the query Identifies instruments that make the desired type of observations Search can includes solar, heliospheric, etc… instruments Dynamic Search Registry determines (at some granularity) which were actually observing Includes pointing, observatory location… User returned list of identified instruments to refine selection Data Registry used to locate archive holding the instrument data and make data request List of files returned that can be retrieved, used to generate data products, etc. Convert to processed products if required

Summary: 

Summary Although mainly presented EGSO approach, structure allows cross-coupling with other VOs Layered metadata provides several entry points Commonality with other VOs differ in detail and need Already exchange of information and resources between the solar VO projects Approaches of the solar VOs very complementary Need to better link into the space plasma, etc. VOs Main objective of this meeting is to find common ground in the way we describe and handle data to facilitate interoperability

Useful URLs: 

Useful URLs General information about EGSO can be found under URL: http://www.egso.org The different components of the EGSO system, including the main entry portal, can be accessed from URL: http://www.egso.org/demo

So, where are we?: 

So, where are we? Release 4a of EGSO recently become available People welcome to try this… Flexible GUI using selectable portlets has been deployed User able to conduct date driven and event driven searches Event driven search accesses EGSO Solar Event Catalogue System searches for datasets that match search criteria User able to make selections at each stage of the search with aid of supporting data Search Registry (SR) is still being developed Simplified version currently installed in GUI More complex version in preparation (Release 4b) Fully functional Search Registry will allow comprehensive selection of the types of instrument, data, region observed, etc Interoperable with the STP & heliospheric observations Support for refining selection of data for time and spatial coordinates using cursors will be added shortly

So, where are we? (cont…): 

So, where are we? (cont…) Special Servers are becoming operational SEC Server is fully integrated into EGSO Web Service accessible relational database - SQL in, VOTable out Several lists are already included: Flares, particle events, CMEs… Hope to add many more in the future DSO and SFC Servers functional, but integration not yet complete Both available as Web interfaces; Web Service interfaces planned Over 3 years of data processed for Solar Feature Catalogue QLK Server is still under development Processing using CoSEC partially integrated GOES X-ray & energetic particle lightcurves generated for GUI In process of adding ability to: generate quicklook images & movies extract and process certain datasets create composite plots for use in searches and publications Several popular data sources have already been integrated Space-based: Yohkoh, SOHO & RHESSI Ground-based: NSO & Global H-alpha Network (GHAN) Planning to add TRACE & numerous GBO sources in near future