W20 Levan Denenberg2

Interoperability Standards & Searching Multiple Repositories: 

Interoperability Standards andamp; Searching Multiple Repositories Ralph LeVan/OCLC Ray Denenberg/Library of Congress

The Problem: 

The Problem How do I provide a common interface for my users? How do I combine results from multiple sources?

How do I provide a common interface for my users?: 

How do I provide a common interface for my users? How do I convert my queries into the Content Provider’s (CP’s) queries? How do I ask for 10 records? How do I ask for more records? How do I interpret their response?

How do I convert my queries into the CP’s queries?: 

How do I convert my queries into the CP’s queries? My user said 'author=twain and title=huck finn' Google expects: +twain +'huck finn' Z39.50: twain/1=1003;4=2 'huck finn'/1=4;4=1 and Lucene: creator:twain and titlePhrase:'huck finn'

How do I ask for 10 records?: 

How do I ask for 10 records? Amazon won’t let you RedLightGreen: MAXRECORDS=n British Library: records=n

How do I ask for more records?: 

How do I ask for more records? Amazon: page=n RedLightGreen: STARTINDEX=n British Library: start=n

How do I interpret their response?: 

How do I interpret their response? How many records did I retrieve? Did something go wrong? How do I convert the CP’s records into something my users will recognize?

How many records did I retrieve?: 

How many records did I retrieve? Amazon: andlt;a href='/gp/search/ref=sr_nr_i_0/002-2019116-8269663?%5Fencoding=UTF8andamp;keywords=pratchettandamp;rh=i%3Aaps%2Ck%3Apratchett%2Ci%3Astripbooksandamp;page=1'andgt;Booksandlt;/aandgt;andlt;span class='narrowValue'andgt;andamp;nbsp;(334)andlt;/spanandgt; RedLightGreen: andlt;bandgt;Viewing:andlt;/bandgt; 1-10 of 239 results British Library andlt;opensearch:totalResultsandgt;190andlt;/opensearch:totalResultsandgt;

Did Something Go Wrong?: 

Did Something Go Wrong? RedLightGreen: andlt;span class=smallTextandgt;We didn't find any matches for andlt;bandgt;dog andandlt;/bandgt;.andlt;/spanandgt; British Library: andlt;item andgt; andlt;title andgt;Nothing found due to an errorandlt;/titleandgt; andlt;description andgt;Too many hits. Refine your request.andlt;/descriptionandgt;andlt;/itemandgt;

How do I convert the records?: 

How do I convert the records? Amazon: andlt;table class='searchresults' border='0' width='100%' cellpadding='0' cellspacing='0'andgt; andlt;trandgt;andlt;td width='100%' class='searchitem' id='Td:0'andgt; andlt;table border='0' width='100%' cellpadding='0' cellspacing='0'andgt;andlt;tr valign='top'andgt; andlt;tdandgt; andlt;table class='n2' border='0' cellpadding='0' cellspacing='0'andgt; andlt;trandgt; andlt;td class='imageColumn' width='88'andgt;andlt;table border='0' cellpadding='0' cellspacing='0'andgt; andlt;trandgt;andlt;td align='center' width='80'andgt; andlt;a href='http://www.amazon.com/gp/product/0060815221/sr=8-1/qid=1142436987/ref=pd_bbs_1/002-2019116-8269663?%5Fencoding=UTF8'andgt;andlt;img src='http://ec1.images-amazon.com/images/P/0060815221.01._PIsitb-st-arrow,TopLeft,-1,-14_SCTHUMBZZZ_.jpg' width='55' alt='Thud! (Discworld, Book 32)' height='82' border='0' /andgt;andlt;/aandgt; andlt;/tdandgt;andlt;td width='8'andgt;andlt;/tdandgt;andlt;/trandgt;andlt;/tableandgt;andlt;/tdandgt; andlt;td class='dataColumn'andgt;andlt;table cellpadding='0' cellspacing='0' border='0'andgt;andlt;trandgt;andlt;tdandgt; andlt;a href='http://www.amazon.com/gp/product/0060815221/sr=8-1/qid=1142436987/ref=pd_bbs_1/002-2019116-8269663?%5Fencoding=UTF8'andgt;andlt;span class='srTitle'andgt;Thud! (Discworld, Book 32)andlt;/spanandgt;andlt;/aandgt; by Terry Pratchett (andlt;span class='binding'andgt;Hardcoverandlt;/spanandgt; - Sep 13, 2005)andlt;/tdandgt;andlt;/trandgt; andlt;trandgt;andlt;td class='brandLink'andgt;andlt;span class='aliasName'andgt;Books:andlt;/spanandgt; andlt;a href='/gp/search/ref=sr_nr_seeall_1/002-2019116-8269663?%5Fencoding=UTF8andamp;keywords=pratchettandamp;rh=i%3Aaps%2Ck%3Apratchett%2Ci%3Astripbooks'andgt;See all 334 itemsandlt;/aandgt;andlt;/tdandgt;andlt;/trandgt; andlt;trandgt;andlt;tdandgt;andlt;span class='priceType'andgt;andlt;a href='http://www.amazon.com/gp/product/0060815221/sr=8-1/qid=1142436987/ref=pd_bbs_1/002-2019116-8269663?%5Fencoding=UTF8'andgt;Buy newandlt;/aandgt;: andlt;/spanandgt;andamp;nbsp;andlt;span class='listprice'andgt;$24.95andlt;/spanandgt; andlt;span class='saleprice'andgt;$15.72andlt;/spanandgt; andamp;nbsp; andlt;span class='priceType'andgt; andlt;a href='http://www.amazon.com/gp/offer-listing/0060815221/sr=8-1/qid=1142436987/ref=pd_bbs_1/002-2019116-8269663?%5Fencoding=UTF8'andgt;Used andamp;amp; newandlt;/aandgt; andlt;/spanandgt; from andlt;span class='otherprice'andgt;$3.76andlt;/spanandgt; andamp;nbsp; andlt;span class='avail'andgt;Usually ships in 24 hoursandlt;/spanandgt; andlt;/tdandgt;andlt;/trandgt;andlt;trandgt;andlt;td colspan='2'andgt;andlt;table cellpadding='0' cellspacing='0' border='0'andgt; andlt;trandgt;andlt;td class='excerptStart'andgt;andlt;span class='excerptLead'andgt;Excerpt fromandlt;/spanandgt; andlt;a href='/gp/reader/0060815221/ref=sib_aps_pg/002-2019116-8269663?%5Fencoding=UTF8andamp;keywords=pratchettandamp;p=S00Eandamp;checkSum=y3glB4NEGJ6Ql3iAWFd6teZptAJmys3Uu8CCW9387%252BA%253D'andgt;page 2andlt;/aandgt;: andamp;quot;andlt;span class='excerpt'andgt;... Terry andlt;bandgt;Pratchettandlt;/bandgt; 'Most of the news is ...andlt;/spanandgt;andamp;quot;andlt;/tdandgt;andlt;/trandgt; andlt;trandgt;andlt;td class='excerptSeeMore'andgt;andlt;a href='/gp/reader/0060815221/ref=sib_aps_ref/002-2019116-8269663?%5Fencoding=UTF8andamp;keywords=pratchettandamp;v=search-inside'andgt;See more referencesandlt;/aandgt; to andlt;span class='excerptUserInput'andgt;pratchettandlt;/spanandgt; in this book.andlt;/tdandgt;andlt;/trandgt;andlt;trandgt;andlt;td style='padding-top: 5px; padding-bottom: 8px;'andgt;andlt;span style='font-weight: bold; color: #339933;'andgt;Surprise me!andlt;/spanandgt; andlt;a href='http://www.amazon.com/gp/reader/0060815221/ref=sib_aps_sup/002-2019116-8269663?%5Fencoding=UTF8andamp;p=random'andgt;See a random pageandlt;/aandgt; in this book.andlt;/tdandgt;andlt;/trandgt;andlt;/tableandgt;andlt;/tdandgt;andlt;/trandgt; andlt;/tableandgt;andlt;/tdandgt;andlt;/trandgt;andlt;/tableandgt; andlt;/tdandgt;andlt;/trandgt;andlt;/tableandgt;andlt;/tdandgt; andlt;/trandgt;

Converting Records Cont.: 

Converting Records Cont. RedLightGreen: andlt;td class='highlightcell'andgt;andlt;span class='titleText'andgt;andlt;bandgt;andlt;a title='View more information about this title.' href='ucw.servlets.UCWController?ACTION=EDITIONandamp;amp;WORKID=21537371andamp;amp;LANGUAGE=ENGandamp;amp;MATERIAL=booksandamp;amp;FROMRSLT=3andamp;amp;FROMWORK=1andamp;amp;lang=english'andgt;Hogfatherandlt;/aandgt;andlt;/bandgt;, by Terry Pratchett andlt;brandgt;3 editions published between 1996 and 1998 in English.andlt;brandgt;Primary Subject: Discworld Imaginary Place - Fictionandlt;brandgt;andlt;img src='/ucwprod/web/images/green.gif' height='3' width='10' alt='A title's position in a search result is based on relevancy (how closely your search terms match the description) andamp;#xA;and availability (how many libraries have a copy of the title).'/andgt;andlt;img src='/ucwprod/web/images/white.gif' height='3' width='1'/andgt;andlt;img src='/ucwprod/web/images/green.gif' height='3' width='10' alt='A title's position in a search result is based on relevancy (how closely your search terms match the description) andamp;#xA;and availability (how many libraries have a copy of the title).'/andgt;andlt;img src='/ucwprod/web/images/white.gif' height='3' width='1'/andgt;andlt;img src='/ucwprod/web/images/green.gif' height='3' width='10' alt='A title's position in a search result is based on relevancy (how closely your search terms match the description) andamp;#xA;and availability (how many libraries have a copy of the title).'/andgt;andlt;img src='/ucwprod/web/images/white.gif' height='3' width='1'/andgt;andlt;img src='/ucwprod/web/images/green.gif' height='3' width='10' alt='A title's position in a search result is based on relevancy (how closely your search terms match the description) andamp;#xA;and availability (how many libraries have a copy of the title).'/andgt;andlt;img src='/ucwprod/web/images/white.gif' height='3' width='1'/andgt;andlt;img src='/ucwprod/web/images/gray.gif' height='3' width='10' alt='A title's position in a search result is based on relevancy (how closely your search terms match the description) andamp;#xA;and availability (how many libraries have a copy of the title).'/andgt;andlt;img src='/ucwprod/web/images/white.gif' height='3' width='1'/andgt;andlt;/spanandgt;andlt;/tdandgt;andlt;/trandgt;andlt;/tableandgt;andlt;table xmlns='http://www.w3.org/TR/REC-html40' border='0' cellpadding='0' cellspacing='0' width='100%'andgt;andlt;trandgt;andlt;td class='recordsepcell' colspan='2'andgt;andlt;img src='/ucwprod/web/images/clear.gif' height='1'/andgt;andlt;/tdandgt;andlt;/trandgt;andlt;/tableandgt;andlt;table xmlns='http://www.w3.org/TR/REC-html40' border='0' cellpadding='3' cellspacing='0' width='100%'andgt;andlt;tr valign='top'andgt;andlt;td width='25' align='right' class='highlightcell'andgt;andlt;span class='titleText'andgt;2.andlt;/spanandgt;andlt;/tdandgt;

Converting Records Cont.: 

Converting Records Cont. British Library: andlt;item andgt;andlt;title andgt;Thud! / Terry Pratchett.andlt;/titleandgt; andlt;link andgt;http://catalogue.bl.uk/F/-?func=direct-doc-setandamp;doc_number=013220851andamp;l_base=BLL01andamp;from=A9OpenSearchandlt;/linkandgt; andlt;description andgt; Pratchett, Terry. ; London : Doubleday, 2005. . ISBN 0385608675 (hbk.) : £17.99 . (Added : 20050614 )andlt;/descriptionandgt;andlt;/itemandgt;

How do I combine results from multiple sources?: 

How do I combine results from multiple sources? Things you might want the server to do for you: Common Record Format Common Sort Order Common Rank Order

Functional Matrix: 

Functional Matrix

The Old Solutions: 

The Old Solutions Screen Scraping Private API’s Z39.50

Screen Scraping: 

Screen Scraping A query has to be generated and embedded in a CP specific URL Code has to be written to examine the HTML returned by a CP Prone to breakage Web sites change formatting frequently Every site is unique Separate code to be maintained for every site

Private API’s: 

Private API’s Often only a slight improvement over screen scraping Provides documentation on how to construct the URL Might provide documentation on how to construct the query Might guarantee a stable response format Still requires unique code for each site

Z39.50: 

Z39.50 Guarantees a standard request and response But… Not HTTP or HTML Binary encoding over raw TCP/IP Complicated 11 services 7 extended services Easy to be compliant and not interoperable Unfriendly The response to a protocol error was to drop the connection

Why Use A Standard API?: 

Why Use A Standard API? Defined requests and responses Reusable code across sites Open Source code

The New Solutions: 

The New Solutions OpenSearch 1.1 MXG Levels 0-2 SRU

OpenSearch 1.1: 

OpenSearch 1.1 From Wikipedia OpenSearch is a collection of technologies that allow publishing of search results in a format suitable for syndication. It is a way for search engines to publish their search results in a standard and accessible format

OpenSearch 1.1 (cont.): 

OpenSearch 1.1 (cont.) Defines a Description Record with information about the CP ShortName and LongName Description Tags URL template Example: http://herbie.bl.uk:9080/opensearch.xml

OpenSearch 1.1 (cont.): 

OpenSearch 1.1 (cont.) URL Template Server Indicates how to specify OpenSearch request parameters Parameters not specified in the template are unavailable The only mandatory parameter is {searchTerms} andlt;Url type='application/rss+xml' template='http://herbie.bl.uk:9080/cgi-bin/OSxml1.cgi/?q={searchTerms}andamp;start={startIndex?}andamp;records={count?}andamp;format=rss' /andgt;

OpenSearch 1.1 (cont.): 

OpenSearch 1.1 (cont.) Request Parameters {searchTerms} {count} {startIndex} {startPage} {language} {outputEncoding} {inputEncoding}

OpenSearch 1.1 (cont.): 

OpenSearch 1.1 (cont.) Uses RSS 2.0 with a few extra elements for the response RSS define title, description and link elements OpenSearch adds the totalResults, startIndex, itemsPerPage, link and Query elements http://herbie.bl.uk:9080/cgi-bin/OSxml1.cgi/?q=levanandamp;format=rss

Functional Matrix: 

Functional Matrix Key: ●==Full Support ○==Limited Support

Cool Feature: 

Cool Feature The RSS mechanism in OpenSearch provides the ability to have persistent and periodic queries!

NISO MetaSearch XML GatewayMXG: 

NISO MetaSearch XML Gateway MXG MXG has been designed to provide a low implementation barrier to content providers that want to make their databases available to metasearch engines.  Interoperability across content providers was explicitly not a goal of MXG

MXG Levels of Support: 

MXG Levels of Support Level 0: Requests are simple URL’s using any query grammar and responses are XML records Level 1: Adds a description record for the database Level 2: Support a limited subset of a standard query grammar: CQL

MXG Request: 

MXG Request Version (mandatory) Query (mandatory) StartRecord MaximumRecords http://alcme.oclc.org/MXG/search/ORPubs?version=1.1andamp;query='levan'andamp;startRecord=1andamp;maximumRecords=10

MXG Response: 

MXG Response andlt;?xml version='1.0' ?andgt; andlt;searchRetrieveResponse xmlns='http://www.loc.gov/zing/srw/'andgt; andlt;versionandgt;1.1andlt;/versionandgt; andlt;numberOfRecordsandgt;10andlt;/numberOfRecordsandgt; andlt;recordsandgt; … andlt;/recordsandgt; andlt;nextRecordPositionandgt;1andlt;/nextRecordPositionandgt; andlt;echoedSearchRetrieveRequestandgt; andlt;versionandgt;1.1andlt;/versionandgt; andlt;queryandgt;andamp;quot;stuffandamp;quot;andlt;/queryandgt; andlt;/echoedSearchRetrieveRequestandgt; andlt;/searchRetrieveResponseandgt;

MXG Response Records: 

MXG Response Records andlt;recordandgt; andlt;recordSchemaandgt; info:srw/schema/1/dc-v1.1 andlt;/recordSchemaandgt; andlt;recordPackingandgt;xmlandlt;/recordPackingandgt; andlt;recordDataandgt; … andlt;/recordDataandgt; andlt;recordPositionandgt;1andlt;/recordPositionandgt; andlt;/recordandgt;

MXG Response recordData: 

MXG Response recordData andlt;srw_dc:dc xmlns='http://www.w3.org/TR/xhtml1/strict' xmlns:dc='http://purl.org/dc/elements/1.1/' xmlns:srw_dc='info:srw/schema/1/dc-v1.1'andgt; andlt;dc:identifierandgt;rrl1234andlt;/dc:identifierandgt; andlt;dc:titleandgt;Dog and Catandlt;/dc:titleandgt; andlt;/srw_dc:dcandgt;

MXG Error Messages: 

MXG Error Messages andlt;diagnosticsandgt; andlt;diagnostic xmlns='http://www.loc.gov/zing/srw/diagnostic/'andgt; andlt;uriandgt;info:srw/diagnostic/1/51andlt;/uriandgt; andlt;detailsandgt;66ntqkandlt;/detailsandgt; andlt;/diagnosticandgt; andlt;/diagnosticsandgt; http://www.loc.gov/z3950/agency/zing/srw/diagnostics-list.html

Functional Matrix: 

Functional Matrix Key: ●==Full Support ○==Limited Support

MXG Level 1: 

MXG Level 1 Add a description record for the database http://www.loc.gov/z3950/agency/zing/srw/explain.html http://alcme.oclc.org/MXG/search/ORPubs

Functional Matrix: 

Functional Matrix Key: ●==Full Support ○==Limited Support

MXG Level 2: 

MXG Level 2 Support a limited subset of a standard query grammar: CQL Supports indexes and Booleans http://www.loc.gov/z3950/agency/zing/cql/ http://alcme.oclc.org/srw/search/ORPublications?version=1.1andamp;query=dc.author=levanandamp;maximumRecords=1

Functional Matrix: 

Functional Matrix Key: ●==Full Support ○==Limited Support

SRU: 

SRU MXG Level 2 Plus: Full Query Grammar (CQL) Full Sort Specification

CQL: Common Query Language: 

CQL: Common Query Language Loosely based on CCL Search Boolean andamp; Proximity Operators Index Sets andamp; Indexes String Indexes vs. Keyword Indexes Truncation Characters ‘*’, ‘#’ andamp; ‘?’ Relations: ‘=‘, all, any, exact, within Example: dc.title='harry potter' or bib1.isbn=123-456-78x

Sort: 

Sort sortKeys parameter with the following comma separated values specified: Xpath (path to the element to be sorted on) Schema (that the xpath comes from) Ascending (value is 1==true or 0==false, default==true) CaseSensitive (value is 1==true or 0==false, default==false) missingValue (values are omit, abort, highValue or lowValue, default==highValue) e.g. andamp;sortKeys=title,onix,0

Functional Matrix: 

Functional Matrix Key: ●==Full Support ○==Limited Support

Cool Feature: 

Cool Feature Combining SRU response data and echoed data with javascript and stylesheets allows for thin, browser based, clients http://alcme.oclc.org/MXG/search/ORPubs?version=1.1andamp;query='levan'andamp;startRecord=1andamp;maximumRecords=10

Functional Matrix: 

Functional Matrix Key: ●==Full Support ○==Limited Support