Purves Judging spatial relevance and document loca

Judging spatial relevance for Geographic Information Retrieval: 

Judging spatial relevance for Geographic Information Retrieval Ross Purves1 and Paul Clough2 1Department of Geography, University of Zurich 2Department of Information Studies, University of Sheffield

Outline: 

Outline Motivation Geographic Information Retrieval Evaluation in IR Methodology SPIRIT system Experiment One -> Spatial relevance Experiment Two -> Spatial and thematic relevance Results Discussion and conclusions

Geographic Information Retreival: 

Geographic Information Retreival Concerned with retrieving information related to a location: e.g. <theme> <spatial relationship> <location> Documents are assigned one or more footprints Documents are retrieved and ranked according to thematic and spatial relevance

Slide4: 

Possible spatial relationships: Inside Near North/ South/ etc. Within distance of... Outside

Is SPIRIT any good?: 

Is SPIRIT any good? Qualitatively SPIRIT seems to work That’s not good enough We must be able to quantify e.g. Performance of SPIRIT in comparison to pure text search Compare SPIRIT to other GIR systems Performance of different types of ranking algorithms (e.g. van Kreveld et al., 2005) Describe types of queries where SPIRIT performs poorly

Learning from Information Retrieval: 

Learning from Information Retrieval GIR is a new, and evolving field IR has long focussed on system-focussed and user-centred evaluation strategies System-focussed methods generate “objective” measures allowing benchmarking and comparison

System-oriented evaluation: 

System-oriented evaluation Focussing on judging relevance - are the documents returned related to my query? Key measures in IR: Precision (Fraction of returned documents relevant) Recall (Fraction of possible documents returned) Standard approaches build large test collections where document relevance is judged (exhaustively) for queries

Document collection: 

Document collection SPIRIT has a 1Tb web collection 1 million geoparsed documents available for spatial search No test collection available (and building one requires lots of resource c.f. TREC/ GeoCLEF) We want to think before doing this!

First steps…: 

First steps… We need to understand how to perform system evaluation of GIR In previous work we showed that annotators felt judging thematic and spatial relevance was useful (Clough et al., 2006) We didn’t show importance of these dimensions of relevance We didn’t explore sensitivity to spatial relationships We found poor interannotator agreements

New experiments – basic methodology: 

New experiments – basic methodology Select suitable queries Perform relevance judgements Evaluate results Two experiments one looking at spatial relevance and footprints one looking at spatial and thematic relevance

Query generation: 

Query generation Queries should test capabilities of GIR not available in standard IR systems e.g. where spatial awareness is required different spatial relationships use locations which occur less frequently use locations of different granularities Mostly tourism related themes (more documents in our collection)

Experiment One: 

Experiment One Two annotators judged spatial relevance of 20 queries with different types of spatial relationship Used a bespoke relevance judgement tool Two elements of relevance judged Is the document spatially relevant to the query? Are the footprints displayed shown correct (context and location)?

Slide13: 

Judging “beaches in Cornwall” Document describing hotels in Cornwall is spatially relevant If rooms are named after beaches and geocoded these footprints are not relevant

Key results from Experiment One: 

Results ordered by annotator agreement Inside – white Near/ directional - grey Key results from Experiment One Mean P@n relatively good for all queries (mean 0.51) but... poor interannotator agreement Interannotator agreement significantly better (p<0.01) for containment than non-containment relationships

Experiment Two: 

Experiment Two For 38 queries, spatial and thematic relevance judged Approximately 10 queries for each of 4 spatial relationships: inside near directional within distance of Annotator had local knowledge for query regions

Slide16: 

No correlation apparent between thematic and spatial relevance Spatial vs. thematic relevance

Mean relevance scores: 

Mean relevance scores No significant differences between relevance scores for either thematic or spatial relevance Directional and …within distance of… queries have poorer spatial relevance scores

Discussion: 

Discussion Interannotator agreement poor for non-containment queries => We contend that, in general, containment is easier to judge without local knowledge Thematic and spatial relevance are not correlated => Judging these dimensions of relevance separately could reveal further interesting results SPIRIT perfomance seems poorer for directional and …within distance of.. queries => Our handling of multiple footprint geometries could be improved

Concluding remarks: 

Concluding remarks Preliminary results, small samples! Ongoing work in GIR is mostly from IR community Arguably, G in GIR is being neglected Our results suggest that this is important Evaluation of such systems is non-trivial and requires careful thought before commitment of resources

Slide20: 

castles east edinburgh camping west fort william islands north inverness camping north inverness cottages north ullapool walking north dunfermline hotels north ullapool beaches east dingwall climbing south aviemore skiing east zurich switzerland cottages within 15km of portree skiing within 20km of fort william skiing within 100km of glasgow museums within 50km of musselburgh hotels within 20km of stirling sailing within 40km of grangemouth music within 50km of horgen switzerland walking within 50km of zurich switzerland beaches in east lothian camping in highland mountaineering in scotland oil industry in aberdeen pubs in edinburgh walking in fife art festivals in edinburgh beaches in highland museums in switzerland switzerland museums in zurich switzerland red kites near munlochy canals near glasgow walking near beauly climbing near aviemore skiing near glencoe beaches near portree skiing near bern switzerland mountains near zurich switzerland walking near luzern switzerland camping near zurich switzerland Queries for Experiment Two