AAAS

Slide1 : Global Discovery: Turning Vision into Reality Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC Symposium: Global Discovery on the Internet: A Grand Challenge AAAS Annual Meeting 16-20 February 2006 St. Louis, MO


Speeding up the Diffusion of New Scientific Knowledge : Speeding up the Diffusion of New Scientific Knowledge What is the goal of Global Discovery? Greatly increase the contact rate between distant communities – through a virtual aggregation or federation of diverse deep web databases. How will we achieve it? Through multiple simultaneous deep web searches with integrated ranking of results. Global Discovery: 2


Slide3 : Mathematician’s Scientific Discovery Biology Researcher’s Scientific Discovery Physics Scientific Discovery Math Community Biology Community Physics Community Biology Databases: Research Papers Correspondence Conferences Physics Databases: Research Papers Correspondence Conferences Knowledge Diffusion in Action 3


Challenges in Working with Thousands of Data Sources : Challenges in Working with Thousands of Data Sources Locate Reliable Sources Categorize Sources by Content Configure Sources for Searching Maintain Sources 4


Challenges in Searching Thousands of Sources : Challenges in Searching Thousands of Sources Automatically Select Sources to Search Perform Many Searches in Parallel Analyze and Organize Results Relevance Rank Cluster/ Visualize 5


The State-of-the-art Federated Search Engine Behind Science.gov : The State-of-the-art Federated Search Engine Behind Science.gov Scalable, grid-computing based federated search engine Sophisticated Search Conductor Multi-tier relevance ranking Framework accepts integration of advanced linguistic, analyses, and visualization modules Sponsored by DOE and Science.gov Alliance ResearchAssistant 6


Slide7 : Grid Computing: Distributing the Workload 7


Slide8 : Search Conductor 8


Multi-tier Relevance Ranking : Multi-tier Relevance Ranking QuickRank – Ranks results based on occurrence of search terms in title and snippet MetaRank – Ranks results utilizing custom algorithms applied to metadata DeepRank – Downloads and indexes full-text documents 9


Source Selection Optimizer : Source Selection Optimizer 10


Summary : Summary It is unknown at this time how many data sources would be searched through a comprehensive Global Discovery portal. Although there are significant challenges, DOE, and the Science.gov alliance with DWT as a partner are well on the road to turning the vision of global discovery into reality 11


Slide12 : Turning Vision into Reality Abe Lederman 122 Longview Drive Los Alamos, NM 87544 abe@deepwebtech.com www.deepwebtech.com 12