Advanced Web Searching, IFEG, 3rd April 2012


Presentation Description

Presentation given to the Information for Energy Group (IFEG) Spring Symposium. Hosted by the Energy Institute, London, 3rd April 2012


Presentation Transcript

Advanced web searching Information for Energy Group (IFEG) Spring Seminar Energy Institute, London 3rd April 2012:

Advanced web searching Information for Energy Group (IFEG) Spring Seminar Energy Institute, London 3 rd April 2012 Karen Blakeman RBA Information Services Slides are available at Twitter: @karenblakeman This presentation is licensed under a Creative Commons Attribution 3.0 License Energy Balance: 100 kW Hydroelectric Turbine at Mapledurham.

Trends in search:

Trends in search No longer straightforward text searching of web pages and documents Localisation Personalisation Social Mobile 19/11/2013 2

General search tools :

General search tools Google - 91% of UK market, 67% US market Bing/Yahoo Yahoo now uses Bing's database and search algorithms for web, image, video and news search Alternatives to the "big two" DuckDuckGo Blekko Increasingly need to be aware of specialist tools 19/11/2013 3

Sanity checking Google:

Sanity checking Google “if I hadn’t searched across more than Google for data on a small, new company that I was asked to research recently, I would have missed out on some very significant information that Google just wasn’t showing me.” 19/11/2013 4

How Google started:

How Google started 19/11/2013 5 11 November 1998 The Internet Archive

How was Google different?:

How was Google different? 19/11/2013 6 Links (citations) a major part of ordering search results

Where is Google now?:

Where is Google now? 19/11/2013 7 2001 Revenues $86,426 thousands Net Income $10,964 thousands 2011 Revenues $37,905 millions Net Income $9,737 millions 2011 – 96% of revenues are from advertising Google is mass market consumer oriented. Serious researchers wanting reliable, structured search are a miniscule fraction of their customer base.

New How People Spend Their Time Online – Stephen's Lighthouse :

New How People Spend Their Time Online – Stephen's Lighthouse 19/11/2013 8

How Google organises and sorts information:

How Google organises and sorts information Has a primary index of higher "quality" documents and a secondary index. Only the primary index is searched when running straightforward searches. Secondary index comes into play with more complex searches and if a small number of results are found. “Dear Bing, We Have 10,000 Ranking Signals To Your 1,000. Love, Google” Over 200 hundred “signals” and each may have over 50 variations 19/11/2013 9

How Google ranks and organises your results:

How Google ranks and organises your results 19/11/2013 10 Google personalizes and tailors your results depending on your location, computer/device, browser, past searches, what you have looked at in the past, your +1s, your Google+ account, what you had for breakfast...and anything else it can find by rummaging around in your Google dashboard To see what's in your dashboard log in to your Google account and go to Also see Google personalisation: web history isn’t the only problem

PowerPoint Presentation:

19/11/2013 11 Google to Launch Third-Party Commenting Platform

PowerPoint Presentation:

Bing does it all as well! "Adaptive search" links with Facebook Sign out of accounts, clear cookies, switch off history DuckDuckGo – no tracking, no personalisation Private browsing/No-tracking options in browser Use Chrome Incognito (Chrome owned by Google!) 19/11/2013 12

Google's new Privacy Policy:

Google's new Privacy Policy 19/11/2013 13 “ Our new Privacy Policy makes clear that, if you’re signed in, we may combine information you’ve provided from one service with information from other services. In short, we’ll treat you as a single user across all our products, which will mean a simpler, more intuitive Google experience. “


What I see on my screen for a search is not what you’ll see on yours. 19/11/2013 14

Google totally loses the plot For 10 days in February 2011: coots = lions:

Google totally loses the plot For 10 days in February 2011: coots = lions 19/11/2013 15 Google decides that coots are really lions Update on coots vs. lions

Coots = lions:

Coots = lions 19/11/2013 16

Three search tricks:

Three search tricks These three techniques can change what Google (and other search engines) decides to give you and also the order of the results. Repeat important search terms coots coots mating behaviour (found coots) Change the order of your terms mating behaviour coots (found coots) Change one of your search terms coots mating behaviour (found lions) coots courtship behaviour (found coots) coots mating ritual ( found coots) 19/11/2013 17

Excluding pages containing words :

Excluding pages containing words Want to exclude pages containing a term? Place a - (minus sign) before the term Use with care as may miss important material Excluding lions from our bizarre coots search coots mating behaviour –lions gave us: 19/11/2013 18

PowerPoint Presentation:

Coots=lions was an extreme example of how Google can work We think Google was doing the following: - assumed a typing error or was running a mobile/smartphone predictive text algorithm (coots=cats)? - ran an automatic variation/synonym search on cats? - used a search frequency rule and found that lions mating behaviour was requested more than cats? 19/11/2013 19

PowerPoint Presentation:

Dear Google, stop messing with my search 19/11/2013 20 Google no longer looks for all of your terms in a page

See what Google sees:

See what Google sees 19/11/2013 21 Hover over a result and a "preview" of the page should appear to the right together with a Cached link – this is Google copy

PowerPoint Presentation:

“When you do a multi-term query on Google (even with quoted terms), the algorithm sometimes backs-off from hard ANDing all of the terms’s clear that people will often write long queries (with anywhere from 5 to 10 terms) for which there are no results. Google will then selectively remove the terms that are the lowest frequency to give you some results (rather than none)....Soft AND is a way to reduce the overall frustration and give the searcher something to examine (and with luck, a chance to reformulate their query).” Dan Russell 19/11/2013 22


Verbatim Forces Google to run an exact match search. Run your search first and then select Verbatim from the left hand menu on your results page Cannot be combined with time options in the side bar Google: Verbatim for exact match search 19/11/2013 23

Google doing its own thing can be good:

Google doing its own thing can be good 19/11/2013 24

PowerPoint Presentation:

Google's new(ish) social network Google Plus (Google+) Google trying forcing people to create a Google+ profile Search Plus Your World (SPYW) referred to as Search+ now available in and is the default. Gives priority to content from people in your Google+ network if you are signed in to your account. (And the next Google killer is….Google! ) 19/11/2013 25 Search gets Social - Resistance is Futile

PowerPoint Presentation:

19/11/2013 26

PowerPoint Presentation:

19/11/2013 27 [Not signed in to a Google account]

PowerPoint Presentation:

19/11/2013 28 Signed in to Google account on

PowerPoint Presentation:

19/11/2013 29

PowerPoint Presentation:

19/11/2013 30 About 4,940,000 results (0.74 seconds) !!!

Google results side bar:

Google results side bar These help you focus your search "Everything" does NOT search everything Vary depending on type of search e.g. web, news, images Open up the "more" options to see everything 19/11/2013 31

Google side bars:

Google side bars 19/11/2013 32 Images Videos News Books Blogs

Google Images:

Google Images 19/11/2013 33 Similar images

Google Images:

Google Images 19/11/2013 34

Google Images – use an existing image:

Google Images – use an existing image 19/11/2013 35 Click on the camera icon in the search box and then either enter the URL of an image or upload it


Flickr Flickr Creative Commons or advanced search screen 19/11/2013 36

Images - other sources for Creative Commons and public domain images :

Images - other sources for Creative Commons and public domain images Wikimedia Commons (check the licence information towards the bottom of the page e.g. ) - public domain Geograph Creative Commons 2.0 Most of the images on US government web sites are public domain (but do check) NASA - public domain 19 November 2013 Karen Blakeman 37

Google Video:

Google Video 19/11/2013 38 Not the same as Youtube


Video Bing Videos YouTube Blinkx - 19 November 2013 Karen Blakeman 39

Google News:

Google News 19/11/2013 40

Google News Archive:

Google News Archive 19/11/2013 41 19/11/2013 42 19/11/2013 43

Google Books:

Google Books 19/11/2013 44

Blogs :

Blogs 19 November 2013 Karen Blakeman 45

Related searches:

Related searches 19/11/2013 46

Translated foreign pages for a different perspective:

Translated foreign pages for a different perspective Google suggests languages from context of search but you can choose your own Your search is translated and the results are translated into your language 19/11/2013 47

Advanced search:

Advanced search Use search commands or Advanced Search screen 19/11/2013 48

Problems finding information on a particular site?:

Problems finding information on a particular site? Use Google's site: command Can combine with date options in side menu 19/11/2013 49

PowerPoint Presentation:

Or if you are interested in UK academic reports 19/11/2013 50

PowerPoint Presentation:

Looking for a particular type of information for example statistics, research report, expert presentation? Use the filetype : command For statistics world oil consumption filetype:xls world oil consumption filetype:xlsx world oil consumption filetype:xlsx OR filetype:xls For government, research, industry reports UK oil consumption forecasts filetype:pdf For conference presentations or trying to locate an expert renewable energy UK filetype:ppt renewable energy UK filetype:pptx renewable energy UK filetype:ppt 19/11/2013 51

Numerical range search:

Numerical range search Anything to do with numbers Use advanced search screen or 1 st number followed by two full stops followed by 2 nd number followed by unit of measurement (if applicable) Norway oil production forecasts 2012..2020 Norway oil production forecasts 2012..2020 filetype:xls OR filetype:xlsx 19/11/2013 52

Advanced commands continued :

Advanced commands continued inurl : for example inurl :"carbon capture" targets intitle : for example intitle :"carbon capture" targets asterisk (*) to search for terms separated by 1-5 words (may have to use quotation marks) solar * panels "solar * panels" Picks up solar PV panels, solar photovoltaic panels, solar water heating panels 19/11/2013 53


Synonyms Google often looks for variations of your terms but you cannot rely on it always happening Use the tilde ~ before a term to look for what Google considers are synonyms ~energy will pick up oil, fuel, gas, electricity No information/documentation on how synonyms are created Very general, consumer oriented rather than scientific Can be used with Verbatim 19/11/2013 54

Google alternatives - Bing and Yahoo:

Google alternatives - Bing and Yahoo Yahoo now uses Bing's database and ranking Many of the Advanced Search commands are similar to Google’s, see Search Tools Summary and Comparison Most of the interesting developments and features are only available in the US version Results tend to be more consumer/retail focused unless using advanced search features Coverage not identical to Google’s - sometimes yields important unique content Sometimes more up to date than Google 19/11/2013 55


DuckDuckGo DuckDuckGo – silly name but a neat little search tool No tracking, no “filter bubble” Commands site: filetype: sort:date to sort by date (uses results from Blekko) Syntax and keyboard shortcuts at 19/11/2013 56 19/11/2013 57 advanced search: advanced search 19/11/2013 58


Blekko slashtags for sorting by date (/date), searching for images (/images) and videos (/videos) create your own to search your specified list of sites (similar to Google Custom Search Engines) wind turbine electricity generation /karenblakeman/renewable “Musings about librarianship: Using Blekko to search across thousands of library sites” 19/11/2013 59


Blekko Cannot do filetype, inurl, intitle searches Drop down menu next to page in results list for site search (or use /site), similar pages (or use /similar), inbound links to the page (or use /links) 19/11/2013 60

Google Scholar:

Google Scholar A useful place to start your research or if you are looking for a specific paper but no source list, not comprehensive and omits many key scientific publications Both peer-reviewed and un-reviewed articles, pre-prints, institutional repositories, references to books, citations Does not use publishers’ meta data, author search unreliable, search on year of publication unreliable Sometimes does strange things with your search terms but you can still use + before a term to force exact match search Sometimes finds unique content 19/11/2013 61

PowerPoint Presentation:

19/11/2013 62

Authors encouraged to claim papers:

Authors encouraged to claim papers 19/11/2013 63

Microsoft Academic Search:

Microsoft Academic Search 19/11/2013 64

Microsoft Academic Search:

Microsoft Academic Search 19/11/2013 65

Microsoft Academic Search:

Microsoft Academic Search Problems coverage sometimes gets the author completely wrong “Will the Real Scott Wilson Please Stand Up, Please Stand Up” 19/11/2013 66 M Edwards should be Martin Edwards not Maria-Benedicta Edwards 19/11/2013 67 19/11/2013 68

Google Public Data Explorer:

Google Public Data Explorer 19/11/2013 69

PowerPoint Presentation:

19/11/2013 70 Statistical Review of World Energy 2011 | BP

Energy Export Databrowser :

Energy Export Databrowser 19/11/2013 71

GasTrends databrowser :

GasTrends databrowser 19/11/2013 72

Department of Energy and Climate Change

Department of Energy and Climate Change 19/11/2013 73


RESTATS 19/11/2013 74


OFFSTATS 19/11/2013 75


OFFSTATS 19/11/2013 76

Priced industry and market research:

Priced industry and market research Aggregators may not be comprehensive use aggregators as an index to see who is publishing on your topic, for example (go to Energy and Utilities) For emerging markets Many sites specialising in energy reports - just search on energy market research reports 19/11/2013 77 19 November 2013 Karen Blakeman 78 19/11/2013 79 : 19/11/2013 80 19/11/2013 81 19/11/2013 82 - create your own newspaper: - create your own newspaper 19/11/2013 83 – individual Twitterstream – individual Twitterstream 19/11/2013 84 – keyword – keyword 19/11/2013 85

Create your own Google custom search engine:

Create your own Google custom search engine For regularly searched sites selected sites on a subject or type of organisation Cannot include password protected sources or sites where you have to fill in a form to access the information Information on setting up a Google Custom Search Engine (CSE) Google's blog on custom search 19 November 2013 Karen Blakeman 86

PowerPoint Presentation:

Think about the type of information you are looking for – news, statistics, experts Get to know the options in Google's sidebar Get to know the advanced search commands for example site: filetype : intitle : numeric range Get to know the alternative search tools and use them! Keep up to date with changes to search tools and in particular Google 19/11/2013 87

Keeping up to date:

Keeping up to date Inside Search Official Google Blog Google Scholar Blog Search Engine Land Search Engine Watch Boolean Black Belt-Sourcing/Recruiting Karen Blakeman’s Blog Phil Bradley's weblog 19/11/2013 88

authorStream Live Help