UMD Rashmi

Slide1: 

User Interfaces for Exploring Large Web-based Collections: Recommender Systems & Metadata-Based Search Interfaces Rashmi Sinha SIMS, UC Berkeley

Structure of Talk: 

Structure of Talk Broad comparison of Recommender Systems & Search/Browse Systems Different User Needs: Levels of control, task (how open-ended), level of personalized need. Focus on Recommender Systems: 2 studies, some design suggestions Quick Overview of Flamenco: A Metadata-Based Search / Browse Interface Summary, Conclusions & Future Plans Interaction Paradigm of Recommender Systems…

Basic Interaction Paradigm of Recommender Systems: 

Basic Interaction Paradigm of Recommender Systems Which book should I read? Input (Ratings of Books): I recently enjoyed: Of Mice and Men, Bias, The Summons & Good to Great Output (Recommendations): Books you might enjoy are… Popularity of Recommender Systems…

I know what you’ll read next summer (Amazon, Barnes&Noble) : 

I know what you’ll read next summer (Amazon, Barnes&Noble) what movies you should watch… (Reel, RatingZone, Amazon) what music you should listen to… (CDNow, Mubu, Gigabeat) what websites you should visit (Alexa) what jokes you will like (Jester) & who you should date (Yenta)

At the heart of Recommender Systems are Collaborative Filtering Algorithms that rely on correlation between individuals: 

At the heart of Recommender Systems are Collaborative Filtering Algorithms that rely on correlation between individuals Meg & James: correlation = .52 Meg & Jim: correlation = -.67 Meg & Nick: correlation = .23 Recommendations For Meg Example Recommender System…

Amazon’s Interaction Paradigm: 

Amazon’s Interaction Paradigm Contrast interaction with Search /Browse Systems

Types of user needs in Exploring Collections: 

Types of user needs in Exploring Collections Looking for some new music… I am tired of my old records, looking for something new. I know what I like, don’t know what else I might like, am open to ideas -more open-ended -often about “individual taste” Example Search/Browse System… Looking for some recipes for cooking I have some needs, (vegan dessert for 6 people). Also I have some strawberries lying around. -less open-ended -less about “individual taste”

Integrated Browse-Search Interaction Paradigm: 

Integrated Browse-Search Interaction Paradigm Chose tomatoes…

Slide9: 

Currently refined by Course/Meal. You can change to Preparation, Cuisine, Season. System informs you what the next choices are, and what subset of items will be left if you take that path User Experience for such systems…

User Experience in such Search/Browse interfaces: 

User Experience in such Search/Browse interfaces More of a controlled experience Every movement (forward, making a turn, backwards) is a conscious choice. (need information at every step) User might make mistakes, and retract (go back) a step or two or start again. Each of these is a conscious choice. Experience is similar to driving a car…

User Experience with Recommender Systems: 

User Experience with Recommender Systems -user has less control over specifics of the interaction. System does not provide information about specifics of action -more of the black box model (some input from user, output from systems). Experience is more similar to riding a roller coaster… What does this mean for Recommender System Interfaces?

Research on Recommender Systems has mostly focused CF Algorithms: 

Research on Recommender Systems has mostly focused CF Algorithms Collaborative Filtering Algorithms Output (Recommendations) Input from user Social Recommendations

Taking a closer look at the Recommendation Process : 

Taking a closer look at the Recommendation Process

Slide14: 

Amazon’s Recommendation Process Input: One artist/author name

Slide15: 

Output: List of Recommendations Explore / Refine Recommendations Search using Recommendations

Slide16: 

Book Recommendation Site: Sleeper Input: Ratings of 10 books for all users Use of continuous Rating Bar (System designed by Ken Goldberg)

Slide17: 

Output: List of items with brief information about each item Degree of confidence in prediction Sleeper: Output

Study 1: Book and Movie Recommender Systems: 

Study 1: Book and Movie Recommender Systems Three book systems Amazon Books Sleeper Rating Zone Three Movie Systems MovieCritic Amazon Movies Reel

Study 2: Looking for music online:Music Recommender Systems : 

Study design similar as before. Recommendations were sampled during study this time. Study 2: Looking for music online:Music Recommender Systems Five systems CDNow Amazon SongExplorer MoodLogic MediaUnbound

General Testing Methodology: 

Not an experiment, but designed like one. Conducted in Lab environment Broad overview to start with, then zero in on some systems Meshing of quantitative and qualitative methods (one informing the other) Pre-test, pre-test, pre-test User motivation ascertained before study Within-subjects design used wherever possible Multiple small studies, rather than one big study General Testing Methodology

Testing Methodology cont.: 

Comprehensive Data Collection: Observation, Behavior logging with time stamps, questionnaires, post-test interviews. Testing Methodology cont. The Slim Logger: Simple Excel Based tool for recording timed observations.

Study Procedure: 

For each of online systems: Rated items Reviewed and evaluated recommendation set Completed questionnaire For Study 1: also reviewed and evaluated sets of recommendations from 3 friends each About 15-20 participants in each study, age:18 to 34 years Study Procedure

Comparing Human Recommenders (user’s friends) to Online Systems: 

How do recommendations from Online Systems compare to that from friends? Popularity of Online Systems indicates that people find such systems useful. What are they useful for? Comparing Human Recommenders (user’s friends) to Online Systems

Human Recommenders & Systems: “Good” & “Useful” Recommendations: 

Human Recommenders & Systems: “Good” & “Useful” Recommendations RS Average Ave. Std. Error (x) No. of Recommendations

However Users Like Online RS…: 

However Users Like Online RS… This result was supported by post test interviews.

Why Systems Over Friends?: 

Why Systems Over Friends? “Suggested a number of things I hadn’t heard of, interesting matches.” “It was like going to Cody’s—looking at that table up front for new and interesting books.” “Systems can pull from a large database—no one person knows about all the movies I might like.”

Recommender Systems broaden horizons: 

Recommender Systems broaden horizons Friends mostly recommend familiar items Movies Books What aspects of systems do users like?

Why do users like particular systems: Searching for reasons: 

Why do users like particular systems: Searching for reasons Previously Liked Items & adequate Item Description are correlated. Time to Receive Recommendations & No. of Items to Rate are not correlated.

Good, Useful & Previously Experienced Recommendations: 

Good, Useful & Previously Experienced Recommendations Post Test Interviews indicate that users “trust” systems if they have already sampled some recommendations Previous Positive Experiences lead to “trust” Previous Negative Experiences lead to mistrust of system Useful Not yet read/ viewed Previously read/viewed (lead to trust) All Good Recommendations

Adequate Item Description: The RatingZone Story: 

Adequate Item Description: The RatingZone Story 0 % of Version 1 and 60% of Version 2 users found item description adequate An adequate item description, and links to other sources about item was a crucial factor in users being convinced by a recommendation. Study 1

System Transparency: 

System Transparency User perception that they understand why an item was recommended Transparent recommendations liked more than not-transparent ones for all five systems Study 2

Slide32: 

Familiar recommendations liked more than unfamiliar ones for all five systems Some Results: Effect of familiarity on liking Study 2

Two Models of Recommender System Success: 

Two Models of Recommender System Success Recommendations from Amazon received highest liking rating for Study 1 (for books & movies) and second highest for Study 2 (Music) Recommendations from MediaUnbound outperformed Amazon in Study 2 (Music) Both systems were well liked but differed dramatically in interaction style…

Amazon’s bare-bones recommendation process: 

Amazon’s bare-bones recommendation process

Media-Unbound’s long, extended (35 questions) recommendation process: 

Genre Selection Media-Unbound’s long, extended (35 questions) recommendation process

Slide36: 

Level of Familiarity Feedback at Every stage Rating some songs

Slide37: 

More feedback about user’s tastes Setting system expectations

Users find MediaUnbound recommendations more useful: 

Users find MediaUnbound recommendations more useful Also, most users preferred MediaUnbound over Amazon But whose recommendations would they buy?

Slide39: 

But, users express more interest in buying Amazon recommendations

Slide40: 

Amazon: Safe, conservative approach to recommendations Recommendations are familiar, few new items Users find system logic transparent Users don’t feel like they learnt anything news MediaUnbound: Verifies from user how familiar they want recommendations Long input process seems to generate trust Recommendations are often new, but well liked

Discussion & Design Suggestions: 

Discussion & Design Suggestions

Justify your Recommendations: 

Justify your Recommendations Adequate Item Information: Providing enough detail about item for user to make choice System Transparency: Generate (at least some) recommendations which are clearly linked to the rated items Explanation: Provide an Explanation, why the item was recommended. Community Ratings: Provide link to ratings / reviews by other users. If possible, present numerical summary of ratings.

Accuracy vs. Less Input: 

Accuracy vs. Less Input Don’t sacrifice accuracy for the sake of generating quick recommendations. Users don’t mind rating more items to receive quality recommendations. Multilevel recommendations: Users can initially use the system by providing one rating, and are offered subsequent opportunities to refine recommendation Provide a happy medium between too little input (leading to low accuracy) and too much input (leading to user impatience) Unlike with Search Engines, users are not willing to try again and again.

Slide44: 

Users like Rec. Systems as they provide information about new, unexpected items. List of recommended items should include new items which the user might not find out in any other way. List could also include some unexpected items (e.g., from other topics / genres) which the user might not have thought of themselves. Include some New Unexpected Items

Slide45: 

Users (especially first time users) need to develop trust in the system. Trust in system is enhanced by the presence of items that the user has already enjoyed. Generating some very popular (which have probably been experienced previously) in the initial recommendation set might be one way to achieve this. Trust Generating Items

Slide46: 

Trust Generating Items: A few very popular ones, which the system has high confidence in Unexpected Items: Some unexpected items, whose purpose is to allow users to broaden horizons. Transparent Items: At least some items for which the user can see the clear link between the items he /she rated and the recommendation. New Items: Some items which are new /just released The right mix of items Question: Should these be presented as a sorted list / unsorted list/ different categories of recommendations?

Slide47: 

Verify degree of familiarity user wants This can help produce the right mix of items for each user.

Slide48: 

To sell as many items as possible or to help users explore their tastes? The two goals are often contradictory, at least in short term. Important for system designer to keep goals in mind while designing system. What kind of a system do you want? Onwards to Search/Browse Systems…

Flamenco: Designing an innovative Search / Browse System for Architectural Images : 

Flamenco: Designing an innovative Search / Browse System for Architectural Images System supports explorations of a large architectural image dataset Our goal was to build a flexible navigation and search using faceted metadata

The Philosophy: 

The Philosophy Information architecture should be designed to integrate search throughout Search results should reflect the information architecture, supporting an interplay between navigation and search This supports the most common human search strategies. Use metadata to show user where to go next. More flexible than canned links Allow users to expand as well as refine

An Important Search Strategy: 

An Important Search Strategy Do a simple, general search / browse Gets results in the generally correct area Look around in the local space of those results If that space looks wrong, start over Akin to Shneiderman’s overview + details Our approach endeavors to supports this search strategy

Questions we are trying to answer: 

Questions we are trying to answer How can Faceted Metadata be effectively used to build an easy to use Search / Browse system? How many facets should be displayed at once? Should facets be mixed and matched? How much is too much? How should hierarchies be revealed (progressively, one step at a time)? How should large categories be displayed? How should refinement & expansion of query be supported?

Image Dataset: 

Image Dataset Architect Image Geo-Region Time/Date ~40,000 images, 9 hierarchical facets, rich faceted hierarchical data Facets

Development Timeline: 

Development Timeline Needs assessment. Interviewed architects and conducted contextual inquiries. Lo-fi prototyping. Showed paper prototype to 3 professional architects. Design / Study Round 1. Simple interactive version. Users liked metadata idea. Design / Study Round 2: Developed 4 different detailed versions; evaluated with 11 architects; results somewhat positive but many problems identified. Matrix emerged as a good idea. Design / Study Round 3. New version based on results of Round 2 Highly positive user response

Architects’ Image Use: 

Architects’ Image Use Common activities: Use images for inspiration Browsing during early stages of design Collage making, sketching, pinning up on walls This is different than illustrating PowerPoint Maintain sketchbooks & shoeboxes of images No formal organization scheme None of 10 architects interviewed about their image collections used indexes Do not like to use computers to find images

Type of tasks we want to support: 

Type of tasks we want to support Your firm wants to enter a competition to design a new central library in downtown Oakland. Find images to support your ideas for creative use of space in crowded urban setting (the site is surrounded by skyscrapers). You're preparing an exhibit to show off the possibilities of "environmentally-friendly" design at an upcoming home & garden show. Find some images that will encourage new-home purchasers to consider building in this way. Your team is doing a pool / pool-house / garden renovation. The lead architect’s preliminary design calls for rough/unfinished materials such as concrete and iron. The client is confused and resistant. Please find 2 or 3 images to use for a collage to help the client explore the idea of “concrete in the garden.”

The Interface: 

The Interface Nine hierarchical facets Matrix SingleTree (control interface inspired by Yahoo) Chess metaphor Opening Middlegame Endgame Tightly Integrated Search Expand as well as Refine Intermediate pages for large categories

Begin Game for SingleTree: 

Begin Game for SingleTree

Middle Game for SingleTree : 

Middle Game for SingleTree

Slide60: 

Chose “cultural landscapes” in structure types Begin Game for Matrix

Slide61: 

Next: Group by Location Middle Game for Matrix

Slide62: 

Next: Click on one image

Slide63: 

Next: Expand from image view

Slide65: 

Searched for “brick” Next: Chose pitched brick vaults

Slide66: 

Next: Expand using breadcrumbs

Results of User Study with 19 architects: 

Results of User Study with 19 architects Users rated Matrix more highly for: Usefulness for design work Seeing relationships between images Flexibility Power On all except “find this image” task, users also rated the Matrix higher for: Feeling “on track” during search Feeling confident about having found all relevant images

User Comments - Matrix: 

User Comments - Matrix “Powerful at limiting and expanding result sets. Easy to shift between searches.” “Keep better track of where I am located as well as possible places to go from there.” “Left margin menu made it easy to view other possible search queries, helped in trouble-shooting research problems.” “Interface was friendlier, easier, more helpful.” “I understood the hierarchical relationships better.”

User Comments – Single Tree: 

User Comments – Single Tree Pro “Simple” “More typical of other search engines I’d use” “Visually simpler and more intuitive…Matrix a bit overwhelming with choices.” Con “I found SingleTree difficult to use when I had to refine my search on a search topic which I was not familiar with. I found myself guessing.” “SingleTree required more thought to use and to find specific images.” “I do not trust my typing and spelling skills. I like having categories.”

Feature Usage (%) Types of Actions: 

Feature Usage (%) Types of Actions

Feature Usage (%) Refining: 

Feature Usage (%) Refining

Feature Usage – Expanding / Starting Over: 

Feature Usage – Expanding / Starting Over

Summary & Conclusions: 

Summary & Conclusions A new approach to web site search Use hierarchical faceted metadata dynamically, integrated with search Many difficult design decisions Iterating and testing was key Design Challenges How to seamlessly integrate metadata previews with search (Show search results in metadata context) How to show hierarchical metadata from several facets (The “matrix” view, Show one level of depth in the “matrix” view) How to handle large metadata categories (Use intermediate pages) How to support expanding as well as refining (Still working on it to some extent)

Overall Summary: 

Overall Summary Two modes of information exploration… User has more control… -Want information at every step -Flexible way to refine, change, expand User has less control… -Wants transparency -Wants new information -Needs to be convinced that recommendation is good

Conclusions & Future Plans: 

Conclusions & Future Plans Recommender Systems and Search/Browse systems support two different modes of information exploration Directly compare both in controlled study Expert / Novice differences in suitability of the two interfaces Task based differences in suitability of interfaces Recommender Systems: Role of explicit community as compared to automated recommendations How to integrate community with recommendations What about Product Advisors? Where do they fit in?

Slide77: 

Recommender Systems Project Kirsten Swearingen SIMS, UC Berkeley For more information sims.berkeley.edu/~sinha Metadata Based Search Users Interfaces Marti Hearst Ame Elliott, Jennifer English, Kirsten Swearingen, Ping Yee COLLABORATORS