rashmi aifia Mar03

User research for information-rich domains: 

User research for information-rich domains Rashmi Sinha, Consultant & Researcher-at-large Uzanto Consulting www.rashmisinha.com www.uzanto.com AIfIA Leadership Seminar, IA Summit 03, Mar 21

Special user research methods for information-rich domains. Why?: 

Special user research methods for information-rich domains. Why? Such domains offer unique challenges and opportunities. design of cell phone vs. design of navigation for online cell phone advisor. Important to understand information representation in user’s minds. Semi-structured methods borrowed from cognitive anthropology and psychology can be of use.

Project Vision: 

Project Vision A portfolio of methods to probe user information representations and needs. Reduced time for user research. Same data collection phase for user categorization and persona creation. Reuse of results. Suggestion not prescription. Reliable methods.

Two key questions for user research: 

Two key questions for user research Part 1: User Categorization Scope & boundaries of the information domain Structure of information domain Differences between groups of people (different user groups, different cultures, stakeholders) Part 2: User information needs Are some needs more important than others? Can users be differentiated into groups on the basis of such needs? Can this grouping be used to form personas?

Part 1: Understanding the categorical structures of the mind: 

Part 1: Understanding the categorical structures of the mind Overview Why people categorize? The structure of semantic memory Is understanding user categorization important for IA’s? Methods Free-listing. Categorization Methods such category allocation & verification. Card Sorting and alternatives. Analysis of sorting data. Identifying dimensions or facets.

Why people categorize: 

Why people categorize Cognitive Economy Russian Psychologist Luria’s case: “Mind of a mnemonic”; Borge’s “Funius the memorius”. Allow generalization from past experiences By treating something as a kind, we can use what we have learned from other examples of that kind. Facilitate communication Describing something as a kind of thing may obviate the need to provide details of the thing itself. Permit different levels of abstraction difference between objects and ideas, animals and plants, dogs and cats, and terriers and collies.

Is understanding categorization useful for Information Architects: 

Is understanding categorization useful for Information Architects Direct use: when user categorization informs IA. Indirect use: good to have broad understanding how users think about product even when user categorization does not directly inform IA. Context is important, getting whole scheme right more important than specific category or node. Important to remember: Categorization is not static. People are good at learning new categories. If you provide the context and the right examples, they can learn new categories or alter boundaries of old categories.

Should interfaces always reflect user categories faithfully?: 

Should interfaces always reflect user categories faithfully? No. Categorization is far too important to depend only on what user thinks. Should also be influenced by business proposition, strategy, brand etc. Different user groups might differ in their perception of domain. No one scheme can serve them all perfectly. Depends on the type of site. More important to get categorization right for an architectural image database, than a small exclusive retail store e.g., Red Envelope which can push its own vision. User research should provide alternatives rather than the one answer, allowing information architects the freedom to make choices.

Structure of human memory and how categorization fits in: 

Structure of human memory and how categorization fits in Implicit / Explicit distinction Short term / working memory limitations Categorization is implicit Terms: categorization and semantic memory used interchangeably

The structure of semantic memory: 

The structure of semantic memory Semantic distance Hierarchical structure Graded structure (some members are more prototypical than others) Semantic organization builds up user expectancies regarding how things are organized Go along with expectancies to help users understand your IA. Violate these expectancies and you leave the user lost and confused.

Not all categories follow these rules: Goal directed categories: 

Not all categories follow these rules: Goal directed categories Example: “things to wear in winter to keep warm”. Central tendency does not play a role in the graded structure of goal directed categories. Graded structure is organized around ideals. For instance, exemplars are better members of category: “things to wear in winter”. The best example of this category (down jacket) is not the exemplar that is most like other category members; rather, it is the exemplar with the most extreme value on the goal-related dimension.

Individual differences in categorization: 

Individual differences in categorization Individual differences in categorization Factors such as experience, learning, culture have an impact. There is enough consistency across people for effective design. Degree of closeness between items might vary between people, but we mostly recognize the idiosyncratic association made by others. Grilled Salmon Shark Diving Aquarium Shark Sushi Fish Fish Person 1 Person 2

Semantic distance as basis of categorization: 

Semantic distance as basis of categorization At root of all categorization techniques is question: “How far is A from B?” Proximity / similarity matrix can help visualize user semantic networks (by using cluster analysis and other statistical techniques).

Semantic Distance can be plotted as a similarity matrix: 

Semantic Distance can be plotted as a similarity matrix Also called proximity matrix. One type of similarity matrix is a correlation matrix (look in Excel guide for instructions for creating correlation matrix) Self correlation (Table 1 in handout)

Plotting similarity / card sorting data: Cluster Analysis for dataset about cars: 

Plotting similarity / card sorting data: Cluster Analysis for dataset about cars Continental Buick Cadillac Mercedes Corvette Jaguar Firebird Camaro Monte Carlo Capri Chevy Vega Dart Volkswagen Cluster Analysis Advantages Suggests a structural solution Disadvantages Prescriptive Averages over different dimensions

Alternative method of analysis: Multidimensional Scaling (MDS): 

Alternative method of analysis: Multidimensional Scaling (MDS)

Advantages of Multidimensional Scaling (MDS): 

Advantages of Multidimensional Scaling (MDS) Hints at possible solutions rather than prescribes. Tells you the possibilities, leaves specifics of solution to you. Dimensions (axis) can suggest facets. Similarity maps are easy to understand Helps identify what dimensions are important cluster analysis can be done on results of MDS. This gives cleaner results. Cluster Analysis and MDS are complementary

Part 1: Understanding the categorical structures of the mind: 

Part 1: Understanding the categorical structures of the mind Overview Why people categorize Is understanding user categorization important for IA’s. The structure of semantic memory Methods Free-listing Categorization methods such category allocation & verification. Card sorting and alternatives Analysis of sorting data Identifying dimensions or facets

Free-listing exercise to explore domain scope and boundaries: 

Free-listing exercise to explore domain scope and boundaries Goals Explore boundaries and scope of domain across a group of people. Gain familiarity with user vocabulary for the domain. Use as a precursor to card-sorting, to define and limit the domain, and frame card items in the user’s language. Method Can be conducted as part of interview, or as written exercise Ask respondent, “Name all the x's you know.” Give sufficient time to do so. How many respondents? Depends on how much agreement there is about the domain. more agreement > fewer respondents. Stability of average rank order is a good index.

Free-listing Exercise: 

Free-listing Exercise Think of the menu at Wendy’s. Write down all the menu items you can think of on a piece of paper. Take one minute to do this. Some items start repeating. Some items occur earlier in the list.

Data from free-listing exercise about McDonald’s menu: 

Data from free-listing exercise about McDonald’s menu User No 1 French fries Cheese burger Shake Hamburger Cheese burger French fries Chicken sandwich Chicken Mcnuggets Fish sandwich Shake Hamburger User No 2 French fries Chicken Cheese burger Shake User No 4 Chicken Mcnuggets Cheese burger Bacon cheese burger French fries User No 5 Hamburger Quarter pounder Big mac Chicken fajita French fries Apple pie User No 3 Hamburger Cheese burger French fries Mc rib Chicken sandwich User No 6 French fries Soda Big mac Quarter pounder Cheese burger Hamburger Double cheese burger Happy meal Chicken Mcnuggets Mc Flurry Ice cream Cookies (Table 2 in handout)

Analysis of data: 

Analysis of data Create a list of all items, sorted by their average rank (of being listed by a respondent). Examine how that rank order changes with the addition of each new respondent. If the ranks are relatively stable, then you can stop adding new respondents. (Table 3 in handout)

Concept structure: 

Concept structure Plot items according to frequency of mention

Similarity from co-occurrence: 

Similarity from co-occurrence How many times did two items occur in the same free-list? Assumption of similarity if same person mentions both Crude similarity metric.

Other uses for free-listing: 

Other uses for free-listing Comparing cultural or other group differences How do two groups perceive the same domain? Comparing two domains How does perception of McDonald’s menu compare with Wendy’s? Segment respondents into types based on familiarity: Find respondents with greater domain familiarity or those who perceive domain in idiosyncratic fashion? Recursive free-listing to explore concept structure

Variants of free-listing: Semantic Association: 

Variants of free-listing: Semantic Association To find out what lies nearest to target items. Helpful in understanding problem categories and items. Exploring at same semantic level, rather than going deeper. Semantic Association What words do you associate these target words with pasta: _____ _____ _____ ______ _____ _____

Part 1: Understanding the categorical structures of the mind: 

Part 1: Understanding the categorical structures of the mind Overview Why people categorize Is understanding user categorization important for IA’s. The structure of semantic memory Methods Free-listing Categorization methods such category allocation & verification. Card Sorting and alternatives Analysis of sorting data Identifying dimensions or facets

Category Generation: 

Category Generation Goal: to understand the overall categorization scheme Method: Open card sort Users given items. Asked to create categories Options: Provide total number of categories to be created (avoid problems with splitters and lumpers). Successive card sorts. Ask for labels for each grouping.

Category Allocation: 

Category Allocation Goal: to understand goodness of existing information architecture and labels Method: Closed card sort Users given items and category labels. Asked to place each item in a category. Do not allow creation of a miscellaneous category. Useful for: Understanding user categorizations when category labels are a given Refining existing categorization scheme. Options: Allowing items to belong to multiple categories. Providing category descriptions rather than category labels.

Category Verification: 

Category Verification Goal: test goodness of an existing categorization scheme. Method: Speed verification tests Create the navigation scheme (the hierarchy / facets etc). Test this scheme by asking users to find items from various levels of the hierarchy in a short time. Note time, number of steps and mistakes. Compare these metrics with another scheme. Search for utility called “Classify”. Useful for: Comparing two designs. Testing overall goodness of an existing scheme.

Part 1: Understanding the categorical structures of the mind: 

Part 1: Understanding the categorical structures of the mind Overview Why people categorize Is understanding user categorization important for IA’s. The structure of semantic memory Methods Free-listing Categorization methods such category allocation & verification. Card sorting and alternatives Analysis of sorting data Identifying dimensions or facets

Example: Card Sorting exercise for online travel guide: 

Example: Card Sorting exercise for online travel guide Example: Designing an online travel guide to help users plan their trips to various locations. Purpose of card sort: How to structure the website to help users find travel information, and create personalized travel guides. Items include lodging, entertainment, local information, When to Go, Travel by Car/Air/Bus, Music Events, Hiking, Day Trips, Skiing, Diving, Golf, Emergency Info.

Alternative for open card sorting: Online survey software: 

Alternative for open card sorting: Online survey software Each item (card) occupies a row Each potential (unnamed category) is represented by a column Disadvantage: Cumbersome with large number of categories Open card sorts: When user generates categories

Alternative for closed card sorting: 

Alternative for closed card sorting Each item (card) occupies a row Each category is represented by a column This can be done on Excel as well Closed card sorts: When user works with given categories

Simple data analysis for card sorting: 

Simple data analysis for card sorting Look at data, one user at a time At a glance you can look down a particular column, and see what items belong in it. Also possible to create hierarchies based on user created categories

Advantages and disadvantages of using online survey software for card-sorting: 

Advantages and disadvantages of using online survey software for card-sorting Advantages Easy to carry out, no software to install and figure out. Can be done remotely using online survey software. Easy to get data into Excel (or your favorite application). Can be used to conduct either open / closed card sorts Can also be adapted to individual / group sessions Disadvantages Feels less intuitive to participants Cumbersome with large number of categories

Options with card sorting and what to from future research: 

Options with card sorting and what to from future research Options Should each item be in only one category? Should card sorts be done individually or with groups? group card sorts provides an energy and enthusiasm that makes light work of an otherwise tedious task. group card sorts can overshadow individual differences, highlighting opinion of a few. Combining card sorting with importance ratings? Future Analyze differences between user groups, different cultures etc. Individual Differences: How much consensus is there between individuals? Stakeholder analysis. Reuse of domain concept maps.

Part 1: Understanding the categorical structures of the mind: 

Part 1: Understanding the categorical structures of the mind Overview Why people categorize Is understanding user categorization important for IA’s. The structure of semantic memory Methods Free-listing Categorization methods such category allocation & verification. Card sorting and alternatives Analysis of sorting data Identifying dimensions or facets

Identifying dimensions: Method 1, 20 questions: 

Identifying dimensions: Method 1, 20 questions The user tries to guess what the item is, by asking about aspects or facets of the object. Facilitator can only answer yes or no. (method proposed by and example from Margaret Hanley on the faceted classification list) Example: Identify facets for microscopy images Questions asked: cellular / non-cellular, animal or not Facets Species Type of electron microscope The magnification of the microscope The biological process being demonstrated Part of the organism or cell being shown The method for taking the image

Identifying dimensions: Method 2, differentiating items: 

Identifying dimensions: Method 2, differentiating items Randomly select two items from your list. Ask user what differentiates them. Repeat question with other pairs till dimensions start repeating frequently. What differentiates pair below. List one or more ways that they are different. Pasta and rice: _______ ______ ______ Plane and train journey: _______ ______ ______ Good way to explore domain you are unfamiliar with. Some of the pairs will sound ridiculous!

Exercise: Identifying dimensions: 

Exercise: Identifying dimensions 20 questions (5-6 minutes) Middle person at each table, think of a favorite movie. Movie should be a well known one. Ask partners to guess by asking yes/no questions. Not important if they get the correct answer or not. Identify dimensions via questions Pair-wise differences (5-6 minutes) Choose one of the pairs of recipes List differences and similarities Identify dimensions Does listing differences work better than listing similarities?

Identifying dimensions: Method 3, Multidimensional Scaling: 

Identifying dimensions: Method 3, Multidimensional Scaling Note: add text here

Conclusion of Part 1: 

Conclusion of Part 1 Overview Why people categorize Is understanding user categorization important for IA’s. The structure of semantic memory Methods Free-listing Categorization methods such category allocation & verification. Card sorting and alternatives Analysis of sorting data Identifying dimensions or facets

Part 2: Understanding user needs, creating personas: 

Part 2: Understanding user needs, creating personas Are some needs more important than others? Can users be differentiated into groups on the basis of such needs? Can this form the basis of personas? Personas as “User Archetypes” who represents needs and goals of many other users. Using “Personas” facilitates design for one specific person. Who should that one person be? How should that person/s be identified? Not the average user Not a real user Derived from background user research (interviews etc.)

Current methods for persona creation: 

Current methods for persona creation Method Conduct interviews with various stakeholders. Find patterns. Pick a nugget, interesting tidbit and build persona around it. Problems with method: Interviews not economical way to find representative users. No tight coupling between user research & personas. Would two designers using same user research create same personas? This method works well when you have time on your hands, resources to interview many people, and skilled persona developers.

Will market segmentation techniques prove useful?: 

Will market segmentation techniques prove useful? Market segmentation is used to identify clusters of people product can appeal to. Mostly on basis of demographics. Sometimes on the basis of psychological variables (also called psychographic variables). Techniques can forecast marketplace acceptance of products and services. Can also help convince executives to build product. Such techniques often not helpful in defining product

Deconstructing marketing techniques: 

Deconstructing marketing techniques Questions focus on like / dislike of product concept what do you think of vanilla coke or green Heinz ketchup? Statistical techniques used in market segmentation cluster users according to demographic variables, not according to user needs from products. Concern themselves with product at a high conceptual level. For interaction design, one needs to focus on the specifics of the interaction: How will product satisfy needs / goals.

What is needed is a method that will…: 

What is needed is a method that will… Ground the personas in reality. Focus on motives/needs of users. Complementary to existing methods e.g., interview, observation. Concrete series of steps that any designer can follow to develop personas. Reliable: When used by other designers, at other times will lead to similar personas.

Creating personas for a Bay Area restaurant finder: 

Creating personas for a Bay Area restaurant finder Goals of site: Become the premier site for Bay Area residents & visitors to find restaurants of their choice. Design quick restaurant-finders (a la product advisors) apart from conventional advanced search / browse options Method: Step 1: Preliminary phone interviews Step 2: Survey of User Needs Step 3: General data analysis Step 4: Identify user archetypes Step 5: Add details to personas through interviews and observation (optional)

Step 1: Preliminary phone interviews: 

Step 1: Preliminary phone interviews Goal: Initial exploration of the domain. Identify questions that need to be asked for next stage. Get acquainted with user vocabulary for a domain. Phone Interviews: Ask people to describe a specific restaurant experience. Note how they chose the restaurant, what factors they indicated as important. (Note: related to concept of schemas and scripts from social psychology) Review of other similar websites to identify factors that are important. User free-listing?

Step 2: Online survey of user needs : 

Step 2: Online survey of user needs Conducted survey on factors that are used in finding restaurants. Rated importance of features in choosing restaurants on 5-point scale. Food quality, décor, wine selection, cuisine, service. Broad and shallow look. An effort to be comprehensive. Tie response to behavior: Ask respondents to recall a specific incident of choosing a restaurant, rather than answer questions in an abstract fashion. Option: Ask about scenarios of usage

Survey: 

Survey

Step 3: General data analysis to broadly understand user needs: 

Step 3: General data analysis to broadly understand user needs Look at Table 4 from Handout

Importance of various factors: 

Importance of various factors What factors are important Most respondents thinks quality of food, type of cuisine, cleanliness is very important Few respondents think valet parking is important Mixed feelings about features such buffet, location, wine selection, waiting in lines etc. Who do you visit with: Most report going to restaurants with friends, or as a couple. Fewer with family. Even fewer alone. Frequency of visiting: Most report going more than once a week.

Relationship between restaurant factors : 

Relationship between restaurant factors Do some factors go together? Exercise: Correlation Matrix with numbers Correlation Matrix with colors Examine some of the relationships

Examining relationships through a numerical correlation matrix : 

Examining relationships through a numerical correlation matrix 

Examining relationships through a shaded correlation matrix: 

Examining relationships through a shaded correlation matrix

Step 4: Identifying user archetypes: 

Step 4: Identifying user archetypes Features that are highly related to each other will form part of the same constellation, or group of needs Factor analysis (Principal Components analysis) techniques can identify such groupings. Each grouping can be said to represent a need type. These are groupings of features not of users.

Exercise: From user archetypes to personas (10 minutes): 

Exercise: From user archetypes to personas (10 minutes) Look at page showing user archetypes Each archetype shows a bunch of factors that formed a group through Principal Components Analysis Examine each. Give a name/title to each Choose one that seems suitable for personas Fill out more details for those Some archetypes seem similar (Archetype 1 and 2).

User Archetype 4: Lin looking for a romantic, relaxed, dining experience…: 

User Archetype 4: Lin looking for a romantic, relaxed, dining experience… Important Features Good decor Romantic setting Relaxed service Trendy Caters to special Requirements (Low-fat, vegan etc.) Bar Not Important Features price, buffet-style Spend more than $20, go out as a couple Date? Food Quality, Type of Cuisine & Cleanliness does not appear in these archetypes because it was important to everyone.

User Archetype 3: Chris, buffet style dining, with kids in tow…: 

User Archetype 3: Chris, buffet style dining, with kids in tow… Important Features Buffet style dining Good for groups Child Friendly Relaxed Service Valet Not Important Features views, outdoor dining, people- watching, bar Many in age group 32-41, dine mostly with family, once a week or lesser

User Archetype 1: Jen, meet friends for drinks, eat some food too…: 

User Archetype 1: Jen, meet friends for drinks, eat some food too… Important Features Outdoor seating People Watching Great view Well stocked bar Crowd Type Valet Conversation Good for groups Not Important Features long Lines, finding Parking nearby

After establishing some user archetypes: 

After establishing some user archetypes Use user archetypes as “candidate personas” Guided interviews and observations: of users who can represent various archetypes Generate sample scenarios, make final decision on primary and secondary personas according to your design goals Do not treat user archetypes generated through survey as final word. Use them in conjunction with other information.

Advantages of technique : 

Advantages of technique Inexpensive in terms of time and expertise. Statistical analysis requires SPSS, or SAS or Matlab. A reliable method, two designers using same method are likely to get similar results. Easier to get buy-in from management since personas are directly derived from user research. Retains creativity and fictional elements of creating personas, while grounding them in real data.

Other uses: 

Other uses Currently used personas (no matter how the personas were created) can be verified for their accuracy. Use the correlation matrix to do so. Personas make some key assumptions regarding what information needs go together. Conduct survey. Verify those individual relationships through correlation matrix. Options Identifying extreme cases Or normative cases Frame survey in terms of scenarios.

Review of talk: 

Review of talk Part 1: User Categorization Scope & boundaries of the information domain Structure of information domain Differences between groups of people (different user groups, different cultures, stakeholders) Part 2: User information needs Are some needs more important than others? Can users be differentiated into groups on the basis of such needs? Can this grouping be used to form personas?

Project Vision: 

Project Vision Semi structured data collection Retain the flexibility of open-ended methods. Some structure allows for deeper probing, easy summaries Can find patterns in data that cannot be explored with the naked eye. Easy returns (looking at data summaries, correlations) Sophisticated answers with more complex analysis Suggestion rather than prescription methods that provide hints rather than one “right answer” Flexible methods Can be used for both design from ground up or to improve existing design (e.g., category verification, persona verification) Reuse of results General domain specific concept maps IA adapts for their own project Reliable methods.

Thanks!: 

Thanks! For more information www.rashmisinha.com www.uzanto.com Email: rashmi@rashmisinha.com Phone: 510-703-9595