User Experience Re-Mastered: Your Guide to Getting the Right Design
86
me what you are thinking as you are grouping the cards. If you go quiet, I will
prompt you for feedback.”
Whenever participants make a change to a card, we strongly encourage them to
tell us about it. It helps us to understand why they are making the change. In a
group session, it offers us the opportunity to discuss the change with the group.
We typically ask questions like
John just made a good point. He refers to a “travel reservation” as a “travel
booking.” Does anyone else call it that?
or
Jane noticed that “couples-only resorts” is missing. Does anyone else book
“couples-only resorts?”

If anyone nods in agreement, we ask him/her to discuss the issue. We then ask
all the participants who agree to make the same change to their card(s). Par-
ticipants may not think to make a change until it is brought to their attention,
otherwise they may believe they are the only ones who feel a certain way and
do not want to be “different.” Encouraging the discussion helps us to decide
whether an issue is pervasive or limited to only one individual.
Participants typically make terminology and defi nition changes while they are
reviewing the cards. They may also notice objects that do not belong and remove
them during the review process. Most often, adding missing cards
and deleting cards that do not belong are not done until
the sorting stage – as participants begin to organize the
information.
Labeling Groups
Once the sorting is complete, the participants
need to name each of the groups. Give the fol-
lowing instructions:
Now I would like for you to name each of your

Card Sorting

CHAPTER 3
87
packages (e.g., SPSS, SAS, STATISTICA ™ ) and spreadsheets. We also show
how to analyze data that computer programs cannot handle. Finally, we
walk you through an example to demonstrate how to interpret the results of
your study.
When testing a small number of participants (four or less) and a limited num-
ber of cards, some evaluators simply “eyeball” the card groupings. This is not
precise and can quickly become unmanageable when the number of partici-
pants increases. Cluster analysis allows you to quantify the data by calculat-
ing the strength of the perceived relationships between pairs of cards, based
on the frequency with which members of each possible pair appear together.
In other words, how frequently did participants pair two cards together in the
same group? The results are usually presented in a tree diagram or dendrogram
(see Figs 3.4 and 3.5 for two examples). This presents the distance between
pairs of objects, with 0.00 being closest and 1.00 being the maximum distance.
A distance of 1.00 means that none of the participants paired the two particu-
lar cards together; whereas 0.00 means that every participant paired those two
cards together.

FIGURE 3.4
Dendrogram for our
travel Web site using
EZCalc.
Books
Links to travel gear sites
Luggage
Travel games

Create a new folder
Delete an existing folder
Rename an existing folder
View another folder
Overview of folders
Delete the trash folder
Move message between folders
Copy message between folders
Overview of messages in folder
0
2000
Complete linkageSingle linkage
4000
6000
8000
10000
12000
14000
16000
18000
20000
22000
24000
26000
28000
FIGURE 3.5
Tree diagram of
WebCAT data analysis
for an e-mail system.
BRIEF DESCRIPTION OF HOW PROGRAMS

Airplane ticket and frequent-fl yer miles ■
Rental auto, pick-up point, and drop-off point ■
Featured destinations ■
Now that you have several groups comprised of items, the question is “How do you con-
tinue to join clusters?” There are several different amalgamation (or linkage) rules available
to decide how groups should next be clustered, and some programs allow you to choose
the rule used. Below is a description of three common rules.
Single Linkage
If any members of the groups are very similar (i.e., small distance score because many
participants have sorted them together), the groups will be joined. So if “frequent-guest
credit” and “frequent-fl yer miles” are extremely similar, it does not matter how different
“hotel reservation” is from “airplane ticket” (see Round 1 groupings above); they will be
grouped in Round 2.
This method is commonly called the “nearest neighbor” method, because it takes only two
near neighbors to join both groups. Single linkage is useful for producing long strings of
loosely related clusters. It focuses on the similarities among groups.
Complete Linkage
This is effectively the opposite of single linkage. Complete linkage considers the most
dissimilar pair of items when determining whether to join groups. Therefore, it doesn’t mat-
ter how extremely similar “frequent-guest credit” and “frequent-fl yer miles” are; if “hotel
reservation” and “airplane ticket” are extremely dissimilar (because few participants sorted
them together), they will not be joined into the same cluster at this stage (see “Round 1”
groupings above).
Not surprisingly, this method is commonly called the “furthest neighbor” method, because
the joining rule considers the difference score of the most dissimilar (i.e., largest difference)
pairs. Complete linkage is useful for producing very tightly related groups.
Average Linkage
This method attempts to balance the two methods above by taking the average of the
difference scores for all the pairs when deciding whether groups should be joined. So
the difference in score between “frequent-guest credit” and “frequent-fl yer miles” may

UserZoom ( http://www.userzoom.com/online-card-sorting-study ) ■
OptimalSort ( http://www.optimalsort.com ) ■
Data analysis using these tools has been found to be quicker and easier than
using manual methods (Zavod, Rickert & Brown, 2002).
Analysis with a Statistics Package
Statistical packages like SAS, SPSS, and STATISTICA are not as easy to use
as specialized card sort programs when analyzing card sort data; but when
you have over 100 cards in a sort, some packages cannot be used. A program
like SPSS is necessary, but any package that has cluster analysis capabilities
will do.
Analysis with a Spreadsheet Package
Most card sort programs have a maximum number of cards that they can
support. If you have a very large set of cards, a spreadsheet (e.g., Microsoft
Excel) can be used for analysis. The discussion of how to accomplish this
is complex and beyond the scope of this book. You can fi nd an excellent,
step-by-step description of analyzing the data with a spreadsheet tool at
http://www. boxesandarrows.com/view/analyzing_card_sort_results_with_a_
spreadsheet_ template .
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Card Sorting

CHAPTER 3
91
Data That Computer Programs Cannot Handle
Computer programs can be great, but they often do not do all the analysis for
you. Below are some of the issues that we have encountered when using differ-
ent electronic programs. Although the data analysis for these elements is a little
awkward, we think the value that the data bring makes them worth collecting.
ADDING OR RENAMING OBJECTS
One of the basic requirements of cluster analysis is that all participants must

programs cannot deal with deleted cards. For these programs, if you have allowed
participants to create a discard or miscellaneous pile of cards that they do not
believe belong in the sort, there is a workaround you need to do. You cannot
enter this collection of discarded cards as a group into a computer program since
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
User Experience Re-Mastered: Your Guide to Getting the Right Design
92
the cluster analysis would treat these cards as a group of objects that participants
believe are related. In reality, these cards are not related to any of the other cards.
Place each rejected card in a group by itself to demonstrate that it is not related
to any other card in the cluster analysis. For example, if participants placed
“Frequent-Flyer Miles,” “Companions,” and “Meal Requests” in the discard pile,
you should enter “Frequent-Flyer Miles” in one group, “Companions” in a sec-
ond group, and “Meal Requests” in a third group.
Interpreting the Results
You now have a collection of rich data. The dendrogram displays groups of
objects that the majority of participants believe belong together.
Changes that participants make to cards can make interpretation of the results
tricky. When a deleted object is repeatedly placed in a group by itself (or left out,
in the case of EZCalc) , you may see it on a branch by itself or loosely attached
to a group that it really doesn’t belong with. Additionally, if participants place
an object in multiple groups, they may not have agreed on the “best” location
to place it. Consequently, you may fi nd the object is living on a branch by itself
or loosely attached to a group that it really doesn’t belong with. You must use
your knowledge of the domain or product to make adjustments when ambigu-
ity exists. Use the additional data you collected like new objects, group names,
changed terminology, and think-aloud data to help interpret the data.
Let’s walk through our travel example and interpret the results of our dendrogram
shown earlier in Fig. 3.4 . Using our domain knowledge and the group labels
participants provided in the card sort, we have named each of the clusters in

and therefore needs to be incorporated – or whether there are several different
possible terms. When several terms exist, you will want to use the most common
term but allow your product to be customized so that it is clear to all your users.
Finally, examine the defi nition changes. Were the changes minor – simply an
issue of clarifi cation? If so, there isn’t anything to change in your product. If, how-
ever, there were many changes, you have an issue. This may mean that the prod-
uct development team does not have a good grasp of the domain or that there is
disagreement within the team about what certain features of the product do.
COMMUNICATE THE FINDINGS
Preparing to Communicate Your Findings
The specifi c data that you communicate to product teams can vary depending
upon the activity you conducted, but some elements of how you communicate
the results are the same regardless of the method.
FIGURE 3.6
Dendrogram of a
travel Web site card
sort with group names
added.
Books
Links to travel gear sites
Luggage
Travel games
Family friendly travel information
Currency
Languages
Tipping information
Featured destinations
Travel alerts
Travel deals
Weekly travel polls

user requirement techniques should be used along the way to capture new require-
ments and verify your current requirements.
MODIFICATIONS
Below are a few modifi cations on the card sorting technique we have presented.
You can limit the number of groups users can create, use computerized tools for
the sort instead of physical cards, provide the groups for users to place the cards
in, ask users to describe the items they would fi nd in a particular category, or
physically place groups that are related closer to each other.
Limit the Number of Groups
You may need to limit the number of groups a participant can create. For exam-
ple, if you are designing a Web site and your company has a standard of no
more than seven tabs, you can ask participants to create seven or fewer groups.
Alternatively, you can initially allow participants to group the cards as they see
fi t; then, if they create more than seven groups, ask them to regroup their cards
into higher-level groups. In the second case, you should staple all the lower-level
groups together and then bind the higher-level groups together with a rubber
band. This will allow you to see and analyze both levels of groupings.
Electronic Card Sorting
There are tools available that allow users to sort the cards electronically rather
than using physical cards (e.g., OptimalSort, WebSort, xSort, and CardZort). Elec-
tronic card sorting can save you time during the data analysis phase because
the sorts are automatically saved in the computer. Another advantage is that,
depending on the number of cards, users can see all the cards available for sort-
ing at the same time. Unless you have a very large work surface for users to
spread their physical cards on, this is not possible for manual card sorts. Elec-
tronic sorting has the disadvantage that, if you run a group session, you will
FIGURE 3.7
Travel card sort table
of recommendations.
Tab name

Some tools support remote testing, which allows you to gather data from users
anywhere. However, users may have a more diffi cult time without a facilitator in
the room to answer questions.
Unfortunately, none of the computer-based programs provides a defi nition with
the objects. Also, they do not allow users to add, delete, or rename the objects.
In our opinion, this is a serious shortcoming of the tools and the reason why we
do not use them.
SUGGESTED RESOURCES FOR ADDITIONAL
READING

Prename the Groups
You may already know the buckets that the objects being sorted must fi t into.
Going back to our Web site example, if you cannot completely redesign your
site, you may want to provide participants with the names of each tab, section,
or page of your site. Provide participants with a “placemat” for each group. The
placemat should state the name of the group and provide a clear description of
it. Participants would then be tasked with determining what objects fi t into the
predetermined groups.
To go one step further, you may have the structure for your entire application
already laid out and simply want to fi nd out whether you are correct.
The article below provides a nice comparison of some of the automated card sorting tools
available (at the time of publication) if electronic card sorting is of interest to you:
Zavod, M. J., Rickert, D. E. & Brown, S. H. (2002). The automated card-sort as an ■
interface design tool: A comparison of products. In : Proceedings of the human
factors and ergonomics society 46th annual meeting, Baltimore, MD,
30 September–4 October, pp. 646–650.
EDITOR’S NOTE: CLOSED AND REVERSE CARD SORTING
The last example where you provide users with the names of categories and then put items
into those categories is called closed card sorting. Closed sorting is useful when you are
verifying an existing hierarchy or structure (e.g., the main menu of an application or Web

users understood the new site’s categorizations and task groupings, compared with only
45 percent on the old design” (Human Factors International, ND, http://www.humanfac-
tors.com/about/arinc.asp ).
Ginny Redish conducted a card sort for the National Cancer
Institute’s Division of Cancer Prevention. Since she does
not work for the National Cancer Institute, she describes
how she worked as a consultant with the development team
and gained the domain knowledge necessary to conduct
the card sort. She describes in wonderful detail the process
of understanding the user profi le, identifying the objects
for sorting, creating the materials, and recruiting the par-
ticipants. She provides a unique perspective because she
conducted the sort individually with think-aloud protocol
and opted not to use cluster analysis software.
Case Study
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Card Sorting

CHAPTER 3
97
various ways to analyze the data and used our travel example to show you how
to interpret and present the results.
Below, Ginny Redish presents a case study to share with readers how she recently
employed a card sort to build the information architecture for a government
Web site.
HOW CARD SORTING CHANGED A WEB SITE TEAM’S
VIEW OF HOW THE SITE SHOULD BE ORGANIZED
Janice (Ginny) Redish
Redish & Associates, Inc.
This case study is about the Web site of the U.S. National Cancer Institute’s

on the site for the general public! (The eighteenth scenario was about a gradu-
ate student seeking a postdoctoral fellowship – a very legitimate scenario for the
Web site.)
The stage was now set for card sorting. The project team agreed that card sorting
was the way to fi nd out how members of the public and medical professionals
would look for information on the site.
Planning and Preparing for the Card Sorting
Members of the project team wrote cards for topics. In addition to the top-
ics from each research group and from the offi ce that handles fellowships, we
added cards for types of cancer and for articles that existed elsewhere in the
many National Cancer Institute Web sites to which we could link.
HOW MANY CARDS?
We ended up with 300 cards – many more than we could expect users to sort in
an hour. How did we winnow them down? We used examples rather than hav-
ing a card for every possible instance of a type of topic or type of document.
For example, although there are many types of cancer, we limited the cards to
about 10 types. For each type of cancer, you might have information about pre-
vention, screening, clinical trials, etc. Instead of having a card for each of these
for each type of cancer, we had these cards for only two types of cancer – and our
card sorters quickly got the point that the fi nal Web site would have comparable
entries for each type of cancer. Instead of having a card for every research study,
we had examples of research studies.
Even with the winnowing, we had about 100 cards – and that was still a lot for
some of our users. An ideal card sorting set seems to be about 40–60 cards.
WHAT DID THE CARDS LOOK LIKE?
Figure 3.9 shows examples of the cards. Each topic went on a separate 3 ϫ 5 inch.
white index card. We typed the topics in the template of a page of stick-on labels,
printed the topics on label paper, and stuck them onto the cards – one topic per card.
We created two “decks” of cards so that we could have back-to-back sessions.
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

singly or in pairs for an hour each; pairs worked together, sorting one
set of cards while discussing what they were doing – like codiscovery in
usability testing
The National Cancer Institute worked with a recruiting fi rm to bring in can-
cer patients/survivors, family members of cancer patients/survivors, members
of the public interested in cancer, doctors, and other health professionals. Our
eight external users included people from each of these categories. The external
people were paid for their time.
CONDUCTING THE CARD SORTING SESSIONS
The only real logistic need for card sorting is a large table so that the par-
ticipant can spread out the cards. We held sessions in an empty offi ce
with a large desk, in a conference room, and on a round conference table
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
User Experience Re-Mastered: Your Guide to Getting the Right Design
100
in another offi ce. The conference room table worked best; one participant
especially liked the chair on wheels so he could roll up and down next to
the table looking at his groupings. Other participants sorted the cards stand-
ing up so they could reach along the table to work with the cards they had
already put out.
In addition to the deck of cards with topics on them, we also had:
Extra white cards for adding topics
■
Sticky notes for indicating cross-links (when participants wanted a ■
topic to be in two places, we asked them to put it in the primary place
and write a sticky note to indicate what other group should have a
link to it)
Cards in a color for putting names on the groups at the end ■
Rubber bands for keeping each group of cards together at the end ■
Pens for writing new topics, cross-links, and group names ■

101
When the participants had sorted all the cards, we gave them the colored cards
and asked them to name each of their groups. We also asked them to place the
groups on the table in the approximate confi guration that they would expect to
fi nd the groups on the home page of a Web site.
The Analysis
In this study, we found that we did not need to do a formal analysis of the data
to meet our goals of understanding at a high level what categories people wanted
on the home page, where on the home page they would put each category, and
the general type of information (topics) that they would expect in each category.
We did not do a formal analysis with complex cluster analysis software for at
least four reasons:
This was a very small study – eight users. ■
We were looking only at the top level of an information architecture. Our in- ■
terest was the home-page categories with names and placement on the page
for those categories and a general sense for
the types of information that would go in
each category. We were not doing an entire
information architecture or putting every
underlying piece of content into a category.
This was just one step in an iterative pro- ■
cess. Our goal was to get input for a proto-
type that we would take to usability testing.
The project continued through several
rounds of prototypes and usability testing.
It was obvious as soon as the sessions ■
were over that there was incredibly high
agreement among the users on the catego-
ries, names, and placements.
If any of these four had not been the case, a for-

Six participants had ■ About NCI DCP or Administration. This category
included the mission statement, organization chart, directory, etc.
Although two of the eight participants also wanted a very brief
mission statement with a link in the upper left corner of the home
page, all six put the About NCI DCP category in the lower right of
the page.
OPENING INTERNAL USERS’ EYES
The technique itself can open the eyes of internal users to the problems with the
way the site is currently designed.
The participants from the Web project team (the internal users) all started by
sorting cards into their organizational groups, creating once again the old Web
site. However, after fi ve to 10 minutes (and sometimes with a bit of prodding
to “think about the users and scenarios you wrote in the meeting”), they made
comments like this: “How would someone from the public know that you have
to look under [this specifi c research group] to fi nd out about that?”; “The public
would want to look up a specifi c type of cancer.”; “The public would want to
look up information about diet or nutrition.”
In the end, each of the internal users came to very similar groupings as the
public. They also realized on their own that information about the organiza-
tion would not be the most important reason people came to the site. Like
the public users, they put the About NCI DCP category in the lower right of
the page.
If you think of internal users as “developers,” you may wonder whether it was
wise to let them do the card sorting. Of course, you do not want to have the
developers (or internal users) be the only card sorters. The primary audience for
the site must be the primary participants in any card sorting study.
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Card Sorting

CHAPTER 3

you are running consecutive sessions – someone to record what each partici-
pant has done and reshuffl e the cards for another participant. The diffi cult part
of card sorting is deciding on the topics to include and limiting the number of
cards by choosing good exemplars of lower-level content rather than including
every single article that might be on the site.
What Happened to the Web site?
Figure 3.11 is the “after” version that was launched in the summer of 2001. (The
current site at http://www.cancer.gov/prevention is a later update following
NCI’s adoption of new look and feel standards.)
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
User Experience Re-Mastered: Your Guide to Getting the Right Design
104
ACKNOWLEDGMENTS
My time as a consultant to the NCI Division of Cancer Prevention (DCP) came
through my work with the NCI Communication Technologies Branch (CTB) in
the NCI Offi ce of Communication. NCI is part of the U.S. National Institutes
of Health, Department of Health and Human Services. I thank Kara Smigel-
Croker (DCP Communications Manager) for leading this project and Madhu
Joshi (who was a CTB Technology Transfer Fellow at the time) for handling all
logistics and support.
FIGURE 3.11
The Web site
after card sorting,
prototyping, and
iterative usability
testing.
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
PART 2
PART 2
Generating Ideas

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
User Experience Re-Mastered: Your Guide to Getting the Right Design
108
Discussing, critiquing, and possibly prioritizing the brainstorming 4.
results for subsequent action (this last step is often called the “conver-
gent” phase where there is a winnowing of all the ideas into the ones
that are judged as most applicable to a problem)
Variations on this group brainstorming procedure can be used to gather ideas
from large groups, geographically dispersed individuals, or participants who are
inhibited by their personality, the social environment, or cultural norms. These
variations are described later in this chapter.
Alex Osborn, an advertising executive, is generally credited with developing
modern organizational brainstorming procedures in the 1940s and 1950s
( Osborn, 1963 ). Osborn’s brainstorming process (originally called “thinking
up”) is described in his classic book, Applied Imagination: Principles and Proce-
dures of Creative Problem-Solving.

There are three fundamental principles for group brainstorming:
1. Aim for sheer quantity . Quantity, not quality, is the sole goal of brain-
storming. The only criterion for the success of brainstorming is the sheer
number of ideas that are generated. Anything that limits the number
of ideas is contrary to the intent of brainstorming. For example, brain-
storming participants should not be taking their own notes because that
reduces their cognitive resources available for generating ideas. Partici-
pants should not be monitoring e-mail (so easy now with wireless con-
nections) or reading reports during brainstorming. All the resources of
the participants should be focused on generating as many ideas as pos-
sible. The principle that “more is always better” is generally supported
in the research literature although there are issues with defi ning exactly
what quality means in brainstorming.

in many cases what people consider “good brainstorming” is often seriously
defi cient.
Osborn’s “structured brainstorming” approach with clear ground rules and
procedures contrasts with “unstructured brainstorming,” in which a group
gets together to generate ideas without a facilitator and clear ground rules.
Ideas that emerge from unstructured brainstorming are often criticized as
they are generated and loud or dominant individuals can exert inordinate
infl uence on the quiet participants, thus limiting the number of ideas that
participants are willing to express. This chapter will focus on structured brain-
storming where there is generally a facilitator and a set of explicit rules for
participants.

When Should You Use Brainstorming?
You can use brainstorming to:
Generate ideas or requirements
■
Find solutions to problems ■
EDITOR’S NOTE: DON’T BELIEVE ALL THAT YOU READ ON
THE WEB: GROUP BRAINSTORMING ISN’T SIMPLE!
Although group brainstorming seems simple, there are many social issues like status
differences, shyness, informal relationships, ego, and cultural factors that can affect the
quantity of ideas. Camacho and Paulus (1995) found, for example, that social anxiety
had a signifi cant effect on brainstorming productivity and suggested that “… interactive
[group] brainstorming may be best suited for people who are low in social anxiety.”
(p. 1,078). A trained facilitator can mitigate some of these factors, but even a good
facilitator won’t have total insight into all the social forces and group dynamics that can
infl uence productivity. Jared Sandberg (2006) summarizes some key requirements for
successful group brainstorming:
“In fact, great brainstorming sessions are possible, but they require the planning of
a state dinner, plenty of rules, and the suspension of ego, ingratiation, and political

It is sometimes less effective than having the same number of partici- ■
pants generating ideas individually. The quantity of ideas can suffer
when one person in the brainstorming group blocks the production of
ideas by other participants by telling “war stories” or whispering to a
colleague and distracting the rest of the group.
It can be chaotic and intimidating to the quiet or shy individual. ■
It can reduce individual recognition for good ideas (though you can ■
compensate for this by being known as a “good brainstormer” and cre-
ative contributor).
It may be diffi cult in some countries or cultures where “wild ideas” ■
may be viewed as inappropriate because those ideas are contrary
to those of more senior colleagues, corporate initiatives, or cultural
norms.
The status or experience differences among participants can reduce brain- ■
storming effectiveness. Mixing senior and junior colleagues can result in
the junior people deferring to their more senior colleagues.
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.