Wiley Publishing, Inc.
Always Be Testing
THE COMPLETE GUIDE TO GOOGLE™ WEBSITE OPTIMIZER
BRYAN EISENBERG
JOHN QUARTO
VONTIVADAR
LISA T. DAVIS
90637ffirs.indd vii90637ffirs.indd vii 6/26/08 12:12:16 AM6/26/08 12:12:16 AM
Advance praise for Always Be Testing: The Complete Guide to Google™ Website Optimizer
I can tell you two things. The first thing is: Testing works. If you’re not doing it well, you’re missing out.
The second? Bryan Eisenberg knows what he’s talking about. Time to get started, time to start testing.
—Seth Godin, Author of Meatball Sundae
Always Be Testing is a very readable and practical guide to landing page testing from an experienced
master practitioner. Bryan has emerged from the trenches with actionable advice and detailed testing tips
that will make your cash register ring more often. Pick up a copy before your competitors do!
—Tim Ash, Author of Amazon’s e-commerce bestseller Landing Page Optimization: The Definitive
Guide to Testing and Tuning for Conversions
If you want to become an expert in website testing, this is the place to start. The book’s depth and breadth
are particularly impressive, from why testing matters to what and how to test. Bryan and John have com-
bined years of experience in persuasive marketing with their deep knowledge of Google Website Optimizer
to deliver actionable insights that will make rock stars out of any marketing team.
—Tom Leung, Senior Business Product Manager, Google Website Optimizer
This is the best collection of practical testing advice that is available today. If you want to get ahead in
testing, then you and everyone involved in building and maintaining your website need to read it. Always
Be Testing offers basic and advanced testing principles, based on years of experience, that help you opti-
mize (with or without Google Website Optimizer) quickly and efficiently. The authors provide practical
exercises to help you evaluate your site objectively, tools to help you identify what to test, and resources
to expand your knowledge. This book should be required reading for anyone interested in helping online
visitors achieve their goals—it easily can save you the time and expense of unproductive results that are
due to common mistakes.

—Sean D’Souza, www.psychotactics.com
A: Bryan Eisenberg delivers the What, Why, and How of closing significantly more online sales. B: This is
where Eisenberg’s “Call to Action” philosophy gets tactical and you get better results from your website.
C: Increase online sales without spending a dime or too much time. This book reveals what it takes and
how to use it. D: “Keep on Testin,’” Eddie Kendricks and Robert Crumb. E: All of the above.
—Jim Sterne, Targeting.com
I have been living and breathing A/B and multivariate testing for the past few years, as illustrated in this
book, and I have found that 25 years of experience in this field and best practices still fails in compari-
son to the feedback we get from our audience through testing. So many “best practices” mistakes can be
prevented by simply putting the subject to a test and letting your audience tell you how they best interact
with your product.
—Dino Bernardi, Vice President of Customer Experience, Experian Interactive
You’ve avoided serious testing on your website because it’s expensive and hard to understand, right? Well,
Google Website Optimizer has made testing free, and this book just took away your other excuse. Put this
book down now, or you just might double your revenue and give your customers the website they’ve always
wanted. Bryan and John tell you why and how to Always Be Testing. If you don’t do it, your competitors
will.
—Mike Moran, Coauthor of Search Engine Marketing, Inc., and Author of Do It Wrong Quickly: How
the Web Changes the Old Marketing Rules
Behind every great sales team is a marketer who understands the power of testing. Let’s face it, sales teams
would get no leads without a marketing team that expertly knew how to convert a visitor into a “close.”
Always Be Testing by Bryan Eisenberg teaches marketers just that … how effective testing leads to a
higher close rate at the very first moment of customer engagement.
—Jim Kukral, Internet Marketing Consultant, www.jimkukral.com
90637ffirs.indd ii90637ffirs.indd ii 6/26/08 12:12:15 AM6/26/08 12:12:15 AM
Everyone talks about the importance of testing, yet few people really do it. Always Be Testing gives you
everything you need to become a testing expert. Tools, methods, theory, and practical examples, plus the
most important aspect: what to test and why. Keep this book on your desk and use it every day.
—John Marshall, Chief Technology Officer, Market Motive
Bryan and John have been optimizing websites for over a decade, helping their clients to earn millions

website works for our clients is to test, measure, and optimize our Persuasion Architecture®. A continuous
improvement process isn’t complete until we reach 100% success. . .in the meantime, read this book, learn
a whole lot about testing, see what others are doing, and get plenty of ideas to get going!
—Stephane Hamel, eBusiness Strategist and Web Analytics Consultant, Immeria.net
90637ffirs.indd iii90637ffirs.indd iii 6/26/08 12:12:15 AM6/26/08 12:12:15 AM
Always Be Testing needs to be every web managers’ and marketers’ mantra. Testing doesn’t have to be
expensive and complicated to be effective. Always Be Testing is more than a how-to guide. Bryan Eisen-
berg and John Quarto-vonTivadar provide a valuable, easy-to-use framework to decide what to test, how
to test, and how to implement the findings. This will be one of the most dog-eared books in your library.
—Seth Romanow, Vice President and Chief Marketing Officer, Rainmaker Systems
During slow economic times, retailers of all shapes and sizes are trying to find affordable ways to bolster
conversion, leapfrog the competition, and enhance the customer experience. The way to achieve this is
with proper A/B and multivariate testing, and Bryan and John do a spectacular job of mapping out test-
ing best practices that are easy to follow and implement. Bryan has been a great speaker and resource to
the Shop.org community for many years, and this is just one more example of Bryan sharing his wealth of
knowledge and experience. Always Be Testing—buy it, live it, love it.
—Larry Joseloff, Vice President of Content, Shop.org
Any marketer who understands the Web will keep a copy of this book on the desk at all times.
—Pinny Gniwisch, Founder EVP Marketing, www.ice.com
For years I have been telling anyone who will listen that “If you’re not doing some type of testing, you’re
not really doing web analytics!” Well, apparently my friend Bryan Eisenberg has been listening! Always
Be Testing is by far the most definitive work on the subject of website testing and optimization ever written.
More importantly, Bryan, John, and Lisa manage to make this critical-but-occasionally-tedious subject
palatable, using clear language and tons of helpful examples. So now instead of jumping up and down
and yelling at people to “start testing,” I can simply pass out copies of this great book. Thanks Bryan!
—Eric Peterson, Chief Executive Officer and Principal Consultant, Web Analytics Demystified, Inc.
Forget best practices says Bryan Eisenberg, every company has a different optimal strategy. To get your
suits and geeks on the same page, you need data, not intuition. Eisenberg’s new book shows how to use
Google Website Optimizer to get that data by refining website alternatives to optimize conversion rates.
With his trademark style and humor, Eisenberg makes the complex math, like adequate sample size and

Production Manager: Tim Tate
Vice President and Executive Group Publisher: Richard Swadley
Vice President and Executive Publisher: Joseph B. Wikert
Vice President and Publisher: Neil Edde
Book Designer: Caryl Gorska
Compositor: Kate Kaminski, Happenstance Type-O-Rama
Proofreader: Kathy Pope, Word One
Indexer: Jack Lewis
Cover Designers: Rob Kirby, Ryan Sneed
Project Coordinator, Cover: Margaret Rowlands
Copyright © 2008 by Wiley Publishing, Inc., Indianapolis, Indiana
Published simultaneously in Canada
ISBN: 978-0-470-29063-7
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic,
mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United
States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the
appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978)
646-8600. Requests to the Publisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc., 10475
Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4355, or online at />Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to
the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation
warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The
advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the
publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the
services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages
arising herefrom. The fact that an organization or Website is referred to in this work as a citation and/or a potential source of
further information does not mean that the author or the publisher endorses the information the organization or Website may
provide or recommendations it may make. Further, readers should be aware that Internet Websites listed in this work may have
changed or disappeared between when this work was written and when it is read.
For general information on our other products and services or to obtain technical support, please contact our Customer Care
Department within the U.S. at (800) 762-2974, outside the U.S. at (317) 572-3993 or fax (317) 572-4002.

Best regards,
Neil Edde
Vice President and Publisher
Sybex, an Imprint of Wiley
90637ffirs.indd ix90637ffirs.indd ix 6/26/08 12:12:17 AM6/26/08 12:12:17 AM
90637ffirs.indd x90637ffirs.indd x 6/26/08 12:12:17 AM6/26/08 12:12:17 AM
For Jeffrey Eisenberg
Your friendship, leadership, humor, and commitment made our mutual
success a reality.
90637ffirs.indd xi90637ffirs.indd xi 6/26/08 12:12:17 AM6/26/08 12:12:17 AM
90637ffirs.indd xii90637ffirs.indd xii 6/26/08 12:12:17 AM6/26/08 12:12:17 AM
More than 10 years ago we began our journey to optimize websites. When all we had
was imagination, our first clients believed us and let us test our theories on their websites and
businesses. We are as grateful to them for allowing us to always be testing as we hope they are
for the improvements. ■ To our past, present, and future clients, thank you for helping us
discover why people do the things they do and for putting that knowledge to good use. ■ We
couldn’t succeed without the FutureNow team. We’d like to thank Jeffrey, Bill, Howard, Brian
Number Two, Brian Number Three, Glen, Cinde, Esther, Jimmy, Robert, Marijayne, Karrie,
Ed, Tegan, Jared, Ryan, and Jeff Number Three for all their work behind the scenes. We are
indebted to our consultants who make our clients successful: Holly, Anthony, Jeff Number
Two, Melissa, Peter, Dan, and Brandon. Josh, Morty, and Ron, you’ll always be part of the team
even if you have moved on to new endeavors. Thank you so much. ■ We’d also like to thank
our friends and advisors Roy Williams, Pat Sullivan, Shmuel Gniwisch, L. Milton Woods, and
Mal Watlington for your insights and encouragement. ■ An old joke goes: “How do writers
feel about their editors?” The punch line is: “Like a fire hydrant feels about a dog.” We beg to
differ! The wonderful editorial staff at Sybex/Wiley—specifically Toni Zuccarini Ackley and
Dassi Zeidel—has been amazing to work with. We’ve appreciated the hard work all the profes-
sional staff members have contributed. In particular, Willem Knibbe has made so much of
this possible. ■ We must thank our friend and evangelist extraordinaire for Google, Avinash
Kaushik, for making the introduction to Willem and for his drum-beating efforts in the wider

ties while working hard to embark on his own life, and made me massive quantities of tea (the
sixth food group). These are the people who have most helped me understand the true meaning
of love.
90637ffirs.indd xiv90637ffirs.indd xiv 6/26/08 12:12:18 AM6/26/08 12:12:18 AM
About the Authors
Bryan Eisenberg Bryan is the cofounder of FutureNow Inc. (www
.futurenowinc.com; OTCBB: FUTR.OB), an interactive marketing optimization firm.
FutureNow helps businesses generate more engagements, leads, subscriptions, and
sales by using its unique framework, Persuasion Architecture. FutureNow intelligently
uncovers improvement opportunities and creates road maps to enable your business to
integrate technology, creativity, and marketing talent to continuously improve your
marketing efforts.
He is a coauthor of Call to Action and Waiting for Your Cat to Bark?, both of which
made the best-sellers lists of the Wall Street Journal, BusinessWeek, USA Today, and
New York Times. He is a sought-after speaker at major business conferences worldwide,
having keynoted events for Search Engine Strategies, Shop.org, Direct Marketing Asso-
ciation, MarketingSherpa, E-consultancy, Webcom, and the Canadian Marketing Asso-
ciation. Bryan is proud that FutureNow’s clients, including NBC Universal, GE, WebEx,
Overstock, and Dell, have consistently enjoyed dramatic improvement in sales using
FutureNow’s Persuasion Architecture

process. Bryan is also the publisher of the popular
marketing optimization blog GrokDotCom (www.grokdotcom.com).
John Quarto-vonTivadar Having worked on NASA’s Hubble
Space Telescope, when John says, “It’s not rocket science,” he does so with authority. An
inventor of Persuasion Architecture and one of the original shareholders in FutureNow,
John also spent many years trading on the floor of the Chicago Board of Trade. John
melds his business and technology background into his role as chief scientist at Future-
Now. He’s a regular speaker at seminars and conferences in North America and Europe,
having written multiple books and white papers on various technology and marketing top-

Chapter 10 ■ Jigsaw Case Study 120
PART II ■ WHAT YOU SHOULD TEST 125
Chapter 11 ■ Understanding What to Test 126
Chapter 12 ■ WIIFM: What’s in It for Me? 135
Chapter 13 ■ Unique Value Proposition/Campaign
Proposition 143
Chapter 14 ■ The Buying Decision 148
Chapter 15 ■ Categorization 154
Chapter 16 ■ Usability 158
Chapter 17 ■ Look and Feel 161
Chapter 18 ■ Searchability 163
Chapter 19 ■ Layout, Visual Clarity, and Eye Tracking 165
Chapter 20 ■ Purchasing 172
Chapter 21 ■ Tools 175
Chapter 22 ■ Error Prevention 178
90637ffirs.indd xvii90637ffirs.indd xvii 6/26/08 12:12:18 AM6/26/08 12:12:18 AM
Chapter 23 ■ Browser Compatibility 180
Chapter 24 ■ Product Presentation 183
Chapter 25 ■ Load Time 186
Chapter 26 ■ AIDAS: Attention, Interest, Desire, Action, and Satisfaction 189
Chapter 27 ■ Security/Privacy 192
Chapter 28 ■ Trust and Credibility 195
Chapter 29 ■ Product Selection/Categorization/Search Results 198
Chapter 30 ■ Navigation/Use of Links 203
Chapter 31 ■ Up-Sell/Cross-Sell 207
Chapter 32 ■ Calls to Action 210
Chapter 33 ■ Point-of-Action Assurances 213
Chapter 34 ■ Persuasive Copywriting 216
Chapter 35 ■ Content 222
Chapter 36 ■ Headlines 225

Headline Copy Strategies 29
Want to Run a Few More Tests? 31
Multivariate Testing: Sections, Variations,
and Combinations 33
More Website Optimizer Tests 36
Avoid Swimming in Data 42
Chapter 3 ■ Understanding Website
Optimizer’s Report 44
Combination Report 45
Page Section Report 48
Chapter 4 ■ Optimization Works: Examples
for Lead Generation and Retail 51
Finding Your Groove 52
Geico’s Disconnect 53
Doctor FootCare Rewind 56
Volvo Excavates Leads 58
Contents
90637ftoc.indd xix90637ftoc.indd xix 6/26/08 12:13:26 AM6/26/08 12:13:26 AM
Danica Meets Go Daddy’s Home Page 58
Dell’s Choices 59
Max Effect’s Show-and-Tell 61
Testing Is for Everyone 62
Chapter 5 ■ The Optimization Life Cycle 63
Miguel and Denise Need a New
Cable Service 64
Hitting the Optimization Wall? 66
Navigating the Phases of the Optimization
Life Cycle 69
Marketing Challenges 71
Chapter 6 ■ Always Be Testing in Action:

State Your Hypothesis 130
Let the Test Rip 130
Measure and Analyze Your Results 130
The Lowly Button 131
The Bottom Line 134
Chapter 12 ■ WIIFM: What’s in It for Me? 135
Competitive/Assertive 136
Spontaneous/Amiable 136
Humanistic/Expressive 137
Methodical/Analytical 138
One Page, Multiple Needs 138
Questions to Ask 140
Exercise 141
What to Test 141
Apply This to Your Site 142
Chapter 13 ■ Unique Value Proposition/
Campaign Proposition 143
Questions to Ask 145
Exercises 146
What to Test 146
Apply This to Your Site 147
Chapter 14 ■ The Buying Decision 148
The Complexity of the Sale 148
Scent Trails 150
Point-of-Action or Shop-with-Confidence
Messaging 150
90637ftoc.indd xxi90637ftoc.indd xxi 6/26/08 12:13:27 AM6/26/08 12:13:27 AM
Questions to Ask 152
Exercise 152
What to Test 153

Exercise 173
What to Test 174
Apply This to Your Site 174
Chapter 21 ■ Tools 175
Questions to Ask 176
Exercise 176
What to Test 177
Apply This to Your Site 177
Chapter 22 ■ Error Prevention 178
Questions to Ask 179
Exercise 179
What to Test 179
Apply This to Your Site 179
Chapter 23 ■ Browser Compatibility 180
Questions to Ask 180
Exercise 180
What to Test 182
Apply This to Your Site 182
Chapter 24 ■ Product Presentation 183
Questions to Ask 183
Exercise 184
What to Test 184
Apply This to Your Site 185
Chapter 25 ■ Load Time 186
Questions to Ask 186
Exercise 187
What to Test 187
Apply This to Your Site 188
90637ftoc.indd xxiii90637ftoc.indd xxiii 6/26/08 12:13:27 AM6/26/08 12:13:27 AM