Maskery
& Associates 2002
..
Bridging the gap between
market research and a successful product by using Success Metrics
Quantifying Customer Value
..
Presentation for SPIN
January 17, 2002
..
Helen Maskery, PhD.
..
+© Maskery
& Associates 2002
Tonight's Presentation
..
What success metrics are and how they define customer value
..
Why success metrics are successful
How to define success metrics to quantify customer value
..
How to use success metrics throughout the development process
..
Where success metrics contribute to software development processes
linked to the Capability
Maturity Model
..
Maskery & Associates 2002
Acknowledgements
..
The concepts and ideas
I'm presenting tonight are the result of work by many people
..
I'd like to acknowledge a few of them
..
..
..
..
Success Metrics: a Definition
..
Success metrics are objective,
measurable targets for the value customers want to experience
In essence, they capture the measures the customers will use when deciding
whether your product or process has delivered to their expectations, once
deployed in their environment
..
Success metrics also define what you need to deliver if you want to win their
business or gain support for process reform
Success metrics should be expressed from the customer's perspective
..
They should be technology-independent
They are often comparative
..
Success Metrics: Examples
Palm made it big in 1996 by identifying that their customers wanted something
more reliable, easier-to-use and costing less than other alternatives
..
This resulted in the following success metrics for the first
release:
..
..
No waiting (for common options)
Under promise, over deliver
..
Eliminate errors and error messages
By the latest release, the value proposition had evolved to:
..
..
..
Palm Latest Success Metrics
..
Simple:
There must be no complicated steps between you and your information - one step/
one touch; easy out of box account activation
..
Wearable:
Handhelds must be
carried everywhere, so you can get your information anywhere, anytime
- therefore must fit easily into pocket or purse without thought
..
Expandable:
Start with the basics,
make them affordable, and let the customers add the functionality they
want
..
Mobile:
..
Handhelds are used anywhere so you must be able to get your important
information anywhere anytime - this means internet and intranet
connectivity
..
Success Metrics: Examples
..
Must be able to complete
process X with 50 % less effort compared to today and in 1 hour instead
of the 10 days it takes today
..
Must be able to use process
X with products a, b and c which are aiming to hit the market Q2, 2002
Must not require documentation
..
Must be able to make a phone call without training
..
Must be able to get 50
people from city A to city B (500 miles apart in the same country) faster
and more cheaply than today
Must generate no waste which requires special disposal
..
Must use commercially
available components that will be available for the next five years
in countries X, Y, Z
..
Must be installable by someone with high school education, no training and no
special tools
How Do You Know if You've Got a Success Metric?
..
When you ask "Why?" you can answer from a customer's business
perspective
..
You can think of ways to measure the metric
When you ask "So what?"
you can answer with a list of benefits the customer expects to get
..
When you look at the
metric, you can identify what parts of the customer's experience will
be impacted
..
e.g. installation versus usage
When you look at the metric, it's technology independent
..
When you're in a development
team meeting, everyone can think of ways in which that success metric
will help them
..
..
By suggesting how to verify the product
..
By thinking of alternative way to meet the objective
..
..
Why Success Metrics?
Solid focus on customer value rather than technology
..
Enables you to be proactive rather than reactive
..
Customer Satisfaction (CSAT) is reactive
Informed decision-making at all levels for all aspects of the product or
process
..
Explicitly balance your business goals with your (potential) customer
values
Explicit, predictable and quantifiable risk management
..
Solid project management boundaries
Reduces product definition and development churn
..
When used as part of a development process, success metrics create a
team-rallying mantra that often leads to patent-receiving
innovations
..
Delivers repeatable successes
When to Use Success Metrics
..
Existing product, service or process
Product evolutions, feature rollouts, low customer loyalty scores, customer
complaints, lost market share
..
New product or process development
..
existing product category or process in the market, or
a human goal that is being accomplished by other means
..
Need to reduce development risks
Market uncertainty (shifting needs, customer requirements)
..
Many market unknowns (new market, new customers)
Churn in product development
..
Unclear or shifting objectives
Lack of focused customer involvement
..
Risk of missing opportunity window
..
How to Use Success Metrics
..
Projects are defined with boundaries or limits, beyond which the project will
not be meeting its defined goals
..
All boundaries should be objective and easily measurable
Quantifying Customer Value is critical to managing that boundary – Success
Metrics define Customer Value
..
<0449
this is the boundary that is often the fuzziest
Schedule
Customer value
..
Total Product
Cost
Total Project cost
..
Product
..
performance
Project
Parameters
..
Customer Value is the boundary that can have the most impact on product
success
Source: Amy Dillon
..
Success vs. Quality Metrics
Guarantees the "right" product to meet customers' needs
..
Technology independent
..
Can be used to compare your service/product to a competitor
..
Will guarantee a quality product
Specific to the product
..
Standard performance measures
Can be "defect""focused
..
Not always directly comparable to competitors
Product Quality Metrics
Success Metrics
..
Maskery & Associates 2002
Focus on Customer Success
The bottom line with
Success Metrics is that they focus the whole team on what’s important
to the customers
..
Will the product perform and deliver as the customer expects when it's in
their hands?
You succeed when customers get the value they expected
..
Time for a Break?
The Quick "How To" Guide for Success Metrics
..
The almost nitty gritty details
..
User-Centered Design Process
Success metrics are most effective when used as part of a customer, citizen or
user-centered design process
..
User-centered design (UCD)
The design methodology which brings customer and user representation into the
design process on an equal footing with technology, schedule, project, product
rollout, marketing, financial and other considerations
..
Complementary to other development methodologies
User-centered design brings focus onto making sure all types of users can
successfully complete their tasks at any stage in their experience with a
product or process
..
Success
..
Metrics
Definition
..
..
The Four Phases
..
The Four Phases
..
Project timeline
..
Phase 1
..
Success Metric Definition
Success
Metrics
..
Definition
..
..
..
Success Metric Definition
..
Project timeline
..
Create a core project
team of key people from all aspects of the development process
..
Get a common view across the development team of what you know,
including:
..
Customer's operational environment
Profiles of all user types
..
Usage scenarios (criticality, frequency, ...)
Baseline performance of comparison products, categories, processes, etc (high
level)
..
Opportunities for making improvements, delivering customer value
..
What you think the value proposition is
..
The other project boundaries (schedule, project cost, product
cost, etc
..
Done through meetings, one-on-one work sessions, brainstorming sessions,
sub-teams, etc
1.2 Generate Success Metrics
..
Generate success metrics
With the customer(s) through a planned customer interaction process
..
By the core development team
If done this way, must be ratified by customers
..
Same process either way
less risk if generated by customers
..
can take longer if generated by customers
Interviews, focus groups, brainstorming sessions to collect and then delve
beyond "easy to use", "intuitive", "simple" to find out why these
attributes are being listed
..
Keep asking "Why?",
"So what?", "How will you know?"
..
Get examples they
think delivers what they're looking for, and then delve...
..
1.3 Ratify Success Metrics
Ratify the success metrics with customers
..
<0872
Especially if they were developed by the development team
Customer interaction process to check:
..
Your understanding of the customer's universe
..
Most/all of that information you collected during the homework phase - suitably
summarized and professionally presented
..
Your understanding of what they think is of value to them
Very Very High Risk if you don't verify with customers
1.4 Agree on Success Metrics
..
Meeting with key decision-makers
Are these the success metrics that we want to deliver to?
..
If we develop a product to meet these metrics, does it fit our business plan
and goals?
..
Revisit and reaffirm the other project boundaries
..
If obvious changes need to be made at this point, make and agree them as
well.
..
If all say "Aye", get signatures and then allow the project team to continue to
Phase 2
..
Phase 2
Product Requirements Definition
Success
..
Metrics
Definition
..
..
..
Product Requirements Definition
..
Project timeline
2.1 Define the Product Requirements
..
Developing the product requirements definition is the same process as you
currently use except for several VITAL additions:
Use brainstorming techniques to explore other ways of delivering to the success
metrics
..
Can do this because success metrics should be technology
independent
..
Develop a view of the "end-to-end process" associated with the
product
..
Frequent project team meetings to ask the questions:
Will this "concept" deliver to the success metrics?
..
Does it stay within all the other project boundaries?
..
Hold meeting with key decision-makers to get OK to proceed to
Phase 3 - Build
the Product
..
Phase 3
..
Build and Release
Success
Metrics
..
Definition
..
..
..
..
Build and Release
Project timeline
..
<1084
3.1 Define a Set of Usage Scenarios
These scenarios will describe key user tasks which can be used to guide the
design process
..
These scenarios should include frequent as well as critical
tasks
They should be tasks which will show the impact of the customer success
metrics
..
These scenarios need to be agreed to by the project team as a whole
Provides a baseline on which all design discussions can be based
..
Reduces the possibilities of churn due to "worst case scenario what
if-ing"
3.2 Define a Set of User Profiles
..
Like the usage scenarios, these user profiles are used to base all design
decisions around the user interface and how to deliver to the success
metrics
..
Get agreement across the whole project team on these scenarios
Sign in blood, if necessary!
..
3.3 Establish the Comparison Baseline
..
In Phase 1, the baseline performance of comparison products or processes was
completed
This baseline needs to be revisited, updated and taken to a deeper level as a
result of the decisions in Phase 1 and 2
..
The baseline is needed to ensure that as design and development progresses, the
yardstick doesn't waggle in the breeze...
This then enables the question to be answered
..
Will this deliver to the success metrics?
..
3.3 Design, Build and Release
Follow your usual design and build process with the addition of that vital
question
..
Will this product deliver to the customer success metrics?
Keep checking with customers
..
To get input on tradeoffs you might need to make
To manage their expectations
..
Phase 4
..
Check the Value Delivered
Success
Metrics
..
Definition
..
..
..
Check that the value has been delivered
..
Project timeline
..
4.1 Validate the Delivery
..
This is a crucial step which ensures that you delivered what the customers were
expecting
Again, your normal field trial and post-installation follow-up techniques
should be used with the addition of a clear focus on:
..
Has this product delivered to the success metrics?
If the customers were involved in the development of the success metrics, you
can use the success metrics to frame the questions.
..
If the customers weren't involved in the success metric generation and/or
ratification, then the questions will take a different form
..
Use terminology understood by the target audience.
..
The Return on Investment
How much to invest in these activities?
..
Levels of Activities
Lite
..
Generate success metrics internally using secondary sources of
information
..
Ratify with customers
Full
..
Generate and ratify success metrics with customers
..
Ultra
Explicitly agreed, shared development objectives based on the success metrics,
between your customers and your organization
..
The Ultra Level
Vendor proposes
a value proposition
..
Customers & Vendor quantify actual
product performance during lab test or field trial
Customer & Vendor acknowledge shared
development objectives based on the success metrics
..
Source: Amy Dillon
Check
..
Vendor helps customers define their success
criteria
Vendor commits to meeting specific success
metrics, quality and features in a Plan of Record
..
Vendor shares product design specifics
and ongoing status updates
Customers provide ongoing input, validation
and lab test or field trial support
..
Vendor commits to the Ultra Level
A business opportunity is identified
..
Ongoing Customer/Vendor touchpoints
Pre-definition
..
Definition
Product Req
..
Build
..
Vendor delivers what is needed when it is needed
When to Use Which Level
..
Low to mid-end consumer products
Simple web sites and applications, and simple software applications
..
Low to mid-end enterprise products
..
Short iterative projects where there are few negative consequences of getting
it wrong
Mid- to high-end enterprise products
..
Carrier products
Web and software applications
..
Government projects which will "impact" large numbers of citizens
..
Large R&D budget projects; high revenue generating products
Negative consequences if get it wrong
..
Life- or business-threatening consequences if get it wrong
..
High-end consumer or enterprise products
Companies which have a few key customers
..
lite
..
Full
Ultra
..
Levels
..
Situations
Other Factors to Consider - 1
..
Organizational
Presence of user-centered design (UCD) Champion in a position
of power
..
Ability to absorb costs that could be avoided
..
Pay less now; pay more later
Ability to manage UCD contributions
..
Project Team
..
Familiarity with UCD / Openness to UCD
Workloads and time pressures
..
Other Factors to Consider - 2
..
The Project
Point in project life cycle
..
Degrees of decision freedom
Size and location of project team(s)
..
Point in Product's Life Cycle
Overall $ investment in project
..
Other Factors to Consider - 3
..
Project's Output
Criticality of tasks being completed
..
<1544
Life-threatening or business-threatening
Installed base / number of people affected
..
Ability to hand-hold customers/users through early releases
Target market and expectations
..
Customer's usage and / or buying decision making priorities
Risk of "missing" goal first time
..
Ability to correct "mistakes"
Speed of expected usage / sales ramp
..
Distribution of users / customers
Success Metrics and CMM
..
Success metrics and user-centered
design can be a significant contributor to Level 2, 3 and 4 of
the Software Capability Maturity Model
..
Initial (1)
Managed (4)
..
Optimizing (5)
Software configuration management
..
Software quality assurance
..
Software project tracking and oversight
Software project planning
..
Software subcontract management
..
Requirements management
Organization process focus
..
Organization process definition
..
Training program
Integrated software management
..
Software product engineering
..
Intergroup coordination
Peer reviews
..
Repeatable (2)
..
Defined (3)
..
Summary
Conclusion
..
If you use success metrics, you will be taking control of defining and
delivering Customer Value
..
Success metrics are an extension of activities that you're probably already
doing in your organizations
..
It does not cost an arm and a leg to define them, nor will it delay your
projects
..
Key thing to remember - you can't do success metrics without some kind of
customer interaction
Start small and build
..
Learn to walk before you try to run - in other words, follow the advice of
the CMM
Even Level 1 organizations will get benefits from defining success
metrics
..
Back up slides
User-centered design perspective
..
User-centered design brings focus onto making sure all types of users can
successfully complete their tasks at any stage in their experience with a
product or process
..
need/want
customize
..
install
choose
..
upgrade
use
..
Buying it
Getting it
..
Using it
purchase
..
The Yah Buts - 1
If you only take development costs or the cost of the components into account,
this may be true. However, if you look at the costs of ongoing user
support, warranties, documentation, limited future product evolution and
possible lawsuits, it may be cheaper to bite the bullet now.
..
It will cost too much to include in the product
Actually, we can save time for your developers by getting a prioritized list of
functionality and getting the design right early on in the product development
cycle so they don't have to waste their time redesigning the product later
on. We can also reduce development churn by delivering a prototype and/or
high level design at the beginning of the implementation phase.
..
You'll take up too much of my developers time
Your answer
..
Excuse
..
Do you have time for supporting user complaints about the product? Do you have
time for the bad product reviews which will come if the reviewers can't
quickly experience the benefits of your product.
Is your product aimed at the consumer market? If yes, then usability is
very important. Consumers don't have much patience and they expect to
be able to use something immediately. They have a huge range of capabilities,
previous experiences, and will often use your product for things you
never intended...
..
I don't have time for this stuff
User interface design is part of software development
..
I can't afford anything
other than software development
The Yah Buts - 2
..
Your answer
..
Excuse
Involving your customers
professionally and appropriately in your design process heavily outweighs
the risk posed by not knowing the technology, plus we always take technical
experts from your company with us if we're dealing with a complex technical
domain
..
You don't know the technology
and will give my customers a bad impression of my company
..
We're used to working
into new domains, we ramp up quickly, we take technical experts with
us when we're talking to customers, and often, being new to the domain
means we come at design with a fresh perspective and no "this is how
it’s always been done" blinkers
..
You don't understand
the domain and therefore can't ask the right questions
..
We don't promise the
users anything. We develop and understand their expectations for the
product and help you prioritize this information in the product development
process.
..
Often, as a consultancy
that you've retained, your customers have no expectation that we can
influence your product development! We can also do "blind" studies
where the customers have no idea which company we're representing.
..
You'll make commitments
to the users that this company can't deliver to so we're not involving
the users in this project!
The Yah Buts - 3
..
They don't know how to design a product, but it is possible to get a pretty
good idea from them of what they want a product to do
..
Users don't know what they need
We can help you prioritize what users really need in the
product
..
Users ask for everything
..
Has that been translated into the product design? Can you identify the
product attributes that will deliver what the users want? Can you build a
prototype or write a specification which will clearly communicate what the
users want to the development team? Can you keep this prototype or
specification up-to-date as design progresses and tradeoffs get
made?
..
I know what the users want
There's always more than one way to deliver to the customer's wants and needs,
let's see what's under "our" control and find an alternative
way
..
Can't do it technically
What about planning for the next release? Do you know what problems they are
going to face in this release? Is your customer support line prepared for
the kinds of user questions they might get?
..
It's too late, the product is being released tomorrow
You'll pay more in customer support How much money are you spending on customer
support, training, documentation, etc?
..
We'll take care of the users once we've got the product out the
door
..
Your answer
Excuse
..
The Yah Buts - 4
Your answer
..
Excuse
..
Users like what they know and don't want to have to learn new stuff --
especially if their status in their job is dependent on their perceived
expertise by their peers. Participative design and careful introduction
of the changes will result in few problems. Customer and user satisfaction will
increase when a user-centered approach is taken.
..
My users won't let me change the user interface
..
With the large installed base, how do you introduce new functionality without
changing anything?
There's a large installed base that means I can't change
anything
..
Even power users benefit from a well designed user interface. You can
design a product to meet the needs of both infrequent and new users, as well as
frequent expert users.
My users are power users and only want a command line interface
..
Are you representative of your full user population?
I can use it
..
True, but has your product been designed so that the user can recover quickly
and easily from the mistakes?
..
Users will always make mistakes
Users are stupid
..
Training costs money and we know from experience that the majority of people
don't get training, even when they're supposed to.
..
Users will be trained so therefore this doesn't have to be easy to
use
The majority of users don't read the documentation so you can't rely on
that
..
We can put it in the documentation