Category: Evidence-based Management

I announced a few weeks in a post that, with some great help from Daphne Chang and Paul Reist at the Stanford Business School Library, and from Management Science & Engineering PhD student Ralph Maurer, Jeff Pfeffer and I had launched www.evidence-basedmanagement.com. We have continued to work on the site, and the greatest improvement has been the work that Daphne and Paul have done in identifying an astounding range of evidence-based movements, and providing links to key information. Check out their list of other evidence-based movements, which include evidence-based conservation, evidence-based crime prevention, evidence-based government (imagine that!), evidence-based medicine (the most completely developed), evidence-based social work, and evidence-based software engineering.

Certainly, the meaning of an evidence-based approach means different things in different places. But the common theme I see across all these movements — and why the term evidence-based is so value and is spreading — is that people across all these areas want the best decisions made and implemented because money and lives are at stake, and rather than taking steps that are based on what has always been done, what sustains the power structure, or provides the largest financial payoff to the most persuasive salespeople, I believe these movements represent a desire and commitment to do the right thing.  Sure, ideology and greed will always bias the decisions that people make and implement, but pressing people to face and follow the best facts is a new, noble, and sometimes effective hurdle.  Indeed, there are signs people with best evidence do and can win — and to see how it is done, if you haven’t already, check out Al Gore’s brilliant and simple arguments about global warming in An Inconvenient Truth.

I’ve written before about how handwashing by medical care workers is one of the most well-documented preventable causes of death and disease in health care settings. One hospital CEO, for example, told me that he wore a button that said "It is OK to Ask," to encourage patients to ask doctors if they had washed their hands — a practice the doctors didn’t like, but that did have positive effects.  Last Sunday’s New York Time Magazine had a splendid article by Dubner and Levitt called "Selling Soap."    The story contains many lessons that go far beyond hand-washing and have implications for evidence-based management on many other things. Consider four:

1. Self-report data can be worse than useless. They describe an Australian study where 73% of doctors reported washing their hands, but when the docs were observed by a researcher only 9% were seen washing their hands. The lesson is that you need to check out what people say, interviewing the people you are trying to help and change produces very suspect data.

2. The story focuses on Cedars-Sinai Medical Center and efforts to get physicians to wash their hands. Nagging doctors with emails and posters didn’t work. But the Hand Hygiene Safety Posse did — they roamed the halls and when they saw a doctor washing his or her hands, they handed out a small reward on the spot, a $10 Starbucks gift certificate. This had substantial effects, getting compliance up to 80% from 65% — but they were aiming to hit 90% for a pending inspection.

3. The way they finally got compliance up to nearly 100% was to have a group of the hospitals more influential  doctors each press their palms on plates that were cultured and photographed, which resulted in images that "were disgusting and and striking, with gobs of colonies of bacteria." One of these was used as a screen saver that was placed on every computer in the hospital. The power of this last intervention not only provides lessons about hospitals, it is a lesson about how to implement any other change that people "know" is good, but aren’t doing:

A. They started by changing some of the most powerful and persuasive people in the system — opinion leaders and people with high status are important for sparking change.

B. They spread the information to everyone in the system, so they could all keep an eye on each other.  This made the norm open and easy to monitor — and vivid.

C. They made the dry statistical information very vivid. The disgusting picture is the perfect kind of image that sticks in people’s minds — check out Heath and Heath’s Made to Stick. This is one of their kep principles.

D. Finally, they created the kind of social pressure where if people did not follow the norm, it would lead to loss of face. Economists and psychologists like financial rewards, and as this tale shows, incentives are effective to a point. But if you can find ways that people feel compelled to follow the evidence, or they will suffer humilation in the eyes of the people around them, that is even more effective. It is a lot easier to measure and dispense financial rewards, but if you want to change human behavior, it is hard to beat pride and shame — as sociologists including Erving Goffman   have demonstrated.

Finally, this story, to me, is another sign that focusing on managing the system is more important than devoting excessive energy to bringing in star talent — that the law of crappy systems trumps the law of crappy people.  This hospital clearly has many smart and well-meaning people. But the compliance rate was only about 65% until management started changing the system, first through rewards (which got them up to 80%), and then through those disgusting pictures (which pushed them to close to %100). These are all the same people, they were just managed differently.

Our evidence-based management website has been up for a week, and thanks largely to the hard work by Daphne Chang and Paul Reist at the Stanford Graduate School of Buinsess Library, a lot of new content has been added. We have a guest column posted by past Academy of Management President Denise Rousseau, who has written two recent articles on evidence-based management and gave her presidential address last year ont he subject.  We will add a new column every week or so, and are expecting contribtions from people including Michael Dearing, a former eBay marketing executive (and now Associate Consulting Professor in Stanford d.school) and Iowa Sociology Professor Michael Lovaglia — you may recall my blog about Lovaglia’s Law . There are also quite a few new articles and news updates. Please visit and make comments and suggestions as the site is brand new.

I exchanged emails with a manager I know today who
was concerned that her company might be heading toward the Otis
Redding Problem in its compensation system. As I said in my post last week, “ We call this The
Otis Redding Problem. Recall the line from his old song: Sitting By the Dock of the Bay,
“Can’t do what ten people tell me to do, so I guess I’ll remain the same.”
That’s the problem with holding people, groups, or businesses to too many
metrics: They can’t satisfy or even think about all of them at once, so they
end-up doing what they want or the one or two things they believe are important
or that will bring them rewards (regardless of senior management’s strategic
intent).”

There are lots of reasons that this problem happens
in organizations, but – at least based on those I’ve studied and worked with –
four jump-out:

1. There are too many groups that have medium power
– so everyone gets a metric to show that what they do is important, but no one
has the power to kill a metric.

2. Senior management does not understand its
strategy, especially is strategic priorities. So they treat everything as
moderately important – the result is that employees can justify virtually
anything they do as important. 

3.  Senior
management does not really understand what the organization’s actual business
model is or what it should be. This means that they can’t figure out the few
key elements that drive many things, so they keep adding more and more items to
the list in hopes that they will figure it out eventually.

4. Senior management can’t say no.  Even if they can articulate their priorities, senior
management lacks the courage to make enemies.  So they cave-in when people act hurt or
threaten to leave the organization unless metrics are added that make them and
their kind look important. The result is that everyone ends-up being unhappy. At one organization I worked with, there was
endless argument over compensation because each general manager would focus on
the subset they performed well on and ignore those metrics where performed did
poorly. Everyone seemed to be #1 at something and everyone used that as
argument that they deserved more compensation. 

Leaders who lack such courage might recall the old
Bill Cosby quote:
“I don’t know the key to success, but the key to failure is trying to please
everybody.”   Otis Redding’s solution was to “remain the
same” because he couldn’t please 10 different people. That is a rational
response to a bad system.  Things get
even worse when you try to please everyone – at least Otis pleased himself!

 

The strong reactions to Marge’s Asshole Management Metric not only got me thinking about the difficulties of managing nasty people (and the times when there might be a need for people to get nastier to defend themselves, an excellent if unfortunate point that Marge made), it started me thinking about metrics.  I have also been thinking about metrics lately because one of my doctoral students has been struggling with the problem of how to measure the effectiveness of user-centered design practices.

Both Hard Facts and The Knowing-Doing Gap review a lot of research about metrics. I confess that wading through the "best practices" claims, theory, and evidence in this area isn’t easy. There is so much written and so many strong beliefs about how performance should be measured and how to motivate people (most unaffected by evidence) that figuring-out how to design an optimal  measurement and incentive system is tough, perhaps impossible.  We’ve even had compensation consultants tell us that it is such a great area to work in because clients always call you back for more work — because no matter what kind of system you help them design or that they install, it never works quite right.

I can’t claim to have any magic answers either (and don’t believe anyone who tells you they do, like the authors of Topgrading or The War for Talent). But there is one guideline that few companies follow, even though they nearly all know they should (there is a reason we wrote a book called The Knowing-Doing Gap): If you measure and reward people on too many different dimensions, they are pulled so many different different directions (since different criteria are often unrelated or negatively related), that it is simply impossible to adjust 10 — or in some cases — 100 diverse behaviors in response to the system.  The result is that that your measurement system becomes useless or worse.

We call this The Otis Redding Problem. Recall the line from his old song: Sitting By the Dock of the Bay, “Can’t
do what ten people tell me to do, so I guess I’ll remain the same.” That’s the problem with holding people,
groups, or businesses to too many metrics: They can’t satisfy or even think about all of them at once, so
they end-up doing what they want or the one or two things they
believe are important or that will bring them rewards (regardless of senior management’s strategic intent). Yet many organizations fail to
implement this well-known and common sense principle.

And the rise of balanced score cards –- which have
many virtues when done right -– have made this problem even worse. As we say in The
Knowing-Doing Gap “In principle, the balanced scorecard makes a great deal of
sense. Rather than just measuring and evaluating
managers on the financial performance of their units, which largely reflects
what has happened in the past, the scorecard emphasizes getting ready for the
future.” But one of the most common unintended side-effects is that people are assessed and rewarded with an absurdly long list of metrics. This happens because so many different groups in the organization want “their” metric to be
measured and linked to incentives and because the groups that design and implement these
systems (often in HR) often don’t have enough
power or will to say “no.” Although HR certainly doesn’t deserve all the blame. I once
went to a talk at Stanford where a supply chain consultant and his client
proudly announced they had just added their 100^th metric to the
performance evaluation system (and this was just for supply chain performance).  I gently asked the executive who was trying to use the system if it created problems and he admitted that it was confusing and he ended-up focusing on just one or two. 

Similarly, we describe another example of the Otis Redding problem in The
Knowing-Doing Gap:

A Harard Business School Case study reported that branch managers at Citibank [now part of Ctigroup]faced the following scorecard measures in 1996:

Financial:

Revenue

Expense

Margin

Strategy Implementation:

Total households

New to bank households

Lost to bank households

Cross-sell, splits, mergers households

Retail asset balances

Market share

Customer Satisfaction

Control:

Audit

Legal/Regulatory

People:

Performance Management

Teamwork

Training/Development

Self

Other

Employee Satisfaction

Standards:

 Leadership

Business Ethics/Integrity

Customer
Interaction/Focus

Community Involvement

Contribution to Overall
Business

Each component
of the Scorecard was scored independently into one of three rating

categories: “below par,” “par,” or “above par”.

Pfeffer and I did an interview with a New York-based Citibank branch manager around that time — we met him because bank executives reported that
he was one of their best managers. The manager told us that, although it required a lot of time to
fill-out all the forms and go to all meetings held under the balanced score-card system, he viewed it as a flavor of the month. He described it as just another program that he had to pretend to care about until management tired of it and became enamored with the next fad (which he would have to pretend to care about as well). This manager also told us that – despite all of the hype from executives about the importance of the new balanced-score card system – he believed
that only one thing still mattered at the bank was generating short-term profits. So that
was what he focused on doing, which was why he was so highly regarded.

Part of the problem was that he didn’t believe that management was committed to the system, because he had endured one internal program after another.  But this program also wasn’t taken seriously because, when a system tries to measure and link rewards to everything, it means nothing because human beings can only think about and do so many things at once.   

The upshot of all this is that I don’t know the secret to designing a successful performance management system
and I am not sure that anyone else does either. But I do know that if you fall
prey to the Otis Redding problem, your system is doomed to fail. 

 

I am pleased to announced that www.evidence-basedmanagement.com went live earlier this week. Jeff Pfeffer and I have been working on this all summer with two fantastic librarians from the Stanford Business School, Daphne Chang and Paul Reist (see my post In Praise of Librarians) and the equally fantastic Ralph Maurer, a doctoral student in the Department of Management Science & Engineering who works with me at the Center for Work Technology and Organization.

The site contains information about what evidence-based management is,examples of evidence-based research and practice, a blog (which has a couple posts and will soon have more),links to information about evidence-based movements in other areas such as medicine and education and course outlines from faculty who are teaching management courses that take an evidence-based management perspective.  Along those lines, you can see the draft outline for the course the Ralph Maurer and I will be teaching in Winter, Organizational Behavior: An Evidence-Based Perspective, which includes links to articles and examples as well.  We also will start having regular guest columns from both academics and practioners about topics that are pertinent to evidence-based management.

I hope you will visit the website for a few reasons. First, as it is a new website, we would be grateful for suggestions about how to make it more compelling and useful.  We just view this as a prototype and will work to make ever better. Second, if you have any materials or information about evidence-based approaches, please pass them our way, and will organize the information and pass it on to others.  Third, finally, please visit the blog and make some comments.

Pfeffer and I have been working together for years, and our motto is "when two people agree, one of them is unnecessary! So please, feel free to explain — using logic and facts of course — why we and others have spouted half-truths or total nonsense. We don’t promise to accept your point of view, but we do promise to respect it!

The answer is that both have had books written about their “leadership secrets.” 

When Jeff Pfeffer and I were writing Hard Facts, Jeff put “leadership secrets
of” into the Amazon search engine, and these were the first two to books to
pop-up. I just did it again, a few months ago, and Leadership
Secrets of Attila the Hun came up #1 and The
Leadership Secrets of Santa Claus came up #2.  The idea of getting leadership advice from
either an infamous tyrant or a fictional character is strange enough as it is, but
(I haven’t read the books, I confess) but the contrast between the two is
pretty funny.  Jeff and I took to
collecting other contrasting titles, with my one of my favorites being
Love is the Killer App versus Business
is Combat.  And don’t forget Tony
Soprano on Management: Leadership Lessons Inspired By America’s Favorite Mobster.
How on earth is a manager supposed
to know who to believe?

All
this would be pretty funny if it wasn’t for the fact that badly managed organizations
do so much harm. Badly managed hospitals have higher mortality rates. Badly
managed military operations waste soldier’s and civilian’s lives.  Badly managed organizations put people out of
work and lose money for shareholders. And
a growing literature on bullying shows that leaders who act like Attila the Hun
damage the physical and mental health of their people, reduce their motivation,
and drive them to find other jobs. 

AsJeff
and I argue, there actually is decent evidence out there about how to make many
managerial decisions; the problem is that is routinely ignored and managers,
boards, and consultants are almost never held accountable for ignoring it. For
example, research
by Columbia’s Joel Brockner and others documents that displaying lack of sensitivity
during layoffs not only has negative effect on those who lose jobs, it also
damages survivors who witness the unfair treatment: These studies suggest that
Radio Shack’s to decision to notify
400 employees VIA EMAIL that they were being laid-off will undermine the
motivation and productivity of those who survive the cuts and their mental
health too. Or as I discussed in an earlier
post on mergers, rather than listening to investment bankers who want
mergers to happen because they make money no matter how things turn out,
perhaps the time has come for executives and boards to document – – based on
the large literature on mergers –- why (other than their irrational
overconfidence) a proposed merger is likely to succeed even though most do not.

If
you go to a doctor and he or she recommends a procedure that existing studies
show will make you sicker or increase your chance of dying, they call it malpractice.
Isn’t it time to start holding managers and their advisors to the same
standards? 

Simon
Caulkin writes a management column in the Observer,
a UK-based paper, and has written a couple columns that draw on ideas from Hard Facts. His first one, back in March, has the kind
charming title that you don’t get in the states “Bosses in Love with Claptrap and Blinded by
Ideologies.” Caulkin recently wrote a column called “You Could be a Genius – If Only You had a
Good System,” which draws a bit on our chapter on talent in
Hard Facts. He uses recent examples of failures by British
sporting teams to show how coaches and critics focus on “naming and shaming”
individual athletes, rather than on problems with the system.

This
tendency to look for individual goats – and heroes – isn’t just a problem that
permeates the world of sports. It is
reflected in many misguided ideologies and management practices, which
focus excessive energy on hiring stars and weeding-out mediocre and poor
performers, and insufficient energy on building a great system that enables
most competent people to succeed.

I agree
– and can show you evidence – that there are huge differences in individual
skill and ability in every occupation. BUT
we’ve also got a lot of evidence that ordinary people can perform at top levels in a
well-designed system, and even a superstar is doomed to fail in a bad
system. Unfortunately, HR and too many
other executives believe the advice in books like The
War For Talent. (In fact, one of the authors is now head of HR at eBay.. perhaps another reason to short the stock). This is one of
the worst management books ever written in my opinion: There is bad evidence
from the authors’ own research, no mention of a massive body of research that
contradicts many of their claims, and excessive claims are made that if leaders
follow the authors’  advice, they can
expect “expect a huge impact in a year.”
 

I will focus on just one claim from this bad book. I quote the authors, “We call it the Rule of Crappy
People: Bad managers hire very, very bad employees, because they are threatened
by anyone who is anywhere near as good as they are.” This claim is bold, but  can’t be supported by any systematic research
that I can find.  There is evidence that people hire others like themselves, so a reasonable inference is that crappy people will hire equally crappy people — but there is no direct evidence on that hypothesis.  I spent weeks and weeks trying to find even a
hint that a single article in a peer reviewed journal supported the belief that
bad performers systematically hire even worse performers.  It is one of those management myths that don’t
appear to have any empirical basis. 

The
worst part about focusing on keeping out crappy people, however, is that
it reflects a belief system that “the people make the place.” The implication is
that, once you hire great people and get
rid of the bad ones, your work is pretty much done. Yet if you look at large
scale studies in everything from automobile industry to the airline industry,
or look at Diane Vaughn’s fantastic book on the space shuttle Challenger explosion and the well-crafted
report written by the Columbia Accident Investigation Board , the evidence is clear:
The “rule of law crappy systems” trumps the “rule of crappy
people.”

Sure,
people matter a lot, but as my colleague Jeff Pfeffer puts it, some systems are
so badly designed that when smart people with a great track record join them,
it seems as if a “brain vacuum” is applied, and they turn incompetent. Jeff
often jokes that this is what happens to many business school deans, and
indeed, these jobs have so many competing and conflicting demands that they are
often impossible to do well.

If
you want to see a more chilling and systematic analysis, read the chapter by the
Columbia
Accident Investigation Board that compares the Challenger explosion with the Columbia
accident. Sally Ride, the first American
woman in space, comments about the “remarkable echoes” of the Challenger
accident that can be seen in the Columbia fiasco. Indeed, there was close to 100%
turnover at NASA between the two accidents, but the system was largely unchanged.
According to the report (which is more useful and better written than most
management books), NASA remained a dysfunctional bureaucracy where, rather than
deferring to people with the greatest expertise, administrators believed that
“an allegiance to hierarchy, procedure, and following the chain of command”
decreased the odds of failure. People
with greater power ignored and stifled, and overturned recommendations people
with more expertise but less power. As a
result, the Board warned “NASA’s problems cannot be solved simply by
retirements, resignations, or transferring personnel.”

As we discuss in Hard Facts, an interesting contrast is
The U.S. civil aviation system. It is one the safest in the world and has
become even safer over time, partly because of its accident and incident
reporting system. This system permits pilots (and others) to report incidents to
the Federal Aviation Administration such as near misses and equipment problems.
The agency follows up on these reports and takes action to repair root causes
of problems.

There are smart people in
both the FAA and NASA (in fact, people from NASA run key parts of the accident
reporting system), but one system is difficult to succeed in because it is
crappy. And the other is comparatively
easy to succeed in because it is well designed. Again, I still believe that people matter. The very best organizations
have both smart people and well-designed systems – Google seem to qualify and
so does Cisco. 

I’ve spent a lot of time this summer on two
web-based projects. The first one is
this blog. The second is developing www.evidence-basedmanagement.com,
which will go live in a week or so. Jeff
Pfeffer and I decided to develop a website and blog to bring together
information about evidence-based management , as well as people in industry and
academia who are interested in fueling an evidence-based management movement.
We soon recruited PhD student Ralph
Maurer (who will be teaching a class on evidence-based management with me
next year), and two librarians from the Stanford Business School Library
to help us with this adventure, Daphne Chang and Paul Reist.

I confess that, when we started this adventure, I
was delighted to have help from just about any smart person to get it
done. I didn’t really think much about whether
their skills as librarians would help. As the summer has progressed, however,
I’ve realized that the rise of the web, blogs, Wikipedia, and all that easily
available information might reduce the need for trips to the library for some
of us, but – although their roles are changing – it has also made the need for
librarians even greater than ever.  There
is an ever growing pile of information out there and it keeps getting harder
and harder to tell what is true and what is not.

Librarians like Daphne and Paul have become even
more important because they care about facts and accuracy and are extremely skilled
at finding sources – so they can actually help stop the spread of claims that
are wrong or inaccurate and can find the true sources of ideas and claims that
are actually correct.  Daphne and Paul
love things that are true, usually
figure what is true and what is false, and if that isn’t possible, can show you
why something that you thought was true can’t be verified. As we’ve developed the site, they constantly find and correct mistakes
(including those on blogs), track down where ideas came from, and find new
sources that we never knew about. We’ve
learned that librarians can play key roles in the evidence-based management
movement, and have learned to view Daphne and Paul as key partners in this
adventure, as people who do the work with us rather than for us.

We aren’t the first to discover how valuable
librarians can be to practicing an evidence-based approach. Dr. David
Sackett , a driving force behind the modern evidence-medicine movement, has
long worked with his colleagues to recruit and train medical librarians to
review and evaluate studies in medical journals. Much as in the management literature, there
are huge numbers of studies published in medical journals, and many (Sackett
says over 90%) are so flawed that the resulting recommendations can’t be
trusted to guide how doctors treat their patients. Physicians are not only too busy to read many
of these studies, they are often not well-enough versed in research methods to
know which studies to believe.  Medical librarians now play a key role in the
evidence-based medicine movement, helping review and judge published studies
and helping  physicians find the best
treatments for their patients.  For
example, the primary goals of The
Evidence-Based Medicine Resource Center in New York include: 1. Provide health
practitioners and librarians with education and training in evidence-based
medicine, and on the information resources and the computer competencies
necessary to teach and practice evidence-based medicine and 2. Provide
librarians with the skills to work in partnership with clinicians in accessing
and managing clinical medical information.   And their
EBM Librarian’s Working Group includes
14 librarians from at least 10 different medical libraries. 

In short, although the rise of the web has changed
what librarians do, it also means that we need them more than ever because
there are so many facts out there now and they are so easy to get, and it is so
hard to tell which ones to believe – and they actually care about facts and
evidence, and know where to get them. Indeed, as I understand it, this partly
why many major universities – like The
University of Michigan and University
of California at Berkeley – have renamed their old library schools to “Information”
schools.

PS: If you want an accessible overview of
evidence-based medicine, check out this article in Family Practice Medicine.

I’ve been delighted with how much response there has
been to Lovaglia’s
Law, and how thoughtful many of the comments have been made on this blog
and elsewhere.  I want to add something
about the difference between “evidence-based management” and what might be
called “slogan-based management” or “faith-based management.”  Michael Lovaglia makes a strong argument that
the law holds and, as I wrote, there are also sound psychological reasons – the
increased public attention, pressure for accountability, and stress – why important
decisions are less likely to be evidence-based than unimportant ones. BUT even
though Michael calls this a law, remember, it really is just a hypothesis –
sort of a well-argued and
strong opinion that is weakly held.  Michael is designing some studies to test the
law. Perhaps other academics will test it as well (Or perhaps it has already
been tested, at least partially, in studies I don’t know about).  This means that, as charming and compelling
as the law is, if the research shows that the law doesn’t hold, Michael – and I
– will kiss this pretty idea good-bye. This commitment to facing the facts and
to discarding dearly held beliefs is a hallmark, perhaps the hallmark of
evidence-based management.

That is the difference between an evidence-based approach
and so much of the snake oil that is sold out there. Note:

1. The law is treated as something
that might be true, not as a proven fact.

2. The law is being subjected to
empirical test.

3. The stronger the evidence uncovered
to support the law, the greater the faith in it will be, and the weaker the
evidence, the weaker the faith will be.

These are, of course, very simple tests. But it
astounds me how often they are not applied.  Exhibit 1, as I’ve written here before, is corporate
mergers, but the list goes on and on.