I’ve been delighted with how much response there has
been to Lovaglia’s
Law, and how thoughtful many of the comments have been made on this blog
and elsewhere.  I want to add something
about the difference between “evidence-based management” and what might be
called “slogan-based management” or “faith-based management.”  Michael Lovaglia makes a strong argument that
the law holds and, as I wrote, there are also sound psychological reasons – the
increased public attention, pressure for accountability, and stress – why important
decisions are less likely to be evidence-based than unimportant ones. BUT even
though Michael calls this a law, remember, it really is just a hypothesis –
sort of a well-argued and
strong opinion that is weakly held.  Michael is designing some studies to test the
law. Perhaps other academics will test it as well (Or perhaps it has already
been tested, at least partially, in studies I don’t know about).  This means that, as charming and compelling
as the law is, if the research shows that the law doesn’t hold, Michael – and I
– will kiss this pretty idea good-bye. This commitment to facing the facts and
to discarding dearly held beliefs is a hallmark, perhaps the hallmark of
evidence-based management.

That is the difference between an evidence-based approach
and so much of the snake oil that is sold out there. Note:

1. The law is treated as something
that might be true, not as a proven fact.

2. The law is being subjected to
empirical test.

3. The stronger the evidence uncovered
to support the law, the greater the faith in it will be, and the weaker the
evidence, the weaker the faith will be.

These are, of course, very simple tests. But it
astounds me how often they are not applied.  Exhibit 1, as I’ve written here before, is corporate
mergers, but the list goes on and on.

The
importance of identifying and testing the assumptions that determine how
organizations and technologies are designed sounds so obvious – yet we’ve
learned that, when we don’t press managers, consultants, and researchers
(including ourselves) to take a hard look at their deeply held beliefs about
what they are doing and why, they will unwittingly do horrible – or at least
very expensive – things over and over. In the management arena, the assumptions held by people who design
organizations are often dangerous half-truths. So, for example, assuming that human beings are always selfish and can’t
be trusted is dangerous because, if you operate on that assumption, you will
design a fear-driven organization that will encourage people to act that way –
and never give people a chance to earn trust.

Another area is teacher incentives and student
test scores. Politicians and school
administrators often argue that teacher’s pay should be linked to student test scores. That all sounds wonderful, until you start
examining the basic assumption: If teachers work harder, their students will do
better on standardized tests. Unfortunately, if you dig into this assumption, you will start realizing
that teachers have little or no control over which students they teach, how
many students are in their classes, what levels of resources they have, and
what materials they use.  So, even if
financial incentives do actually encourage teachers to work harder (another
suspect assumption), increasing motivation doesn’t increase student performance
much, if at all, because teachers don’t have enough control over the work. And,
at least in the short-term, incentives don’t affect teacher’s knowledge and
skill.

That
is why studies going back nearly 100 years show that teacher incentive pay has
little if any impact on student achievement scores. It does have other predictable
effects: teachers and school administrators will try to change things that
they can control: Like cheating on the tests to get their students higher
scores, either by changing the forms themselves or telling students the right
answers. And, as I’ve heard from researchers and parents in the Chicago school
system, teachers respond to these incentives by moving their weakest students
into special education classes (which are overflowing with kids who really
aren’t well-suited to those classes) and when they have a gifted child who
should probably skip a grade or move to a classroom or school of gifted kids,
they squelch efforts to take those kids out of their classes. In other words, the incentive pay does affect
teacher effort: They focus on ways to get their test scores up that they can
control, even though those changes have nothing to do with student learning,
and in fact, may actually undermine learning. Yes, incentives do drive behavior, but sometimes the wrong kind (See
Chapter 5 of Hard Facts for an
in-depth discussion of incentives).

What
does this have to do with escape from submarines? Perhaps the clearest, and most troubling,
case I know of a deadly assumption (that was held for over 50 years) has to do
with the problem of escaping from a sunken submarine. C.B. “Swede” Momsen was a colorful and
charismatic U.S. Naval officer who unwittingly perpetuated false and deadly
beliefs about the best way to ascend to the surface. Check out The
Terrible Hours to read about this maverick. Momsen was deeply disturbed by
several incidents where submariners were trapped at depths of 100 to 200 feet beneath
the surface, with no apparent means for escape. They all died waiting for a rescue that never came. “Swede” dedicated years of his life to
developing the Momsen lung in the 1930’s, a complicated apparatus that –- by
the time development was completed — included a mouth piece, a breathing bag,
a canister of soda lime, goggles, a nose clip, and a marker buoy attached to
500 feet of rope that had a knot every ten feet. The idea was “Escaping submariners were to
pause every ten feet, where they found a knot, so as to ascend no faster than
fifty feet per minute.”  The Germans had
used a similar device called the Dräger breathing set, going back to World War
I.

The
perceived need for these cumbersome devices – and the actions of people on
submarines –were based on the assumption that simply exiting the submarine and
swimming to the surface meant certain death. But research done after World War II showed that this assumption was
false: at depths of less than 300 feet, a trapped submariner’s best chance of
survival was a “free ascent.”  As Ann
Jensen’s 1986 article Why
the Best Technology for Escaping from a Submarine is No Technology reports:

The solution was a British suggestion. Inflate a life jacket while
in the submarine. The jacket would be designed with a flapper valve to release
the expanding air as it carried its wearer upward. “Once you’re out, you start
blowing as hard as you can blow,” said Schlech. “The jacket takes you up and
out of the water like a shot. We called the system ‘Blow and Go.’ There was a
lot of opposition at first, but eventually it got rid of the Momsen Lungs and
all the other equipment, and it’s still in use in depths of up to three hundred
feet.”

Moreover,
Jensen reports that German experience going back to World War I showed that –-even
without a life jacket — simply exhaling while making the ascent is effective
as well, although such experience was not known or was ignored as research focused
on developing better devices for escaping submarines, even though none were
needed. To quote Jensen’s article:

The case of the German U-57, which hit a
mine north of Scotland
in 1915, is typical. The U-boat went down in 128 feet of water with twenty men
alive inside. The air in the submarine quickly filled with chlorine gas as
seawater flooded the boat’s electric batteries. The fumes burned the men’s eyes
and made breathing nearly impossible. Their ears ached as pressure increased.
They found only four Dräger units aboard; believing their situation hopeless,
two of the men shot themselves. Then the captain decided to make one last,
desperate effort to save the remainder of his crew by opening a hatch.

To his amazement the hatch flew open and he was drawn out and
upward. He had no time to inflate his life jacket or even to take a breath. As
he recounted later, “I had no desire to inhale, but to forcibly exhale so that
I constantly had to blow air out.” The air in his lungs had expanded as he
rose. If he hadn’t exhaled, his lungs probably would have burst. The rest of
the crew followed him; seven survived the ascent and were later picked up.

The
upshot is that –- whether you are talking about teachers or technology –-
stopping to identify and test your assumptions is something that isn’t done
often enough, and can save a lot of money and a lot of lives. And a related
lesson, as I’ve written about before, is that the most effective people and
organizations are often masters
of the obvious.

I mentioned Rakesh
Khurana’s book Searching
for a Corporate Savior in my last post. After I wrote it, I realized that
Rakesh may have found an instance of Lovaglia’s Law: "The more
important the outcome of a decision, the more people will resist using evidence
to make it."

Rakesh’s book is so compelling because he blends
impressive quantitative data with insights from his in-depth research on how
senior executives are selected and evaluated by corporate boards. Rakesh described how directors of huge
companies had enormous faith in the power of CEOs that went beyond anything
that could be justified by any research, how they spent vast amounts of money
and time searching for new corporate saviors, and paid out huge sums to
executive search firms and to the CEOs they ultimately hired. Following Lovaglia’s Law, perhaps because
these decisions were so important, Rakesh found that when he asked corporate
directors if CEOs are worth all that money, they reacted with anger and
surprise, as if he had raised a taboo subject. He found that they had
“virtually religious” convictions on the subject, which led them to dismiss any
evidence showing that CEO quality is not a primary and powerful cause of company performance.
My hunch is that they would have been more receptive to evidence about more
trivial decisions, such as what colors to paint the walls or what music to play
in the elevators.

Today’s
Wall Street Journal contains an
entertaining story about the recent financial setbacks suffered by hedge fund
manager Robert Chapman. It makes for good reading, as Mr. Chapman is quoted as
saying that “We serve eviction notices to incompetent executives” The Journal reports how Mr. Chapman once
described the 78 year-old Chairman of a company as a “helpless Mr. Magoo-like
character" and its CEO as “The Dummy.” And they tell us that “Mr. Chapman’s office features a toy guillotine, a
shark skull and other symbols of gruesome destruction.”

Whether
Mr. Chapman is an effective intimidator and turn around artist is unclear at
the moment, but what struck me about the story was that it was yet another
indication that CEOs, or perhaps CEOs and their top teams, are treated as the
most powerful factor that drives organizational performance. The message in the business press – and from
headhunting firms too – is that if things are going good, than it is because a
company has great leadership. And if things are going badly, then it is largely
because of bad leadership – so if you replace the CEO or perhaps the senior
team (serving them with “eviction notices”) then everything will be fixed. As we show in our leadership chapter in Hard Facts, a related conclusion is
accepted by economists like Michael Jensen, who argue that if you get the CEO’s
incentives “properly aligned” with organizational goals, then, boom,
performance will improve because they will be properly motivated to act in the
firm’s best interest.  Indeed, many experts who draw
on economic perspectives like agency theory complain that the problem with CEO
pay is that they are paid well when things go wrong or right, and if companies
would just provide CEOs with the right incentives, then the CEO pay controversy
would – or at least should — go away.

The
problem with these arguments is that they all are based on the assumption that
CEOs have a lot of influence over performance. After all, if CEOs didn’t have much influence, tying their rewards to
performance or getting rid of bad leaders wouldn’t matter much. It is
instructive to compare this unwavering faith in the power of CEOs with the best
evidence. Yes, leaders do have some
impact, but far less than most people think. My colleague Jeff Pfeffer published a paper in 1977 in the Academy of Management Review showing
that leader’s actions rarely account for more than 10% of the variation in organizational
performance, and often, account for much less. Subsequent studies have confirmed this general pattern. Leaders do have
a somewhat stronger effect on performance when companies are small and
young. But a host of experiments and
field studies show that the business press and other observers consistently
give leaders far more credit and far more blame than they deserve. Apparently this
happens because people are brainwashed by cultural myths about the power of
leaders and because it is requires less mental effort – and is more comforting
– to view, say, Procter & Gamble as AG Lafley or Oracle as Larry Ellison, then to think about the long list of factors that actually cause performance.

Jeff
Pfeffer emphasized that leaders have, at best, modest control because there are so many
factors that are simply impossible for them to control – economic conditions,
industry structure, fixed costs, what competitors do, what happened before the
CEO took over, and what a nuances of what the other hundreds and thousands of
people in the company do. A few years
ago, I was talking with Spencer Clark, a former GE executive who had led a
large business during Jack Welch’s reign about the limits of CEO power. Clark  remarked “Jack did a good job, but everyone seems
to forget that the company had been around for over a hundred years before he
ever took the job, and he had 70,000 other people to help him.” Indeed,
although CEOs can certainly make a difference, especially founders, many of the
greatest companies – take Toyota
or Procter & Gamble – have succeeded because they have such great systems
that it makes it easy for competent people to succeed, not because of the work of leaders with magical powers.

Another
reason that leaders don’t have as much effect as most people think is what
statisticians call “the restriction of range” problem. Although, as Mr. Chapman
says, evicting incompetent leaders is no doubt important, some researchers –
notably Stanford’s James March – have argued that nearly all people that are seriously
considered for senior management positions are competent (as they are heavily
screened, trained, and all have shown that they can do the work) so it doesn’t
make much difference which finalist you choose. March, the master of challenging conventional wisdom with logic and
evidence – and his charm – once wrote:

Management may be extremely difficult and important even though
managers are indistinguishable. It is hard to tell the difference between two
different light bulbs also; but if you take all the light bulbs away, it is
difficult to read in the dark. What is hard to demonstrate is the extent to
which high performing managers (or light bulbs that endure for an exceptionally
long time) are something more than one extreme end of a probability
distribution generated by essentially equivalent individuals

March’s
perspective is bolstered by evidence that corporate boards place irrational
faith in magical superstar CEOs, as a savior who will ride in on a white horse and transform a struggling company
into an industry leader. See Rakesh
Khurana’s lovely book Searching
for a Corporate Savior for some of this work.

What does this all mean in practice? There are a
lot of nuances, but the most important implication is that bringing in
a new CEO is rarely a magical quick fix. Executive search firms love CEO
changes –especially external hires – because they make money on each search and
hire. But the weight of the evidence suggests that bringing in a new CEO
doesn’t help that much, often makes
no difference at all, and can even do moderate damage.

Venture capitalist Steve Dow is one of the few
people in industry I’ve talked to who actually seems to get and act on this
evidence. As we wrote in Hard Facts:

Start-ups have notoriously high failure
rates, no matter how well they are managed. Dow has been a general partner at VC firm Sevin Rosen since 1983 and served on
dozens of boards over the years. He tells us that many board members,
especially young venture capitalists who lack operational experience, are quick
to talk about replacing the CEO at the first hint of trouble. Dow asks them,
“Now, suppose you were CEO, what would you do differently than the one we have right now?” Dow
says that most of the time they can’t think of much, if anything, they would
change. Like everyone else, until they think about it carefully, these venture
capitalists can’t separate the CEO’s performance from the firm’s performance.

Michael Lovaglia
is a Professor and Department Chair in the Sociology Department at the Iowa. Michael wrote Jeff Peffer and me a charming and
insightful email a few weeks back about evidence-based management that I can’t
stop thinking about.

Michael proposed:

Lovaglia’s
Law: The more important the outcome of a decision, the more people will resist
using evidence to make it.

I can’t stop thinking about this law because, although it
sounds like a joke, if I think about the basics of sociology, economics, and
psychology, many signs are that Michael is right.  Sociologists are really obsessed with power
and status. The upshot of much of their work is that, the more is at stake, the
more that people will be motivated to push for solutions that increase their
power and decrease other’s power – and not motivated to take steps that help
other people or other groups, let alone that help the system as a whole.

The concept that drives much economic research is
that people strive to do what is in their individual self-interest (Indeed, some
people claim that there is no underlying theme in Freakonomics.
I always beg to differ because each chapter is a brilliant example of how Steve
Levitt has applied the concept of self-interest in a different setting.) Clearly, the more important the decision, the
more people involved stand to lose and gain, and the more strongly they push for outcomes that enhance their
self-interest rather than are best for everyone involved.

Finally, psychologists have documented many ways
that human decisions are flawed, swayed by biases rather than rationality (see
Max Bazerman’s book on decision-making)
whether the decision is important or not.  But if you look at research on what happens to
human beings when we are under stress, a host of studies show that we cling
even more tightly to what we know and can do best, and that is especially hard
for us to process unfamiliar ideas and to change how we do things when we work under severe time pressure and public scrutiny or when the stakes are high.

A partial, and paradoxical, solution is implied by
Karl Weick’s work on small wins (see my last post): If important decisions
provoke so much greed, distress, and irrationality, it might be best to try to reframe big decisions as small ones –- to fool yourself and others into
believing that what seems big is really small!   I
confess, this is one of those ideas that sound really dumb when you start applying it: How would you convince the
players in the Lebanon
nightmare that, actually, the terms for the cease-fire really aren’t all that
important?  Pretty stupid, huh? But it may prove more practical in everyday decisions made by organizations: who to hire,
what business strategies to pursue, what products to release, and so on. Perhaps framing
them as less important to performance than they actually are can increase your success
rate!

I throw out this weird idea for fun; I am not sure
I believe it.  I also confess that part
of me doesn’t want to believe it even if it is true.  But if Lovaglia’s Law and my fretting about
basic behavioral science theory are right, then it just might work.  Perhaps it is time for a management book called something like "It Doesn’t Matter: The Power of Indifference." I think I am joking about that, but I am not sure.