Ronny Kohavi is the General Manager of the
Experimentation Platform at Microsoft, his group has developed and is spreading
the use of tools that allow people within
Microsoft and external developers to do controlled experiments with users,
and that way, potentially develop better and more usable software . You can
read about this EXP Platform here. Clearly, Ronny is someone who cares about and
understands the power of evidence-based practices, and as such, we have been
exchanging emails about the work that Jeff Pfeffer and I have done on
evidence-based management in our book
and our website.
Ronny is generally supportive of our approach,
but has pushed me to emphasize that controlled experiments are the gold
standard of science and that we should have pushed people to use them more, and
included more examples. I partly agree
with Ronny, as there are times when experiments aren’t used but can yield
better practices – as we talk about in the book, Gary Loveman at Harrah’s did
use experiments as one of the methods (along with data mining) to overturn
deeply held assumptions in the gambling industry. But I argued that there are
also times when an experiment wasn’t possible, for example, in the case of the
now apparently aborted Microsoft/Yahoo! merger, the only decent evidence I know
of that provides guidance are correlational studies, as no one has yet been allowed
to do controlled experiments of the conditions under which mergers fail or
succeed,
BUT
I do agree completely with Ronny that we need more experiments and, as the
history of medicine shows, it is important to have the strongest possible data
because – even when you have it – convincing people to abandon a bad practice
can be remarkably difficult when they believe in its efficacy, are skilled at,
and everyone around them has always done and believed in it. We wrote about
bloodletting as an example, but Ronny wrote me about an even more interesting
and terrifying case:
Dear Bob,
One of the examples was
the bloodletting trials by Pierre Louis, but I was reminded of a better
example: Semmelweis’s Childbed Fever. The Semmelweis Reflex is the
dismissing or rejecting out of hand any information, automatically, without
thought, inspection, or experiment. Anyway, here’s a summary.
Ignaz Semmelweis’s
Childbed Fever
The story below is
mostly from the book Leadership and
Self-Deception. The story is corroborated by multiple sources
including Encyclopedia
Britannica, Childbed Fever: A
Scientific Biography of Ignaz Semmelweis, and Wikipedia.
Semmelweis was a
European doctor, an obstetrician, in the mid 1800s. He worked at Vienna’s
General Hospital, an important research hospital. The mortality rate in
the ward where he practiced was one in 10 – one in every ten women giving birth
there died! The reputation of Vienna General was so bad that women
preferred to give birth on the street and then went to the hospital. In
the book Childbed Fever, they estimated that 2,000 women died each year
from childbed fever in Vienna alone.
The collection of
symptoms associated with these deaths was known as “childbed fever” or
Puerperal fever. More than half the women who contracted the
disease died within days. Patients begged to be moved to a second section
of the maternity ward where the mortality rate was one in fifty – still
horrific, but far better than one-in-ten in Semmelweis’s section.
Semmelweis became obsessed
with the problem. He tried to control for all factors, including
birthing positions, ventilation, diet, and even the way laundry was done.
The one obvious difference between the sections was that Semmelweis’s section
was attended by doctors, while the other section was attended by midwives.
After a four-month leave
to visit another hospital, he discovered that the death rate had fallen
significantly in his section of the ward in his absence. His inquiries
led him to think about the possible significance of research done by him and
the doctors on cadavers. Yes, cadavers. Semmelweis spent far more
time doing research on cadavers than other doctors.
Vienna General was a
teaching and research hospital and many doctors split their time between
research on cadavers and treatment of live patients. The doctors in his
section performed autopsies each morning on women who had died the previous
day, but the midwives were not required or allowed to perform such
autopsies. They hadn’t seen any problem with that practice because there
was as yet no understanding of germs.
Semmelweis concluded
that ‘particles’ from cadavers and other diseased patients were being
transmitted to healthy patients on the hands of the physicians. He
experimented with various cleansing
agents and instituted a policy requiring physicians to wash their
hands thoroughly in a chlorine and lime solution before examining any
patient. The death rate fell to one in a hundred!
What is surprising about
this story isn’t the discovery through attempts to control for factors, which
led to the unthinkable conclusion (at the time) that there was something
invisible that was transferred by the doctors. What is really
shocking is how long it took the community of doctors to accept the results.
According to Encyclopedia
Britannica, the mortality rate in Semmelweis’s division fell from
18.27% to 1.27% in 1848. That was not enough to generate sufficient
recognition and in 1849 he was dropped from his post at the clinic and turned
down for a teaching post. Semmelweis spent the next six years at a
Hospital in Pest, Hungary, where he reduced mortality rate in the obstetrics
department to 0.85% while in Prague and Vienna the rate was still about 10% to
15%.
Vienna continued to
ignore his recommendations. In 1861, he published a book, but the
community rejected his doctrine. In 1865 he suffered a nervous breakdown
and was taken to a mental hospital, where he was beaten by asylum personnel and
died. It took another 14 years for the discovery to be accepted, after
Louis Pasteur, in 1879, showed the presence of Streptococcus in the blood of
women with child fever. Semmelweis is now recognized as a pioneer of antiseptic
policy.
This story is instructive on many levels; the
first thing that comes to mind is that developing the best evidence and
practices is often even less than half the battle. Ideas that spread and stick
need to be sold well too. That is the
main idea behind our d.school class on Creating Infectious Action, and great
books like Influence
and Made
to Stick.
P.S.
Ronny, thanks for all your great ideas and for sharing this story.
