Work Matters

  • New York City Halts Teacher Bonus Program: Another Blow to Evidence-Resistant Ideology

    The New York Times reports that the school system has abandoned their teacher bonus system because it is ineffective. I quote:

    A New York City program that distributed $56 million in performance bonuses to teachers and other school staff members over the last three years will be permanently discontinued, the city Department of Education said on Sunday. The decision was made in light of a study that found the bonuses had no positive effect on either student performance or teachers’ attitudes toward their jobs.

    The research appears to be quite careful and the RAND Corporation is highly respected:

    The study, commissioned by the city, is to be published Monday by the RAND Corporation, the public policy research institution. It compared the performance of the approximately 200 city schools that participated in the bonus program with that of a control group of schools. Weighing surveys, interviews and statistics, the study found that the bonus program had no effect on students’ test scores, on grades on the city’s controversial A to F school report cards, or on the way teachers did their jobs.  “We did not find improvements in student achievement at any of the grade levels,” said Julie A. Marsh, the report’s lead researcher and a visiting professor at the University of Southern California. “A lot of the principals and teachers saw the bonuses as a recognition and reward, as icing on the cake. But it’s not necessarily something that motivated them to change.”

    Are you surprised? I am not, and if the people running the New York City school system had actually read a large body of existing research, they would never have wasted all this money in the first place. In our opening chapter of Hard Facts, Dangerous Half-Truths, and Total Nonsense, Jeff Pfeffer and I reviewed the extensive literature on the links between incentives and teacher performance, and it turns out that although there always have been people with great faith in pay for performance systems for teachers — going back to at least 1918 — careful studies show over and over again that they do not improve student performance.  The New York Times article suggests that despite the ideology supporting pay for performance systems, there is growing evidence that the current round of incentive-based teacher pay isn't working — just as it never had worked:

    The results add to a growing body of evidence nationally that so-called pay-for-performance bonuses for teachers that consist only of financial incentives have no effect on student achievement, the researchers wrote. Even so, federal education policy champions the concept, and spending on performance-based pay for teachers grew to $439 million nationally last year from $99 million in 2006, the study said.

    To be clear, pay for performance schemes do appear to have some effects in schools — most of which are bad. One of the most well-documented (see this post on findings in Freakomomics and related research) is that some teachers and administrators start cheating when their pay is linked to performance on student's standardized tests.  Their are strong hints that this is exactly what happened in Washington, D.C. and other cities where financial incentives for teachers and administrators are linked to student test scores. 

    Note that I am not arguing against pay for performance systems in general.  They work in other settings –sports, sales, lots of other places,as we show in Hard Facts.  But they don't work for teachers for a host of reasons, perhaps paramount among them are that teachers rarely have enough control over key student behaviors before, during, and outside of class, over class composition (and when they do, they sometimes use it to cheat the tests.. such as by sending poor perfomers to special education classes), and over other resources they need to have a strong enough impact on student learning.   Also, giving students a test once a year probably isn't a very good way to measure what students are learning.  As The New York Times report argues, another problem with pay for performance schemes is that it turns teachers' attention away from intrinsic rewards (the reason most go into the profession in the first place) and toward extrinsic rewards (See Dan Pink's Drive to learn more about the trouble with extrinsic rewards). 

    To be clear, I am NOT a general supporter of the policies of teacher's unions.  Although I do think that way too much blame and way too little credit is given to teachers, I do have an evidence-based pet peeve against how vehemently teacher's union's defend the jobs of bad apples, the rotten and incompetent teachers.  This argument is consistent with the work on "Bad is stronger than good" that I've discussed here before… while it is tough for even a good teacher to overcome a lousy system and have strong positive impact on students, it is pretty clear that really lousy teachers can make a bad system worse, and dampen the positive effects of a good one.   I believe that if unions changed policy here and became even more vehement about reforming and removing bad teachers than their critics it would improve their reputations and the quality of education — and earn them political capital to battle lousy policies such as tying teacher pay to student test performance.  (See this great discussion and debate at The New York Times).

    To return the dismal record of pay for performance systems in schools, some years back, I had an interesting conversation with Tony Bryk, a prestigious educational researcher who is now heads the Carnegie Foundation.  We were were at a think tank, a place called the Center for Advanced Study in the Behavioral Sciences, and I asked Tony why — even though there is so much evidence against practices like pay for performance for teachers — they remain popular and come back in waves… until overwhelming evidence emerges again that in fact they are bad.  Tony suggested two reasons.  The first has to do with ideology — that people hold some assumptions so strongly (like economics and business minded folks who believe that incentives are the best answer to changing any kind of human behavior) that they refuse to accept any evidence that runs counter to their beliefs — no matter how strong those findings might be.  The second reason was what Tony called "collective amnesia."  He argued that, in the history of educational policy, the same bad ideas seem to come around every 10 or 20 years, and policy makers and their staffs either don't remember or make no effort to dig-up relevant research to guide their decisions, regardless of their ideologies.  In the case of pay for performance, it appears that both of these factors are operating. 

    Practicing evidence-based management isn't easy given our various human flaws.  But we sure could save a lot of money, a lot of heartache, and make people's lives a lot better if we all tried a lot harder to do it.  There are plenty of outcomes in life that are impossible to predict.  Unfortunately, what happened in New York was completely predictable, even if people with blind faith in linking test scores to financial rewards for schools and teachers remain unwilling to believe this well-established truth now. 

    P.S. A comment below suggests that some recent studies do show a positive effect of financial rewards on student performance.  My reading of the Rand Report suggests that for studies done in the U.S. there are a a few studies that show a positive impact, but the weight of the evidence supports the historical pattern of no effects or negative effects.  The more rigorous studies in particular find no ot weak effects on test scores, and little effect on teacher motivation, although there is some evidence that teachers devote more evidence to teaching to the test and less to teaching other things (not a surprise).  There is also some evidence that teacher cooperation goes down a bit and evidence that teachers game the system more to boost test scores.   A researcher from Chicago explained to me that in the schools she was studying (this was about 10 years ago) that scores were going up but she believed that it was not so much because incentives motivated teachers to work harder, but because it motivated them to get rid of their weakest students (often by sending them to special ed classes) and to refuse to "skip" gifted students because they pumped-up the average test scores in a class.  Finally, the most obvious effects of pressure on teachers and administrators to pump up test scores is cheating on the tests (by the teachers and administrators), as we have seen from evidence from Chicago, Philadelphia, and Atlanta, and just today, a probe started into cheating at schools in New Jersey.

    I am not rejecting the value of financial rewards as motivational tools for teachers outright, and it does appear that there are some special conditions under which they may be of some value. But the weight of the evidence suggests that most of the money spent on such incentives could have been to better use, that the ideological support for them is much stronger than than the evidence in support, and that one of the most consistent effects is bad — teachers and administrators cheat either to get the incentives or because they fear losing their jobs.

    This is my conclusion. You may reach a different one.  Here is a link to the 300 page Rand study of the failed New York program, which contains an excellent and very current review of the research.

  • Is Your Future Boss Horrible? A 10 Point Reference Check

    The film Horrible Bosses  opens on July 8th.  The basic plot, as I understand it, is that three guys who hate their bosses, played by  Jason Bateman, Charlie Day and Jason Sudeikis, conspire to murder them.  I don't recommend that way of dealing with a bosshole, and have been suggesting more constructive approaches (see this ABC interview).  As part of the film's release, I have been getting quite a few media calls about bosses. This reminded me of a checklist that I worked on with the folks at LinkedIn and Guy Kawasaki a few years back to help  assesses if a prospective boss is likely to be an asshole.   The list builds on the ideas in The No Asshole Rule and some ideas that appeared in Good Boss, Bad Boss.   

    We developed ten "reference check"  questions that you can ask people who have worked with and for your prospective boss — or perhaps had him or her as a client — to help determine if you are at risk of going to work for an asshole.

    Discovering the answers to these questions before you take a job can save you a lot of heartache. One of the key points in The No Asshole Rule is that one of the most effective ways to avoid being harmed by assholes — and becoming one yourself — is (to steal a phrase from Leonardo da Vinci) "to resist at the beginning," to avoid working for an asshole boss (or joining an asshole infested workplace) in the first place.  Here is our 10 point checklist:

    1. Kisses-up and kicks-down: “How does the prospective boss respond to feedback from people higher in rank and lower in rank?” “Can you provide examples from experience?” One characteristic of certified assholes is that they tend to demean those who are less powerful while brown-nosing their superiors.

    2. Can’t take it: “Does the prospective boss accept criticism or blame when the going gets tough?” Be wary of people who constantly dish out criticism but can’t take a healthy dose themselves.        

    3. Short fuse    : “In what situations have you seen the prospective boss lose his temper?” Sometimes anger is justified or even effective when used sparingly, but someone who “shoots-the-messenger” too often can breed a climate of fear in the workplace. Are co-workers scared of getting in an elevator with this person?  

    4. Bad credit: “Which style best describes the prospective boss: gives out gratuitous credit, assigns credit where credit is due, or believes everyone should be their own champion?” This question opens the door to discuss whether or not someone tends to take a lot of credit while not recognizing the work of his or her team.

    5. Canker sore: “What do past collaborators say about working with the prospective boss?” Assholes usually have a history of infecting teams with nasty and dysfunctional conflict. The world seems willing to tolerate talented assholes, but that doesn’t mean you have to.              

    6. Flamer: What kind of email sender is the prospective boss? Most assholes cannot contain themselves when it comes to email: flaming people, carbon-copying the world, blind carbon copying to cover his own buttocks. Email etiquette is a window into one’s soul.

    7. Downer: “What types of people find it difficult to work with the prospective boss? What type of people seem to work very well with the prospective boss?” Pay attention to responses that suggest “strong-willed” or “self-motivated” people tend to work best with the prospective boss because assholes tend to leave people around them feeling de-energized and deflated.

    8. Card shark: “Does the prospective boss share information for everyone’s benefit?” A tendency to hold cards close to one’s chest—i.e., a reluctance to share information—is a sign that this person treats co-workers as competitors who must be defeated so he or she can get ahead.                    

    9. Army of one: “Would people pick the prospective boss for their team?” Sometimes there is upside to having an asshole on your team, but that won’t matter if the coworkers refuse to work with that person. Use this question to help determine if the benefit of having the prospective boss on your team outweighs any asshole behaviors.

    10. Open architecture: “How would the prospective boss respond if a copy of The No Asshole Rule appeared on her desk?” Be careful if the answer is, “Duck!”

    Those are our 10 questions. I would love to hear other tips about what has helped you avoid taking a job with an asshole boss — or warning signs that you wish you would have noticed before going to work for a demeaning creep.   

  • How Many Pilots Does a 737 Need? Evidence-Based Management in Action

    I have been reading a lot about group and organizational size lately because it is a key issue for understanding the "scaling problem" that Huggy Rao and I are currently tackling.  After all, if you want to grow a large organization or network, it is crucial to understand how large "the building blocks" should be, how many people a leader can lead, and the the upper limits of organization and network size.   You will likely see other posts on this issue here as I am fretting over this question a lot.  But I couldn't resist a quick post drawn from J. Richard Hackman's fantastic book Leading Teams. 

    Richard reports an astounding solution to a disagreement between United Airlines and the pilots union when United was making its first big purchase of the 737 aircraft.  Boeing designed the cockpit so that it could be flown by either a two or three person crew.  Of course, United wanted two pilots because of the enormous savings in labor expenses; the union wanted three pilots because they argued that, since the planes would be flown in busy air spaces, it would be better to have a third person on board to help with demanding work and to keep an eye out for problems.   Well, this kind of disagreement didn't surprise me and I am sure it doesn't surprise me.  But what shocked me as that United and the union jointly sponsored an independent group to study the differences between two and three person crews during actual flight operations.  The study found no consistent differences between two and three-pilot crews.  As Hackman reports on page 119 of Leading Teams:

    "Members of the three-person crews did leave the cockpit more frequently to visit the cabin, which may have helped strengthen the work relationship between pilots and flight attendants. But they caught no more potentially conflicting traffic called to their attention by air traffic control than did the two-person crews."

    Chalk-up one for management on this one; as Richard points out, 737s are now exclusively flown by pairs of pilots, not trios.  I love this story because it appears to be an actual evidence-based decision. Unfortunately, this happens far less than it should.  As a process, it fascinates me because asking an objective third party to study problem when two parties have conflicting ideologies, goals, or incentives –and agreeing in advance to a decision that fits the evidence — seems like the right way to go about things.  I am not even sure if that is what happened in this case, but I fancy the notion.  I know that it usually won't be feasible in real life because we human-beings aren't that cooperative and rational, but it would lead to more effective and safer organizations.

    P.S. Check out Hackman's awesome recent post at HBR on group effectiveness. He has been studying this problem for over 40 years and I believe knows more about both the academic and practical challenges than anyone on the planet.

     

  • A Rough But Intriguing Metric for School Assessing a School Principal

    Yesterday, I did an interview for the BAM network on Good Boss, Bad Boss.  The content expert on line was Justin Snider, who teaches at Columbia and has in-depth knowledge about K-12 schools, as that was the focus of the conversation.  Justin had great questions and comments about bosses in general (see this recent post) and about school principals in particular.  I thought he made especially good comments about how the best principals are PRESENT, constantly interacting with teachers, students, and parents. He especially suggested that school principals think about where their offices are located.. are they in a place that essentially requires them to keep bumping into teachers and parents, or are they in some corner of campus that reduces the amount of interaction.

    I like Justin's point about the office because it reminds me of the design for Pixar's building in Emeryville, which was inspired by Steve Jobs' assertion that they needed to make sure that everyone was basically forced to bump into each other as a result of the placement of the food and bathrooms.  At one point, Jobs half-seriously suggested that there be just one central bathroom so that everyone had to run into everyone else and there would be a lot of random encounters as people walked to and from that crucial location. The ultimate design resulted in more than one bathroom , but the food and bathrooms were located so that people need to walk through this central area constantly — one of those little things that has helped fuel Pixar's creativity over the years.

    After the interview, Justin and I exchanged emails,  I told him a story about how I saw the difference between the impact of a good versus a bad principal at my daughter's middle school, how there was a great principal who seemed to know every students name and was widely loved.   He retired and was replaced by a bad one who seemed to not know any student's name and was so out of touch that his lack of soul and other more objective acts of incompetence provoked widespread despair among students and parents, and quite a few teachers complained about his lousy leadership openly.    I was reminded of this difference between the two principals just a few weeks ago when, even though it is has been a few years since the good boss last saw my daughter, he greeted her by name in a local restaurant. In contrast, my daughter is still annoyed that the bad one mispronounced so many student names, including hers, at graduation (Her name is "Eve," he called her something that sounded like "Ev.")

    Justin had an interesting reaction to my little story:

    Actually, right after our call concluded, I realized I should have said that a great back-of-the-envelope measure of whether a principal is generally doing a good job is how many students' names he or she knows.  In my experience, there's a strong correlation between principals who know almost all students by name and those who are respected (and seen as effective) by students, parents and teachers.  It's not a perfect measure, of course, but I think it's probably a fairly good indicator of a school's climate and a leader's effectiveness.

    I like Justin's observation.  Of course, some us are better at remembering names than others and we all have cognitive limits. But Justin's argument is compelling to me because knowing people's names seems like a good sign that a boss is directing attention to those he or she leads and is responsible for helping and is not overly focused on him or herself, or on kissing-up to the superintendent, board of education, or other superiors.

    What do you think of this metric?  Is it right for schools? What about other workplaces?

  • Indira Gandhi on Doing Work Versus Taking Credit For It

    I had a meeting today with my colleague Huggy Rao where we were batting around various ideas about systems that are effective versus ineffective at scaling good ideas. Huggy brought up this cool quite from Indira Gandhi:

    My grandfather once told me that there were two kinds of people: those who do the work and those who take the credit. He told me to try to be in the first group; there was much less competition.

    He then went on to argue that systems that bring-in, develop, and reward people in the first group  — and that expel, reform, and punish people in the second group — are likely to be more effective at spreading and implementing constructive action. Sounds right to me.

  • Secret Features of Apple’s Proposed New Campus

    Apple has proposed a most inventive new campus in Cupertino. The folks as joyoftech.com had good fun imagining the "hidden features." I especially like the empty part of the building where no one is allowed to go — and is meant to create mystery.  That is VERY Apple.  Thanks to Alistair Davidson for sending this my way:

     

    Real Scoop on Apple HQ

  • I’m on BNET’s “The Live One” Webcast Today

    I will be interviewed on BNET's new webcast show, "The Live One" today at 10AM pacific.  I plan to talk about Good Boss, Bad Boss and related stuff including Google's recent research differentiating their best and worst managers (technical skills didn't matter nearly as much as people skills, which surprised a lot of people at Google), a cool new study that shows having more women on your team will make it act smarter (in fact, it is more important than having people with higher IQ's), and the recent appearance of The No Asshole Rule in Doonesbury.  At least that is what I am planning on talking about.  It will be a fairly informal conversation, so who knows exactly what will happen.  I hope you can tune in.  Again, the URL is here — check out the past interviews, with people including Peter Sims and Penelope Trunk.

  • Evidence that “Retail Therapy” is Effective

    Just as I feared!  BPS Research reports a recent article containing a series of small studies that shows "retail therapy" does work — at least in a sample of young American consumers.  And they found little evidence of regret or guilt after the purchases.  Here is the description of the third and most compelling of the studies:

    "Lastly the researchers had 69 undergrads complete two retrospective consumption diaries, two weeks apart, documenting their purchasing behaviour, mood and regrets. All the participants admitted in the first diary to having bought themselves a treat (mostly clothes, but also food, electronics, entertainment products and so on). Sixty-two per cent of these purchases had been motivated by low mood, 28 per cent as a form of celebration. Surprisingly perhaps, treats bought as a form of mood repair were generally about half the value of treats bought for celebration, reinforcing the notion that retail therapy is constrained, not out of control. Moreover, according to the diaries, the retail therapy purchases were overwhelmingly beneficial, leading to mood boosts and no regrets or guilt, even when they were unplanned. Only one participant who'd made a retail therapy purchase said that she would return it, given the opportunity."

    I better not show this study to my teenage daughters.  They love retail therapy and I have never seen a hint of guilt from them… only the motivation to do more in the future!

    The citation is: Atalay, A., and Meloy, M. (2011). Retail therapy: A strategic effort to improve mood. Psychology and Marketing, 28 (6), 638-659 DOI: 10.1002/mar.20404.

    P.S. As I have said before,  BPS Research Digest is one of my favorite places on the web.  Check it out.

     

  • Taking The Path of Most Resistance: The Virtues

    I am blogging only intermittently as I am pretty focused on reading, talking to people, and generally fretting, worrying, and trying to structure the book on scaling constructive action that Huggy Rao and I are trying to write. I have been reading everything from psychological experiments on how different metaphors affect our perceptions and action, to studies of the mathematical and administrative challenges of scaling computer systems, to research on cities of different sizes (especially some interesting stuff that suggests bigger is better). But the area where scaling has been studied perhaps most directly is in  education, including studies of how to replicate great charter schools and how to substitute effective practices for ineffective practices in large school systems.  

    This weekend, I read an old (1993) but excellent study commissioned by the Casey foundation on what it takes comprehensive school reform in large school systems.  I was taken with its counter-intuitive title "The Path of Most Resistance"  (see the PDF here), in part, because it ran counter to some of the (evidence-based) assumptions that we have developed about scaling, including the notion that scaling depends on finding ways to simplify things and reduce cognitive load on people, and the notion that changes that are consistent with local cultures and traditions are easier to implement than those that run counter to embedded beliefs. 

    As I read the report, however, I realized that the authors agreed with some of these points, as they weren't arguing that leaders should TRY to make things harder on themselves, but rather, to do large scale change right, there argument was that a lot of very hard things need to get done.  They argued that taking the easy way out — expecting instant results; not taking the time to engage with parents, students, administrators, local politicians and other key crucial actors; doing it on the cheap; expecting everything to go smoothly–  and a host other "easy solutions  — simply weren't realistic or wise for would-be change agents. The examples of successful large scale change they examined all took pretty much the opposite approach — there was a lot of patience and a long term perspective, time was taken to involve major constituencies, lots of resources were devoted to the effort, and a host of other tactics that entailed doing things the hard way rather than the easy way. 

    More broadly, I think it is intriguing to use their title to flip assumptions about change.  Sometimes the tougher road is the better road, as people go in with a more realistic mindset, they are ready for setbacks,  and expect to spend the time and money necessary.  And, as an added bonus, any social psychologist will tell you that the more effort and sacrifice people make toward something, the more committed they will be to it.   Indeed, as I watch successful innovators — ranging from the teams we teach at Stanford's design school to Pixar's amazing journey — the most successful tend to have this "it is going to be tough, but I can and will do it" mindset.

    On the other hand, I think there is an important caveat, one the Jeff Pfeffer and I have written about in Hard Facts. One of the impediments to successful change is that people use the belief that "it is difficult and takes a long time" to avoid trying to make necessary changes at all.  Or, worse yet, they  propose a long-term change process, but only start working on it just before the "due date" — perhaps proposing a two-year project, but doing all the work in the final months (much like my students who, even though I assign a paper months in advance, don't start it until the night before).  In addition, there are many constructive changes that are not difficult and do not take a long time — such as changing small rules or procedures, experimenting with a new and delimited program, and so on.   Unfortunately, all too often, large scale change is slowed or stopped because people delay or fail to complete the array of small and easy steps required to accomplish any large change (In other words, they fail to focus on the daily small wins).

    Finally, there is an old but interesting lesson in creative thinking here, one consistent with the notion of "having strong opinions, weakly held."  The challenges of doing successful change look a lot different when you assume that "taking the path of least resistance" is best versus assuming that "taking the path of most resistance" is best.  Indeed, although they are pretty much exact opposites, you can learn a lot about change when you look for conditions under which each statement is true and false.  More generally, a good way to spark creativity is to take your most dearly held assumptions and ask "suppose the opposite were true?"

     

  • Caffeine: It Undermines Performance on Collaborative Tasks for Men, Enhances It For Women

    I can't believe that I missed this study reported by BPS research last January.  Way cool.  It compared the performance of men working in pairs to women working pairs.  The researchers placed them under performance pressure, and varied whether they drank caffeinated or decaffeinated coffee.   The "caffeinated" men performed worse, while the women performed better.  Here is the opening paragraph from BPS, which suggested that the stimulant has these varying effects because, when cranked-up physiologically, people tend towered their most natural and well-rehearsed behavior — which means that men get more aggressive and women become more collaborative:

    If a meeting becomes stressful, does it help, or make things worse, if team members drink lots of coffee? A study by Lindsay St. Claire and colleagues that set out to answer this question has uncovered an unexpected sex difference. For two men collaborating or negotiating under stressful circumstances, caffeine consumption was bad news, undermining their performance and confidence. By contrast, for pairs of women, drinking caffeine often had a beneficial effect on these same factors. The researchers can't be sure, but they think the differential effect of caffeine on men and women may have to do with the fact that women tend to respond to stress in a collaborative, mutually protective style (known as 'tend and befriend') whereas men usually exhibit a fight or flight response.

    Clearly, this is a "more research is needed" situation.  But, if it generalizes to real life, the implication is that, if you are running a meeting and it is attended by all women, give them caffeinated drinks, but if it is all men, or perhaps a blend of men and women, given them the decaf if you want cooperation and better performance.  

    Here is the reference:

    St. Claire, L., Hayward, R., and Rogers, P. (2010). Interactive Effects of Caffeine Consumption and Stressful Circumstances on Components of Stress: Caffeine Makes Men Less, But Women More Effective as Partners Under Stress. Journal of Applied Social Psychology, 40 (12), 3106-3129 DOI: