Strategies for Learning from Failure

The wisdom of learning from failure is incontrovertible. Yet organisations that do it well are extraordinarily rare. This gap is not due to a lack of commitment to learning. Managers in the vast majority of enterprises that I have studied over the past 20 years – pharmaceutical, fi nancial services, product design, telecommunications, and construction companies; hospitals; andNASA’s space shuttle programme, among others – genuinely wanted to help their organisations learn from failures to improve future performance. In some cases they and their teams had devoted many hours to after-action reviews, post-mortems, and the like. But time after time I saw that these painstaking efforts led to no real change. The reason: Those managers were thinking about failure the wrong way.

Most executives I’ve talked to believe that failure is bad (of course!). They also believe that learning from it is pretty straightforward: Ask people to reflect on what they did wrong and exhort them to avoid similar mistakes in the future – or, better yet, assign a team to review and write a report on what happened and then distribute it throughout the organisation.

These widely-held beliefs are misguided. First, failure is not always bad. In organisational life it is sometimes bad, sometimes inevitable, and sometimes even good. Second, learning from organisational failures is anything but straightforward. The attitudes and activities required to effectively detect and analyse failures are in short supply in most companies, and the need for context-specifi c learning strategies is underappreciated. Organisations need new and better ways to go beyond lessons that are superficial (“Procedures weren’t followed”) or self-serving (“The market just wasn’t ready for our great new product”). That means jettisoning old cultural beliefs and stereotypical notions of success and embracing failure’s lessons. Leaders can begin by understanding how the blame game gets in the way.

The Blame Game

Failure and fault are virtually inseparable in most households, organisations, and cultures. Every child learns at some point that admitting failure means taking the blame. That is why so few organisations have shifted to a culture of psychological safety in which the rewards of learning from failure can be fully realised.

Executives I’ve interviewed in organsations as different as hospitals and invesment banks admit to being torn: How can they respond constructively to failures without giving rise to an anything-goes atitude? If people aren’t blamed for failures, what will ensure that they try as hard as possible to do their best work?

This concern is based on a false dichotomy. In actuality, a culture that makes it safe to admit and report on failure can– and in some organisational contexts must – coexist with high standards for performance. To understand why, see A Spectrum of Reasons for Failure, which lists causes ranging from deliberate deviation to thoughtful experimentation.

Which of these causes involve blameworthy actions? Deliberate deviance, fi rst on the list, obviously warrants blame. But inattention might not. If it results from a lack of effort, perhaps it’s blameworthy. But if it results from fatigue near the end of an overly long shift, the manager who assigned the shift is more at fault than the employee. As we go down the list, it gets more and more difficult to find blameworthy acts. In fact, a failure resulting from thoughtful experimentation that generates valuable information may actually be praiseworthy.

When I ask executives to consider this spectrum and then to estimate how many of the failures in their organisations are truly blameworthy, their answers are usually in single digits – perhaps two per cent to fi ve per cent. But when I ask how many aretreated as blameworthy, they say (after a pause or a laugh) 70 per cent to 90 per cent. The unfortunate consequence is that many failures go unreported and their lessons are lost.

Not All Failures Are Created Equal

A sophisticated understanding of failure’s causes and contexts will help to avoid the blame game and institute an effective strategy for learning from failure. Although an infinite number of things can go wrong in organisations, mistakes fall into three broad categories: preventable, complexityrelated, and intelligent.

Preventable failures in predictable operations. Most failures in this category can indeed be considered “bad”. They usually involve deviations from spec in the closely defined processes of highvolume or routine operations in manufacturing and services. With proper training and support, employees can follow those processes consistently. When they don’t, deviance, inattention, or lack of ability is usually the reason. But in such cases, the causes can be readily identified and solutions developed. Checklists (as in the Harvard surgeon Atul Gawande’s recent best seller The Checklist Manifesto) are one solution. Another is the vaunted Toyota Production System, which builds continual learning from tiny failures (small process deviations) into its approach to improvement. As most students of operations know well, a team member on a Toyota assembly line who spots a problem or even a potential problem is encouraged to pull a rope called the andon cord, which immediately initiates a diagnostic and problem-solving process. Production continues unimpeded if the problem can be remedied in less than a minute. Otherwise, production is halted – despite the loss of revenue entailed – until the failure is understood and resolved.

Unavoidable failures in complex systems. A large number of organisational failures are due to the inherent uncertainty of work: A particular combination of needs, people, and problems may have never occurred before. Triaging patients in a hospital emergency room, responding to enemy actions on the battlefi eld, and running a fast-growing start-up all occur in unpredictable situations. And in complex organisations like aircraft carriers and nuclear power plants, system failure is a perpetual risk.

Although serious failures can be averted by following best practices for safety and risk management, including a thorough analysis of any such events that do occur, small process failures are inevitable. To consider them bad is not just a misunderstanding of how complex systems work; it is counterproductive. Avoiding consequential failures means rapidly identifying and correcting small failures. Most accidents in hospitals result from a series of small failures that went unnoticed and unfortunately lined up in just the wrong way.

Intelligent failures at the frontier. Failures in this category can rightly be considered “good”, because they provide valuable new knowledge that can help an organisation leap ahead of the competition and ensure its future growth – which is why the Duke University professor of management Sim Sitkin calls them intelligent failures. They occur when experimentation is necessary: when answers are not knowable in advance because this exact situation hasn’t been encountered before and perhaps never will be again. Discovering new drugs, creating a radically new business, designing an innovative product, and testing customer reactions in a brand-new market are tasks that require intelligent failures. “Trial and error” is a common term for the kind of experimentation needed in these settings, but it is a misnomer, because “error” implies that there was a “right” outcome in the fi rst place. At the frontier, the right kind of experimentation produces good failures quickly. Managers who practice it can avoid the unintelligent failure of conducting experiments at a larger scale than necessary.

Leaders of the product design fi rm IDEO understood this when they launched a new innovation-strategy service. Rather than help clients design new products within their existing lines – a process IDEO had all but perfected – the service would help them create new lines that would take them in novel strategic directions. Knowing that it hadn’t yet figured out how to deliver the service effectively, the company started a small project with a mattress company and didn’t publicly announce the launch of a new business.

Although the project failed – the client did not change its product strategy – IDEO learned from it and fi gured out what had to be done differently. For instance, it hired team members with MBAs who could better help clients create new businesses and made some of the clients’ managers part of the team. Today strategic innovation services account for more than a third of IDEO’s revenues.

Tolerating unavoidable process failures in complex systems and intelligent failures at the frontiers of knowledge won’t promote mediocrity. Indeed, tolerance is essential for any organisation that wishes to extract the knowledge such failures provide. But failure is still inherently emotionally charged; getting an organisation to accept it takes leadership.

Building a Learning Culture

Only leaders can create and reinforce a culture that counteracts the blame game and makes people feel both comfortable with and responsible for surfacing and learning from failures. They should insist that their organisations develop a clear understanding of what happened – not of“who did it”– when things go wrong. This requires consistently reporting failures, small and large; systematically analysing them; and proactively searching for opportunities to experiment.

Leaders should also send the right message about the nature of the work, such as reminding people in R&D,“We’re in the discovery business, and the faster we fail, the faster we’ll succeed.” I have found that managers often don’t understand or appreciate this subtle but crucial point. They also may approach failure in a way that is inappropriate for the context. For example, statistical process control, which uses data analysis to assess unwarranted variances, is not good for catching and correcting random invisible glitches such as software bugs. Nor does it help in the development of creative new products. Conversely, though great scientists intuitively adhere to IDEO’s slogan, “Fail often in order to succeed sooner,” it would hardly promote success in a manufacturing plant.

Often one context or one kind of work dominates the culture of an enterprise and shapes how it treats failure. For instance, automotive companies, with their predictable, high-volume operations, understandably tend to view failure as something that can and should be prevented. But most organisations engage in all three kinds of work discussed above – routine, complex, and frontier. Leaders must ensure that the right approach to learning from failure is applied in each. All organisations learn from failure through three essential activities: detection, analysis, and experimentation.

Detecting Failure

Spotting big, painful, expensive failures is easy. But in many organisations any failure that can be hidden is hidden as long as it’s unlikely to cause immediate or obvious harm. The goal should be to surface it early, before it has mushroomed into disaster.

Shortly after arriving from Boeing to take the reins at Ford, in September 2006, Alan Mulally instituted a new system for detecting failures. He asked managers to colour code their reports green for good, yellow for caution, or red for problems – a common management technique. According to a 2009 story in Fortune, at his fi rst few meetings all the managers coded their operations green, to Mulally’s frustration. Reminding them that the company had lost several billion dollars the previous year, he asked straight out: “Isn’t anything not going well?” After one tentative yellow report was made about a serious product defect that would probably delay a launch, Mulally responded to the deathly silence that ensued with applause. After that, the weekly staff meetings were full of colour.

That story illustrates a pervasive and fundamental problem: Although many methods of surfacing current and pending failures exist, they are grossly underutilised. Total Quality Management and soliciting feedback from customers are well-known techniques for bringing to light failures in routine operations. Highreliability-organisation (HRO) practices help prevent catastrophic failures in complex systems like nuclear power plants through early detection. Electricité de France, which operates 58 nuclear power plants, has been an exemplar in this area: It goes beyond regulatory requirements and religiously tracks each plant for anything even slightly out of the ordinary, immediately investigates whatever turns up, and informs all its other plants of any anomalies.

Such methods are not more widely employed because all too many messengers –even the most senior executives – remain reluctant to convey bad news to bosses and colleagues. One senior executive I know in a large consumer products company had grave reservations about a takeover that was already in the works when he joined the management team. But, overly conscious of his newcomer status, he was silent during discussions in which all the other executives seemed enthusiastic about the plan. Many months later, when the takeover had clearly failed, the team gathered to review what had happened. Aided by a consultant, each executive considered what he or she might have done to contribute to the failure. The newcomer, openly apologetic about his past silence, explained that others’ enthusiasm had made him unwilling to be “the skunk at the picnic.”

In researching errors and other failures in hospitals, I discovered substantial differences across patient-care units in nurses’ willingness to speak up about them. It turned out that the behaviour of midlevel managers – how they responded to failures and whether they encouraged open discussion of them, welcomed questions, and displayed humility and curiosity – was the cause. I have seen the same pattern in a wide range of organisations.

A horrifi c case in point, which I studied for more than two years, is the 2003 explosion of the Columbia space shuttle, which killed seven astronauts (see Facing Ambiguous Threats, by Michael A. Roberto, Richard M.J. Bohmer, and Amy C. Edmondson, HBR, November 2006). NASA managers spent some two weeks downplaying the seriousness of a piece of foam’s having broken off the left side of the shuttle at launch. They rejected engineers’ requests to resolve the ambiguity(which could have been done by having a satellite photograph of the shuttle or asking the astronauts to conduct a space walk to inspect the area in question), and the major failure went largely undetected until its fatal consequences 16 days later. Ironically, a shared but unsubstantiated belief among programme managers that there was little they could do contributed to their inability to detect the failure. Postevent analyses suggested that they might indeed have taken fruitful action. But clearly leaders hadn’t established the necessary culture, systems, and procedures.

One challenge is teaching people in an organisation when to declare defeat in an experimental course of action. The human tendency to hope for the best and try to avoid failure at all costs gets in the way, and organisational hierarchies exacerbate it. As a result, failing R&D projects are often kept going much longer than is scientifi cally rational or economically prudent. We throw good money after bad, praying that we’ll pull a rabbit out of a hat. Intuition may tell engineers or scientists that a project has fatal fl aws, but the formal decision to call it a failure may be delayed for months.

Again, the remedy – which does not necessarily involve much time and expense – is to reduce the stigma of failure. Eli Lilly has done this since the early 1990s by holding “failure parties” to honour intelligent, high-quality scientifi c experiments that fail to achieve the desired results. The parties don’t cost much, and redeploying valuable resources – particularly scientists – to new projects earlier rather than later can save hundreds of thousands of dollars, not to mention kickstart potential new discoveries.

Analysing Failure

Once a failure has been detected, it’s essential to go beyond the obvious and superficial reasons for it to understand the root causes. This requires the discipline – better yet, the enthusiasm – to use sophisticated analysis to ensure that the right lessons are learned and the right remedies are employed. The job of leaders is to see that their organisations don’t just move on after a failure but stop to dig in and discover the wisdom contained in it.

Why is failure analysis often shortchanged? Because examining our fail- ures in depth is emotionally unpleasant and can chip away at our self-esteem. Left to our own devices, most of us will speed through or avoid failure analysis altogether. Another reason is that analysing organisational failures requires inquiry and openness, patience, and a tolerance for causal ambiguity. Yet managers typically admire and are rewarded for decisiveness, efficiency, and action – not thoughtful refl ection. That is why the right culture is so important.

The challenge is more than emotional; it’s cognitive, too. Even without meaning to, we all favour evidence that supports our existing beliefs rather than alternative explanations. We also tend to downplay our responsibility and place undue blame on external or situational factors when we fail, only to do the reverse when assessing the failures of others –a psychological trap known as fundamental attribution error.

My research has shown that failure analysis is often limited and ineffective –even in complex organisations like hospitals, where human lives are at stake. Few hospitals systematically analyse medical errors or process fl aws in order to capture failure’s lessons. Recent research in North Carolina hospitals, published in November 2010 in the New England Journal of Medicine, found that despite a dozen years of heightened awareness that medical errors result in thousands of deaths each year, hospitals have not become safer.

Fortunately, there are shining exceptions to this pattern, which continue to provide hope that organisational learning is possible. At Intermountain Healthcare, a system of 23 hospitals that serves Utah and southeastern Idaho, physicians’ deviations from medical protocols are routinely analysed for opportunities to improve the protocols. Allowing deviations and sharing the data on whether they actually produce a better outcome encourages physicians to buy into this programme. (See Fixing Health Care on the Front Lines, by Richard M.J. Bohmer, HBR, April 2010.)

Motivating people to go beyond firstorder reasons (procedures weren’t followed) to understanding the second- and third-order reasons can be a major challenge. One way to do this is to use interdisciplinary teams with diverse skills and perspectives. Complex failures in particular are the result of multiple events that occurred in different departments or disciplines or at different levels of the organisation. Understanding what happened and how to prevent it from happening again requires detailed, team-based discussion and analysis.

A team of leading physicists, engineers, aviation experts, naval leaders, and even astronauts devoted months to an analysis of the Columbia disaster. They conclusively established not only the fi rst-order cause –a piece of foam had hit the shuttle’s leading edge during launch – but also second-order causes: A rigid hierarchy and scheduleobsessed culture at NASA made it especially difficult for engineers to speak up about anything but the most rock-solid concerns.

Promoting Experimentation

The third critical activity for effective learning is strategically producing failures – in the right places, at the right times– through systematic experimentation. Researchers in basic science know that although the experiments they conduct will occasionally result in a spectacular success, a large percentage of them will fail. How do these people get out of bed in the morning? First, they know that failure is not optional in their work; it’s part of being at the leading edge of scientifi c dis- covery. Second, far more than most of us, they understand that every failure conveys valuable information, and they’re eager to get it before the competition does.

In contrast, managers in charge of piloting a new product or service – a classic example of experimentation in business– typically do whatever they can to make sure that the pilot is perfect right out of the starting gate. Ironically, this hunger to succeed can later inhibit the success of the offi -cial launch. Too often, managers in charge of pilots design optimal conditions rather than representative ones. Thus the pilot doesn’t produce knowledge about what won’t work.

In the very early days of DSL, a major telecommunications company I’ll call Telco did a full-scale launch of that highspeed technology to consumer households in a major urban market. It was an unmitigated customer-service disaster. The company missed 75 per cent of its commitments and found itself confronted with a staggering 12,000 late orders. Customers were frustrated and upset, and service reps couldn’t even begin to answer all their calls. Employee morale suffered. How could this happen to a leading company with high satisfaction ratings and a brand that had long stood for excellence?

A small and extremely successful suburban pilot had lulled Telco executives into a misguided confi dence. The problem was that the pilot did not resemble real service conditions: It was staffed with unusually personable, expert service reps and took place in a community of educated, techsavvy customers. But DSL was a brandnew technology and, unlike traditional telephony, had to interface with customers’ highly variable home computers and technical skills. This added complexity to the service-delivery challenge in ways that Telco had not fully appreciated before the launch.

A more useful pilot at Telco would have tested the technology with limited support, and unsophisticated customers. It would have been designed to discover everything that could go wrong – instead of proving that under the best of conditions everything would go right. (See Designing Successful Failures.) Of course, the managers in charge would have to have under- stood that they were going to be rewarded not for success but, rather, for producing intelligent failures as quickly as possible.

In short, exceptional organisations are those that go beyond detecting and analysing failures and try to generate intelligent ones for the express purpose of learning and innovating. It’s not that managers in these organisations enjoy failure. But they recognise it as a necessary by-product of experimentation. They also realise that they don’t have to do dramatic experiments with large budgets. Often a small pilot, a dry run of a new technique, or a simulation will suffi ce.

The courage to confront our own and others’ imperfections is crucial to solving the apparent contradiction of wanting neither to discourage the reporting of problems nor to create an environment in which anything goes. This means that managers must ask employees to be brave and speak up – and must not respond by expressing anger or strong disapproval of what may at fi rst appear to be incompetence. More often than we realise, complex systems are at work behind organisational failures, and their lessons and improvement opportunities are lost when conversation is stifl ed.

Savvy managers understand the risks of unbridled toughness. They know that their ability to fi nd out about and help resolve problems depends on their ability to learn about them. But most managers I’ve encountered in my research and consulting work are far more sensitive to a different risk – that an understanding response to failures will simply create a lax work environment in which mistakes multiply.

This common worry should be replaced by a new paradigm – one that recognises the inevitability of failure in today’s complex work organisations. Those that catch, correct, and learn from failure before others do will succeed. Those that wallow in the blame game will not.