Report from a Workshop on
Assembling the Scientific Basis
for Progress on Patient Safety

Appendix B
 

 
Incident Reporting Systems in Medicine and Experience With the
Aviation Safety Reporting System

Charles Billings, MD
Institute for Ergonomics
Ohio State University

This appendix contains the talk by Charles Billings, MD, Chief Scientist (retired), NASA Ames, on the lessons learned from incident reporting in aviation. Dr. Billings designed, started and managed the Aviation Safety Reporting System 22 years ago when at NASA's Ames Research Center. His talk framed the discussion on the second day of the workshop. The lessons he abstracted from the aviation experiences represent the best guidance available on incident reporting. Medicine is quite different from aviation in many ways. What proved successful in aviation is not likely to transfer directly and literally to medicine. However, the lessons Dr. Billings has derived are generic and can serve as a guide to develop successful systems in medicine.
 

This is only a brief digest of what I think are some of the most compelling and important issues regarding formalized incident reporting. I hesitate to use the term systems to describe the many different approaches to incident reporting. To call them systems would dignify them unjustifiably, at least at this point. But I acknowledge that there are various requirements that, in the past, have shaped the systems during design and implementation. The experience with the aviation safety reporting system, in which I developed the ideas that form the basis for this presentation, exemplifies these requirements. The first and most critical requirement for a successful incident reporting system is a demonstrated, tangible, widely agreed upon need for more and better information. If a substantial portion of a community believes that it already knows what needs to be known about incidents, then it is unlikely to give more than lip service to finding out what incident reporting can discover. Strong, widely held consensus that more and better information is needed, is essential for the development of successful incident reporting. The second requirement is for a respected body, one independent of the influences of other stakeholders, to conduct the collection and analysis of data. This is an absolute requirement. Some disagree with this, but many incident reporting schemes have come to grief over the years by being installed in that body which was charged with oversight or in some other body that was subservient to the body charged with oversight, of the community whose activities were being assessed by the incident reporting system.

Two other factors have led to incident reporting schemes coming to grief in the recent past. One is adequate funding to permit expertise to be brought to bear. These systems cannot be run with a couple of clerks and a keypunch operator. For any useful degree of understanding of the reports, incident reporting requires expertise at the South end equal to that which was on the North end, that is, there must be expertise used in evaluating the reports as they are obtained. The other factor necessary is time. There must be adequate time to establish the system, to gather data and, more importantly to sell the system, its input, data, information, reports, and conclusions, to those authorities who represent the ultimate recipients of its products.

To some, these prerequisite requirements may seem obvious, but the simple fact is that incident reporting schemes have sometimes failed for want of them.

Let me address very briefly the question of whether incident reporting should be mandated. Those of you who have read the two volumes of materials that were sent in preparation for the conference (and who were still reading carefully by the time you got to the end of them!), will have recognized that the New York hospital incident reporting system is a mandatory system. But the other documents in these volumes you find in several points the observation that incidents are underreported. Well, that will come as a less than an overwhelming surprise to most. Some claim that mandatory reporting is necessary.

They say that if you don't have mandatory reporting everyone will protect themselves. I add only that if you do have mandatory reporting, everyone else will protect themselves. Despite mandatory reports in some locales, the number of incidents reported is small. For example, according to the materials, Carolina had 15 reports in its first year and Colorado had 17 reports in 2 years. I leave it to you to decide whether you think that is adequate or not. I will here make the claim that, in some form, in one way or another, all incident reporting becomes voluntary. It either becomes voluntary because of inertia on the part of reporters, or it becomes voluntary because of constraints within the establishment and the environment, or it becomes voluntary because hospitals (and there are at least one or two in your two source books) decide that they are not required to report this particular event because of the fine print in that particular incident reporting regulation or statute.

Underreporting is a recognized problem. But I'm not at all sure that that is the critical problem. Staying within the medical context and referring to the materials from your source books, there are enough reports of mishaps with potassium chloride, lidocaine, vincristine and other drugs and devices to have made it very clear that a problem with these exists. The information that these events occur is already present. We may well ask what it is that keeps us from making progress on safety, given that we already know about the existence of these problems. What is added by more formal, elaborate (and expensive) incident reporting? We may hypothesize that if the events had become known to a central organization, there would have been discovery of trends and systematic evaluation of what was going on that, in turn, would have led to some unspecified but important change. But it is clear, even at this stage, that the hypothetical, safe, non-punitive incident reporting system is not the primary obstacle to making progress on safety in medicine. Existing agencies can take action once a problem is adequately defined, explored, and explained (and, I would have to add, publicized). A central question facing us is not really how many there are, but how many is enough. That is to say, there are already many signals that point to a variety of failures. Part of the consensus that needs to be formed for successful incident reporting is consensus about what is a sufficiently strong signal to warrant action. The sourcebook materials suggest that such a consensus remains to be developed.

Stakeholders influence the likelihood of developing consensus. Now once again I've dipped into the sourcebook to the CEO's retrospective report on the Ben Kolb case, which is volume 1 tab 4. I've added two here because they were spoken of elsewhere but I found that the CEO listed a rather substantial number of stakeholders in that two-page report.

That, if you will forgive me, is a lot of people, organizations, interests, stakes and conflicting goals. If I am prepared to argue to you that incident reporting requires consensus, it seems reasonable to ask whether consensus can be reached among these stakeholders on anything including whether it is day or night, whether the sun is moving to the west or the east.

It will surely be asked by many why consensus among these stakeholders is necessary. What does it buy if you have it? First, it buys a substantial number of people and organizations as advocates for incident reporting and the system that supports it. This is essential to keeping the system a working, living entity in a contentious, politically driven environment. Second, it buys the participation of those people who can, if they wish, do something with the information you produce.

It might be useful to turn the question of consensus on its head. What does it cost you if you don't have it? At this point I will diverge just slightly to answer Richard Cook's question, posed at the beginning of this conference. He asked what critical failures we now recognize after a lifetime of research. This led me to think back on the history of the aviation safety reporting system and to identify the most significant failure in its development. The NASA Aviation Safety Reporting System was established in response to the cries of virtually everybody in the community, directly at the order of the FAA Administrator working through the Administrator of the National Aeronautics and Space Administration. NASA was chosen because we represented a respected and presumably objective third party. We were given one month to establish the system, guide it through about as many stakeholders as are shown in the table, and present it to the Administrator as a done deal. We (or, rather, I), in my naiveté made an assumption. I assumed that since the chief of the FAA was asking for it, the FAA wanted it. This was a bad mistake.

Over the past 21 years, the NASA Aviation Safety Reporting System has had the support of virtually everybody in the United States aviation community to a greater or lesser extent with one notable exception. The exception is the people under the FAA Administrator-and there are roughly 22,000 of them-whom he did not consult before he came to NASA and asked for an incident reporting system. We do well to remember that a primary issue is who may be hurt by reporting. This is especially of concern where use of immunity (and, originally in the ASRS, transactional immunity as well) is a prominent feature.

What was the largest segment of the FAA aside from the air traffic control system? Divisions involving regulations and enforcement. Whose enmity did we earn the day this thing was announced? Those who had to make it work in the community. That is the worst mistake I made in 40 years.

So the absence of consensus about the need for and characteristics of incident reporting was a critical flaw in the development of ASRS. What does it cost you if you don't have consensus? It costs you in passivity; resistance to acting on a lot of the recommendations derived from data received; delays in implementing; even ridicule. "You know it isn't a problem," was one form this passivity and resistance took. Another was "You guys just don't understand." Consensus is critical and it must include all the stakeholders, not just a few or a special team or a division or an agency or a company.

I also want to point out that consensus isn't enough. It's necessary but it is not sufficient. Incident reporting also requires understanding and that is even tougher to establish. Incident reports are unique sets of data. Each incident is unique and not easily classified or pigeonholed. Generalizations may be possible in retrospect, given enough detailed data and enough understanding of the data. But this means understanding details of the task, the context, the environment, and its constraints. This is why you have got to have experts looking at reports. Simply constructing taxonomies is grossly insufficient and it permits only counting of incidents that fall under phrase a, b or c of the taxonomy.

Counting incidents is a waste of time. Why? Because incident reporting is inherently voluntary. Because the population from which the sample is drawn is unknown and therefore can not be characterized, and because you lose too much information and gain too little in the process of condensing and indexing these reports unless you do what we were fortunate enough to do blindly, and that is keep all the narratives. Every ASRS report is in fact a narrative rather than a categorization. And the ASRS keeps every word, except those necessary to be discarded to de-identify the reports. The evaluation of incidents in such a system requires an understanding of all that. A deep understanding.

Let me give you an example. We got a report about 2 years after the ASRS started which, paraphrased, said "I had a frightening experience this morning. I took my airplane, a Lockheed 1011, off at Los Angeles headed for Vancouver, and it took full right aileron to keep it in the air. The airplane wanted very badly to roll sharply to the left." Fortunately, it was a cold day at Los Angeles, the load in the airplane was light, the pilot, carrying full right aileron all the way and with help from the first officer, managed to struggle around the airport and get it back on the ground at Los Angeles and no one was hurt. Whereupon it was found by the flight engineer that the two outboard spoilers on the right wing were both fully extended and had been since the airplane left the ramp.

Why was this not caught? Neither of those two spoilers carries sensors for spoiler deflection, so the crew, once the airplane was powered up, had no indication in the cockpit that those devices were extended. You can't see them from the cockpit. The airplane was powered down at the time the first officer made the walk around, and in an all-hydraulic airplane, controls can be in any position whatever when there is no hydraulic power on. We had three other similar reports after that, but we really didn't wait for a second report. We immediately got why that happened and reported it, de-identified, to those people we thought needed to know about it. As a matter of fact, the reporting back was conducted by telephone and that incident was handled initially within 24 hours.

Counts are not what you are after. You can't, in a voluntary system, determine the incidence or prevalence of a problem. But if you get two or three reports like that, you can be pretty sure that you have a problem, and that is what you are after in this kind of a system. On the other hand, having extolled the kinds of things incident reporting can do for you more briefly than I usually do, incident reporting is not the whole answer. I think this is particularly germane to your considerations here. In aviation accident reporting is mandatory, and accidents are very precisely defined. Yesterday we discussed the problems associated with defining and characterizing what an accident is. In aviation we have a much easier job, and the distinction between an accident, which must be reported, and an incident is easier to make than is the case in health care. Keep in mind that incident reporting in aviation is voluntary under most circumstances. Incident reporting and accident reporting are not substitutes one for the other. They are complementary.

Incident reports, properly interpreted, provide new knowledge. It is important to remember that this is all they provide. They are not a panacea. They only provide knowledge about what is going on in a particular domain or area of purview. The incident reports themselves tell you little. They generally do not tell you how it could be done better. The analysts, looking at many incidents, may or may not be able to tell you how it may be done better. They are more likely to know, in many cases, more than the incident reporters. Certainly they are more likely, over time, through reading these things, to gain some experience and understanding. But like descriptive epidemiology anywhere else, incident reports can provide only descriptions of phenomena. Analytic studies and other research remain critical to a full understanding of the phenomena.

Acting on the new knowledge that comes from combining incident reporting with analytical studies and detailed research is not and should not be within the purview of an incident reporting system. I emphasize this because too many people have thought that incident reporting was the core and primary component of what was needed. These people thought that simply from the act of collecting incidents, solutions and fixes would be generated sui generis and that this would make safety better. Although much is unclear about incident reporting systems, this one fact is quite clear: incident reporting is only one component of what you need. Using new knowledge gained from these systems must be the responsibility of the stake-holders I have listed. This says that the use of this knowledge to spur new analysis, new research, to guide regulation, to inform management decision making, to change performance, must be the responsibility of everybody else in the domain. It must not be the job of the people who run the incident reporting system, who have got to remain objective and disinterested in order to do this job as it needs to be done.

The incident reporting system must remain an objective and disinterested third party to retain its effectiveness. Otherwise, sooner or later, it will be accused, properly, of bias. There are enough biases we cannot get rid of; it clearly does not make sense to add more. So, I ask you to consider among other things, what should be reported. Is harm required? How much harm? What kind of harm? Is an adverse event-and that is the term Kaplan and his colleagues are using in Texas-determined by processes or by outcomes? Can incidents be differentiated from accidents in these settings? The difference is very critical with respect to legal liability. Unfortunately as we see the increasing trend, referred to in your sourcebook, toward criminal prosecution as well as civil liability, this question becomes increasingly important. It becomes increasingly important because it is entirely possible that if someone decides to go after a criminal indictment following such an incident or accident or mishap, you may be harboring evidence. That is not a particularly desirable position for the hapless expert working in one of these outfits to be in. I believe the legal term for it is obstruction of justice. This was a serious problem in Canada, where a violation of what in the US is called an air navigation order is not only a civil violation, it is also a criminal offense. The people who work within the Canadian confidential air safety reporting system had to be specifically immunized by Parliment to allow them to do this task. That would be true in any nation that was governed by the Napoleonic code. It is increasingly a potential threat in the common-law nations as well, given what I said yesterday about corporate manslaughter, corporate crimes involving damage or injury to people. So does it matter? Twenty years ago I would certainly have said, "Probably not." Now I would say, "Yes, I think it probably does." I am not sure yet of the way to get around this, and I'm doing a fairly extensive study at the present time to try to figure out how much it matters. But there is certainly, in aviation at least, an increasing trend toward imprisoning or invoking other criminal sanctions against people for offenses committed in the course of their employment in this industry.

Should records ever be protected? There are certainly going to be differences of opinion with respect to that. Can they ever be protected? One of the most treasured bits of data within the aviation industry, are the cockpit voice recorder (CVR) tapes. The police in New Zealand, after an accident there in 1995, subpoenaed, then sought a court order, then went to trial to get the CVR tape from that accident in connection with prosecution of the pilot and co-pilot, one of whom lived and the other of whom did not. The police prevailed at the appellate level and were given the CVR tape as criminal evidence. That has led to a very substantial effort in New Zealand to decide what are the limits of evidence gathering in this environment. And, as a matter of fact, the New Zealand Law Commission is looking into that at the present moment. This is a very serious question. Another question, related to it, is should the analytical records of an incident reporting system be protected? Sufficiently detailed, these analytical records may make it almost unnecessary to get to the raw data. In this country, thus far, raw data, from the aviation safety reporting system once de-identified has been considered hearsay and therefore inadmissible in the vast majority of cases. That may be the balm that can be spread upon the wounds in some of these other nations within the common-law states but in any event, this question remains an important one.

And finally, is an incident reporting system possible in this, that is to say, the medical environment? Thinking of the number of stakeholders involved, thinking of the enormity of the hypothesized problem, I think that we all need to reserve judgment. We need to have more evidence, gathered by different means-converging evidence to bring to bear on this question. Until this problem is scoped, neither the National Patient Safety Foundation nor anyone else is going to be able to figure out how to tackle it, so I think that that should clearly be one of your front-end research objectives. But however large the problem, the diversity amongst the stakeholders is very worrying. Now in the supplementary sourcebook, I've included some view graphs describing the NASA Aviation Safety Reporting System. A couple of those deal with our advisory committee, which represents a broad segment of the large, complex, diverse, aviation industry. But the size, complexity, and diversity of aviation are simply not on the scale of medicine; in fact, aviation is nowhere near. I would offer you some hope that it may be possible to define areas within medicine in which there is a somewhat smaller constituency, a somewhat smaller group of stakeholders and within which, therefore, the problem may be slightly more tractable than it will be if you decide that your purview is all of medicine. I am not sure I would know at this point, even after 20-plus years of experience with this business, how to design a system for medicine. I think I could perhaps design a system for some subsegments of medicine, that are more tightly circumscribed, but I hasten to add that I am not even sure of that.

These are some of the questions you are going to have to face as you begin to think about incident reporting systems in this particular domain. They make success in medicine much less certain, in contrast with the undoubted success that incident reporting systems have had in the aviation environment and a very few others. There are many other detailed issues to consider in developing incident reporting within medicine, issues that you will need to address in detail, should you chose to pursue this line of work. All these other questions deserve attention. But you will find, I think, that the ones I have presented here today are among the largest, most important ones that will be critical to the success or failure of any new incident reporting system in medicine. I think those kinds of questions need to be thought about very carefully, deliberately, intelligently, and thoroughly before beginning.

Question 1: Would you comment about your efforts over the years to keep the ASRS alive?

Well, that was the first thing we had to do. In some ways, this was actually fairly straightforward because the industry, by and large, recognized that they had a problem. The Administrator of the FAA had a political problem, but this is not particularly germane in our discussion today. The larger aviation industry, including all the stakeholders, had a real problem and recognized it. So this particular seed fell upon reasonably fertile ground. We got along reasonably well initially until we began to recognize the depths of the disinterest of the FAA working staff in this program. We did everything we could to interest them. We went out and we talked to as many of them as we could get to. We worked within FAA headquarters to try to get them to understand what they had put us into.

This was not entirely successful, but we got along by and by fairly well, until political problems arose concerning a freedom of information act request asking for an evaluation and discussion of air traffic congestion in terminal areas. The ASRS was obligated to respond to this, and it did. We put together a report whose principal finding of concern was that near mid-air collisions appeared from our data to be most likely in the highest density terminal control areas in the United States. The next most likely areas for near mid-air collisions were what were called Airport Radar Service Areas. These were the terminal areas around secondary but still busy airports. Near mid-air collisions were less likely to occur anywhere else.

This would probably not surprise you all much and it did not surprise us. But, it was of course a direct affront to the agency which regulates air space in the United States, and which had been saying, as part of its campaign to establish some 10 or 12 new terminal control areas, that they were the safest place in the air. The internal political response to this report was complex but amounted to an effort to remove the legal immunity that protected reports to the ASRS. Although this was forestalled by the intervention of Congress, it is clear in retrospect that the survival of the system as a working entity depended on the consensus within the community that had been reached early in the development of the system. It is one example of the importance of stakeholders who are directly and actively involved.

Question 2: Does the fact that pilots are at personal risk when flying have something to do with the success of the system? Is that a reason that you think this system might work much better in aviation than it might be made to work in medicine?

Well, it is certainly true that pilots are normally the first people at the scene of an accident. But the answer in my view is no, it is not the sense of personal risk that has driven pilots to participate in the system. I read the first 30,000 of these reports that came in. I have not read the ensuing 340,000 or so. One is continually amazed by the number of people who do far more than they need in order to qualify for immunity under this system. People write pages of descriptions, they send tapes, they volunteer to come to our offices. They make clear in a number of ways that they want us to understand, in all its rich detail, the complexity of the incident in which they had been involved and which they had already reported by mail. One indication of the sincerity and dedication of these people is that some have had such close interaction that they have become personal friends. There is no question about the motivation of the pilot community in general with respect to safety issues, none whatever. The reports are not grudging acknowledgement or pro forma filings but rather quite rich and human descriptions of troubling, often frightening events. I believe that the reporting to this system is motivated not by the sense of personal risk that attaches to flying but rather from two major factors: (1) the sincere interest in improving safety by identifying hazards and (2) the sincere (and, in my view, well grounded) belief that the system to which they are reporting uses that information productively and deliberately to improve safety rather than simply as a means of counting failures.

Parenthetically, I know that the system has been "gamed" in the course of adjudication of labor issues. I think it would be very surprising if it were not. But that becomes relatively easy to pick out; one notices it. One occasionally gets some very good information out of those reports incidentally, because some of that gaming is over issues that are real and substantive. The fatigue and jet lag issue is one that I think of immediately although that is by no means the only one. But once again you have to have a human-a smart human and a motivated human-who understands and has been there at the sharp end of the process from which reports come as well as at the analytical end. And I do not believe there is any substitute for that.

Question 3: Could you comment on expertise issues in the procedure for examining incidents and looking for patterns?

Pilots get rusty after about so long; as you know, many of our analytical people are retired pilots. We have instituted a set of rotations for these people, whereby they would come and work for us for a period of time and then they would move on and we would replace them with others coming out of the cockpit. Now that has been in place for probably fifteen years now, and it is also true with respect to our air traffic controllers. No one can stay adequately familiar with the niceties in an industry this complex unless he or she has been actively working in it until comparatively recently.

Interestingly, although we have been overwhelmed at times by the number of reports, the system for handling those reports has changed remarkably little. They are all hand-read by an expert who is appropriate to the field of consideration in the report. The reports are still identified at that stage and are being handled as classified material, so that if more data is required we can get it directly from the reporter. We still have the pilot's name, phone numbers, things of that sort. We will attempt to call the pilot back and discuss in more detail the incident that was reported and sometimes secondary issues as well, before the reports are de-identified. We do classification on a number of classification fields. The reports are processed, the narratives are de-identified with respect to person, flight numbers, and things of that sort, and then those narrative reports are keyed directly into a data base as well as of 60 or 70 fields of coded data. So the classification system, which is primarily used for indexing, provides access to the narrative. It is also possible to search the narrative fields by words, by phrases, and considerable amount of research has been done using that, as well. We would have lost all of that had we destroyed the narrative.

Question 4: What does it cost to run this system now?

To the best of my knowledge it's about two million dollars per year for about 35,000 to 40,000 reports.

Question 5: Currently in the medical field, legal requirements for discoverability drive access to these incident reports. In Louisiana and Arizona, for instance, between 50% and 100% of incident reports are discoverable in civil cases. This has created a substantial incentive to make these reports narrow and short. There is a strong emphasis, stated explicitly in most cases as directions to the reporting individual, to limit what is described to objective rather than subjective information. For example, regarding a fall in the hospital, what people are instructed to record and report is the bare fact that someone slipped and fell. If the individual said, "I slipped on water" the incident reporter can put that in quotes but needs to avoid concluding that they slipped on water. Thus these reports are limited to observations of the scene, noting the times, locations and so on. Are the ASRS reports specifically limited in those ways-that is, narrow, succinct, objective and without any opinions about the incident or how it occurred? Will such a system produce results similar to those obtained via the ASRS?

Absolutely not. We rely on preservation of this material as classified material initially, on rapid de-identification, and on blurring enough of the details of such a report so that it would at least be difficult to introduce it as evidence. But we also need the reports of people, of pilots, who we understand are experts in their field and whose opinions, thoughts, and observations we value. We do not try to limit them or channel their descriptions in a particular way. Indeed, it would surely limit the value of the reports if we were to artificially constrain the narratives to eliminate any conclusions or opinions from the pilots. But these reports are data, and we all recognize that data has biases and limits and the important thing is to try to understand these. This goes back to the problem of throwing away data when you throw away the narratives-counting doesn't work, that is not what you are after in incident reporting. Limiting the narratives in the way you describe would be throwing away the data even before you got it, a serious mistake in my view.

Now with respect to de-identifying the events in the database-that can be a bit hard. I think particularly of this city in which we find ourselves, and arriving here early one morning shortly before I retired from NASA, to see a 747 sitting ingloriously in the dirt, between Runway 10-Right and the taxiway adjacent to it. Now that was not, as it happened, an accident. The monetary value of it was not high enough to require defining it that way, partly because it was soft mud and there was little damage to the plane. But I would have hated to try to de-identify that report sufficiently so that it would be in the database but unidentifiable. A 747 is one big moose when it is sitting out there where everyone can see it.