James S. Todd Memorial Award for Patient Safety Research

Quantitative Measurement of the Progression of Clinical Expertise

Matthew B. Weinger, MD, Alison G. Vredenburgh, PhD, Martin P.Paulus, MD, Craig Mckenzie, PhD

This project measured the acquisition of clinical expertise throughout anesthesia residency training and compared resident performance with that of more expert anesthesia providers. The original hypothesis was that clinical expertise could be described by specific behavioral and cognitive processes. The original Aims of the project were: 1) Examine the progression of residents’ technical skills over the course of anesthesia residency by measuring actual clinical performance; 2) Examine how cognitive models of a complex clinical decision evolve throughout residency training; and 3) Define the critical differences in technical and cognitive performance among anesthesia residents, CRNA, and Board-Certified anesthesiologists.

During the original project period, we collected task, workload, and vigilance data from hundreds of anesthesia cases performed by clinicians having a range of experience and training. The results suggest that, in similar cases, the residents’ task patterns, workload, and vigilance changes over the course of their training. Experienced clinicians are better able to allocate their cognitive, perceptual, and manual resources with changing clinical demands. We also examined how brand new anesthesia residents are taught patient care in the first weeks of training.

The training of novices initially focuses on manual tasks with later addition of cognitive tasks. The workload of early trainees is very high and their capability to assimilate new data/tasks may be limited.

The NPSF funding allowed us to successfully undertake several related studies. Because our initial work suggested that task analysis could be an important way to assess what clinicians do during actual patient care (as well as during simulated care), we conducted a study to evaluate the intra- and inter-rater reliability of this technique. We also specifically examined the nature of intravenous drug preparation and administration tasks in cardiac and non-cardiac general anesthetics. During the case-specific data collection funded by the NPSF, we began to develop and refine our conceptual model and methods to capture “non-routine events” (or NRE). NRE are any events identified by clinicians or observers that represent deviations from optimal care for that specific patient in that specific clinical situation.

In conclusion, NPSF funding had a real impact on my career and on patient safety research through the development of more rigorous methods of measuring care processes and their effects on safety. In addition to the results cited above, the early support
of NPSF, and of the Anesthesia Patient Safety Foundation (APSF), fostered my team’s formative research agenda around the time of the release of the Institute of Medicine’s To Err Is Human. Since then, we have actively participated in 10 federally funded patient safety projects amounting to almost $5 million in direct costs, from the Agency for Healthcare Research and Quality (AHRQ), Veterans Administration Health Services
Research and Development (HSR&D), National Library of Medicine, National Heart Lung and Blood Institute, and the United States Food and Drug Administration (FDA). My team has used these techniques to conduct high-quality research on various relevant patient safety topics (e.g., the effects of workload, fatigue, and distractions, as well as event capture) that has been published in a wide range of journals . We have also extended NPSF-funded research in anesthesiology in San Diego to other disciplines (e.g., intensive care and operating room nursing, surgeons, inpatient medicine and pediatrics, ambulatory medicine). We have developed collaborations across the United States and our tools are also being used or emulated by investigators in other countries.

Auditory Warning Signals in Critical Care Settings

Yan Xiao, PhD; F. Jacob Seagull, PhD, Christopher D. Wickens, PhD, Colin Mackenzie, MB

Alarms are critical defenses against preventable harms to patients, and alarms are ubiquitous because of ease in attaching them to almost any device. Yet instead of being a mechanism safeguarding the patient, alarms often increase workload, make communications difficult, and produce a hostile work environment.
Care providers do not always respond to alarms. Why?

A multi-disciplinary team studied auditory alarms using methodologies of cognitive engineering to understand interactions with auditory alarms. Eye-tracking devices were used to understand both auditory and visual sources of information during critical care. Clinicians were found to ignore alarms and concentrate directly on the patient when they were overloaded. Patient monitors are scattered around the patient, making it difficult to gather visual information, yet auditory alarms provide little useful information. Interviews were used to understand reasons why alarms were ignored. Alarms were reported as compensatory solutions to poorly designed floor layouts, and physicians frequently overused monitored beds even for relatively healthy patients. These factors increased the occurrences of false alarms. This project demonstrated the value of cognitive engineering methods in understanding challenges to patient safety.

Theory and Methods for Minimizing Name Confusion Errors

Bruce L. Lambert, Ph.D., Michael Cohen, RPh, MS, Prahlad Gupta, PhD, Gordon Schiff, MD, Clement Yu, PhD

We conducted a study of the effect of similarity and prescribing frequency on pharmacists’ and consumers’ visual perception of drug names. We found that accuracy in visual perception of drug names was strongly influenced by both prescribing frequencyand spelling similarity. The most commonly prescribed drugs were perceived much more accurately than the least commonly prescribed drugs. Also, we found that drug names with many similar “neighbors” were perceived less accurately than names with few neighbors. Findings were published in Social Science & Medicine in 2003. The study provided further validation for our computerized measures of similarity and further bolstered our claim that objective measures of similarity should be used as part of FDA’s pre-approval screening process for new drug names. In December 2003, the FDA announced that they would begin using a computer system that incorporated the measures we had validated in our NPSF-funded study. In an indirect way, our NPSF project led to changes in the way drug names are screened and approved in the United States. Health Canada has announced that they will be following FDA’s example, and they plan to adopt computerized name screening as part of their screening process as well.

In 2003, an offshoot of the NPSF work was funded by a four-year grant from the Agency for Healthcare Research and Quality. The results were published in Social Science & Medicine in 2010. We conducted auditory perception experiments to assess the impact of similarity, familiarity, background noise and other factors on clinicians’ and laypersons’ ability to identify spoken drug names. Accuracy increased significantly as the signal-tonoise (S/N) ratio increased, as subjective familiarity with the name increased and as the national prescribing frequency of the name increased. For clinicians only, similarity to other drug names reduced identification accuracy, especially when the neighboring names were frequently prescribed. When onename was substituted for another, the substituted name was almost always a more frequently prescribed drug. We concluded that objectively measurable properties of drug names could be used to predict confusability. The magnitude of the noise and familiarity effects suggested that they may be important targets for intervention.

Looking for Trouble in All the Right Places: Electronic Decision Support for Error Reduction in a Large HMO

Gabriel J. Escobar, MD, Mary Anne Armstrong, MA, Larry I. Palmer, JD, Linda J. Nozick, PhD, Andrea Kabcenell, MPH

The original hypothesis of this study was that it is possible to use commonly available hospital information systems to identify situations that have a high probability of being associated with human error. The objectives were to develop and validate a set of computer algorithms that would detect specific patterns of clinical, laboratory, and/or administrative data associated with extremely high-risk events in obstetrics. The investigators have found that electronic scanning techniques can be highly effective in detecting high-risk events in perinatal care. The results have also confirmed that these techniques are superior to existing methods (e.g., voluntary incident reporting) used by hospital quality assurance departments.