Cognitive task analysis (CTA) methods are traditionally used to conduct small-sample, in-depth studies. In this case study, CTA methods were adapted for a large multi-site study in which 102 anesthesiologists worked through four different high-fidelity simulated high-consequence incidents. Cognitive interviews were used to elicit decision processes following each simulated incident. In this paper, we highlight three practical challenges that arose: (1) standardizing the interview techniques for use across a large, distributed team of diverse backgrounds; (2) developing effective training; and (3) developing a strategy to analyze the resulting large amount of qualitative data. We reflect on how we addressed these challenges by increasing standardization, developing focused training, overcoming social norms that hindered interview effectiveness, and conducting a staged analysis. We share findings from a preliminary analysis that provides early validation of the strategy employed. Analysis of a subset of 64 interview transcripts using a decompositional analysis approach suggests that interviewers successfully elicited descriptions of decision processes that varied due to the different challenges presented by the four simulated incidents. A holistic analysis of the same 64 transcripts revealed individual differences in how anesthesiologists interpreted and managed the same case.

Effective decision-making in crisis events is challenging due to time pressure, uncertainty, and dynamic decisional environments. We conducted a systematic literature review in PubMed and PsycINFO, identifying 32 empiric research papers that examine how trained professionals make naturalistic decisions under pressure. We used structured qualitative analysis methods to extract key themes. The studies explored different aspects of decision-making across multiple domains. The majority (19) focused on healthcare; military, fire and rescue, oil installation, and aviation domains were also represented. We found appreciable variability in research focus, methodology, and decision-making descriptions. We identified five main themes: (1) decision-making strategy, (2) time pressure, (3) stress, (4) uncertainty, and (5) errors. Recognition-primed decision-making (RPD) strategies were reported in all studies that analyzed this aspect. Analytical strategies were also prominent, appearing more frequently in contexts with less time pressure and explicit training to generate multiple explanations. Practitioner experience, time pressure, stress, and uncertainty were major influencing factors. Professionals must adapt to the time available, types of uncertainty, and individual skills when making decisions in high-risk situations. Improved understanding of these decisional factors can inform evidence-based enhancements to training, technology, and process design.

Despite the increased importance attributed to distributed improvisation in major crises, few studies investigate how central authorities can promote a harmonic, coordinated national response while allowing for distributed autonomy and improvisation. One idea implicit in the literature is that central authorities could help track and tackle common decision bottlenecks as they emerge across "improvising" local authorities as a result of shared, dynamic external constraints. To explore this idea we map central functions needed to roll-out vaccines to local populations and identify and classify bottlenecks to decision-making by local authorities managing COVID-19 vaccine roll-out in Norway. We found five bottlenecks which emerged as vaccine roll-out progressed, three of which could feasibly have been addressed by changing the local authorities' external constraints as the crisis developed. While the national crisis response strategy clearly allowed for distributed improvisation, our overall findings suggest that there is potential for central authorities to address external constraints in order to ease common bottlenecks as they emerge across local authorities responding to the crisis. More research is to explore alternative centralized response strategies and assess how well they effectively balance centralized and distributed control. The study contributes to the growing literature examining the interaction between local and centralized response in crisis management.

In the coming years, artificial intelligence (AI) will pervade almost every aspect of the health care delivery system. AI has the potential to improve patient safety (e.g. diagnostic accuracy) as well as reduce the burden on clinicians (e.g. documentation-related workload); however, these benefits are yet to be realized. AI is only one element of a larger sociotechnical system that needs to be considered for effective AI application. In this paper, we describe the current challenges of integrating AI into clinical care and propose a sociotechnical systems (STS) approach for AI design and implementation. We demonstrate the importance of an STS approach through a case study on the design and implementation of a clinical decision support (CDS). In order for AI to reach its potential, the entire work system as well as clinical workflow must be systematically considered throughout the design of AI technology.

We identify the value and usage of a cognitive artifact used by hospital nurses. By analyzing the value and usage of workaround artifacts, unmet needs using intended systems can be uncovered. A descriptive study employed direct observations of registered nurses at two hospitals using a paper workaround ("brains") and the Electronic Health Record. Field notes and photographs were taken; the format, size, layout, permanence, and content of the artifact were analyzed. Thirty-nine observations, spanning 156 hr, were conducted with 20 nurses across four clinical units. A total of 322 photographs of paper-based artifacts for 161 patients were collected. All participants used and updated "brains" during report, and throughout the shift, most were self-generated. These artifacts contained patient identifiers in a header with room number, last name, age, code status, and physician; clinical data were recorded in the body with historical chronic issues, detailed assessment information, and planned activities for the shift. Updates continuously made during the shift highlighted important information, updated values, and tracked the completion of activities. The primary functional uses of "brains" are to support nurses' needs for clinical immediacy through personally generated snapshot overviews for clinical summaries and updates to the status of planned activities.

Cognitive work analysis is useful to develop displays for complex situations, but it has not been well explored in providing support for human-automation coordination. To fill this gap, we propose a degree of automation (DOA) layering approach, demonstrated by modeling an automated financial trading domain, with a goal of supporting interface design in this domain. The abstraction hierarchy and the decision ladder each adopted an additional layer, mapping functions allocated to the trader and to the automation. In addition to the mapping, we marked the four stages of automation on the decision ladder to provide guidance on representing the function allocation at the task level. Next, we compared the DOA layering approach to how automation was represented in the cognitive work analysis literature. We found that a DOA-layered decision ladder, which included well-developed knowledge of the stages and levels of automation, can be suited to modern automated systems with different DOAs. This study suggests that the DOA layering approach has important implications for designing automation displays and deciding stages and levels of automation and may be a useful approach for modeling adaptive automation.

The affordability of wearable psychophysiological sensors has led to opportunities to measure the mental workload of operators in complex sociotechnical systems in ways that are more objective and less obtrusive. This study primarily focuses on the sensors themselves by investigating low-cost and wearable sensors in terms of their accuracy, obtrusiveness, and usability for research purposes. Two sensors were assessed on their accuracy as tools to measure mental workload through heart rate variability (HRV): the E3 from Empatica and the emWave Pro from HeartMath. The BioPatch from Zephyr Technology, which is an U.S. Food and Drug Administration-approved device, was used as a gold standard to compare the data obtained from the other 2 devices regarding their accuracy on HRV. Linear dependencies for 6 of 10 HRV parameters were found between the emWave and BioPatch data and for 1 of 10 for the E3 sensor. In terms of research usability, both the E3 and the BioPatch had difficulty acquiring either sufficiently high data recording confidence values or normal distributions. However, the BioPatch output files do not require postprocessing, which reduces costs and effort in the analysis stage. None of the sensors was perceived as obtrusive by the participants.

The design and adoption of decision support systems within complex work domains is a challenge for cognitive systems engineering (CSE) practitioners, particularly at the onset of project development. This article presents an example of applying CSE techniques to derive design requirements compatible with traditional systems engineering to guide decision support system development. Specifically, it demonstrates the requirements derivation process based on cognitive work analysis for a subset of human spaceflight operations known as . The results are presented in two phases. First, a work domain analysis revealed a comprehensive set of work functions and constraints that exist in the extravehicular activity work domain. Second, a control task analysis was performed on a subset of the work functions identified by the work domain analysis to articulate the translation of subject matter states of knowledge to high-level decision support system requirements. This work emphasizes an incremental requirements specification process as a critical component of CSE analyses to better situate CSE perspectives within the early phases of traditional systems engineering design.

The effectiveness of different instrument approach charts to deliver minimum visibility and altitude information during airport equipment outages was investigated. Eighteen pilots flew simulated instrument approaches in three conditions: (a) normal operations using a standard approach chart (), (b) equipment outage conditions using a standard approach chart (), and (c) equipment outage conditions using a prototype decluttered approach chart (). Errors and retrieval times in identifying minimum altitudes and visibilities were measured. The standard-outage condition produced significantly more errors and longer retrieval times versus the standard-normal condition. The prototype-outage condition had significantly fewer errors and shorter retrieval times than did the standard-outage condition. The prototype-outage condition produced significantly fewer errors but similar retrieval times when compared with the standard-normal condition. Thus, changing the presentation of minima may reduce risk and increase safety in instrument approaches, specifically with airport equipment outages.

We describe a phenomenon viewed through the conceptual lens of a naturalistic decision making perspective: a loss of system resilience, due to increased difficulty in performing macrocognition functions, associated with the implementation of new information technology. Examples of the phenomenon collected in a targeted literature review are characterized by stakeholder groups, technology, typical changes in workflow before and after implementation, and potential impacts on macrocognition and patient outcomes for four clinical care environments. The loss of system resilience is due to increased difficulty in performing macrocognition functions: 1) sensemaking due to less effective cognitive warm-up and collaborative framing strategies, 2) detecting events due to missing trends in data and changes to orders, and 3) coordinating due to less clinical knowledge during scheduling and updating information, and less effective cross-checks. Potential impacts to patient safety include an increase in unnecessary care, missed care, delays in diagnoses and treatment, redundant care, inaccurate diagnoses, medication errors, and adverse events. We recommended future conceptually-driven research in other complex, sociotechnical settings order to develop useful metrics and reduce the risk of incurring undesirable and unnecessary impacts on cognitive work associated with new technology.

Adoption of clinical decision support has been limited. Important barriers include an emphasis on algorithmic approaches to decision support that do not align well with clinical work flow and human decision strategies, and the expense and challenge of developing, implementing, and refining decision support features in existing electronic health records (EHRs). We applied decision-centered design to create a modular software application to support physicians in managing and tracking colorectal cancer screening. Using decision-centered design facilitates a thorough understanding of cognitive support requirements from an end user perspective as a foundation for design. In this project, we used an iterative design process, including ethnographic observation and cognitive task analysis, to move from an initial design concept to a working modular software application called the Screening & Surveillance App. The beta version is tailored to work with the Veterans Health Administration's EHR Computerized Patient Record System (CPRS). Primary care providers using the beta version Screening & Surveillance App more accurately answered questions about patients and found relevant information more quickly compared to those using CPRS alone. Primary care providers also reported reduced mental effort and rated the Screening & Surveillance App positively for usability.

The objective of this work was to assess the functional utility of new display concepts for an emergency department information system created using cognitive systems engineering methods, by comparing them to similar displays currently in use. The display concepts were compared to standard displays in a clinical simulation study during which nurse-physician teams performed simulated emergency department tasks. Questionnaires were used to assess the cognitive support provided by the displays, participants' level of situation awareness, and participants' workload during the simulated tasks. Participants rated the new displays significantly higher than the control displays in terms of cognitive support. There was no significant difference in workload scores between the display conditions. There was no main effect of display type on situation awareness, but there was a significant interaction; participants using the new displays showed improved situation awareness from the middle to the end of the session. This study demonstrates that cognitive systems engineering methods can be used to create innovative displays that better support emergency medicine tasks, without increasing workload, compared to more standard displays. These methods provide a means to develop emergency department information systems-and more broadly, health information technology-that better support the cognitive needs of healthcare providers.

The decision sciences are increasingly challenged to advance methods for modeling analysts, accounting for both analytic strengths and weaknesses, to improve inferences taken from increasingly large and complex sources of data. We examine whether psychometric measures-personality, cognitive style, motivated cognition-predict analytic performance and whether psychometric measures are competitive with aptitude measures (i.e., SAT scores) as analyst sample selection criteria. A heterogeneous, national sample of 927 participants completed an extensive battery of psychometric measures and aptitude tests and was asked 129 geopolitical forecasting questions over the course of 1 year. Factor analysis reveals four dimensions among psychometric measures; dimensions characterized by differently motivated "top-down" cognitive styles predicted distinctive patterns in aptitude and forecasting behavior. These dimensions were not better predictors of forecasting accuracy than aptitude measures. However, multiple regression and mediation analysis reveals that these dimensions influenced forecasting accuracy primarily through bias in forecasting confidence. We also found that these facets were competitive with aptitude tests as forecast sampling criteria designed to mitigate biases in forecasting confidence while maximizing accuracy. These findings inform the understanding of individual difference dimensions at the intersection of analytic aptitude and demonstrate that they wield predictive power in applied, analytic domains.

Electronic health record systems (EHRs) can improve safety and reliability of health care, but they can also introduce new vulnerabilities by failing to accommodate changes within a dynamic EHR-enabled health care system. Continuous assessment and improvement is thus essential for achieving resilience in EHR-enabled health care systems. Given the rapid adoption of EHRs by many organizations that are still early in their experiences with EHR safety, it is important to understand practices for maintaining resilience used by organizations with a track record of success in EHR use. We conducted interviews about safety practices with 56 key informants (including information technology managers, chief medical information officers, physicians, and patient safety officers) at two large health care systems recognized as leaders in EHR use. We identified 156 references to resilience-related practices from 41 informants. Framework analysis generated five categories of resilient practices: (a) sensitivity to dynamics and interdependencies affecting risks, (b) basic monitoring and responding practices, (c) management of practices and resources for monitoring and responding, (d) sensitivity to risks beyond the horizon, and (e) reflecting on risks with the safety and quality control process itself. The categories reflect three functions that facilitate resilience: reflection, transcending boundaries, and involving sharp-end practitioners in safety management.

Displaying both the strategy that information analysis automation employs to makes its judgments and variability in the task environment may improve human judgment performance, especially in cases where this variability impacts the judgment performance of the information analysis automation. This work investigated the contribution of providing either information analysis automation strategy information, task environment information, or both, on human judgment performance in a domain where noisy sensor data are used by both the human and the information analysis automation to make judgments. In a simplified air traffic conflict prediction experiment, 32 participants made probability of horizontal conflict judgments under different display content conditions. After being exposed to the information analysis automation, judgment achievement significantly improved for all participants as compared to judgments without any of the automation's information. Participants provided with additional display content pertaining to cue variability in the task environment had significantly higher aided judgment achievement compared to those provided with only the automation's judgment of a probability of conflict. When designing information analysis automation for environments where the automation's judgment achievement is impacted by noisy environmental data, it may be beneficial to show additional task environment information to the human judge in order to improve judgment performance.

Although access to Internet health information can be beneficial, solving complex health-related problems online is challenging for many individuals. In this study, we investigated the performance of a sample of 60 adults ages 18 to 85 years in using the Internet to resolve a relatively complex health information problem. The impact of age, Internet experience, and cognitive abilities on measures of search time, amount of search, and search accuracy was examined, and a model of Internet information seeking was developed to guide the characterization of participants' search strategies. Internet experience was found to have no impact on performance measures. Older participants exhibited longer search times and lower amounts of search but similar search accuracy performance as their younger counterparts. Overall, greater search accuracy was related to an increased amount of search but not to increased search duration and was primarily attributable to higher cognitive abilities, such as processing speed, reasoning ability, and executive function. There was a tendency for those who were younger, had greater Internet experience, and had higher cognitive abilities to use a bottom-up (i.e., analytic) search strategy, although use of a top-down (i.e., browsing) strategy was not necessarily unsuccessful. Implications of the findings for future studies and design interventions are discussed.