Extending the Wingspan of Data: Increasing Lift Through Artificial Intelligence for In-Time Aviation Safety
By Commercial Aviation Data Analysis Safety Team, NASA System-Wide Safety Project
Today’s airliners are information behemoths, collecting data at incredible volumes and speeds. Pilots are undoubtedly aware of how much data just one aircraft generates. An A350 has as many as 400,000 data parameters. A B-737-800 can generate as much as 20 terabytes of engine data per hour. Airlines and their pilots have embraced data collection and analysis through safety data programs such as Flight Operations Quality Assurance (FOQA) and voluntary safety reporting programs such as the Aviation Safety Action Program (ASAP) and are continuously innovating to improve safety.
But how much analysis can possibly be done with all this data? Realistically, not enough. Today, safety management has been primarily achieved through safety data mining. Data mining, defined here as “the practice of analyzing large databases to generate new information”—isn’t enough for the dynamic air transportation system of the future. NASA believes that safety can be advanced even further if we harness all this data with innovative use of machine learning (ML) and data science to make sense of large and disparate data sets and then apply the information it provides to hazard identification and safety risk mitigation.
For a large span of aviation’s history, improvements to safety have been made reactively, after a fatal accident has occurred. In the late 1990s, the industry began incorporating more proactive changes, relying on emerging data sources to feed safety advancements. In the future safety data analysis realm, NASA has a vision that optimized algorithms will be used to translate data into knowledge to help monitor, assess, mitigate, and ensure against threats and hazards and the risks they present. This is different from the industry’s current process of manually tagging and verifying events. In today’s system, there’s a massive human workload to satisfy the level of data analysis required in our safety systems, taking valuable time and resources that could be devoted to other aspects of safety management.
The future of operational safety will likely rely heavily on the development and use of “intelligent” algorithms that can provide valuable proactive—and even predictive—insights to safety performance parameters.
To that end, NASA launched the System-Wide Safety (SWS) project in 2018. The SWS project identifies and addresses safety threats by leveraging advances in ML and data science for better in-time risk management and safety assurance. The massive amount of data the aviation industry is collecting can be intimidating, however, if this data deluge and ML (a subdiscipline of artificial intelligence [AI]) are harnessed, it would further enhance the future air transportation system to remain the safest one in history. This is the overarching goal of NASA’s SWS project.
SWS data science work includes the following activities:
- Assessing how well different data sources can identify vulnerabilities and accident or incident precursors individually and jointly. Current sources of data that pilots provide to the system include their ASAP reports and inputs through FOQA, but NASA is adding significantly more sources.
- Determining which ML algorithms best apply to various data sources.
- Identifying what visualization techniques will most intuitively reveal these vulnerabilities to domain experts to make changes to the operations (e.g., pilots, cabin crew, dispatchers, etc.).
- Identifying which data subsets are most relevant and incorporating domain expert feedback on the operational significance of the identified vulnerabilities.
NASA has a 2050 vision for the future of commercial aviation known as “Sky for All.” A core capability of this is In-Time System-Wide Safety Assurance, which requires mastering data analysis. The superordinate goal of SWS is to deploy numerous programs and data systems to gain what many in the aviation community have termed “safety intelligence.” NASA envisions that this safety data could be leveraged as part of the flight-planning process—that in the future, pilots will preflight their aircraft with an already heightened awareness of the threats that are present for that particular flight, even when the threats aren’t intuitively obvious or detectable to them.
SWS data scientists have been working collaboratively with ALPA and airline partners to develop ML algorithms that identify patterns and relationships previously undetectable by exceedance-based (i.e., rules-based) methods or traditional statistical models. Vulnerability detection and mitigation uses different types of data from different sources to monitor operations, perform analyses that assess the nature of operations and determine if safety issues are present or likely to arise, and then take actions to mitigate any safety issues.
Today, this vulnerability-detection process at each airline uses a combination of primarily FOQA and ASAP data to identify safety concerns, safety performance trends, or outlier events. Once the potential vulnerability is identified, it then gets vetted for action by human subject-matter experts (SMEs). This process, while valuable, can create an undesirable lag of days, weeks, or even longer in safety data reporting and action.
NASA envisions a future in which ML algorithm “detectives” are trained to identify anomalies and precursors that can be difficult to detect in large and varied data sets. The algorithms are designed to look for patterns in large amounts of data to quickly discover potential risks so that they may be mitigated and managed before becoming threats. These risks could be clues to hidden issues that haven’t yet caused a safety event or emergent risks that are complex and difficult to understand. This allows human SMEs to spend their time on investigating root causes, gathering operational context, and working on mitigations to implement on the line.
Computerized anomaly-detection tools identify those few data points that stick out when compared to most of the data generated during normal operations. These detection methods aid in the vulnerability-discovery process, because both known and unknown problems are relatively rare and are, therefore, among statistically anomalous data points. Vulnerability discovery involves using an algorithm to find the statistical anomalies, removing data points representing known problems, and examining the remaining statistical anomalies to separate the vulnerabilities from the false alarms (see Figure 1—statistically anomalous data is magnified to better enable detection of potential safety-critical risks).
The NASA SWS researchers use a process called active learning with inputs from domain experts (specifically from pilots) to train the algorithm. SWS contends with one of the most laborious and expensive challenges in the quest for safety intelligence: labeling data. Pilots are an important piece of the development because they must spend time tagging data points as operationally anomalous or false alarms, and active learning ensures that they only label data points that are most helpful in training ML how to distinguish between the labels.
Once a NASA SWS algorithm is trained, it’s constantly reviewing new data. If it appears operationally anomalous, the data point is handed over to the freshly trained SWS classifier software application where the algorithms, wise from pilots’ expert tagging, can decide whether it’s a real issue or just another false alarm. SWS seeks to develop ML algorithms that can identify these “data mavericks” before they emerge as risks. Working together with human experts who make the final assessment, ML can recommend mitigations that may require little to no intervention by pilots and other operators. This is called “predictive safety intelligence.” Ultimately, NASA believes that these advanced algorithms will help the industry quickly leverage a mountain of data into a resource that can help us make operational changes before safety events occur.
Airlines collect every speck of data from flight ops to maintenance, and safety intelligence helps to address the gap between what airlines think is happening and what’s actually unfolding. To better understand the data, efforts are focused on sharing data sets and enabling “decision fusion” across various types of airspace system data. Decision fusion significantly improves systemwide safety intelligence for risk management and safety assurance. SWS is developing ways to perform fusion and to display complex data and information optimized for human interpretation, understanding, and use. The integration of advanced data analytics with human expertise and operational insight will advance the concept of in-time safety intelligence, and we expect that it will become an invaluable tool in the pursuit of continually improving safety. The feedback from thousands of ALPA pilots has been paramount to advancing NASA’s work in this area.
NASA’s data analytics collaborations allow airlines to manage data already being collected to better understand operational safety-related performance challenges, and we anticipate that it will enable in-time safety actions and mitigations in the future. This will allow safety professionals to more accurately assess how beneficial these safety risk-mitigation actions are and when these actions may need to change. Whether that’s achieved through updated training programs, changes to policy, or even aircraft design, enhanced AI-based predictive in-time safety will help the industry keep focused on our ever-changing operating environment.
A just culture and a nonpunitive reporting environment are critical components to collecting secured and protected quality data. Key to success is commitment by all stakeholders, including ALPA, airlines, the FAA, and many others, to incorporate shared responsibility for safety into internal practices to ensure the protection of data submitters and the safety data itself. ALPA has been an unrelenting advocate for strong data protections throughout the evolution of voluntary safety reporting programs. NASA has been proactively engaged with ALPA to develop these new technological capabilities in support of aviation and its stakeholders’ enduring quest to explore how aviation can maintain, and even improve upon, its commendable safety record.
Ongoing collaboration between NASA and airlines will allow SWS to continue to develop ML and visualization tools that are more helpful to the U.S. airline industry than what’s currently in use and will yield greater value in terms of safety, efficiency, and passenger comfort. As we extend the wingspan on data, the increased lift will come not just from better ways to understand complex data and its use to advance aviation safety. Equally important is the advocacy and leadership of ALPA and its members who’ve contributed to and will continue to contribute to safety science and advancing responsible pilot-centered AI technology.
Harnessing Pilot Data for Safer Operations
By Capt. Steve Jangelis (Delta), Chair, ALPA Aviation Safety Group
Every airline flight is an opportunity to share valuable safety data via Flight Operations Quality Assurance (FOQA)/flight data monitoring and safety reporting programs like the Aviation Safety Action Program (ASAP) or safety managements systems (SMSs). FOQA and ASAP data is deidentified, and protections are in place against punitive company action and regulator enforcement. The insights garnered from this data provide a unique opportunity to identify system vulnerabilities, share lessons learned, and support safety risk-management and safety-assurance processes.
Both FOQA and ASAP provide otherwise unavailable insight into not only what happened, but the all-important “why.” This is the foundation and the value of nonpunitive safety data sharing, and we must continue to evolve and improve data collection, sharing, and analytical capabilities as we pursue our goal of continually enhancing safety performance. But how does this work in practice, and where do we need to go?
As ALPA’s Aviation Safety chair and the cochair of the Aviation Safety Information Analysis Sharing (ASIAS) Executive Board, we, as stakeholders, work to ensure that we’re proactively addressing safety issues emerging from the analysis of data and that government and industry collaborative programs are collecting the right data to improve aviation safety. We also ensure that pilot safety data is always used in a confidential, nonpunitive environment while also broadly sharing critical safety intelligence. While participating in programs such as ASIAS, we focus on how the data is used at a national level to improve systemic safety though all aspects of the operation. And with that information, ALPA advocates for data-driven decision-making.
The ASIAS program is a joint industry-government initiative to collect and analyze aviation safety data voluntarily shared from the aviation community. Our shared data can provide a more complete picture of emerging trends that might not be visible at each individual carrier. The Association has actively participated in ASIAS since the program’s inception in 2007, advocating for the protection of safety data and those who report safety events. This ensures that pilot perspective and experience are taken into consideration throughout the process.
Collaboration is key, and these groups not only work with airlines, but also with representatives from the National Air Traffic Controllers Association, aircraft manufacturers, the FAA, and technical groups such as NASA. By working together, we can advance safety, and contributing to industry groups and pooling our data have been key contributors to the impressive safety record in the United States.
The industry has historically viewed safety data through ASAP reports and FOQA data, but there’s an untapped wealth of data outside of these traditional sources to be harnessed to enhance safety. We have yet to incorporate sources such as fatigue reports, Line Operations Safety Assessments, individual airline safety risk assessments, and other emerging resources. As industry stakeholders evolve their SMSs, more emphasis will be placed on a diverse pool of data to ensure we’re learning from all aspects of airline operations.
Even “uneventful” flights can help produce safety information if we leverage data appropriately. Our industry has done a good job investigating what goes wrong on the flight deck, but we haven’t historically focused on what goes right on the flight deck and how that positively affects safety. How do pilots intervene to create a safe outcome when automation, ATC, or weather challenge the flight?
We must use a combined approach of identifying both the good and the bad in the system to understand and reinforce the positive, while also designing better solutions to address any issues. This is why emerging data analytics such as NASA’s In-Time System-Wide Safety Assurance are so valuable. We need more efficient data analysis techniques to investigate our ever-growing data set and comprehensive data management and improved capabilities to share them in a timelier way. With finite resources and time, aviation subject-matter experts need to focus on building safety solutions rather than digging through mountains of data to identify trends. But we need to do this the right way, and it will take time.
At the national level, work is under way to organize a workflow process that mirrors an SMS, based on FAA guidance and internal airline practices for hazard identification and risk management. The new SMS-based process will help to systematically organize emerging issues, risk analysis, and mitigation strategies so that all relevant stakeholders will be able to monitor developing trends. The goal is to provide airline and industry decision-makers with the high-level information they need to make changes within their organizations.
Our industry is rapidly evolving, with record-high traffic, new entrants sharing the airspace, and technology that both supports and potentially threatens aviation safety. The need for vigilance and proactive safety analysis has never been more critical. We must operate safely every day, and data is the key component to supporting this effort. Strong airline and federal data protection policies will ensure that this data is able to flow from pilots to the appropriate sources. It’s often said in safety circles that our next accident is already in our data set somewhere. ALPA is proactively evolving its safety management practices to ensure that we have the capabilities and support mechanisms in place to prevent a data point from becoming an incident or accident.
