Patent Application
20230373654
The present application claims the benefit of India Provisional Patent Application 202241029171, filed May 20, 2022, titled “SYSTEM AND METHOD FOR PROCESSING AUDIO DATA OF AIRCRAFT CABIN ENVIRONMENT”, naming NITIN KUMAR GOYAL et al. as inventors, which is incorporated herein by reference in the entirety.
Currently, some aircraft are equipped with image-based video devices, which stream video to the crew panel, inside cockpit displays, and recorders for monitoring and capturing of events inside the cabin for direct surveillance and for later replay. These video cameras are currently not equipped with analytics systems and do not provide alerts to crew members. Such video devices have some limitations. For example, video cameras have physical restrictions of where the cameras can be installed. Additionally, it is not practical to display a complete view of the cabin on a display all at the same time.
In one aspect, embodiments of the inventive concepts disclosed herein are directed to a system. The system may include an array of audio devices installed in an aircraft cabin of an aircraft, each of the audio devices including a microphone, wherein each of the audio devices is configured to output microphone audio data on a channel. The system may include an audio processing device communicatively coupled to the array of audio devices, the audio processing device installed in the aircraft. The audio processing device may be configured to receive the microphone audio data from each of the array of audio devices. The audio processing device may include a processor. The audio processing device may be configured to: monitor the channel of microphone audio data from each of at least two of the audio devices; analyze the monitored channels of microphone audio data to identify at least one of: a gender of a speaker captured by a given microphone, an estimated age of the speaker captured by the given microphone, a number of speakers captured by the given microphone, a language of the speaker captured by the given microphone, an estimated location of the speaker captured by the given microphone, at least one selective word captured by the given microphone, a loud noise having a predetermined loudness, or a threat scenario; identify an event to be an occurrence of a predetermined event based at least on the analyzing of the monitored channels of microphone audio data; based at least on the identified event and the analyzing of the monitored channels of microphone audio data, generate an event report associated with at least one channel of microphone audio data associated with the identified event; output the event report to a crew panel; and output at least a portion of microphone audio data of the at least one channel of microphone audio data associated with the identified event to the crew panel.
In a further aspect, embodiments of the inventive concepts disclosed herein are directed to a system. The system may include audio processing device including a processor. The audio processing device may be configured to: obtain or generate processed audio data, the processed audio data having been processed from monitored channels of microphone audio data from an array of audio devices, the array of audio devices installed in an aircraft cabin of an aircraft, each of the audio devices including a microphone, wherein each of the audio devices is configured to output microphone audio data on a channel; obtain or generate an event report associated with at least one channel of microphone audio data associated with an identified event, the identified event to being an occurrence of a predetermined event; obtain or generate at least a portion of the processed audio data of the at least one channel of microphone audio data associated with the identified event; analyze the processed audio data and the event report to label portions of the processed audio data as at least one of: a threat scenario, an unusual sound captured by a given microphone, at least one selective word captured by the given microphone, a temperament of a speaker captured by the given microphone, a speaker profile, a language of the speaker captured by the given microphone, a location of the speaker captured by the given microphone, or an aircraft system failure; store the labeled portions of the processed audio data; and output at least some of the labeled portions of the processed audio data to a crew panel or an offboard device.
In a further aspect, embodiments of the inventive concepts disclosed herein are directed to a system. The system may include a post event investigation (PEI) device including a processor. The post event investigation device may be configured to: obtain or generate processed audio data, the processed audio data having been processed from monitored channels of microphone audio data from an array of audio devices installed in an aircraft cabin of an aircraft, wherein each of the audio devices is configured to output microphone audio data on a channel; obtain or generate an event report associated with at least one channel of microphone audio data associated with an identified event, the identified event to being an occurrence of a predetermined event; obtain or generate at least a portion of the processed audio data of the at least one channel of microphone audio data associated with the identified event; obtain labeled portions of the processed audio data or analyze the processed audio data and the event report to label portions of the processed audio data as the labeled portions of the processed audio data, wherein each of the labeled portions of the processed audio data is labeled as at least one of: a threat scenario, an unusual sound captured by a given microphone, at least one selective word captured by the given microphone, a temperament of a speaker captured by the given microphone, a speaker profile, a language of the speaker captured by the given microphone, a location of the speaker captured by the given microphone, or an aircraft system failure; and recreate the identified event at least by replaying, for a user, at least one of the labeled portions of the processed audio data.
In a further aspect, embodiments of the inventive concepts disclosed herein are directed to a method. The method may include: obtaining or generating, by at least one post event investigation (PEI) device, processed audio data, the processed audio data having been processed from monitored channels of microphone audio data from an array of audio devices installed in an aircraft cabin of an aircraft, wherein each of the audio devices is configured to output microphone audio data on a channel, wherein each of the at least one PEI device comprises at least one processor; obtaining or generating, by the at least PEI device, an event report associated with at least one channel of microphone audio data associated with an identified event, the identified event to being an occurrence of a predetermined event; obtaining or generating, by the at least one PEI device, at least a portion of the processed audio data of the at least one channel of microphone audio data associated with the identified event; by the at least one PEI device, obtaining labeled portions of the processed audio data or analyzing the processed audio data and the event report to label portions of the processed audio data as the labeled portions of the processed audio data, wherein each of the labeled portions of the processed audio data is labeled as at least one of: a threat scenario, an unusual sound captured by a given microphone, at least one selective word captured by the given microphone, a temperament of a speaker captured by the given microphone, a speaker profile, a language of the speaker captured by the given microphone, a location of the speaker captured by the given microphone, or an aircraft system failure; and recreating, by the at least one PEI device, the identified event at least by replaying, for a user, at least one of the labeled portions of the processed audio data.
Implementations of the inventive concepts disclosed herein may be better understood when consideration is given to the following detailed description thereof. Such description makes reference to the included drawings, which are not necessarily to scale, and in which some features may be exaggerated and some features may be omitted or may be represented schematically in the interest of clarity. Like reference numerals in the drawings may represent and refer to the same or similar element, feature, or function. In the drawings:
Before explaining at least one embodiment of the inventive concepts disclosed herein in detail, it is to be understood that the inventive concepts are not limited in their application to the details of construction and the arrangement of the components or steps or methodologies set forth in the following description or illustrated in the drawings. In the following detailed description of embodiments of the instant inventive concepts, numerous specific details are set forth in order to provide a more thorough understanding of the inventive concepts. However, it will be apparent to one of ordinary skill in the art having the benefit of the instant disclosure that the inventive concepts disclosed herein may be practiced without these specific details. In other instances, well-known features may not be described in detail to avoid unnecessarily complicating the instant disclosure. The inventive concepts disclosed herein are capable of other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.
As used herein a letter following a reference numeral is intended to reference an embodiment of the feature or element that may be similar, but not necessarily identical, to a previously described element or feature bearing the same reference numeral (e.g., 1, 1a, 1b). Such shorthand notations are used for purposes of convenience only, and should not be construed to limit the inventive concepts disclosed herein in any way unless expressly stated to the contrary.
Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by anyone of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
In addition, use of the “a” or “an” are employed to describe elements and components of embodiments of the instant inventive concepts. This is done merely for convenience and to give a general sense of the inventive concepts, and “a” and “an” are intended to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.
Finally, as used herein any reference to “one embodiment,” or “some embodiments” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the inventive concepts disclosed herein. The appearances of the phrase “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiment, and embodiments of the inventive concepts disclosed may include one or more of the features expressly described or inherently present herein, or any combination of sub-combination of two or more such features, along with any other features which may not necessarily be expressly described or inherently present in the instant disclosure.
Broadly, embodiments of the inventive concepts disclosed herein may be directed to a system and a method configured to provide audio analytics onboard and/or offboard of an aircraft for safety and surveillance of an aircraft cabin.
Some embodiments may include audio analytics onboard an aircraft for safety and surveillance of an aircraft cabin and may provide a crew alert (e.g., on a crew panel, which may also be interchangeably referred to as a crew call panel or a crew management system). The audio data may also be analyzed onboard and/or offboard of the aircraft, such as on the ground or in the air, to further extract information from live-streamed and/or on-demand recorded audio to recognize any unusual event during flight and/or for training of crew and optimization of services provided in the cabin. For example, the audio data may be transmitted via satellite communication (SATCOM) to a ground site that provides analytics in real time, such as if there is a high threat unusual event is perceived.
Some embodiments may include an array (e.g., a network) of distributed audio devices, which may include microphones, that live stream (e.g., by a wired and/or wireless connection) audio data to a bridge device, such as an aircraft interface computing device and/or an audio processing device, which in turn can communicate a stream of data to the crew panel, to a recording device, and/or to cockpit and other communication systems. The communication systems may further transmit such data to ground equipment for further detailed analysis.
Some embodiments provide an automatic system that provides advanced situation analysis, reports, and alarms, which can increase safety and security inside the cabin. In some embodiments, the automatic system may reduce a workload of the crew. For example, an unusual sound coming from an aircraft system, such as landing gear, an engine, a fan cowl, or any other moving body of the aircraft, and from high vibrations may help identify an impending aircraft system failure. The system may help gauge the situation at a location where a camera blind spot exists.
Referring now to
For example, the user may be a pilot or crew member. The user may interface with the system 100 via any suitable user interfaces, such as the crew panel 112. The at least one user interface may be implemented as any suitable user interface, such as a control panel, a touchscreen (e.g., of a display unit computing device and/or the crew panel 112), a multipurpose control panel, a control panel integrated into a flight deck, a cursor control panel (CCP) (sometimes referred to as a display control panel (DCP)), a keyboard, a mouse, a trackpad, at least one hardware button, a switch, an eye tracking system, and/or a voice recognition system. The user interface may be configured to receive at least one user input and to output the at least one user input to a computing device (e.g., 106, APD 108, and/or AAPD 110). For example, a pilot or crew member of the aircraft 102 may be able to interface with the user interface 110 to: view and/or listen to an event report and/or an alert; and/or listen to a portion of aircraft cabin audio provided by the APD 108 and/or AAPD 110.
The array 104 of audio devices 202 may include a plurality of audio devices 202 installed in an aircraft cabin of the aircraft 102 and may be implemented as any suitable array of any suitable audio devices. In some embodiments, some or all of the audio devices 202 may be in-flight entertainment (IFE) devices; for example, as shown in
The at least one aircraft interface computing device 106 may be implemented as any suitable computing device, such as an aircraft router computing device (e.g., a flight operation and maintenance exchange (FOMAX) computing device or a smart router). The at least one aircraft interface computing device 106 may include any or all of the elements, as shown in
The at least one APD 108 may be implemented as any suitable audio processing computing device. The APD 108 may be installed in the aircraft 102. The APD 108 may be a line replaceable unit (LRU). As shown in
In some embodiments, the at least one APD 108 may be further configured to: based at least on the monitoring of the channel of microphone audio data from at least two of the audio devices 202, assign a severity level of a plurality of severity levels to a portion of the channel of the microphone audio data from at least two of the audio devices, wherein the plurality of severity levels are predetermined severity levels and include a highest severity level and at least one other severity level, each of the at least one other severity level having a relative severity that is less than the highest severity level. In some embodiments, the at least one APD 108 may be further configured to: store the portion of the channel of microphone audio data from at least two of the audio devices 202 for a particular duration based on the assigned severity level, wherein a particular portion having an assigned highest severity level is stored longer than another particular portion having another assigned severity level.
In some embodiments, the at least one APD 108 may be further configured to: output a command, to at least one of the array 104 of audio devices 202, to enable or disable at least one audio stream of at least one microphone 204 of the at least one of the array 104 of audio devices 202.
In some embodiments, the at least one APD 108 may be further configured to: based at least on the identified event and the analyzing of the monitored channels of microphone audio data, generate a threat alarm associated with the identified event; and output the threat alarm to the crew panel 112.
In some embodiments, the at least one APD 108 may be further configured to: enhance an audio quality of the monitored channels of microphone audio data, for example, by performing noise cancellation operations on the monitored channels of microphone audio data to remove background noise.
In some embodiments, the at least one APD 108 may be further configured to: output processed audio data to the at least one second APD 110 (e.g., an AAPD), the processed audio data having been processed from the monitored channels of microphone audio data; and output the event report to the at least one second APD 110.
The at least one second APD 110 (e.g., an AAPD) may be implemented as any suitable audio processing computing device. The AAPD 110 may be installed in the aircraft 102 or offboard of the aircraft. In some embodiments, the AAPD 110 may be a line replaceable unit (LRU). As shown in
In some embodiments, the at least one AAPD 110 may be further configured to: output the at least some of the labeled portions of the processed audio data to the crew panel 112 based on a severity level of a plurality of severity levels, wherein the plurality of severity levels are predetermined severity levels and include a highest severity level and at least one other severity level, each of the at least one other severity level having a relative severity that is less than the highest severity level.
In some embodiments, the at least one AAPD 110 may be further configured to: update the event report based at least on the labeled portions of the processed audio data; and output the updated event report to the crew panel 112 or the offboard device (e.g., PEI 118).
In some embodiments, the at least one AAPD 110 may be further configured to: based at least on the labeled portions of the processed audio data, generate a threat alarm associated with at least one of the labeled portions; and output the threat alarm to the crew panel 112.
In some embodiments, the at least one AAPD 110 may be further configured to: enhance an audio quality of the labeled portions of the processed audio data, for example, by performing noise cancellation operations on the labeled portions of the processed audio data to remove background noise.
In some embodiments, the at least one AAPD 110 may be further configured to: generate a textual transcript corresponding to at least one of the labeled portions of the processed audio data, and output the textual transcript to the crew panel 112 or the offboard device (e.g., PEI 118).
The at least one PEI device 118 may be implemented as any suitable audio processing computing device. The PEI device 118 may be installed in the aircraft 102 or offboard of the aircraft 102 (e.g., in a cloud computing environment). In some embodiments, the PEI device 118 may be a line replaceable unit (LRU). As shown in
In some embodiments, the at least one PEI device 118 may be further configured to: recreate the identified event at least by replaying (e.g., via the display 608 and/or the at least one speaker 610), for the user, the at least one of the labeled portions of the processed audio data with a displayed textual transcript corresponding to the at least one of the labeled portions of the processed audio data.
In some embodiments, the at least one PEI device 118 may be further configured to: recreate the identified event at least by replaying (e.g., via the display 608 and/or the at least one speaker 610), for the user, the at least one of the labeled portions of the processed audio data with at least one displayed video segment corresponding to captured video of a particular location of the identified event on the aircraft 102 associated with the at least one of the labeled portions of the processed audio data.
In some embodiments, the at least one PEI device 118 may be further configured to: access a virtual rendering of the aircraft 102; and virtually recreate the identified event at least by replaying (e.g., via the display 608 and/or the at least one speaker 610), for the user, the at least one of the labeled portions of the processed audio data with the virtual rendering of the aircraft 102 showing a location of the identified event. In some embodiments, the virtual rendering of the aircraft 102 includes a virtual rendering of the aircraft cabin layout including seating zones and seating arrangements.
In some embodiments, one or more of the at least one of the labeled portions of the processed audio data include a voice of a selected crew member, wherein the at least one PEI device 118 may be further configured to: virtually recreate the identified event at least by replaying (e.g., via the display 608 and/or the at least one speaker 610), for the user, the at least one of the labeled portions of the processed audio data with a visual depiction of at least one status characteristic of the selected crew member's voice, the at least one status characteristic including at least one of: a loudness, a temperament, an accent, or an indication of yelling. In some embodiments, the at least one PEI device 118 may be further configured to: generate a report card of behavior of the selected crew member during the identified event based at least on the at least one status characteristic of the selected crew member's voice. In some embodiments, the at least one PEI device 118 may be further configured to: generate a report card of behavior of the selected crew member during the identified event based at least on the at least one status characteristic of the selected crew member's voice and a phase of flight during the identified event; and store the report card.
In some embodiments, the at least one PEI device 118 may be further configured to: use the recreated identified event as a training exercise at least by replaying (e.g., via the display 608 and/or the at least one speaker 610), to a crew member in training, at least one of the labeled portions of the processed audio data.
In some embodiments, the APD 108 and the AAPD 110 may be implemented as a single audio processing device having the functionality of the APD 108, the AAPD 110, and/or the PEI device 118.
Referring now to
In some embodiments, the audio data may be processed, and analytics may be performed on multiple human verbal and other factors, such as selective word (e.g., critical word) detections, pitch and frequency analysis, volume, abusive tone, screaming, screeching, shouting, panic, and exceptional acoustic events like crack, bang, thud, gun shots and sounds from product or mechanism failures, pressurization change indicator events, outside engine audio qualities, doors opening or closing, etc.
In some embodiments, the recorded audio data may also be transmitted to the PEI device 118 (e.g., at a ground station) over a wireless link or through physical medium transfer for further processing of the audio data to detect any anomaly and/or signature of anomaly coming from various systems on board the aircraft 102. Additionally, the recorded audio data may be used for service optimization and training of the crew members.
Referring now to
Some embodiments may include an APD 108 configured to receive the audio data channels and to provide commands to audio interface for audio channel selection (e.g., for some or all of microphones 204 to be enabled or disabled) as required for processing. The APD 108 may be an edge computing device for processing of the audio data for multiple type of enhancements and analytics like gender, language, selective words, threat scenarios, and unusual events like loud sound.
In some embodiments, the aircraft 102 also includes an AAPD 110. The APD 108 may provide processed audio data to the AAPD 110. In some embodiments, the AAPD 110 may include multiple analytics modules and advanced audio processing modules. The audio processing modules can process the data for spectrogram improvements like voice selection, speech selection, noise reduction, etc. The analytics modules of the AAPD 110 may include pre-trained modules (e.g., artificial intelligence (AI) modules, machine learning (ML) modules, and/or neural network modules) for performing operations like selective word (e.g., critical word) detections, pitch and frequency analysis, volume, abusive tone, screaming, screeching, shouting and exceptional acoustic events detection, LRU sound detection etc. In some embodiments, the APD 108 may also include such pre-trained modules.
The onboard cockpit and communication system 116 (which may include an alarm system) and the crew panel 112 may take inputs from analytics module(s) and provides detailed safety and security assessment of the aircraft cabin to crew and pilot. In case any safety and/or security threat exists, such device(s) may annunciate an alarm for better cognizance by pilot and crew.
The processed audio data and/or labels (e.g., markers) may be stored in on board data storage and/or a database. Also, the raw data may be stored on the on-board data storage and/or the database. In some embodiments, once the aircraft 102 touches ground the on-board data storage and/or database information may be transmitted and/or physically transferred to the PEI device 118 for further processing.
The PEI device 118 may process the audio data further for enhanced detection on events (e.g., incidents), items of interest for improving operational efficiency, training etc. In some embodiments, for this, the PEI device 118 may include at least one high performance computing (HPC) processor and at least one GPU, at least one module with pre-trained neural networks for analytics, at least one report module for generating detailed safety and security analysis reports, and at least one database for comprehensive storage of all the required information. In some embodiments, a PEI application can be hosted in a cloud computing environment.
In some embodiments, the APD 108, the AAPD 110, and/or the PEI device 118 may include a service request module, which may analyze the voice of passengers for their specific needs and requests like comfort, cabin temperature regulation needs (e.g., higher or lower temperature), light on or off, illness or not well condition, and/or noticing of any hazardous or unusual condition. This module can be further enhanced based on airliner specific requirements and audio analytics needs.
The system 100 may include a distributed network of receivers (e.g., the array 104 of audio devices 202) that may act independently or be closely calibrated to video devices installed in the aircraft cabin. These audio receivers may have unique identification and may be analyzed to know from where the audio cues are coming. The location of the audio device may be used in the analysis process and to guide the crew to the location of the event in real time.
Referring now to
In some embodiments, the APD 108 may include receive buffers 902, at least one signal processing module 904, at least one access and channel control module 906, at least one program and control module 908, at least one configuration memory module 910 for analytics module selection, at least one analytics module 912, at least one module 914, at least one avionics communication interface 916, and/or at least one on demand storage device and access control 918, some or all of which may be communicatively coupled at any given time. In some embodiments, the receive buffers 902, the at least one signal processing module 904, the at least one access and channel control module 906, the at least one program and control module 908, the at least one configuration memory module 910 for analytics module selection, the at least one analytics module 912, the at least one module 914, the at least one avionics communication interface 916, and/or the at least one on demand storage device and access control 918 may be implemented as a single device or any number of devices.
In some embodiments, each of the receive buffers 902 and/or the at least one on demand storage device and access control 918 may at least in part include the memory 404 and/or the storage 406 of
In some embodiments, each of the at least one signal processing module 904, the at least one access and channel control module 906, the at least one program and control module 908, the at least one configuration memory module 910 for analytics module selection, the at least one analytics module 912, the at least one module 914 (e.g., a threat status and analytics report alarm module), the at least one avionics communication interface 916, and/or the at least one on demand storage device and access control 918 may at least in part include the at least one processor 402 of
In some embodiments, the APD 108 may be an LRU that acts as a first point of interface for the streams of digital audio data coming from the audio sensors (e.g., microphones 204) in the aircraft 102. The APD 108 may be an edge computing device with capabilities to perform audio processing and analytics to provide the aircraft safe condition and threat assessment to the crew and pilot. The APD 108 may also include a communication module called access & channel control that coordinates the communication from audio source channels, channel selection of the audio stream based on feedback from the analytics modules 912 to select and/or deselect the audio source and provides the real time basic output, triggers and/or events and reports to the crew panel 112 and avionics in the cockpit.
In some embodiments, the APD 108 may also coordinate with the crew panel 112 for alarm trigger on the crew panel 112 to bring the cognizance of threat situation and severity to the crew. For this, the APD 108 may communicate the threat status, analytics report, and alarm trigger to the crew panel 112 and avionics system using the avionics communication interface 916.
The APD 108 may include a program and control module 108 that controls the sub-systems and their functionalities like communication interfaces, signal processing module 904, and analytics modules 912. The different analytics modules 912 may be configurable and controlled through a configuration memory module 910 and may be configurable based on customer's choice.
The signal processing module 904 may generate different types of spectrograms and help with dedicated computing resources to perform feature extraction, generate spectral features and associated properties for providing the inputs to different analytics modules 912. The signal processing module 904 may include at least one digital signal processor and at least one GPU for processing. The signal processing module 904 may also help in speech enhancements, noise reduction, and/or specific sound selection.
The APD 108 may include software configurable and selectable analytics modules 912, which may be pre-trained analytics modules for variety of services based on customer preference. The APD 108 may include classification pretrained analytics modules 912 for gender classification, unusual event/trigger detection like loud noise, bang sound, etc., languages, selective word detection and threat scenario detection.
The APD 108 may include a communication bridge and access control through which the APD 108 allows access to the AAPD 110 and passes on the processed audio data stream received from signal processing module 904 and the reports including triggers/events generated by analytics modules 912.
Referring now to
In some embodiments, the AAPD 110 may be an LRU that may be equipped with a cluster of processors 502 (e.g., CPU(s) and GPU(s) to achieve a high performance computing environment. The AAPD 110 may takes processed audio stream data and aircraft cabin operating environment status reports from the APD 108 and perform advanced analytics to provide output (e.g., aircraft cabin threat scenario, voice, selectable audio channel, speaker, gender, age, tone, temperament, language, selected threat words, unusual events sounds like thud, bang, crack, scream, screech, splash, etc).
In some embodiments, the AAPD 110 may include at least one data buffer 1002, at least one security management module 1004, at least one analytics application module 1006, at least one storage services module 1008, at least one dashboard service module 1010, at least one trigger and event marking service module 1012, at least one speech to text service module 1014, at least one print service module 1016, and/or at least one application management module 1018, some or all of which may be communicatively coupled at any given time. In some embodiments, the AAPD 110 may be communicatively coupled to the crew panel 112, a cockpit display, and/or a crew device 1022 (e.g., a mobile and/or handheld computing device).
In some embodiments, each of the at least one data buffer 1002 and/or the at least one storage services module 1008 may at least in part include the memory 504 and/or the storage 506 of
In some embodiments, each of the at least one security management module 1004, the at least one analytics application module 1006, the at least one storage services module 1008, the at least one dashboard service module 1010, the at least one trigger and event marking service module 1012, the at least one speech to text service module 1014, the at least one print service module 1016, and/or the at least one application management module 1018 may at least in part include the at least one processor 502 of
The AAPD 110 may include a data buffer 1002 to capture the audio data stream coming from the APD 108 and provide the audio data to the analytics modules of the analytics application module 1006. The data buffer 1002 may be enable with wired and/or wireless connection(s) to the APD 108. The connection establishment and security may be handled by the security management module 1004.
The analytics application module 1006 may include the pre-trained analytics modules responsible for detection and classification of various sensitive information related to safe operating environment of the cabin and to detect unusual events and triggers. These analytics modules may be capable of sharing data with each other for leveraging the data available from other modules for analytics. Examples of various suitable analytics modules of the AAPD 110 are mentioned below:
Such exemplary analytics modules along with “trigger and event marking services” module 1012 may have capabilities to label the audio data for the identified events and triggers so that the audio can be stored along with these labels for later retrievable based on instance of interest of voice in the time sequence. Once the analytics module processes the audio the analytics module may provide the labeled data to be stored with labels to the storage service—“Data and Event Storage” of the storage services module 1008.
The AAPD 110 may include the “storage service” module 1008 which may support with storage and retrieval of data required for the functioning of the analytics modules in the “training database”. Along with this the storage services module 1008 may store the audio data along with associated labels and events identification for later access and retrieval for post event analysis which can be conducted either on board the aircraft or on the ground station using “Data and Event Storage”.
The “Data and Event Storage” storage service of the storage services 1008 may provide capabilities for the complete and partial backup and download the data to a ground station through various means like wireless, wired or transfer through any physical medium for “post event investigation” by the PEI device 118.
The analytics application module 1006 and/or the “dashboard service” module 1010 may provide the cabin environment a condition assessment output in presentable format to the cabin crew, the crew panel 112, crew device 1022, and to the cockpit display 1020. The dashboard service module 1010 may generate the metadata and data packet for such presentation.
The AAPD 110 may have capabilities to print the cabin operating environment status report at different instances of time for offline analysis using the “printer service” module 1016. The AAPD 110 may also be configured for speech to text conversion using the “speech to text service” module 1014 for display and print based on inputs from the crew on the crew panel 112.
The AAPD 110 may include an analytics module to identify the threats emerging from different zones in the cabin and associate the threats to identify any pattern emerging from these different locations to co-relate and estimate potential collaborated threat situation.
The AAPD 110 may also be configured to differentiate crew member voice versus passenger voice and to identify any inappropriate behavior of crew member to be reported to airline operations and other concerned authorities, for example, if airlines may maintain the voice samples of all the crew members.
Referring again to
The PEI device 118 may be configured to replay the audio by selecting a track and event labels from the audio data retrieved from a database. For this, the PEI device 118 may provide visual representation of the labelled audio track over time series and events for replay.
The PEI device 118 may be configured to provide status and feedback by selecting the voice of crew members from audio stream and showing status like loud voice, bad temperament, neutral accent, and yelling, which may be helpful for training purposes. The PEI device 118 may generate a report card of the crew member for their behavior in the aircraft 102 based on their voice profile in the aircraft 102. The PEI device 118 may also monitor the response and auditability of the crew announcement in different locations of the aircraft 102 for different phases of flight, seat belt related, ascent, descent, emergency, and other general announcements.
The PEI device 118 may have access to an external audio database accessible for specific needs like training the modules, identification of any specific event which are not captured by current training modules in the internal database.
In some embodiments, the PEI device 118 may provide seamless wireless connectivity with the AAPD 110 and/or the APD 108.
In some embodiments, on-ground infrastructure may support real time incremental training and validation of the analytics modules of the APD 108, the AAPD 110, and/or the PEI device 118 with new events and inputs. The on-ground infrastructure may also support the deployment of such newly trained modules periodically.
In some embodiments, the PEI device 118 may provide capabilities for training and increasing the operational efficiency of the crew.
In some embodiments, the APD 108 and/or the AAPD 110 may be enabled with self-learning using edge computing capabilities to improve their accuracy of analytics and over-the-air upgrade and employment of modules.
In some embodiments, the APD 108 and/or the AAPD 110 may use a selection of specific analytical model based on a flight phase, such as filters can be changed during higher disturbances from engine sound. In some embodiments, the APD 108 and/or the AAPD 110 may use multilevel data processing and filtering and analytics based on empirical cabin profiling during development phase.
In some embodiments, an automatic system may provide advanced situation analysis and report along with an alarm (e.g., an aural alarm, such as a beep or a voice alarm) to increase the safety and security of the aircraft 102. In some embodiments, the automatic system may reduce crew workload. In some embodiments, an unusual sound coming from an aircraft system, such as landing gear, engine, fan cowl, or any other moving body of the aircraft and high vibrations helps identify any impending aircraft system failure.
In some embodiments, the system may help gauge, using audio, the situation at a location where blind spot exists for cameras and hence increases the coverage space on the aircraft.
Referring now to
A step 1102 may include obtaining or generating, by at least one post event investigation (PEI) device, processed audio data, the processed audio data having been processed from monitored channels of microphone audio data from an array of audio devices installed in an aircraft cabin of an aircraft, wherein each of the audio devices is configured to output microphone audio data on a channel, wherein each of the at least one PEI device comprises at least one processor.
A step 1104 may include obtaining or generating, by the at least PEI device, an event report associated with at least one channel of microphone audio data associated with an identified event, the identified event to being an occurrence of a predetermined event.
A step 1106 may include obtaining or generating, by the at least one PEI device, at least a portion of the processed audio data of the at least one channel of microphone audio data associated with the identified event.
A step 1108 may include by the at least one PEI device, obtaining labeled portions of the processed audio data or analyzing the processed audio data and the event report to label portions of the processed audio data as the labeled portions of the processed audio data, wherein each of the labeled portions of the processed audio data is labeled as at least one of: a threat scenario, an unusual sound captured by a given microphone, at least one selective word captured by the given microphone, a temperament of a speaker captured by the given microphone, a speaker profile, a language of the speaker captured by the given microphone, a location of the speaker captured by the given microphone, or an aircraft system failure.
A step 1110 may include recreating, by the at least one PEI device, the identified event at least by replaying, for a user, at least one of the labeled portions of the processed audio data.
Further, the method 1100 may include any of the operations disclosed throughout.
Referring generally to
The training of the classification models may be a multi-step process where sample speeches or sounds 1202 are collected. The sample speech or sound 1202 may be captured to sample speech or sound within the aircraft. Optionally, once preprocessing happens words are extracted using tokenization 1203. The sample speeches or sounds 1202 are pre-processed 1204 for digital conversion, noise elimination, filtering and transformation to make the speeches or sounds suitable for feature extraction 1206. The feature extraction 1206 may include extracting one or more features based on the desired classification model. For example, the feature extraction 1206 may include extracting the following features: spectrum frame, age, speaker count, spectrogram, spectral centroids, zero crossing rate, Mel-frequency rate, threat scenarios, unusual sounds, temperament, anomalies, and the like. The features may then be labelled in a data labelling 1208 step. The labelled data may then be provided for neural network training and validation 1210. The network training may include training a neural network. The neural network used may be an artificial neural network with multiple layers. The neural network training and validation 1210 may then be considered to form one or more trained classification model 1212.
The trained classification model 1212 may be trained to classify various real-time speech or sound 1214. For example, the real-time speech or sound 1214 may be received and the processed 1216. Feature extraction 218 may then be performed on the processed sound or speech to extract one or more features. The feature extraction 218 may be similar to the feature extraction 1206. The extracted features may then be provided to the trained classification model 1212 in a step 1220. The trained classification model 1212 may then provide a classification 1222 classifying the one or more features. The features may be classified as one or more of gender, age (e.g., a value or in a range), speaker count, language, location, coordinates, row, seat number, selective words, loudness (e.g., loud, scream, normal), threat scenarios, threat words, events, types of sound, temperament (e.g., aggressive, calm, dangerous, enraged, etc.), a component identifier, and/or a condition of the component (e.g., normal, anomalous, etc.).
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Once the anomalous sound detection model 1212m is trained, the anomalous sound detection model 1212m may be used in real time for LRUs/Systems/Components identification and different working condition of normal and malfunctioning with different scenarios of failure conditions. The real time sound may be preprocessed and utilized for features extraction in terms of spectrogram. The spectrogram may be convoluted and applied to the trained anomalous sound detection model 1212m. The anomalous sound detection model 1212m may classifies the LRUs/Systems/Components from where sound is coming. The anomalous sound detection model 1212m may also provide the working condition of the LRUs/Systems/Components in terms of normal functioning or malfunctioning with identification of failure conditions along with a confidence score. The different anomalous sounds detected may include: Engine Rattling, Engine Cranking, Loose Fan cowl, Damaged Engine Blades, Engine Exhaust, Damaged Tires, Damaged Wheels and Brakes, Potable Water System Compressor Malfunction, Environmental Control System Inflow and Outflow Valve Function, Environment Control System Compressors Malfunction, Loose connections, Loose Rivets, Damaged Surface, Structural Damages, Windshield Cracking, Landing Gear Deployment and Retraction, Actuators movement, Electrical Motor, Gear Failures, and the like. Thus, at least one audio processing device may be configured to analyze the processed audio data and the event report to label portions of the processed audio data as an aircraft system failure.
Referring generally again to
As used throughout and as would be appreciated by those skilled in the art, “at least one non-transitory computer-readable medium” may refer to as at least one non-transitory computer-readable medium (e.g., at least one computer-readable medium implemented as hardware; e.g., at least one non-transitory processor-readable medium, at least one memory (e.g., at least one nonvolatile memory, at least one volatile memory, or a combination thereof; e.g., at least one random-access memory, at least one flash memory, at least one read-only memory (ROM) (e.g., at least one electrically erasable programmable read-only memory (EEPROM)), at least one on-processor memory (e.g., at least one on-processor cache, at least one on-processor buffer, at least one on-processor flash memory, at least one on-processor EEPROM, or a combination thereof), or a combination thereof), at least one storage device (e.g., at least one hard-disk drive, at least one tape drive, at least one solid-state drive, at least one flash drive, at least one readable and/or writable disk of at least one optical drive configured to read from and/or write to the at least one readable and/or writable disk, or a combination thereof), or a combination thereof).
As used throughout, “at least one” means one or a plurality of; for example, “at least one” may comprise one, two, three, . . . , one hundred, or more. Similarly, as used throughout, “one or more” means one or a plurality of; for example, “one or more” may comprise one, two, three, . . . , one hundred, or more. Further, as used throughout, “zero or more” means zero, one, or a plurality of; for example, “zero or more” may comprise zero, one, two, three, . . . , one hundred, or more.
In the present disclosure, the methods, operations, and/or functionality disclosed may be implemented as sets of instructions or software readable by a device. Further, it is understood that the specific order or hierarchy of steps in the methods, operations, and/or functionality disclosed are examples of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the methods, operations, and/or functionality can be rearranged while remaining within the scope of the inventive concepts disclosed herein. The accompanying claims may present elements of the various steps in a sample order, and are not necessarily meant to be limited to the specific order or hierarchy presented.
It is to be understood that embodiments of the methods according to the inventive concepts disclosed herein may include one or more of the steps described herein. Further, such steps may be carried out in any desired order and two or more of the steps may be carried out simultaneously with one another. Two or more of the steps disclosed herein may be combined in a single step, and in some embodiments, one or more of the steps may be carried out as two or more sub-steps. Further, other steps or sub-steps may be carried in addition to, or as substitutes to one or more of the steps disclosed herein.
From the above description, it is clear that the inventive concepts disclosed herein are well adapted to carry out the objects and to attain the advantages mentioned herein as well as those inherent in the inventive concepts disclosed herein. While presently preferred embodiments of the inventive concepts disclosed herein have been described for purposes of this disclosure, it will be understood that numerous changes may be made which will readily suggest themselves to those skilled in the art and which are accomplished within the broad scope and coverage of the inventive concepts disclosed and claimed herein.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202241029171 | May 2022 | IN | national |
