Embodiments of the invention relate generally to cardiac auscultation, particularly in the case of systems and methods for automatically classifying detected heart sounds on an embedded diagnostic device.
Cardiac auscultation is performed in virtually every clinical examination and can identify heart disease before the symptoms are present and apparent to the patient. However, in practice, the majority of heart murmurs are missed, due to their subtlety or other conditions during the examination. Even significant valvular heart disease is identified by auscultation in less than 43% of actual cases.
Devices thus far require a connection to a mobile app and/or the cloud to complete the intensive processing involved in running a classification algorithm. UI/UX and visual feedback in these cases is provided on the mobile app, after processing is completed. These devices generally require connection to a mobile app or computer, a high-speed internet connection, and the use of a smartphone. This significantly limits the conditions under which they can be used. Additionally, these devices typically operate at a delay, record data, and transmit this data to another device or devices for analysis prior to providing a result.
The present invention's electronic device can be attached to a standard or regular stethoscope and can be used to automatically flag or provide an indication to the physician or medical personnel that heart murmurs have been detected in the patient in real-time, during routine examinations. No other devices provide heart murmur detection in real-time, on a standalone device. This device performs the processing required to run one or more detection algorithms on-board in real-time. For this reason, the device does not require a connection to external computing resources. The device can signal a user directly once a prediction or analysis is reached. This allows the device's use to be more readily integrated into standard auscultation exams, improves ease of use, and efficiency.
Thus, the need exists for real-time detection of internal medical issues using embedded electronic devices which can be coupled with standard stethoscopes and/or other medical equipment.
This summary is provided to introduce a variety of concepts in a simplified form that is disclosed further in the detailed description of the embodiments. This summary is not intended for determining or limiting the scope of the claimed subject matter.
The example embodiments provided herein relate to and disclose techniques for artificial intelligence detection of abnormal heart sounds classified as “heart murmurs” (HM). The detection of HMs in the medical field is difficult and leveraging technology to assist in detecting said HMs in the patient population when physicians listen to a patient's heart sounds in real time (also referred to as “auscultation” for this application).
In some embodiments described herein, systems, methods, and devices that automatically detect heart murmurs during auscultation includes an apparatus attached to a stethoscope during use, circuitry configured to determine if a heart murmur is present, and a sensory method such as lights or audio feedback to communicate to the user if a heart murmur is present.
In some embodiments described herein, systems, methods and devices for automatic detection of an internal body signal of interest in a stream of diagnostic data using a trained classifier deployed on an embedded electronic device. These systems, methods, and devices can include classification data that classifies the internal body signal, or heart sound as an internal body signal of interest, or normal heart sound or a certain type of heart murmur, based on a preestablished training set of selected types of heart murmurs, and analyzing the heart sounds in real time by sending into an input layer of a neural network. The neural network and associated weights are trained on this preestablished training set of select types of heart murmurs prior to implementation in the analyzation method during real time analysis during auscultation.
Other objects and advantages of the various embodiments of the present invention will become obvious to the reader and it is intended that these objects and advantages are within the scope of the present invention. To the accomplishment of the above and related objects, this invention may be embodied in the form illustrated in the accompanying drawings, attention being called to the fact, however, that the drawings are illustrative only, and that changes may be made in the specific construction illustrated and described within the scope of this application.
A more complete understanding of the embodiments, and the attendant advantages and features thereof, will be more readily understood by references to the following detailed description when considered in conjunction with the accompanying drawings wherein:
The specific details of the single embodiment or variety of embodiments described herein are set forth in this application. Any specific details of the embodiments described herein are used for demonstration purposes only, and no unnecessary limitation(s) or inference(s) are to be understood or imputed therefrom.
Before describing in detail exemplary embodiments, it is noted that the embodiments reside primarily in combinations of components related to particular devices and systems. Accordingly, the device components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
The present embodiments include embedded electronic devices designed to connect or couple in-line to a stethoscope so that it can record audio signals being transmitted in the airway of the stethoscope tubing. An example device can comprise: one or more processors; non-transitory computer readable memory, which stores a compressed deep neural network and instructions for the microprocessor; at least one audio receiver or transceiver, such as a microphone, operable to receive and/or record the sound transmitted through stethoscope tubing; a means of sensory output to communicate the results to the physician; and an operable coupled battery to power the device.
The present embodiments also include systems, methods, and devices for automatic detection of internal body signal of interest (e.g. a heart sound, lung sound, digestive sound, electrical sound, or other sound), in a stream of audio data, recorded by the device when connected to a stethoscope, using a trained classifier deployed on an embedded electronic device. An example embodiment method can comprise:
A) Training by an audio recognition system that includes at least one computer, a deep neural network to determine probabilities that data received by the deep neural network has features similar to key features of heart sounds of interest, the training comprising: providing the deep neural network with a first set of feature values for heart sound data; adjusting values for each of a plurality of weights included in the neural network; and compressing the plurality of weights and optimizing based on a balance of performance and size for deployment onto resource-constrained embedded systems.
B) Deploying the deep neural network, which was previously trained and compressed, on an embedded electronic device.
C) Acquiring on the embedded electronic device, streaming data detected by a sensor (e.g. audio from a microphone on the device, electrical signals from an EKG sensor, or others), and providing said data stream to the on-device deep neural network.
D) Using the trained deep neural network to determine a probability that data received by the deep neural network has features similar to key features of a body signal of interest (e.g. one or more heart sounds, lung sounds, digestive sounds, electrical signals, or others), the deep neural network to detect only those of the one or more body signals of interest encoded in a stream of audio data. These features can be time-frame data, frequency, amplitude, irregularities, or others.
E) Sending a notification of the detection of an internal body signal of interest to an output device when the probability of detection exceeds a pre-established threshold value or another trigger occurs.
In an example embodiment, sensor 103, which can be a microphone or other transducer or signal detecting/receiving component(s) in various embodiments, detects, records and/or senses sounds in, from, or through a stethoscope or phonendoscope. These sounds are then transduced into electrical signals and provided as data to communicatively coupled processor 101. Processor 101 can in turn run one or more processes that are stored in non-transitory memory 102 that may compare the electrical signals against, with, or through the deep neural network 108 to determine whether they match or indicate a particular type of internal body process or condition of the individual being monitored. In some embodiments thresholds can be used, as well as markers, quantities of matching indicators, or others to determine whether the signal matches known body processes.
Conditions can be classified in some instances. For example, a heart sound may be classifiable as one or more types of heart murmurs, which can be associated with a particular diagnosis. More generally heart murmurs can be considered abnormal heart sounds. Classified murmurs may further be classifiable by their intensity into different grades.
To elaborate, the computer readable memory 102 stores the trained, compressed neural network 108, and the instructions for running the microprocessor 101. The processor 101 prepares the streaming audio data for the trained, compressed deep neural network 108 by performing feature extraction and creating a feature vector which creates one or more virtual models of the audio data. The feature vector can be provided to the deep neural network 108, which in turn provides a prediction regarding the occurrence of the heart sounds of interest in the streaming data. See
A user input 109 can include one or more buttons or screens in various embodiments that allow a user to interact with the device. Functions can include power on/off, standby, activate, deactivate, acknowledge signal, change mode, reset, or others, in various embodiments. See
In various embodiments, the device may not require any outside data or direct power connection to function. Battery 105 can be charged prior to dissemination to a physician and may or may not be rechargeable through wireless or wired connection in various embodiments. In some embodiments, one or more network interface(s) can be included such that the embedded electronic device can receive software updates and to extract or send stored information via a network connection to a receiving device such as a computer, mobile device, server, or other device.
The embedded electronic device is able to perform its classification activities/processes on received data signals using the pre-trained (used interchangeable with trained, herein) deep neural network (DNN) 108. In many embodiments, the DNN 108 has been compressed (quantized) for running on resource constrained platforms. Incoming monitored or sensed audio data from sensor 103 is checked using the deep neural network 108 can also be compressed in various embodiments. In various embodiments, the data stream can be compressed to a standard format (e.g. .wav, .mp3, .mp4, or many others, known or later developed) after being processed to save for later review on a separate device (e.g. a computer, smartphone, tablet computer, or other computing device). The DNN 108 can be designed and optimized specifically for this resource constrained application on the embedded electronic device in some embodiments. Moving the DNN 108 from the cloud, where it would typically be stored, and directly onto the device can be important to functionality of this device, as it allows the feedback, in the form of output 104, to be provided in real-time or near real-time to the physician or other user, and it eliminates the standard use requirement of a peripheral devices, such as a smartphones.
The deep neural network 108 needs to be trained specifically for this resource constrained application and compressed for deployment onto the device. The DNN is trained using a database of heart sounds prior to deployment on the device. The DNN can be used to determine probabilities that data received by the DNN has features similar to key features of heart or other internal body sounds of interest. Training of the DNN can be performed by at least one computer, following the steps of providing the DNN with a first set of feature values for heart (or other) sound data, adjusting values for one or more weights included in the neural network, and compressing the plurality of weights and optimizing based on balance of performance and size for deployment onto resource constrained embedded systems.
There are two approaches to training, supervised and unsupervised. In some embodiments, supervised training can be used for embedded electronic device DNN training. In other embodiments, unsupervised training can be used for embedded electronic device DNN training.
In supervised training embodiments, both inputs and the outputs can be provided. The DNN then processes the inputs and compares resulting outputs against the desired outputs. Errors can then be propagated back through the system, causing the system to adjust the weights which control the network. This process can iteratively occur numerous times over and over as the weights are continually tweaked. The set of data which enables the training is called a “training set.” During the training of a network the same set of data is processed many times as the connection weights can be ever refined.
In unsupervised training, the DNN can be provided with inputs but not with desired outputs. The DNN itself can then decide what features it will use to group the input data. This is often referred to as self-organization or adaption. See
Critically in our application we implemented learning under resource constraints. This departs from traditional machine learning in that model features are accompanied by costs (e.g. memory required, processing time, etc.). This is what allows us to deploy our trained model on small, embedded platforms.
In some embodiments, a network interface 118 can be provided that allows the embedded device to receive and/or send data via network 112 to and/or from one or more devices 114. Devices can include smartphones, tablets, desktop and/or laptop computers, servers, proprietary computing devices, wearable devices such as smartwatches and smart glasses and others in various embodiments. In some embodiments, one or more databases 116 can be stored in non-transitory memory on device 114. Memory and/or databases 116 of device 114 can also store DNN and other information such as patient information, measurements, medical data, algorithms, and others, as appropriate and necessary.
In some embodiments, stethoscope tubing can or may be punctured, whereafter positioning a microphone in, at, or otherwise adjacent or near the opening can allow it to adequately detect audio signals in the tubing. Potting with silicone or similar sealant can be used to create an airtight seal between the punctured tube and the housing around or near the microphone(s). See
In some embodiments, an airtight seal may not be required for adequate signal detection by a microphone (e.g. for audio signals) or other sensor (e.g. electrical signals for EKG detection). In such embodiments no puncture, hole, or other access to the interior of a stethoscope tube may be required. Clasp(s), clamp(s), and/or other coupling mechanisms can be used in such embodiments. In some embodiments, more than one type of internal signals can be monitored and analyzed by the embedded electronic device (e.g. heart signals, electrical signals, lung signals, and/or others).
In some embodiments digital stethoscopes can be used with embedded electronic devices. In such embodiments, the digital stethoscope may capture data on its own, which can be communicatively coupled with an embedded electronic device in order to employ DNN(s) to achieve the outcomes outlined herein.
One or more housing, which can be plastic in some embodiments, can include an upper housing 202a and a lower housing 202b. These upper and lower housings 202a, 202b can be permanently coupled together in some embodiments or removably coupled using any manner of detents, buttons, latches, seals, glues, resins, epoxies, screws and receiving holes, or other appropriate coupling mechanisms. Upper and lower housings 202a, 202b can be contain the PCBA(s) 206 and tubing section 207. When coupled, housings 202a, 202b can provide at least one hole 211 that tubing section 207 passes through, but which is flush with the exterior surface of tubing section 207. As such, the components on the interior of the housing can be protected from moisture, dirt, dust, or other corrosive or damaging elements.
As shown, at least one battery 205 can be included and housed within housing 202a, 202b of an assembled device. Battery 205 can be charged through induction in some embodiments, while in other embodiments a plug or hole can be provided to allow for removably coupling a wire to charge the battery, as is known in the art. Battery 205 can be coupled directly to PCBA 206 to provide power.
An indicator 213 can include one or more visual, audio, mechanical, or other mechanisms to alert a user and/or indicate to a user a particular operating status or state of the device is currently in (e.g. on, activated (processing data), condition identified, incorrect use (e.g. not at appropriate site, moved during use, or others), unknown condition (please retry), low power, charging, prediction confidence level, software updating, audio recording, standby, monitoring, resetting device, paired with other device (e.g. via Bluetooth or other wireless connection), device error state, second body signal detected (e.g. heart murmur, lung sound, arrhythmia, or others) or others). As such, the output provided to the user could take different forms, lights, audio, tactile feedback. In the example embodiment, PCBA 206 can have one or a plurality of LED indicator light(s) 213 included, that are able to shine through holes, a membrane, or clear or opaque section or surface of upper housing 202a to indicate a condition to the user. The resulting output can also be communicated to a separate device which provides an indication in some embodiments (e.g. wirelessly transmitted signal to a related and communicatively coupled device such as a speaker, mechanical indicator, and/or audible indicator.
One or more user input mechanisms 209 can be included in various embodiments. As shown, input mechanism 209 can be a button can be a separate from an upper housing 202a, or could be integrated in some embodiments. When actuated or engaged, the button can cause the processor of the PCBA 206 to perform and/or cease a function. Input mechanism 209 could also be a touchscreen display or other mechanism as appropriate to allow a user to interact with and control the device.
In various embodiments the device is able to recognize a number of sounds and/or types of sounds and is not limited to heart sounds. These can include lung sounds, digestive tract sounds, or others, as appropriate.
In various embodiments, diagnostic data being provided to the device is not limited to that which could be picked up with a microphone. As such, electrical signals produced by internal organs, such as those picked up by electrocardiograms, can be detected.
In an example embodiment, one or more usage steps after assembly can include: 1. Physician positions the stethoscope diaphragm at a first auscultation site. 2. Physician presses an activation button 209 on the device to start processing. 3. Physician listens to the heart sounds while the device processes data. This step is expected to take a particular amount of time (e.g. about milliseconds, fractions of a second, multiple seconds, five seconds, or other orders of magnitude may be employed in various embodiments) or range of time, during which the physician keeps the diaphragm pressed to the auscultation site. 4. The device signals (e.g. visually by flashing or shining an LED light) one of three possible results. a. Heart Murmur detected. b. Heart murmur not detected. or c. Unknown results, please try again. 5. Physician can then move the stethoscope diaphragm to a next auscultation site and repeats steps 1-4, if additional measurements are desired.
In some embodiments, additional features can be included. In some embodiments, one or more additional microphones can be included that are outward facing and signals captured thereby be used by the processor to perform noise-cancelling operations on the stethoscope audio recording input data stream to provide more accurate overall results. In some embodiments, a multi-tiered neural network approach can be implemented. In such instances, a first deep neural network can identify a snippet of captured data of interest and a second (or multi-leveled operating) deep neural network can function as a classifier or other mechanism.
In a multi-tiered deep neural network a first network can be used to segment streaming audio, so as to identify or pull a relevant snippet of the streaming data out so that it can be run through or otherwise used by a second neural network to obtain prediction(s) about conditions which may indicated by signals or data present in that snippet.
To elaborate, in some embodiments: An audio heart sound can include of a first heart sound, S1, followed by a second heart sound, S2. A time existing between these sounds, and in between successive groupings of sounds can be related to heart rate and may vary by patient. A first stage of the deep neural network can be used to recognize S1 and S2 sounds, so as to decompose or otherwise break down the streaming audio signal into meaningful cardiac events. These identified events can then be provided to a second stage of the deep neural network for recognition of conditions (i.e. heart murmurs or other conditions). Providing cleanly isolated cardiac events can improve the accuracy of the system in some instances.
In some embodiments, a chamber 808 between the exit side of two barbed connectors 804 is provided. The top of the chamber 808 can including an opening for a microphone 810 of or coupled with PCBA 802.
A rigid structure 816 within housing 812 designed with an internal tube structure 814. This tube structure or chamber 814 can be integrated in a monolithic structure with or otherwise be coupled to barbed connectors 804 on either end to connect to the stethoscope tubing. The PCBA 802 can be mounted within housing 812 to the top of this rigid structure 816, and the structure 816 can have an opening or access to the interior or into the tube section 814 which aligns with the position of the microphone 810 on the PCBA 802. Securing the PCBA 802 to the top of this rigid structure can create a seal, for example an airtight seal, in some embodiments.
In some embodiments at least one noise cancelling microphone can be included that has access to and can receive external sound signals that can be processed and used to improve the accuracy of results and predictions by countering and/or removing background noise from an audio sample.
In some embodiments, combinations of multiple sensors (e.g. multiple microphones, one microphone and one EKG monitoring sensor, or others) can be included in a single electronic embedded device. These may be in line or at strategic locations and can be used to increase the accuracy of results.
Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety to the extent allowed by applicable law and regulations. The systems and methods described herein may be embodied in other specific forms without departing from the spirit or essential attributes thereof, and it is therefore desired that the present embodiment be considered in all respects as illustrative and not restrictive. Any headings utilized within the description are for convenience only and have no legal or limiting effect.
Many different embodiments have been disclosed herein, in connection with the above description and the drawings. It will be understood that it would be unduly repetitious and obfuscating to literally describe and illustrate every combination and subcombination of these embodiments. Accordingly, all embodiments can be combined in any way and/or combination, and the present specification, including the drawings, shall be construed to constitute a complete written description of all combinations and subcombinations of the embodiments described herein, and of the manner and process of making and using them, and shall support claims to any such combination or subcombination.
The foregoing is provided for purposes of illustrating, explaining, and describing embodiments of this disclosure. Modifications and adaptations to these embodiments will be apparent to those skilled in the art and may be made without departing from the scope or spirit of this disclosure.
As used herein and in the appended claims, the singular forms “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise.
It should be noted that all features, elements, components, functions, and steps described with respect to any embodiment provided herein are intended to be freely combinable and substitutable with those from any other embodiment. If a certain feature, element, component, function, or step is described with respect to only one embodiment, then it should be understood that that feature, element, component, function, or step can be used with every other embodiment described herein unless explicitly stated otherwise. This paragraph therefore serves as antecedent basis and written support for the introduction of claims, at any time, that combine features, elements, components, functions, and steps from different embodiments, or that substitute features, elements, components, functions, and steps from one embodiment with those of another, even if the description does not explicitly state, in a particular instance, that such combinations or substitutions are possible. It is explicitly acknowledged that express recitation of every possible combination and substitution is overly burdensome, especially given that the permissibility of each and every such combination and substitution will be readily recognized by those of ordinary skill in the art.
In many instances entities are described herein as being coupled to other entities. It should be understood that the terms “coupled” and “connected” (or any of their forms) are used interchangeably herein and, in both cases, are generic to the direct coupling of two entities (without any non-negligible (e.g., parasitic) intervening entities) and the indirect coupling of two entities (with one or more non-negligible intervening entities). Where entities are shown as being directly coupled together, or described as coupled together without description of any intervening entity, it should be understood that those entities can be indirectly coupled together as well unless the context clearly dictates otherwise.
While the embodiments are susceptible to various modifications and alternative forms, specific examples thereof have been shown in the drawings and are herein described in detail. It should be understood, however, that these embodiments are not to be limited to the particular form disclosed, but to the contrary, these embodiments are to cover all modifications, equivalents, and alternatives falling within the spirit of the disclosure. Furthermore, any features, functions, steps, or elements of the embodiments may be recited in or added to the claims, as well as negative limitations that define the inventive scope of the claims by features, functions, steps, or elements that are not within that scope.
An equivalent substitution of two or more elements can be made for any one of the elements in the claims below or that a single element can be substituted for two or more elements in a claim. Although elements can be described above as acting in certain combinations and even initially claimed as such, it is to be expressly understood that one or more elements from a claimed combination can in some cases be excised from the combination and that the claimed combination can be directed to a subcombination or variation of a subcombination.
It will be appreciated by persons skilled in the art that the present embodiment is not limited to what has been particularly shown and described herein. A variety of modifications and variations are possible in light of the above teachings without departing from the following claims.
The present application claims priority to U.S. Provisional Application No. 63/165,018 filed Mar. 23, 2021, titled “AUTOMATIC CLASSIFICATION OF HEART SOUNDS ON AN EMBEDDED DIAGNOSTIC DEVICE,” which is hereby incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
63165018 | Mar 2021 | US |