Patent Grant
10863296
With the advancement of technology, the use and popularity of electronic devices has increased considerably. Electronic devices are commonly used to capture and process audio data.
For a more complete understanding of the present disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.
Electronic devices may be used to capture and process audio data. The audio data may be used for voice commands and/or as part of a communication session, and may be generated using a single microphone or multiple microphones. In some examples, a device may capture input audio data using a single microphone in a microphone array, while in other examples the device may capture input audio data using multiple microphones in the microphone array and perform beamforming to generate directional audio data corresponding to a plurality of directions.
Performance of the device is negatively impacted when a microphone fails and is unusable by the device. If the device selects a single microphone to generate the audio data, microphone failure results in a negative user experience as the device is unable to generate audio data using the selected microphone. If the device generates the audio data using multiple microphones, microphone failure results in decreased performance as the nonfunctional microphone is relied upon during audio processing, such as beamforming, adaptive beamforming, acoustic echo cancellation, and/or the like.
To improve audio processing despite microphone failure, devices, systems and methods are disclosed that detect microphone failure and re-optimize settings using the remaining functional microphones. The system may detect a nonfunctional (e.g., dead) or malfunctioning (e.g., attenuated) microphone when an energy level for the microphone is lower than a threshold value for a period of time. For example, a first energy level of the microphone may be lower than an average energy level of functional microphones by more than a threshold (e.g., 6 dB), although the disclosure is not limited thereto. After detecting a defective microphone, the system may update configuration settings by reassigning microphones (e.g., selecting a functional microphone instead of the defective microphone) and/or by calculating new configuration settings without the defective microphone. For example, the system may recalculate beamformer coefficients using the remaining microphones, which may improve a performance of the device (e.g., wakeword detection, automatic speech recognition (ASR), etc.) and result in only a marginal performance degradation relative to fully-functioning microphones.
As illustrated in
In some examples, the remote system 120 may be configured to process voice commands (e.g., voice inputs) received from the device 110. For example, the device 110 may capture input audio 11 corresponding to a voice command from the user 5 (e.g., an utterance), may generate input audio data representing the audio 11 using the one or more microphone(s) 112, and may send the input audio data to the remote system 120 for speech processing. However, the disclosure is not limited thereto and in other examples the device 110 may send the input audio data to a remote device (not illustrated) as part of a communication session. For example, the device 110 may capture input audio 11 corresponding to the communication session, may generate input audio data representing the audio 11 using the one or more microphone(s) 112, and may send the input audio data to the remote device.
Performance of the device 110 is negatively impacted when a microphone 112 fails and is unusable by the device 110. For example, if the device 110 selects a single microphone 112 to generate input audio data, microphone failure results in a negative user experience as the device 110 is unable to generate the input audio data using the selected microphone 112 (e.g., the input audio data does not represent the audio 11). Additionally or alternatively, if the device 110 generates the input audio data using multiple microphones 112, microphone failure results in decreased performance as the nonfunctional microphone is relied upon during audio processing, such as beamforming, adaptive beamforming, acoustic echo cancellation, and/or the like.
To improve audio processing despite microphone failure, the system 100 is configured to detect microphone failure and re-optimize configuration settings using the remaining functional microphones 112. As will be described in greater detail below, the system may detect a defective (e.g., nonfunctional or malfunctioning) microphone when an energy level for the defective microphone is outside of a desired range (e.g., lower than a threshold value) for a period of time (e.g., time range). For example, a first energy level of the defective microphone may be lower than an average energy level of the functional microphones by more than a threshold (e.g., 6 dB), although the disclosure is not limited thereto. After detecting the defective microphone, the system 100 may update configuration settings by reassigning microphones (e.g., selecting a functional microphone instead of the defective microphone) and/or by calculating new configuration settings accordingly (e.g., excluding a nonfunctional microphone or compensating for an attenuation or phase shift of a malfunctioning microphone). For example, the system 100 may recalculate beamformer coefficients using the remaining microphones, which may improve a performance of the device (e.g., wakeword detection, automatic speech recognition (ASR), etc.) and result in only a marginal performance degradation relative to fully-functioning microphones.
As used herein, a defective microphone may be nonfunctional (e.g., does not capture audio and/or generates audio data with energy levels below a minimum threshold value) or malfunctioning (e.g., generates audio data with energy levels above the minimum threshold value but outside of a desired range, such as attenuated relative to a second threshold value corresponding to normal operation), whereas a functional microphone (e.g., microphone that generates audio data with energy levels within the desired range, such as above the second threshold value) may be referred to as a functional microphone, remaining microphone(s), and/or the like. In some examples, the system 100 may treat both nonfunctional microphones and malfunctioning microphones similarly and update settings to ignore both. However, the disclosure is not limited thereto and in other examples, the system 100 may update settings to ignore the nonfunctional microphones and to compensate for an attenuation and/or phase shift associated with the malfunctioning microphones without departing from the disclosure.
As illustrated in
The device 110 may determine (136) that a first energy level does not satisfy a condition (e.g., outside of a desired range, such as below a threshold value, whether an absolute threshold value or a relative threshold value), may determine (138) that a first microphone corresponding to the first energy level is defective, and may generate (140) output data 111 indicating that the first microphone is defective. In some examples, the device 110 may optionally send (142) the output data 111 to the remote system 120 and receive (144) first data 121 from the remote system 120, although the disclosure is not limited thereto. The device 110 may then update (146) settings based on the first microphone being defective. For example, the device 110 may perform microphone reassignment to deselect the first microphone and/or select a functional microphone, may calculate updated beamformer coefficients without the first microphone, and/or the like, as will be described in greater detail below.
For ease of illustration, some audio data may be referred to as a signal, such as a far-end reference signal x(t), a microphone signal z(t) (e.g., input signal), error signal m(t) or the like. However, the signals may be comprised of audio data and may be referred to as audio data (e.g., far-end reference audio data x(t), microphone audio data z(t), error audio data m(t)) without departing from the disclosure.
During a communication session, the device 110 may receive a far-end reference signal x(t) (e.g., playback audio data) from a remote device/remote server(s) (not illustrated) via the network(s) 199 and may generate output audio (e.g., playback audio) based on the far-end reference signal x(t) using the one or more loudspeaker(s) 114. Using one or more microphone(s) 112 in the microphone array, the device 110 may capture input audio as microphone signal z(t) (e.g., near-end reference audio data, input audio data, microphone audio data, etc.) and may send the microphone signal z(t) to the remote device/remote server(s) and/or the remote system 120 via the network(s) 199.
In some examples, the device 110 may send the microphone signal z(t) to the remote device as part of a Voice over Internet Protocol (VoIP) communication session. For example, the device 110 may send the microphone signal z(t) to the remote device either directly or via remote server(s) and may receive the far-end reference signal x(t) from the remote device either directly or via the remote server(s). However, the disclosure is not limited thereto and in some examples, the device 110 may send the microphone signal z(t) to the remote system 120 in order for the remote system 120 to determine a voice command. For example, during a communication session the device 110 may receive the far-end reference signal x(t) from the remote device and may generate the output audio based on the far-end reference signal x(t). However, the microphone signal z(t) may be separate from the communication session and may include a voice command directed to the remote system 120. Therefore, the device 110 may send the microphone signal z(t) to the remote system 120 and the remote system 120 may determine a voice command represented in the microphone signal z(t) and may perform an action corresponding to the voice command (e.g., execute a command, send an instruction to the device 110 and/or other devices to execute the command, etc.). In some examples, to determine the voice command the remote system 120 may perform Automatic Speech Recognition (ASR) processing, Natural Language Understanding (NLU) processing and/or command processing. The voice commands may control the device 110, audio devices (e.g., play music over loudspeaker(s) 114, capture audio using microphone(s) 112, or the like), multimedia devices (e.g., play videos using a display, such as a television, computer, tablet or the like), smart home devices (e.g., change temperature controls, turn on/off lights, lock/unlock doors, etc.) or the like.
The device 110 may operate using microphone(s) 112 comprising multiple microphones, where beamforming techniques may be used to isolate desired audio including speech. In audio systems, beamforming refers to techniques that are used to isolate audio from a particular direction in a multi-directional audio capture system. Beamforming may be particularly useful when filtering out noise from non-desired directions. Beamforming may be used for various tasks, including isolating voice commands to be executed by a speech-processing system.
One technique for beamforming involves boosting audio received from a desired direction while dampening audio received from a non-desired direction. In one example of a beamformer system, a fixed beamformer unit employs a filter-and-sum structure to boost an audio signal that originates from the desired direction (sometimes referred to as the look-direction) while largely attenuating audio signals that original from other directions. A fixed beamformer unit may effectively eliminate certain diffuse noise (e.g., undesireable audio), which is detectable in similar energies from various directions, but may be less effective in eliminating noise emanating from a single source in a particular non-desired direction. The beamformer unit may also incorporate an adaptive beamformer unit/noise canceller that can adaptively cancel noise from different directions depending on audio conditions.
In audio systems, acoustic echo cancellation (AEC) processing refers to techniques that are used to recognize when a device has recaptured sound via microphone(s) after some delay that the device previously output via loudspeaker(s). The device may perform AEC processing by subtracting a delayed version of the original audio signal (e.g., far-end reference signal x(t)) from the captured audio (e.g., microphone signal z(t)), producing a version of the captured audio that ideally eliminates the “echo” of the original audio signal, leaving only new audio information. For example, if someone were singing karaoke into a microphone while prerecorded music is output by a loudspeaker, AEC processing can be used to remove any of the recorded music from the audio captured by the microphone, allowing the singer's voice to be amplified and output without also reproducing a delayed “echo” of the original music. As another example, a media player that accepts voice commands via a microphone can use AEC processing to remove reproduced sounds corresponding to output media that are captured by the microphone, making it easier to process input voice commands.
As an alternative to generating the reference signal based on the playback audio data, Adaptive Reference Algorithm (ARA) processing may generate an adaptive reference signal based on the input audio data. To illustrate an example, the ARA processing may perform beamforming using the input audio data to generate a plurality of audio signals (e.g., beamformed audio data) corresponding to particular directions. For example, the plurality of audio signals may include a first audio signal corresponding to a first direction, a second audio signal corresponding to a second direction, a third audio signal corresponding to a third direction, and so on. The ARA processing may select the first audio signal as a target signal (e.g., the first audio signal includes a representation of speech) and the second audio signal as a reference signal (e.g., the second audio signal includes a representation of the echo and/or other acoustic noise) and may perform Adaptive Interference Cancellation (AIC) (e.g., adaptive acoustic interference cancellation) by removing the reference signal from the target signal. As the input audio data is not limited to the echo signal, the ARA processing may remove other acoustic noise represented in the input audio data in addition to removing the echo. Therefore, the ARA processing may be referred to as performing AIC, adaptive noise cancellation (ANC), AEC, and/or the like without departing from the disclosure.
As discussed in greater detail below, the device 110 may be configured to perform AIC using the ARA processing to isolate the speech in the input audio data. The device 110 may dynamically select target signal(s) and/or reference signal(s). Thus, the target signal(s) and/or the reference signal(s) may be continually changing over time based on speech, acoustic noise(s), ambient noise(s), and/or the like in an environment around the device 110. In some examples, the device 110 may select the target signal(s) based on signal quality metrics (e.g., signal-to-interference ratio (SIR) values, signal-to-noise ratio (SNR) values, average power values, etc.) differently based on current system conditions. For example, the device 110 may select target signal(s) having highest signal quality metrics during near-end single-talk conditions (e.g., to increase an amount of energy included in the target signal(s)), but select the target signal(s) having lowest signal quality metrics during far-end single-talk conditions (e.g., to decrease an amount of energy included in the target signal(s)).
Additionally or alternatively, the device 110 may select the target signal(s) by detecting speech, based on signal strength values or signal quality metrics (e.g., signal-to-noise ratio (SNR) values, average power values, etc.), and/or using other techniques or inputs, although the disclosure is not limited thereto. As an example of other techniques or inputs, the device 110 may capture video data corresponding to the input audio data, analyze the video data using computer vision processing (e.g., facial recognition, object recognition, or the like) to determine that a user is associated with a first direction, and select the target signal(s) by selecting the first audio signal corresponding to the first direction. Similarly, the adaptive beamformer may identify the reference signal(s) based on the signal strength values and/or using other inputs without departing from the disclosure. Thus, the target signal(s) and/or the reference signal(s) selected by the adaptive beamformer may vary, resulting in different filter coefficient values over time.
As discussed above, the device 110 may perform beamforming (e.g., perform a beamforming operation to generate beamformed audio data corresponding to individual directions). As used herein, beamforming (e.g., performing a beamforming operation) corresponds to generating a plurality of directional audio signals (e.g., beamformed audio data) corresponding to individual directions relative to the microphone array. For example, the beamforming operation may individually filter input audio signals generated by multiple microphones 112 in the microphone array (e.g., first audio data associated with a first microphone, second audio data associated with a second microphone, etc.) in order to separate audio data associated with different directions. Thus, first beamformed audio data corresponds to audio data associated with a first direction, second beamformed audio data corresponds to audio data associated with a second direction, and so on. In some examples, the device 110 may generate the beamformed audio data by boosting an audio signal originating from the desired direction (e.g., look direction) while attenuating audio signals that originate from other directions, although the disclosure is not limited thereto.
To perform the beamforming operation, the device 110 may apply directional calculations to the input audio signals. In some examples, the device 110 may perform the directional calculations by applying filters to the input audio signals using filter coefficients associated with specific directions. For example, the device 110 may perform a first directional calculation by applying first filter coefficients to the input audio signals to generate the first beamformed audio data and may perform a second directional calculation by applying second filter coefficients to the input audio signals to generate the second beamformed audio data.
The filter coefficients used to perform the beamforming operation may be calculated offline (e.g., preconfigured ahead of time) and stored in the device 110. For example, the device 110 may store filter coefficients associated with hundreds of different directional calculations (e.g., hundreds of specific directions) and may select the desired filter coefficients for a particular beamforming operation at runtime (e.g., during the beamforming operation). To illustrate an example, at a first time the device 110 may perform a first beamforming operation to divide input audio data into 36 different portions, with each portion associated with a specific direction (e.g., 10 degrees out of 360 degrees) relative to the device 110. At a second time, however, the device 110 may perform a second beamforming operation to divide input audio data into 6 different portions, with each portion associated with a specific direction (e.g., 60 degrees out of 360 degrees) relative to the device 110.
These directional calculations may sometimes be referred to as “beams” by one of skill in the art, with a first directional calculation (e.g., first filter coefficients) being referred to as a “first beam” corresponding to the first direction, the second directional calculation (e.g., second filter coefficients) being referred to as a “second beam” corresponding to the second direction, and so on. Thus, the device 110 stores hundreds of “beams” (e.g., directional calculations and associated filter coefficients) and uses the “beams” to perform a beamforming operation and generate a plurality of beamformed audio signals. However, “beams” may also refer to the output of the beamforming operation (e.g., plurality of beamformed audio signals). Thus, a first beam may correspond to first beamformed audio data associated with the first direction (e.g., portions of the input audio signals corresponding to the first direction), a second beam may correspond to second beamformed audio data associated with the second direction (e.g., portions of the input audio signals corresponding to the second direction), and so on. For ease of explanation, as used herein “beams” refer to the beamformed audio signals that are generated by the beamforming operation. Therefore, a first beam corresponds to first audio data associated with a first direction, whereas a first directional calculation corresponds to the first filter coefficients used to generate the first beam.
Prior to sending the microphone signal z(t) to the remote device/remote system 120, the device 110 may perform acoustic echo cancellation (AEC), adaptive interference cancellation (AIC), residual echo suppression (RES), and/or other audio processing to isolate local speech captured by the microphone(s) 112 and/or to suppress unwanted audio data (e.g., echoes and/or noise). For example, the device 110 may receive the far-end reference signal x(t) (e.g., playback audio data) and may generate playback audio (e.g., echo signal y(t)) using the loudspeaker(s) 114. The far-end reference signal x(t) may be referred to as a far-end reference signal (e.g., far-end reference audio data), a playback signal (e.g., playback audio data) or the like. The one or more microphone(s) 112 in the microphone array may capture a microphone signal z(t) (e.g., microphone audio data, near-end reference signal, input audio data, etc.), which may include the echo signal y(t) along with near-end speech s(t) from the user 10 and noise n(t).
To isolate the local speech (e.g., near-end speech s(t) from the user 10), the device 110 may include an AIC component that selects target signal(s) and reference signal(s) from the beamformed audio data and generates an error signal m(t) by removing the reference signal(s) from the target signal(s). As the AIC component does not have access to the echo signal y(t) itself, the reference signal(s) are selected as an approximation of the echo signal y(t). Thus, when the AIC component removes the reference signal(s) from the target signal(s), the AIC component is removing at least a portion of the echo signal y(t). In addition, the reference signal(s) may include the noise n(t) and other acoustic interference. Therefore, the output (e.g., error signal m(t)) of the AIC component may include the near-end speech s(t) along with portions of the echo signal y(t) and/or the noise n(t) (e.g., difference between the reference signal(s) and the actual echo signal y(t) and noise n(t)).
An audio signal is a representation of sound and an electronic representation of an audio signal may be referred to as audio data, which may be analog and/or digital without departing from the disclosure. For ease of illustration, the disclosure may refer to either audio data (e.g., far-end reference audio data or playback audio data, microphone audio data, near-end reference data or input audio data, etc.) or audio signals (e.g., playback signal, far-end reference signal, microphone signal, near-end reference signal, etc.) without departing from the disclosure. Additionally or alternatively, portions of a signal may be referenced as a portion of the signal or as a separate signal and/or portions of audio data may be referenced as a portion of the audio data or as separate audio data. For example, a first audio signal may correspond to a first period of time (e.g., time range, such as 30 seconds) and a portion of the first audio signal corresponding to a second period of time (e.g., 1 second) may be referred to as a first portion of the first audio signal or as a second audio signal without departing from the disclosure. Similarly, first audio data may correspond to the first period of time (e.g., 30 seconds) and a portion of the first audio data corresponding to the second period of time (e.g., 1 second) may be referred to as a first portion of the first audio data or second audio data without departing from the disclosure. Audio signals and audio data may be used interchangeably, as well; a first audio signal may correspond to the first period of time (e.g., 30 seconds) and a portion of the first audio signal corresponding to a second period of time (e.g., 1 second) may be referred to as first audio data without departing from the disclosure.
As used herein, audio signals or audio data (e.g., far-end reference audio data, near-end reference audio data, microphone audio data, or the like) may correspond to a specific range of frequency bands. For example, far-end reference audio data and/or near-end reference audio data may correspond to a human hearing range (e.g., 20 Hz-20 kHz), although the disclosure is not limited thereto.
Far-end reference audio data (e.g., far-end reference signal x(t)) corresponds to audio data that will be output by the loudspeaker(s) 114 to generate playback audio (e.g., echo signal y(t)). For example, the device 110 may stream music or output speech associated with a communication session (e.g., audio or video telecommunication). In some examples, the far-end reference audio data may be referred to as playback audio data, loudspeaker audio data, and/or the like without departing from the disclosure. For ease of illustration, the following description will refer to the playback audio data as far-end reference audio data. As noted above, the far-end reference audio data may be referred to as far-end reference signal(s) x(t) without departing from the disclosure.
Microphone audio data corresponds to audio data that is captured by the microphone(s) 112 prior to the device 110 performing audio processing such as AIC processing. The microphone audio data may include local speech s(t) (e.g., an utterance, such as near-end speech generated by the user 10), an “echo” signal y(t) (e.g., portion of the playback audio captured by the microphone(s) 112), acoustic noise n(t) (e.g., ambient noise in an environment around the device 110), and/or the like. As the microphone audio data is captured by the microphone(s) 112 and captures audio input to the device 110, the microphone audio data may be referred to as input audio data, near-end audio data, and/or the like without departing from the disclosure. For ease of illustration, the following description will refer to microphone audio data and near-end reference audio data interchangeably. As noted above, the near-end reference audio data/microphone audio data may be referred to as a near-end reference signal or microphone signal z(t) without departing from the disclosure.
Output audio data corresponds to audio data after the device 110 performs audio processing (e.g., AIC processing, ANC processing, AEC processing, and/or the like) to isolate the local speech s(t). For example, the output audio data r(t) corresponds to the microphone audio data z(t) after subtracting the reference signal(s) (e.g., using an adaptive interference cancellation (AIC) component), optionally performing residual echo suppression (RES) (e.g., using a RES component), and/or other audio processing known to one of skill in the art. As noted above, the output audio data may be referred to as output audio signal(s) without departing from the disclosure, and one of skill in the art will recognize that the output audio data may also be referred to as an error audio data m(t), error signal m(t) and/or the like.
Further details of the device operation are described below following a discussion of directionality in reference to
As illustrated in
Using such direction isolation techniques, a device 110 may isolate directionality of audio sources. As shown in
To isolate audio from a particular direction the device may apply a variety of audio filters to the output of the microphones where certain audio is boosted while other audio is dampened, to create isolated audio data corresponding to a particular direction, which may be referred to as a beam. While in some examples the number of beams may correspond to the number of microphones, the disclosure is not limited thereto and the number of beams may vary from the number of microphones without departing from the disclosure. For example, a two-microphone array may be processed to obtain more than two beams, using filters and beamforming techniques to isolate audio from more than two directions. Thus, the number of microphones may be more than, less than, or the same as the number of beams. The beamformer unit of the device may have a fixed beamformer (FBF) unit and/or an adaptive beamformer (ABF) unit processing pipeline for each beam without departing from the disclosure.
The device 110 may use various techniques to determine the beam corresponding to the look-direction. For example, if audio is first detected by a particular microphone, the device 110 may determine that the source of the audio is associated with the direction of the microphone in the array. Other techniques may include determining which microphone detected the audio with a largest amplitude (which in turn may result in a highest strength of the audio signal portion corresponding to the audio). Other techniques (either in the time domain or in the sub-band domain) may also be used such as calculating a signal-to-noise ratio (SNR) for each beam, performing voice activity detection (VAD) on each beam, or the like.
To illustrate an example, if audio data corresponding to a user's speech is first detected and/or is most strongly detected by microphone 212g, the device 110 may determine that a user 301 is located at a location in direction 7. Using a FBF unit or other such component, the device 110 may isolate audio data coming from direction 7 using techniques known to the art and/or explained herein. Thus, as shown in
One drawback to the FBF unit approach is that it may not function as well in dampening/canceling noise from a noise source that is not diffuse, but rather coherent and focused from a particular direction. For example, as shown in
Conventional systems isolate the speech in the input audio data by performing acoustic echo cancellation (AEC) to remove the echo signal from the input audio data. For example, conventional acoustic echo cancellation may generate a reference signal based on the playback audio data and may remove the reference signal from the input audio data to generate output audio data representing the speech.
As an alternative to generating the reference signal based on the playback audio data, Adaptive Reference Algorithm (ARA) processing may generate an adaptive reference signal based on the input audio data. For example, the device 110 may perform beamforming using the input audio data to generate a plurality of audio signals (e.g., beamformed audio data) corresponding to particular directions (e.g., a first audio signal corresponding to a first direction, a second audio signal corresponding to a second direction, etc.). After beamforming, the device 110 may optionally perform adaptive interference cancellation using the ARA processing on the beamformed audio data. For example, after generating the plurality of audio signals, the device 110 may determine one or more target signal(s), determine one or more reference signal(s), and generate output audio data by subtracting at least a portion of the reference signal(s) from the target signal(s). For example, the ARA processing may select the first audio signal as a target signal (e.g., the first audio signal includes a representation of speech) and the second audio signal as a reference signal (e.g., the second audio signal includes a representation of the echo and/or other acoustic noise), and may perform AIC by removing (e.g., subtracting) the reference signal from the target signal.
To improve noise cancellation, the AIC component may amplify audio signals from two or more directions other than the look direction (e.g., target signal). These audio signals represent noise signals so the resulting amplified audio signals may be referred to as noise reference signals. The device 110 may then weight the noise reference signals, for example using filters, and combine the weighted noise reference signals into a combined (weighted) noise reference signal. Alternatively the device 110 may not weight the noise reference signals and may simply combine them into the combined noise reference signal without weighting. The device 110 may then subtract the combined noise reference signal from the target signal to obtain a difference (e.g., noise-cancelled audio data). The device 110 may then output that difference, which represents the desired output audio signal with the noise removed. The diffuse noise is removed by the FBF unit when determining the target signal and the directional noise is removed when the combined noise reference signal is subtracted.
The device 110 may dynamically select target signal(s) and/or reference signal(s). Thus, the target signal(s) and/or the reference signal(s) may be continually changing over time based on speech, acoustic noise(s), ambient noise(s), and/or the like in an environment around the device 110. For example, the adaptive beamformer may select the target signal(s) by detecting speech, based on signal strength values (e.g., signal-to-noise ratio (SNR) values, average power values, etc.), and/or using other techniques or inputs, although the disclosure is not limited thereto. As an example of other techniques or inputs, the device 110 may capture video data corresponding to the input audio data, analyze the video data using computer vision processing (e.g., facial recognition, object recognition, or the like) to determine that a user is associated with a first direction, and select the target signal(s) by selecting the first audio signal corresponding to the first direction. Similarly, the device 110 may identify the reference signal(s) based on the signal strength values and/or using other inputs without departing from the disclosure. Thus, the target signal(s) and/or the reference signal(s) selected by the device 110 may vary, resulting in different filter coefficient values over time.
As discussed above, microphone failure negatively impacts a performance of the device 110. To improve performance of the device 110, the device 110 may detect defective microphones, such as nonfunctional (e.g., dead) and/or malfunctioning (e.g., attenuated) microphone(s), and re-optimize configuration settings of the device 110 accordingly. The device 110 may detect defective microphone(s) by measuring energy levels for each of the microphones 112 in the microphone array for a period of time (e.g., 30 seconds). For example, the device 110 may detect a defective microphone when an energy level for the defective microphone is lower than an absolute threshold value. Additionally or alternatively, the device 110 may detect a defective microphone when an energy level for the defective microphone is lower than a reference energy level (e.g., lowest energy level for remaining microphones 112, average energy level for remaining microphones 112, and/or the like) by more than a threshold offset (e.g., 6 dB).
Microphone energy chart 410 illustrates an example of detecting the defective microphone using an absolute threshold value 412. For example, the device 110 may detect a defective microphone when an energy level for a microphone 112 is lower than the absolute threshold value 412. Thus, any microphones 112 associated with an energy level above the absolute threshold value 412 are considered functional, and any microphones 112 associated with an energy level below the absolute threshold value 412 are considered defective, regardless of a magnitude of the output audio being captured. The absolute threshold value 412 may be selected to distinguish low energy levels (e.g., near zero) from active energy levels of functional microphones 112.
As illustrated in
In contrast, microphone energy chart 420 illustrates an example of detecting the defective microphone using a relative threshold value 426. For example, the device 110 may detect a defective microphone when an energy level for a microphone 112 is lower than the relative threshold value 426. Thus, any microphones 112 associated with an energy level above the relative threshold value 426 are considered functional, and any microphones 112 associated with an energy level below the relative threshold value 426 are considered defective. Thus, the relative threshold value 426 takes into account a magnitude of the output audio being captured, as the relative threshold value 426 is determined based on energy levels of functional microphone(s) 112. For example, the relative threshold value 426 may be lower when a magnitude of the output audio is lower and may be higher when a magnitude of the output audio is higher.
The relative threshold value 426 may be determined by determining a reference level 422 and subtracting a threshold offset 424 from the reference level 422. The device 110 may determine the reference level 422 based on energy levels associated with a majority of the microphones 112. For example, the reference level 422 may be determined based on a second lowest energy value, a lowest energy value for a group of microphones 112, an average energy level for a group of microphones 112, and/or the like without departing from the disclosure, although the disclosure is not limited thereto.
As illustrated in
Using the relative threshold value 426, the device 110 may determine that a third energy level of the third microphone (Mic3) is below the relative threshold value 426 and therefore the third microphone (Mic3) is defective, even though the third energy level may be above the absolute threshold value 412. Thus, even though the third microphone generates some audio data, the device 110 may improve performance by determining that the third microphone has a lower energy level than the remaining microphones and re-optimizing configuration settings to exclude the third microphone.
Periodic detection 520 corresponds to performing microphone failure detection periodically based on a period of time (e.g., every day, every week, etc.). In some examples, the periodic detection 520 may be performed using a fixed interval 522 (e.g., fixed period of time), such that the device 110 performs microphone failure detection consistently at the same time. For example, the fixed interval 522 may result in the device 110 performing microphone failure detection every day at a scheduled time, every week at a scheduled time, and/or the like.
Additionally or alternatively, the periodic detection 520 may be performed using a variable interval 524 (e.g., variable period of time), such that the device 110 performs microphone failure detection at approximately the same time depending on activity of the device 110. For example, the variable interval 524 may correspond to a desired period of time (e.g., daily, weekly, etc.), but the device 110 may delay performing microphone failure detection until the device 110 generates output audio using the loudspeaker(s) 114. Thus, the device 110 may perform microphone failure detection every day or week at variable times depending on when output audio is being generated by the device.
In contrast, activity-based detection 530 corresponds to performing microphone failure detection based on an activity being performed, without regard to a duration of time or frequency of the activity. For example, the device 110 may perform microphone failure detection while outputting audio 532, which corresponds to performing microphone failure detection as a background process every time the device 110 generates output audio. Additionally or alternatively, the device 110 may perform microphone failure detection prior to establishing a communication session 534, which corresponds to performing microphone failure detection as an initialization step before establishing the communication session (e.g., using VoIP or the like).
Performing periodic detection 520 is beneficial as the device 110 regularly verifies whether the microphones 112 are functional and optimizes configuration settings based on the functional microphones 112, regardless of the activities performed by the device 110. In contrast, performing activity-based detection 530 is beneficial as the device 110 verifies whether the microphones 112 are functional prior to and/or while performing an activity, ensuring that a performance of the device 110 is optimized for the activity and avoiding a negative user experience caused by a defective microphone. The device 110 may perform periodic detection 520 and activity-based detection 530, although the disclosure is not limited thereto.
As illustrated in
Passive detection 560 corresponds to performing microphone failure detection based on output audio generated by other applications or tasks running on the device 110. For example, the device 110 may be generating output audio corresponding to music and may perform microphone failure detection using the output audio. Thus, performing the microphone failure detection may depend on other activities (e.g., when output audio is being generated), but may not be detected by the user 5 as the music is being used as a test signal with which to measure energy levels for the microphones 112.
As illustrated in
While
Active detection 570 corresponds to generating output audio in order to perform microphone failure detection. For example, the device 110 may generate output audio corresponding to a test signal and may perform microphone failure detection using the test signal. Thus, the device 110 may perform microphone failure detection regardless of whether the device 110 is currently active and/or already generating output audio.
As illustrated in
Additionally or alternatively,
While the first multi-microphone implementation 620 indicates that all functional microphones 112 are selected, the disclosure is not limited thereto. For example,
As mentioned above, the microphone configurations illustrated in
In some examples, the device 110 may select one or more first microphones as a target signal and select one or more second microphones as a reference signal. For example, the device 110 may arbitrarily select a single microphone as the reference signal and use the remaining microphones to generate the target signal. Thus, the device 110 may perform echo cancellation by removing the reference signal from the target signal. During microphone reassignment, the device 110 may update the one or more first microphones and/or the one or more second microphones to remove defective microphones and/or adjust the microphone assignment accordingly. For example, if the single reference microphone is defective, the device 110 may select one of the functional microphones as a new reference microphone and recalculate settings (e.g., beamformer coefficient values) for the remaining microphones.
Adjusting front-end DNN selection (670) may correspond to selecting a different front-end DNN to process input audio data based on a number of input channels received from functional microphones. For example, each front-end DNN may be configured to process input audio data from a fixed number of microphones, such that input audio data associated with four microphones is sent to a four-input DNN, input audio data associated with three microphones is sent to a three-input DNN, and so on. Therefore, if one or more microphones is determined to be defective, the device 110 may select a different front-end DNN to process the input audio data based on the number of functional microphones remaining.
Disabling noise cancellation using a reference microphone (680) may correspond to performing noise cancellation using a dedicated reference microphone. For example, the reference microphone may be positioned in proximity to a noise source and/or directed at a noise source and the device 110 may perform a first stage of noise cancellation using audio data generated by the reference microphone. If the device 110 determines that the reference microphone is defective, the device 110 may disable the first stage of noise cancellation.
Updating multi-microphone algorithms (690) may correspond to particular algorithms that are performed using audio data generated by multiple microphones 112 of the device 110. For example, an algorithm may reduce wind noise or the like based on audio data generated by several microphones 112. If the device 110 determines that one of the microphones is defective, the device 110 may update the algorithm to deselect the defective microphone(s).
As illustrated in
The remote system 120 may determine (716) device identification unique to the device 110 and may determine (718) microphone configuration data corresponding to the device identification. For example, the output data and/or audio data associated with the device 110 may include the unique identification associated with the device 110 and the remote system 120 may determine the device identification based on the output data and/or the audio data. The remote system 120 may use the device identification to identify the microphone configuration data associated with the device 110 in a device dictionary stored on the remote system 120. The remote system 120 may identify (720) defective microphone(s) indicated by the output data, may generate (722) first data based on the defective microphone(s) and may send (724) the first data to the device 110. For example, the first data may correspond to beamformer coefficient values calculated using functional microphones (e.g., excluding the defective microphone(s)) and/or the like.
The device 110 may receive the first data, may update (726) settings (e.g., configuration settings) stored on the device using the first data, and may optionally reset (728) a flag in the output data to indicate that the settings are updated.
While
In some examples, the microphone dictionary may indicate the defective microphone(s) using binary values, as illustrated by microphone dictionary 820. For example, microphone dictionary 820 illustrates that the update required flag 822 and the second microphone flag 824b are set using a binary value of one, with all other flags corresponding to a binary value of zero. However, the disclosure is not limited thereto and selected flags may be indicated using a binary value of zero and/or using any other technique known to one of skill in the art without departing from the disclosure.
In some examples, the microphone dictionary can indicate additional information beyond whether a microphone is defective. For example, the device 110 may detect an attenuation and/or phase shift associated with the microphone, enabling the system 100 to compensate for the attenuation and/or phase shift without considering the microphone defective. As will be described in greater detail below, the device 110 may determine the attenuation and/or phase shift by comparing input audio data received from an individual microphone to expected audio data associated with the microphone, such as by calculating a cross correlation and/or the like. In some examples, the device 110 may determine the expected audio data based on reference audio data, such as output audio data sent to the loudspeaker(s) 114. For example, the device 110 may calculate the expected audio data based on the output audio data and a transfer function associated with the microphone (e.g., acoustic echo transfer function), although the disclosure is not limited thereto. In other examples, the device 110 may determine the expected audio data based on the remaining microphones 112 in the microphone array. For example, if the microphone array includes eight microphones 112a-112h, the device 110 may determine expected audio data for a first microphone 112a based on input audio data associated with the remaining seven microphones 112b-112h.
As illustrated in
To illustrate an example, the microphone dictionary 830 indicates that a first microphone Mic1 is operating normally (e.g., not defective, with no measurable attenuation or phase shift), a second microphone Mic2 is functional but experiencing an attenuation of 10 dB and a phase shift of 60 degrees, and a third microphone MicN is defective. In the example illustrated in
As the device 110 determined that one of the microphones is attenuated (e.g., Mic2) and/or defective (e.g., MicN), the device 110 may set the update required flag 832 and send the microphone dictionary 830 as output data to the remote system 120. Using the microphone dictionary 830, the remote system 120 may calculate updated parameters and/or settings that improve a performance of the device 110. For example, the remote system 120 may calculate updated beamformer coefficients for the seven functional microphones 112 (e.g., including the second microphone Mic2, but ignoring the third microphone MicN completely) that compensate for the attenuation value and/or phase shift value associated with the second microphone Mic2.
The device 110 and/or the remote system 120 may store the microphone dictionary 810/820/830 and may use the microphone dictionary 810/820/830 to determine which microphone(s) 112 are defective and/or whether updated settings are required. For example, once the remote system 120 calculates updated settings (e.g., first data) and sends the first data to the device 110, the device 110 may update the microphone dictionary 810/820/830 to unselect the update required flag 812/822/832.
In some examples, the remote system 120 may store additional information in the device dictionary, as illustrated by device dictionary 850. For example, device dictionary 850 may indicate a device class, a model, a version, a number of microphones, a microphone configuration (e.g., arrangement or orientation), an indication of defective microphone(s), additional microphone data (e.g., attenuation and/or phase shift for a functional microphone), and/or the like. However, this is intended as an illustrative example and the disclosure is not limited thereto. Thus, the device dictionary may include any information associated with identifying a particular device and/or microphones 112 of the particular device 110 without departing from the disclosure.
In some examples, the device 110 may store preconfigured settings corresponding to potential microphone configurations and update settings locally using the preconfigured settings. For example, the remote system 120 may calculate the preconfigured settings (e.g., beamformer coefficients and/or the like) for each microphone configuration and the device 110 may store the preconfigured settings for later reference. Thus, if the device 110 determines that a microphone is defective, the device 110 may identify a current microphone configuration based on the remaining functional microphones, identify preconfigured settings corresponding to the current microphone configuration, and update settings on the device 110 using the preconfigured settings. As a result, the device 110 may update the settings based on the defective microphone without sending the output data to the remote system 120 or receiving the first data from the remote system 120.
As illustrated in
As illustrated in the configuration dictionary 860, a first microphone configuration that corresponds to all eight microphones being functional (e.g., no defective microphones) may be associated with first preconfigured settings (e.g., Settings 1), a second microphone configuration that corresponds to seven functional microphones (e.g., one defective microphone) may be associated with second preconfigured settings (e.g., Settings 2), a third microphone configuration that corresponds to six functional microphones (e.g., two neighboring defective microphones) may be associated with third preconfigured settings (e.g., Settings 3), a fourth microphone configuration that corresponds to six functional microphones (e.g., two defective microphones separated by a functional microphone) may be associated with fourth preconfigured settings (e.g., Settings 4), a fifth microphone configuration that corresponds to six functional microphones (e.g., two defective microphones separated by two functional microphones) may be associated with fifth preconfigured settings (e.g., Settings 5), a sixth microphone configuration that corresponds to six functional microphones (e.g., two defective microphones separated by three functional microphones) may be associated with sixth preconfigured settings (e.g., Settings 6), and so on until a final microphone configuration that corresponds to one functional microphone (e.g., seven defective microphones) may be associated with final preconfigured settings (e.g., Settings Z).
While the configuration dictionary 860 shows a simplified version of preconfigured settings, the disclosure is not limited thereto and in some examples the system 100 may generate preconfigured settings for every potential combination of defective microphones without departing from the disclosure. As illustrated in
While
The device 110 may determine (916) a first plurality of energy levels that do not satisfy a condition (e.g., below a threshold value, such as an absolute threshold value or a relative threshold value), identify (918) first microphones associated with the first plurality of energy levels, and determine (920) that the first microphones are defective. In some examples, the device 110 may optionally determine (922) a second plurality of energy levels that satisfy the condition (e.g., above a threshold value, such as an absolute threshold value or a relative threshold value), identify (924) second microphones associated with the second plurality of energy levels, and determine (926) that the second microphones are functional. However, the disclosure is not limited thereto and the device 110 may only determine the defective microphones without departing from the disclosure. The device 110 may then generate (928) output data indicating the defective microphones.
As illustrated in
The device 110 may determine (966) whether there is an additional microphone and, if so, may loop to step 952 and repeat steps 952-964 for the additional microphone. If there isn't an additional microphone, the device 110 may generate (968) output data indicating the defective microphone(s).
In some examples, in addition to or instead of determining that an individual microphone is defective, the device 110 may identify an attenuation and/or phase shift associated with the individual microphone. This enables the system 100 to compensate for the attenuation and/or phase shift without considering the microphone defective. The device 110 may determine the attenuation and/or phase shift by comparing input audio data received from an individual microphone to expected audio data associated with the microphone, such as by calculating a cross correlation and/or the like. In some examples, the device 110 may determine the expected audio data based on reference audio data, such as output audio data sent to the loudspeaker(s) 114. For example, the device 110 may calculate the expected audio data based on the output audio data and a transfer function associated with the microphone (e.g., acoustic echo transfer function), although the disclosure is not limited thereto. In other examples, the device 110 may determine the expected audio data based on the remaining microphones 112 in the microphone array. For example, if the microphone array includes eight microphones 112a-112h, the device 110 may determine expected audio data for a first microphone 112a based on input audio data associated with the remaining seven microphones 112b-112h.
As illustrated in
The device 110 may select (972) a microphone, determine (974) an energy level for the selected microphone, and determine (976) whether the energy level is above the first energy threshold value. If the energy level is below the first energy threshold value, the device 110 may determine (978) that the selected microphone is defective and store (980) an indication that the selected microphone is defective, as described in greater detail above.
If the energy level is above the first energy threshold value, the device 110 may determine (982) whether the energy level is above the second energy threshold value. If the energy level is above the second energy threshold value, the device 110 may determine (984) that the selected microphone is functional and may store (986) an indication that the microphone is functional. If the energy level is below the second energy threshold value, the device 110 may determine (988) an attenuation and/or phase shift corresponding to the selected microphone and may store (990) an indication of the attenuation and/or phase shift.
The device 110 may then determine (992) whether there is an additional microphone and, if so, may loop to step 972 and repeat steps 972-990 for the additional microphone. If there are no additional microphones, the device 110 may generate (994) output data indicating that an individual microphone is functional, defective, and/or additional information (e.g., attenuation and/or phase shift) associated with the microphone.
The device 110 may select (1014) a microphone, determine (1016) a portion of the input audio data corresponding to the selected microphone, and determine (1018) expected audio data based on the output audio data and a transfer function (e.g., echo estimate transfer function) unique to the selected microphone. For example, the device 110 may estimate what portion of the output audio data is captured by the selected microphone.
The device 110 may determine (1020) an attenuation and/or phase shift between the portion of the input audio data and the expected audio data and store (1022) an indication of the attenuation and/or the phase shift. For example, the device 110 may generate cross-correlation data (e.g., calculate a cross correlation between the portion of the input audio data and the expected audio data) and determine the attenuation and/or phase shift based on the cross-correlation data.
The device 110 may determine (1024) whether there is an additional microphone and, if so, may loop to step 1014 and repeat steps 1014-1022 for the additional microphone. If there is not an additional microphone, the device 110 may generate (1026) output data indicating defective microphone(s), attenuation, and/or phase shifts. For example, the device 110 may determine that any attenuation above a threshold value corresponds to a defective microphone, although the disclosure is not limited thereto.
While
As illustrated in
The device 110 may select (1014) a microphone, determine (1016) a portion of the input audio data corresponding to the selected microphone, and determine (1050) expected audio data based on remaining input audio data. For example, the device 110 may determine first expected audio data for a first microphone Mic1 using microphones Mic2-Mic8, second expected audio data for a second microphone Mic2 using microphones Mic1 and Mic3-Mic8, and so on.
The device 110 may determine (1020) an attenuation and/or phase shift between the portion of the input audio data and the expected audio data and store (1022) an indication of the attenuation and/or the phase shift. For example, the device 110 may generate cross-correlation data (e.g., calculate a cross correlation between the portion of the input audio data and the expected audio data) and determine the attenuation and/or phase shift based on the cross-correlation data.
The device 110 may determine (1024) whether there is an additional microphone and, if so, may loop to step 1014 and repeat steps 1014-1022 for the additional microphone. If there is not an additional microphone, the device 110 may generate (1026) output data indicating defective microphone(s), attenuation, and/or phase shifts. For example, the device 110 may determine that any attenuation above a threshold value corresponds to a defective microphone, although the disclosure is not limited thereto.
In some examples, the device 110 may store preconfigured settings locally on the device 110 and may update the settings using the preconfigured settings. As illustrated in
As illustrated in
In some examples, the device 110 may determine an attenuation and/or phase shift in addition to and/or instead of determining which microphone(s) are defective. As illustrated in
While the above description refers to determining that microphone(s) are defective, in some situations a microphone may be intermittently functional or defective and/or may regain functionality. Thus, in some examples the system 100 may determine that a previously defective microphone is functional or that an attenuation and/or phase shift has changed significantly and update settings accordingly.
As illustrated in
While
Similarly, while
Multiple servers may be included in the remote system 120. In operation, each of these devices (or groups of devices) may include computer-readable and computer-executable instructions that reside on the respective device (110/120), as will be discussed further below.
Each of these devices (110/120) may include one or more controllers/processors (1304/1404), which may each include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (1306/1406) for storing data and instructions of the respective device. The memories (1306/1406) may individually include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magnetoresistive memory (MRAM), and/or other types of memory. Each device (110/120) may also include a data storage component (1308/1408) for storing data and controller/processor-executable instructions. Each data storage component (1308/1408) may individually include one or more non-volatile storage types such as magnetic storage, optical storage, solid-state storage, etc. Each device (110/120) may also be connected to removable or external non-volatile memory and/or storage (such as a removable memory card, memory key drive, networked storage, etc.) through respective input/output device interfaces (1302/1402).
Computer instructions for operating each device (110/120) and its various components may be executed by the respective device's controller(s)/processor(s) (1304/1404), using the memory (1306/1406) as temporary “working” storage at runtime. A device's computer instructions may be stored in a non-transitory manner in non-volatile memory (1306/1406), storage (1308/1408), or an external device(s). Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device in addition to or instead of software.
Each device (110/120) includes input/output device interfaces (1302/1402). A variety of components may be connected through the input/output device interfaces (1302/1402), as will be discussed further below. Additionally, each device (110/120) may include an address/data bus (1324/1424) for conveying data among components of the respective device. Each component within a device (110/120) may also be directly connected to other components in addition to (or instead of) being connected to other components across the bus (1324/1424).
Referring to
Via antenna(s) 1314, the input/output device interfaces 1302 may connect to one or more networks 199 via a wireless local area network (WLAN) (such as WiFi) radio, Bluetooth, and/or wireless network radio, such as a radio capable of communication with a wireless communication network such as a Long Term Evolution (LTE) network, WiMAX network, 3G network, 4G network, 5G network, etc. A wired connection such as Ethernet may also be supported. Through the network(s) 199, the system may be distributed across a networked environment. The I/O device interface (1302/1402) may also include communication components that allow data to be exchanged between devices such as different physical servers in a collection of servers or other components.
The components of the device(s) 110 and the remote system 120 may include their own dedicated processors, memory, and/or storage. Alternatively, one or more of the components of the device(s) 110 and the remote system 120 may utilize the I/O interfaces (1302/1402), processor(s) (1304/1404), memory (1306/1406), and/or storage (1308/1408) of the device(s) 110 and remote system 120, respectively. Thus, the ASR component 250 may have its own I/O interface(s), processor(s), memory, and/or storage; the NLU component 260 may have its own I/O interface(s), processor(s), memory, and/or storage; and so forth for the various components discussed herein.
As noted above, multiple devices may be employed in a single system. In such a multi-device system, each of the devices may include different components for performing different aspects of the system's processing. The multiple devices may include overlapping components. The components of the device 110 and the remote system 120, as described herein, are illustrative, and may be located as a stand-alone device or may be included, in whole or in part, as a component of a larger device or system.
The concepts disclosed herein may be applied within a number of different devices and computer systems, including, for example, general-purpose computing systems, speech processing systems, and distributed computing environments.
The above aspects of the present disclosure are meant to be illustrative. They were chosen to explain the principles and application of the disclosure and are not intended to be exhaustive or to limit the disclosure. Many modifications and variations of the disclosed aspects may be apparent to those of skill in the art. Persons having ordinary skill in the field of computers and speech processing should recognize that components and process steps described herein may be interchangeable with other components or steps, or combinations of components or steps, and still achieve the benefits and advantages of the present disclosure. Moreover, it should be apparent to one skilled in the art, that the disclosure may be practiced without some or all of the specific details and steps disclosed herein.
Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture such as a memory device or non-transitory computer readable storage medium. The computer readable storage medium may be readable by a computer and may comprise instructions for causing a computer or other device to perform processes described in the present disclosure. The computer readable storage medium may be implemented by a volatile computer memory, non-volatile computer memory, hard drive, solid-state memory, flash drive, removable disk, and/or other media. In addition, components of system may be implemented as in firmware or hardware, such as an acoustic front end (AFE), which comprises, among other things, analog and/or digital filters (e.g., filters configured as firmware to a digital signal processor (DSP)).
Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements, and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without other input or prompting, whether these features, elements, and/or steps are included or are to be performed in any particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list.
Disjunctive language such as the phrase “at least one of X, Y, Z,” unless specifically stated otherwise, is understood with the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to each be present.
As used in this disclosure, the term “a” or “one” may include one or more items unless specifically stated otherwise. Further, the phrase “based on” is intended to mean “based at least in part on” unless specifically stated otherwise.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9984675 | Schuster | May 2018 | B2 |
| 20040165735 | Opitz | Aug 2004 | A1 |
| 20150317981 | Yliaho | Nov 2015 | A1 |
| 20160050488 | Matheja | Feb 2016 | A1 |
| 20160134966 | Fitzgerald | May 2016 | A1 |
| 20170272878 | Partio | Sep 2017 | A1 |
| 20180270565 | Ganeshkumar | Sep 2018 | A1 |
| 20190173446 | Knode | Jun 2019 | A1 |
