Patent Grant
11887574
The disclosure relates to a wearable electronic apparatus and a controlling method thereof, and, more particularly, to a wearable electronic apparatus identifying a dialog situation based on a user's speech and changing its operation mode, and a controlling method thereof.
In recent years, technology for wearable earphones worn on user's ears has been developed, and active noise cancellation (ANC) technology for wearable earphones has thus also been developed.
The ANC technology is technology for canceling or blocking an external noise that may be interference if the user listens to music by the wearable earphones. In detail, it is possible to receive the external noise by a microphone of the wearable earphone and convert the noise into a data signal, generate a reverse-phase wavelength corresponding thereto and provide the wavelength to a speaker of the wearable earphone, thereby canceling or blocking the external noise.
However, in the related art, the user has been required to directly control the ANC function by turning on or off this function.
Embodiments provide a wearable electronic apparatus identifying a dialog situation based on a user's speech and changing its operation mode based on the dialog situation, and a controlling method thereof.
In accordance with an aspect of the disclosure, there is provided a controlling method of a wearable electronic apparatus worn on user's ears. The controlling method includes: receiving, by an inertial measurement unit sensor, a bone conduction signal corresponding to vibration generated in the user's face, while the wearable electronic apparatus is operated in an active noise cancellation (ANC) mode; identifying a presence or an absence of the user's voice based on the bone conduction signal while the wearable electronic apparatus is operated in the ANC mode; based on the identifying the presence of the user's voice while the wearable electronic apparatus is operated in the ANC mode, controlling an operation mode of the wearable electronic apparatus to be a different operation mode from the ANC mode; while the wearable electronic apparatus is operated in the different operation mode, identifying a presence or an absence of the user's voice based on the bone conduction signal; and based on the absence of the user's voice being identified for a predetermined time while the wearable electronic apparatus is operated in the different operation mode, controlling the different operation mode to return to the ANC mode.
The controlling method further includes: based on the identifying the absence of the user's voice while the wearable electronic apparatus is operated in the ANC mode, controlling the ANC mode to be maintained.
The controlling method further includes: based on the presence of the user's voice being identified within the predetermined time while the wearable electronic apparatus is operated in the different operation mode, controlling the different operation mode to be maintained.
The identifying the presence or the absence of the user's voice while the wearable electronic apparatus is operated in the ANC mode further includes: identifying a probability indicating whether the user's voice exists in a plurality of frame units, respectively, that are included in the bone conduction signal, wherein the bone conduction signal is split into the plurality of frame units each having a predetermined duration; and identifying a frame unit among the plurality of frame units, as a current frame in which the user's voice exists based on the identifying that the probability for the frame unit has a predetermined value or more.
The controlling the operation mode to be the different operation mode further includes: identifying whether the current frame corresponds to a humming based on the identifying that the user's voice exists in the current frame; and based on the identifying that the current frame does not correspond to the humming, controlling the operation mode of the wearable electronic apparatus to be the different operation mode from the ANC mode.
The controlling method further includes: based on the identifying that the current frame corresponds to the humming, controlling the operation mode of the wearable electronic apparatus to be maintained as the ANC mode.
The different operation mode includes a normal operation mode in which an external noise is output as is, an AMBIENT mode in which the external noise is emphasized, and a Noise Focusing mode in which an external voice is emphasized.
The controlling the operation mode to be the different operation mode further includes: based on the identifying that the user's voice exists in the current frame, identifying a noise level of the current frame by using a microphone; and controlling the operation mode of the wearable electronic apparatus to be the Noise Focusing mode based on the noise level being identified to have a predetermined value or more.
The controlling method further includes: controlling the operation mode of the wearable electronic apparatus to be the AMBIENT mode based on the noise level being identified to have a value less than the predetermined value.
In accordance with an aspect of the disclosure, there is provided a wearable electronic apparatus worn on user's ears. The wearable electronic apparatus includes: a memory configured to store at least one instruction; an inertial measurement unit (IMU) sensor; and a processor which is, by executing the at least one instruction stored in the memory, configured to control the IMU sensor to receive a bone conduction signal corresponding to vibration generated in the user's face while the wearable electronic apparatus is operated in an active noise cancellation (ANC) mode, identify a presence or an absence of the user's voice based on the bone conduction signal while the wearable electronic apparatus is operated in the ANC mode, based on the identifying the presence of the user's voice while the wearable electronic apparatus is operated in the ANC mode, control an operation mode of the wearable electronic apparatus to be a different operation mode from the ANC mode, while the wearable electronic apparatus is operated in the different operation mode, identify a presence or an absence of the user's voice based on the bone conduction signal, and based on the absence of the user's voice being identified for a predetermined time while the wearable electronic apparatus is operated in the different operation mode, control the different operation mode to return to the ANC mode.
The processor is further configured to, based on the identifying the absence of the user's voice while the wearable electronic apparatus is operated in the ANC mode, control the ANC mode to be maintained.
The processor is further configured to, based on the presence of the user's voice being identified within the predetermined time while the wearable electronic apparatus is operated in the different operation mode, control the different operation mode to be maintained.
The processor is further configured to identify a probability indicating whether the user's voice exists in a plurality of frame units, respectively, that are included in the bone conduction signal, wherein the bone conduction signal is split into the plurality of frame units each having a predetermined duration, and identify a frame unit among the plurality of frame units, as a current frame in which the user's voice exists based on the identifying that the probability for the frame unit has a predetermined value or more.
The processor is further configured to identify whether the current frame corresponds to a humming based on the identifying that the user's voice exists in the current frame, and based on the identifying that the current frame does not correspond to the humming, control the operation mode of the wearable electronic apparatus to be the different operation mode from the ANC mode.
The processor is further configured to, based on the identifying that the current frame corresponds to the humming, control the operation mode of the wearable electronic apparatus to be maintained as the ANC mode.
In accordance with an aspect of the disclosure, there is provided anon-transitory computer-readable storage medium storing at least one instruction which, when executed by a processor of a wearable electronic apparatus, causes the processor to execute a method including: receiving, by an inertial measurement unit sensor of the wearable electronic apparatus, a bone conduction signal corresponding to vibration generated in the user's face, while the wearable electronic apparatus is operated in an active noise cancellation (ANC) mode; identifying a presence or an absence of the user's voice based on the bone conduction signal while the wearable electronic apparatus is operated in the ANC mode; based on the identifying the presence of the user's voice while the wearable electronic apparatus is operated in the ANC mode, controlling an operation mode of the wearable electronic apparatus to be a different operation mode from the ANC mode; while the wearable electronic apparatus is operated in the different operation mode, identifying a presence or an absence of the user's voice based on the bone conduction signal; and based on the absence of the user's voice being identified for a predetermined time while the wearable electronic apparatus is operated in the different operation mode, controlling the different operation mode to return to the ANC mode.
The method executed by the processor further includes: based on the identifying the absence of the user's voice while the wearable electronic apparatus is operated in the ANC mode, controlling the ANC mode to be maintained.
The method executed by the processor further includes: based on the presence of the user's voice being identified within the predetermined time while the wearable electronic apparatus is operated in the different operation mode, controlling the different operation mode to be maintained.
In the identifying the presence or the absence of the user's voice while the wearable electronic apparatus is operated in the ANC mode, the method executed by the processor further includes: identifying a probability indicating whether the user's voice exists in a plurality of frame units, respectively, that are included in the bone conduction signal, wherein the bone conduction signal is split into the plurality of frame units each having a predetermined duration; and identifying a frame unit among the plurality of frame units, as a current frame in which the user's voice exists based on the identifying that the probability for the frame unit has a predetermined value or more.
In the controlling the operation mode to be the different operation mode, the method executed by the processor further includes: identifying whether the current frame corresponds to a humming based on the identifying that the user's voice exists in the current frame, and based on the identifying that the current frame does not correspond to the humming, controlling the operation mode of the wearable electronic apparatus to be the different operation mode from the ANC mode.
The method executed by the processor further includes: based on the identifying that the current frame corresponds to the humming, controlling the operation mode of the wearable electronic apparatus to be maintained as the ANC mode.
According to embodiments, the wearable electronic apparatus may provide the operation mode based on the user's dialog situation, thereby having improved convenience.
The above and other aspects, features and advantages of certain embodiments of the present disclosure will be more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which:
Hereinafter, the disclosure is described in detail with reference to the accompanying drawings.
Referring to
The microphone 110 may be configured to receive noise around the wearable electronic apparatus. In detail, the microphone 110 may use a microphone to receive the noise around the wearable electronic apparatus 100 and convert the received noise into an electrical data signal. In this case, the microphone 110 may transmit the converted data signal to the processor 150.
In an embodiment, the microphone 110 may include an external microphone 112 disposed on the wearable electronic apparatus 100 to be positioned outside the user's ear. The external microphone may be disposed to be positioned outside the user's ear, and configured to receive the external noise.
In addition, the wearable electronic apparatus 100 may further include an internal microphone 160. The internal microphone may be positioned inside the user's ear, and configured to receive the user's spoken voice. For example, two external microphones may be implemented, and one internal microphone may be implemented. However, an embodiment is not limited thereto, and the various numbers of external microphones and internal microphones may be implemented.
The IMU sensor 120 may be configured to receive a bone conduction signal corresponding to vibration generated in the user's face. That is, the IMU sensor 120 may receive information on the vibration generated from the user's skin or bone and convert the received vibration into a waveform signal. In this case, the IMU sensor 120 may transmit the converted waveform signal to the processor 150. For example, the IMU sensor 120 may include an acceleration sensor capable of measuring the bone conduction signal. However, an embodiment is not limited thereto, and may include various sensors capable of measuring the bone conduction signal.
For example, if the wearable electronic apparatus 100 is worn on the user's ear, the IMU sensor 120 may be positioned in the wearable electronic apparatus 100 to be inserted in the user's ear canal. In addition, the IMU sensor 120 may receive the bone conduction signal conducted by the user's skin or bone. However, an embodiment is not limited thereto, and the IMU sensor 120 may be disposed to be in contact with an outer housing of the wearable electronic apparatus 100 that is inserted in the user's ear canal.
The speaker 130 is configured to output audio data. The speaker 130 according to an embodiment may output audio data from which the external noise is canceled or blocked (ANC mode), or output audio data in which the external noise is emphasized (AMBIENT mode), based on various operation modes of the wearable electronic apparatus 100. It is possible to output audio data in a normal operation mode in which external noise is output to the speaker as it is, not in the ANC mode or in the AMBIENT mode.
The memory 140 may store at least one instruction or data related to at least one another component of the wearable electronic apparatus 100. In particular, the memory 140 may be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SDD), etc. The memory 140 may be accessed by the processor 150, and the processor 150 may perform readout, recording, correction, deletion, update and the like of data therein.
In an embodiment, the term memory may include the memory 140, a read only memory (ROM, not shown) and a random access memory (RAM, not shown) in the processor 150, or a memory card (not shown) mounted on the wearable electronic apparatus 100 (e.g., micro secure digital (SD) card or memory stick).
As described above, the memory 140 may store at least one instruction. Here, the instruction may be for controlling the wearable electronic apparatus 100. For example, the memory 140 may store the instruction related to a function changing an operation mode based on a user's dialog situation. In detail, the memory 140 may include a plurality of components (or modules) changing the operation mode based on the user's dialog situation according to an embodiment, which is described below.
The processor 150 may be electrically connected to the memory 140 and may control the overall operation and function of the wearable electronic apparatus 100. In particular, the processor 150 may provide an operation mode change function changing the operation mode based on the user's dialog situation. As shown in
In addition, a plurality of modules 1000 through 5000 may be loaded into the memory (e.g., volatile memory) included in the processor 150 to perform the operation mode change function. That is, in order to perform the operation mode change function, the processor 150 may load the plurality of modules 1000 through 5000 from the non-volatile memory to the volatile memory, and then execute respective functions of the plurality of modules 1000 through 5000. Loading may indicate an operation of loading and storing data stored in the non-volatile memory into the volatile memory for the processor 150 to access the data.
In an embodiment, the operation mode change function am be implemented by the plurality of modules 1000 through 5000 stored in the memory 140 as shown in
The plurality of modules 1000 through 5000 according to an embodiment may be each implemented in software. However, an embodiment is not limited thereto, and some modules may be implemented as a combination of hardware and software. In another embodiment, the plurality of modules 1000 through 5000 may be implemented as a single software module. In addition, some modules may be implemented in the wearable electronic apparatus 100, while others may be implemented in an external apparatus.
The external voice identification module 1000 is configured to identify information on an external voice through the microphone 110. In detail, the external voice identification module 1000 may identify whether ambient noise data received by the microphone 110, via the external microphone 112, is the external voice. Here, the external voice may be an external voice which is different from the voice spoken by the user of the wearable electronic apparatus 1X). That is, the external voice may be a voice of a talker performing a speech with the user of the wearable electronic apparatus 100.
In an embodiment, the external voice identification module 1000 may identify whether the external voice is included in a noise signal received by the microphone 110 using a voice activity detection (VAD) technique. The VAD technique is a technique for distinguishing a voice and silence from each other in a noise signal, and may also be referred to as a “speech detection” technique.
In detail, the external voice identification module 1000 may identify whether the external voice is included in each frame of the noise signal using the VAD technique. For example, the external voice identification module 1000 may identify whether or not the external voice exists in the noise signal in a binary manner using the VAD technique.
In addition, if it is identified whether the external voice is included in each frame of the audio data, the external voice identification module 1000 may provide the identified information to the noise level identification module 3000.
The user voice identification module 2000 is configured to identify information on the voice of the user of the wearable electronic apparatus 100 based on the bone conduction signal of the user of the wearable electronic apparatus 100, obtained by the IMU sensor 120.
In detail, the user voice identification module 2000 may identify whether the user's voice is included in each frame of the bone conduction signal using a wearer speech detection (WSD) technique which uses the user's bone conduction signal obtained by the IMU sensor 120. The wearer speech detection (WSD) technique is a technique for obtaining a probability whether a voice exists in a frequency domain of the bone conduction signal based on energy of each frequency band. In detail, it is possible to estimate a noise spectrum of the bone conduction signal, and analyze the gain and signal to noise ratio (SNR) for the estimated spectrum, thereby identifying the probability whether the voice exists in each frame of the bone conduction signal. The user voice identification module 2000 may identify the probability whether the voice exists in each frame of the bone conduction signal by distinguishing a stationary voice signal and a non-stationary voice signal from each other in the bone conduction signal using the WSD technique.
In an embodiment, the user voice identification module 2000 may identify the probability whether the user's voice exists in each frame having a predetermined interval, e.g., a duration, by dividing the bone conduction signal into a plurality of frame units having the predetermined frame intervals, e.g., durations, (e.g., frame intervals of 10 ms). For example, the user voice identification module 2000 may identify the frame unit in which the probability that the voice exists has a predetermined value (e.g., 0.7) or more as the frame including the voice, e.g., a current frame including the voice.
The description describes the WSD technique which uses the user's bone conduction signal obtained by the IMU sensor 120 below with reference to
If it is identified that the probability whether the user's voice exists in the bone conduction signal, the user voice identification module 2000 may provide the identified probability whether the user's voice exists in each frame of the bone conduction signal to the noise level identification module 3000 and the dialog situation identification module 4000.
The noise level identification module 3000 is configured to identify an external noise level. In detail, the noise level identification module 3000 may identify the noise level except for the voice, based on: information on whether the external voice exists, which is provided by the external voice identification module 1000; information on the probability whether the user's voice exists, which is provided by the user voice identification module 2000; and the noise signal received from the microphone 110. That is, among the noise signal received from the microphone 110, the noise level identification module 3000 may identify the noise level of the other frame except: the frame identified as including the external voice by the external voice identification module 1000; and the frame identified as including the probability that the user's voice exists therein based on the predetermined value (e.g., 0.7) or more by the user voice identification module 2000.
In an embodiment, the noise level identification module 3000 may calculate the noise level with low complexity by using the lowest sampling frequency in a range including the maximum attenuation frequency. A method of calculating the noise level is described below with reference to
In an embodiment, the operation mode change function according to an embodiment may be implemented without the operation of the external voice identification module 1000. That is, among the noise signal received from the microphone 110, the noise level identification module 3000 may identify the noise level of the other frame except the frame having the probability that the user's voice exists therein based on the predetermined value (e.g., 0.7) or more, obtained by the user voice identification module 2000.
The dialog situation identification module 4000 is configured to identify whether a current situation is the dialog situation based on information on the probability whether the user's voice exists, provided by the user voice identification module 2000.
In an embodiment, the dialog situation identification module 4000 may identify whether the current situation is the dialog situation using only the information on the probability whether the user's voice exists, which is obtained by the user voice identification module 2000. In detail, in case that the frame identified as including the voice has the predetermined frame duration (e.g., frame duration of 500 ms) or more, based on the information on the probability that the user's voice exists, the dialog situation identification module 4000 may identify that the user has a dialog in the frame.
In addition, in case that the frame identified as including the voice has a frame duration less than the predetermined frame duration (e.g., less than 500 ms), based on the information on the probability that the user's voice exists, the dialog situation identification module 4000 may identify that the user simply makes an exclamation in the frame. That is, the dialog situation identification module 400) may identify that the user has no dialog in the frame.
However, an embodiment is not limited thereto. It is assumed that the user's voice is simply the user's humming even though it is identified that the frame identified as including the voice has the predetermined frame interval (frame interval of 500 ms). In this case, the dialog situation identification module 4000 may identify that the frame is the frame of a humming situation and not the frame of the dialog situation. For example, the dialog situation identification module 4000 may identify whether the user's voice is humming using a signal feature extraction technique. In detail, the dialog situation identification module 4000 may identify whether the user's voice is the humming by analyzing energy of the bone conduction signal. In addition, the dialog situation identification module 4000 may identify the frame identified as including the humming made by the user as that of the humming situation, and may thus identify that the user has no dialog.
For example, the dialog situation identification module 400) may identify the frame as that of the humming situation by identifying that the user makes the humming in the frame in case that a ratio between high-band energy and low-band energy has a value less than a predetermined value using a difference between the high-band energy and the low-band energy for each frame of the bone conduction signal.
For example, the dialog situation identification module 4000 may identify whether the user's voice is the humming by further using a zero crossing rate, which represents the periodic frequency of a waveform for each frame of the bone conduction signal.
That is, the humming may include a voiced sound accompanying the vibration of a vocal cord, and the dialog situation identification module 400) may thus identify the frame as that of the humming situation by identifying that the user makes the humming in the frame in case that a ratio of a frame size and the zero crossing rate for each frame of the bone conduction signal (zero crossing rate/frame size) has a value less than a predetermined value.
In addition, the dialog situation identification module 4000 may identify the frame as that of the humming situation by using both the difference between the high-band energy and the low-band energy for each frame of the bone conduction signal and the zero crossing rate for each frame of the bone conduction signal.
In addition, the dialog situation identification module 4000 may identify a start point of the identified frame as a dialog start point in case that the frame identified as including the dialog made by the user exists in the bone conduction signal.
In addition, the dialog situation identification module 4000 may identify a dialog end point after the dialog start point. In detail, the dialog situation identification module 4000 may identify the dialog end point depending on whether the user is identified as having a dialog again within a predetermined time (e.g., five seconds) from a point where the user's voice ends, after the dialog start point. That is, the dialog situation identification module 4000 may identify the dialog situation as continuing after the dialog start point in case that the user is identified as having the dialog within the predetermined time (e.g., five seconds) from the point where the user's voice ends, after the dialog start point. In addition, the dialog situation identification module 4000 may identify that the dialog is over in case that the user's dialog is not identified within the predetermined time (e.g., five seconds) from the point where the user's voice ends. Here, the predetermined time may be five seconds, but is not limited thereto, and may be determined or changed by the user or manufacturer of the wearable electronic apparatus 100 such as three seconds, seven seconds, nine seconds, etc.
The operation mode determination module 5000 is configured to determine the operation mode based on the external noise level identified by the noise level identification module 3000 and the dialog situation identified by the dialog situation identification module 4000.
In detail, the operation mode may include the ANC mode, the AMBIENT mode and the normal operation mode. However, the operation mode is not limited thereto, and may be implemented to only include the ANC mode and the AMBIENT mode.
In addition, the operation mode may further include an operation mode different from the ANC mode, the AMBIENT mode and the normal operation mode. For example, the operation mode may further include a Noise Focusing mode controlling the operation mode to be the ANC mode for a low frequency band and the AMBIENT mode for a high frequency band. While being operated in the Noise Focusing mode, the wearable electronic apparatus 100 may be operated to control the external noise corresponding to the low frequency band to be canceled and a voice corresponding to the high frequency band to be emphasized.
The ANC mode is a mode for outputting the audio data from which the external noise is canceled or blocked. The AMBIENT mode is a mode for outputting the audio data in which the external noise is emphasized. The normal operation mode is a mode for outputting the audio data as is without emphasizing or blocking the external noise.
In an embodiment, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 to be the AMBIENT mode if the current situation is identified as the dialog situation by the dialog situation identification module 4000. In addition, if the dialog is identified as being over, the operation mode determination module 5000 may control the operation mode to return to an original operation mode if the dialog is identified as being over. For example, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 to be the AMBIENT mode if the user is identified as having the dialog while the wearable electronic apparatus 100 is operated in the ANC mode. In addition, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 to return to the ANC mode if the dialog is identified as being over while the wearable electronic apparatus 100 is operated in the AMBIENT mode.
In an embodiment, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 to be the ANC mode if the noise level is identified as having a predetermined value (e.g., 80 dB) or more by the noise level identification module 3000. In addition, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 to return to the original operation mode if the noise level is identified as having a value lower than the predetermined value (e.g., 80 dB) while the wearable electronic apparatus 100 is operated in the ANC mode. For example, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 to be the ANC mode if the noise level is identified as having the predetermined value (e.g., 80 dB) or more while the wearable electronic apparatus 100 is operated in the normal operation mode. In addition, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 to return to the normal operation mode if the noise level is identified as having the value less than the predetermined value (e.g., 80 dB) while the wearable electronic apparatus 100 is operated in the ANC mode.
In an embodiment, the operation mode change function may be implemented without the operation of the external voice identification module 1000. In this case, the noise level identification module 3000 may identify the noise level of the other frame except the frame identified as including the user's voice by the user voice identification module 2000. In addition, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 based on the noise level obtained by the noise level identification module 3000 and the dialog situation obtained by the user voice identification module 2000. In detail, the operation mode determination module 5000 may control the operation mode of the wearable electronic apparatus 100 as shown in Tables 1, 2 and 3.
Table 1 is a table showing a method to control the operation mode in case that two operation modes, i.e., ANC mode and AMBIENT mode, are implemented according to an embodiment. Referring to Table 1, situation nos. 1, 2, 3 and 4 may each indicate a case of an operation in the ANC mode (ANC status=1), and situation nos. 5, 6, 7 and 8 may each indicate a case of no operation in the ANC mode (ANC status=0), i.e. operation in the AMBIENT mode.
In addition, a case where “dialog situation=1” may indicate the dialog situation, and a case where “dialog situation=0” may indicate no dialog situation, which are identified by the dialog situation identification module 4000.
In addition, a case where “noise level=1” may indicate a case where the noise level has the predetermined value (e.g., 80 dB) or more, and a case where “noise level=0” may indicate a case where the noise level has the value less than the predetermined value (e.g., 80 dB), which are identified by the noise level identification module 3000.
Referring to Table 1, situation no. 1 may indicate a case where the dialog situation is detected during the operation in the ANC mode. In this case, the operation mode determination module 5000 may change the operation mode from the ANC mode to the AMBIENT mode, and may then control the operation mode to return to the ANC mode again if the dialog situation is detected as being over.
In addition, situation no. 5 may indicate a case where the noise level is identified as having the predetermined value or more and the dialog situation is detected during the operation in the AMBIENT mode. In this case, the operation mode determination module 5000 may control the operation mode to be maintained in the AMBIENT mode.
In addition, situation no. 7 may indicate a case where the noise level is identified as having the predetermined value or more during the operation in the AMBIENT mode. In this case, the operation mode determination module 5000 may change the operation mode from the AMBIENT mode to the ANC mode, and may then control the operation mode to return to the AMBIENT mode again if the noise level is identified as having a value less than the predetermined value.
Table 2 is a table showing a method to control the operation mode in case that three operation modes, i.e., ANC mode, AMBIENT mode and Noise Focusing mode, are implemented according to an embodiment. Referring to Table 2, situation no. 1 may indicate a case where the dialog situation is detected during the operation in the ANC mode, and the noise level is identified as having the predetermined value or more. In this case, the operation mode determination module 5000 may change the operation mode from the ANC mode to the Noise Focusing mode, and may then control the operation mode to return to the ANC mode again if the dialog situation is detected as being over.
Referring to Table 2, situation no. 5 may indicate a case where the noise level is identified as having the predetermined value or more and the dialog situation is detected during the operation in the AMBIENT mode. In this case, the operation mode determination module 5000 may control the operation mode to be changed from the AMBIENT mode to the Noise Focusing mode and may then control the operation mode to return to the AMBIENT mode again if the dialog situation is detected as being over. However, the operation mode determination module 5000 is not limited thereto. The operation mode determination module 5000 may control the operation mode to be changed from the Noise Focusing mode to the ANC mode if the dialog situation is detected as being over, and the noise level is continuously identified as having the predetermined value or more, after the operation mode is changed from the AMBIENT mode to the Noise Focusing mode.
Table 3 is a table showing a method to control the operation mode in case that three operation modes, i.e., ANC mode, AMBIENT mode and normal operation mode, are implemented according to an embodiment. Referring to Table 3, situation no. 5 may indicate a case where the noise level is identified as having the predetermined value or more and the dialog situation is detected during the operation in the AMBIENT mode or in the normal operation mode. In this case, the operation mode determination module 5000 may control the operation mode to be changed from the AMBIENT mode or the normal operation mode to the ANC mode. In addition, the operation mode determination module 5000 may control the operation mode to be changed from the AMBIENT mode to the ANC mode if the dialog situation is detected as being over, and the noise level is continuously identified as having the predetermined value or more.
According to the various embodiments described above, the wearable electronic apparatus 100 may have the changed operation mode based on the user's dialog situation and the noise level.
In an embodiment, the wearable electronic apparatus 100 may pre-process the bone conduction signal received by the IMU sensor 120 and convert the signal into a signal in a 2 kHz band (operation 300). For example, the wearable electronic apparatus 100 may convert the bone conduction signal in a 16 kHz band into the signal in the 2 kHz band by using a band-pass filter (BPF) and a sampling rate conversion (SRC). In addition, the wearable electronic apparatus 100 may identify the probability whether the user's voice exists in each frame of the signal based on the signal in the 2 kHz band converted using the WDS (operation 302). In an embodiment, the wearable electronic apparatus 100 may identify the probability whether the voice exists in each frame unit having the predetermined interval (e.g., frame interval of 10 ms) among the plurality of frames of the signals in the 2 kHz band.
In an embodiment, the user voice identification module 2000 of
In addition, the wearable electronic apparatus 100 may extract a parameter for detecting the humming by performing the signal feature extraction on the converted 2 kHz band signal (operation 304). That is, the wearable electronic apparatus 100 may identify whether the user's voice is the humming by analyzing the energy of the signal in the 2 kHz band, as described in
In addition, the wearable electronic apparatus 100 may identify whether the dialog situation exists in each frame of the signal (dialog detection) based on the probability whether the user's voice exists in each frame of the signal obtained by the WSD and the information on whether the user's voice obtained by the signal feature extraction is the humming (operation 306). For example, the wearable electronic apparatus 100 may identify the other frame except the frame identified as that of the humming situation as the frame corresponding to the dialog situation among the frames of the signals, having the probability that the user's voice exists therein based on the predetermined value (e.g., 0.7) or more, using the signal feature extraction. For example, the operation mode determination module 5000 of
In an embodiment, the wearable electronic apparatus 100 may pre-process the audio signal received by the microphone 110 and convert the signal into a signal in a 4 kHz band (operation 400). For example, the wearable electronic apparatus 100 may convert the audio signal in a 16 kHz band into the signal in the 4 kHz band by using a low-pass filter (LPF) and the SRC.
In addition, the wearable electronic apparatus 100 may identify whether the external voice exists in each frame of the signal, based on the signal in the 4 kHz band converted using the VAD (operation 402). In an embodiment, the wearable electronic apparatus 100 may identify whether the external voice exists in each frame having the predetermined interval (e.g., frame interval of 10 ms) among the plurality of frames of the signals in the 4 kHz band. In an embodiment, the external voice identification module 1000 of
In addition, the wearable electronic apparatus 100 may identify the noise level of each frame (operation 404) based on the information on whether the external voice exists in each frame obtained by the VAD, the converted 4 kHz band signal, and the probability whether the user's voice exists in each frame of signal obtained by the WSD described in
For example, the wearable electronic apparatus 100 may identify the noise level of the other frame except the frame identified as including the user's voice by the WSD among the plurality of frames based on the signal in the 4 kHz band.
For example, the wearable electronic apparatus 100 may identify the noise level of the other frame except: the frame identified as including the user's voice obtained by the WSD; and the frame identified as including the external voice obtained by the VAD, among the plurality of frames based on the signal in the 4 kHz band. In an embodiment, the noise level identification module 3000 of
As described in
In detail, in operation 500, the wearable electronic apparatus 100 may determine whether to change its operation mode (switching decision) using: the information on whether the dialog situation exists in each frame of the signal based on the dialog detection using the IMU sensor, as described in
In addition, the operation mode may be controlled by the user of the wearable electronic apparatus 100. For example, the user of the wearable electronic apparatus 100 may change the operation mode using an operation mode control application installed in the wearable electronic apparatus 100.
In addition, the wearable electronic apparatus 100 may identify a result of the switching decision (operation 502) and whether the operation mode is changed in the operation mode control application, and may generate a signal to cancel the noise included in the audio signal received by the external microphone and the internal microphone based on the operation mode and provide the generated signal to the speaker.
For example, in case that music is played through a music player application while being operated in the ANC mode, the wearable electronic apparatus 100 may control the signal to cancel the audio signal received by the external microphone to be generated, and may control the generated signal to be output to the speaker together with the music.
In an embodiment, the wearable electronic apparatus 100 may determine its operation mode using the external microphone and the internal microphone.
In an embodiment, the wearable electronic apparatus 100 may identify whether the dialog situation exists in each frame of the signal by using the internal microphone without using the IMU sensor. In detail, the wearable electronic apparatus 100 may convert music audio data in a 48 kHz band received by the internal microphone into the signal in the 4 kHz band using the SRC (operation 600). In addition, the wearable electronic apparatus 100 may convert audio data in a 16 kHz band (e.g., users voice) except for the music audio data received by the internal microphone into the signal in the 4 kHz band using the SRC (operation 602). In addition, the wearable electronic apparatus 100 may remove an echo from the converted 4 kHz band signal by using acoustic echo canceling (AEC) (operation 604), and may identify the probability whether the user's voice exists in each signal frame of the signal in the 4 kHz band from which the echo is removed by using the WSD (operation 302). In an embodiment, the wearable electronic apparatus 100 may identify the probability whether the voice exists in each frame unit having the predetermined interval (e.g., frame interval of 10 ms) among the plurality of frames of the signals in the 4 kHz band.
In addition, the wearable electronic apparatus 100 may extract the parameter for detecting the humming by performing the signal feature extraction on the signal in the 4 kHz band from which the echo is removed (operation 304). That is, the wearable electronic apparatus 100 may identify whether the user's voice is the humming by analyzing the energy of the signal in the 4 kHz band, from which the echo is removed.
In addition, the wearable electronic apparatus 100 may identify the noise level of each frame (noise level calculation) for the other frame except the frame identified as including the user's voice based on a result of the WDS obtained using the internal microphone, among the plurality of frames of signals received by the external microphone (operation 404).
In addition, the wearable electronic apparatus 100 may identify whether the dialog situation exists in each frame of the signal (dialog detection) based on the probability whether the user's voice exists in each frame of the signal obtained by the WSD and the information on whether the user's voice obtained by the signal feature extraction is the humming (operation 306).
For example, the wearable electronic apparatus 100 may identify the other frame except the frame identified as that of the humming situation as the frame corresponding to the dialog situation among the frames of the signals, having the probability that the user's voice exists therein based on the predetermined value (e.g., 0.7) or more, using the signal feature extraction.
In addition, in an embodiment, the wearable electronic apparatus 100 may identify whether the dialog situation exists in each frame of the signal by further using a result of the noise level calculation of operation 404. That is, if a large noise exists in the frame of the signal, the result of the noise level calculation may be further used to correct a result of the dialog detection.
In addition, in operation 500, the wearable electronic apparatus 100 may determine whether to change the operation mode of the wearable electronic apparatus 100 (switching decision) by using: the information on whether the dialog situation exists in each frame of the signal based on the dialog detection using the internal microphone; and the noise level of each frame of the signal based on the noise level calculation using the external microphone and the internal microphone. In detail, the wearable electronic apparatus 100 may determine whether to change its operation mode in the same manner as shown in Tables 1, 2 and 3.
In addition, the operation mode may be controlled by the user of the wearable electronic apparatus 100. For example, the user of the wearable electronic apparatus 100 may change the operation mode using the operation mode control application installed in the wearable electronic apparatus 100.
In addition, in operation 502, the wearable electronic apparatus 100 may identify a result of the switching decision and whether the operation mode is changed in the operation mode control application, and may generate a signal to cancel the noise included in the audio signal received by the external microphone and the internal microphone based on the operation mode and provide the generated signal to the speaker.
For example, while being operated in the ANC mode, if music is played through a music player application, the wearable electronic apparatus 100 may control the signal to cancel the audio signal received by the external microphone to be generated, and may control the generated signal to be output to the speaker together with the music.
First, the wearable electronic apparatus 100 may identify whether the voice is included in the current frame (operation S705). In detail, the wearable electronic apparatus 100 may identify whether the voice is included in the frame of the signal obtained by the microphone 110 and the IMU sensor 120. In an embodiment, the wearable electronic apparatus 100 may identify whether the voice is included in each frame having the predetermined interval (e.g., frame interval of 10 ms) among the plurality of the frames of the signals. In addition, if the voice is included in the current frame, the wearable electronic apparatus 100 may identify that the frame is the frame corresponding to the voice (speaking=1). On the other hand, if no voice is included in the current frame, the wearable electronic apparatus 100 may identify that the frame is not the frame corresponding to the voice (speaking=0).
In addition, the wearable electronic apparatus 100 may identify whether a prior frame is the frame of the dialog situation (operation S710). In detail, the wearable electronic apparatus 100 may identify that whether the prior frame is the frame of the dialog situation (dialog_detection_old=?) based on the result obtained by the dialog situation identification module 4000 of
If the prior frame is identified as a frame of no dialog situation (dialog_detection_old=0), the wearable electronic apparatus 100 may identify whether the current frame is the frame of the dialog situation based on the WDS (dialog detection) (operation S715).
If it is identified that the current frame is not the dialog situation (operation S715-N), the wearable electronic apparatus 100 may identify the current situation as no dialog situation (dialog detection=0) (operation S720). In addition, the wearable electronic apparatus 100 may identify the region of the current frame as no dialog region (dialog=0) (operation S750).
In addition, if the current frame is identified as the frame of the dialog situation (operation S715-Y), the wearable electronic apparatus 100 may identify whether the region of the current frame is a humming region (operation S725).
In addition, if it is identified that the region of the current frame is the humming region (operation S725-Y), the wearable electronic apparatus 100 may identify the current situation as the humming situation (dialog detection=−1) (operation S730). In addition, the wearable electronic apparatus 100 may identify the region of the current frame as no dialog region (dialog=0) (operation S750).
In addition, if it is identified the region of the current frame as no humming region (operation S725-N), the wearable electronic apparatus 100 may identify the current situation as the dialog situation (dialog detection=1) (operation S735). In addition, the wearable electronic apparatus 100 may identify the region of the current frame as a dialog region (dialog=1) (operation S755).
If it is identified that the prior frame is the frame of the humming situation (dialog_detection_old=−1) in S710, the wearable electronic apparatus 100 may identify whether the current frame includes no voice and the prior frame includes the voice (operation S740).
If it is identified that the current frame includes no voice and the previous frame includes the voice (operation S740-Y), the wearable electronic apparatus 100 may identify the current situation as no dialog situation (dialog detection=0) (operation S720). In addition, if it is identified that the current frame includes the voice or the prior frame includes no voice (operation S740-N), the wearable electronic apparatus 100 may identify that the current situation is the humming situation (dialog detection=−1) (operation S730).
If the prior frame is identified as the dialog situation (dialog_detection_old=1) in S710, the wearable electronic apparatus 100 may identify whether the current frame includes the voice (speaking=1) or whether the predetermined time (e.g., 5 seconds) is not elapsed from the dialog start point (operation S745).
That is, if the current frame includes the voice (speaking=1) (operation S745-Y), the wearable electronic apparatus 100 may identify that the current situation is the dialog situation (dialog detection=1) (operation S735). In addition, if the predetermined time (e.g., 5 seconds) is not elapsed from the dialog start point (operation S745-Y), the wearable electronic apparatus 100 may identify that the current situation is the dialog situation (dialog detection=1) (operation S735). In addition, the wearable electronic apparatus 100 may identify the region of the current frame as the dialog region (dialog=1) (operation S755).
On the other hand, if the current frame includes no voice (speaking=0), and the predetermined time (e.g., 5 seconds) is elapsed from the dialog start point (operation S745-N), the wearable electronic apparatus 100 may identify the current situation as no dialog situation (dialog detection=0)(operation S720). In addition, the wearable electronic apparatus 100 may identify the region of the current frame as no dialog region (dialog=0) (operation S750).
In addition, the wearable electronic apparatus 100 may update the result data of operations S750 and S755 (operation S760). That is, the wearable electronic apparatus 100 may update whether the current frame includes the dialog situation (dialog_detection) to whether the prior frame includes the dialog situation (dialog_detection_old), and whether the current frame includes the voice (speaking) to whether the prior frame includes the voice (speaking).
In addition, the wearable electronic apparatus 100 may continuously repeat operations S705 through S760.
In an embodiment, the wearable electronic apparatus 10) may control the ANC mode to be turned on or off based on whether the current situation is the dialog situation and based on the noise level of the current frame.
In detail, it is possible to identify whether the wearable electronic apparatus 100 is currently operated in the ANC mode (ANC_status_user==on?) (operation S805). For example, the user of the wearable electronic apparatus 100 may identify whether its current operation mode is determined as the ANC mode.
If the wearable electronic apparatus 100 is currently operated in the ANC mode (operation S805-Y), the wearable electronic apparatus 100 may identify whether the current situation is the dialog situation (dialog_detect_status>0 ?) (operation S810). For example, as described above in
If it is identified that the current situation is the dialog situation (dialog_detect_status=1) (operation S810-Y), the wearable electronic apparatus 100 may control the ANC mode to be turned off (ANC_status_auto=off) (operation S815). For example, the wearable electronic apparatus 100 may control its operation mode to be the normal operation mode. However, the wearable electronic apparatus 100 is not limited thereto, and may control its operation mode to be the AMBIENT mode or the Noise Focusing mode.
If it is identified that the current situation is not the dialog situation (dialog_detect_status=0 or −1), the wearable electronic apparatus 100 may control the ANC mode to be maintained (ANC_status_auto=on) (operation S820). For example, if it is identified that the current situation as no dialog situation (dialog_detect_status=0) or the humming situation (dialog_detect_status=−1), the wearable electronic apparatus 100 may control the ANC mode to be maintained.
In addition, if the wearable electronic apparatus 100 is not currently operated in the ANC mode (operation S805-N), the wearable electronic apparatus 100 may identify whether the current situation is the dialog situation (dialog_detect_status>0 ?) (operation S825).
If it is identified that the current situation is the dialog situation (dialog_detect_status=1) (operation S825-Y), the wearable electronic apparatus 100 may control its operation mode to be maintained as the operation mode in which the wearable electronic apparatus 100 is currently operated (ANC_status_auto=off) (operation S830). That is, the wearable electronic apparatus 100 may control its current operation mode different from the ANC mode to be continuously maintained.
If it is identified that the current situation is not the dialog situation (dialog_detect_status=0 or −1), the wearable electronic apparatus 100 may identify whether the noise level has the predetermined value or more (operation S835). For example, the wearable electronic apparatus 100 may identify whether the noise level of the frame of the signal received by the external microphone is 80 dB or more.
If the noise level has a value less than the predetermined value (operation S835-N), the wearable electronic apparatus 100 may control its current operation mode to be maintained (ANC_status_auto=off) (operation S830). That is, the wearable electronic apparatus 100 may control its current operation mode different from the ANC mode to be continuously maintained.
If the noise level has the predetermined value or more (operation S835-Y), the wearable electronic apparatus 100 may control its operation mode to be the ANC mode (ANC_status_auto=on) (operation S840).
First, a wearable electronic apparatus 100 may be operated in an ANC mode (operation S910). For example, a user of the wearable electronic apparatus 100 may determine its operation mode as the ANC mode. However, the wearable electronic apparatus 100 is not limited thereto, and its current operation mode may be determined as the ANC mode according to an embodiment of
In addition, the wearable electronic apparatus 100 may receive a bone conduction signal corresponding to vibration generated in the user's face by an IMU sensor while the wearable electronic apparatus is operated in the ANC mode (operation S920).
In addition, the wearable electronic apparatus 100 may identify the user's voice based on the bone conduction signal (operation S930).
In an embodiment, the wearable electronic apparatus 100 may receive the bone conduction signal by the IMU sensor, and may identify a probability whether the user's voice exists in a frame unit of the bone conduction signal, the frame unit having a predetermined interval, e.g., a predetermined duration. In addition, the wearable electronic apparatus 100 may identify that the identified frame unit is the frame in which the voice exists if it is identified that the probability indicating whether the user's voice exists in the frame unit having the predetermined interval has a predetermined value (e.g., 0.7) or more.
In addition, the wearable electronic apparatus 100 may control the operation mode of the wearable electronic apparatus 100 to be a different operation mode from the ANC mode if the user's voice is identified (operation S940). On the other hand, the wearable electronic apparatus 100 may control the ANC mode to be maintained if the user's voice is not identified.
In an embodiment, the different operation mode may include a normal operation mode in which an external noise is output as it is, an AMBIENT mode in which the external noise is emphasized, and a Noise Focusing mode in which an external voice is emphasized. However, an embodiment is not limited thereto, and may further include various operation modes.
In an embodiment, the wearable electronic apparatus 100 may identify whether a current frame is the frame of a humming situation if it is identified that the current frame is the frame in which the voice exists. In addition, the wearable electronic apparatus 100 may control the operation mode of the wearable electronic apparatus 100 to be the operation mode different from the ANC mode if it is identified that the current frame is not the frame of the humming situation as a result of the identification. On the other hand, the wearable electronic apparatus 100 may control the operation mode of the wearable electronic apparatus 100 to be maintained as the ANC mode if it is identified that the current frame is the frame of the humming situation as a result of the identification.
In an embodiment, the wearable electronic apparatus 100 may identify a noise level of the current frame by a microphone if it is identified that the current frame is the frame in which the voice exists. In addition, the wearable electronic apparatus 100 may control the operation mode of the wearable electronic apparatus 100 to be the Noise Focusing mode if the identified noise level has a predetermined value or more as a result of the identification. On the other hand, the wearable electronic apparatus 100 may control the operation mode of the wearable electronic apparatus 100 to be the AMBIENT mode if the identified noise level has a value less than the predetermined value.
In addition, while being operated in the operation mode different from the ANC mode, the wearable electronic apparatus 100 may identify that the user's voice is not identified for a predetermined time based on the bone conduction signal (operation S950). Here, the predetermined time may be five seconds, but is not limited thereto, and may be determined or changed by the user or manufacturer of the wearable electronic apparatus 100 such as three seconds, seven seconds, nine seconds, etc.
In addition, the wearable electronic apparatus 100 may control the operation mode to return to the ANC mode if it is identified that the user's voice is not identified for the predetermined time (operation S960). On the other hand, the wearable electronic apparatus 100 may control the different operation mode to be maintained if the user's voice is identified within the predetermined time.
Referring to
The external microphone 112 may be implemented as the external microphone disposed on the wearable electronic apparatus 100 to be positioned outside the user's ear like the external microphone of
According to an embodiment, the processor 150 may control the wearable electronic apparatus 100 to be operated in the ANC mode. For example, the user of the wearable electronic apparatus 100 may determine its operation mode as the ANC mode. However, the wearable electronic apparatus 100 is not limited thereto, and its current operation mode may be determined as the ANC mode according to an embodiment of
In addition, while being operated in the ANC mode, the processor 150 may control the IMU sensor 120 to receive a bone conduction signal corresponding to vibration generated in the user's face. In addition, the processor 150 may identify the user's voice based on the bone conduction signal.
In an embodiment, the processor 150 may control the IMU sensor 120 to receive the bone conduction signal, and may identify the probability whether the user's voice exists in the frame unit of the bone conduction signal, the frame having the predetermined interval. In addition, the processor 150 may identify that the identified frame unit is the frame in which the voice exists if it is identified that the probability indicating whether the user's voice exists in the frame unit having the predetermined interval has the predetermined value (e.g., 0.7) or more.
In addition, the processor 150 may control the operation mode of the wearable electronic apparatus 100 to be an operation mode different from the ANC mode if the user's voice is identified. On the other hand, the processor 150 may control the ANC mode to be maintained if the user's voice is not identified.
In an embodiment, the different operation mode may include the normal operation mode in which the external noise is output as it is, the AMBIENT mode in which the external noise is emphasized, and the Noise Focusing mode in which the external voice is emphasized. However, an embodiment is not limited thereto, and may further include various operation modes.
In an embodiment, the processor 150 may identify whether the current frame is the frame of the humming situation if it is identified that the current frame is the frame in which the voice exists. In addition, the processor 150 may control the operation mode of the wearable electronic apparatus 100 to be the operation mode different from the ANC mode if it is identified that the current frame is not the frame of the humming situation as a result of the identification. On the other hand, the processor 150 may control the operation mode of the wearable electronic apparatus 100 to be maintained as the ANC mode if it is identified that the current frame is the frame of the humming situation as a result of the identification.
In an embodiment, the processor 150 may identify the noise level of the current frame by the external microphone 112 if it is identified that the current frame is the frame in which the voice exists. In addition, the processor 150 may control the operation mode of the wearable electronic apparatus 100 to be the Noise Focusing mode if the identified noise level has the predetermined value or more as a result of the identification. On the other hand, the processor 150 may control the operation mode of the wearable electronic apparatus 100 to be the AMBIENT mode if the identified noise level has a value less than the predetermined value.
In addition, while the wearable electronic apparatus 100 is operated in the operation mode different from the ANC mode, the processor 150 may identify that the user's voice is not identified for the predetermined time based on the bone conduction signal. Here, the predetermined time may be five seconds, but is not limited thereto, and may be determined or changed by the user or manufacturer of the wearable electronic apparatus 100 such as three seconds, seven seconds, nine seconds, etc.
In addition, the processor 150 may control the operation mode of the wearable electronic apparatus 100 to return to the ANC mode, if it is identified that the user's voice is not identified for the predetermined time. On the other hand, the processor 150 may control the different operation mode of the wearable electronic apparatus 100 to be maintained if the user's voice is identified within the predetermined time.
The internal microphone 160 may be disposed inside the user's ear, as described in
The communication interface 170 is a configured to perform communication with the external apparatus. Meanwhile, the communicative connection of communication interface 170 and the external apparatus may include the communication therebetween performed via a third device (e.g., repeater, hub, access point, server or gateway). According to an embodiment, wireless communication may include at least one of, for example, wireless fidelity (Wi-Fi), Bluetooth, Bluetooth low energy (BLE), ZigBee, near field communication (NFC), magnetic secure transmission, radio frequency (RF) or a body area network (BAN).
In particular, the communication interface 170 may receive the audio data provided from an external electronic apparatus by performing communication with the external electronic apparatus. In addition, the processor 150 may control the speaker 130 to output the audio data.
Embodiments described herein may be variously modified and/or combined, and certain embodiments are thus illustrated in the drawings and described. However, it is to be understood that technologies mentioned in herein are not limiting, but include various modifications, equivalents, and/or alternatives. Throughout the accompanying drawings, similar components may be denoted by similar reference numerals.
In addition, embodiments described above may be modified in several different forms, and the scope and spirit of the disclosure are not limited to the embodiments. Rather, embodiments are provided to transfer the spirit of the disclosure to those skilled in the art.
Terms used herein are to describe the specific embodiments rather than limiting the scope of the disclosure. Singular forms used herein are intended to include plural forms unless explicitly indicated otherwise.
In an embodiment, an expression ‘have’, ‘may have’, ‘include’, ‘may include’ or the like, indicates existence of a corresponding feature (for example, a numerical value, a function, an operation, a component such as a part or the like), and does not exclude existence of an additional feature.
As used herein, an expression “A or B”, “least one of A and/or B” or “one or more of A and/or B” or the like, may include all possible combinations of items enumerated together. For example, “A or B”, “least one of A and B,” or “at least one of A or B” may indicate all of 1) a case in which at least one A is included, 2) a case in which at least one B is included, or 3) a case in which both of at least one A and at least one B are included.
As used herein, the terms such as “1st” or “first,” “2nd” or “second,” etc., may modify corresponding components regardless of importance or order and are used to distinguish one component from another without limiting the components. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. Therefore, the meanings of the elements are not limited by the terms, and the terms are also used just for explaining the corresponding embodiment.
If it is mentioned that any component (for example, a first component) is (“operatively or communicatively”) coupled with/to or is connected to another component (for example, a second component), it is to be understood that any component is directly coupled to another component or may be coupled to another component through the other component (for example, a third component).
On the other hand, if it is mentioned that any component (for example, a first component) is “directly coupled” or “directly connected” to another component (for example, a second component), it is to be understood that the other component (for example, a third component) is not present between any component and another component.
An expression “configured (or set) to” used in an embodiment may be replaced by an expression “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to” or “capable of” based on a situation. A term “configured (or set) to” may not necessarily mean “specifically designed to” in hardware.
Instead, an expression “an apparatus configured to” may mean that the apparatus may “perform-” together with other apparatuses or components. For example, “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (for example, an embedded processor) for performing the corresponding operations or a generic-purpose processor (for example, a central processing unit (CPU) or an application processor) that may perform the corresponding operations by executing one or more software programs stored in a memory apparatus.
In embodiments, a ‘module’ or a ‘˜er/or’ may perform at least one function or operation, and be implemented by hardware or software or be implemented by a combination of hardware and software. In addition, a plurality of ‘modules’ or a plurality of ‘˜ers/ors’ may be integrated in at least one module and be implemented by at least one processor except for a ‘module’ or an ‘˜er/or’ that needs to be implemented by specific hardware.
Meanwhile, various elements and regions in the drawings are schematically illustrated. Therefore, the spirit of the disclosure is not limited by relative sizes or intervals illustrated in the accompanying drawings.
Meanwhile, embodiments described herein may be implemented in a computer or a computer readable recording medium using software, hardware, or a combination of software and hardware. According to a hardware implementation, the embodiments described in an embodiment may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors or electric units for performing other functions. In some cases, an embodiment may be implemented by the processor itself. According to a software implementation, embodiment may be implemented by separate software modules. Each of the software modules may perform one or more functions and operations described in an embodiment.
Embodiments may be implemented as software containing one or more instructions that are stored in machine-readable (e.g., computer-readable) storage medium (e.g., internal memory or external memory). A processor may call instructions from a storage medium and is operable in accordance with the called instructions. When the instruction is executed by a processor, the processor may perform the function corresponding to the instruction, either directly or under the control of the processor, using other components. The instructions may contain a code made by a compiler or a code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium.
The non-transitory readable medium is not a medium that temporarily stores data therein, such as a register, a cache, a memory or the like, and indicates a medium that semi-permanently stores data therein and is readable by an apparatus. In detail, programs for performing the various methods described above may be stored and provided in the non-transitory readable medium such as a compact disc (CD), a digital versatile disc (DVD), a hard disc, a Blu-ray disc, a universal serial bus (USB), a memory card, a read only memory (ROM) or the like.
According to an embodiment, the methods may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a purchaser. The computer program product may be distributed in the form of a storage medium (for example, a compact disc read only memory (CD-ROM)) that may be read by the machine or online through an application store (for example, PlayStore™). In case of the online distribution, at least portions of the computer program product may be at least temporarily stored in a storage medium such as a memory of a server of a manufacturer, a server of an application store or a relay server, or be temporarily generated.
While certain embodiments have been particularly shown and described with reference to the drawings, embodiments are provided for the purposes of illustration and it will be understood by one of ordinary skill in the art that various modifications and equivalent other embodiments may be made from the disclosure. Accordingly, the true technical scope of the disclosure is defined by the technical spirit of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2021-0014234 | Feb 2021 | KR | national |
This application is a bypass continuation of International Application No. PCT/KR2021/015726, filed on Nov. 2, 2021, which is based on and claims priority to Korean Patent Application No. 10-2021-0014234, filed on Feb. 1, 2021, in the Korean Intellectual Property Office, the disclosures of which are incorporated by reference herein in their entireties.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9055366 | Azmi et al. | Jun 2015 | B2 |
| 10397687 | Watts et al. | Aug 2019 | B2 |
| 10564925 | An et al. | Feb 2020 | B2 |
| 10757501 | Mehra et al. | Aug 2020 | B2 |
| 11308977 | Moon et al. | Apr 2022 | B2 |
| 11317188 | Mehra et al. | Apr 2022 | B2 |
| 20070088544 | Acero | Apr 2007 | A1 |
| 20140205131 | Azmi et al. | Jul 2014 | A1 |
| 20170193978 | Goldman | Jul 2017 | A1 |
| 20190342647 | Mehra et al. | Nov 2019 | A1 |
| 20200152225 | Numata et al. | May 2020 | A1 |
| 20200219525 | Moon et al. | Jul 2020 | A1 |
| 20200258539 | Shim et al. | Aug 2020 | A1 |
| 20200389716 | Mehra et al. | Dec 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 2010019876 | Jan 2010 | JP |
| 2020076923 | May 2020 | JP |
| 10-2013-0036867 | Apr 2013 | KR |
| 101631192 | Jun 2016 | KR |
| 10-2015-0108907 | Sep 2017 | KR |
| 101898911 | Oct 2018 | KR |
| 10-2020-0085030 | Jul 2020 | KR |
| 10-2020-0098323 | Aug 2020 | KR |
| 10-2021-0005168 | Jan 2021 | KR |
| Entry |
|---|
| Written Opinion (PCT/ISA/237) dated Feb. 21, 2022 issued by the International searching Authority for International Application No. PCT/KR2021/015726. |
| Search Report (PCT/ISA/210) dated Feb. 21, 2022 issued by the International searching Authority for International Application No. PCT/KR2021/015726. |
| Number | Date | Country | |
|---|---|---|---|
| 20220246129 A1 | Aug 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/KR2021/015726 | Nov 2021 | US |
| Child | 17578164 | US |
