Patent Application
20240427013
The present invention is based on the detection of a presence of a user or object, including the movements and gestures of the user.
Detecting the presence of a user to an electronic device is well known, e.g. as disclosed in U.S. Pat. No. 9,678,559 where a number of different sensors, including acoustic, vibration and visual sensors, are suggested to for detecting a user presence, where machine learning is used for recognizing activity close to the device. The solution is, however, vulnerable to noise and unexpected situations. In EP 2389598 a system is described where mobile phone or similar, possibly carried by a user, is configured to transmit a signal which may be detected. The described solution uses signals directly from the mobile phone and reflections in the room to detect the presence and position of the mobile phone. The monitored zone thus includes a number of passive acoustic receivers connected to the installation.
Active detection is also well known, for example including an ultrasound probe signal is sent out and when the ultrasound echos from objects in the detection zone are picked up by ultrasound receivers in the device, the signals are compared and analyzed to determine whether someone is present or not. This way the user is not required to carry a transmitter. There needs to be at least one ultrasound receiver and at least one ultrasound transmitter for this scheme to work. The ultrasound receiver and transmitter are in some cases combined in the same component capable of both transmitting and receiving the ultrasound signal at the same time. Such active presence detection is discussed at length in WO2021/045628 and NO20210304 as well as in simultaneously filed patent applications Ser. No. 20/211,332 and NO20211333 which are incorporated here by way of reference. The devices may be any electronic device such as computers, phones, TVs or tablets having a user interface that should be responsive to a user within an area close to the device.
The main issues for presence detection sensors based on ultrasound are interference, noisy environments, power consumption and intermodulation effects from playing audio concurrently on the same transmitting device. Another problem is related to the power consumption of the device using an active transmitter and receivers.
The present invention relates to an innovative approach to for achieving a presence detection while maintaining a low power consumption. This is obtained as specified in the accompanying claims.
The present invention is thus based on the concept using any audible or inaudible sound (e.g. infrasound, ultrasound) in the environment including output from any device that play out sound explicitly (e.g. radio, TV, smart speaker, hifi-system, smartphone, etc) or any object generating sound just by operating (e.g. coffee grinder, PC fan, dishwasher, etc). It is also possible to use sounds from living things (e.g. dogs, people, cats, etc) too including vocal sounds they emit or the noise they make moving around (e.g. creaking floor, bumping things, breaking things, playing, etc). The idea is that the presence detection device should use any detectable acoustic signal close to or within the detection zone of the presence detection sensor that can be picked up by at least one acoustic receiver.
The invention will be described more in detail below, referring to the enclosed drawing illustrating the invention by way of examples.
As is illustrated in
The present invention is based on the presence of at least one acoustic source 13 in the vicinity of the device, for example a radio 13 with a speaker 2 emitting a signal. The signal from the speaker will propagate through the space and be reflected by any person or object 7 in the area, and part of the reflection being received by the microphones. Even if there are other objects in the room the sound patter received by the microphones will change at the entrance of a user representing a change in the patterns and the analysis may define the change as a probable presence of a user 7. By comparing the previously measured patterns with the new pattern it is also possible to calculate the direction to the user or object. As part of the detection the processor may filter out certain frequency ranges, for example at high frequency and in the ultrasound range, where it is easier to distinguish features and objects in the room, and to calculate the presence in that range. The processing as such is related to room correction algorithms adjusting sound levels from measured controlled signals in a room.
The acoustic source may be of any kind generating a sound in the audible or ultrasound range. While a controlled signal may be preferable for accuracy the decision whether a person is present may allow more complex signals with less accuracy. According to one embodiment the signal is known or recognizable, for example music being recognized by the processor and making it possible to compare the recognized, stored music with the music detected by the microphones.
The acoustic receiver 1,9 could thus run all the time or be duty-cycled for power saving if necessary. The presence detection sensor could also use on-device, on-the-fly training to identify the primary user of the device through voice recognition. Identifying the main user to be within the detection zone could add additional information to the presence detection device. Due to the risk of spoofing, voice recognition should preferably not be used by itself to allow the user to enter the detection. If the detection zone is a security zone, using voice recognition to disable the presence detection alarm could be configurable by user or organization in charge of the protected space.
If the presence detection device is controlled by an access system, the voice recognition system should only do on-the-fly training to identify the main users of the device when the access system has not logged any intrusion event. Similarly, a HPD sensor on a laptop should only do on-the-fly training when the user is logged into the device when sitting in front of it talking while interacting with the device using at least one acceptable input method requiring proximity to the device (e.g. touch screen, keyboard, mouse, etc).
As mentioned above the presence device should preferably triangulate the position of available external sound sources continuously or in a duty-cycle manner if necessary. Alternatively, the presence detection device could use a probe signal (e.g. music, tone, speech, etc) emitted from a transmitter within its own device if available, or being coupled and synchronized with the device so as to reduce the necessary processing. In this case the transmitted signal from the speaker may be fed directly to the processor with a wired or wireless connection 11.
More in detail it is possible with the present invention to select the most suitable sound source from a set of available sound sources based on a set of relevant criteria such as amplitude, up-time, relative position, distance, identifiability, etc. The presence detection device should use these selected sound sources and the echos they create when bouncing off any user in the detection zone to identify if someone is present in the detection zone. The presence detection sensor will receive the direct path 14 signal from the sound sources 13 as well as the resulting echos 16 of the sound signals 15 reflected by the user 7 and use the received information 14,16 to find out if the user is present. For laptops with HPD sensors, the analysis of both the sound source echos bouncing off all parts of the user and the direct path sound from at least one sound source will find out if the user is in front of the laptop or not. The same principles apply to other presence detection devices with specified detection zones. The analysis will potentially include both advanced signal processing and serial or parallel execution of one or more neural network engines either executing in software or possibly accelerated in ML-optimized hardware engines.
As discussed above, triangulation of sound sources requires at least two acoustic receivers that are either separated to make triangulation possible if the physical layout of the receivers is known by the triangulation method. The frequency ranges discussed here is usually 0-384 KHz. Higher frequencies are also possible and will allow use of more bandwidth at the cost of memory, processing requirements and power consumption. There are other configurations of receivers that are ideal for triangulating sound sources, that is, receiver arrays. Since the distance between microphones in a microphone array is usually frequency dependent, the layout of the microphone array dictates the frequency range that can utilize the capabilities of a microphone array. Separating the acoustic receivers as much as possible is an advantage. Their position may also be important and should be selected with care. Adding additional acoustic receivers will in general improve the triangulation and thereby the presence detection solution.
If the device is in a space or room without any acoustic sound that can be utilized for presence detection in the target detection zone, the presence detection sensor may start using a probe signal either playing an audible signal (e.g. soothing music) or an ultrasound signal as discussed in WO2021/045628 and NO20210304, or by emitting audible sounds (e.g. music, video, audio, fan noise, etc), so that the presence detection solution can detect that and select an ideal part of the generated acoustic signal.
If the sound is emitted by at least one speaker in the device, the echo reference signal copy which is normally fed back into an echo cancellation module and could be forwarded to the presence detection module 10 providing information about the emitted signal. Comparing the echo reference signal to the received signal can be used to implement a presence detection solution based on echo analysis based on these to signals.
It should be noted that the phrase “presence detection” not only includes the verification of the user being present or not, but may also within the limitations of the available transducers, microphones and transmitted signals, also be configured to detect or recognize the nature of the detected object, e.g. by detecting the reflection characteristics compared to the transmitted sounds to detect whether or not the object is hard (possibly not a person), or soft, or by sampling a sequence measuring the speed or direction relative to the device. By analyzing the signal it may also be possible to recognize the posture of the user, e.g. leaning over the device or relaxing.
As mentioned above it may be advantageous to use a filter in the processing to select and use higher frequencies, preferably into the ultrasound range, to detect the presence, as it is well known that higher frequencies will increase the accuracy and resolution. Thus a detection system based on ultrasound or high frequency audible sound utilizing a set of high frequency transducers can be used to detect multiple objects close to the device.
For example, if an electronic device with at least one ultrasound output transducers sends out a broadband ultrasound signal (e.g. chirp, random modulation, frequency-stepped sines, etc), it can receive the ultrasound signal in at least one ultrasound input transducer and identify multiple objects in the targeted detection area. The different techniques to do this processing is known in the prior art as described in more detail in WO2017/137755, WO2009/122193, WO2009/115799 and WO2021/045628.
The resolution of the identified echos depends on bandwidth and frequency range of the signal. Higher sampling rates supported already by some consumer electronics (e.g. 96 KHz, 192 KHz, 384 KHz, etc) allows an increased signal bandwidth (e.g. more than 10 KHz) in a frequency range above the audible frequency range. With an increased signal frequency range and signal bandwidth, it is possible to identify multiple users (e.g. objects) and for each of them separate the different body parts such as fingers, hands, arms, head, torso, legs, etc. High frequency signals may also be affected by turbulence, e.g. cause by breathing, which means that variations in the received signal such as fluctuations in the signal phase, frequency and amplitude may be used to distinguish between a breathing person and an object.
In one embodiment of this invention, a laptop could send out a high-frequency, broadband signal to detect user presence. It could also detect user posture and breathing pattern while the user is sitting in front of the laptop whether he/she is interacting with it or not. The echo information could be combined with sensor data (e.g. hinge angle sensor, IMU sensor, light sensor, pressure sensor, ambient light sensor, etc) to provide more accurate information related to the detection. Identifying users peeking over the shoulder of the main laptop user is also possible with the increased resolution described here.
In another embodiment, a presence detection device could send out a high-frequency broadband signal to detect user presence. Since the resolution of the echos will be significantly higher and more details can be extracted, the presence detection device could monitor user movement and fed the data into an incremental, on-device ML-training process to create a continuously updated system such as deep neural network (DNN) that can be used to detect anomalies in user movement and gait.
To summarize the present invention is related to a presence detecting electronic device including at least one microphone configured to receive acoustic signals from the environment, and a processor connected to the microphone for analyzing the received signal. The device also includes means such as microphones and acoustic transducers and processors for directly detecting the transmitted signal from an acoustic source being positioned separate from the device and comparing the directly transmitted signal with the received signal for identifying if an object or user in the vicinity of the device based on the comparison between the directly transmitted signal and the signal reflected from the object or user. The device preferably includes at least two microphones, the processor being configured to detect the direction of the incoming signals, including the transmitted signal and the reflected signal.
According to one embodiment the acoustic source is independent of the device and the processor is configured to detect the direct acoustic signal as well as the first reflection and from the difference in time of arrival calculate whether a user is likely to be in the vicinity of the device.
The processor may be configured to detect the direction to the acoustic source and the direction of the reflected signal as well as the time difference between the signal arrivals at the detectors, and to analyze the likelihood of a user being present. Further analysis may be made from the received signals so as to analyse the nature of the assumed user, e.g reflectivity, hardness etc. The processor may also be configured to compare the time history of previous measurements to detect changes is the received signal possibly indicating the presence of a new object or user in the vicinity.
The processor may also be configured to filter the received acoustic signals at the microphones, the filter being a high pass filter selecting frequencies above a certain limit, such as 1 kHz, improving the accuracy of the distinction between the different acoustic propagation paths.
Instead of detecting the transmitted signals through the at least one microphone the processor may also be configured to directly receive the transmitted signal from a wireless or wired electronic connection for comparing with the received signal. In that case the wired or wireless connection is connected to the speaker or close to the speaker so as to compensate for any distortions caused by speaker protection processes or limitations in the signal processing and amplification.
The processor may be configured to recognize the transmitted acoustic signal by comparison with a databased including a set of samples, such as a streaming service, and to compare the received signals with the sample to analyze the received signal and detecting reflected signals in the received signals. In that case the processor may be configured to identify a user based on the distortions in the received signals compared with the original.
Depending on the processing capacity and microphone accuracy the processor may also be configured to analyze the characteristics of the received signals, such as amplitude and frequency range, so as to evaluate the size, position, posture or any gestures of the detected user.
The present invention also relates to a method for identifying a user in the vicinity of a device, after the transmission of an acoustic signal including the steps of:
The device preferably includes two microphones, and thus the method includes a step of calculating the direction of at the reflected acoustic signal and providing an indication if the position of the user is within a predefined area relative to the device. If the transmitted signal is not directly connected to the processor the first received signal is received at the microphone.
| Number | Date | Country | Kind |
|---|---|---|---|
| 20211335 | Nov 2021 | NO | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2022/080720 | 11/3/2022 | WO |
