This relates generally to the field of privacy protection, and more specifically to an apparatus for shaping audio masking signal.
Smartphones have sensors for collecting information of or about a user. For example, microphones on smartphones can be used to record a user's conversation. Often, smartphones also have radios for local or remote communications, e.g., a cellular radio, a WiFi radio, and/or a Bluetooth radio. Together, the sensors and radios can reveal a wealth of user information to third parties, e.g., the third parties can eavesdrop from a remote location with the help of the microphones and communication devices. Currently, smartphones are not capable of masking the recorded audio signals. Accordingly, smartphones are inadequate in user privacy protection.
So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description can be had by reference to aspects of some illustrative embodiments, some of which are shown in the accompanying drawings.
In accordance with common practice the various features illustrated in the drawings cannot be drawn to scale. Accordingly, the dimensions of the various features can be arbitrarily expanded or reduced for clarity. In addition, some of the drawings cannot depict all of the components of a given system, method or device. Finally, like reference numerals can be used to denote like features throughout the specification and figures.
Accordingly, described herein is an apparatus (also known as a smart case or a sound shaping apparatus) for providing a shaped audio masking signal to audio input devices on user equipment, e.g., to microphones on a personal communication device. In some embodiments, the apparatus includes an audio output device (e.g., a speaker) that outputs sound directed to an audio input device of the user equipment being protected by the apparatus. The audio output device is mated with the audio input device through an audio coupling interface (e.g., an audio seal). In some embodiments, the audio coupling interface provides a cavity for the audio output device and the audio input device, such that the physical barrier provided by the audio coupling interface attenuates sound in both directions. For example, in one direction, the ambient sound from outside the cavity is attenuated before reaching the microphone on the user equipment; and in the other direction, the masking signal from inside the cavity is attenuated in order to reduce the obtrusiveness of leaky masking signals. In some embodiments, to further reduce the obtrusiveness of the leaky masking signal, the apparatus includes a spectral shaper to apply a spectral envelope to an audio signal (e.g., the masking signal). By applying the spectral envelope, the spectral shaper produces a shaped audio signal to be selectively coupled to the audio output device. The shaped audio signal has characteristics that are less obtrusive to the surroundings. Thus, the apparatus disclosed herein reduces the obtrusiveness of the masking signal to the environment, while maintaining the effectiveness of the masking signal.
In accordance with some embodiments, the apparatus comprises an audio output device that is arranged to output sound directed to an audio input device of a second device. In some embodiments, the apparatus further includes an audio coupling interface arranged to provide a cavity for the audio output device and the audio input device of the second device. In some embodiments, the apparatus also includes a spectral shaper that is coupled to the audio output device, the spectral shaper being operable to apply a spectral envelope to an audio signal in order to produce a shaped audio signal, wherein the shaped audio signal is selectively coupled to the audio output device.
In accordance with some embodiments, a device includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing performance of the operations of any of the methods described herein. In accordance with some embodiments, a non-transitory computer readable storage medium has stored therein instructions which, when executed by one or more processors of a device, cause the device to perform or cause performance of the operations of any of the methods described herein. In accordance with some embodiments, a device includes means for performing or causing performance of the operations of any of the methods described herein.
Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
It will also be understood that, although the terms first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact, unless the context clearly indicates otherwise.
The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes”, “including”, “comprises”, and/or “comprising”, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As used herein, the term “if” is, optionally, construed to mean “when”, “upon”, “in response to determining”, or “in response to detecting”, depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining”, “in response to determining”, “upon detecting [the stated condition or event],” or “in response to detecting [the stated condition or event],” depending on the context.
It should be appreciated that in the development of any actual embodiments (as in any development project), numerous decisions must be made to achieve the developers' specific goals (e.g., compliance with system and business-related constraints), and that these goals will vary from one embodiment to another. It will also be appreciated that such development efforts might be complex and time consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art of image capture having the benefit of this disclosure.
Referring to
To that end, as a non-limiting example, in some embodiments, the system 100 includes an audio input device 102 (e.g., a microphone) providing audio signal to the audio conditioner 120 and a noise generator 104 providing the noise signal 110 to the spectral shaper 130. In some embodiments, the audio input device 102 records ambient sound from the surroundings. The ambient sound (including voice conversations) as the audio signal 110 is conditioned by the audio conditioner 120. For example, the audio signal 110 can be conditioned to sound like a voice pattern different from the voice pattern of the user of the system 100. As a result, it is more difficult to identify or extra information related to the user based on the conditioned audio signal.
In some embodiments, the noise generator 104 provides noise signal 110 to the spectral shaper 130. In some embodiments, the noise signal 110 can be mixed with the sound recorded by the audio input device 102 and the mixed signal are then coupled to the audio output device 150. As such, the sound recorded by the audio input device 102 is masked and not identifiable or intelligible for privacy protection. In some embodiments, the noise signal 110 from the noise generator 104 is associated with a random (or pseudo-random) number sequence. In some embodiments, the noise signal 110 is generated within a digital signal processor (DSP), field programmable gate array (FPGA), application-specific integrated circuit (ASIC), microprocessor, and/or by the firmware/software (e.g., through the use of pseudo random number generators and/or algorithms such as AES encryption with various key lengths etc.). In other embodiments, the noise signal 110 is generated by external or dedicated electronic components, such as a diode or a resistor that generates electronic noise. In some embodiments, the noise signal 110 is generated by applying power to the resistor and/or by the diode in breakdown mode and measuring and/or sampling the noise created. In some embodiments, the noise signal 110 can be used as a random seed to generate multiple uncorrelated audio signal streams, e.g., by re-using the same seed or sampling such seed at pre-determined or randomized intervals to produce uncorrelated noise signal streams for masking two or more audio input devices. As used herein, a random seed is a number (or vector) used to initialize a random or pseudorandom number generator.
In some embodiments, the spectral shaper 130 processes the noise signal 110, including obtaining a spectral envelope with target spectral characteristics and applying the spectral envelope in order to modify the spectral characteristics of the noise signal 110. In some embodiments, the spectral shaper 130 receives feedbacks 108 for calibration. The spectral shaper 130 realizes a desired output frequency response through the feedback loop 108. The spectral shaper 130 is further described in detail below with reference to
In some embodiments, the audio coupling interface 152 mates the audio output device 150 with the audio input device 162. The audio coupling interface 152 can be made of audio seals, structures, baffles, and/or sound isolating techniques known in the art to help reduce external audio energy from reaching the audio input device 162. In some embodiments, mechanical or electro-mechanical mechanism known in the art can be used to apply the audio coupling interface 152 to its mated surface. In some embodiments, differing compression levels based on the surface material of the user equipment 160 can be used for the audio coupling interface 152 in order to form an audio-sealing pathway. The audio coupling interface 152 reduces the amount of leaked sound 154-1 and 154-2 from escaping the audio-sealing pathway and reduces the amount of ambient sound from entering the audio-sealing pathway.
For example,
In
Though
Turning to
To that end, as a non-limiting example, in some embodiments, the housing 315 is a housing assembly. In some embodiments, the housing assembly further includes sub-assemblies, e.g., a plurality of both moveable parts and non-moveable parts that can form an enclosure when assembled together. The housing 315 thus allows a user to insert the user equipment 160 into the active case 310 for more protection of sensitive information (e.g., in a work mode) or take the user equipment 160 out of the active case 310 for less monitoring of the personal communication by a business organization (e.g., in a personal mode). In some embodiments, the housing assembly also causes the DSP 320 to selectively couple the shaped audio signal to the speaker 150 of the active case 310. In other words, by pressing a button at least partially supported by the housing assembly or changing the hood position, the DSP 320 generates and provides different control signals to the switch 106 (
In some embodiments, the sub-assemblies include a base and a hood assembly that is moveable. In some embodiments, when the hood assembly is in a first position (e.g., hood down/lowered or clamped), the active case 310 is in a first mode of operation (e.g., a privacy protection mode or a work mode). On the other hand, when the hood assembly is in a second position (e.g., hood up or unclamped), the active case 310 is in a second mode of operation (e.g., an unprotected mode or a personal mode). When the active case 310 is in the first mode of operation (e.g., the work mode), the hood assembly coordinated with the base engages the audio seals 152 to mate the speakers 150 with the microphones 162. Once engaged, the audio seals 152 provide parts of audio-sealing pathways between the speakers 150 and the microphones 162. In some embodiments, a button at least partially supported by the housing 315 is provided to turn on or off the sound masking function, so that the active case 310 selectively couples the shaped audio signal to the speakers 150.
In some embodiments, at least the audio output device 150 is connectable to the user equipment 160. For example, the speaker 150 is clipped on a smart device and the audio seal 152 is coupled to the opening of the microphone 162 of the user equipment 160 to seal the audio-sealing pathway between the speaker 150 and the microphone 162.
In some embodiments, the user can activate, deactivate, tune or change the level, volume, power or capabilities of the electronic audio repeating and/or masking capability through the use of switches, buttons or other such physical interface included in or at least partially supported by the housing 315, so that the active case 310 can selectively couple the shaped audio signal to the speakers 150. In some other embodiments, such features or capabilities can be activated, deactivated, tuned or changed by movements, motion, remote control(s) such as radio frequency (RF), infrared, or other wired or wireless technology or sound such as a spoken keyword or phrase. In some embodiments of the invention, the use of electronic audio repeating, manipulation, jamming, masking, attenuating, and/or blocking can provide features or capabilities beyond audio protection, such as amplification, audio enhancement, noise or echo canceling, audio mixing and other forms of audio manipulation to name a few.
In some embodiments, the speakers 150 output sound generated based on the shaped audio signal. The sound is passed to the microphones 162 along the audio-sealing pathway, including passing through the cavity inside the audio seals 152. Through the audio seals 152, the speakers 150 are mated with the microphones 162, e.g., the audio seal 152-1 mates the speaker 150-1 with the microphone 162-1, the audio seal 152-2 mates the speaker 150-2 with the microphone 162-2, the audio seal 152-3 mates the speaker 150-3 with the microphone 162-3, and the audio seal 152-4 mates the speaker 150-4 with the microphone 162-4. In some embodiments, one end of each audio-sealing pathway is the speaker 150 on the active case 310, e.g., the audio seal 152 is made of flexible materials such that it extends from the surrounding edge of the speaker membrane. In some embodiments, the other end of each audio-sealing pathway covers a respective microphone 162 (or the opening of the respective microphone 210 as shown in
As described above with reference to
In some embodiments, the one or more audio seals 152 can remain stationary relative to the housing 315. In some other embodiments, the one or more audio seals 152 can be moveable, thus sealing and unsealing the sealing path at different points in time. In some embodiments, the ability to seal or unseal one or more microphones 162 of the user equipment 160 is available on a microphone-by-microphone basis.
In some embodiments, to further reduce the detectability and/or obtrusiveness of the leaked audio signal, the DSP 320 shapes the audio signal prior to outputting the shaped audio signal by the speakers 150. Embodiments of the DSP 320 include hardware, software, firmware, or a combination thereof. In some embodiments, the DSP 320 executes instructions stored in non-transitory memory to perform at least certain functions of noise source generation (e.g., the instructions for the noise generator 104) and sound shaping (e.g., the instructions for the spectral shaper 130,
In some embodiments, to reduce the detectability of the leaked audio signal, the DSP 320 instructs different audio output devices 150 to play different audio content (e.g., pass-through, noise, modified, processed, manipulated or otherwise changed content) for different microphones 162. As such, the audio content played is uncorrelated, and thus it is more difficult to derive the protected audio content (e.g., through elaborated efforts), such as by comparing the audio signals from different microphones 162.
In some embodiments, the active case 310 also includes a plurality of audio input devices of its own, e.g., the microphones 102-1, 102-2, 102-3, and 102-4. The microphones 102 are at least partially supported by the housing 315. In some embodiments, the microphones 102 record sound independently of the sound recording by the microphones 162 on the user equipment 160. In some embodiments, the active case 310 transmits the independently recorded sound to an external electronic device through a secure channel for secure communication. Further, the ambient sound (including voice conversations) recorded by the microphones 102 can be shaped by the DSP 320 to obfuscate the ambient sound before being outputted by the audio output device 150 and directed at the audio input device 162 of the user equipment 160. The obfuscation makes it more difficult to derive the audio content from the shaped audio signal.
In some embodiments, an envelope detector 322 derives the sound envelope from the sound recorded by the microphone(s) 102. In some embodiments, the envelope detector 322 is coupled to the microphones 102 and the DSP 320. In some embodiments, the envelope detector 322 is an electronic circuit that takes the audio signal representing the ambient sound recorded by one or more of the microphones 102 as input and provides an output, which is the envelope of the audio signal. The envelope detector 322 thus detects the amplitude variations of the incoming signal. In some embodiments, the envelope detector 322 provides the envelope information to the DSP 320, so that the DSP 320 directs the speakers 150 to adjust the volume of the output masking signal from the speakers 150 appropriate for the level of ambient sound. Though
To that end, as a non-limiting example, in some embodiments, the spectral shaper 130 is coupled to the audio output device 150. In some embodiments, the spectral shaper 130 maintains the spectral envelope profiles 410. In some embodiments, a spectral envelope profile of the spectral envelope profiles 410 specifies spectral characteristics or other desirable characteristics for audio signal shaping. For example, a spectral envelope profile 410 can comprise frequency patterns of noise (e.g., white, blue, pink, gray, etc.) for shaping the audio signal. In another example, a spectral envelope profile 410 comprises masking signal characteristics corresponding to wind noise, traffic sound, music, or other voice for shaping or morphing the audio signal. In yet another example, a spectral envelope profile 410 specifies audio signal shaping parameters for different frequency bands. In some embodiments, the spectral shaper 130 is programmable, so that the user can activate, deactivate, obtain, or select a spectral envelope profile 410 through the use of switches, buttons or other such physical interface included in or at least partially supported by the housing 315. In some other embodiments, such features or capabilities can be triggered by movements, motion, remote control(s) such as RF, infrared, or other wired or wireless technology or sound such as a spoken keyword or phrase.
In some embodiments, the spectral shaper 130 increases or decreases different frequency bands according to a spectral envelope profile retrieved from the spectral envelope profile 410. In order to shape different frequency bands, in some embodiments, the spectral shaper 130 includes a spectral divider 420 that breaks the audio signal into a set of frequency bands. The spectral shaper 130 then uses a spectral selector to select at least one of the set of frequency bands to apply the spectral envelope corresponding to the retrieved spectral envelope profile 410. As such, different frequency bands are shaped or adjusted according to the parameters specified by the spectral envelope.
For example, as shown in
In another example,
In yet another example,
Though
Referring back to
In some embodiments, the spectral validator 432 is operable to obtain a frequency response to the audio signal from the envelope detector 322, which is further coupled to the audio input device(s) 102 of the active case 310. The audio input device(s) 102, in some embodiments, captures the audio signal outputted by the audio output device(s) 150 of the active case 310, and provides the audio signal to the envelope detector 322 for measurement. The spectral validator 434 then obtains the measurement from the envelope detector 322 in order to derive the frequency response. The frequency response is then provided to the spectral shaper 130 and causes the spectral shaper 130 to adjust the shaped audio signal as a function of the frequency response.
In some embodiments, the amplitude validator 434 obtains the level of ambient sound from the envelope detector 322. The amplitude validator 434 then compares the level of ambient sound with the level or amplitude of the output noise signal in order to determine whether or not the shaped audio signal is at the appropriate level for masking the ambient sound. Based on the comparison result, in some embodiments, through the coupling with the speakers 150, the amplitude validator 434 directs the speakers 150 to adjust the output sound level in case the output sound level is not appropriate (e.g., too high or too low) for masking ambient sound.
For example, along the audio-sealing pathway, components including the audio output devices 150, the audio coupling interface 152 (
Using the validation engine 430 disclosed herein, the active case 310 balances the effectiveness of privacy protection and obtrusiveness. For example, in a quiet room, where the ambient sound level is low, it is obtrusive for the active case to output loud masking signal. On the other hand, when the ambient sound level is high, e.g., when people are yelling, it is necessary to increase the masking signal level in order to shield the loud conversation. Thus, through the validation engine 430, the active case 310 balances the effectiveness of privacy protection and obtrusiveness by varying the level of the audio jamming in accordance with the ambient sound level.
To that end, as represented by block 610, the method 600 includes obtaining an audio signal (e.g., the audio signal from 110 from the microphone 102 and/or the noise generator 104 as shown in
As represented by block 620, in some embodiments, the method 600 includes applying a spectral envelope to the audio signal in order to produce a shaped audio signal, wherein the shaped audio signal has a diminishing propagation pattern. In some embodiments, as represented by block 622, the spectral envelope is associated with a pink noise profile. For example, a pink noise profile is characterized by lower the frequency components/content above certain frequency levels. The pink noise profile is desirable due to its low power consumption, effective masking, and less obtrusive or noticeable. An exemplary shaped audio signal generated by applying a pink noise profile is shown in
In some embodiments, the spectral envelope application includes splitting the audio signal into a plurality of frequency bands and adjusting at least one of the plurality of frequency bands in accordance with the spectral envelope in order to produce the shaped audio signal. As a result of shaping the frequency bands, the shaped audio signal has the diminishing propagation pattern when propagating.
For example, as represented by block 624, the diminishing propagation pattern specifies a threshold distance beyond which the shaped audio signal diminishes below a threshold level. Because low frequencies travel farther, by applying a spectral envelope to lower the level of low frequency bands between, for example, [F1, F2] as shown in
In another example, as represented by block 626, the diminishing propagation pattern specifies a threshold rate and the shaped audio signal diminishes faster than the threshold rate. Because high frequencies are absorbed better than low frequencies, the amplitude of high frequency signals falls off faster. By applying a spectral envelope to lower the level of high frequency bands between, for example, [F2, F3] as shown in
As represented by block 630, in some embodiments, the method 600 includes selectively coupling the shape audio signal to an audio output device (e.g., the speaker 150) of the first device (e.g., the active case 310). For example, in one mode of operation, the shaped audio signal is the noise signal shaped from the noise provided by the noise generator 104. In such mode, the shaped noise signal is coupled to the speaker 150. In a different mode of operation, the shaped audio signal is generated by shaping the audio signal captured by the microphone 102 of the active case 310, e.g., by changing the frequency of the audio signal so that the voice sounds like a different person. In such embodiments, the modified voice signal is coupled to the speaker 150.
In some embodiments, as represented by block 640, the method 600 includes causing the audio output device (e.g., the speaker 150) to output sound as a function of the shaped audio signal directed to an audio input device (e.g., the microphone 162) of a second device (e.g., the user equipment 160). In such embodiments, the shaped audio signal is passed through a cavity provided by an audio coupling interface (e.g., the audio seal 152) for the audio output device (e.g., the speaker 150) of the first device (e.g., the active case 310 and the audio input device (e.g., the microphone 162) of the second device (e.g., the user equipment 160). In some embodiments, the cavity is formed by the audio coupling interface (e.g., the hole inside the wall of the audio seal 152 as shown in
In some embodiments, the method 600 includes estimating (e.g., by the envelope detector 322) a level of ambient sound in which the apparatus (e.g., the active case 310) is operating, determining (e.g., by the validation engine 430) whether or not a level of the shaped audio signal is appropriate for the level of ambient sound, and as represented by block 642, causing (e.g., by the validation engine 430) the audio output device to adjust the level of the shaped audio signal based on a determination that the level of the shaped audio signal is not appropriate for the level of ambient sound. The level of the shaped signal after the adjustment thus is appropriate for the level of ambient sound in order to balance the effectiveness of privacy protection and obtrusiveness.
In some embodiments, the audio signal includes uncorrelated audio signals. In such embodiments, as represented by block 644, the method 600 includes applying the spectral envelope to the uncorrelated audio signals to produce uncorrelated shaped audio signals, selectively coupling the uncorrelated shaped audio signals to a plurality of audio output devices (e.g., the speakers 150) of the first device (e.g., the active case 310), and directing the plurality of audio output devices (e.g., the speakers 150) to output uncorrelated sounds based on the uncorrelated shaped audio signals, wherein the uncorrelated shaped audio signals are passed through a plurality of cavities provided by the plurality of audio coupling interfaces (e.g., the cavities inside the seals 152) for the plurality of audio output devices (e.g., the speakers 150) of the first device (e.g., the active case 310) and the plurality of audio input devices (e.g., the microphones 162) of the second device (e.g., the user equipment 160). As such, the audio content recorded by the microphones 162 is uncorrelated, and it is more difficult to derive the protected audio content through elaborated efforts, such as by comparing the audio signal from different microphones 162.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best use the invention and various described embodiments with various modifications as are suited to the particular use contemplated.
This application is a continuation of U.S. patent application Ser. No. 16/272,946 filed on Feb. 11, 2019, which further claims priority to U.S. provisional patent application No. 62/630,128 filed on Feb. 13, 2018, the contents of which are hereby incorporated by reference.
Number | Name | Date | Kind |
---|---|---|---|
5732143 | Andrea | Mar 1998 | A |
7260221 | Atsmon | Aug 2007 | B1 |
7346654 | Weiss | Mar 2008 | B1 |
9525765 | Moser | Dec 2016 | B2 |
10074353 | Cook | Sep 2018 | B2 |
20030091199 | Horrall | May 2003 | A1 |
20030219133 | Horrall | Nov 2003 | A1 |
20060136544 | Atsmon | Jun 2006 | A1 |
20080281588 | Akagi | Nov 2008 | A1 |
20090067291 | Atsmon | Mar 2009 | A1 |
20100030838 | Atsmon | Feb 2010 | A1 |
20110103614 | Cheung | May 2011 | A1 |
20110182445 | Atsmon | Jul 2011 | A1 |
20140006017 | Sen | Jan 2014 | A1 |
20140064526 | Otto | Mar 2014 | A1 |
20150271341 | Kleiner | Sep 2015 | A1 |
20160041808 | Pelland | Feb 2016 | A1 |
20180115839 | Eichfeld | Apr 2018 | A1 |
Number | Date | Country |
---|---|---|
2006267174 | Oct 2006 | JP |
2006267174 | Oct 2006 | JP |
Number | Date | Country | |
---|---|---|---|
20200366993 A1 | Nov 2020 | US |
Number | Date | Country | |
---|---|---|---|
62630128 | Feb 2018 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16272946 | Feb 2019 | US |
Child | 16983780 | US |