Patent Application
20240267678
The present application relates to capturing immersive audio for providing an immersive user experience with rendering of the immersive audio.
Immersive audio may be utilized for an enhanced user experience that comprises rendering of the immersive audio. The immersive audio may be binaural audio or spatial audio for example. The immersive audio may be rendered for a call that may be a voice call or a video call, or the immersive audio may be rendered as part of rendering media content that comprises the immersive audio.
The scope of protection sought for various embodiments is set out by the independent claims. Dependent claims define further embodiments included in the scope of protection. The exemplary embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.
According to a first aspect there is provided an apparatus comprising means for: capturing, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognizing, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognizing the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capturing the immersive audio using the second capturing mode.
In some example embodiments according to the first aspect, the means comprises at least one processor, and at least one memory storing instructions that, when executed by the at least one processor, to cause the performance of the apparatus.
According to a second aspect there is provided an apparatus comprising at least one processor, and at least one memory storing instructions that, when executed by the at least one processor, to cause the apparatus at least to perform: capture, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognize, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognize the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capture the immersive audio using the second capturing mode.
According to a third aspect there is provided a method comprising: capturing, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognizing, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognizing the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capturing the immersive audio using the second capturing mode.
In some example embodiments according to the third aspect the method is a computer implemented method.
According to a fourth aspect there is provided a computer program comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: capture, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognize, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognize the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capture the immersive audio using the second capturing mode.
According to a fifth aspect there is provided a computer program comprising instructions stored thereon for performing at least the following: capturing, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognizing, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognizing the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capturing the immersive audio using the second capturing mode.
According to a sixth aspect there is provided a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: capture, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognize, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognize the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capture the immersive audio using the second capturing mode.
According to a seventh aspect there is provided a non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following: capturing, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognizing, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognizing the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capturing the immersive audio using the second capturing mode.
According to an eighth aspect there is provided a computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: capture, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognize, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognize the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capture the immersive audio using the second capturing mode.
According to a ninth aspect there is provided a computer readable medium comprising program instructions stored thereon for performing at least the following: capturing, using a first capturing mode, immersive audio using a first capturing device comprising a first microphone and a second capturing device comprising a second microphone, recognizing, based on obtaining data from one or more sensors, movement of the first capturing device, wherein the movement is with respect to the second capturing device, recognizing the movement as movement for changing from the first capturing mode to a second capturing mode, wherein the second capturing mode is for capturing immersive audio, and capturing the immersive audio using the second capturing mode.
The following embodiments are exemplifying. Although the specification may refer to “an”, “one”, or “some” embodiment(s) in several locations of the text, this does not necessarily mean that each reference is made to the same embodiment(s), or that a particular feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments.
As used in this application, the term ‘circuitry’ refers to all of the following: (a) hardware-only circuit implementations, such as implementations in only analog and/or digital circuitry, and (b) combinations of circuits and software (and/or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s)/software including digital signal processor(s), software, and memory(ies) that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present. This definition of ‘circuitry’ applies to all uses of this term in this application. As a further example, as used in this application, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and/or firmware. The term ‘circuitry’ would also cover, for example and if applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device. The above-described embodiments of the circuitry may also be considered as embodiments that provide means for carrying out the embodiments of the methods or processes described in this document.
As used herein, the term “determining” (and grammatical variants thereof) can include, not least: calculating, computing, processing, deriving, measuring, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), obtaining and the like. Also, “determining” can include resolving, selecting, choosing, establishing, and the like.
Immersive audio may be used to enhance a user experience when audio is rendered. Such enhanced user experience may be present for example during a voice call, a video call, or when rendering audio as such or as part of other media content such as video or still images. Immersive audio may be understood to be binaural audio, spatial audio, or a combination thereof. For example, when immersive audio is rendered using a headset worn by a user, the binaural audio may be utilized. Binaural audio may be captured using for example two microphones and it may be used for providing the user with a perceptually three-dimensional headphone-reproduced stereo sound that provides a user experience mimicking the user actually being present in the room from which the binaural audio was captured. Spatial audio on the other hand may comprise a full sphere surround-sound that mimics the way the user would perceive the rendered immersive audio in real life. Spatial audio may comprise audio that is perceived by the user to originate from a certain direction and/or distance, and thus the rendered spatial audio is perceived to change with the movement of the user or with the user turning. Spatial audio may comprise audio that is perceived to originate from one or more sound sources, ambient sound or a combination thereof. Ambient sound may comprise audio that might not be identifiable in terms of a sound source such as traffic humming, wind or waves.
To capture immersive audio, a plurality of capturing devices, that are for capturing audio, and comprise one or more microphones per capturing device, may be used. The plurality of capturing devices may comprise for example ear-worn devices such as earbuds and in-ear headphones. For example, if a headset is worn by a user and the headset comprises two earbuds that each comprise at least one microphone, the headset may be used to capture immersive audio such as binaural audio around the user. Such a headset may be a true wireless stereo headset with integrated microphones.
If a user is in a call that is an immersive call such as described above, it may be difficult to capture the immersive audio, without the voice of the user drowning, when the environmental sounds are loud. While the environmental sounds are relevant for capturing the immersive audio and providing the user experience for the immersive call, having environmental sounds that are too loud can cause the voice of the user to be drowned and thus it may be difficult for another party in the call to hear the user well enough. The user could of course talk louder, but that may be inconvenient due to privacy of the call being compromised or others around the user perceiving talking loudly as rude. Beamforming techniques may be used to better capture the voice of the user and thus cancel some of the environmental sounds perceived as disturbing. Further, microphones with a narrow directive capture pattern may be utilized to mitigate the disturbing environmental sounds. Further, additionally or alternatively, machine learning methods may be used to mitigate the disturbing environmental sounds.
When capturing immersive audio during a call, earbuds comprised in a head set and worn by a user may be used as capturing devices for capturing immersive audio. The earbuds comprising microphones may be located such that they are in good placements in terms of capturing surrounding spatial sounds. Yet, in some example embodiments, such placement may not be most optimal if the speech of the user is to be captured and thus it may be useful to move at least one of the two earbuds closer to the mouth of the user to better capture the speech of the user.
Then, in situation 222, the user 200 moves close to a noisy crowd 230 while the immersive call is still ongoing. Thus, the environmental sounds captured by the capturing devices, that is, by the earbuds 210 and 215, become loud enough to be perceived as disturbing as the at least one other party in the immersive call is having trouble hearing the voice of the user 200 because of the loud environmental sounds that may also be referred to as background noise. Thus, in this example embodiment, the user 200 removes the earbud 210 from his ear and places it closer to his mouth. Thus, the earbud 210 is moved with respect to the earbud 215, which is still worn in the other ear of the user 200. This movement may be interpreted as a trigger to switch to another mode of capturing immersive audio for the ongoing call. In this other mode, the earbud 210 captures a mono object-stream of the voice of the user 200. The residual of the object signal and the signal captured by the earbud 215 may be used to extract ambient sound. As the earbud 210 is moved such that it is now closer to the mouth of the user 200, this placement increases the signal to noise ratio of the audio captured for the voice of the user 200, which allows capturing spatial audio differently, in other words, allows capturing the audio for the immersive call using another mode for capturing. The earbud 210 may comprise one or more microphones and as they are close to the mouth, the captured audio may be processed as an object source. As mentioned, in this example embodiment, the voice of the user 200 is captured as a mono stream that is rendered to the immersive audio during the playback of the captured immersive audio. The object may thus be for example panned to different locations in the sound scene of the immersive audio that is captured during the rendering of the immersive audio. The earbud 215 may also comprise one or more microphones which in the second mode of capturing may be used to capture ambient sound that may also be understood as background noise. It is to be noted that the one or more microphones from both earbuds 210 and 215 may also be used for capturing ambient sound in the other capturing mode as well, which may have the benefit of making it easier to remove the voice of the user 200 from the ambience sound captured.
In the second mode of capturing the voice of the user 200 captured using the earbud 210 may comprise that the voice of the user 200 is removed from the ambient sound. This may be performed in any suitable manner, and it may be performed by any suitable device such as by the mobile device to which the earbuds 210 and 215 are connected to and which is used for enabling the immersive call. It is also to be noted that the mobile device may be the device that recognizes the movement of the earbud 210 as a movement based on which the capturing mode is changed from the first capturing mode to the second capturing mode.
For example, the speech signal captured by the earbud 210, the voice of the user 200, that is the speech signal, can be used for removing the speech signal from the ambient sound captured by the earbud 215. The removal of the voice of the user 200 may be performed for example by first defining a speech signal corresponding to the voice of the user 200 based on the sound captured by the earbud 210. For example, the speech signal may be directly the microphone signal captured by the earbud 210. Alternatively, the speech signal obtained from the earbud 210 may be enhanced using for example ambient noise reduction methods, and/or by using machine-learning based speech enhancement methods. After defining the speech signal, the acoustic path of the speech signal to the at least one other microphone comprised in the earbud 215 is estimated using any suitable method such as those used in the field of acoustic echo cancellation. Next, the speech signal may be processed with the estimated acoustic path, and subtracted from the audio captured by the earbud 215 using the at least one other microphones. As a result, a mono speech signal and a mono remainder, such as the ambience, signals are obtained as separate signals. It is then possible to synthesize spatial ambience based on the remainder signal by means of decorrelating the remainder signal to two incoherent remainder channels and cross-mixing these channels as a function of frequency to obtain the required binaural cross-correlation. Then, the speech signal may be processed further for example with a pair of head-related transfer functions (HRTF) to a desired direction, to front for example, and the further processed speech signal may then be added to the binaural ambience signal. The levels of the speech signal and the remainder signal may be set to desired levels, for example, by ensuring at least a defined speech-to-remainder energy ratio. Alternatively, the speech signal and the remainder signals may be transmitted separately to the recipient device, for example to the device of the at least one other party used for participating in the immersive call, so that the speech position can be controlled at the decoder of the recipient device.
As illustrated in the situation 222 of the
When the capturing mode is changed, the type of immersive audio captured changes and the other parties present in the immersive call are to be informed regarding this change. Also, the audio stream that is transmitted to the other parties in the call changes. To address this, the device of the user 200, which is the device allowing the immersive call to occur for the user 200, detects that the capturing mode is to be changed based on the movement of the earbud 210, which is also detected as described above. Then the device may determine parameters for the second mode of capturing, for example, it may be determined which of the earbuds 210 and 215 has been moved and to which position. Next, the change of the capturing mode and configurations related to the second capturing mode may be indicated to the at least one other party in the call. Based on the configurations for the second capturing mode, the mobile device then sets up accordingly codecs to capture the object, that is the speech of the user 200, and the ambient sound and then begins to transit those as an immersive audio stream to the at least one other party present in the immersive call.
In this example embodiment, the switching of the capturing mode may cause a small interruption in the immersive audio stream transmitted to the at least one other party present in the call. Such interruption may be mitigated for example by fading out the immersive audio captured using the current mode of capture and fading in the immersive audio captured using the new capture mode after it has started. As an alternative example, both capture modes may be running simultaneously for a while, in other words, for a pre-determined time period, and also crossfading between them may be performed.
In this example embodiment, the user 200 then moves away from the crowd 230 as illustrated in situation 224. As a consequence, the environment is not noisy anymore, so the user 200 moves the earbud 210 back into his ear. Now the device to which the earbuds 210 and 215 are connected to again recognizes the movement of the earbud 210 that is movement different than the movement of the earbud 215 and is also with respect to the earbud 215. Based on recognizing this movement, the device reverts back to the original binaural capture mode, that is, back to the first capturing mode.
It is to be noted that in some example embodiments, further movement of a capture device such as an earbud may be used for further interaction than changing the capturing mode from a first capturing mode to a second capturing mode. The further interaction may comprise for example controlling of the capturing of the immersive audio. For example, if the second capturing mode is as described in the example embodiment of
The example embodiments described above may have benefits such as having the voice of a user audible in various situations when immersive audio is captured for an immersive call or for other immersive media capturing purpose. Also, the immersive capturing may be maintained with minimal effect on the quality of the captured immersive audio. The user interaction may also be perceived as intuitive.
The processor 410 is coupled to a memory 420. The processor is configured to read and write data to and from the memory 420. The memory 420 may comprise one or more memory units. The memory units may be volatile or non-volatile. It is to be noted that in some example embodiments there may be one or more units of non-volatile memory and one or more units of volatile memory or, alternatively, one or more units of non-volatile memory, or, alternatively, one or more units of volatile memory. Volatile memory may be for example RAM, DRAM or SDRAM. Non-volatile memory may be for example ROM, PROM, EEPROM, flash memory, optical storage or magnetic storage. In general, memories may be referred to as non-transitory computer readable media. The memory 420 stores computer readable instructions that are execute by the processor 410. For example, non-volatile memory stores the computer readable instructions and the processor 410 executes the instructions using volatile memory for temporary storage of data and/or instructions.
The computer readable instructions may have been pre-stored to the memory 420 or, alternatively or additionally, they may be received, by the apparatus, via electromagnetic carrier signal and/or may be copied from a physical entity such as computer program product. Execution of the computer readable instructions causes the apparatus 400 to perform functionality described above.
In the context of this document, a “memory” or “computer-readable media” may be any non-transitory media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.
The apparatus 400 further comprises, or is connected to, an input unit 430. The input unit 430 comprises one or more interfaces for receiving a user input. The one or more interfaces may comprise for example one or more motion and/or orientation sensors, one or more cameras, one or more accelerometers, one or more microphones, one or more buttons and one or more touch detection units. Further, the input unit 430 may comprise an interface to which external devices may connect to.
The apparatus 400 also comprises an output unit 440. The output unit comprises or is connected to one or more displays capable of rendering visual content such as a light emitting diode, LED, display, a liquid crystal display, LCD and a liquid crystal on silicon, LCoS, display. The output unit 440 further comprises one or more audio outputs. The one or more audio outputs may be for example loudspeakers or a set of headphones.
The apparatus 400 may further comprise a connectivity unit 450. The connectivity unit 450 enables wired and/or wireless connectivity to external networks. The connectivity unit 450 may comprise one or more antennas and one or more receivers that may be integrated to the apparatus 400 or the apparatus 400 may be connected to. The connectivity unit 450 may comprise an integrated circuit or a set of integrated circuits that provide the wireless communication capability for the apparatus 400. Alternatively, the wireless connectivity may be a hardwired application specific integrated circuit, ASIC.
It is to be noted that the apparatus 400 may further comprise various component not illustrated in the
Example embodiments described herein may be implemented using software, hardware, application logic or a combination of them. Also, if desired, different functionalities discussed herein may be performed in a different order, some functionalities may be performed concurrently, and, if desired, some of the above-mentioned functionalities may be combined. Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and/or dependent claims with features of the independent claims and not solely the combinations explicitly set out in the claims.
It will be appreciated that the above described example embodiments are purely illustrative and are not limiting on the scope of the invention. Other variations and modifications will be apparent to persons skilled in the art upon reading the present specification.
| Number | Date | Country | Kind |
|---|---|---|---|
| 23155563.2 | Feb 2023 | EP | regional |
