Patent Grant
10448192
The present application relates to apparatus for the processing and stabilizing of audio signals. The application further relates to, but is not limited to, portable or mobile apparatus for processing and stabilizing audio signals from mobile devices.
The capture or recording of audio or audio video content is well known. Many handheld devices are equipped with both cameras and microphones configured to capture or record audio and/or audio video signals for storage or transmission. Furthermore such devices are more commonly being equipped with spatial audio capture technology. Spatial audio capture technology uses an array of microphones (two or more) for recording or capturing the audio environment. The captured audio signals are analysed to extract the spatial co-ordinates/positions of any relevant or dominant sources in the captured audio environment. The spatial co-ordinates can then be defined with regards or reference to the orientation of the capturing device and typically the orientation of the centre of the camera viewpoint. Typically the spatial co-ordinates of the audio sources relative to the orientation of the electronic device are in the form of an angle with respect to a device centre axis or axes defined by the camera orientation. These coordinates are then used in some situations to synthesize a stereo audio signal at a listening device. The synthesis involves imparting frequency and angle dependent inter aural time and level difference cues through a head related transfer function (HRTF) or head related impulse response (HRIR). These combinations of HRTF/HRIR values when replayed on a stereo headphone set allow the signal to represent an audio sound field which is perceptually similar to the recorded audio environment. Alternatively synthesis based on custom panning rules can be applied for replaying it on a multi-channel loudspeaker set up.
Often the presentation of spatial audio signals is performed in conjunction with a replay of a video feed captured by a camera on the device.
Such portable devices unlike tripod mounted apparatus are prone to translational and rotational motion while they are recording audio and video. These motions can be the result of motion of the person holding the device (produced such as by device ‘shake’, or movement such as walking, running, and changing hands), or by the motion of a vehicle on which the device is mounted while recording. The motion is unintentional and unavoidable but can result in an unpleasant video playback experience. Video stabilization to overcome such translational and rotational values for video in many commercially available video recorders as this motion is typically constrained to relatively small translational and rotational values.
The recorded audio signal is similarly affected by motion. Specifically any motion could generate an incorrect estimate of the positions of the sources in the audio environment. Furthermore the estimation could further assign an audio source an incorrect motion. For example stationary sources at a given coordinate would when affected by motion can lead to variations in the estimated positions which causes the replayed sound sources to ‘oscillate’ or ‘wobble’ around their position whilst the video image maintained stationary.
Aspects of this application thus provide audio source stabilization in recording audio signals.
There is provided according to the application a method comprising: determining a spatial audio signal; determining an apparatus motion parameter; and stabilizing the spatial audio signal dependent on the apparatus motion parameters.
The method may further comprise: capturing at least two video images at different time instances; determining from the at least two video images a video image motion value; and wherein determining an apparatus motion parameter comprises determining an apparatus motion parameter based on the video image motion value.
Determining from the at least two video images a video image motion value may comprise: determining at least two local motion values; and combining the at least two local motion values to generate the video image motion value.
Determining a spatial audio signal may comprise: capturing at least one audio signal; and determining from the at least one audio signal at least one audio signal source location relative to a capturing apparatus.
Stabilizing the spatial audio signal dependent on the apparatus motion parameters may comprise changing the at least one audio signal source location relative to the capturing apparatus dependent on the video image motion value.
Determining an apparatus motion parameter may comprise filtering the apparatus motion parameter to remove lower frequency values.
Determining the apparatus motion parameter may comprise: determining an apparatus motion value from a positional sensor.
The positional sensor may comprise at least one of: accelerometer, gyroscope, electronic compass, a satellite positional estimator.
Determining an apparatus motion parameter based on the video image motion value may comprise: converting the video image motion value into at least one of a translational motion parameter of the apparatus and a rotational motion parameter of the apparatus.
Stabilizing the spatial audio signal dependent on the apparatus motion parameters may comprise rotationally stabilizing the spatial audio signal based on the rotational motion parameter of the apparatus.
Stabilizing the spatial audio signal dependent on the apparatus motion parameters may comprise translationally stabilizing the spatial audio signal based on the translational motion parameter of the apparatus.
Translationally stabilizing the spatial audio signal based on the translational motion parameter of the apparatus may comprise translationally stabilizing the spatial audio signal dependent on the translational motion parameter of the apparatus being greater than a translational parameter threshold.
According to a second aspect there is provided an apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to at least perform: determining a spatial audio signal; determining an apparatus motion parameter; and stabilizing the spatial audio signal dependent on the apparatus motion parameters.
The apparatus may further perform: capturing at least two video images at different time instances; determining from the at least two video images a video image motion value; and wherein determining an apparatus motion parameter comprises determining an apparatus motion parameter based on the video image motion value.
Determining from the at least two video images a video image motion value may cause the apparatus to perform: determining at least two local motion values; and combining the at least two local motion values to generate the video image motion value.
Determining a spatial audio signal may cause the apparatus to perform: capturing at least one audio signal; and determining from the at least one audio signal at least one audio signal source location relative to the apparatus.
Stabilizing the spatial audio signal dependent on the apparatus motion parameters may cause the apparatus to perform changing the at least one audio signal source location relative to the apparatus dependent on the video image motion value.
Determining an apparatus motion parameter may cause the apparatus to perform filtering the apparatus motion parameter to remove lower frequency values.
Determining the apparatus motion parameter may cause the apparatus to perform: determining an apparatus motion value from a positional sensor.
The positional sensor may comprise at least one of: accelerometer, gyroscope, electronic compass, a satellite positional estimator.
Determining an apparatus motion parameter based on the video image motion value may cause the apparatus to perform: converting the video image motion value into at least one of a translational motion parameter of the apparatus and a rotational motion parameter of the apparatus.
Stabilizing the spatial audio signal dependent on the apparatus motion parameters may cause the apparatus to perform rotationally stabilizing the spatial audio signal based on the rotational motion parameter of the apparatus.
Stabilizing the spatial audio signal dependent on the apparatus motion parameters may cause the apparatus to perform translationally stabilizing the spatial audio signal based on the translational motion parameter of the apparatus.
Translationally stabilizing the spatial audio signal based on the translational motion parameter of the apparatus may cause the apparatus to perform translationally stabilizing the spatial audio signal dependent on the translational motion parameter of the apparatus being greater than a translational parameter threshold.
According to a third aspect of the application there is provided an apparatus comprising: an audio capturer configured to determine a spatial audio signal; an apparatus motion determiner configured to determine an apparatus motion parameter; and an audio stabilizer configured to stabilize the spatial audio signal dependent on the apparatus motion parameters.
The apparatus may further comprise: a camera configured to capture at least two video images at different time instances; a video motion determiner configured to determine from the at least two video images a video image motion value; and wherein the apparatus motion determiner is configured to determine an apparatus motion parameter based on the video image motion value.
The video motion determiner may comprise: a local motion determiner configured to determine at least two local motion values; and a global motion determiner configured to combine the at least two local motion values to generate the video image motion value.
The audio capturer may comprise: at least one microphone configured to determine at least one audio signal; and a spatial audio signal determiner configured to determine from the at least one audio signal at least one audio signal source location relative to the apparatus.
The audio stabilizer may comprise a spatial audio signal processor configured to change the at least one audio signal source location relative to the apparatus dependent on the video image motion value.
The apparatus motion determiner may comprise a filter configured to filter the apparatus motion parameter to remove lower frequency values.
The apparatus motion determiner may comprise a sensor configured to determine an apparatus motion value.
The sensor may comprise at least one of: accelerometer, gyroscope, electronic compass, a satellite positional estimator.
The video motion determiner may comprise: a converter configured to convert the video image motion value into at least one of a translational motion parameter of the apparatus and a rotational motion parameter of the apparatus.
The audio stabilizer may comprise a rotational stabilizer configured to rotationally stabilize the spatial audio signal based on the rotational motion parameter of the apparatus.
The audio stabilizer may comprise a translational stabilizer configure to translationally stabilize the spatial audio signal based on the translational motion parameter of the apparatus.
The translational stabilizer may comprise a translation significance determiner configured to enable the translational stabilizer to translationally stabilize the spatial audio signal dependent on the translational motion parameter of the apparatus being greater than a translational parameter threshold.
According to a fourth aspect of the application there is provided an apparatus comprising: means for determining a spatial audio signal; means for determining an apparatus motion parameter; and means for stabilizing the spatial audio signal dependent on the apparatus motion parameters.
The apparatus may further comprise: means for capturing at least two video images at different time instances; means for determining from the at least two video images a video image motion value; and wherein the means for determining an apparatus motion parameter comprises means for determining an apparatus motion parameter based on the video image motion value.
The means for determining from the at least two video images a video image motion value may comprise: means for determining at least two focal motion values; and means for combining the at least two local motion values to generate the video image motion value.
The means for determining a spatial audio signal may comprise: means for capturing at least one audio signal; and means for determining from the at least one audio signal at least one audio signal source location relative to a capturing apparatus.
The means for stabilizing the spatial audio signal dependent on the apparatus motion parameters may comprise means for changing the at least one audio signal source location relative to the capturing apparatus dependent on the video image motion value.
The means for determining an apparatus motion parameter may comprise means for filtering the apparatus motion parameter to remove lower frequency values.
The means for determining the apparatus motion parameter may comprise: means for determining an apparatus motion value from a positional sensor.
The positional sensor may comprise at least one of: accelerometer, gyroscope, electronic compass, a satellite positional estimator.
The means for determining an apparatus motion parameter based on the video image motion value may comprise: means for converting the video image motion value into at least one of a translational motion parameter of the apparatus and a rotational motion parameter of the apparatus.
The means for stabilizing the spatial audio signal dependent on the apparatus motion parameters may comprise means for rotationally stabilizing the spatial audio signal based on the rotational motion parameter of the apparatus.
The means for stabilizing the spatial audio signal dependent on the apparatus motion parameters may comprise means for translationally stabilizing the spatial audio signal based on the translational motion parameter of the apparatus.
The means for translationally stabilizing the spatial audio signal based on the translational motion parameter of the apparatus may comprise means for translationally stabilizing the spatial audio signal dependent on the translational motion parameter of the apparatus being greater than a translational parameter threshold.
A computer program product stored on a medium may cause an apparatus to perform the method as described herein.
An electronic device may comprise apparatus as described herein.
A chipset may comprise apparatus as described herein.
Embodiments of the present application aim to address problems associated with the state of the art.
For better understanding of the present application, reference will now be made by way of example to the accompanying drawings in which:
The following describes in suitable apparatus and possible mechanisms for the provision of effective audio stabilization. In the following examples audio signals and audio capture stabilization is described as part of an audio and video capture apparatus.
The concept described herein is related to assisting in audio recording or process capture and an associated audio presentation process. Furthermore although the following examples are described with regards to the camera or video capture apparatus generating sensor data for committing or enabling audio stabilization to occur it would be understood that in some embodiments other sensor data such as compass data, gyroscope data or accelerometer data can be used to produce suitable motion vector values or assist in the generation of suitable motion vectors values. Similarly in some embodiments sensor data from positional and/or orientation estimation such as estimated motion using satellite signals can similarly be used.
With respect to
The capturing of these audio sources can be processed in such a way that the audio signals when processed and synthesised for the listener device 771 the first synthesised audio source S1 711 is located directly ahead of the listener, the second synthesised audio source S2 713 is located at approximately 45 degrees and to the right of the listener and the third audio source 715 S3 is located at approximately 135 degrees and to the left of the listener at the locations and in line with the determined audio source locations.
As is shown in the right hand side of
This error furthermore is shown in the synthesised replayed audio sound field by a similar rotational error shown in the direction of the synthesised first 711 S1 second 713 S2 and third 715 S3 audio sources in the replayed audio sound field from the expected position. In other words the first synthesized audio source S1 712 has a first angle error 721 the second synthesized audio source has a second angle error 723 and the third synthesized audio source has a third angle error 725. It would be understood that where the audio signal sources are far field (where the displacement distance is much smaller than the distance between the source and the apparatus) then the errors will be substantially the same, however for the field audio signal sources the errors will differ dependent on the distance to the source from the capture apparatus.
With respect to
The example rotational angle 855 of the capture device causes the perceived reference direction to move (approximately 30 degrees shift right) with respect to the first (and other) audio sources as shown in the right hand side of
The electronic device 10 may for example be a mobile terminal or user equipment of a wireless communication system. In some embodiments the apparatus can be an audio player or audio recorder, such as an MP3 player, a media recorder/player (also known as an MP4 player), or any suitable portable device suitable for recording audio or audio/video camcorder/memory audio or video recorder.
The apparatus 10 can in some embodiments comprise an audio subsystem. The audio subsystem for example can comprise in some embodiments a microphone or array of microphones 11 for audio signal capture. In some embodiments the microphone or array of microphones can be a solid state microphone, in other words capable of capturing audio signals and outputting a suitable digital format signal. In some other embodiments the microphone or array of microphones 11 can comprise any suitable microphone or audio capture means, for example a condenser microphone, capacitor microphone, electrostatic microphone, Electret condenser microphone, dynamic microphone, ribbon microphone, carbon microphone, piezoelectric microphone, or micro-electrical-mechanical system (MEMS) microphone. The microphone 11 or array of microphones can in some embodiments output the audio captured signal to an analogue-to-digital converter (ADC).
In some embodiments the apparatus can further comprise an analogue-to-digital converter (ADC) configured to receive the analogue captured audio signal from the microphones and outputting the audio captured signal in a suitable digital form. The analogue-to-digital converter can be any suitable analogue-to-digital conversion or processing means.
In some embodiments the apparatus 10 audio subsystem further comprises a digital-to-analogue converter for converting digital audio signals from a processor 21 to a suitable analogue format. The digital-to-analogue converter (DAC) or signal processing means can in some embodiments be any suitable DAC technology.
Furthermore the audio subsystem can comprise in some embodiments a speaker. The speaker can in some embodiments receive the output from the digital-to-analogue converter and present the analogue audio signal to the user. In some embodiments the speaker can be representative of a headset, for example a set of headphones, or cordless headphones.
Although the apparatus 10 is described having both audio capture and audio presentation components, it would be understood that in some embodiments the apparatus 10 can comprise the audio capture or recording parts of the audio subsystem only such that in some embodiments the microphones (for audio capture) and/or ADC are present only.
In some embodiments the apparatus 10 comprises a processor 21. The processor 21 is coupled to the audio subsystem and specifically in some examples the analogue-to-digital converter for receiving digital signals representing audio signals from the microphones 11. The processor 21 can be configured to execute various program codes. The implemented program codes can comprise for example audio stabilization routines.
In some embodiments the apparatus further comprises a memory 22. In some embodiments the processor is coupled to memory 22. The memory can be any suitable storage means. In some embodiments the memory 22 comprises a program code section 23 for storing program codes implementable upon the processor 21. Furthermore in some embodiments the memory 22 can further comprise a stored data section 24 for storing data, for example data that has been encoded in accordance with the application or data to be encoded via the application embodiments as described later. The implemented program code stored within the program code section 23, and the data stored within the stored data section 24 can be retrieved by the processor 21 whenever needed via the memory-processor coupling.
In some further embodiments the apparatus 10 can comprise a user interface 15. The user interface 15 can be coupled in some embodiments to the processor 21. In some embodiments the processor can control the operation of the user interface and receive inputs from the user interface 15. In some embodiments the user interface 15 can enable a user to input commands to the electronic device or apparatus 10, for example via a keypad, and/or to obtain information from the apparatus 10, for example via a display which is part of the user interface 15. The user interface 15 can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the apparatus 10 and further displaying information to the user of the apparatus 10.
In some embodiments the apparatus further comprises a transceiver 13, the transceiver in such embodiments can be coupled to the processor and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network. The transceiver 13 or any suitable transceiver or transmitter and/or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupling.
The transceiver 13 can communicate with further devices by any suitable known communications protocol, for example in some embodiments the transceiver 13 or transceiver means can use a suitable universal mobile telecommunications system (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA).
In some embodiments the apparatus comprises a position or orientation sensor 16 configured to estimate the position of the apparatus 10. The position sensor 16 can, in some embodiments be a satellite positioning sensor such as a GPS (Global Positioning System), GLONASS or Galileo receiver.
In some embodiments the positioning sensor can be a cellular ID system or an assisted GPS system.
In some embodiments the apparatus 10 further comprises a direction or orientation sensor. The orientation/direction sensor can in some embodiments be an electronic compass, accelerometer, a gyroscope or be determined by the motion of the apparatus using the positioning estimate.
In some embodiments the apparatus comprises a visual imaging subsystem. The visual imaging subsystem can in some embodiments comprise at least one camera configured to capture image data. The at least one camera can comprise a suitable lensing or image focus element configured to focus an image on a suitable image sensor. In some embodiments the image sensor can be further configured to output digital image data to processor 21.
It is to be understood again that the structure of the electronic device 10 could be supplemented and varied in many ways.
With respect to
The apparatus concept uses motion parameters separated from video stabilization or video coding to assist in audio stabilization. These motion parameters can be converted into rotational and translational parameters using geometric transformations. The derived rotational and translational parameters can be in some embodiments filtered to differentiate between intended motion and unintended motion suitable for stabilization. The captured spatial audio can then be compensated for these motions.
In some embodiments as discussed herein the apparatus 10 can comprise a microphone 11 arrangement which includes a microphone array. The microphone arrangement or array can be configured to capture the acoustic waves and convert or generate suitable electrical audio signal representations of the captured acoustic waves. In the example shown in
With respect to
In some embodiments the apparatus comprises an audio capturer/recorder 101. The audio capturer/recorder 101 can be configured in some embodiments to process the microphone data to determine a spatial audio signal.
For example in some embodiments a spatial audio signal can include one audio channel signal and additional audio parameters which can then be used to artificially manipulate the single audio channel signal to form a two or more channel audio signals. In other words the audio capturer/recorder spatial audio signal is produced by signal processing which can determine or generate an audio signal which at some point later can be configured to restore multiple audio channels by using a stereo widening effect from a single audio signal.
The signal processing techniques can for example be the introduction of one or more of: ITD (interaural time difference), IPD (interaural phase difference), and IID (interaural intensity difference) values. It would be understood that in some embodiments other suitable signal processing techniques using different audible cues can be implemented.
In some embodiments the audio signals are provided by recording or capturing the acoustic waves at more than one microphone to generate at least two audio signals generated from different microphone or microphone output combinations. For example where at least two channel audio signals are required for playback of spatial audio signals (for example for playback within headsets or even loudspeakers) and the apparatus is configured to generate these audio signals representing the acoustic signal defining the sound field (the sound field itself being defined by one or more sound sources), then at least two microphone audio signals are provided to the processor.
In some embodiments the audio capturer can be configured to determine estimated audio source directions or orientations with respect to the apparatus based on the audio signals generated or determined by the microphones. In some embodiments the audio capturer/recorder 101 comprises an analogue-to-digital converter (ADC) to convert the analogue output from the microphones into a suitable digital format for processing.
In some embodiments the audio capturer/recorder 101 can further be configured to receive the digital audio signal from the audio sub-system arrangement. The audio capturer/recorder 101 can also be configured to perform any other suitable processing of the microphone data prior to the microphone data being passed to the audio stabilization apparatus 100. For example as shown in
In some embodiments the audio capturer/recorder 101 can be configured to perform spatial audio capture, in other words estimation of the direction of various audio sources.
Given three microphones in a plane it is possible to estimate the direction of a single source using geometry.
The example source S1 601 and the delay path between the source S1 601 and the microphones M1 M2 and M3 are also shown in
Where there is more than one source active along various directions the audio capturer/recorder 101 and in some embodiments the direction estimator 305 can be configured to solve the problem of determining directions by performing source estimation for each sub band within a given time frame. In such examples a dominant source for each of the sub bands are determined and the direction associated with the source for each sub band is further determined. When the sources are moving the estimated directions have to be estimated at periodic intervals.
In some embodiments the direction estimator 305 can be configured to comprise a framer configured to receive the audio signals from each audio signal and generate frames or groups of sample data. The frame can for example be 20 ms long. In some embodiments the frames can overlap with neighbouring frames. For example each frame can overlap neighbouring frames by 10 ms. In some embodiments the frames are windowed with a suitable windowing function.
In some embodiments the direction estimator 305 further comprises a time to frequency domain transformer. The time to frequency domain transformer can be any suitable transformer configured to perform a suitable transformation on the time domain frames audio signals from each microphone from the microphone array. In some embodiments the transformation can be a discrete Fourier Transform (DFT), a discrete cosine transform (DCT), a modified discrete cosine transform (MDCT), a quadrature mirror filter (QMF) or similar.
In some embodiments the direction estimator 305 filter comprises a sub-band filter configured to receive the frequency domain signals from each microphone and divide these into sub-bands. The sub-bands can be in some embodiments discrete or overlapping. In some embodiments the sub-bands are determined by psychoacoustic experimentation or can be divided into linear or non-linear band distributions.
In some embodiments the direction estimator 305 comprises a correlator, configured to receive each sub-hand for at least pairs of microphones. Furthermore the correlator can be configured to determine the delay value which produces the largest correlation value for a microphone audio signal pair sub-band.
In some embodiments the direction estimator comprises a pairwise direction estimator configured from the correlator delay value to be configured to determine a direction for the dominant source signal for the sub-band being analysed. Furthermore in some embodiments, the pairwise direction estimator can be configured to compare each dominant source sub-band direction estimate to determine a direction for each source.
The spatial analysis therefore can be configured in some embodiments to estimate the direction of audio sources by this method. However it would be understood that correlation can be performed in the time domain and spatial analysis estimate audio source directions using the time domain correlation values.
Furthermore it would be understood that by offsetting microphones in elevation as well as azimuth then the elevation as well as azimuth positions can be estimated in some embodiments. Furthermore more accurate and front/back discrimination can be estimated by comparing different pairs of microphones to produce more than one estimate.
The operation of spatial audio capture is shown in
As discussed herein where the device or camera is in motion then the estimated audio direction source will be affected by this motion. Using the direction estimation it is not possible to resolve whether all of the sources are moving during the capture or if the capture device is itself in motion without having additional information. This ambiguity can be solved as discussed in some embodiments by motion sensors or by analysing the video. Motion sensors can clearly indicate camera motion quantitatively and implementation can be readily made by looking at the integrated motion sensors. In some embodiments the motion of the device is determined by the video stabilizer 105 as described herein.
The apparatus furthermore can comprise at least one camera 33 the at least one camera is shown in this example by an array of cameras C1 to Cj. The camera output can be passed to a video stabilizer 105. The camera data can be in any suitable format.
The operation of capturing video from the camera or cameras is shown in
In some embodiments the apparatus comprises a video stabilizer 105. The video stabilizer can be configured to analyse the images received from the cameras 33 and determine image motion suitable to further perform video stabilization on the camera image data. The video stabilizer 105 can be configured to in some embodiments to perform global motion estimation.
In some embodiments the video stabilizer 105 can be configured to perform video motion estimation and compensation as part of a video coding standard such as mpeg4. The video stabilizer 105 motion estimation can in such embodiment compute or determine a localised motion vector for a block or macroblock of pixel data where neighbouring frames are analysed to provide information as to how much the macroblock is ‘moved’ to match an area in at least one of the neighbouring frames. The video stabilizer 105 and furthermore any motion estimation module in other words can in some embodiments as part of a video coding scheme generate several motion vector values for individual regions of the frame in the video. In some embodiments the video stabilizer can be configured to generate a global motion vector from the several local motion vectors. In some embodiments this can be determined by averaging the localised motion values across a frame. In some embodiments an outlier local motion vector value filter can furthermore be performed prior to averaging the localised motion values to reduce the effect of object motion errors within the apparatus caused motion estimation.
The determination of video motion is shown in
The global motion estimation can furthermore be used by the video stabilizer 105 the global motion is modelled as arising from the motion of the capture device or apparatus 10.
The global motion estimation therefore can in some embodiments be used to compensate for the motion be resynthesizing the later image frame to reduce the changes in the audio source ‘movement’ arising from the device motion. In some embodiments the global motion vector estimated parameters are passed to the motion vector to translational and rotation parameters convertor 107 to convert the global motion vector into translational, zoom and rotational motions of the camera. In some embodiments two successively captured image frames from the camera can then be analysed to estimate these parameters. However it would be appreciated that in some embodiments more than two captured image frames can be used.
With respect to
Track—moving left or right along the X axis
Boom—moving up or down along the Y axis
Dolly—moving forwards or backwards along the Z axis
Similarly rotational types of motion can be defined such as
Tilt—rotation about the X axis
Pan—rotation about the Y axis
Roll—rotation about the Z axis
Furthermore the operation of generating motion vectors from the video motion is shown in
In some embodiments the apparatus comprises a motion vector to translation and rotation parameter converter 107 which can be configured to perform parameter estimation by modelling the motion as a parametric motion model. In such examples the intensity of a pixel at location x, y of a capture frame j is defined as I1(x,y). A subsequent frame pixel at the same location x, y has the intensity defined by Ij+1(x, y). In some embodiments an affine transformation perspective model can be performed. The affine transformation perspective model requires eight parameters (a0, a7). The motion of the pixel from x to x′ and y to y′ is defined as
x′=(a0+a2.x+a3.y)/(a6.x+a7.y+1)
y′=(a1+a4.x+a5.y)/(a6.x+a7.y+1)
The video converter 107 can then be configured to estimate the parameter values of a0 to a7 to minimise the error value, in other wise such that the sum of the squared or sum of the absolute deviation between I1(x, y) and Ij+1(x′, y′) is minimised.
This for example is summarised in the following mathematical expression
It would be understood that the parameter estimation or conversion is mathematically under constrained and in some embodiments the converter 107 can solve this by implementing additional constraints such as defining a simpler affine, a pure translation or a translation zoom rotation constraint. These can mathematically represented by
In some embodiments these constraints allow the converter 107 to for find the optimal solution that can be implemented in real time. The parameters of the parametric motion model can be used to determine the rotations and translations of the device.
The algorithms used to estimate the parameters are generally known and will not be described in detail further.
The converter can then be configured to output to the audio stabilizer 103 suitable motion estimation values for translation and rotation of the apparatus.
The operation of generating rotational and translational parameters is shown in
In some embodiments the apparatus comprises an audio stabilizer 103. The audio stabilizer can be configured to receive the audio captured (or recorded) information and the translation and rotation parameters to determine whether to apply audio stabilization and to apply audio stabilization to the audio source directional estimator where determined to be required.
With respect to
The translational and rotational parameter converter 107 as described herein can be configured to generate translational and rotational parameters passed to the audio stabilizer 103.
The operation of receiving the translational and rotational parameters is shown in
Furthermore the audio stabilizer 103 can be configured to comprise in some embodiments a rotational high pass filter 301 configured to receive the rotational parameters and a translational high pass filter 303 configured to receive the translational parameters.
The rotational high pass filter 301 and the translational high pass filter 303 can be configured to high pass filter the rotational and translational parameters respectively in order to remove any intended motion which is mostly smooth and low frequency from the unintended or ‘jerky’ high frequency motion. The Rotational high pass filter 301 can therefore be configured to output filtered rotational parameters to a rotation offset compensator 307. The translational high pass filter 303 can be configured to output filtered translational parameters to a translational offset compensator 304.
The operation of high pass filtering the rotational and translational parameters are shown in
The directional estimator 305 can be configured to output directional estimations for each (audio band) audio source to the rotational offset compensator 307.
The operation of receiving the directional estimates for each source is shown in
In some embodiments the audio stabilizer 103 comprises a rotational offset compensator 307. The rotational offset compensator 307 is configured to receive the output of the rotational high pass filter 301 in the form of a rotational parameter estimating the rotation experienced by the apparatus and the directional estimations for each source from the directional estimator 305. The compensation performed by the rotational offset compensator 307 can be a constant offset for all of the angles of the estimated direction. In other words each of the determined sources are offset with the inverse of the filtered rotational parameter value.
The output of the rotational offset compensator 307 can then be passed to a (non-linear) translational offset compensator 309.
The operation of compensating for rotational offset is shown in
The audio stabilizer 103 in some embodiments comprises a translational (non-linear) offset compensator 309 can be configured to determine whether an offset is to be applied with regards to any translational error and to determine and apply an offset when required. The translational offset compensator 309 is configured to receive the output of the rotational offset compensator 307 and the high pass filtered translational parameters from the high pass filter 303.
The non-linear offset compensator 309 can be configured to perform compensation, depending on the type of motion.
With respect to
Mathematically these determination and offset application operations can be summarised as
Where the original coordinates of source are defined by: r cos(θ)+j r sin(θ)
Let the displacement along x-axis be x.
The new angle (wrongly estimated because of displacement)
ϕ=a tan(r sin(θ)/(r cos(θ)−x))
In some embodiment the radius of the sources r can be defined as being the Far field radius, r as the value of the displacement x is typically small with respect to the value of r.
So θ can be computed by solving the above equation.
The operation of compensating for non-linear translational offset is shown in
The output of the non-linear offset compensator 309 can then be passed to the HRTF or HRIR processor 311. In some embodiments the positional information or other stabilized spatial audio signal may be output or stored in memory. The output stabilized spatial audio signals can in some embodiments be processed by a further or listening device to regenerate the spatially stabilized audio signal which can be listened to by a user.
In some embodiments the apparatus comprises a HRTF/HRIR processor configured to select a set of HRTF/HRIR values for each audio source/sub-band depending on the positional estimate of each audio source/sub-band. These can be determined by a stored database or look up table. The HRTF/HRIR values for each can then be output to a listening device where they can be summed to create a synthesized audio signal.
The operation of generating and applying a suitable head related transfer function or head related impulse response per audio source/sub-band is shown in
In some embodiments the parameters for stabilization can be estimated using any other suitable sensor.
Furthermore the actual stabilization of audio signals can be performed using various means.
Although the above has been described with regards to audio signals, or audio-visual signals it would be appreciated that embodiments may also be applied to audio-video signals where the audio signal components of the recorded data are processed according to embodiments of the application.
It shall be appreciated that the term user equipment is intended to cover any suitable type of wireless user equipment, such as mobile telephones, portable data processing devices or portable web browsers.
Furthermore elements of a public land mobile network (PLMN) may also comprise apparatus as described above.
In general, the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
The embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.
The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), gate level circuits and processors based on multi-core processor architecture, as non-limiting examples.
Embodiments of the inventions may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
Programs, such as those provided by Synopsys, Inc. of Mountain View, Calif. and Cadence Design, of San Jose, Calif. automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules. Once the design for a semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or “fab” for fabrication.
The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of the exemplary embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention as defined in the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 4248/CHE/2011 | Dec 2011 | IN | national |
This application is a continuation of U.S. application Ser. No. 14/362,663, filed on Jun. 4, 2014, which was originally filed as PCT Application No. PCT/FI2012/051209 filed on Dec. 5, 2012, which claims priority to IN application 4248/CHE/2011 filed on Dec. 7, 2011, the disclosures of which are incorporated by reference in their entireties.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5687239 | Inanaga | Nov 1997 | A |
| 5973733 | Gove | Oct 1999 | A |
| 20030227476 | Wilcock | Dec 2003 | A1 |
| 20040155968 | Cheatle | Aug 2004 | A1 |
| 20050147261 | Yeh | Jul 2005 | A1 |
| 20060170783 | Stavely et al. | Aug 2006 | A1 |
| 20070009120 | Algazi et al. | Jan 2007 | A1 |
| 20080243278 | Dalton et al. | Oct 2008 | A1 |
| 20090028347 | Duraiswami | Jan 2009 | A1 |
| 20100128892 | Chen et al. | May 2010 | A1 |
| 20100150355 | Kon | Jun 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 0676899 | Oct 1995 | EP |
| 2257943 | Aug 2010 | EP |
| 2342290 | Aug 2011 | EP |
| WO-2006103581 | Oct 2006 | WO |
| Entry |
|---|
| Deo, S. et al., “Experiments in Spatial Mobile Audio-Conferencing”, in proc. Int. Conference on mobile technology, Applications and Systems, Sep. 2007, pp. 447-451, Singapore. |
| Number | Date | Country | |
|---|---|---|---|
| 20180310117 A1 | Oct 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14362663 | US | |
| Child | 16017079 | US |
