Patent Application
20230007435
The present invention relates to audio processing and, particularly, to audio signal processing of sound scenes occurring, for example, in virtual reality or augmented reality applications.
Geometrical Acoustics are applied in auralization, i.e., real-time and offline audio rendering of auditory scenes and environments. This includes Virtual Reality (VR) and Augmented Reality (AR) systems like the MPEG-I 6-DoF audio renderer. For rendering complex audio scenes with six degrees of freedom (DoF), the field of Geometrical Acoustics is applied, where the propagation of sound data is modeled using methods known from optics such as ray-tracing. Particularly, the reflections at walls are modeled based on models derived from optics, in which the angle of incidence of a ray that is reflected at the wall results in a reflection angle being equal to the angle of incidence.
Real-time auralization systems, like the audio renderer in a Virtual Reality (VR) or Augmented Reality (AR) system, usually render early reflections based on geometry data of the reflective environment. A Geometrical Acoustics method like the image source method in combination with ray-tracing is then used to find valid propagation paths of the reflected sound. These methods are valid, if the reflecting planar surfaces are large compared to the wave length of incident sound. The distance of the reflection point on the surface to the boundaries of the reflecting surface also has to be large compared to the wave length of incident sound.
Sound in Virtual Reality (VR) and Augmented Reality (AR) is rendered for a listener (user). The inputs to this process are (typically anechoic) audio signals of sound sources. A multitude of signal processing techniques is then applied to these input signals, simulating and incorporating relevant acoustic effects such as sound transmission through walls/windows/doors, diffraction around and occlusion by solid or permeable structures, the propagation of sound over longer distances, reflections in half-open and enclosed environments, Doppler shifts of moving sources/listeners, etc. The output of the audio rendering are audio signals that create a realistic, three-dimensional acoustic impression of the presented VR/AR scene when delivered to the listener via headphones or loudspeakers.
The rendering is performed listener-centric and the system has to react to user motion and interaction instantaneously, without significant delays. Hence the processing of the audio signals has to be performed in real-time. User input manifests in changes of the signal processing (e.g., different filters). These changes are to be incorporated in the rendering without audible artifacts.
Most audio renderers used a pre-defined fixed signal processing structure (block diagram applied to multiple channels, see for example [1]) with a fixed computation time budget for each individual audio source (e.g. 16× object source, 2× third-order Ambisonics). These solutions enable rendering dynamic scenes, by updating location-dependent filters and reverb parameters, but they do not allow for sources to be dynamically added/removed during runtime.
Moreover, a fixed signal processing architecture can be rather ineffective when rendering complex scenes, as a large number of sources has to be processed in the same way. Newer rendering concepts facilitate clustering and level-of-detail concepts (LOD), where, depending on the perception, sources are combined and rendered with different signal processing. Source clustering (see [2]) can enable renderers to handle complex scenes with hundreds of objects. In such a setup, the cluster budget is still fixed which may lead to audible artifacts of extensive clustering in complex scenes.
According to an embodiment, an apparatus for rendering a sound scene may have a first pipeline stage including a first control layer and a reconfigurable first audio data processor, wherein the reconfigurable first audio data processor is configured to operate in accordance with a first configuration of the reconfigurable first audio data processor; a second pipeline stage located, with respect to a pipeline flow, subsequent to the first pipeline stage, the second pipeline stage including a second control layer and a reconfigurable second audio data processor, wherein the reconfigurable second audio data processor is configured to operate in accordance with a first configuration of the reconfigurable second audio data processor; and a central controller for controlling the first control layer and the second control layer in response to the sound scene, so that the first control layer prepares a second configuration of the reconfigurable first audio data processor during or subsequent to an operation of the reconfigurable first audio data processor in the first configuration of the reconfigurable first audio data processor, or so that the second control layer prepares a second configuration of the reconfigurable second audio data processor during or subsequent to an operation of the reconfigurable second audio data processor in the first configuration of the reconfigurable second audio data processor, and wherein the central controller is configured to control the first control layer or the second control layer using a switch control to reconfigure the reconfigurable first audio data processor to the second configuration for the reconfigurable first audio data processor or to reconfigure the reconfigurable second audio data processor to the second configuration for the reconfigurable second audio data processor at a certain time instant.
Another embodiment may have a method of rendering a sound scene using an apparatus including a first pipeline stage including a first control layer and a reconfigurable first audio data processor, wherein the reconfigurable first audio data processor is configured to operate in accordance with a first configuration of the reconfigurable first audio data processor; a second pipeline stage located, with respect to a pipeline flow, subsequent to the first pipeline stage, the second pipeline stage including a second control layer and a reconfigurable second audio data processor, wherein the reconfigurable second audio data processor is configured to operate in accordance with a first configuration of the reconfigurable second audio data processor, the method having the steps of: controlling the first control layer and the second control layer in response to the sound scene, so that the first control layer prepares a second configuration of the reconfigurable first audio data processor during or subsequent to an operation of the reconfigurable first audio data processor in the first configuration of the reconfigurable first audio data processor, or so that the second control layer prepares a second configuration of the reconfigurable second audio data processor during or subsequent to an operation of the reconfigurable second audio data processor in the first configuration of the reconfigurable second audio data processor, and controlling the first control layer or the second control layer using a switch control to reconfigure the reconfigurable first audio data processor to the second configuration for the reconfigurable first audio data processor or to reconfigure the reconfigurable second audio data processor to the second configuration for the reconfigurable second audio data processor at a certain time instant.
Another embodiment may have a non-transitory digital storage medium having a computer program stored thereon to perform the method of rendering a sound scene using an apparatus including a first pipeline stage including a first control layer and a reconfigurable first audio data processor, wherein the reconfigurable first audio data processor is configured to operate in accordance with a first configuration of the reconfigurable first audio data processor; a second pipeline stage located, with respect to a pipeline flow, subsequent to the first pipeline stage, the second pipeline stage including a second control layer and a reconfigurable second audio data processor, wherein the reconfigurable second audio data processor is configured to operate in accordance with a first configuration of the reconfigurable second audio data processor, the method having the steps of: controlling the first control layer and the second control layer in response to the sound scene, so that the first control layer prepares a second configuration of the reconfigurable first audio data processor during or subsequent to an operation of the reconfigurable first audio data processor in the first configuration of the reconfigurable first audio data processor, or so that the second control layer prepares a second configuration of the reconfigurable second audio data processor during or subsequent to an operation of the reconfigurable second audio data processor in the first configuration of the reconfigurable second audio data processor, and controlling the first control layer or the second control layer using a switch control to reconfigure the reconfigurable first audio data processor to the second configuration for the reconfigurable first audio data processor or to reconfigure the reconfigurable second audio data processor to the second configuration for the reconfigurable second audio data processor at a certain time instant, when said computer program is run by a computer.
The present invention is based on the finding that, for the purpose of rendering a complex sound scene with many sources in an environment, where frequent changes of the sound scene can occur, a pipeline-like rendering architecture is useful. The pipeline-like rendering architecture comprises a first pipeline stage comprising a first control layer and a reconfigurable first audio data processor. Furthermore a second pipeline stage that is located, with respect to a pipeline flow, subsequent to the first pipeline stage is provided.
This second pipeline stage again comprises a second control layer and a reconfigurable second audio data processor. Both the first and the second pipeline stages are configured to operate in accordance with a certain configuration of the reconfigurable first audio data processor at a certain time in the processing. In order to control the pipeline-architecture, a central controller for controlling the first control layer and the second control layer is provided. The control takes place in response to the sound scene, i.e., in response to an original sound scene or a change of the sound scene.
In order to achieve a synchronized operation of the apparatus among all pipeline stages, and when a reconfiguration task for the first or the second reconfigurable audio data processor is required, the central controller controls the control layers of the pipeline stages so that the first control layer or the second control layer prepares another configuration such as a second configuration of the first or the second reconfigurable audio data processor during or subsequent to an operation of the reconfigurable audio data processor in the first configuration. Hence, a new configuration for the reconfigurable first or second audio data processor is prepared while the reconfigurable audio data processor belonging to this pipeline stage is still operating in accordance with a different configuration or is configured in a different configuration in case the processing task with the earlier configuration is already done. In order to make sure that both pipeline stages operate synchronized in order to obtain the so-called “atomic operation” or “atomic updates”, the central controller controls the first and the second control layers using a switch control to reconfigure the reconfigurable first audio data processor or the reconfigurable second audio data processor to the second different configuration at a certain time instant. Even when only a single pipeline stage is reconfigured, embodiments of the present invention nevertheless guarantee that due to the switch control at the certain time instance, the correct audio sample data is processed in the audio workflow via the provision of the audio stream input or output buffers included in the corresponding render lists.
Advantageously, the apparatus for rendering the sound scene has a higher number of pipeline stages than a first and a second pipeline stage, but already in a system with a first and a second pipeline stage and no additional pipeline stage, the synchronized switching of the pipeline stages in response to the switch control is necessary for obtaining an improved high quality audio rendering operation that, at the same time, is highly flexible.
In particular, in complex virtual reality scenes, where a user can move in three directions and where, additionally, the user can move her or his head in three additional directions, i.e., in a six degrees of freedom (6-DoF) scenario, frequent and sudden changes of filters in the rendering pipeline, for example for switching from one head-related transfer function to another head-related transfer function in case of the moving of the listener's head or the walking around of the listener requires such a change of head-related transfer functions will take place.
Other problematic situations with respect to a flexible rendering with high quality are that when a listener moves around in a virtual or augmented reality scene, the number of sources to be rendered will change all the time. This can for example occur due to the fact that certain image sources become visible at a certain position of the user or due to the fact that additional diffraction effects have to be considered. Furthermore, other procedures are that in certain situations, a clustering of many different closely spaced sources is possible while, when the user moves closer to these sources, then the clustering is not feasible anymore, since the user is so close that it is necessary that each source is rendered at its distinct position. Thus, such audio scenes are problematic in that changing filters or a changing number of sources to be rendered or, in general, changing parameters is required all the time. On the other hand, it is useful to distribute the different operations for rendering onto different pipeline stages so that an efficient and high speed rendering is possible, in order to make sure that a real time rendering in complex audio environments is achievable.
A further example for a thoroughly changing parameter is that as soon as a user comes closer to a source or an image source, the frequency-dependent distance attenuation and propagation delay changes with the distance between the user and sound source. Similarly, the frequency-dependent characteristics of the reflective surface may change depending on the configuration between the user and a reflecting object. Furthermore, depending on whether a user is close to a diffracting object or further away from the diffracting object or at a different angle, the frequency-dependent diffraction characteristics will also change. Thus, if all these tasks are distributed to different pipeline stages, continuing changes of these pipeline stages have to be possible and have to be performed synchronously. All this is achieved by means of the central controller that controls the control layers of the pipeline stages to prepare for a new configuration during or subsequent to an operation of the corresponding configurable audio data processor in the earlier configuration. In response to a switch control for all stages in the pipeline effected by a control update via the switch control, the reconfiguration takes place a certain time instant being identical or being at least very similar among the pipeline stages in the apparatus for rendering the sound scene.
The present invention is advantageous since it allows a high quality real-time auralization of auditory scenes with dynamically changing elements, for example moving sources and listeners. Thus, the present invention contributes to the achievement of perceptually convincing soundscapes that are a significant factor for the immersive experience of a virtual scene.
Embodiments of the present invention apply separate and concurrent workflows, threads or processes that very well fit to the situation of rendering dynamic auditory scenes.
Executions of the control workflow vary in run time, depending on which necessary computations a change is triggered, similar to the frame loop in visual computing. Advantageous embodiments of the invention are advantageous in that such variations of executions of the control workflow do not at all adversely affect the processing workflow, which is concurrently executed in the background. As real-time audio is processed block-wise, the acceptable computation time of the processing workflow is typically limited to usually a few milliseconds.
The processing workflow that is concurrently executed in the background is processed by the first and the second reconfigurable audio data processors, and the control workflow is initiated by the central controller and is then implemented, on the pipeline stage level, by the control layers of the pipeline stages parallel to the background operation of the processing workflow. The interaction workflow is implemented, on the pipelined rendering apparatus level, by an interface of the central controller to external devices such as a head tracker or a similar device or is controlled by the audio scene having a moving source or geometry that represents a change of the sound scene as well as a change in the user orientation or location, i.e., generally in the user position.
The present invention is advantageous in that multiple objects in the scene can be changed coherently and sample synchronously due to the centrally controlled switch control procedure. Furthermore, this procedure allows so-called atomic updates of multiple elements that have to be supported by the control workflow and the processing workflow in order to not interrupt the audio processing due to changes on the highest level, i.e., in the interaction workflow or in the intermediate level, i.e., the control workflow.
Advantageous embodiments of the present invention relate to the apparatus for rendering the sound scene implementing a modular audio rendering pipeline, where the necessary steps for auralization of virtual auditory scenes are partitioned into several stages which are each independently responsible for certain perceptual effects. The individual partitioning into at least two or advantageously even more individual pipeline stages depends on the application and is advantageously defined by the author of the rendering system as is illustrated later.
The present invention provides a generic structure for the rendering pipeline that facilitates parallel processing and dynamic reconfiguration of the signal processing parameters depending on the current state of the virtual scene. In that process, embodiments of the present invention ensure
Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:
In response to the sound scene means in response to a whole scene input at a certain initialization or beginning time instant or in response to sound scene changes that, together with a preceding scene existing before the sound scene changes again, represent a full sound scene that is to be processed by the central controller 100. In particular, the central controller 100 controls the first and the second control layers and if available, any other control layers such the n-th control layer 401 so that a new or second configuration of the first, the second and/or the n-th reconfigurable audio data processor is prepared while the corresponding reconfigurable audio data processor operates in the background in accordance with an earlier or first configuration. For this background mode, it is not decisive whether the reconfigurable audio data processor still operates, i.e., receives input samples and calculates output samples. Instead, it can also be the situation that a certain pipeline stage has already completed its tasks. Thus, the preparation of the new configuration takes place during or subsequent to an operation of the corresponding reconfigurable audio data processor in the earlier configuration.
In order to make sure that atomic updates of the individual pipeline stages 200, 300, 400 are possible, the central controller outputs a switch control 110 in order to reconfigure the individual reconfigurable first or second audio data processors at a certain time instant. Depending on the specific application or sound scene change, only a single pipeline stage can be reconfigured at the certain time instant or two pipeline stages such as pipeline stages 200, 300 are both reconfigured at the certain time instant or all pipeline stages of the whole apparatus for rendering the sound scene or only a subgroup having more than two pipeline stages but less than all pipeline stages can also be provided with the switch control to be reconfigured at the certain time instant. To this end, the central controller 100 has a control line to each control layer of the corresponding pipeline stage in addition to the processing workflow connection serially connecting the pipeline stages. Furthermore, the control workflow connection that is discussed later can either be provided also via the first structure for the central switch control 110. In advantageous embodiments, however, the control workflow is also performed via the serial connection among the pipeline stages so that the central connection between each control layer of the individual pipeline stage and the central controller 100 is only reserved for the switch control 110 to obtain atomic updates and, therefore, a correct and high quality audio rendering even in complex environments.
The following section describes a general audio rendering pipeline, composed of independent render stages, each with separated, synchronized control and processing workflows (
Every render stage has a control part and a processing part with separate inputs and outputs corresponding to the control and processing workflow respectively. In the pipeline, the outputs of one render stage are the inputs of a succeeding render stage, while a common interface guarantees that render stages can be reorganized and replaced, depending on the application.
This common interface is described as a flat list of render items that is provided to the render stage in the control workflow. A render item combines processing instructions (i.e., metadata, such as position, orientation, equalization, etc.) with an audio stream buffer (single- or multichannel). The mapping of buffers to render items is arbitrary, such that multiple render items can refer to the same buffer.
Every render stage ensures that succeeding stages can read the correct audio samples from the audio stream buffers corresponding to the connected render items at the rate of the processing workflow. To achieve this, every render stage creates a processing diagram from the information in the render items that describes the necessary DSP steps and its input and output buffers. Additional data may be required to construct the processing diagram (e.g., geometry in the scene or personalized HRIR sets) and is provided by the controller. The processing diagrams are lined up for synchronization and handed over to the processing workflow simultaneously for all render stages, after the control update is propagated through the whole pipeline. The exchange of processing diagrams is triggered without interfering with the real-time audio block rate, while the individual stages have to guarantee that no audible artifacts occur due to the exchange. If a render stage only acts on metadata, the DSP workflow can be a no-operation.
The controller maintains a list of render items corresponding to actual audio sources in the virtual scene. In the control workflow, the controller starts a new control update by passing a new list of render items to the first render stage, atomically cumulating all metadata changes resulting from user interaction and other changes in the virtual scene. Control updates are triggered at a fixed rate that may depend on the available computational resources, but only after the previous update is finished. A render stage creates a new list of output render items from the input list. In that process, it can modify existing metadata (e.g., add an equalization characteristic), as well as add new and deactivate or remove existing render items. Render items follow a defined life cycle (
In a real-time application, the processing workflow is triggered by the callback from the audio hardware. When a new block of samples is requested, the controller fills the buffers of the render items it maintains with input samples (e.g., from disk or from incoming audio streams). The controller then triggers the processing part of the render stages sequentially, which act on the audio stream buffers according to their current processing diagrams.
The render pipeline may contain one or more spatializers (
Compared to the state of the art, the inventive audio rendering pipeline can handle highly dynamic scenes with the flexibility to adapt processing to different hardware or user requirements. In this section, several advances over established methods are listed.
A practical example for a rendering pipeline to create virtual acoustic environments for VR applications may contain the following render stages in the given order (see also
Processing part: filtering, fractional delay lines, etc.
Subsequently,
The pipeline stage 200 receives, as an input via an input interface, an input render list 500 and outputs, via an output interface, an output render list 600. In case of a directly subsequent connection of the second pipeline stage 300 in
Each render list 500 comprises a selection of render items illustrated by a column in the input render list 500 or the output render list 600. Each render item comprises a render item identifier 501, render item metadata 502 indicated as “x” in
Alternatively, the render list can comprise audio stream buffers representing physical memory portions, but it is advantageous to implement the audio stream buffers 503 as said references to a certain physical memory.
Similarly, the output render list 600 again has one column for each render item and the corresponding render item is identified by a render item identification 601, corresponding metadata 602 and audio stream buffers 603. Metadata 502 or 602 for the render items can comprise a position of a source, a type of a source, an equalizer associated with a certain source or, generally, a frequency-selective behavior associated with a certain source. Thus, the pipeline stage 200 receives, as an input, the input render list 500 and generates, as an output, the output render list 600. Within the DSP 202, audio sample values identified by the corresponding audio stream buffers are processed as required by the corresponding configuration of the reconfigurable audio data processor 202, for example as indicated by a certain processing diagram generated by the control layer 201 for the digital signal processor 202. Since the input render list 500 comprises, for example, three render items, and the output render list 600 comprises, for example, four render items, i.e., more render items than the input, the pipeline stage 202 could perform an upmix, for example. Another implementation could, for example, be that the first render item with the four audio signals is downmixed into a render item with a single channel. The second render item could be left untouched by the processing, i.e., could, for example, be only copied from the input to the output, and the third render item could also be, for example, left untouched by the render stage. Only the last output render item in the output render list 600 could be generated by the DSP, for example, by combining the second and the third render items of the input render list 500 into a single output audio stream for the corresponding audio stream buffer for the fourth render item of the output render list.
If a render item is active, it is processed and it can be either immediately deactivated or normally deactivated. In the latter case, a deactivate state 514 is obtained and a fade out procedure is performed in order to come from the deactivate state 514 to the inactive state 513. In case of an immediate deactivation, a direct transition from state 512 to state 513 is performed. The inactive state can either come back to an immediate reactivation or into a reactivate instruction in order to arrive at the activate state 511 or, if neither a reactivate control nor an immediate reactivation control is obtained, control can proceed to the disposed output node 515.
Subsequently,
Two object audio sources are represented as two Render Items (RIs). The Directivity Stage is responsible for directional filtering of the sound source signal. The Propagation Stage is responsible for rendering a propagation delay based on the distance to the listener. The Binaural Spatializer is responsible for binauralization and downmixing the scene to a binaural stereo signal.
At a certain control step, the RI positions change with regard to previous control steps, thus requiring changes in the DSP processing of each individual stage. The acoustic scene should update synchronously, so that e.g. the perceptual effect of a changing distance is synchronous with the perceptual effect of a change in the listener-relative angle of incidence.
The Render List is propagated through the complete pipeline at each control step. During the control step, the parameters of the DSP processing stay constant for all stages, until the last Stage/Spatializer has processed the new Render List. After that, all Stages change their DSP parameters synchronously at the beginning of the next DSP step.
It is each Stage's responsibility to update the parameters of the DSP processing without noticeable artifacts (e.g. output crossfade for FIR filter updates, linear interpolation for delay lines).
RIs can contain fields for metadata pooling. This way, for example the Directivity stage does not need to filter the signal itself, but can update an EQ field in the RI metadata. A subsequent EQ stage then applies the combined EQ field of all preceding stages to the signal.
Particularly, the input render list is the same as the output render list 500 in the
In the first render or pipeline stage 200 which is the directivity stage in this example, a first FIR filter 211 is applied to the first render item and another directivity filter or FIR filter 212 is applied to the second render item 512. Furthermore, within the second render stage or second pipeline stage 33, which is the propagation stage in this embodiment, a first interpolating delay line 311 is applied to the first render item 511, and another second interpolating delay line 312 is applied to the second render item 512.
Furthermore, in the third pipeline stage 400 connected subsequent to the second pipeline stage 300, a first stereo FIR filter 411 for the first render item 511 is used, and a second FIR filter 412 or the second render item 512 is used. In the binaural specializer, a downmix of the two filter output data is performed in the adder 413 in order to have the binaural output signal. Thus, the two object signals indicated by the render items 511, 512, binaural signal at the output of the adder 413 (not illustrated in
In a complex auditory scene, the Render List may contain many RIs that are perceptually close-by, i.e. their difference in position cannot be distinguished by the listener. To reduce the computational load for subsequent Stages, a Clustering Stage may replace multiple individual RIs by a single representative RI.
At a certain control step, the scene configuration may change so that the clustering is no longer perceptually feasible. In this case, the Clustering Stage will become inactive and passes the Render List unchanged.
When some incoming RIs are clustered, the original RIs are deactivated in the outgoing Render List. The reduction is opaque for subsequent Stages and the Clustering Stage needs to guarantee that as soon as the new outgoing Render List becomes active, valid samples are provided in the buffers associated with the representative RI.
When the cluster becomes infeasible, the new outgoing Render List of the Clustering stage contains the original, unclustered RIs. Subsequent stages need to process them individually starting with the next DSP parameter change (e.g. by adding a new FIR filter, delay line, etc. to their DSP diagram).
In the
The first render item 521 comes from an output of the FIR filter 221. The second render item 522 is generated by an output of the FIR filter 222 of the directivity stage, and the third render item 523 is obtained at the output of the FIR filter 223 of the first pipeline stage 200 being the directivity stage. It is to be noted that, when it is outlined that a render item is at the output of a filter, this refers to the audio samples for the audio stream buffer of the corresponding render item.
In the example in
Again, the third pipeline stage in
In geometric room acoustics, it may be beneficial to model reflected sound as image sources (i.e. two point sources with the same signal and their position mirrored on a reflective surface). If the configuration between listener, source and a reflecting surface in the scene is favorable for reflection, the Early Reflections Stage adds a new RI to its outgoing Render List that represents the image source.
The audibility of image sources typically changes rapidly when the listener moves. The Early Reflections Stage can activate and deactivate the RIs at each control step and subsequent Stages should adjust their DSP processing accordingly.
Stages after the Early Reflections Stage can process the reflection RI normally, as the Early Reflections Stage guarantees that the associated audio buffer contains the same samples as the original RI. This way, perceptual effects like propagation delay can be handled for original RIs and reflections alike without explicit reconfiguration. For increased efficiency when the activity status of RIs changes often, the Stages can keep required DSP artifacts (like FIR filter instances) for reuse.
Stages can handle Render Items with certain properties differently. For example, a Render Item created by a Reverb Stage (depicted by item 532 in
The render list 500 comprises a first render item 531 and a second render item 532. Each has a single audio stream buffer that can carry mono or a stereo signal, for example.
The first pipeline stage 200 is a reverb stage that has, for example, generated render item 531. The render list 500 additionally has render item 532. In the earlier deflection stage 300, render item 531 and, particularly, the audio samples thereof are represented by an input 331 for a copy operation. The input 331 of the copy operation is copied into the output audio stream buffer 331 corresponding to the audio stream buffer of render item 631 of the output render list 600. Furthermore, the other copied audio object 333 corresponds to the render item 633. Furthermore, as stated, render item 532 of the input render list 500 is simply copied or fed through to the render item 632 of the output render list.
Then, in the third pipeline stage that is, in the above example, binaural spatializer, the stereo FIR filter 431 is applied to the first render item 631, the stereo FIR filter 433 is applied to the second render item 633, and the third stereo FIR filter 432 is applied to the third render item 632. Then, the contributions of all three filters are correspondingly added, i.e., channel-by-channel by the adder 413 and the output of the adder 413 are a left signal on the one hand and a right signal on the other hand for a headphone or, generally, for a binaural reproduction.
At certain instants that may be time instants that are irregular and depend on a listener behavior, as, for example, determined by a head tracker, a central controller receives an audio scene or an audio scene change as indicated by step 91. In step 92, the central controller determines a render list for each pipeline stage under the control of the central controller. Particularly, the control updates that are then sent from the central controller to the individual pipeline stages are triggered at regular rates, i.e., with a certain update rate or update frequency.
As illustrated in step 93, the central controller sends the individual render list to each respective pipeline stage control layer. This can be done centrally via the switch control infrastructure, for example, but it is advantageous to perform this serially via the first pipeline stage and from there to the next pipeline stage and so on as indicated by the control workflow line 130 of
In step 95, the control layer receives the switch control from the central controller and reconfigures its associated reconfigurable audio data processor to the new configuration. This control layer switch control reception in step 95 can take place in response to a reception of a ready message of all pipeline stages by the central controller or can be done in response to a sending out of the central controller of the corresponding switch control instruction after a certain time duration with respect to the update trigger as done in step 93. Then, in step 96, the control layer of the corresponding pipeline stage cares for the fade-out of items that do not exist in the new configuration or cares for the fade-in of new items that have not existed in the old configuration. In case of the same objects in the old configuration and the new configuration, and in case of meta data changes such as with respect to the distance to a source or a new HRTF filter due to a movement of the listener's head and so on, a cross-fade of filters or a cross-fade of filtered data in order to smoothly come from one distance, for example, to the other distance is also controlled by the control layer in step 96.
The actual processing in the new configuration is started via a call back from the audio hardware. Thus, in other words, the processing workflow is triggered subsequent to the reconfiguration to the new configuration in a advantageous embodiment. When a new block of samples is requested, the central controller fills the audio stream buffers of the render items it maintains with input samples such as from a disc or from incoming audio streams. The controller then triggers the process part of the render stages, i.e., the reconfigurable audio data processors sequentially, and the reconfigurable audio data processors act on the audio stream buffers according to their current configuration, i.e., to their current processing diagrams. Thus, the central controller fills the audio stream buffers of the first pipeline stage in the apparatus for rendering a sound scene. However, there will also be the situation, where input buffers of other pipeline stages are to be filled from the central controller. This situation can, for example, arise when there have not been spatially extended sound sources in earlier situations of the audio scene. Thus, in this earlier situation, stage 300 of
While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations and equivalents as fall within the true spirit and scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 20163153.8 | Mar 2020 | EP | regional |
This application is a continuation of copending International Application No. PCT/EP2021/056363, filed Mar. 12, 2021, which is incorporated herein by reference in its entirety, and additionally claims priority from European Application No. EP 20 163 153.8, filed Mar. 13, 2020, which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/EP2021/056363 | Mar 2021 | US |
| Child | 17940871 | US |
