Patent Application
20230274756
One aspect of the disclosure relates to dynamically changing audio properties that are associated with an application.
Computer systems, including mobile devices or other electronic systems, can run one or more applications that play audio to a user. For example, a computer can launch a movie player application that, during runtime, plays sounds from the movie to the user. Other applications, such as video calls, phone calls, alarms, and more, can be associated with audio playback.
An operating system can present a user interface or display to a user that shows one or objects to a user, where that object (e.g., an icon, a window, a picture, an animated graphic, etc.) is representative of the application. For example, a movie player application may play in a ‘window’ that allows the user to view and control playback. Operating systems can manage multiple applications at a given time.
System level rules can be enforced for adjusting audio parameters of an application based on size of an object. The object, e.g., an icon, a window, a picture, an animated graphic, etc., can represent the underlying application. The object can be presented on a 2D display, or as a virtual object in an extended reality (XR) environment.
Further, audio that is associated with the application can be rendered spatially so that the object represents one or more sound sources. For example, if a media player window is presented to a user that shows a movie, and the media player window is shown as a small window, then the audio parameters can be determined so that audio that is associated with the media-player window (e.g., a movie audio track) is rendered so as to be perceived to originate from a small source. If a user adjusts a size of the media player window to be bigger, then audio parameters are dynamically adjusted to reflect the size of the window. In this case, the movie audio can sound like it originates from a larger, more complex, or impressive sound source. Audio parameters that are determined based on object size can include, for example, dynamic range, directivity pattern, frequency response, sound power, and/or other audio parameters.
In some aspects, a method, system or computing device that performs the method is described. The method includes maintaining metadata associated with one or more applications. The metadata specifies a size of an object (e.g., an icon, a window, a picture, a computer generated graphic, an animation, and/or other object) that is associated with the application. The object is presented to a user, for example, on a display. Based on a size of the object, one or more audio parameters are determined or modified. An audio parameter can include at least one of: a dynamic range, a directivity pattern, a frequency response, a sound power, a frequency range, a pitch, a timbre, a number of output audio channels, and a reverberation.
The audio parameter can be applied to render and/or mix audio that is associated with the application. In such a manner, objects that are shown to a user that appear to be large can also sound as if they are large (e.g., multiple sound sources, large dynamic range, bass, etc.). Conversely, objects that are shown to a user that are tiny can sound tiny (e.g., a single point source, small dynamic range, etc.). By enforcing these rules, real world acoustic behaviors of objects are mimicked to maintain plausibility. A user can also resize objects to make them sound ‘bigger’ or ‘smaller’. The system level rules can be enforced at an operating system level. In some aspects, these rules can be enforced on multiple applications concurrently.
The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the Claims section. Such combinations may have particular advantages not specifically recited in the above summary
Several aspects of the disclosure here are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” aspect in this disclosure are not necessarily to the same aspect, and they mean at least one. Also, in the interest of conciseness and reducing the total number of figures, a given figure may be used to illustrate the features of more than one aspect of the disclosure, and not all elements in the figure may be required for a given aspect.
Several aspects of the disclosure with reference to the appended drawings are now explained. Whenever the shapes, relative positions and other aspects of the parts described are not explicitly defined, the scope of the invention is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some aspects of the disclosure may be practiced without these details In other instances, well-known circuits, algorithms, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.
A person can interact with and/or sense a physical environment or physical world without the aid of an electronic device. A physical environment can include physical features, such as a physical object or surface. An example of a physical environment is physical forest that includes physical plants and animals A person can directly sense and/or interact with a physical environment through various means, such as hearing, sight, taste, touch, and smell. In contrast, a person can use an electronic device to interact with and/or sense an extended reality (XR) environment that is wholly or partially simulated. The XR environment can include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, and/or the like. With an XR system, some of a person's physical motions, or representations thereof, can be tracked and, in response, characteristics of virtual objects simulated in the XR environment can be adjusted in a manner that complies with at least one law of physics. For instance, the XR system can detect the movement of a user's head and adjust graphical content and auditory content presented to the user similar to how such views and sounds would change in a physical environment. In another example, the XR system can detect movement of an electronic device that presents the XR environment (e.g., a mobile phone, tablet, laptop, or the like) and adjust graphical content and auditory content presented to the user similar to how such views and sounds would change in a physical environment. In some situations, the XR system can adjust characteristic(s) of graphical content in response to other inputs, such as a representation of a physical motion (e.g., a vocal command)
Many different types of electronic systems can enable a user to interact with and/or sense an XR environment. A non-exclusive list of examples include heads-up displays (HUDs), head mountable systems, projection-based systems, windows or vehicle windshields having integrated display capability, displays formed as lenses to be placed on users' eyes (e.g., contact lenses), headphones/earphones, input systems with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop/laptop computers. A head mountable system can have one or more speaker(s) and an opaque display. Other head mountable systems can be configured to accept an opaque external display (e.g., a smartphone). The head mountable system can include one or more image sensors to capture images/video of the physical environment and/or one or more microphones to capture audio of the physical environment. A head mountable system may have a transparent or translucent display, rather than an opaque display. The transparent or translucent display can have a medium through which light is directed to a user's eyes. The display may utilize various display technologies, such as uLEDs, OLEDs, LEDs, liquid crystal on silicon, laser scanning light source, digital light projection, or combinations thereof. An optical waveguide, an optical reflector, a hologram medium, an optical combiner, combinations thereof, or other similar technologies can be used for the medium. In some implementations, the transparent or translucent display can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection technology that projects images onto users' retinas. Projection systems can also project virtual objects into the physical environment (e.g., as a hologram or onto a physical surface).
Various examples of electronic systems and techniques for using such systems in relation to various XR technologies are described.
Referring to
The operating system can manage objects (e.g., user interface elements) that are shown to a user. Each of the objects can be associated with or represent a corresponding application. In some aspects, the object represents an actively running application (e.g., an open media player window) rather than a selectable icon that, when selected, causes the operating system to launch an application.
At operation 10, the method includes maintaining metadata associated with an application running on the computing device, the metadata including a size of an object (e.g., a user interface object) that is associated with the application. For example, the metadata can include dimensions of a media player window that is being shown to a user. In some aspects, the metadata can include the position of the object relative to display coordinates, which can vary from one display environment to another.
At operation 12, the method includes presenting the object associated with the application. In some aspects, the object is presented through a two-dimensional display, such as, for example, a computer monitor, a television, a display of a tablet computer, a mobile phone, or other two-dimensional display. In some aspects, the object is presented on a device that supports three-dimensional XR, such as, for example, a head mounted display, a head-up display, or other equivalent technology. In some aspects, a user or user head position is tracked relative to objects and spatial audio is rendered based on the tracked position.
At operation 14, the method includes determining an audio parameter based on a size of the object. The audio parameter is applied to render audio associated with the application. The audio parameter, in some cases, includes at least one of a dynamic range, a directivity pattern, a frequency response, a sound power, a frequency range, a pitch, a timbre, a number of output audio channels (or a channel layout), and a reverberation. In some aspects, as described in greater detail below with respect to
In some aspects, at operation 14, the method includes determining at least two audio parameters based on size of the object, where one of the at least two audio parameters is a sound power, and at least one other of the at least two audio parameters includes a dynamic range, a directivity pattern, a frequency response, a frequency range, a pitch, a timbre, a number of output audio channels, and a reverberation. The determination (or adjustment) of sound power and at least one other audio parameter can enhance a perceived relationship between the size of the object and audio that is associated with the object.
In some aspects, the method can be performed continuously to dynamically determine or modify one or more of the of the plurality of audio parameters if the size of the object is modified. In some aspects, size of the object can be modified through a user input. A user input can be received from an input device such as a touch screen display, a mouse input, an XR user input sensor (e.g., recognizing hand gestures with computer vision and/or 2D or 3D image sensor technology), or input device. In some aspects, the size of the object can be automatically modified (e.g., based on an automated rearrangement of active ‘windows’). The method can be performed by an operating system that manages one or more applications.
As shown in
The windows manager can manage metadata of each of the applications, which can include a size of the object that represents the application. Based on the size of the object (e.g., a size of an active window), the spatial audio controller 22 can determine one or more audio parameters as described in other sections, that are applied to audio content of the application.
For example, as shown in
Referring back to
In some aspects, these audio parameters are independent of other controls that affect audio parameters. For example, user-level controls can allow for increase and decrease in volume, or modifications to bass, treble, etc. This can be independent of the audio parameters that are determined based on size of the object.
Further, application audio can have metadata that describes configured audio settings such as dynamic range, loudness, frequency range, channel layout, etc. This application level metadata can also be independent of the audio parameters that are determine based on object size.
In some aspects, if there is a conflict between the user level controls, the audio parameters based on object size, or the application level metadata, the operating system arbitrates to determine how audio is rendered based on the competing audio parameters. This arbitration can apply one or more algorithms or logic that is capable of being determined and adjusted based on routine test and experimentation.
For example, object A can represent audio of application A, which is a media player. On the same display, object B can be representative of application B, which is a music player. Object C can be representative of application C, which is a web browser. Each of these applications can be actively running and managed by the operating system. Audio that is associated with one or more of the applications can be played through the speakers. Based on size of the movie player window, music player, and web browser, their corresponding audio parameters can be determined.
If the size of the movie player is small, the audio parameters of the audio content associated with the movie player can be ‘small’ sounding. If the size of the movie player is large, the audio content associated with the movie player can have a ‘large’ sound. The size of objects can be changed (e.g., automatically by the operating system, or through user input). The audio parameters can adjust accordingly based on the updated size of the objects. Thus, if the object size is increased, the audio parameters can be adjusted so that associated audio sounds larger. Conversely, if the object size is decreased, the audio parameters can be adjusted so that the audio sounds smaller The audio output for each of the applications can be rendered separately and then combined to form output audio channels that are used to drive output speakers to produce sound.
As discussed, audio parameters are determined based on size of an object associated with application audio. These audio parameters can include one or more of a dynamic range, a directivity pattern, a frequency response (e.g., an on-axis frequency response), a sound power, a frequency range, a pitch, a timbre, a number of output audio channels, and a reverberation.
In some aspects, the directivity pattern associated with audio of an application is determined based on size of an object. For example, if an object is a virtual guitar is small, the directivity pattern can have a reduced number of lobes or be omnidirectional. In the case of an omnidirectional directivity pattern, audio associated with the virtual guitar can be spatially rendered equally in all directions around the virtual guitar. If the virtual guitar is large, however, then the directivity pattern can have an increased number of lobes or variance, giving the spatially rendered audio more variance at different directions relative to the virtual guitar. In the case of XR, the directivity pattern can mimic a directivity pattern of a physical guitar.
In some aspects, the directivity pattern becomes more directional (e.g., narrower or more concentrated in one or more directions) as the size of the object is increased and becomes more omnidirectional (e.g., round or equal in all directions) as the size of the object decreases. For a single chosen frequency, a physical object that is acoustically small for low frequencies can be acoustically large for high frequencies. Also, an object that is acoustically large for low frequencies can also be acoustically large for high frequencies. An acoustically small object can be defined as an object the size of which is small compared to the wavelength of the radiated sound wave. If an object is acoustically small, it is “invisible” to the wave - the effect of reflection and diffraction can be ignored, the shape and presence of the sound source does not affect the radiation pattern, and the source can be treated as a monopole (omnidirectional). As such, an acoustically small object can represent a large source at a very small frequency, or a tiny source at a high frequency. An acoustically large object is an object the size of which is much greater than wavelength of the radiated sound wave. The object and its geometry becomes visible to the wave (for example, an asymptotically large object will be seen as close to an infinitely large wall reflecting the sound) and has effect on the radiation pattern of sounds emitted from it. In such case, the source can become more directional. This relationship can be imagined as the source's body casting a shadow towards the back of the source, not letting the acoustic energy to the back, and radiating a bigger part of it to the front.
To a wave typically referred to as “low-frequency” (e.g. a frequency of 100 Hz, wavelength of which is equal to 3.43 meters (air, normal conditions)), an object with dimensions much smaller than its wavelength value (e.g. a cubical loudspeaker with a driver on one wall, with an edge length of a few dozen centimeters) will be acoustically small (invisible), thus yielding an omnidirectional pattern. If a “high-frequency” wave is under consideration (e.g. a frequency of 8 kHz, the wavelength is equal to 4.3 centimeters), the same example cubical loudspeaker will be acoustically large, yielding a more directional pattern. If the same example cubical loudspeaker gets larger, at some point it will become acoustically large for the low frequency. In such a case, the pattern will no longer be omnidirectional and the high frequency pattern will become even more directional than before.
Multiplication of the wavenumber k (k=2*Pi*frequency/speed of sound) with the characteristic dimension of the source ‘a’ (e.g. the radius of a sphere enclosing the physical asset or the radius of the source's membrane) can determine if an object is acoustically small or acoustically large. When the ka value of an object is small, the object is acoustically small When the ka value is large, the object is acoustically large. The smaller the ka value, the more omnidirectional the source. The larger the ka value, the more directional the source.
In some aspects, if the size of the object is small, then the dynamic range has a reduced range. If the size of the object is large, then the dynamic range has an increased range. In this manner, if the object size increases, the envelope with which you can hear the sound is larger (meaning that audio that is associated with the object can become louder and quieter). Conversely, if the object is small, then the audio that is associated with the object will be limited to a smaller range. Additionally, or alternatively, the dynamic range can be offset (e.g., raised or lowered) based on size of object. For example, an offset of the sound raised so that the both maximum and minimum levels of audio are higher when the object is large, and/or lowered when the object is small
In some aspects, frequency response (e.g., an on-axis frequency response) of audio associated with an object is determined based on the object size. Frequency response can be the quantitative measure of the output spectrum of a system or device in response to a stimulus, and is used to characterize the dynamics of the system. Frequency response can be represented as a measure of magnitude and phase of audio output of a system as a function of frequency, as compared to the audio input of the system. On-axis frequency refers to the frequency response of the sound source on the sound source's axis (e.g., at its origin) as opposed to the off-axis frequency response of the sound source, which can vary as a function of direction and frequency. The frequency response can be determined to mimic frequency responses of a large sound source when the object is large. Conversely, the frequency response can be determined to mimic frequency response of a small sound source when the object is small.
In some aspects, the frequency response (e.g., on-axis frequency response) is changed if the size of the object is modified, such that a low frequency cut-off for the audio is raised if the size of the object is decreased and the low frequency cut-off for the audio is lowered if the size of the object is increased. This effectively cuts off more frequencies below the frequency cut-off. The on-axis sound pressure in the far field (which can be referred to as the level of the source) depends on the volume velocity generated by a loudspeaker's vibrating diaphragm. As the diaphragm oscillates back and forth (assuming sinusoidal displacement), both of those quantities depend on the amplitude of the diaphragm's displacement (in meters) and the time that it takes for it to achieve that displacement. Each of the frequencies are characterized by its period, inverse to the frequency value. A half of this period is the time in which the diaphragm must move from its minimum to its maximum displacement. For high frequencies, the period is very short. In such a case, to achieve a high velocity the displacement of the diaphragm does not have to be large. For low frequencies, the period is very long. In such a case, to achieve a high velocity, the displacement of the diaphragm must be large. Volume velocity is a value obtained by multiplying the surface area of the diaphragm and its velocity. The sound pressure in the far field is proportional to the volume velocity.
Large sources have large diaphragm surface areas, which, in combination with their physical construction that facilitates larger displacements, allows them to be good low frequency radiators. Small sources have small diaphragm surface areas. In order to generate sufficient amounts of low frequency energy, the displacement of the diaphragm would have to be very large—which is physically difficult for tiny objects. For example, a small cube with an edge of a few centimeters having a diaphragm moving back and forth by a dozen centimeters, would be unnatural and structurally implausible. Therefore, the system can simulate inability of small sources to generate low frequency energy. The smaller the source, the higher its cut-off frequency (no sound below this cut-off frequency).
As discussed, sound power (also known as acoustic power) of audio can be determined based on the object size. Sound power refers to a power of acoustic energy emitted from a sound source, irrespective of an environment (e.g., a room) of the sound source, which can have an effect on the sound pressure level that measures the sound power in the environment. Sound power can be measured as the rate at which sound energy is emitted (or in some cases, reflected, transmitted or received), per unit time. If an object is small, the sound power of the audio that is associated with the object can be determined to be small. If the object is large, the sound power of the audio object can be determined to be large.
In some aspects, frequency range of audio can be determined based on the object size. For example, as shown in
In some aspects, pitch of audio is determined based on the object size. Pitch refers to a perceived quality of how high or low a sound is and relates to a frequency of the sound. The higher the frequency of a sound, the higher the pitch. In some aspects, a pitch is determined as higher for smaller objects, and lower for larger objects. In some aspects, bass of audio is determined based on the object size. For example, lower frequencies (e.g., in the bass range) can be introduced or emphasized when the object is large, and de-emphasized when the object is small
In some aspects, a number of output audio channels or a channel layout associated with audio is determined based on the size of the object. For example, if an object is small, the output audio channel can be a single audio channel (e.g., mono). If the object size is large or increased, the output audio channels can include binaural audio having spatialized sound rendered in left and right audio channels. In some aspects, the number of sound sources can be determined based on the size of the object. For example, if a window player is small, the window player can represent a single sound source from which the user perceives audio to be emanating from. If, however, the window player is large, then multiple sounds in audio that is associated with the movie player can be rendered at different virtual locations.
For example, if a movie scene has two people speaking at opposite sides of the scene, then voice of each person can be rendered at separate virtual locations when the movie player window is large. If the movie player window is small, then audio of the movie is rendered as a single sound source. In some aspects, the number of channels or layout is determined based on object size. For example, based on a large object size, the channel layout can be determined to be a surround sound layout (e.g., 5.1, 7.2, etc.). For a small object, the channel layout can be mono or stereo.
In some aspects, reverberation of audio is determined based on the object size. A large object can have greater reverberation and a small object can have little or no reverberation. If the object size increases, reverberation of the audio associated with the object can be increased (e.g., proportional to the size of the audio object). If the object size is decreased, then reverberation of the audio associated with the object can be decreased.
In some aspects, a timbre of audio (also known tone quality) is determined based on the object size. Timbre can be mainly determined by the harmonic content of a sound and the dynamic characteristics of the sound such as vibrato and the attack-decay envelope of the sound, frequency spectrum and envelope. Timbre characteristics can be varied based on the size of the object, so that large objects have enhanced tone quality.
It should be understood that the object that is associated with the audio application can be representative of a sound source in a spatial audio environment and/or in an XR environment. Thus, as the object size is increased, the audio associated with the object is modified with audio parameters to make the object sound bigger or smaller. The audio can be spatialized so that it appears to originate from or near the object that is shown to the user. For example, audio associated with a movie player (an object) that is shown to a user will sound as if the audio is emanating from the movie player. In some aspects, the object can represent multiple sound sources, for example, if audio of an application contains more than one sound source (e.g., two people speaking).
It should be understood that the terms small and large can vary based on application, e.g., depending on whether the display is a two display or an XR display, or how big a display is. In some aspects, the audio parameters can be determined proportional to the size of the object. In such a case, the object size is a gradient from small to large. In some aspects, thresholds can be used to determine, in a discrete manner, whether an object is small, medium, large, extra-large, etc. For example, if the object has a dimension (e.g., area, height, width, length, diameter, etc.) smaller than a threshold x, then it is deemed to be small. If the object has a dimension greater than a threshold y, it is deemed to be large. If the object has a dimension than a threshold z, then it is deemed to be extra-large, and so on. The one or more thresholds can be determined based on test and experimentation and can vary from one object or another.
In some aspects, an application can be categorized and these categories can affect how the audio parameters of these applications are treated relative to the object. In some aspects, determining or modifying the audio parameters based on the size of the object can be contingent on a categorization of the application. Categories can include, for example, a media or multi-media category (e.g., movie player, music player, videogames), a communication category (e.g., for phone calls or video chat), and/or a utility (e.g., an alarm clock, camera, a calendar, etc.) category. In some aspects, audio parameters for applications that fall under media are determined dynamically based on object size, while applications in the other categories (e.g., utility or communication) do not have their respective audio parameters determined dynamically based on object size.
The audio processing system (for example, a laptop computer, a desktop computer, a mobile phone, a smart phone, a tablet computer, a smart speaker, a head mounted display (HMD), a headphone set, or an infotainment system for an automobile or other vehicle) includes one or more buses 162 that serve to interconnect the various components of the system. One or more processors 152 are coupled to bus 162 as is known in the art. The processor(s) may be microprocessors or special purpose processors, system on chip (SOC), a central processing unit, a graphics processing unit, a processor created through an Application Specific Integrated Circuit (ASIC), or combinations thereof. Memory 151 can include Read Only Memory (ROM), volatile memory, and non-volatile memory, or combinations thereof, coupled to the bus using techniques known in the art. A head tracking unit 158 can include an IMU (e.g., gyroscope and/or accelerometers) and/or camera (e.g., RGB camera, RGBD camera, depth camera, etc.) and tracking algorithms that are applied to sensed data to determine position or location of a user. The audio processing system can further include a display 160 (e.g., an HMD, HUD, a computer monitor, a television, or touchscreen display).
Memory 151 can be connected to the bus and can include DRAM, a hard disk drive or a flash memory or a magnetic optical drive or magnetic memory or an optical drive or other types of memory systems that maintain data even after power is removed from the system. In one aspect, the processor 152 retrieves computer program instructions stored in a machine readable storage medium (memory) and executes those instructions to perform operations described herein.
Audio hardware, although not shown, can be coupled to the one or more buses 162 in order to receive audio signals to be processed and output by speakers 156. Audio hardware can include digital to analog and/or analog to digital converters. Audio hardware can also include audio amplifiers and filters. The audio hardware can also interface with microphones 154 (e.g., microphone arrays) to receive audio signals (whether analog or digital), digitize them if necessary, and communicate the signals to the bus 162.
Communication module 164 can communicate with remote devices and networks. For example, communication module 164 can communicate over known technologies such as Wi-Fi, 3G, 4G, 5G, Bluetooth, ZigBee, or other equivalent technologies. The communication module can include wired or wireless transmitters and receivers that can communicate (e.g., receive and transmit data) with networked devices such as servers (e.g., the cloud) and/or other devices such as remote speakers and remote microphones.
It will be appreciated that the aspects disclosed herein can utilize memory that is remote from the system, such as a network storage device which is coupled to the audio processing system through a network interface such as a modem or Ethernet interface. The buses 162 can be connected to each other through various bridges, controllers and/or adapters as is well known in the art. In one aspect, one or more network device(s) can be coupled to the bus 162. The network device(s) can be wired network devices (e.g., Ethernet) or wireless network devices (e.g., WI-FI, Bluetooth). In some aspects, various aspects described (e.g., simulation, analysis, estimation, modeling, object detection, etc.,) can be performed by a networked server in communication with the capture device.
A shape of the directivity pattern, which can include the shape, direction, and/or number of lobes of the directivity pattern, can be determined based on the geometry and/or size of a) the model, and/or b) the one or more portions that radiates acoustic energy. The directivity pattern can be determined through acoustic simulation of the sound source 180 in virtual environment (e.g., a room). For example, the larger the model of the sound source, the more complicated the directivity pattern can be become (e.g., having increased directivity and/or larger quantity of lobes).
Different sound sources can be modeled differently. Further, some models can have multiple portions that produce sound. For example, if the sound source is a person, the model can have a first portion that vibrates at a first frequency (e.g., approximating a mouth), and a second portion that vibrates at a lower frequency (e.g., approximating the throat). In other examples, a sound source such as a vehicle can be modeled with a first portion that vibrates like an engine and a second portion that vibrates like an exhaust. Thus, a model can have one or more portions that produce sound differently.
From the acoustic simulation with the model, audio filters 190 can be extracted and applied to one or more audio signals to produce an output audio having directivity pattern 182. In some aspects, the audio filters include a) a first filter that is associated with direct sound (to model sound travel from the source directly to the listener), b) a second filter that is associated with early reflections (to model sound that typically reflects off one or two surfaces before arriving at the listener), and c) a third filter that is associated with reverberation (to model sound that arrives at a listener after multiple bounces off of surfaces, typically after 100 ms from the origin of the sound). The filters can define frequency response (e.g., magnitude and phase) for different frequencies at different directions relative to a listener.
In some aspects, the model of the sound source, which can be described as a ‘physical model’, is associated with an object 190. The object can be a visual representation of the sound source that the model is modelling. For example, the object can be a graphic, a video, an animation, an avatar, etc. The sound source can be any sound source such as a loudspeaker, a person, an animal, a movie, a computer application, a videogame, a vehicle, etc. As described the object can be presented in an ER setting and/or on a traditional two-dimensional display.
The model of the sound source can be determined and/or modified based on the object. For example, depending on the orientation, size, or type of the object, the geometry or size of the model can be determined. If the orientation or size of the object changes (e.g., based on input from a user, or an automated action taken by the operating system), then the model can be modified accordingly, thus resulting in another (e.g., a second or modified) set of audio filters. The adjustment of the model can attempt to realistically follow the adjustment of the object that represents the sound source. A reduction in size of the object can result in a reduction in size of the model Similarly, an increase in size of the object can result in an increase in size of the model. For example, a 50% increase or decrease in size of the sound source or object can result in a 50% increase or decrease in size of the physical model. The model can be changed proportionate to a change in the object. In some embodiments, the mapping between the model and the object can be defined (e.g., in user settings), thus allowing for a user to artistically define the relationship between the model and the object.
In some aspects, geometrical attributes of the model can be exposed to a user. For example, the user can configure settings that define the size, shape, or direction of the model. In some aspects, a user can configure the portion of the model that radiates acoustic energy, such as its size, shape, quantity, and/or location on the model. Audio filters can be generated based on the modified geometrical attributes. As such, the user can tune the model according to taste or application.
Thus, based on size or geometry of the model (or of the object that the model is associated with), audio filters 190 are determined. These audio filters can be applied to render audio associated with the sound source. For example, referring to
Similar to the discussion in other sections, the modeling of the sound source can be associated with an application that is managed by an OS. Thus, an application can have an object that visually represents the application as well as sound of the application. The sound of the application can be modeled to automatically produce audio filters that can vary depending on the geometry and/or size of the model, which can be determined based on the geometry, type, or size of the object. Thus, different applications that are managed by the OS can each have corresponding models. A movie application may have a different model than a conferencing application. Further, in some aspects, audio for some sound sources and/or applications are produced using a model, while others are produced ‘artistically’, as described in other sections, without using a model. In some aspects, audio for some sound sources and/or applications can be produced using both a model as described with respect to
Various aspects described herein may be embodied, at least in part, in software. That is, the techniques may be carried out in an audio processing system in response to its processor executing a sequence of instructions contained in a storage medium, such as a non-transitory machine-readable storage medium (e.g. DRAM or flash memory). In various aspects, hardwired circuitry may be used in combination with software instructions to implement the techniques described herein. Thus the techniques are not limited to any specific combination of hardware circuitry and software, or to any particular source for the instructions executed by the audio processing system.
In the description, certain terminology is used to describe features of various aspects. For example, in certain situations, the terms “manager”, “application”, “engine”, controller”, “module”, “processor”, “unit”, “renderer”, “system”, “device”, “filter”, “localizer”, and “component,” are representative of hardware and/or software configured to perform one or more processes or functions. For instance, examples of “hardware” include, but are not limited or restricted to an integrated circuit such as a processor (e.g., a digital signal processor, microprocessor, application specific integrated circuit, a micro-controller, etc.). Thus, different combinations of hardware and/or software can be implemented to perform the processes or functions described by the above terms, as understood by one skilled in the art. Of course, the hardware may be alternatively implemented as a finite state machine or even combinatorial logic. An example of “software” includes executable code in the form of an application, an applet, a routine or even a series of instructions. As mentioned above, the software may be stored in any type of machine-readable medium.
Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the audio processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as those set forth in the claims below, refer to the action and processes of an audio processing system, or similar electronic device, that manipulates and transforms data represented as physical (electronic) quantities within the system's registers and memories into other data similarly represented as physical quantities within the system memories or registers or other such information storage, transmission or display devices.
The processes and blocks described herein are not limited to the specific examples described and are not limited to the specific orders used as examples herein. Rather, any of the processing blocks may be re-ordered, combined or removed, performed in parallel or in serial, as necessary, to achieve the results set forth above. The processing blocks associated with implementing the audio processing system may be performed by one or more programmable processors executing one or more computer programs stored on a non-transitory computer readable storage medium to perform the functions of the system. All or part of the audio processing system may be implemented as, special purpose logic circuitry (e.g., an FPGA (field-programmable gate array) and/or an ASIC (application-specific integrated circuit)). All or part of the audio system may be implemented using electronic hardware circuitry that include electronic devices such as, for example, at least one of a processor, a memory, a programmable logic device or a logic gate. Further, processes can be implemented in any combination hardware devices and software components.
While certain aspects have been described and shown in the accompanying drawings, it is to be understood that such aspects are merely illustrative of and not restrictive on the broad invention, and the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those of ordinary skill in the art.
To aid the Patent Office and any readers of any patent issued on this application in interpreting the claims appended hereto, applicants wish to note that they do not intend any of the appended claims or claim elements to invoke 35 U.S.C. 112(f) unless the words “means for” or “step for” are explicitly used in the particular claim
It is well understood that the use of personally identifiable information should follow privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of users. In particular, personally identifiable information data should be managed and handled so as to minimize risks of unintentional or unauthorized access or use, and the nature of authorized use should be clearly indicated to users.
This application claims the benefit of the U.S. Provisional application No. 63/073,175 filed Sep. 1, 2020 and 63/172,963 filed Apr. 9, 2021.
| Number | Date | Country | |
|---|---|---|---|
| 63172963 | Apr 2021 | US | |
| 63073175 | Sep 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/US2021/045969 | Aug 2021 | US |
| Child | 18110298 | US |
