Patent Application
20250008287
The present disclosure relates to the generation of three-dimensional sound, and in particular to systems and methods for capturing and processing mixed sound tracks into separate sound types and then applying transfer functions to the separated sound to generate three-dimensional sound that contains spatial information about the sounds sources to recreate a three-dimensional (3D) sound field that has been configured by users.
Billions of people listen to music worldwide, but most listeners may only have access to music in a mono or stereo sound format. Stereo is a method of sound reproduction that may use multiple independent audio channels played using two or more speakers (or headphones) so that the sound from the speakers appears to be coming from various directions, as in natural hearing. However, stereo sound usually refers to just two audio channels to be played using two speakers or headphones. More immersive sound technologies like surround sound need to record and save multiple sound tracks (e.g., 5.1 or 7.1 surround sound configurations), and the sound must be played through an equivalent number of speakers. In any case, each of the audio channels or sound tracks consists of mixed sound from multiple sound sources. Therefore, stereo sound is different from “real” sound (e.g., a listener in front of a stage at a concert) because spatial information regarding the individual sound sources (e.g., instruments and vocals) is not reflected in the sound.
With two cars, a person may perceive spatial information and hear “real” three-dimensional (3D) sound as binaural sound (e.g., sound represented by a left car and a right ear), such as how music is perceived by two cars in a music hall, theater or at a sporting event at a stadium or arena. However, as noted above, today's music technology usually provides only mono or stereo sound without spatial cues or spatial information. For this reason, music and other sounds may be experienced differently and often more enjoyably in theaters, arenas, and music halls than it is through headphones or earbuds or on loudspeakers or even on multiple-channel, multiple-loudspeaker surround systems. Currently, the generation of 3D sound may be accomplished, for example, by many loudspeakers mounted on the walls of a movie theater with each loudspeaker being driven by a separate sound track recorded during manufacturing of a movie. However, this kind of 3D audio system may be very expensive and cannot be realized in mobile devices as an app (application software) or even in most home theater or in-car configurations. Therefore, in today's music and entertainment industry, most of music or other audio data is stored and played as mono or stereo sound, where all sound sources, such as vocals and different kinds of instruments, are pre-mixed into just one (mono) or two (stereo) sound tracks.
Most audio/sound from a video conferencing device, such as a computer, laptop, smartphone, or tablet, is in mono sound. Although on a display screen, a user (e.g., an attendee or participant), may see all attendees of the conference in separate windows, the audio is usually only one channel mono with a narrow bandwidth. Using video of each of the different attendees, a virtual conference room may be accomplished, but the audio component cannot match the video component because it does not have the 3D sound which is necessary for providing a more accurate (e.g., spatially) virtual reality sound experience. Furthermore, when two attendees have similar sounding voices, the user may not be able to distinguish between voices when they are talking at the same time or even separately. This may happen, for example, when the user is watching shared documents on another screen or video window while the user is not looking at the attendees' faces. The problem may be even worse when more attendees are in a video conference, such as a remote learning classroom. The user may need spatial information, like 3D sound, to help identify which attendee is speaking based on the conference sound alone.
The disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments of the disclosure. The drawings, however, should not be taken to limit the disclosure to the specific embodiments, but are for explanation and understanding only.
Described herein are three-dimensional (3D) configurable soundstage audio systems and applications and implementations. A three-dimensional (3D) sound field refers to sound that includes discrete sound sources located at different spatial locations. The 3D soundstage is the sound representing the 3D sound field. For example, soundstage music may allow a listener to have an auditory perception of the isolated locations of instruments and vocal sources when listening to a given piece of music either through earphones, headphones, or loudspeakers. In general, the 3D soundstage may have embedded cues for the listener's perception of the spatial information. The soundstage may also be configurable so that it may be configured by the listener, a DJ, software, or audio systems. For example, the location of each instrument in the 3D sound field may be moved while the listener's location in the 3D sound field may be dynamic or static at the location of a preferred instrument.
In order to listen to or to play the 3D soundstage a listener may use binaural sound represented by two tracks, one for the left ear and one for the right car, with embedded cues for listener perception of spatial information associated with sound sources. Binaural sound may be experienced as 3D sound (e.g., as if coming from different locations) through earphones, headsets or other such devices. Alternatively, direct 3D sound may be used to play the 3D soundstage. In direct 3D sound, the sound is played from a group of loudspeakers located in different 3D locations (e.g., corresponding to desired locations for individual sound sources in the 3D sound field). Each loudspeaker may play one isolated sound track, e.g., one speaker for drum and another for bass. The listener may hear the 3D sound field from the loudspeakers directly since they are at different locations in a real world 3D sound field. In both the binaural and direct 3D sound use cases, the listener's brain may perceive the 3D sound field and may recognize and track the discrete sound sources like in the real world, which may be referred to as acoustic virtual reality throughout the present disclosure.
Furthermore, another way to achieve the 3D sound field may be to record binaural sound directly with a specialized binaural/3D microphone. Most existing binaural microphones are just a dummy human head with microphones installed in the cars which may be too big in the size and/or too expensive for many applications. Accordingly, described herein is a 3D microphone that may have a small form factor by using an array of very small microphones and signal processing technology. This small form 3D microphone may be used with any handheld recording device such as a smartphone or tablet. The output of the sound captured by the 3D microphone may be presented as binaural, stereo, or multi-track recordings, with one track for each spatial direction associated with a sound source for the 3D sound field.
Also, in the present disclosure, three techniques are described to enhance the signal-to-noise ratio (SNR) of audio signals as follows. Noise reduction is the process of reducing the background noise in an audio channel based on temporal information, such as the statistical properties between signal and noise or the frequency distributions of different kinds of signals. A microphone array uses one or multiple acoustic beam patterns to enhance the sound coming from one beam direction while canceling the sound coming from outside the beam direction. An acoustic echo canceller (AEC) uses one or more reference signals to cancel the corresponding signals mixed in the microphone captured signals. The reference signal(s) is/are correlated to the signal(s) which the AEC will cancel.
Referring to
In the example of
Each of the separated sound tracks may then be processed by signal processing unit 106A using a pair HRTF filters from storage unit 104A to output two audio channels representing left and right car channels, respectively, for each separated sound track. In one implementation, the above-noted process may be performed in parallel for each of the input mixed sound tracks.
Each HRTF filter (e.g., a pair of left and right HRTF filters 200B of
As noted below with respect to
All of the output left audio tracks may then be mixed to generate the left channel of the binaural sound (e.g., Binaural L), while all the right channels may be mixed to generate the right channel of the binaural sound (e.g., Binaural R). When both the L and R channels are played through earphones or a headset, a listener may experience 3D binaural sound and perceive the spatial locations of the sound sources in the 3D sound field.
Furthermore, the listener may configure the locations and/or volume of each sound source and/or of the listener in the 3D sound field through the GUI 110A. Virtually (e.g., in the acoustic virtual reality), the listener and the sound source(s) may be located in any location within the 3D sound field and the volume of each of the sound source(s) may be proportional to the distance from the location of the listener to the location of the sound source in the 3D sound field. For example, the sound source location and/or volume may be configured through the GUI 110A which may be presented via a user interface device. The user interface device may be, for example, in the form of a touch screen on a smartphone (
In one implementation, to generate the binaural output (e.g., Binaural L+R) as shown in
Referring to
In the example of
As noted above with respect to
An implementation of GUI 110A or GUI 112B may be seen in
A head related transfer function (HRTF) filter (e.g., like those stored in storage unit 104A of
With two cars (e.g., binaural hearing), a listener may localize sounds in three dimensions: in range (distance); in direction above and below; and in front and to the rear, as well as to either side. This is possible because the brain, inner ear and the external cars (pinna) work together to make inferences about location. The listener may estimate the location of a sound source by taking cues derived from one ear (monaural cues), and by comparing cues received at both ears (difference cues or binaural cues). Among the difference cues are time differences of arrival at each car and intensity differences at each car. The monaural cues come from the interaction between the sound source and the listener's human anatomy, in which the original source sound is modified by the inner ear and the external cars (pinna) before it enters the car canal for processing by the cochlea and the brain. These modifications encode the sound source location, and may be captured via a relationship between the sound source location and the listener's location. A sound track filter based on this relationship is referred to herein as the HRTF filter. Convolution of [[an]] a sound track with a pair of HRTF filters converts the sound to generate binaural signals for left and right ears respectively, wherein the binaural sound signals (e.g., Binaural L+R of
A pair of binaural tracks for the left and right ears of the listener may be used to generate a binaural sound, from mono or stereo, which seems to come from a particular location in space. A HRTF filter is a transfer function, describing how a sound from a specific location in a 3D space will arrive at the listener's location (generally at the outer end of the listener's auditory canal). The HRTF filter may be implemented as convolutional computation in the time domain or multiplication in the frequency domain to save computation time as shown in
Referring to
Referring to
In an implementation, for multiple sound sources like sound source 204A, the system may repeat the above filter retrieval and filtering operation for each sound source and then combine (e.g., mix) the filtered sound tracks together for the final binaural output or stereo-kind (superior to mono) outputs to two loudspeakers.
As noted above with respect to
Although music may be recorded on multiple tracks using multiple microphones, where each individual track represents each instrument or vocal recorded in a studio, the music streams that consumers most often get are mixed into stereo sound. The costs of recording, storage, bandwidth, transmission, and playing of multi-track audio may be very high, so most existing music recordings and communication devices (radio or smartphones) are configured for either mono or stereo sound. To generate the 3D soundstage from conventional mixed sound track formats (mono and stereo), the system (e.g., system 100A of
Machine learning in this disclosure refers to methods implemented on a hardware processing device that uses statistical techniques and/or artificial neural networks to give computer the ability to “learn” (i.e., progressively improve performance on a specific task) from data without being explicitly programmed. The machine learning may use a parameterized model (referred to as “machine learning model”) that may be deployed using supervised learning/semi-supervised learning, unsupervised learning, or reinforced learning methods. Supervised/semi-supervised learning methods may train the machine learning models using labeled training examples. To perform a task using supervised machine learning model, a computer may use examples (commonly referred to as “training data”) to train the machine learning model and to adjust parameters of the machine learning model based on a performance measurement (e.g., the error rate). The process to adjust the parameters of the machine learning model (commonly referred to as “train the machine learning model”) may generate a specific model that is to perform the practical task it is trained for. After training, the computer may receive new data inputs associated with the task and calculate, based on the trained machine learning model, an estimated output for the machine learning model that predicts an outcome for the task. Each training example may include input data and the corresponding desired output data, where the data can be in a suitable form such as a vector of numerical values or alphanumerical symbols as representation of sound tracks.
The learning process may be an iterative process. The process may include a forward propagation process to calculate an output based on the machine learning model and the input data fed into the machine learning model, and then calculate a difference between the desired output data and the calculated output data. The process may further include a backpropagation process to adjust parameters of the machine learning model based on the calculated difference.
The parameters for a machine learning model 308 for separating mixed sound tracks may be trained by machine learning, statistical, or signal processing technology. As shown in
The feature extraction unit 302 may extract features from the original input data (e.g., mixed sound) in order to facilitate the training and separation computations. The training data may be processed in the time domain (raw data), frequency domain, feature domain, or time-frequency domain through the fast Fourier transform (FFT), short-time Fourier transform (STFT), spectrogram, auditory transform, wavelets, or other transforms.
The model structure and training algorithms for machine learning model 308 may be neural network (NN), convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), long short-term memory (LSTM), Gaussian mixture model (GMM), hidden Markov model (HMM), or any model and/or algorithm which may be used to separate sound sources in a mixed sound track. After training, in the separation session, the input music data may be separated into multiple tracks by the trained separation model computation unit 310, each separated sound track corresponding to one kind of isolated sound. In an implementation, the multiple separated sound tracks may be mixed in different ways for different sound effects through user configuration (e.g., via GUI 110A of
In one implementation, machine learning model 300 may be a DNN or CNN that may include multiple layers, in particular including an input layer (e.g., training session) for receiving data inputs, an output layer (e.g., separation session) for generating outputs, and one or more hidden layers that each includes linear or non-linear computation elements (referred to as neurons) to perform the DNN or CNN computation propagated from the input layer to the output layer that may transform the data inputs to the outputs. Two adjacent layers may be connected by edges. Each of the edges may be associated with a parameter value (referred to as a synaptic weight value) that provide a scale factor to the output of a neuron in a prior layer as an input to one or more neurons in a subsequent layer.
Shown in
The training data (e.g., time-domain mixed sound signals) may be processed separation unit 404 (like sound separation unit 102A of
Shown in
Shown in
Shown in
Shown in
Shown in
In an implementation of the present disclosure, the mixed sound track was separated using the trained machine learning model 308. The separation computation may be performed according to the system 400 described above with respect to
As an initial matter, microphone array 602 may capture a voice command. The pre-amplifiers/analog to digital converters (ADC) 604 may enlarge the analog signal and/or convert it to a digital signal. Both the pre-amplifier and ADC are optional depending on what kind of microphones are used in microphone array 602. For example, they may not be needed by digital microphones.
The acoustic beamformer 606 forms acoustic beam(s) to enhance the voice or voice command and to suppress any background noise. An acoustic echo canceller (AEC) 608 further cancels the loudspeaker sound (e.g., from loudspeakers 630) captured by the microphone array 602 using reference signals. The reference signal may be captured by one or more reference microphones 610 near the loudspeakers 630 or from the audio signals (e.g., from configuration/equalizer unit 624) prior to sending to sending them to the amplifier 608 for the loudspeakers 630. The output from the AEC may then be sent to the noise reduction unit 612 to further reduce the background noise.
The clean speech is then sent to the wakeup phrase recognizer 614 which may recognize a pre-defined wakeup phrase for system 600. The system 600 may mute the loudspeakers 630 to further improve voice quality. The automatic speech recognizer (ASR) 616 may then recognize the voice command, such as a song music title, and then instructs a music retrieval unit 618 to retrieve the music from a music library 620. In an implementation, the wakeup phrase recognizer 614 and ASR 616 may be combined as one unit. Furthermore, the retrieved music may then be separated by the sound separation unit 622 that may be like sound separation unit 102A of
A configuration/equalizer unit 624 may then adjust the volume of each sound source and/or conduct equalization (gain of each frequency band or each instrument or vocal) of each sound track. Finally, the separated music sound tracks may be played from the loudspeakers 630 (via amplifier 628) as direct 3D sound as shown in system 100B of
A few implementations of loudspeaker layouts are shown in
In an implementation, the GUI 700 may be configured so that all sound sources (e.g., from a music band on stage) are represented by band member icons on a virtual stage and the listener is represented by a listener head icon (wearing headphones to accentuate the position of the left and right ears) that may be moved freely around the stage by a user of GUI 700. In another implementation, all the icons in
In
In
In
In
The GUI 700 may also be applicable on a remote to control a TV with direct 3D sound systems, or other such applications. For example, when a user is watching a move, she or he may move the listener head icon closer to a vocal icon so that the volume of the voice is increased while the volume of other background sounds (e.g., music) may be reduced so that the user may hear a clearer voice.
The system 800 may be described as a 3D microphone system which may capture and output 3D and binaural sound directly. As referred to herein, a 3D microphone system may comprise a microphone array system which may capture sounds from different directions together with spatial information regarding the location of the sources of the sounds. The system 800 may produce two kinds of outputs: (1) multiple tracks, each corresponding to the sound from one direction where each of the multiple tracks may drive a group of loudspeakers to represent a 3D sound field; and (2) binaural L and R tracks for earbuds or earphones to virtually represent the 3D sound field.
Each microphone of microphone array 802 may have their signals processed by a pre-amplifier/ADC unit 804. The pre-amplifiers and analog to digital converters (ADC) may enlarge the analog signal and/or convert it to a digital signal. Both the pre-amplifier and ADC are optional and may depend on the selected microphone components for microphone array 802. For example, they may not be necessary for digital microphones.
The acoustic beamformer 806 may form acoustic beam patterns pointing to different directions or different sound sources, simultaneously, as shown in
In order to generate direct 3D sound, the multiple tracks may drive multiple amplifiers and loudspeakers to construct a 3D sound field for listeners.
In order to generate binaural output, the multiple sound tracks may go through multiple pairs of selected HRTF filters 810 to convert the spatial sound track to binaural sound. The HRTF filters may be selected based on a user's configuration (e.g., via output audio configuration unit 814) or based on the actual spatial locations of the sound sources in the real world. Furthermore, a mixer 812 may then combine the HRTF outputs to a pair of binaural output for left and right ears, respectively. The final binaural output represents the 3D sound field recorded by the microphone array 802.
Based on the microphone array 802 only having two acoustic beam patterns, pointing to left and right respectively, the microphone array works as a stereo microphone, which is a special case of the 3D microphone.
The conference system 1000 may include a signal processing and computation unit 1002, a bank 1004 of head related transfer functions (HRTF) filters, a display unit with graphical user interface (GUI) 1006, amplifiers 1008, headset or earphones 1010, and loudspeakers 1012. The system 1000 may be implemented, for example, as software on a user's laptop, tablet, computer, or smartphone with a connected headset. The video and audio conference, hereinafter referred to as the “conference”, may also be referred to as a teleconference, virtual conference, web conference, webinar, or video conference. One such conference may include multiple local and/or multiple remote attendees. In an implementation, the attendees may be connected by internet and telephone networks 1014. In an implementation the conference may be controlled by cloud servers or remote servers via the internet and telephone networks 1014.
A user of system 1000 may be one of the attendees of a conference or virtual concert. She or he is the owner of the laptop, tablet, computer, or smartphone running the conference software with video and audio and possibly wearing headset 1010. The terms “speakers” or “attendees” refer to persons attending the conference. The loudspeakers 1012 may be any devices which can convert an audio signal to audible sound. The amplifiers 1008 may be an electronic device or circuit to increase the signal power to drive the loudspeakers 1012 or the headset 1010. The headset 1010 may be headphones, car caps, or in-car audio devices.
The input signals (e.g., from the cloud via 1014) may include video, audio and the speaker's identification (ID). The speaker's ID may associate video and audio input to an attendee who is speaking. Based on a speaker's ID not being available, a new speaker ID may be generated by the speaker ID unit 1016 as described below.
The speaker ID unit 1016 may obtain a speaker ID from the conference software based on the speaker ID used for the speaker's videoconference session. Furthermore, the speaker ID unit 1016 may obtain a speaker ID from a microphone array (e.g., microphone array 802 of
Based on an attendee speaking, a video window associated with the attendee may be highlighted visually in the display/GUI 1006, so the user knows which attendee of the conference is speaking, e.g., Attendee 2 in
Based on multiple display/GUIs 1006 and multiple loudspeakers 1012 being used in a real conference room, each loudspeaker 1012 may be dedicated to one speaker's sound in one display/GUI 1006 at one location. In this situation, the user does not need to use a headset 1010 and she or he may experience 3D sound from the loudspeakers 1012. The polarity of loudspeakers can be placed in a home theater, a movie theater, a soundbar, a TV set, a smart speaker, a smartphone, a mobile device, a handheld device, a laptop computer, a PC, an automobile vehicle or anywhere with more than one loudspeakers or sound generators.
The virtual conference room 1100 may have multiple windows (1102-1112) including video of the user and meeting attendees. The locations of the windows (1102-1112) may be assigned by the conference software (e.g., running on laptop) or by the user (e.g., via a display/GUI 1006 of
As noted above, Furthermore, the virtual conference room 1100 may be configured by the user so that the video windows (1104-1112) of the attendees may be placed virtually anywhere in the virtual conference room 1100 with a mouse, keypad, or touch screen, etc. From the relative location of a speaker (e.g., attendee 2) to the user (e.g., angle from video window 1106 of attendee 2 to video window 1102 of the user), related HRTF's may be selected and applied automatically for attendee 2 when they are speaking.
For simplicity of explanation, the methods of this disclosure are depicted and described as a series of acts. However, acts in accordance with this disclosure can occur in various orders and/or concurrently, and with other acts not presented and described herein. Furthermore, not all illustrated acts may be needed to implement the methods in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the methods could alternatively be represented as a series of interrelated states via a state diagram or events. Additionally, it should be appreciated that the methods disclosed in this specification are capable of being stored on an article of manufacture to facilitate transporting and transferring such methods to computing devices. The term “article of manufacture,” as used herein, is intended to encompass a computer program accessible from any computer-readable device or storage media.
The methods may be performed by processing devices that may comprise hardware (e.g., circuitry, dedicated logic), computer readable instructions (e.g., run on a general purpose computer system or a dedicated machine), or a combination of both. The methods and each of their individual functions, routines, subroutines, or operations may be performed by one or more processors of the computer device executing the method. In certain implementations, the methods may be performed by a single processing thread. Alternatively, the methods may be performed by two or more processing threads, each thread executing one or more individual functions, routines, subroutines, or operations.
In one implementation, method 1200 may be performed by the signal processing units of system 100A of
At 1202, the method includes receiving a specification of a three-dimensional space (e.g., 200A of
At 1204, the method includes determining (e.g., by sound separation unit 102A of
At 1206, the method includes representing a listener (e.g., listener 202A of
At 1208, the method includes generating, responsive to a user configuration (e.g., via GUI 110A of
At 1210, the method includes applying each of the plurality of HRTF filters (e.g., 200B of
At 1212, the method includes generating the three-dimensional sound based on the filtered sound tracks.
At 1302, the method includes capturing sound from the plurality of sound sources with a microphone array (e.g., microphone array 802 of
At 1304, the method includes rendering the three-dimensional sound with one or more loudspeakers (e.g., loudspeakers 108B of
At 1306, the method includes removing echoes in the plurality of sound tracks with an acoustic echo cancellation unit (e.g., AEC 608 of
At 1308, the method includes reducing a noise component in the plurality of sound tracks with a noise reduction unit (e.g., noise reduction unit 612 of
At 1310, the method includes processing the plurality of sound tracks with a sound equalizer unit (e.g., configuration/equalizer unit 624 of
At 1312, the method includes capturing a reference signal with a reference sound capture circuit (e.g., reference microphone 610 of
At 1314, the method includes recognizing voice commands with a speech recognition unit (e.g., speech recognizer 616 of
In certain implementations, computer system 1400 may be connected (e.g., via a network, such as a Local Area Network (LAN), an intranet, an extranet, or the Internet) to other computer systems. Computer system 1400 may operate in the capacity of a server or a client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. Computer system 1400 may be provided by a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, a computing device in vehicle, home, room, or office, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, the term “computer” shall include any collection of computers, processors, or SoC, that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods described herein.
In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of either a server or a client machine in server-client or cloud network environments, or it may act as a peer machine in peer-to-peer (or distributed) network environments. The machine may be an onboard vehicle system, wearable device, personal computer (PC), a tablet PC, a hybrid tablet, a personal digital assistant (PDA), a mobile telephone, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. Similarly, the term “processor-based system” shall be taken to include any set of one or more machines that are controlled by or operated by a processor (e.g., a computer or cloud server) to individually or jointly execute instructions to perform any one or more of the methodologies discussed herein.
Example computer system 1400 includes at least one processor 1402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both, processor cores, compute nodes, cloud server, etc.), a main memory 1404 and a static memory 1406, which communicate with each other via a link 1408 (e.g., bus). The computer system 1400 may further include a video display unit 1410, an alphanumeric input device 1412 (e.g., a keyboard), and a user interface (UI) navigation device 1414 (e.g., a mouse). In one embodiment, the video display unit 1410, input device 1412 and UI navigation device 1414 are incorporated into a touch screen display. The computer system 1400 may additionally include a storage device 1416 (e.g., a drive unit), a sound production device 1418 (e.g., a speaker), a network interface device 1420, and one or more sensors 1422, such as a global positioning system (GPS) sensor, accelerometer, gyrometer, position sensor, motion sensor, magnetometer, or other sensors.
The storage device 1416 includes a machine-readable medium 1424 on which is stored one or more sets of data structures and instructions 1426 (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 1426 may also reside, completely or at least partially, within the main memory 1404, static memory 1406, and/or within the processor 1402 during execution thereof by the computer system 1400, with main memory 1404, static memory 1406, and processor 1402 comprising machine-readable media.
While the machine-readable medium 1424 is illustrated in an example embodiment to be a single medium, the term “machine-readable medium” may include a single medium or multiple media (e.g., a centralized, cloud, or distributed database, and/or associated caches and servers) that store the one or more instructions 1426. The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing, encoding or carrying instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media include volatile or non-volatile memory, including but not limited to, by way of example, semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; and CD-ROM and DVD-ROM disks.
The instructions 1426 may further be transmitted or received over a communications network 1428 using a transmission medium via the network interface device 1420 utilizing any one of a number of well-known transfer protocols (e.g., HTTP). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, mobile telephone networks, plain old telephone (POTS) networks, and wireless data networks (e.g., Bluetooth, Wi-Fi, 3G, and 4G LTE/LTE-A or WiMAX networks). The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding, or carrying instructions for execution by the machine, and includes digital or analog signals or other intangible medium to facilitate communication of such software.
Example computer system 1400 may also include an input/output controller 1430 to receive input and output requests from the at least one central processor 1402, and then send device-specific control signals to the device they control. The input/output controller 1430 may free the at least one central processor 1402 from having to deal with the details of controlling each separate kind of device.
Unless specifically stated otherwise, terms such as “receiving,” “associating,” “determining,” “updating” or the like, refer to actions and processes performed or implemented by computer systems that manipulates and transforms data represented as physical (electronic) quantities within the computer system registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. Also, the terms “first,” “second,” “third,” “fourth,” etc. as used herein are meant as labels to distinguish among different elements and may not have an ordinal meaning according to their numerical designation.
Examples described herein also relate to an apparatus for performing the methods described herein. This apparatus may be specially constructed for performing the methods described herein, or it may comprise a general purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.
The methods and illustrative examples described herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform method 500 and/or each of its individual functions, routines, subroutines, or operations. Examples of the structure for a variety of these systems are set forth in the description above.
The above description is intended to be illustrative, and not restrictive. Although the present disclosure has been described with references to specific illustrative examples and implementations, it will be recognized that the present disclosure is not limited to the examples and implementations described. The scope of the disclosure should be determined with reference to the following claims, along with the full scope of equivalents to which the claims are entitled.
This application is a continuation of U.S. application Ser. No. 18/121,452 filed on Mar. 14, 2023, which is a continuation of U.S. application Ser. No. 17/568,343 filed on Jan. 4, 2022, which is a continuation of U.S. application Ser. No. 17/227,067 filed on Apr. 9, 2021, which claims the benefit of the following patent applications: U.S. provisional application Ser. No. 63/008,723, filed on Apr. 11, 2020; and U.S. provisional application Ser. No. 63/036,797, filed on Jun. 9, 2020. The contents of the above-mentioned applications are hereby incorporated by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63036797 | Jun 2020 | US | |
| 63008723 | Apr 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18121452 | Mar 2023 | US |
| Child | 18816295 | US | |
| Parent | 17568343 | Jan 2022 | US |
| Child | 18121452 | US | |
| Parent | 17227067 | Apr 2021 | US |
| Child | 17568343 | US |
