Patent Application
20250142251
The present disclosure is directed to deep audio zooming with beamwidth-controllable neural beamformer features.
Audio zooming, a signal processing technique, enables selective focusing and enhancement of sound signals from a specified region, attenuating others. While traditional beamforming and neural beamforming techniques, centered on creating a directional array, necessitate the designation of a singular target direction, those techniques may overlook the concept of a field of view (FOV), that defines an angular area.
That is, audio zooming technology refers to a signal processing technique that allows a user to selectively focus on and enhance the audio signals originating from a specific region of interest in a sound field, while attenuating signals from other directions. Audio zooming technology finds applications in various fields, including teleconferencing, surveillance, broadcasting and video filming. In teleconferencing, audio zooming can improve the intelligibility of speech by isolating the sound signals from the target speakers' region and reducing background noises and interfering speakers. In surveillance, audio zooming can be used to monitor specific regions of interest by focusing on sounds originating from those regions, while suppressing sounds from other directions. In broadcasting, audio zooming can enhance the experience of viewers by allowing them to selectively listen to commentary or sound effects while reducing the volume of other sounds in the broadcast. With the audio zooming feature, a user can adjust the video recording's focus using a pinch motion on the screen. When the user zooms in, the audio from the users focal point becomes more pronounced. Conversely, as the user zooms out, the ambient sounds become more evident and are no longer diminished.
Beamforming is the most closely related field of study to this concept. It is a computational technique that involves constructing a directional microphone using an array of omnidirectional microphones. By exploiting the different time delays of signals arriving from different directions, beamforming combines the microphone signals into an output signal that amplifies the sound from the target direction while suppressing all other sounds for monoaural directional sound filtering. In nearly all beamforming techniques, it is necessary to specify or estimate a single target direction, which plays a crucial role in the mathematical formulation of beamforming. Most beamforming techniques share the same objective of improving the sound quality from a single direction. However, those techniques do not take into account the concept of a FOV. In addition, beamforming typically requires a large microphone array to create a narrow beam that captures sound from a specific direction. The width of the enhanced beam in beamforming is fixed and determined by several factors, including the microphone configuration, signal frequency, and type of beamformer used. There has been proposed an audio zooming implementation that combines several robust adaptive beamformers with a derived post-processing algorithm to further enhance the targeted sound source, and the enhanced targeted sound source is weighted mixed with the original microphone signal to create the audio zoom effect. There has also been proposed to calculate the sound spectral covariance matrices, considering both the desired audio signals inside the FOV and those outside it, and by addressing a generalized eigenvalue problem with these estimated matrices, the solved beamformer can enhance sounds originating within the FOV. However, one constraint of these techniques is that in environments with sound reflections, audio waves from a source outside the FOV can reach the microphone after bouncing off surfaces from within the FOV. Under these circumstances, their audio zooming technique would continue to amplify these reflected sound signals, which is a technical problem arising in that technology.
And for any of those reasons there is therefore a desire for technical solutions to such problems that arose in computer audio technology.
There is included a method and apparatus comprising memory configured to store computer program code and a processor or processors configured to access the computer program code and operate as instructed by the computer program code. The computer program is configured to cause the processor implement receiving code configured to cause the at least one processor to receive multiple audio signals obtained from ones of a plurality of microphones of a microphone array, implementing code configured to cause the at least one processor to implement an audio zooming based on the audio signals by selectively focusing and enhancing first ones of the audio signals and by attenuating other ones of the audio signals; and controlling code configured to cause the at least one processor to control an output of audio based on the audio zooming, wherein the audio zooming comprises a consolidating of a plurality of directional features of the first ones of the audio signals within a field around the microphone array and a countering based on determining directional aspects of the other ones of the audio signals from outside of the field.
The audio zooming may include sampling the audio signals along a pre-set number of directions evenly partitioned around the microphone array.
The pre-set number may be 36.
Consolidating the plurality of directional features may be based on determining
where in represents an index set of sectors encircling the microphone array, dθ
The countering may be based on determining
where out represents a second index set of the sectors encircling the microphone array.
The audio zooming may be based on consolidating field-of-view feature vectors based on a concatenation represented as =[
,
]∈
T×2F, where dθ
T×F represents each of the directional features.
The audio zooming may be based on post-processing determined as
The audio zooming may be applied to a 3D space by modifying dθ
The audio zooming may be based on a neural network.
Further features, nature, and various advantages of the disclosed subject matter will be more apparent from the following detailed description and the accompanying drawings in which:
The proposed features discussed below may be used separately or combined in any order. Further, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program that is stored in a non-transitory computer-readable medium.
In
A streaming system may include a capture subsystem 203, that can include a video source 201, for example a digital camera, creating, for example, an uncompressed video sample stream 213. That sample stream 213 may be emphasized as a high data volume when compared to encoded video bitstreams and can be processed by an encoder 202 coupled to the video source 201, which may be for example a camera as discussed above. The encoder 202 can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream 204, which may be emphasized as a lower data volume when compared to the sample stream, can be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 can access the streaming server 205 to retrieve copies 208 and 206 of the encoded video bitstream 204. A client 212 can include a video decoder 211 which decodes the incoming copy of the encoded video bitstream 208 and creates an outgoing video sample stream 210 that can be rendered on a display 209 or other rendering device (not depicted). In some streaming systems, the video bitstreams 204, 206 and 208 can be encoded according to certain video coding/compression standards. Examples of those standards are noted above and described further herein.
A receiver 302 may receive one or more codec video sequences to be decoded by the decoder 300; in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent from other coded video sequences. The coded video sequence may be received from a channel 301, which may be a hardware/software link to a storage device which stores the encoded video data. The receiver 302 may receive the encoded video data with other data, for example, coded audio data and/or ancillary data streams, that may be forwarded to their respective using entities (not depicted). The receiver 302 may separate the coded video sequence from the other data. To combat network jitter, a buffer memory 303 may be coupled in between receiver 302 and entropy decoder/parser 304 (“parser” henceforth). When receiver 302 is receiving data from a store/forward device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer 303 may not be needed, or can be small. For use on best effort packet networks such as the Internet, the buffer 303 may be required, can be comparatively large and can advantageously of adaptive size.
The video decoder 300 may include a parser 304 to reconstruct symbols 313 from the entropy coded video sequence. Categories of those symbols include information used to manage operation of the decoder 300, and potentially information to control a rendering device such as a display 312 that is not an integral part of the decoder but can be coupled to it. The control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI messages) or Video Usability Information (VUI) parameter set fragments (not depicted). The parser 304 may parse/entropy-decode the coded video sequence received. The coding of the coded video sequence can be in accordance with a video coding technology or standard, and can follow principles well known to a person skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so forth. The parser 304 may extract from the coded video sequence, a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based upon at least one parameters corresponding to the group. Subgroups can include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs) and so forth. The entropy decoder/parser may also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, and so forth.
The parser 304 may perform entropy decoding/parsing operation on the video sequence received from the buffer 303, so to create symbols 313. The parser 304 may receive encoded data, and selectively decode particular symbols 313. Further, the parser 304 may determine whether the particular symbols 313 are to be provided to a Motion Compensation Prediction unit 306, a scaler/inverse transform unit 305, an Intra Prediction Unit 307, or a loop filter 311.
Reconstruction of the symbols 313 can involve multiple different units depending on the type of the coded video picture or parts thereof (such as: inter and intra picture, inter and intra block), and other factors. Which units are involved, and how, can be controlled by the subgroup control information that was parsed from the coded video sequence by the parser 304. The flow of such subgroup control information between the parser 304 and the multiple units below is not depicted for clarity.
Beyond the functional blocks already mentioned, decoder 300 can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can, at least partly, be integrated into each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the functional units below is appropriate.
A first unit is the scaler/inverse transform unit 305. The scaler/inverse transform unit 305 receives quantized transform coefficient as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, etc. as symbol(s) 313 from the parser 304. It can output blocks comprising sample values, that can be input into aggregator 310.
In some cases, the output samples of the scaler/inverse transform 305 can pertain to an intra coded block; that is: a block that is not using predictive information from previously reconstructed pictures, but can use predictive information from previously reconstructed parts of the current picture. Such predictive information can be provided by an intra picture prediction unit 307. In some cases, the intra picture prediction unit 307 generates a block of the same size and shape of the block under reconstruction, using surrounding already reconstructed information fetched from the current (partly reconstructed) picture 309. The aggregator 310, in some cases, adds, on a per sample basis, the prediction information the intra prediction unit 307 has generated to the output sample information as provided by the scaler/inverse transform unit 305.
In other cases, the output samples of the scaler/inverse transform unit 305 can pertain to an inter coded, and potentially motion compensated block. In such a case, a Motion Compensation Prediction unit 306 can access reference picture memory 308 to fetch samples used for prediction. After motion compensating the fetched samples in accordance with the symbols 313 pertaining to the block, these samples can be added by the aggregator 310 to the output of the scaler/inverse transform unit (in this case called the residual samples or residual signal) so to generate output sample information. The addresses within the reference picture memory form where the motion compensation unit fetches prediction samples can be controlled by motion vectors, available to the motion compensation unit in the form of symbols 313 that can have, for example X, Y, and reference picture components. Motion compensation also can include interpolation of sample values as fetched from the reference picture memory when sub-sample exact motion vectors are in use, motion vector prediction mechanisms, and so forth.
The output samples of the aggregator 310 can be subject to various loop filtering techniques in the loop filter unit 311. Video compression technologies can include in-loop filter technologies that are controlled by parameters included in the coded video bitstream and made available to the loop filter unit 311 as symbols 313 from the parser 304, but can also be responsive to meta-information obtained during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, as well as responsive to previously reconstructed and loop-filtered sample values.
The output of the loop filter unit 311 can be a sample stream that can be output to the render device 312 as well as stored in the reference picture memory 557 for use in future inter-picture prediction.
Certain coded pictures, once fully reconstructed, can be used as reference pictures for future prediction. Once a coded picture is fully reconstructed and the coded picture has been identified as a reference picture (by, for example, parser 304), the current reference picture 309 can become part of the reference picture buffer 308, and a fresh current picture memory can be reallocated before commencing the reconstruction of the following coded picture.
The video decoder 300 may perform decoding operations according to a predetermined video compression technology that may be documented in a standard, such as ITU-T Rec. H.265. The coded video sequence may conform to a syntax specified by the video compression technology or standard being used, in the sense that it adheres to the syntax of the video compression technology or standard, as specified in the video compression technology document or standard and specifically in the profiles document therein. Also necessary for compliance can be that the complexity of the coded video sequence is within bounds as defined by the level of the video compression technology or standard. In some cases, levels restrict the maximum picture size, maximum frame rate, maximum reconstruction sample rate (measured in, for example megasamples per second), maximum reference picture size, and so on. Limits set by levels can, in some cases, be further restricted through Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.
In an embodiment, the receiver 302 may receive additional (redundant) data with the encoded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the video decoder 300 to properly decode the data and/or to more accurately reconstruct the original video data. Additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and so on.
Embodiments herein may be applied in such environments, such as 2 or more dimensional video conferencing, surveillance, or hearing aids or karaoke environments or theatre environments or the like that may experience acoustic deterioration and may be improved by audio zoom.
where NL(.) denotes the nonlinear distortion introduced by the loudspeaker, h(t) represents the acoustic path from loudspeaker to microphone, and * denotes linear convolution.
where n(t) represents the background noise, Δt denotes the system delay from microphone to loudspeaker, and G the gain of amplifier. The recursive relationship between y(t) and y(t−Δt) causes re-amplifying of playback signal and leads to a feedback loop that results in an annoying, high-pitched sound, which is known as a form of acoustic deterioration.
Viewing the example 600 of
where m1 and m2 are two microphones of the m-th microphone pair out of M selected microphone pairs.
A directional feature (DF) is incorporated at S604 as a target speaker bias. According to embodiments, this feature computes the averaged cosine distance between the target speaker steering vector and IPD on all selected microphone pairs as
where ∠vθ(m)(f):=2πfΔ(m)cos θ(m)/c is phase of the steering vector for target speaker from θ at frequency f with respect to m-th microphone pair, Δ(m) is the distance between the m-th microphone pair, c is the sound velocity, and vector e(·):=[cos(·), sin(·)]T.
If the T-F bin (t, f) is dominated by the source from θ, then dθ (t, f) will be close to 1, otherwise it deviates towards −1. As a result, dθ (t, f) indicates if a speaker from a desired direction θ dominates in each T-F bin, which drives the network to extract, at S605, the target speaker from the mixture.
According to embodiments, when there is even division of the space into several sectors, such as shown in the demonstration of audio zooming in a 2D FOV feature extraction example 700 of
According to embodiments, a neural network-driven multi-look enhancement employs directional features from various look directions as input. This enables the model to concurrently enhance multiple sampled look directions. A set of K directions in the horizontal plane is sampled. For example, K=36 and the horizontal space is evenly partitioned to 36 sectors. The azimuths of look directions θ1, 2, . . . , 36={5°, 15°, . . . 345°, 355°}, corresponding to the sectors [0, 10], [10, 20], . . . , [340, 350], and [350, 360], respectively. The azimuths of look directions θ1, 2, . . . , K result in K directional feature vectors dθ, k=1, 2, . . . , K. The value of directional feature in a T-F bin is large if a speaker close to the desired direction is dominant in this bin. Furthermore, this value decreases as the source deviates from the desired look direction.
According to exemplary embodiments, see example 800 of in. On the other hand, sectors that fall outside these boundaries are represented by
out. And this difference may be determined by the user input, such as by correlating touched or selected portions of a screen with a displayed area that may be audio-zoomed on that screen. Each of the directional feature dθ
T×F where T and F are the total number of frames and frequency bands of the complex spectrogram, respectively. By concatenating the values of dθ
in, the resulting dimension becomes considerably large. Moreover, this dimension fluctuates with variations in the field size, making it unsuitable for model training. Therefore, embodiments herein introduce a method to consolidate the high-dimensional directional feature sets by employing, at S802, a max operation across the sectors, as detailed in Eq. 5
At the same time, embodiments incorporate, at S803, a counter FOV feature to represent all directional attributes beyond the angular field. Using both these features sharpens the sound capture, particularly near the boundaries of the FOV. This enhances the model's ability to differentiate between sound sources within and outside the targeted field. The counter FOV feature is computed similarly as
There are two methods to further consolidate the two FOV feature vectors according to exemplary embodiments. A first approach, at S804, involves concatenation, represented as
And a second, at S805, is through post-processing, as calculated in
The post-processing aims to remove the components of speakers outside the desired region, thereby enhancing the prominence of the speakers within the region. The feature 2D extraction in horizontal plane is shown in the demonstration of audio zooming in 3D FOV feature extraction example 900 of
To adapt the aforementioned feature extraction to a 3D space, exemplary embodiments modify the original directional feature dθ in Eq. 4 such that ∠vθ(m)(f):=2πfΔ(m)cos θ(m)cos α(m)/c with α representing the elevation angle in the vertical dimension. Concurrently, the definitions of in and
out will also account for the preferred elevation span. As illustrated in
in, shape the pyramid defined by
Embodiments, such as with example 1000 of
vm(t) denotes the noise signal received by the m-th microphone. By, at S1002, computing STFT (512 window size and 256 hop size), a reference channel, e.g. the first channel complex spectrogram Y1, is used to compute, at S1003, logarithm power spectrum (LPS) by LPS=log(|Y1|2)∈T×F.
The single-channel spectral feature LPS, combined with the introduced audio zooming FOV feature , are, at S1004, merged and utilized as the input for the neural network.
Embodiments exclude the direct current (DC) frequency band, specifically the initial frequency band. As a result, F is set to 256.
For each time frame, the input feature vector, Ψ, undergoes dimension projection to F via a linear projection layer, transitioning from ψ∈T×3F to
T×F. Using the FOV feature outlined in Eq. 7 as a reference, the total input feature dimension along the frequency axis equates to 3F according to embodiments.
Subsequent to processing, at S1005, through a gated recurrent unit (GRU) encoder layer, the model, according to embodiments, deduces, at S1006, T×F dimensional complex-valued masks: in and
out for source estimations within and outside the FOV, respectively, via four linear layers. Noise signals are associated with the sources outside the FOV since noise reduction is integral to our objective. The two masks are multiplied with each microphone spectrum Ym to, at S1007, estimate the target signals and interfering signals at each microphone m, resulting Yinm and Youtm, respectively. M channels of real and imaginary parts of Yin and Yout are then concatenated, represented as Φ. According to embodiments, layer normalization may be utilized to normalize Φ, which in turn equalizes the training loss and gradients among various chunks within a mini-batch. After a linear projection with leakyRELU activation, a RNN layer together with a linear layer receive the sub-band embedding of individual frequency bands Φproj(f)∈
T×E, where f=1, 2, . . . F and E=32 is the dimension of the sub-band embedding, to estimate the enhancement filters at the corresponding frequency as W(f)∈
T×M. Embodiments compute, at S1008, the output signal Ŝ by WH(f)Y(f). In the end, the output's DC band is padded, at S1009, with zeros to, at S1010, compute the inverse STFT, facilitating the production of a time-domain signal and thereby output of more appropriately audio-zoomed audio. The model structure is shown in the subband audio zooming model diagram 1100 of
As such, embodiments herein represent a straightforward and potent techniques regarding angular region feature that allows the neural network model to perform audio zooming efficiently. The methods of random field of view sampling, converting field boundaries into look directions, and using a silence ratio-dependent loss function are all essential components for achieving beam-width adjustable neural beamforming. Experimental results have shown the promise and industrial applicability of the pioneering deep learning-based audio zooming approach according to embodiments herein.
That is, with the increasing deployment of microphone arrays, multi-channel techniques have gained greater importance. An array of microphones can produce several recordings, which carry spatial information about the source of a sound. The effectiveness of established spatial features, such as inter-channel phase difference (IPD), has been demonstrated when integrated with monaural spectral features at the input stage for speech separation methods that use time-frequency (T-F) masking. Additionally, to further improve the extraction of source signals from a specific direction, meticulously crafted directional features indicating the dominant directional source in each time-frequency (T-F) bin have been introduced as the input features to the neural network models. These directional features are contingent upon a specific direction of interest. Drawing inspiration from previous multi-look neural beamformer and the multi-zone neural beamformer techniques, embodiments herein have introduced a FOV feature that consolidates all directional features within the desired field of interest. Simultaneously, embodiments integrate a counter FOV feature to signify all the directional features outside the angular field. Employing both these features refines sound capture, especially around the FOV's edges. Contrary to a directional feature, the FOV feature signifies the prevalence of all audio sources within the user-specified field of view. When this feature is input into the neural network, this features according to embodiments ensures the capture of all sound sources or speakers within the designated angular field, while attenuating those outside this zone.
As such, key features of embodiments of this disclosure may be understood as representing technical improvements to (1) Field of View (FOV) Features (Embodiments herein introduce a unique FOV feature that consolidates all directional attributes within a user-defined region or angular field. It encapsulates sound signals from a designated region, enhancing the capture of desired sound sources), (2) Counter FOV Features (This is a complementary feature to the FOV Features. The counter FOV features capture and represent directional characteristics outside the desired angular field or region. It aids in refining the sound capture, particularly around the edges of the FOV.), (3) Integration with Neural Network Models (The FOV and counter FOV features are designed to be input features for neural network models. This integration ensures efficient extraction of sound sources within the designated FOV, emphasizing the capture of all desired sound sources and attenuating undesired ones), and (4) Low-Power Subband Model for Real-time Applications (Embodiments herein seamlessly incorporates the angular FOV feature into a low-power subband model, and is thereby applied to real-time applications according to embodiments.)
To achieve such features, the following improved functions are provided by exemplary embodiments: (1) Selective Sound Enhancement (Embodiments herein allow users to selectively focus on and enhance audio signals from a specific region while attenuating signals from undesired directions), (2) Isolation of Targeted Sounds (Especially useful in applications like teleconferencing, the feature can isolate sounds emanating from target speakers, thereby improving speech clarity and reducing background noises), (3) Sound Monitoring in Specific Regions (In surveillance scenarios, the audio zooming feature focuses on sounds originating from specific regions of interest, suppressing unwanted noises from other areas.), (4) Enhanced Audio-Visual Experience (In broadcasting, viewers can choose to listen selectively to specific sound effects or commentaries, enhancing their overall experience.), and (5) Adaptive Sound Capture (Embodiments herein adapt based on user preferences, ensuring the sound from the desired region is pronounced, and as users zoom out, ambient sounds are no longer diminished.).
Exemplary embodiments herein are applied to various scenarios such as (1) Teleconferencing (Improve the intelligibility of speech by isolating the sound signals from the target speakers' region and reducing background noises.
Accordingly, embodiments herein provide a simple yet effective FOV feature, amalgamating all directional attributes within the user-defined field. In conjunction, embodiments have introduced a counter FOV feature capturing directional aspects outside the desired field. Such advancements ensure refined sound capture, particularly emphasizing the FOV's boundaries, and guarantee the enhanced capture of all desired sound sources inside the user-defined filed. Experimental results have demonstrated the efficacy of the introduced angular FOV feature and its seamless incorporation into a low-power subband model suited for and applied to real-time applications.
The techniques described above, can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media or by a specifically configured one or more hardware processors. For example,
The computer software can be coded using any suitable machine code or computer language, that may be subject to assembly, compilation, linking, or like mechanisms to create code comprising instructions that can be executed directly, or through interpretation, micro-code execution, and the like, by computer central processing units (CPUs), Graphics Processing Units (GPUs), and the like.
The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, internet of things devices, and the like.
The components shown in
Computer system 1200 may include certain human interface input devices. Such a human interface input device may be responsive to input by one or more human users through, for example, tactile input (such as: keystrokes, swipes, data glove movements), audio input (such as: voice, clapping), visual input (such as: gestures), olfactory input (not depicted). The human interface devices can also be used to capture certain media not necessarily directly related to conscious input by a human, such as audio (such as: speech, music, ambient sound), images (such as: scanned images, photographic images obtain from a still image camera), video (such as two-dimensional video, three-dimensional video including stereoscopic video).
Input human interface devices may include one or more of (only one of each depicted): keyboard 1201, mouse 1202, trackpad 1203, touch screen 1210, joystick 1205, microphone 1206, scanner 1208, camera 1207.
Computer system 1200 may also include certain human interface output devices. Such human interface output devices may be stimulating the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human interface output devices may include tactile output devices (for example tactile feedback by the touch-screen 1210, or joystick 1205, but there can also be tactile feedback devices that do not serve as input devices), audio output devices (such as: speakers 1209, headphones (not depicted)), visual output devices (such as screens 1210 to include CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch-screen input capability, each with or without tactile feedback capability—some of which may be capable to output two dimensional visual output or more than three dimensional output through means such as stereographic output; virtual-reality glasses (not depicted), holographic displays and smoke tanks (not depicted)), and printers (not depicted).
Computer system 1200 can also include human accessible storage devices and their associated media such as optical media including CD/DVD ROM/RW 1220 with CD/DVD 1211 or the like media, thumb-drive 1222, removable hard drive or solid state drive 1223, legacy magnetic media such as tape and floppy disc (not depicted), specialized ROM/ASIC/PLD based devices such as security dongles (not depicted), and the like.
Those skilled in the art should also understand that term “computer readable media” as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.
Computer system 1200 can also include interface 1299 to one or more communication networks 1298. Networks 1298 can for example be wireless, wireline, optical. Networks 1298 can further be local, wide-area, metropolitan, vehicular and industrial, real-time, delay-tolerant, and so on. Examples of networks 1298 include local area networks such as Ethernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LTE and the like, TV wireline or wireless wide area digital networks to include cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial to include CANBus, and so forth. Certain networks 1298 commonly require external network interface adapters that attached to certain general-purpose data ports or peripheral buses (1250 and 1251) (such as, for example USB ports of the computer system 1200; others are commonly integrated into the core of the computer system 1200 by attachment to a system bus as described below (for example Ethernet interface into a PC computer system or cellular network interface into a smartphone computer system). Using any of these networks 1298, computer system 1200 can communicate with other entities. Such communication can be uni-directional, receive only (for example, broadcast TV), uni-directional send-only (for example CANbusto certain CANbus devices), or bi-directional, for example to other computer systems using local or wide area digital networks. Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.
Aforementioned human interface devices, human-accessible storage devices, and network interfaces can be attached to a core 1240 of the computer system 1200.
The core 1240 can include one or more Central Processing Units (CPU) 1241, Graphics Processing Units (GPU) 1242, a graphics adapter 1217, specialized programmable processing units in the form of Field Programmable Gate Areas (FPGA) 1243, hardware accelerators for certain tasks 1244, and so forth. These devices, along with Read-only memory (ROM) 1245, Random-access memory 1246, internal mass storage such as internal non-user accessible hard drives, SSDs, and the like 1247, may be connected through a system bus 1248. In some computer systems, the system bus 1248 can be accessible in the form of one or more physical plugs to enable extensions by additional CPUs, GPU, and the like. The peripheral devices can be attached either directly to the core's system bus 1248, or through a peripheral bus 1249. Architectures for a peripheral bus include PCI, USB, and the like.
CPUs 1241, GPUs 1242, FPGAs 1243, and accelerators 1244 can execute certain instructions that, in combination, can make up the aforementioned computer code. That computer code can be stored in ROM 1245 or RAM 1246. Transitional data can be also be stored in RAM 1246, whereas permanent data can be stored for example, in the internal mass storage 1247. Fast storage and retrieval to any of the memory devices can be enabled through the use of cache memory, that can be closely associated with one or more CPU 1241, GPU 1242, mass storage 1247, ROM 1245, RAM 1246, and the like.
The computer readable media can have computer code thereon for performing various computer-implemented operations. The media and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.
As an example and not by way of limitation, the computer system having architecture 1200, and specifically the core 1240 can provide functionality as a result of processor(s) (including CPUs, GPUs, FPGA, accelerators, and the like) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as introduced above, as well as certain storage of the core 1240 that are of non-transitory nature, such as core-internal mass storage 1247 or ROM 1245. The software implementing various embodiments of the present disclosure can be stored in such devices and executed by core 1240. A computer-readable medium can include one or more memory devices or chips, according to particular needs. The software can cause the core 1240 and specifically the processors therein (including CPU, GPU, FPGA, and the like) to execute particular processes or particular parts of particular processes described herein, including defining data structures stored in RAM 1246 and modifying such data structures according to the processes defined by the software. In addition or as an alternative, the computer system can provide functionality as a result of logic hardwired or otherwise embodied in a circuit (for example: accelerator 1244), which can operate in place of or together with software to execute particular processes or particular parts of particular processes described herein. Reference to software can encompass logic, and vice versa, where appropriate. Reference to a computer-readable media can encompass a circuit (such as an integrated circuit (IC)) storing software for execution, a circuit embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.
While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents, which fall within the scope of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods which, although not explicitly shown or described herein, embody the principles of the disclosure and are thus within the spirit and scope thereof.
