The present description generally relates to frequency-aware masked autoencoders for multimodal pretraining on frequency-based signals.
Various physiological parameters of a user can be measured and analyzed to estimate other physiological measures indicative of the user's physiological state. Computer hardware has been utilized to make improvements across different industry applications including applications used to assess and monitor physiological activities.
Certain features of the subject technology are set forth in the appended claims. However, for purpose of explanation, several embodiments of the subject technology are set forth in the following figures.
The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology can be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein and can be practiced using one or more other implementations. In one or more implementations, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.
Machine learning has seen a significant rise in popularity in recent years due to the availability of training data, and advances in more powerful and efficient computing hardware. Machine learning may utilize models that are executed to provide predictions in particular applications.
Recent advancements in wearable device technology have led to the capability to record various physiological signals, which can be utilized for monitoring the overall wellness of users including mental and/or emotional states. Two of the most commonly collected physiological signals from wearable devices are photoplethysmography (PPG) and electrocardiogram (ECG). Cardiac electrical activity is measured by ECG, containing information about cardiovascular health, while volumetric changes in arterial blood flow are measured by PPG, encompassing a wide range of biological information. These physiological signals are employed on a wearable electronic device for the detection of certain health conditions, such as atrial fibrillation, and the monitoring of specific health metrics, such as blood oxygen. The manifestation of physical and mental states in a user can be achieved through a variety of complementary physiological responses or frequency-based signals. For instance, brain activities indicative of human emotions can be detected through electroencephalography (EEG), while facial expressions like smiling can be decoded by recording muscle contractions using electromyography (EMG). Moreover, a combination of these modalities enables the decoding of emotions, such as happiness.
Inspired by advancements in foundation models for language and vision, the use of transformers and large-scale pretraining for frequency-based signals is explored in the present disclosure. An architecture for frequency-based signals is introduced, which can be trained on multiple modalities and easily adapted to new modalities or tasks. The proposed model incorporates three key features: (i) an efficient identification of local and global information from frequency-based signals by leveraging global filters in the frequency space through a frequency-aware architecture; (ii) the sharing of encoder weights across different channels using either general-purpose or modality-specific filters, achieved by a channel-independent design; and (iii) the effective combination of an arbitrary number of modalities through a modality-fusion transformer. The robustness of the proposed architecture is demonstrated on multiple frequency-based signal datasets, exhibiting superior performance compared to single-modality models and outperforming in transfer learning tasks. In the present disclosure, the utilization of the multi-modal information not only facilitates the construction of enhanced and resilient representations of the human body and mental states but also provides valuable insights to researchers about the contributions of each frequency-based signal to specific predictions and the extent of their informational overlap.
In the domain of multi-modal learning, challenges in data acquisition often confront task-specific datasets. As a result, models trained on such datasets tend to exhibit fragility and sensitivity to data distribution and task specification. Recently, remarkable generalization and zero-shot capabilities have been demonstrated in the language-vision domains through large-scale multimodal pre-training. Considering these advancements, an investigation is undertaken to explore the application of similar pre-training in the frequency-based signal domain. However, performing multimodal pre-training on frequency-based signals proves to be particularly challenging due to significant distributional shifts between the pre-training and downstream datasets. These challenges may fall into two categories: (i) substantial distributional shifts within each modality for frequency-based signals, where data variations occur across tasks, subjects, and even recording sessions within subjects, attributed to slight changes in sensor placement and recording conditions, or, (ii) multimodal frequency-based signals may encounter strong domain shifts across modalities, leading to alterations in the connections between different modalities. These cross-modal domain shifts may arise from unimodal shifts, where changes in inputs within a single modality can affect the connections across different modalities. Moreover, multi-modal frequency-based signals often face scenarios of modality mismatch, wherein modalities may be dropped or replaced with new modalities providing consistent bio-information. Effectively addressing these issues is crucial to harness the potential of multi-modal pre-training in frequency-based signals.
The subject technology provides techniques for an architecture that incorporates frequency information in time series to mitigate distributional shifts and enable multimodal pre-training on frequency-based signals. The model includes two key components: (i) a frequency-aware learning module and (ii) a frequency-preserving pre-training module. The effectiveness of using frequency domain information to address generalization issues has been shown in previous works, either relying on encoders from both the time and frequency domains, or utilizing complicated sampling and combining modules to incorporate frequency information. A multi-head frequency filter layer is provided and coupled inside a transformer architecture to build a frequency-aware learning module. Furthermore, to extend this frequency awareness into a multimodal pre-training setting, it is coupled with a frequency-preserving pre-training design. The method can modify legacy masked autoencoding strategies to preserve frequency awareness during reconstruction, and the corresponding channel independence design ensures robustness to domain shifts in multimodal connections. The model presents a pure reconstruction-based multimodal pre-training architecture. Coupled with the channel-independence design, the model demonstrates robustness for domain shifts across modalities. The subject technology is systematically evaluated over several datasets and benchmarks. It facilitates transfer learning on frequency-based signals. The architecture can integrate and leverage information across different modalities, demonstrating robustness to modality mismatch scenarios commonly encountered in real-world cases.
A robust and computationally efficient frequency-aware transformer encoder with a multi-head frequency filter operation is proposed for performing frequency-aware learning of frequency-based signals. A frequency-preserving pre-training module is introduced to effectively integrate and utilize information across different modalities, creating a representation that is resilient to domain shifts. The method under systematic evaluation can demonstrate its robustness for two types of domain shifts: shifts within each modality and across modalities. Additionally, pre-training on multiple modalities can enhance transfer/generalization ability, making it practical for various real-world scenarios.
Implementations of the subject technology improve the ability of a given electronic device to provide sensor-based, machine-learning generated feedback to a user (e.g., a user of the given electronic device). These benefits therefore are understood as improving the computing functionality of a given electronic device, such as an end user device which may generally have less computational and/or power resources available than, e.g., one or more cloud-based servers.
The network environment 100 includes an electronic device 110, an electronic device 112, an electronic device 114, an electronic device 116, an electronic device 118, and a server 120. The network 106 may communicatively (directly or indirectly) couple the electronic device 110 and/or the server 120. In one or more implementations, the network 106 may be an interconnected network of devices that may include, or may be communicatively coupled to, the Internet. For explanatory purposes, the network environment 100 is illustrated in
The electronic device 110 may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like. In
The electronic device 112 may be, for example, desktop computer, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera, headphones), a tablet device, or a wearable device such as a head mountable portable system, that includes a display system capable of presenting a visualization of an extended reality environment to a user. In
The electronic device 114 may be, for example, desktop computer, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like. In
The electronic device 116 may be, for example, desktop computer, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like. In
The electronic device 118 may be, for example, desktop computer, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like. In
In one or more implementations, one or more of the electronic devices 110-118 may provide a system for training a machine learning model using training data, where the trained machine learning model is subsequently deployed to one or more of the electronic devices 110-118. Further, one or more of the electronic devices 110-118 may provide one or more machine learning frameworks for training machine learning models and/or developing applications using such machine learning models. In an example, such machine learning frameworks can provide various machine learning algorithms and models for different problem domains in machine learning. In an example, the electronic device 110 may include a deployed machine learning model that provides an output of data corresponding to a prediction or some other type of machine learning output. In one or more implementations, training and inference operations that involve individually identifiable information of a user of one or more of the electronic devices 110-118 may be performed entirely on the electronic devices 110-118, to prevent exposure of individually identifiable data to devices and/or systems that are not authorized by the user.
The server 120 may form all or part of a network of computers or a group of servers 130, such as in a cloud computing or data center implementation. For example, the server 120 stores data and software, and includes specific hardware (e.g., processors, graphics processors and other specialized or custom processors) for rendering and generating content such as graphics, images, video, audio and multi-media files. In an implementation, the server 120 may function as a cloud storage server that stores any of the aforementioned content generated by the above-discussed devices and/or the server 120.
The server 120 may provide a system for training a machine learning model using training data, where the trained machine learning model is subsequently deployed to the server 120 and/or to one or more of the electronic devices 110-118. In an implementation, the server 120 may train a given machine learning model for deployment to a client electronic device (e.g., the electronic device 110, the electronic device 112, the electronic device 114, the electronic device 118). In one or more implementations, the server 120 may train portions of the machine learning model that are trained using (e.g., anonymized) training data from a population of users, and one or more of the electronic devices 110-118 may train portions of the machine learning model that are trained using individual training data from the user of the electronic devices 110-118. The machine learning model deployed on the server 120 and/or one or more of the electronic devices 110-118 can then perform one or more machine learning algorithms. In an implementation, the server 120 provides a cloud service that utilizes the trained machine learning model and/or continually learns over time.
In the example of
Electronic devices (e.g., 110-118) including wearable devices (e.g., 112, 114, 118) can facilitate the provision of improved and objective data concerning the timing and frequency of seizures. It has been observed that patient-reported seizure counts do not consistently offer precise information, with a notable number of all seizures remaining undocumented. This underreporting stems from factors beyond the patient's control, such as postictal seizure unawareness or amnesia. For instance, seizures occurring during sleep and focal impaired awareness seizures pose the highest risk of going unrecorded. Accurate seizure counts play a crucial role in medication adjustments, monitoring changes, and other relevant aspects. Notably, status epilepticus, defined as seizure activity lasting longer than 5 minutes, can lead to permanent brain damage. Timely notification of caregivers regarding seizure activity holds the potential to prevent permanent brain damage caused by status epilepticus.
In one or more implementations, any one of the electronic devices 110-118 may include a seizure detection system that incorporates (1) multimodal frequency-based signal data acquired from any one of the electronic devices 110-118 and (2) a detection algorithm as will be described in more detail in
In one or more implementations, the frequency-based signals may include electromyography data recorded by at least one of the electronic devices 110-118, such as the electronic device 114, electroencephalography data recorded by at least one of the electronic devices 110-118, such as the electronic device 118, electrocardiography data recorded by at least one of the electronic devices 110-118, such as the electronic device 114, electrooculography data recorded by at least one of the electronic devices 110-118, such as the electronic device 118, and respiration data recorded by at least one of the electronic devices 110-118, such as the electronic device 118, among others.
As illustrated, the electronic device 200 includes training data 210 for training a machine learning model. In an example, the server 120 may utilize one or more machine learning algorithms that uses training data 210 for training a machine learning (ML) model 220. Machine learning model 220 may include one or more neural networks, which will be described in more detail below with reference to
Training data 210 may include activity information associated with measurable frequency-based signals or electrical impulses generated within a user. These signals are produced by various physiological processes in the body and carry important information about the user's health, function, and state. These frequency-based signals can be broadly categorized into different types, including: (1) Electrocardiogram (ECG/EKG), (2) Electroencephalogram (EEG); (3) Electromyogram (EMG); (4) Electrooculogram (EOG); (5) Electrodermal Activity (EDA); (6) Electroretinogram (ERG); (7) Electrogastrogram (EGG); and (8) Electrocorticography (ECoG). In some aspects, ECG/EKG may refer to a frequency-based signal that measures the electrical activity of the heart. It is commonly used to assess heart rate, rhythm, and detect abnormalities in the heart's function. In some aspects, EEG measures the electrical activity of the brain. EEG can be used for diagnosing and studying neurological conditions, monitoring brain activity during sleep, and understanding cognitive processes. In some aspects, EMG may refer to a frequency-based signal measures the electrical activity of muscles. It is useful in diagnosing neuromuscular disorders, assessing muscle function, and monitoring physical rehabilitation progress. In some aspects, EOG records the electrical activity of the muscles around the eyes and is commonly used to monitor eye movements and detect abnormalities related to vision and sleep. In some aspects, EDA may refer to a galvanic skin response (GSR) that measures the electrical conductance of the skin, which can provide information about emotional responses, stress, and arousal. In some aspects, ERG measures the electrical responses of the retina in the eye, aiding in the diagnosis of various visual disorders. In some aspects, EGG records the electrical activity of the stomach and can help in understanding gastric motility and digestive disorders. In some aspects, ECoG involves placing electrodes directly on the brain's surface, used for research and certain clinical applications like epilepsy monitoring. Training data 210 may also include demographic information (e.g., age, gender, BMI, etc.) for a user of the electronic device 110, and/or a population of other users.
A goal of artificial intelligence is the creation of computer agents capable of understanding and processing information from multiple modalities. Recently, considerable research attention has been devoted to training large-scale models capable of effectively utilizing multimodal information, facilitated by the emergence of effective pre-training methods and access to large-scale datasets. Multimodal systems have been developed across various domains, encompassing vision-language systems, vision-audio systems, language-audio systems, as well as their various combinations. Furthermore, robustness has been demonstrated in learning from data with different structures and from diverse sources sharing similar underlying structures, such as different visual modalities like depth or semantic segmentations. Theoretical research has also shown that utilizing multimodal information can provably enhance the quality of latent representations.
Effective and sufficient pre-training plays a vital role in multimodal learning, given the increased diversity and dimensionality associated with multimodal inputs. Pre-training methods for learning on multiple modalities can be categorized as follows: (i) Training separate encoders for each modality, utilizing contrastive methods such as CLIP or ALIGN, which introduce novel objectives to align or fuse representations from different modalities. (ii) Designing a unified architecture for multiple modalities, with either completely shared encoders per-modality or a few shared layers for decoding, often relying on the transformer architecture. The advantage of employing a unified architecture is the ability to share weights across different modalities, reducing additional computation overhead. The subject technology provides for training a unified architecture for multimodal frequency-based signals, incorporating an effective frequency-awareness design.
Frequency-based signals are multivariate time series capturing various physiological processes within the human body. Although crucial for diverse applications like sleep stage detection and disease detection, acquiring labeled frequency-based signals requires labor-intensive efforts with domain experts. To address the need for labeled data, researchers have proposed self-supervised methods to pre-train models with large-scale unlabeled datasets. These methods encompass (i) contrastive approaches, building latent representations based on similarity across samples with different augmentations, (ii) reconstruction-based techniques involving either feature reconstruction or data reconstruction, and (iii) hybrid methodologies. Pre-training on extensive data has shown to enhance downstream task performance, especially when labeled data is scarce at test time. However, existing works have primarily focused on unimodal pre-training and neglected effective utilization of multimodal information. Some even demonstrated that pre-training on multimodal data could lead to performance degradation due to significant modality variations. In contrast, the subject technology provides for multimodal pre-training on frequency-based signals, considering distributional shifts within and across modalities.
Frequency-aware approaches have been demonstrated as effective in learning multivariate time series. For instance, FED-former employs frequency space sampling to enhance time domain learning for time series processing and forecasting. Recently, the use of frequency-aware techniques has extended to frequency-based signals, where electrophysiological signals exhibit rich frequency information through periodic oscillations and analogous patterns. For instance, consistency between time and frequency spaces can be used to guide frequency-based signal learning, leading to robust performance in transfer learning scenarios by increasing network generalizability. Another approach leverages the connection between time and spectral spaces for cross reconstruction. However, prior methods face limitations: to effectively use frequency information, one must rely on a separate encoder in the frequency space or specific-purpose modules for frequency space sampling, making it computationally heavy and infeasible for irregularly sampled time series in real-world settings. Additionally, the complicated architecture hinders pre-training adaptability in a self-supervised manner without affecting frequency information. In contrast, the subject technology deviates from previous methods through its performance with a simplistic frequency-aware design and frequency-maintaining pre-training strategy.
The effectiveness of operations in the frequency space is also demonstrated in other domains, such as computer vision and long sequence processing. In computer vision, combining frequency-aware approaches into learning architectures proves effective for enhancing image fidelity and facilitating effective token mixing, particularly when integrated with transformer architectures to reduce excessive focus on local information. Additionally, mapping signals into the frequency space is found to be advantageous in long-sequence modeling, with applications in state-space models and structured global convolution networks. This can be attributed to the connection between Fourier transform and global circular convolution.
To overcome these distributional shifts and enhance the processing of frequency-based signal data, the subject technology provides for a frequency-aware multi-modal autoencoder (FAMAE). FAMAE incorporates two key strategies: a frequency-aware encoder and frequency-preserving pre-training techniques. The frequency-aware encoder may be configured to handle the frequency information present in the frequency-based signals efficiently. It allows the model to capture important frequency-related patterns and features within the data, which can facilitate addressing domain shifts and improving generalization. Furthermore, the frequency-preserving pre-training strategies are employed to combine information from various modalities. By preserving the frequency awareness during the pre-training phase, the model can build a robust and coherent representation of the frequency-based signal data that is less affected by domain shifts.
The effectiveness of the transformer architecture 400 has been demonstrated in computer vision, natural language processing, and more recently, time series processing domains. The transformer architecture 400 can receive input data containing frequency-based signal information associated with one or more modalities. The individual embedding of each data patch, denoted as X0, is obtained through an initial patchify Multi-Layer Perceptron (MLP) layer 432 as illustrated in
Frequency-Aware Transformer with Multi-Head Frequency Filters
The subject technology provides for the use of the Discrete Fourier Transform (DFT) for processing frequency-based signals and vision data. Given a tokenized sequence of N numbers x∈RN with elements xn, where 0≤n≤N−1, a one-to-one mapping exists in the frequency space z∈CN. The time representation x and the frequency representation z can be interconverted using DFT F(·) and Inverse Discrete Fourier Transform (IDFT) F−1(·), respectively, as illustrated below:
The imaginary unit, denoted by i, is utilized in the context of the Fast Fourier Transform (FFT) algorithms, which capitalize on the periodicity properties and lead to a significant reduction in computational complexity from quadratic to O (N log N).
Consider a tokenized sequence X=[x1, . . . , xN]T2 of N tokens of D-dimensions, transformers aim to learn the interactions across tokens, typically through the self-attention operation. Recently, mixing tokens with frequency-based operations through DFT and IDFT is shown to be a computationally efficient alternative as it considers global-wise information mixing. The token mixing process is theoretically grounded by the Fourier Neural Operators, which is often implemented in its discrete form (denote as K) as such:
Ideally, it is desired for R to be the Fourier transform of a periodic function that allows a Fourier series expansion. To simplify matters, the implementation often takes the form of learnable weights with a shape of .
The transformer architecture includes a frequency-aware transformer encoder 434 as illustrated in
In this regard, instead of employing the multi-head self-attention operation, the subject technology utilizes the multi-head frequency filter layer 440 to facilitate information mixing across a sequence of tokens. The signal is passed through the transformer layer in the following manner:
Each transformer layer is a combination of the multi-head frequency filter layer 440 (Freq-L(·)), the projection (FF(·)), and residual connections.
For a sequence of tokens X∈RN×D, where N represents the total number of tokens, and D denotes the dimensionality of each token, a DFT is initially performed along the patch dimension to generate representations in the frequency space, resulting in Z∈CN×D. To ensure learnability and transferability across frequency-based signals with varying input lengths, the frequency representation is manipulated using multi-head frequency filters K∈CH×D, where H signifies the total number of heads.
Two variants of the model are developed to achieve the manipulated features in frequency space, denoted as {tilde over (Z)}∈CN×D. In the first variant, the system computes queries Q=ZrealW based on the real values of Z, where W E RD×H is a learnable matrix in real space, facilitating simpler manipulation. These resulting queries are utilized to re-weight the kernels, thus obtaining {tilde over (Z)} through the following operation:
In one or more implementations, additional intuition for the multi-head frequency filter layer 440 can be provided by breaking down the computation for each individual filter. For each k-th filter K[k] inside K∈, given latent representation Z=[z1, z2 . . . ZN]T∈
, the expression Z(k)=[z1⊙K[k], z2⊙K[k], . . . , ZN⊙K[k]]T can be computed, where ⊙ represents the Hadamard product between each representation and the learnable filter weights. To learn the combination between different filters, the weights w can be defined that compute {tilde over (Z)}=Σk=1HwkZ(k). In one or more other implementations, to increase the expressiveness of the filtering operation, instead of learning a linear combination of different filters, intuition can be borrowed from the computation of self-attention to compute the queries for the kernel weights w through w=zW, where W∈
. Thus, {tilde over (Z)}[i, j] can be defined as follows:
This expression can yield expression (4) described above. Embodiments of the subject technology utilize the real values of latents to learn the weights of the combiner through w=ZrealW. Similarly, based on the same intuition of combining filtered matrices, the max pooling operation can be realized.
In one or more implementations, the aforementioned operation is equivalent to a self-guided weighted summation between each query and its corresponding modulated frequency representation. Some approaches may utilize filters K of the same shape as Z for element-wise multiplication, which is may not be suitable for time series due to potential significant changes in input length. In contrast, the subject technology employs a flexible multi-head kernel K with a size of N, ensuring model transferability across different subjects and tasks.
Alternatively, the modulated spectral feature matrix {tilde over (Z)} can be obtained by applying each row of K to every row of Z, followed by a max-pooling operation:
The max-pooling operation is performed based on the absolute value of the complex features to obtain the resulting feature in frequency space, denoted as {tilde over (Z)}. Subsequently, the feature is converted back to a representation in time space using the Inverse Discrete Fourier Transform (IDFT), resulting in {tilde over (X)}=F−1(Z). The entire process is denoted as Freq-L(·), ensuring computational efficiency, transferability, and robustness.
The self-supervised pre-training framework, masked autoencoder (MAE) implemented as a frequency-maintain pre-training circuit 436 as illustrated in
To preserve frequency information while enabling pre-training based on the masked autoencoding strategy, masked autoencoding is performed in the latent space using the frequency-maintain pre-training circuit 436. The frequency-aware transformer encoder 434, denoted as FA-Enc(·), learns the full sequence of frequency-based signals S to obtain XL=[x1L, x2L, . . . , xNL]. Patches are randomly sampled with a fixed masking ratio to obtain Zmask, and then processed by a lightweight transformer encoder. The resulting latents are padded with mask tokens and fed to a lightweight transformer decoder to reconstruct {tilde over (s)}i, corresponding to the original signal si. In this regard, the subject technology can produce a trained machine learning model by training a neural network to predict one or more masked frequency components of the frequency-embedded latent representation.
Referred to as MAE(·), FAMAE is designed to optimize the objective below:
In the expression above, Ω is the set of masked tokens. The computation of offsets between original signal patches and reconstructed patches is accomplished using the error term l, set as mean squared error (MSE) in experiments. In some aspects, the performance remains robust when MAE(·) is removed, and only keep FA-Enc(·) is retained during test time. The novelty lies in the observation that utilizing the masked autoencoding objective alone, without any contrastive terms, proves to be effective for frequency-based signals.
Frequency-based signals are multivariate time series that often face channel-wise and modality-wise mismatch scenarios at test time. To obtain robust transfer performance, a channel-independent design may be implemented to model multimodal frequency-based signals.
A multi-channel frequency-based signal [S1, S2, . . . , SC] is provided, where C represents the total number of channels. In one or more implementations, channel independence can be performed through FA-Enc (Si) for channel i∈[1, C]. The channel independence design is implemented by passing each Sξ into FA-Enc(·), and MAE(·) is utilized as illustrated below:
In the expression, Ω is the union of masked tokens for each channel, which is independently determined based on a fixed masking ratio for each channel. The transformer architecture 400 incorporates MAE, featuring one lightweight transformer encoder for information mixing across channels and separate lightweight decoders for reconstruction. The weight of the frequency-aware transformer FA-Enc(·) is shared across channels. By integrating this channel independence into the multi-modal masked autoencoding objective, the subject technology gains the advantage of robustness against multimodal domain shifts, encompassing (i) shifts in connections across channels or modalities and (ii) scenarios with empty or additional channels or modalities, referred to as “channel dropout” and “channel substitute,” respectively.
In one or more implementations, in transfer tasks, embodiments of the subject technology utilize the average of tokens to extract the final representations for the downstream classification. In one or more other implementations, when having multimodal information, fixing the dimensionality of the latent representation when many modalities are present can narrow down the information from each modality, causing potential information loss. Thus, in multimodal tasks, embodiments of the subject technology average the representations from each individual modality, and then concatenate the representations across modalities before performing the downstream classification.
As illustrated in
At block 504, the apparatus transforms the input data from a time domain to a frequency domain.
At block 506, the apparatus generates a frequency-embedded latent representation of the input data comprising time-domain and frequency-domain information.
At block 508, the apparatus generates a masked frequency-embedded latent representation by masking one or more frequency components in the frequency-embedded latent representation.
At block 510, the apparatus produces a trained machine learning model by training a neural network to predict one or more masked frequency components of the frequency-embedded latent representation.
The bus 608 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 600. In one or more implementations, the bus 608 communicatively connects the one or more processing unit(s) 612 with the ROM 610, the system memory 604, and the permanent storage device 602. From these various memory units, the one or more processing unit(s) 612 retrieves instructions to execute and data to process in order to execute the processes of the subject disclosure. The one or more processing unit(s) 612 can be a single processor or a multi-core processor in different implementations.
The ROM 610 stores static data and instructions that are needed by the one or more processing unit(s) 612 and other modules of the electronic system 600. The permanent storage device 602, on the other hand, may be a read-and-write memory device. The permanent storage device 602 may be a non-volatile memory unit that stores instructions and data even when the electronic system 600 is off. In one or more implementations, a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) may be used as the permanent storage device 602.
In one or more implementations, a removable storage device (such as a flash drive, and its corresponding solid state drive) may be used as the permanent storage device 602. Like the permanent storage device 602, the system memory 604 may be a read-and-write memory device. However, unlike the permanent storage device 602, the system memory 604 may be a volatile read-and-write memory, such as random access memory. The system memory 604 may store any of the instructions and data that one or more processing unit(s) 612 may need at runtime. In one or more implementations, the processes of the subject disclosure are stored in the system memory 604, the permanent storage device 602, and/or the ROM 610. From these various memory units, the one or more processing unit(s) 612 retrieves instructions to execute and data to process in order to execute the processes of one or more implementations.
The bus 608 also connects to the input device interface 614 and output device interface 606. The input device interface 614 enables a user to communicate information and select commands to the electronic system 600. Input devices that may be used with the input device interface 614 may include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output device interface 606 may enable, for example, the display of images generated by electronic system 600. Output devices that may be used with the output device interface 606 may include, for example, printers and display devices, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information. One or more implementations may include devices that function as both input and output devices, such as a touchscreen. In these implementations, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
Finally, as shown in
Implementations within the scope of the present disclosure can be partially or entirely realized using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. The tangible computer-readable storage medium also can be non-transitory in nature.
The computer-readable storage medium can be any storage medium that can be read, written, or otherwise accessed by a general purpose or special purpose computing device, including any processing electronics and/or processing circuitry capable of executing instructions. For example, without limitation, the computer-readable medium can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. The computer-readable medium also can include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.
Further, the computer-readable storage medium can include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more implementations, the tangible computer-readable storage medium can be directly coupled to a computing device, while in other implementations, the tangible computer-readable storage medium can be indirectly coupled to a computing device, e.g., via one or more wired connections, one or more wireless connections, or any combination thereof.
Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be realized as executable or non-executable machine code or as instructions in a high-level language that can be compiled to produce executable or non-executable machine code. Further, instructions also can be realized as or can include data. Computer-executable instructions also can be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those of skill in the art, details including, but not limited to, the number, structure, sequence, and organization of instructions can vary significantly without varying the underlying logic, function, processing, and output.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits, such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions that are stored on the circuit itself.
Those of skill in the art would appreciate that the various illustrative blocks, modules, elements, components, methods, and algorithms described herein may be implemented as electronic hardware, computer software, or combinations of both. To illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, components, methods, and algorithms have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. Various components and blocks may be arranged differently (e.g., arranged in a different order, or partitioned in a different way) all without departing from the scope of the subject technology.
It is understood that any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that all illustrated blocks be performed. Any of the blocks may be performed simultaneously. In one or more implementations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
As used in this specification and any claims of this application, the terms “base station”, “receiver”, “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms “display” or “displaying” means displaying on an electronic device.
As used herein, the phrase “at least one of” preceding a series of items, with the term “and” or “or” to separate any of the items, modifies the list as a whole, rather than each member of the list (i.e., each item). The phrase “at least one of” does not require selection of at least one of each item listed; rather, the phrase allows a meaning that includes at least one of any one of the items, and/or at least one of any combination of the items, and/or at least one of each of the items. By way of example, the phrases “at least one of A, B, and C” or “at least one of A, B, or C” each refer to only A, only B, or only C; any combination of A, B, and C; and/or at least one of each of A, B, and C.
The predicate words “configured to”, “operable to”, and “programmed to” do not imply any particular tangible or intangible modification of a subject, but, rather, are intended to be used interchangeably. In one or more implementations, a processor configured to monitor and control an operation or a component may also mean the processor being programmed to monitor and control the operation or the processor being operable to monitor and control the operation. Likewise, a processor configured to execute code can be construed as a processor programmed to execute code or operable to execute code.
Phrases such as an aspect, the aspect, another aspect, some aspects, one or more aspects, an implementation, the implementation, another implementation, some implementations, one or more implementations, an embodiment, the embodiment, another embodiment, some implementations, one or more implementations, a configuration, the configuration, another configuration, some configurations, one or more configurations, the subject technology, the disclosure, the present disclosure, other variations thereof and alike are for convenience and do not imply that a disclosure relating to such phrase(s) is essential to the subject technology or that such disclosure applies to all configurations of the subject technology. A disclosure relating to such phrase(s) may apply to all configurations, or one or more configurations. A disclosure relating to such phrase(s) may provide one or more examples. A phrase such as an aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to other foregoing phrases.
The word “exemplary” is used herein to mean “serving as an example, instance, or illustration”. Any embodiment described herein as “exemplary” or as an “example” is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the term “include”, “have”, or the like is used in the description or the claims, such term is intended to be inclusive in a manner similar to the term “comprise” as “comprise” is interpreted when employed as a transitional word in a claim.
All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed under the provisions of 35 U.S.C. § 112 (f) unless the element is expressly recited using the phrase “means for” or, in the case of a method claim, the element is recited using the phrase “step for”.
The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more”. Unless specifically stated otherwise, the term “some” refers to one or more. Pronouns in the masculine (e.g., his) include the feminine and neuter gender (e.g., her and its) and vice versa. Headings and subheadings, if any, are used for convenience only and do not limit the subject disclosure.
This application claims the benefit of U.S. Provisional Application Ser. No. 63/529,666, entitled “FREQUENCY-AWARE MASKED AUTOENCODERS FOR MULTIMODAL PRETRAINING ON FREQUENCY-BASED SIGNALS,” and filed on Jul. 28, 2023, the disclosure of which is expressly incorporated by reference herein in its entirety.
Number | Date | Country | |
---|---|---|---|
63529666 | Jul 2023 | US |