Patent Grant
11989268
The disclosed embodiments generally relate to techniques for using sensor data to authenticate users in computing systems. More specifically, the disclosed embodiments relate to a technique that uses a dummy class framework to facilitate continual supervised classification to authenticate users in a computer system.
There presently exist a number of techniques for authenticating computer system users. At present, most techniques rely on knowledge-based factors, such as passwords. However, passwords have limitations, especially with respect to the user's experience. Human users are simply not good at creating new passwords, which include strong and unique combinations of characters that are also memorable. Furthermore, passwords are commonly phished or stolen. Moreover, the password-creation rules that websites and services enforce are ever-changing and growing increasingly more complex. To keep up with this complexity, users often reuse passwords across multiple services, or make only small, predictable changes among passwords for different services. Also, because passwords are hard to remember, users often write them down or store them in a file for easy access, which also makes them easier to steal. (Note that the term “authenticate” as used in this specification the appended claims refers to both the process of “authenticating” a user and the process of “re-authenticating” a user.)
Some systems rely on a two-factor authentication technique, which requires a user to carry an extra device to authenticate themselves. However, this can be burdensome to the user because the device can be forgotten, run out of power, or break.
One promising technique is to authenticate a user based on sensor data gathered from sensors located in proximity to the user. For example, accelerometer data from a smartphone can be used to recognize a characteristic gait of a user while the user is walking. Other types of sensor data can be used to authenticate a user, including data from: a multi-touch measurement sensor; a force-touch measurement sensor; a sensor in a mouse; a sensor in a trackpad; or an image sensor.
This sensor-data-based authentication technique operates by first training a machine-learning model on sensor data, and then using the trained model to perform classification operations to recognize sensor data associated with specific users. However, in many use cases, not all of the sensor data is available during the initial training of the model. This may be due to the fact that the sensor data has not been collected, processed, or labeled yet. Or, some of the sensor data may only be available on particular devices, and consequently may not be available for initial training. When additional sensor data eventually becomes available, it is generally necessary to retrain the model from scratch to recognize new classes associated with the additional data. This process can be quite time-consuming, especially for mobile computing devices that have limited computing resources.
Hence, what is needed is a technique for efficiently training a machine-learning model to accommodate additional sensor data without the need to start the training process over from the beginning.
The disclosed embodiments provide a system that authenticates a user of an electronic device. During a training mode, the system uses an initial training data set, comprising sensor data from electronic devices associated with a set of initial users, to train a multilayer neural network model to authenticate the initial users. Next, the system uses an additional training data set, which includes sensor data from electronic devices associated with one or more new users, to update a portion of the weights in the trained model so that the updated model can be used to authenticate both the initial users and the one or more new users. During a subsequent mode, the system uses the updated model to authenticate a user of the electronic device based on sensor data contemporaneously received from the electronic device.
In some embodiments, the multilayer neural network model includes one or more dummy nodes, which are not associated with specific users during the initial training process, and which become associated with the one or more new users during the updating process.
In some embodiments, the one or more dummy nodes are located in one or more last layers of the multilayer neural network model.
In some embodiments, while updating the weights in the trained model, the system updates weights in one or more last layers in the multilayer neural network model while maintaining weights in other layers unchanged.
In some embodiments, while updating the weights in the trained model, the system uses a least-squares regression technique to update weights in a last layer in the multilayer neural network model.
In some embodiments, while updating the weights in the trained model, the system uses a standard neural network training technique to update weights in the one or more last layers in the multilayer neural network model.
In some embodiments, the additional training data set comprises the sensor data associated with the one or more new users combined with a sampled subset of the sensor data in the initial training data set.
In some embodiments, while authenticating the user, the system additionally identifies the user.
In some embodiments, the contemporaneously received sensor data includes accelerometer data gathered while the user was walking, wherein the accelerometer data reflects a characteristic gait of the user while walking.
In some embodiments, while training the multilayer neural network model, the system makes use of a federated learning technique and/or a differential learning technique to maintain confidentiality of the sensor data.
In some embodiments, the sensor data is obtained from one or more of the following types of sensor: an accelerometer; a touchscreen; a gyroscope; an inertial sensor; an ambient light sensor; an image sensor; a camera; an odor sensor; a temperature sensor; a barometric-pressure sensor; a cellular radio signal strength sensor; a Bluetooth radio signal strength sensor; a near-field communication (NFC) sensor; a network-proximity sensor; an infrared sensor; a magnetometer; a battery sensor; a force-touch measurement sensor; a multi-touch measurement sensor; a sensor in a mouse; a sensor in a trackpad; and a keyboard-timing sensor.
The disclosed embodiments also provide a system that performs a classification operation on sensor data from an electronic device. During a training mode, the system uses an initial training data set, comprising sensor data received from one or more electronic devices, to initially train a multilayer neural network model to perform a classification operation for an initial set of classes associated with the initial training data set. Next, the system uses an additional training data set, which includes sensor data associated with one or more additional classes, to update a portion of the weights in the trained model so that the updated model can perform classification operations for the initial set of classes and the one or more additional classes. During a subsequent mode, the system uses the updated model to perform a classification operation based on sensor data contemporaneously received from the electronic device.
The following description is presented to enable any person skilled in the art to make and use the present embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present embodiments. Thus, the present embodiments are not limited to the embodiments shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.
The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing computer-readable media now known or later developed.
The methods and processes described in the detailed description section can be embodied as code and/or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and/or data stored on the computer-readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the computer-readable storage medium. Furthermore, the methods and processes described below can be included in hardware modules. For example, the hardware modules can include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), and other programmable-logic devices now known or later developed. When the hardware modules are activated, the hardware modules perform the methods and processes included within the hardware modules.
Computing Environment
These computing devices typically include sensors that can be used to authenticate the user, or verify that an action is legitimately performed by the user. For example, mobile devices, such as smartphone 104, wearable device 108, a tablet computer or other personal electronic device, include sensors such as: a touchscreen; an accelerometer; a gyroscope; an inertial sensor; a virtual keyboard-timing sensor; an ambient light sensor; a camera; an odor sensor; a temperature sensor; a barometric-pressure sensor; a cellular radio signal strength sensor; a Bluetooth radio signal strength sensor; a near-field communication (NFC) sensor; an infrared sensor; a battery sensor; a force-touch measurement sensor; a multi-touch measurement sensor; and a microphone. (Note that instead of using a microphone to collect sound and voice, sound and voice can alternatively be captured using motion sensors, such as an accelerometer or a gyroscope.) Laptop computer 106 or a desktop computer may include sensors, such as: a keyboard-timing sensor; a mouse sensor; a trackpad sensor; a Bluetooth radio signal strength sensor; an image sensor or camera; a network proximity sensor; an infrared sensor; and a battery sensor.
Computing environment 100 can include other devices with sensors, which are located in, on or near a person's body, including: medical indigestibles, subdermal implants, electronic tattoos, smart clothing or accessories, and physical authentication devices.
In addition, computing environment 100 can include other devices in the vicinity of user 102 having sensors and network connectivity, such as: a camera 103, a Wi-Fi® access point 110, a Bluetooth™ beacon 112, a home thermostat, automobile components, security systems, televisions and other entertainment devices, and IoT (Internet of Things) devices.
Note that the sensors can also provide side-channel information gleaned from measurements not necessarily designed to directly measure the element in question, such as: an accelerometer; a gyroscope; an ambient light sensor; a magnetometer; a barometer; a proximity sensor; a thermometer; and an image sensor.
Moreover, the data obtained from the sensors can be related to the interaction of multiple devices, such as: (1) the presence of different devices within wireless range or on the same local network; (2) the relative signal strength, noise, packet loss, bandwidth readings, or other statistics among multiple devices, or between a mobile device such as a mobile phone and a fixed device such as a wireless access point; (3) a combination of audio data from multiple microphones that measure ambient noise; (4) audio signals generated by playing sounds on speakers and listening for those sounds or echoes of those sounds on nearby microphones; (5) signals obtained by recording ambient light data from multiple light sensors; and (6) signals that indicate when a device is attached/paired with another device, or comes into range or communication with the other device (or when a device is detached/unpaired or goes out of range).
Also, data obtained from sensors can include semantic information about the usage of a service, such as: (1) a specific sequence of interactions with the service and the timings between the interactions; (2) when there exist multiple ways to perform a task, the particular user flow used to perform the task; (3) mistakes, typos, or misclicks, or other operations that are typically ignored by the service; (4) metadata about specific interactions, such as what part of a button or screen element was clicked, or readings from other sensors before, during, or after a significant event; and (5) responses and changes in user behavior due to perceptible or imperceptible changes in how a service is rendered or results are presented.
Exemplary Authentication Process
Prior to the authentication process taking place, the user performs a number of initialization operations, such as installing software agents on the user's devices, and importing password data from a password-management system. Next, the system learns familiar user behaviors based on information obtained from the user's devices over time to produce one or more models of the user's behaviors. After these initialization operations are complete, the system is ready to perform the authentication process.
Next, as illustrated by the example in
Dummy Class Framework
The disclosed embodiments provide a framework for training machine-learning models to be deployed in supervised classification scenarios where not all of the labeled data is available during the initial training of the model, and there is a need to supplement, fine-tune, or customize a model given additional data without the need to start the training process over from scratch. This may be due to the fact that the data has not been collected, processed, or labeled yet. Another scenario is that some of the data is only available in particular locations or on particular devices, and is therefore not available during initial training. Hence, in general, our system addresses the problem of sequential supervised classification that falls within the field of lifelong learning (LL) or continual learning (CL) where new classes appear sequentially and a model needs to be trained and deployed for each class.
The main idea is to “pre-allocate” a number of dummy output nodes that will be used to represent the classes for which there is no data available at initial training time. Because the actual data for those classes is not available, we can use synthetic data (for example, noise, zeros, or data that is synthesized based on statistical information about the data classes that are anticipated) for those classes during the initial training process. Once the actual class data is later available, we can use the data from the new classes to fine-tune the initial network, using training strategies to avoid catastrophic forgetting and to make the fine-tuning process more efficient than retraining the entire network. This has the benefit that the original data is not needed during the fine-tuning process.
In one example, our system collects an in-domain background labeled N-class data set (such as accelerometric gait) that is used to train a universe/template model (UTM). In this example, there also exists a continual stream of new users/classes coming into the system drawn from outside of the existing data set. Hence, each user/class requires an individual model to include the user/class as one of the output classes. (Because we discuss an exemplary embodiment that uses a neural network model to recognize users, in this application the terms “classes” and “users” are used interchangeably.)
To handle this example, we pre-allocate a set of dummy nodes/classes while training the UTM and use a warm-start pre-training strategy, which leverages statistical information about the data classes that are anticipated in the future. Then, when new user/class data arrives, we use this ‘enrollment’ data to fine-tune the warm-started UTM while using a set of strategies to reduce the risk of catastrophic forgetting (often associated with continual learning systems) to generate a user-specific model (USM). A model thus generated can then be used in an inference pipeline during a subsequent mode. Within the subsequent mode, classification is performed by passing the input data tensor through the USM and performing a weighted likelihood ratio binary hypothesis test.
In the context of deep neural networks (DNNs), this methodology can be implemented by pre-inserting dummy/ghost nodes in the output softmax layer of the UTM. Note that these dummy/ghost nodes become useful when we are trying to enroll and deploy a model for a cohort of classes or the modes (sub-classes) of a single class to attain more granular classification.
Our solution generally operates as follows. Step 1: We first train a UTM with dummy nodes dangling at the output layer. (
Step 2: Next, we train the dummy nodes with enrollment data. When a new user/class data arrives for the k dummy nodes, we train the weights associated with the last k classes using an appropriate transfer learning strategy, while minimizing catastrophic forgetting. Note that these k nodes may map to either k different users or k different modes of an incoming new user.
Step 3: Finally, we perform an inference during a subsequent mode. When the subsequent mode tensors arrive, they are passed through the neural network and a tensor is classified as belonging to the newly enrolled user's class if the softmax score(s) pertaining to the dummy-node(s) is greater than a specific threshold. (See the exemplary code illustrated in
Our new system provides a number of advantages. (1) No change from a conventional DNN architecture is required. (2) The same enrollment and inference code can be reused and shared across all the users. (3) Pre-training of weights can be used to accelerate enrollment. (4) Finally, training of new classes can happen in a distributed manner, and it is not necessary to transmit the original data to the location where the training of new classes occurs; you only need to send the universe/template model (UTM).
Efficiently Determining Weights for the Last Layer
In step 2 above, while training weights associated with the last k classes, the weights can be determined much more efficiently than by using conventional neural network training techniques for the special case of the last layer of the neural network. During this process, we: take the training data for the new classes; pass it through the NN; and cut off the NN before the last layer and you get the features F. So, we have the features F and we know what the ground truth (for the outputs) Y is. The only thing that is missing are the weights W of the last layer. More specifically, for a set of outputs Y and a set of features F, we want to determine a weight matrix W, so that Y=FW. This problem can be solved very efficiently using a least-squares regression technique as is described in more detail below.
The least-squares regression technique proceeds as follows. Start with the following data items and model.
Step 1: First obtain the feature-extraction model by slicing the “template mod” trained previously.
Step-2: Next, use the feature-extraction model to extract the features.
Step 3: Prepare the combined tensors for LS regression.
Step 4: Perform the LS regression and extract the associated weights and biases.
Step 5: Populate the last layer weights with the newly estimated weights and biases.
Step-6: Save the model.
The above-described authentication process can also make use of a “federated learning technique” to train the neural network model. Federated learning is a machine-learning-based methodology that trains an inferential model across multiple decentralized devices or servers holding local data, without exchanging the local data. (See Kairouz, Peter, et al. “Advances and Open Problems in Federated Learning” arXiv:1912.04977 [cs.LG], 10 Dec. 2019.) By limiting the exchange of local sensor data, federated learning facilitates keeping the local sensor data confidential.
The authentication process can also make use of a “differential privacy” system while training the neural network model. Differential privacy refers to a methodology for publicly sharing information about a dataset by describing the patterns of groups within the dataset while withholding information about individuals in the dataset. (See Irit Dinur and Kobbi Nissim. 2003. Revealing information while preserving privacy. In Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems (PODS '03). ACM, New York, N.Y., USA, 202-210.) By withholding information about individuals in the dataset, differential privacy also facilitates keeping the local sensor data confidential.
Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
The foregoing descriptions of embodiments have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the present description to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present description. The scope of the present description is defined by the appended claims.
This application claims the benefit of U.S. Provisional Patent Application Ser. No. 62/975,670, entitled “Dummy Class Framework for Continual Supervised Classification Problems,” by inventors Vinay Uday Prabhu and John C. Whaley, filed on 12 Feb. 2020, the contents of which are incorporated by reference herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 11126716 | Wang | Sep 2021 | B2 |
| 11127403 | Medalion | Sep 2021 | B2 |
| 11170320 | Garg | Nov 2021 | B2 |
| 11436344 | Juch | Sep 2022 | B1 |
| 11514171 | Dinh | Nov 2022 | B2 |
| 11526405 | Fisher | Dec 2022 | B1 |
| 11551173 | Bertoni Scarton | Jan 2023 | B2 |
| 11650749 | Tran | May 2023 | B1 |
| 20180189466 | Raviv | Jul 2018 | A1 |
| 20190164054 | Lee | May 2019 | A1 |
| 20200098461 | Macoviak | Mar 2020 | A1 |
| 20200125706 | Adir | Apr 2020 | A1 |
| 20210089878 | Greenewald | Mar 2021 | A1 |
| 20210157945 | Cobb | May 2021 | A1 |
| 20210160687 | Ross | May 2021 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20210248215 A1 | Aug 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62975670 | Feb 2020 | US |
