The present invention relates generally to method an system for identifying a user of a handheld device, e.g. remote control systems. Many systems would benefit from easy, non-intrusive user identifications. For reference only, many aspects of the invention and the background relating to the inventions are described in relation to a remote control system, suitable for control of consumer electronic products and home appliances, that includes a touch sensitive handheld remote control unit that detects holding and grabbing patterns of the user as well as other characteristics such as the trajectory at which the user raises the remote and first touches the remote to identify the user.
Handheld remote control units, typically featuring a large plurality of push buttons, are now quite commonplace on coffee tables throughout the world. With most consumer electronic products, it is customary for the manufacturer to furnish such a handheld remote control with each unit. Thus, most consumers own a collection of various different remote control units, each associated with a particular product or appliance.
In an effort to simplify matters, the Applicants' assignee has developed several different embodiments of a touch-sensitive remote control unit that features a reduced number of push buttons and one or more touch-sensitive touchpads that may be manipulated by the user's fingers or thumb to interact with information on a display screen. The touch pads may be manipulated, for example, to move a selection indicator (such as a cursor or other graphical element) across a control region upon a display screen. In some applications, the display screen will be separate from the handheld remote control unit, and thus the user manipulates the selection indicator by watching the display screen while manipulating the keypad with a finger or thumb. Preferably, the touchpad or touchpads are disposed on the remote control unit so that they can be manipulated by the user's thumb while the user is holding the unit in one hand. Furthermore, the remote control has touch sensitive sensors on its outer casing sensitive to a user's touch.
As multiple users may use a single remote control and corresponding host device, there is a growing demand for a means of identifying which user is using the remote control. Methods such as logging in with a user name and password are time consuming and annoying to users. Thus, there is a need for a means to identify a user that requires minimal user interaction. Optimally, it would be beneficial to allow a user to be identified without the user having to enter or perform any identification tasks. Such a passive means of identification would result in virtually no user interaction on the part of the user.
We have therefore developed a remote control system that implements various passive and semi-passive user identification methods. The methods range from the grab/hold patterns by which the user holds a remote, to the trajectory that the remote follows when grabbed by the user.
This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.
In one sense, the present invention relates to a system and method for identifying a user of a handheld device. The handheld electronic device comprises a housing and a sensor system disposed along a periphery of the housing. The sensor system is responsive to a plurality of simultaneous points of contact between a user's hand and the device to generate observation signals indicative of the plurality of contact points between the user's hand and the device. The handheld electronic device further includes a user identification database storing data corresponding attributes of a plurality of known users, wherein the attributes of the plurality of known users are used to identify a user. The device further comprises a user identification module configured to receive the observation signals from the sensor system and identify the user from the observation signals and the attributes of the plurality of users.
In a second sense, the present relates to a handheld electronic device comprising a housing and a touchpad responsive to a finger movement of a user that generates a touchpad signal corresponding to the finger movement. The device further includes a touchpad processing module that receives the touchpad signal and generates finger movement data based on said touchpad signal. The handheld electronic device further includes a user identification database that stores user identification data corresponding to physical attributes of a plurality of known users, wherein physical attributes includes finger movement of a user drawing a predetermined object. The device is further comprised of a user identification module that receives finger movement data of the user and identifies the user based on the finger movement data and the user identification data, wherein the finger movement data is the user drawing the predefined shape.
In a third sense, a handheld electronic device is comprised of a housing and a touch sensor system disposed along a periphery of the housing. The touch sensor system is responsive to a plurality of simultaneous points of contact between a user's hand and the device to generate observation signals indicative of the plurality of contact points between the user's hand and the device. The device also includes a touch sensor processing module configured to receive the observation signals from the touch sensor system and determine a user's holding pattern. The device is further comprised of an inertial sensor embedded in the housing which is responsive to movement of the device by the user's hand to generate inertial signals and a trajectory module configured to receive the inertial signals from the inertial sensor and determine a trajectory for the movement of the device. The device also includes a touchpad located along an external surface of the housing that is responsive to the user's finger movement along the external surface of the touchpad to generate touchpad signals and a touchpad processing module that receives the touchpad signals and determines user finger movement data. The device further includes a user identification database storing data corresponding to attributes of a plurality of known users, wherein the attributes of the plurality of the known users are used to identify a user, and wherein the attributes include holding patterns of the plurality of known users, trajectories corresponding to movement of the device by each of the plurality of known users, and user finger movement data of the plurality of known users. The device is further comprised of a user identification module configured to receive identification information of the user and identify the user based on the identification information and the attributes of the plurality of known users, wherein the identification information includes the user's holding pattern, the user's trajectory for movement of the device, and the user's finger movement data.
Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.
The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.
Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.
Example embodiments will now be described more fully with reference to the accompanying drawings.
A system and method for user identification is herein disclosed. The system combines one or more user identification techniques to authenticate and/or identify a user. The techniques may be passive techniques such as identifying a user by the way which the user grabs the handheld device, the trajectory that the device follows when the user picks up the remote, the user's first touch of the device, and the user's heartbeat. The techniques may also be semi-passive, such as having the user draw a shape, e.g. a circle, on a touch-sensitive surface of the handheld device. For simplicity, the techniques are first explained as applied to a remote control 12 that may be used with a television, a set-top box, a computer, an entertainment center or other host device. It will be apparent that the techniques are also applied to all handheld devices where user identification would benefit the user. Such applications are described in greater detail below.
Referring to
Any type of communication interface between the handheld remote control 12 unit and the electronic product can be utilized. For purposes of illustration, a wireless transmitting device, shown diagrammatically at 24 and a wireless receiving device, shown diagrammatically at 22, are illustrated. It will be appreciated that wireless communication can be accomplished using infrared, ultrasonic and radio frequencies, and further utilizing a variety of different communication protocols, including infrared communication protocols, Bluetooth, WiFi, and the like. Communication can be unilateral (from remote control unit 12 to electronic product 14) or bilateral.
In the illustrated embodiment, a control region is defined on the screen, within which a user-controlled selection indicator may be visually displayed. In
The system herein disclosed may be used, for example, to identify a mapping for the remote based on the user identification. Although the illustrated embodiment uses a one-to-one mapping between the touchpad surface and the control region, this mapping is altered to accommodate the hand size characteristics of the user. Referring to
The user identification may be used to configure other aspects of the host device controlled by the remote control. For example, if the host device is a set-top box for a television, the user identification may be used to restrict access to certain channels for certain users. Additionally, a list of pre-programmed favorite channels or settings may be loaded onto the set-top box. More examples are provided below.
Referring to
For example, a sensor that provides a resistance relative to the contact points between the user and the device may be used. Such sensors are currently being developed and provide a higher dimensional data set, which is advantageous when identifying a user out of many users. These electrode matrix sensors have one sensor that transmits a signal, e.g. an electric current, and a plurality of receptors that receive the signal via the user's hand (or other body part). The transmitter and the plurality of receptors are placed along the exterior surface of the remote control 12. The plurality of receptors are oriented spatially around the transmitter. The distance between each receptor and the transmitter is known, as is the current and voltage of the transmitted electrical signal. When the electrical signal is transmitted and subsequently received by the receptors, the resistance of the user's hand at the contact points may be determined. As can be appreciated each transmission increases the dimensionality of the data by a factor of X, where X is the ratio of receptors to a transmitter, as each receptor will generate a resistance value. These sensors are particularly helpful if datasets of higher dimensionality are preferred. Furthermore, transfer functions may be performed on the transmitted signals resulting in the communications between the transmitters and the corresponding receptors to further increase the dimensionality.
The remote may also include acoustic (not shown) and optical sensors (not shown), inertial sensors (not shown), a pulse oximiter (not shown) and thermal sensors (not shown).
It is appreciated that the sensors may receive signals from a user holding the remote control 12 with one hand or with two hands. For example, the user may hold the remote control 12 in an operative position with one hand. The user may also hold the remote control 12 with both hands, much like a video game controller, for purposes of identification.
It is appreciated that user identification module 50 may reside on the remote control 12 or the host device. Due to the fact that the system implements powerful learning techniques, remote control 12 may not have the processing power to handle such calculations. If this is the case, then remote control 12 may communicate the input data to the recognition module 50 residing on the host device.
The various processing modules receive raw data from the respective sensor and process the data to a form that may be used by recognition module 90, which may use one or more of a k-means clustering method, a support vector machines (SVM) method, a hidden Markov model method, and a linear regression method to identify a user based on the processed data. As can be appreciated, the various sensors will produce high dimensional data sets, which is beneficial for identification. User identification module 50 may also perform feature extraction on the input data set, so that the high dimensional data set can be more easily processed. Dimensionality reduction methods such as principle component analysis (PCA) and isomap may be implemented by recognition module. The recognition module 90 uses the processed data and the datasets contained in user identification database 62 to classify the user to be identified via various statistical learning and/or clustering methods. The recognition module 90 will then determine the user whose feature data, e.g. hold/grab, trajectory, or first touch, most resembles the input data received from the various sensors. A confidence score may also be associated with the user identification. Furthermore, recognition module 90 may generate a list of the n-nearest matches in the user identification database 62. Additionally, as is described later, recognition module 90 may generate a user identification.
Recognition module 90 may take as parameters, a data set and data type. Based on the data type, recognition module 90 may select which learning or clustering technique to use and which data types to access in user identification database 62.
Alternatively, touch sensor processing module 80 may extrapolate additional data from the raw data. For example, it is discernable which hand (left or right) is grabbing the remote. It is also discernable which sensors were activated by the palm and which sensors are activated by the fingers, due to the fact that the palm is continuous and the fingers have gaps between them. Based on which hand is grabbing the remote and the position of the fingers and the palm, the touch sensor processing module 80 may extrapolate an estimated hand size. Additional feature data may be extrapolated from the initial grab, such as a holding pattern. For example, some users will use three fingers to grab the remote on the side, while other user will use a four finger hold. Also, information relating to the pressure applied to each sensor may also be included. The collection of extrapolated feature data may be used to find a match in the user identification database 64.
Referring now to
At step S122, the finger occlusion patterns are calculated. Similar to the palm occlusion patterns, pressure and pressure points may be determined from the received touch sensor data. From this data, touch processing module 80 can determine the amount of fingers used to grab the remote and the spacing between fingers. The finger occlusion patterns may be combined with the results of step S118 and S116, which provide an estimate of the palm portion of the hand, to determine a hold pattern. This data, along with the hand size data, may be communicated to recognition module 90 and used to match a user in the user identification database at step S126. It is envisioned that many different methods of matching a user may be used. For example, a k-means clustering may be performed on the processed data and the user identification data. Other data mining techniques and statistical learning techniques may also be used to determine a user identification. Furthermore, a confidence score may be attached to the identification, or an n-nearest match list of possible users. In the event a confidence score is used, the system may require a confidence score to exceed a predetermined threshold to identify a user. In the event an n-nearest match list is produced, other identification techniques may be used to pare down the list.
In an alternative embodiment, the method of identifying a user may not initiate until the remote has reached a resting point. Thus, the user will grab the remote control 12, pick up the remote control 12, and then reach the hold position of the remote control 12. Once the inertial sensors indicate that the remote control 12 has reached a steady position, e.g. acceleration or velocity are below a predetermined threshold, then the user identification process may commence. In this embodiment, variations in the initial grab of the remote are entirely ignored, as the grab pattern may be equally dependent on the location of the remote control and the user, e.g. user will grab a remote differently if behind the user or lodged between two couch cushions.
Hold/grab pattern matching may be used as a sole means of user identification, a primary means of user identification or a partial means of user identification. Preliminary research reveals that in a small user group (5 users or the size of a family), hold/grab patterns result in about an 87.5% accuracy in user identification. Thus, depending on the scale and the application of the underlying system, 87.5% may be a sufficient identification accuracy. However, for more sensitive login environments more accuracy may be needed and thus, hand/grab pattern matching may be used as one of a number of matching techniques.
As mentioned, the trajectory processing module 82 may increase prediction accuracy if a starting location is known. It is appreciated that one underlying reason is that the starting location and the trajectory are dependent on one another. The dependency, however, is not necessarily the exact geographic location, but rather the relative location of the remote. For example, a user picking up the remote from the far right end will likely take a similar trajectory when picking the remote up off of the center of the coffee table. The trajectory will differ, however, when the user picks up the remote control 12 off of the couch. Thus, step S148, described above, does not require a pinpoint location. Rather a general location, or a cluster of locations may be used as the starting location. Trajectory processing module may use a k-means clustering algorithm of known locations and an estimated starting location to determine the general starting location.
It should be noted that in certain embodiments the remote control 12 may ask a user to verify a location periodically. Furthermore, in some embodiments, there may be a location registration phase, where the user preprograms the n-most likely locations of a remote. In this embodiment, the user could enter a coffee table, a couch, a side table, the floor, the entertainment center, etc. In such a registration process, the user would have to define the locations with respect to each other.
In other embodiments, the motion processing module 82 will determine a location based on the motion itself. In these embodiments, the motion processing module 82 receives the sensor data and determines a reference trajectory using dead reckoning techniques at step S142. It should be noted that the trajectory is a reference trajectory because it assumes a starting point of (0, 0, 0) and further is used as a reference to determine a starting location. At step S144, the motion processing module 82 may use a k-means cluster algorithm, using the received trajectory as input, to determine the most likely starting location.
Once a starting location is determined, the motion processing module may use the resting location (starting point), the hold position (the end point), and the sensor data, to determine a trajectory. The module processing module 82 will then attempt to find a match in the user identification database 62 for the trajectory, based on the starting point and the trajectory itself. The reason that both parameters may be of importance is that the categorization of a trajectory is dependent on the starting point. For example, two identical trajectories may be reported to motion sensor processing module 72, despite one trajectory beginning on the coffee table and one trajectory beginning on the center of the floor. Without a starting location, it may be very difficult to differentiate the two trajectories. However, with an estimated or known starting location, the motion processing module 72 may differentiate between the trajectories because one started from the floor, while the other started from the coffee table. Thus, it may be determined, for example, that a shorter user (e.g. a child) picked up the remote from the floor in a standing position, and that a taller (e.g. an adult) user picked up the remote from the coffee table. It is envisioned that learning methods such as support vector machines or k-means clustering may be used to determine a user identification based on the calculated trajectory and starting point. It should be noted that the starting point, e.g. couch or coffee table, may be used to pare down the set of trajectories that the input trajectory is compared with. For example, if the motion processing module 82 determines that the user picked the remote control 12 up from the coffee table, i.e. the input trajectory originated from the coffee table, only the set of trajectories originating from the coffee table are used to generate a user identification.
In an alternative embodiment, the starting point of the trajectory is ignored. Rather, a vector representing the relative motion of the remote control is used to identify the user. Thus, the trajectory is assumed to always begin at a (0,0,0) position.
It should be noted that trajectory matching may be used as a sole means of user identification, a primary means of user identification or a partial means of user identification. Depending on the scale of the system and the application of the underlying system, trajectory matching may provide sufficient identification accuracy. However, for more sensitive login environments more accuracy may be needed and thus, trajectory matching may be used as one of a number of identification techniques.
In embodiments of the remote control 12 that include a touchpad, additional identification methods may be enabled.
It should be noted that first touch data may be used as a sole means of user identification, a primary means of user identification or a partial means of user identification. Depending on the scale of the system and the application of the underlying system, first touch data may provide sufficient identification accuracy. However, for more sensitive login environments more accuracy may be needed and thus, first touch data may be used as one of a number of identification techniques.
In some embodiments, the timing data is initially removed and only the coordinate data, i.e. the (x,y) components of the data are used in steps S172-S178. In these embodiments, the results of the k-means clustering may be rescored using the timing data at step S180. Once rescored, a user may be identified at step S182.
It should be noted that shape drawing may be used as a sole means of user identification, a primary means of user identification or a partial means of user identification. In fact, shape drawing typically provides very high identification accuracy rates. Shape drawing, however, is not a passive approach and may, therefore, be implemented as a back up method when the system is unsure of the user's identity after using the passive identification techniques. Depending on the scale of the system and the application of the underlying system, shape drawing may be an advantageous means of protecting more sensitive login environments.
It is envisioned that additional sensors may also be used to identify a user. For example, an acoustic sensor 58 may be used to detect a user's heartbeat. The acoustic sensors 58 may be strategically placed along-side the outer covering of the remote control, whereby the entire remote control 12 acts as an acoustic antenna. When a user holds the device tightly, the acoustic sensors detect the heartbeat and transmit the data to an acoustic processing module 86. Acoustic processing module 86 may process the received data so that a frequency and amplitude of the heartbeat may be determined. The recognition module 90 may then use one or more of the statistical learning or data mining techniques described above to determine if there exists a matching user in the user identification database 62 based on the user's heartbeat characteristics. It is envisioned that other types of sensors may be used to monitor a user's heartbeat or related statistics. For example, a pulse oximeter may be used to measure a patient's pulse or blood-oxygen levels. Additionally, an ultra-sensitive accelerometer may be used to detect vibrations resulting from the user's pulse. Finally, an impulse response system may further be used. An impulse response system is essentially comprised of a speaker and a microphone. The microphone emits a high-frequency sound wave that reverberates through the user's hand. The sound wave may be deflected back to the microphone, where the sensor is able to discern the augmentation of the sound wave. The impulse-response sensors may also be used to measure a user's pulse.
Another additional sensor is an optical sensor 56. The optical sensor 56 may be located on the remote control or on the host device. The optical sensor 56 may be used to receive image data of the user. The image processing module 86 may perform face-recognition or torso recognition on the user for purposes of identifying the user. Yet another sensor is a thermal sensor placed on the outside covering of the remote control 12. The thermal sensors may be used to determine a user's body temperature. Typically, an identification based solely on body temperature may not be reliable. Body temperature data, however, may be useful in increasing the dimensionality of the data sets so that greater separation results in the collection of user attribute data sets.
Individual methods for user identification have been disclosed. All the methods are either passive or semi-passive, as they do not require the user to remember or enter a username, passcode, or other unique identifier. Rather, the techniques rely on a user's natural kinematic tendencies when performing subconscious tasks. In an alternative embodiment, a combination of two or more of the above-described techniques may be used to increase the accuracy of a user identification. As mentioned earlier, each of the individual techniques may have a confidence score associated with an identification. Additionally, an n-nearest match list may also be generated for each identification technique. Based on either the confidence scores or the n-nearest neighbors of multiple user identification efforts, a more accurate user identification may be realized. Taking a sample size of five users, the grab/hold identification method resulted in an 87.5% accuracy rate, the accelerometer-only based trajectory identification resulted in a 77.5% accuracy rate and the gyroscope-only based trajectory identification resulted in a 65% accuracy rate. Taking the three passive identification methods in combination, however, results in a 90% accuracy rate. It should be noted that the circle-drawing based identification resulted in a 97.5% authentication accuracy.
As can be seen in
General reference has been made to the processing of data.
A segmentation module 208 separates portions of the feature vector 204 into segments representing different states. For example, in the example of the remote control being raised in
As mentioned, the segment selection module 214 will select the state of the feature data, and may select the relevant segments 212a-212n for classification. Referring back to the previous example, the segment selection module may be configured to only select trajectories of the remote when picked up and when at the rest position. The feature vector 204 or the selected segments of the feature vector 204 are communicated to a classifier 216. The classifier 216 may use a clustering analysis to determine a user identification. In the cluster analysis, the selected segments are analyzed with user models 220a-220n. The user models 220a-220n represent the attributes of the various users. The segment selection module 214 can also communicate the selected segment to classifier 224 for purposes of classifying the feature vector with the most relevant data. For example, when segment selection module 214 determines that the feature vector 204 is primarily trajectory data corresponding to a remote control pickup, classifier, when accessing the user models, will only retrieve the segments of user models 220a-220n, that correspond to trajectory data. Because the user models have been previously classified, the classifier only needs to determine the cluster, i.e. which user model 220a-220n, that the feature vector or selected segment of the feature vector belongs to. As previously mentioned, classifier 224 may execute a clustering algorithm, such as k-means clustering, to determine the cluster that the feature vector most belongs to. The determined cluster will correspond with the user's identify. Thus, a user identification 222 may be made by the system.
As is apparent from the disclosure, the various methods of identification all rely on at least one matching, learning or classification method, such as support vector machines, k-means clustering, hidden Markov models, etc. Thus, it has been assumed that the various data types in user identification database 62 are actually present in said database. The data sets, such as grab/hold data, trajectory data, first touch data, arcuate trajectory data, and heartbeat data, may be collected using unsupervised or supervised learning techniques.
A first method of collecting the various data sets is by implementing a training session. Each user may register with the system, e.g. the host device. The registering user will be asked to repeatedly perform various tasks such as grabbing the remote control, picking up the remote control, or drawing a circle on the touch pad of the remote control. The collected training data are used to define a user's tendencies for purposes of identification. When the system is in an operational mode, the input data, used for identification, may be added to the training data upon each successful identification. Furthermore, the user can verify a correct user identification and correct an improper identification to increase the robustness of the system.
A second method of collecting the various data sets is by implementing an unsupervised learning process that differentiates the users over the course of the remote control's usage. Take for example a family that owns a Digital Video Recorder (DVR). The father has large hands and grabs the remote control using three fingers. The wife has a small hands and grabs the remote with four fingers. The child has small hands and the remote with three fingers. Furthermore, the father records sports-related programming and reality television. The mother records sitcoms and police dramas. The child records cartoons and animal shows. Over the course of an initial period, the system will differentiate the three users based on the differences of hand size and hold patterns. Over the initial period, the identification system will also learn that the user with large hands and a three finger grab is associated with the sports and reality television programming. The system can then map the user preferences or profile to the extrapolated hold pattern data. Thus, a user may grab the remote control and have his or her preferences readily set based only on past usage and the initial picking up of the remote. Using this method, the user will never actually engage in training the system, but user identification will be realized over the course of time.
For example only,
While reference has been made to a remote control 12, it should be appreciated that the sensors described above, as well as the identification methods described above will be used in various handheld devices such as cell phones, portable phones, mp3 players, personal DVD players, PDAs, and computer mice. For example, with a cell phone or portable phone, using any of the above techniques, the phone may determine that a first user or a plurality of users is using the phone. Based on this, specific settings such as a phonebook, saved text messages, saved emails, volume settings, screen settings, wall paper and saved files such as photos will become available to the user. Similarly, in a device such as an MP3 player, the first user's music library may be accessible to the user after identification. In the PDA, schedules and contacts personal to the first user are made available to the user, only after the user grabs the PDA and is identified.
With a computer mouse or a laptop mouse pad, the methods disclosed above may be used to identify and authenticate the user. The user may then be automatically logged onto her user profile. Further, the user may leave the computer and upon another user touching the mouse or mouse pad, the device will be able to determine that the user has changed. At this point, the second user may be locked out of the first user's profile until an explicit override instruction is provided by the first user.
It should be apparent that the disclosed methods and devices will allow the sharing of devices once thought to be personal devices without having to risk the privacy or intimacy typically associated with these devices.
As used herein, the term module may refer to, be part of, or include an Application Specific Integrated Circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and/or memory (shared, dedicated, or group) that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality. It is should be understood that when describing a software or firmware program, the term module may refer to machine readable instructions residing on an electronic memory.
The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the invention, and all such modifications are intended to be included within the scope of the invention. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order discussed or illustrated, unless specifically identified as an order of performance. It is also to be understood that additional or alternative steps may be employed.
This application claims the benefit of U.S. Provisional Application No. 61/046,578, filed on Apr. 21, 2008, the entire disclosure of which is incorporated herein by reference.
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/US09/41227 | 4/21/2009 | WO | 00 | 10/20/2010 |
Number | Date | Country | |
---|---|---|---|
61046578 | Apr 2008 | US |