1. Field of the Invention
This invention relates generally to interacting with electronic devices via a touch-sensitive surface.
2. Description of the Related Art
Many touch pads and touch screens today are able to support a small set of gestures. For example, one finger is typically used to manipulate a cursor or to scroll the display. Another example is using two fingers in a pinching manner to zoom in and out of content, such as a photograph or map. However, this is a gross simplification of what fingers and hands are capable of doing. Fingers are diverse appendages, both in their motor capabilities and their anatomical composition. Furthermore, fingers and hands can also be used to manipulate tools, in addition to making gestures themselves.
Thus, there is a need for better utilization of the capabilities of fingers and hands to control interactions with electronic devices.
The present invention allows users to interact with touch-sensitive surfaces in a manner that distinguishes different touch types. For example, the same touch events performed by a finger pad, a finger nail, a knuckle or different types of instruments may result in the execution of different actions on the electronic device.
In one approach, a user uses his finger(s) or an instrument to interact with an electronic device via a touch-sensitive surface, such as a touch pad or a touch screen based on capacitive sensing (e.g., projective capacitive, mutual capacitance). The electronic device based on capacitive sensing often uses an arrangement of electrodes to sense capacitance at multiple locations on the touch-sensitive surface. The information from these electrodes form a two dimensional representation (a “capacitive image”) that includes capacitance measurements taken at each location of the electrode arrangement. The capacitive image is typically computed inside a touch sensor of the electronic device, but other chips or locations associated with the electronic device for generating the capacitive image are possible.
The capacitive image can be used to determine the location of touch inputs (potentially, multiple touch inputs, e.g., “multi-touch”). The capacitive image can also be used to derive a series of features. These features can then be passed to a classification engine which uses these features to determine touch types. For example, the classification engine determines whether a fingertip, knuckle, fingernail, stylus, eraser or other instrument has been used to touch the surface of the electronic device.
In an alternative approach, other touch data are also passed to the classification engine along with the features derived from the capacitive image. For example, this other touch data can include position and touch size detected by other sensors of the electronic device. Features can further be extracted from the touch data and used to determine touch type in combination with the features derived from the capacitive image. For example, these features extracted based upon touch data can be from acoustic sensors, vibro-acoustic sensors, accelerometers, gyroscopes, microphones, magnetometers, barometers, etc. In one exemplary embodiment, these features as well as the features from the capacitive image are processed by multiple classifiers to generate multiple results and the multiple results are combined in a voting manner to determine the type of the touch event.
In another aspect, the touch type can be reported to applications running on a computing platform of the electronic device, which can make use of the touch type information. Accordingly, an action is taken on the electronic device in response to the touch type. That is, different types of touches can result in execution of different actions.
Other aspects of the invention include methods, devices, systems, components and applications related to the approaches described above.
The invention has other advantages and features which will be more readily apparent from the following detailed description of the invention and the appended claims, when taken in conjunction with the accompanying drawings, in which:
The figures depict embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.
The figures and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.
In a common architecture, the data storage 106 includes a machine-readable medium which stores the main body of instructions 124 (e.g., software). The instructions 124 may also reside, completely or at least partially, within the memory 104 or within the processor 102 (e.g., within a processor's cache memory) during execution. The memory 104 and the processor 102 also constitute machine-readable media.
In this example, the different components communicate using a common bus, although other communication mechanisms could be used. As one example, the processor 102 could act as a hub with direct access or control over each of the other components.
The device 100 may be a server computer, a client computer, a personal computer (PC), or any device capable of executing instructions 124 (sequential or otherwise) that specify actions to be taken by that device. Further, while only a single device is illustrated, the term “device” shall also be taken to include any collection of devices that individually or jointly execute instructions 124 to perform any one or more of the methodologies discussed herein. The same is true for each of the individual components. For example, the processor 102 may be a multicore processor, or multiple processors working in a coordinated fashion. It may also be or include a central processing unit (CPU), a graphics processing unit (GPU), a network processing unit (NPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), or combinations of the foregoing. The memory 104 and data storage 106 may be dedicated to individual processors, shared by many processors, or a single processor may be served by many memories and data storage.
As one example, the device 100 could be a self-contained mobile device, such as a cell phone or tablet computer with a touch screen. In that case, the touch screen serves as both the touch-sensitive surface 110 and the display 120. As another example, the device 100 could be implemented in a distributed fashion over a network. The processor 102 could be part of a cloud-based offering (e.g., renting processor time from a cloud offering), the data storage 106 could be network attached storage or other distributed or shared data storage, and the memory 104 could similarly be distributed or shared. The touch-sensitive surface 110 and display 120 could be user I/O devices to allow the user to interact with the different networked components.
Returning to
In the case that the touch-sensitive surface 110 is based on capacitive sensing (e.g., projected capacitive, mutual capacitance), the touch sensor 112A may often use an arrangement of electrodes to sense capacitance at multiple locations on the touch-sensitive surface 110. For example, the touch-sensitive surface 110 may include a grid of horizontal and vertical electrodes (“lines” or “electrode lines”). The touch sensor 112A measures the capacitance at each intersection of the “lines.” In this way, the touch sensor 112A creates a two-dimensional (2D) capacitance profile, which is referred to as a “capacitive image.” Accordingly, a capacitive image includes capacitive signals corresponding to capacitances measured at a plurality of locations on the touch-sensitive surface 110 (e.g., intersections of lines). In the above example, the capacitive image is a 2D grid of capacitance values. The capacitive image includes image points corresponding to the intersections of the horizontal and vertical electrodes lines, and values at the image points corresponding to the capacitances measured at the intersections. The underlying grid is not required to be rectilinear. Any arrangement of electrodes in which the capacitance is sampled at various locations on the touch-sensitive surface 110 can give rise to a 2D capacitive image. For example, it is possible that the touch sensor 112A extracts an image from a sensing arrangement using lines running diagonally across the surface 110 (e.g., at a 45 degree angle to the edges of the surface 110).
The capacitances measured at locations of the touch-sensitive surface 110 vary in response to a touch event (e.g., a physical touch) on the touch-sensitive surface 110. Further, a touch event may correspond to a plurality of locations of the touch-sensitive surface 110 and thus affect a plurality of capacitances measured at the plurality of locations. For example, capacitances sampled at locations where a physical touch occurs may be different from capacitances sampled at other locations where no physical touch occurs. Moreover, in response to different types of touch event occurring on the touch-sensitive surface 110, the capacitances measured may be different. The generated capacitive image, including the capacitances measured at multiple locations of the touch-sensitive surface 110, can be used to analyze and classify touch events.
The touch sensor circuitry 112A may also produce a touch event trigger, which indicates the occurrence of a touch event. Touch event triggers could appear in different forms. For example, the touch event trigger might be an interrupt from a processor controlling the touch sensing system. Alternately, the touch event trigger could be a change in a polled status of the touchscreen controller. It could also be implemented as a modification of a device file (e.g., “/dev/input/event6”) on the file system, or as a message posted to a driver work queue. As a final example, the touch event trigger could be implemented as an onTouchDown( ) event in a graphical user interface program.
The sensor circuitry 112 can optionally include vibro-acoustic sensor 112B or other types of sensors, as depicted using dashed line in
Whatever the underlying principle of operation, touches on the touch-sensitive surface 110 will result in signals—both 2D capacitive image signals and possibly also vibro-acoustic or other signals. However, these raw signals typically are not directly useable in a digital computing environment. For example, the signals may be analog in nature. The sensor circuitry 112A-B typically provides an intermediate stage to process and/or condition these signals so that they are suitable for use in a digital computing environment. As shown in
Touch types can be defined according to different criteria. For example, different touch types can be defined depending on the number of contacts. A “uni-touch” occurs when the touch event is defined by interaction with a single portion of a single finger (or instrument), although the interaction could occur over time. Examples of uni-touch include a simple touch (e.g., a single tap), touch-and-drag, and double-touch (e.g., a double-tap—two taps in quick succession). In multi-touch, the touch event is defined by combinations of different fingers or finger parts. For example, a touch event where two fingers are simultaneously touching is a multi-touch. Another example would be when different parts of the same finger are used, either simultaneously or over time.
Touch types can also be classified according to which part of the finger or instrument touches. For example, touch by the finger pad, finger nail or knuckle could be considered different touch types. The finger pad is the fleshy part around the tip of the finger. It includes both the fleshy tip and the fleshy region from the tip to the first joint. The knuckle refers to any of the finger joints. The term “finger” is also meant to include the thumb. It should be understood that the finger itself is not required to be used for touching; similar touches may be produced in other ways. For example, the “finger pad” touch type is really a class of touch events that have similar characteristics as those produced by a finger pad touching the touch-sensitive surface, but the actual touching object may be a man-made instrument or a gloved hand or covered finger, so long as the touching characteristics are similar enough to a finger pad so as to fall within the class.
The touch type is determined in part by a classification of extracted features from the capacitive image (hereinafter, referred to as “capacitive features”) in response to the touch event. The capacitive features can be used alone or in combination with features derived from other sensor data to determine the touch type. The other sensor data can include data from acoustic sensors, vibro-acoustic sensors, accelerometers, gyroscopes, microphones, magnetometers, barometers, etc. For example, the touch type is determined in part by a classification of features from the capacitive image as well as features from vibro-acoustic signals from the touch event. When an object strikes a certain material, vibro-acoustic waves propagate outward through the material or along the surface of the material. As such, when respective finger parts touch or contact the surface of the touch-sensitive surface 110, vibro-acoustic responses are produced. The vibro-acoustic characteristics of the respective finger parts are unique, mirroring their unique anatomical compositions.
For example,
The capacitive image data processing module 252 can also process the capacitive image data to extract locations corresponding to the physical touches on the touch-sensitive surface 110. For example, the capacitive image data processing module 252 extracts a region of the capacitive image data representing a physical touch on the touch-sensitive surface 110. The region includes capacitive image data for the location where the physical touch occurs. Such processing may be typically performed by the touch sensor 112A in the touch circuitry 112. However, it can be recognized that other component of the device 100 can also perform such or similar processing to extract the region of a physical touch. After a specific region of the capacitive image data corresponding to a single physical touch is identified, the capacitive image data processing module 252 may crop or isolate the specific region from regions representing other physical touches. As one example, the touch sensor 112A reports a touch position (e.g., a touch ellipse) based on touch data and the processing module 252 extracts a region of the capacitive image data corresponding to the touch ellipse. As another example, the processing module 252 uses a flood-filling algorithm to extract significant values near an initial seed point (e.g., gathered from peak analysis or from a touch position reported by the touch sensor 112A).
The capacitive image data processing module 252 can also perform transform on the capacitive image data to reveal additional informational content or to facilitate further processing of the image data. For example, the capacitive image data processing module 252 applies log transform on the capacitive image data to increase contrast and improve algorithmic sensitivity to small differences in the capacitive image. Through various transformations, the capacitive image data processing module 252 can obtain a number of derivative images from the capacitive image data, as shown in
The capacitive image data processing module 252 can also isolate the region of the capacitive image data in each derived image using the aforementioned region extraction procedure. For sake of conciseness and convenience, the derivation of the capacitive image in which the region corresponding to the physical touch has been isolated is referred to hereinafter, as “derivative touch image data.”
The conversion module 254 is optional as depicted using dashed line in
The feature extraction module 256 then generates various features from the processed capacitive image data as well as possibly other sensor data, including vibro-acoustic signals and other touch data. In one embodiment, the feature extraction module 256 computes a number of features based on the derivative touch image data. As one example, the feature extraction module 256 computes covariance matrix of the image points in the derivative touch image data. The values of this covariance matrix are obtained as features. In addition, the ratio of each element of the covariance matrix to the total variance, and eigenvalues of covariance matrix can be obtained as features. As another example, the feature extraction module 256 extracts various-sized neighborhoods around the touch contact origin of the physical touch from the derivative touch image data. The various-sized neighborhoods can be regions including various numbers of points in the derivative touch image data. The sizes of the neighborhoods can include, but not limited to, for example, 5×5 points, 7×7 points, 9×9 points, 11×11 points, etc. The feature extraction module 256 then computes statistical features over these neighborhoods. The statistical features can include, but not limited to, mean, range, standard deviation, dispersion, skewness, root mean square (RMS), etc. Additionally, a vector of individual values in the derivative touch image data may also be used as a set of features. The individual values in the vector can be sorted by a comparison function. The total sum of individual values in the derivative touch image data can also be used as a feature.
Furthermore, the feature extraction module 256 may fit a multivariate Gaussian function over the distribution of the values in the derivative touch image data. This can be done through an expectation-maximization algorithm or by simple sample-weighting estimates. The feature extraction module 256 then computes features associated with the Gaussian function. These features can include magnitude (weight) of the Gaussian function relative to the mathematical normalized Gaussian function, value of the Gaussian function at the touch contact origin, and statistics on the covariance matrix of the Gaussian function.
The number of nonzero values in the derivative image data can be used as a simple estimation of the touch contact area. A similar value is commonly reported as “pressure” in many touch sensors. The sum of these nonzero values can be used as a feature. The feature extraction module 256 also compute statistical features, such as those previously computed from extracted neighborhoods, over these nonzero values.
The feature extraction module 256 can also compute shape features based on the derivative touch image data. For example, the feature extraction module 256 extracts a boundary around the touch region using contour analysis and then computes features based on the boundary. These features can include, but not limited to, average contour turning angle, contour perimeter, contour area, etc.
Optionally, the feature extraction module 256 can also extract features from other sensor data. For example, these features can include time domain and/or frequency domain representations of the vibro-acoustic signal (or its filtered versions), as well as first, second, and higher order derivatives thereof. These features can also include down-sampling the time and frequency domain data into additional vectors (e.g., buckets of ten), providing different aliasing. Additional vibro-acoustic features include linear prediction-based cepstral coefficients (LPCC), perceptual linear prediction (PLP) cepstral coefficients, cepstrum coefficients, mel-frequency cepstral coefficients (MFCC), and frequency phases (e.g., as generated by an FFT).
The feature extraction module 256 can also generate features from the touch data. Examples include location of the touch (2D, or 3D in the case of curved glass or other non-planar geometry), size and shape of the touch (some touch technologies provide an ellipse of the touch with major and minor axes, eccentricity, and/or ratio of major and minor axes), orientation of the touch, surface area of the touch (e.g., in squared mm or pixels), number of touches, pressure of the touch (available on some touch systems), and shear of the touch. For example, the touch ellipse is also used by other module (such as the processing module 252) to process the capacitive image data as described above with reference to the processing module 252. Another possible feature is an image of the hand pose (as imaged by e.g., an optical sensor, diffuse illuminated surface with camera, near-range capacitive sensing).
The classification module 258 classifies the touch event using extracted features from the capacitive image data as well as possibly other non-capacitive sensor features, including vibro-acoustic features and touch data features. Additionally, conventional touch information is also passed to the classification module 258. For example, many touch sensors provide touch position in X/Y coordinate, major and minor axes, size, pressure and many other attributes regarding a physical touch. These features can also be classified by the classification module 258 to determine a touch type of the touch event.
The classification module 258 can use a large number of approaches, including, but not limited to, basic heuristics, decision trees, support vector machine, random forest, naïve Bayes, elastic matching, template matching, k-means clustering, k-nearest neighbors algorithm, neural network, multilayer perception, multinomial logistic regression, Gaussian mixture models, AdaBoost, logistic boosting (LogitBoost), etc. In addition, the classification module 258 can also combine results from several different classifiers through, for example, a voting scheme. Moreover, the classification module 258 can use different classifiers based on different features. For example, two classifiers can be employed, one for classifying touch events with small contact areas, and another for classifying touch events with large contact areas. To aid classification, the user can provide supplemental training samples to the classifiers.
In one exemplary embodiment, to reduce computation time both for creating the classifier and evaluating the classifier, the classification module 258 can optionally apply an attribute selection process to reduce the set of features used in classification to only those with the highest discriminatory power. The attribute selection process can be based on a variety of approaches, including, for example, forward subset evaluation, information gain, support vector machine (SVM) weight analysis, etc.
Returning to
This approach allows the same touch event to control more than one action. This can be desirable for various reasons. First, it increases the number of available actions for a given set of touch events. For example, if touch types are not distinguished, then a single tap can be used for only one purpose, because a single tap by a finger pad, a single tap by a finger nail and a single tap by an instrument cannot be distinguished. However, if all three of these touch types can be distinguished, then a single tap can be used for three different purposes, depending on the touch type.
Conversely, for a given number of actions, this approach can reduce the number of user inputs needed to reach that action. Continuing, the above example, if three actions are desired, by distinguishing touch types, the user will be able to initiate the action by a single motion—a single tap. If touch types are not distinguished, then more complex motions or a deeper interface decision tree may be required. For example, without different touch types, the user might be required to first make a single tap to bring up a menu of the three choices. He would then make a second touch to choose from the menu.
Although the detailed description contains many specifics, these should not be construed as limiting the scope of the invention but merely as illustrating different examples and aspects of the invention. It should be appreciated that the scope of the invention includes other embodiments not discussed in detail above. Various other modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus of the present invention disclosed herein without departing from the spirit and scope of the invention as defined in the appended claims. Therefore, the scope of the invention should be determined by the appended claims and their legal equivalents.
The term “module” is not meant to be limited to a specific physical form. Depending on the specific application, modules can be implemented as hardware, firmware, software, and/or combinations of these. Furthermore, different modules can share common components or even be implemented by the same components. There may or may not be a clear boundary between different modules.
Depending on the form of the modules, the “coupling” between modules may also take different forms. Dedicated circuitry can be coupled to each other by hardwiring or by accessing a common register or memory location, for example. Software “coupling” can occur by any number of ways to pass information between software components (or between software and hardware, if that is the case). The term “coupling” is meant to include all of these and is not meant to be limited to a hardwired permanent connection between two components. In addition, there may be intervening elements. For example, when two elements are described as being coupled to each other, this does not imply that the elements are directly coupled to each other nor does it preclude the use of other elements between the two.