The present disclosure relates to a multi-touch surface providing detection and tracking of multiple touch points.
Human machine interactions for consumer electronic devices are gravitating towards more intuitive methods based on touch and gestures and away from the existing mouse and keyboard approach. For many applications touch sensitive surface is used for users to interact with underlying systems. Same touch surface can also be used as display for many applications. Consumer electronics displays are getting thinner and less expensive. Hence there is need for a touch surface that is thin and inexpensive and provides multi-touch experience.
In order to provide multi-touch interaction on a surface, several different sensors, such as IR sensors, camera sensors and pressure sensors, have been sued. These sensors can be expensive, complex and take more space resulting in thicker display and bulkier end products. Capacitive sensors provide a cheaper and thinner alternative. Two-dimensional capacitive sensors have been used for multi-touch application having smaller surface area. Employing capacitive sensors for multi-touch application having large size surface, however, can be difficult due to increased need for information processing. The complexity of two-dimensional capacitive sensors grows exponentially as size of the surface area increases. Along with complexity, costs for producing for the two-dimensional capacitive sensors also increase.
The present disclosure provides two independent arrays of orthogonal linear capacitive sensors. One or more embodiments of the present disclosure can provide a simpler and less expensive alternative to two-dimensional capacitive sensors for multi-touch applications with larger surfaces. One or more embodiments of the present disclosure can be packaged in a very thin foil at lower costs than using other sensors for multi-touch solutions. One or more embodiments of the present disclosure aim to accurately detect and track multiple touch points.
The inventors of the present disclosure propose an apparatus detecting at least one touch point. The apparatus has a surface having a first dimension and second dimension. A first plurality of sensors are deployed along the first dimension and generating a first plurality of sensed signals caused by the at least one touch point. The first plurality of sensors provide a first dataset indicating the first plurality of sensed signals as a first function of position on the first dimension. A second plurality of sensors are deployed along the second dimension and generating a second plurality of sensed signals caused by the at least one touch point. The second plurality of sensors provide a second dataset indicating the second plurality of sensed signals as a second function of position on the second dimension. The first plurality of sensors and the second plurality of sensors operate independently to each other. A trained-model based processing unit processes the first and second datasets to determine a position for each of the at least one touch point.
The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.
Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.
Example embodiments are provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.
The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms “a”, “an” and “the” may be intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises,” “comprising,” “including,” and “having,” are inclusive and therefore specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order discussed or illustrated, unless specifically identified as an order of performance. It is also to be understood that additional or alternative steps may be employed.
When an element or layer is referred to as being “on”, “engaged to”, “connected to” or “coupled to” another element or layer, it may be directly on, engaged, connected or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly engaged to”, “directly connected to” or “directly coupled to” another element or layer, there may be no intervening elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between” versus “directly between,” “adjacent” versus “directly adjacent,” etc.). As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.
Although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another region, layer or section. Terms such as “first,” “second,” and other numerical terms when used herein do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.
Spatially relative terms, such as “inner,” “outer,” “beneath”, “below”, “lower”, “above”, “upper” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. Spatially relative terms may be intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the example term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
Referring to
The capacitive sensor 14 is sensitive to conductive objects like human body parts when the objects are near the surface of the interactive foil 12. The capacitive sensors 13 read sensed capacitance values on the x-axis and y-axis independently. When an object, e.g. a finger, comes close enough to the surface, the capacitance values on the corresponding x-axis and y-axis increase. The values at the x-axis and y-axis thus make possible the detection of a single or multiple touch points on the interactive foil 12. Giving a specific example, the foil 12 can be 32 inches long diagonally, and the ratio of the long and short sides can be 16:9. Therefore, the corresponding sensor distance in the x-axisaxis is about 22.69 mm and that in the y-axis is about 13.16 mm.
Referring now to
Before discussing detecting multiple touch points from the detection readings of the two independent arrays of capacitive sensors 13, it is helpful to briefly discuss detection of a single touch point on the interactive foil 12. Referring now to
Referring now to
Referring now to
Considering a situation for detecting two touch points, because the background noise may also be modeled as a Gaussian, a sum of three Gaussian functions will be fitted. Two of the three component Gaussians can be identified as correlating to the tow touch points to be detected. The third one having a very small peak value comparing to the other two can be rejected as noise.
A discussion regarding detecting and searching one or more peak points from the capacitance detection readings of the capacitive sensors 12 has been presented. However, before one or more embodiments of the present disclosure can utilize the above techniques to search for peak points from the capacitance detection readings, the number of touch points for which the capacitance detection readings are generated needs to be known. The reason is simple: if only one touch point is on the interactive foil, only one peak point needs to be searched; if two or more touch points are on the interactive foil, two or more peak points need to be searched.
Referring now to
The capacitance detection readings from the capacitive sensors 13 are first passed to the touch point classifier 61, which was trained off-line to classify between single touch point and two touch points. The classification results are then fed into a Hidden Markov Model 62 to update the posterior probability. Once the posterior probability reaches a predetermined threshold, the corresponding number of the touch points is confirmed and a peak detector 63 searches the readings to find the local maxima. A Kalman tracker 64 is then used to track the movement of the touch points.
Referring now to
One goal of one or more embodiments of the present disclosure is to analyze the capacitance detection readings and determine if the reading is from a single touch point or two touch points. The inventors of the present disclosure propose using a computational mechanism to analyze the capacitance detection readings and for example statistically determine if the capacitance detection readings are from a single touch point or two touch points. The computational mechanism can be a trained-model based mechanism. The inventors of the present disclosure further propose employing a classifier for this analysis.
A classifier can be defined as a function that maps an unlabeled instance to a label identifying a class according to an internal data structure. For example, the classifier can be used to label the capacitance detection readings as single touch point or two touch points. The classifier extract significant features from the information received (the capacitance detection readings in this example) and labels the information received based on those features. These features can be chosen in such way that clear classes of the information received can be identified.
A classifier needs to be trained by using training data in order to accurately label later received information. During training, underlying probabilistic density functions of the sample data are being estimated.
Referring now to
The inventors of the present disclosure further propose using a Gaussian density classifier. During training, for example, point clouds received from the two-dimensional capacitive sensors is to be labeled by the Gaussian density classifier as one of two classes: one-touch-point class and two-touch-point class. In a Gaussian density classifier, a probabilistic density function of received data (e.g., point clouds) with respect to the different classes is modeled as a linear combination of multivariate Gaussian probabilistic density functions.
Suppose samples of each group are from a multivariate Gaussian density N(μk, Σk), k=1, 2. Let xik ε Rd be the i-th sample point for the k-th group, i=1, . . . , Nk. For each group, the Maximum Likelihood (ML) estimation of the mean μk and covariance matrix σk is
With this estimation, the boundary is then defined as the equal Probabilistic Density Function (PDF) curve, and is given by xTQx+Lx+K=0, where Q=Σ1−1−Σ2−1, L=−2(μ1Σ1−1−μ2Σ2−1), K=μ1TΣ1−1μ1−μ2TΣ2−1μ2−log|Σ1|+log|Σ2|.
The present disclosure now describes determining features from the capacitance detection readings need to be extracted for the Gaussian density classifier in one or more embodiments of the present disclosure. The inventors of the present disclosure propose to use statistics of the capacitance detection readings, such as mean, the standard deviation and the normalized higher order central moments, as features to be extracted. Note that the statistics of the reading may be stable even though the position of the peak and the value of the each individual sensor may vary. Features are then selected as the statistics of the capacitance detection readings on each axis. The inventors of the present disclosure then propose to determine a suitable set of and/or number of features by using K-fold cross validation on a training dataset with features up to the 8th normalized central moment.
Generally speaking, in a K-fold cross validation, a training dataset is randomly split into K mutually exclusive subsets of approximately equal size. Of the K subsets, a single subset is retained as the validation data for testing the model, and the remaining K−1 subsets are used as training data. The cross-validation process is then repeated K times (the folds), with each of the K subsets used exactly once as the validation data. The K results from the folds then can be averaged (or otherwise combined) to produce a single estimation.
In one or more embodiments of the present disclosure, K-fold cross validation is employed to train and validate the Gaussian density classifier. The estimated false positive and false negative rates are shown in
Thus, one or more embodiments of the present disclosure can extract mean, standard deviation and skewness of capacitance detection readings received from the capacitive sensors at a given time t. The Gaussian density classifier then determines whether the capacitance detection readings received is from a single touch point or from two touch points based on the extracted features.
Further, the inventors of the present disclosure recognize that results from the Gaussian density classifier (i.e., a single touch point or two touch points) can be connected over time to smooth the detection over time in a probabilistic sense and to confirm the results determined by the Gaussian density classifier. In one or more embodiments of the present disclosure, a confirmation module receives current result signals from the touch point classifier 61 and determines a probability of occurrence of the current result (i.e., either a single touch point or two touch points) based on result signals previously received. If the probability reaches a predetermined threshold, then the current result from the touch point classifier 61 is confirmed. The inventors of the present disclosure further propose to employ a Hidden Markov Model in the confirmation module.
Referring now to
The sequence of observations may be modeled as a probabilistic function of an underlying Markov chain having state transitions that are not directly observable. For example, the HMM can have two hidden states. At the given time t, the hidden states can be represented as Z1 ε{S1, S2}, wherein S1 and S2 represent two states: a single-touch-point state and a two-touch-point state respectively. Because only a scenario having one or two touch points is considered now, two hidden states are adapted for the HMM. In a scenario where more than two touch points need to be detected, more than two hidden states can be adapted for the HMM.
The probability of transition from state Zt at time t to state Zt+1 at time (t+1) is represented as: P(Zt+1|Zt).
At time t, the probability of observing Xt if the HMM is at state Zt is represented at P(Xt|Zt).
The inventors of the disclosure discover that a homogeneous HMM can be applied to one or more embodiments of the disclosure. In a homogeneous HMM, the possibilities of transition at two consecutive time points are the same: P(Zt
P(Xt+δ|Zt+67 )=P(Xt|Zt), ∀δε Z+.
At time 0 the possibility of transition is assumed to be P(Z0)=0.5 based on Bernoulli distribution. At a given time t, suppose the probability of state P(Zt−1) at time (t−1) is known and observation Xt is received from the touch point classifier 61, the hidden state is then updated by the Bayesian rule as
The inventors of the disclosure discover that decisions can be made based on the posterior probability P(Zt|Xt, Zt−1) instead of based on maximizing the joint likelihood to find the best sequence of the state transitions. A threshold can be predefined to verify the observations from the touch point classifier. If the calculated posterior probability P(Zt|Xt, Zt−1) is higher than the predefined threshold, the state at time t is confirmed by the posterior probability P(Zt|Xt, Zt−1). A high threshold can be set to obtain higher accuracy.
The result from the touch point classifier 61 is now confirmed by the confirmation module. In other words, the capacitance detection readings from the capacitive sensors are analyzed by the touch point classifier and confirmed to be either from a single touch point or two touch points in this example.
Referring now again back to
To achieve a subpixel accuracy, the inventors of the present disclosure propose to employ a parabola fitting process for each local maximum pair (xm,f(xm)) on each axis (i.e.: x-axis and y-axis) of the interactive foil, where xm is the position and f(xm) is the capacitance detection reading value. The local maximum pair (xm,f(xm)) together with one point on each side of the peak position, (xm−1,f(xm−1)) and (xm+1,f(xm+1)), are fit into a parabola f(x)=ax2+bx+c. This is equivalent to solving a linear system
Thus, the peak position is refined to
In one or more embodiments of the disclosure, by using the above described techniques, the peak detector 63 can determine one or two peak positions for each of x-axis and y-axis of the touch screen. In some other embodiments, more than two peak points on each axis can be similarly determined.
Because the two arrays of capacitive sensors 13 on the interactive foil 12 are independent, positions on x-axis and y-axis need to be associated together to determine the touch points in the two-dimensional plane of the interactive foil. When there are two peaks on both x-axis (x1,x2) and y-axis (y1,y2), there are two pair of possible associations (x1,y1), (x2,y2) and (x1,y2), (x2,y1), which have equal probability. The inventors of the present disclosure recognize that the two possible associations can pose ambiguity at the very beginning of detection when no other data has been collected to assist determination of the association. Thus, the inventors of the present disclosure propose to restrict the detection to start from a single touch point.
In one or more embodiments of the present disclosure, the history of detected touch points is stored in a data store of the embodiment. The data store for example can be deployed within the processing unit. A table in the data store records the x and y values for each touch point at each time point. This history data is then utilized by the tracker 19 to determine movements of the touch points. The tracker 19 based on the history data can predict and assign one or more trajectories to a touch point. Based on the determined trajectories, the tracker 19 can determine an association of current peaks on the x-axis and y-axis detected by the peak detector 63. In this way, the processing unit can more accurately determine the current position for each touch point.
The inventors of the present disclosure further propose a technique to enhance the detection results as well as to smooth the trajectory as the touch point moves. No matter what detection methods are used, it will inevitably include missed detections, both in term of false positive and false negative. Missed detections can happen either due to system or environment noise or the way a person touches the surface. For example, if a person intended to touch the surface with index finger, but the middle finger or the thumb is very close to the surface, then those fingers can be falsely detected. To enhance the detection results as well as to smooth the trajectory as the touch point moves, a tracking method is employed.
In one or more embodiments of the disclosure, the inventors of the present disclosure propose to employ a Kalman filter as the underlying model for a touch point tracker. Kalman filter provides a prediction based on previous observations and after the detection is confirmed it can also update the underlying model. Kalman filter records the speed at which the touch point moves and the prediction is made based on the previous position and the previous speed of the touch point.
Referring now to
The touch point tracker 110 can use the Kalman filter 111 as the underlying motion model to output a prediction based on previously detected touch points. Based on the prediction, a match finder 112 is deployed to search a best match in a detection dataset. Once a match is found, a new measurement is taken and the underlying model 113 is updated according the measurements.
Referring now to
In one or more embodiments of the present disclosure, to associate touch points at different time frames as well as smooth the movement, a Kalman filter with a constant speed model is employed. A state vector is defined as z=(x,y,Δx,Δy), where (x,y) are the position on the touch screen, (Δx,Δy) are the change in position between adjacent time frames, and
The transition of the Kalman filter satisfies
where in our problem,
are the transition and measurement matrix, w˜N(0,R) and v˜N(0,Q) are white Gaussian noises with covariance matrices R and Q.
Given prior information from past observations zt˜N(μt,Σ), the update once the measure is available is given by
z
t
post=μt+ΣMT(MΣMT+R)−1(
Σpost=Σ−MT(MΣMT+R)−1M
μt+1=Hztpost
Σ=HΣpostHT+Q
where ztpost is the correction when the measurement
Although for simplicity a scenario where only a single and two touch points are detected is described, the proposed systems and techniques, however, can be extended to handle more than two touch points by simply adding classes when training the classifier as well as increasing the states in the simplified Hidden Markov Model as described above. For example, in order to detect and track three points, three classes are defined in the classifier during training and three states are defined in the simplified Hidden Markov Model.
The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the invention, and all such modifications are intended to be included within the scope of the invention.