A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
The present invention relates generally to the field of touch screen technology and more particularly to the addition of an edge touch classifier to increase the accuracy of the analysis of touch screen events.
The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also be inventions.
Various electronic devices today are typically operated by a user interacting with a touch sensitive screen. This feature is particularly a characteristic of the recent generation of smart phones. Typically, the touch sensitive screens respond to finger contact to activate the display for further processes. Contact may also be made using tools such as a stylus or other parts of the hands. The fingers and other contacts made to the touch sensitive screen generally appear as an activated point or blob (i.e., region). However, when touch contacts occur on the edge of the touch screen, only a portion of the touch contact can be digitized.
The touch sensitive screen may be associated with a classification engine which is normally trained on real world touch event data from users. However, because edge contacts are rare in ordinary use, classifiers (in the classification engine) receive few edge training instances. In response, the classification accuracy of edge touch events tends to be lower. Further, because less of the touch contact is visible, there is less data to work with. This leads to several problems, most notably that classification algorithms may over-fit to the limited data. Secondly, edge contacts appear very different to full-contact, ordinary touches, leading to bi-modal (or even multi-modal) distributions of key characteristics to which some classification algorithms are ill-suited to accommodate. There is therefore a need to mitigate the potential problems associated with edge touch events that could otherwise reduce the accuracy of such classification analysis.
For some embodiments, an apparatus for improving accuracy of touch screen event analysis comprises two separate and distinct classifiers including a full touch classifier and an edge touch classifier. In order to determine which such classifier is appropriate for each touch event, the touch screen is provided with two similarly distinct sensing regions. One such sensing region is derived from the conventional touch screen and is generally the bulk of the central area of the touch screen, but somewhat reduced in size relative to the entire physical extent of the touch screen. The balance of the touch screen around the periphery, forms the second sensing region which is everywhere proximate to the screen edge. The precise shape and size of this edge touch sensing region is chosen to enable a high probability that any finger touch that might extend beyond the edge of the touch screen would be treated as an edge touch event which will instigate edge touch classifier operation instead of full touch classifier operation during the ensuing analysis of the event.
Other aspects and advantages of the present invention can be seen on review of the drawings, the detailed description and the claims, which follow.
The included drawings are for illustrative purposes and serve only to provide examples of possible structures and process steps for the disclosed techniques. These drawings in no way limit any changes in form and detail that may be made to embodiments by one skilled in the art without departing from the spirit and scope of the disclosure.
Applications of methods and apparatus according to one or more embodiments are described in this section. These examples are being provided solely to add context and aid in the understanding of the present disclosure. It will thus be apparent to one skilled in the art that the techniques described herein may be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order to avoid unnecessarily obscuring the present disclosure. Other applications are possible, such that the following examples should not be taken as definitive or limiting either in scope or setting.
In the following detailed description, references are made to the accompanying drawings, which form a part of the description and in which are shown, by way of illustration, specific embodiments. Although these embodiments are described in sufficient detail to enable one skilled in the art to practice the disclosure, it is understood that these examples are not limiting, such that other embodiments may be used and changes may be made without departing from the spirit and scope of the disclosure.
The aforementioned issues that arise from a user contacting the edges of touch screens are addressed in embodiments of the present invention by employing a classification system that treats edge contacts as a separate and distinct class of touch events. Touches occurring near the edge of the screen are processed by a classifier that is better suited to process edge contacts (e.g., it has been trained on edge contacts). On the other hand, touches that occur in the middle of the screen (and thus are wholly digitized) are processed by another classifier.
One or more embodiments may be implemented in numerous ways, including as a process, an apparatus, a system, a device, a method, a computer readable medium such as a computer readable storage medium containing computer readable instructions or computer program code, or as a computer program product comprising a computer usable medium having a computer readable program code embodied therein.
The disclosed embodiments may include apparatus for improving accuracy of touch screen event analysis and may comprise two separate and distinct classifiers. A first classifier may be a full touch classifier. A second classifier may be an edge touch classifier. In order to determine which of the first or second classifier is appropriate for a touch event, the touch screen is provided with two similarly distinct sensing regions. One such sensing region is derived from the conventional touch screen and is generally the bulk of the central area of the touch screen, but somewhat reduced in size relative to the entire physical extent of the touch screen. The balance of the touch screen around the periphery, forms the second sensing region which is everywhere proximate to the screen edge. The precise shape and size of this edge touch sensing region is chosen to enable a high probability that any finger touch that might extend beyond the edge of the touch screen would be treated as an edge touch event which will instigate edge touch classifier operation instead of full touch classifier operation during the ensuing analysis of the event.
The disclosed embodiments may include a method for improving accuracy of touch screen event analysis and may comprise detecting a touch event on a touch sensitive screen, said surface having at least two touch regions; generating a vibro-acoustic waveform signal using at least one sensor detecting such touch event; converting the waveform signal into at least one other form; extracting distinguishing features from said converted waveform signal; and classifying said features to analyze the features of the converted touch event waveform signal by employing one of at least two different classification processes depending on which of the two distinct touch regions was touched during the touch event.
The disclosed embodiments may include a machine-readable medium carrying one or more sequences of instructions for providing social information, which instructions, when executed by one or more processors, cause the one or more processors to detect a touch event on a touch sensitive screen, said surface having at least two touch regions, generate a vibro-acoustic waveform signal using at least one sensor detecting such touch event, convert the waveform signal into at least one other form, extract distinguishing features from said converted waveform signal, and classify said features to analyze the features of the converted touch event waveform signal by employing one of at least two different classification processes depending on which of the two distinct touch regions was touched during the touch event.
In general, a user may make contact with a touch sensitive screen of a touch sensitive device or computer system using a stylus or other parts of the hands such as the palm and various parts of the finger, i.e., pad, nail, knuckle, etc. Each such different type of touch mechanism produces a different type of digital signature. Moreover, each user of a touch sensitive device may have his or her own unique touch event characteristics resulting from anatomical differences such as fleshiness, finger size, finger shape, BMI and the like. These differences in touch event characteristics, whether the result of different user anatomies or different touch mechanisms, may be used advantageously to improve the touch screen technology by reducing ambiguities, distinguishing between users, responding only to intentional touch events and the like. Such advantageous uses are derived from sophisticated sensor-based analysis of the touch event coupled with one or more algorithms designed to provide further analytical characteristics otherwise hidden or not readily apparent in the data generated by the touch event.
By way of example, one such apparatus is disclosed in pending U.S. patent application Ser. No. 14/483,150 filed on Sep. 11, 2014 by the Applicant hereof and entitled “Method And Apparatus For Differentiating Touch Screen Users Based On Touch Event Analysis”. This co-pending application discloses that when a user touches a touch screen a mechanical force is applied to the screen resulting in mechanical vibrations that may be captured by a variety of sensors such as impact sensors, vibration sensors, accelerometers, strain gauges or acoustic sensors such as a microphone.
Once the vibro-acoustic signal has been captured, it can be converted into a series of features, for example: Average acoustic power, Standard Deviation, Variance, Skewness, Kurtosis, Absolute sum, Root Mean Square (RMS), Dispersion, Zero-crossings, Spectral centroid, Spectral density, Linear Prediction-based Cepstral Coefficients (LPCC), Perceptual Linear Prediction (PLP), Cepstral Coefficients Cepstrum Coefficients, Mel-Frequency Cepstral Coefficients (MFCC), Frequency phases (e.g., as generated by an FFT).
Simultaneously, many touch screen technologies are able to digitize several aspects of a touch event, such as the shape, size, capacitance, orientation, pressure, etc. The latter may be used as distinguishing features, or such features can be derived from them.
Because human fingers vary in their anatomical composition, their acoustic and touch properties can vary between humans. Moreover, the way users touch a screen can also be distinguishing (e.g., what finger, what part of the finger, how flat, how hard). Thus, the vibro-acoustic features and touch features contain properties that can be characteristic of different users and different parts of user's hands (e.g., fingertip, knuckle, and nail).
It is thus possible to provide a classifier that can run on a touch computing device that upon receipt of a touch event, makes a guess about which user is operating the device, or whether the user is authorized or has any personalized features. Alternatively, it is also possible to provide a classifier that can run on a touch computing device that upon receipt of a touch event, makes a guess about what part of the finger was used to contact the screen.
For some embodiments, the disclosed method may include the following operations and may employ the following components:
(a) a sensing system that may be configured to continuously sample vibro-acoustic data, saving it into a buffer. This buffer can be of many lengths such as, for example 50 milliseconds;
(b) a touch sensitive screen may be configured wait for a touch event to occur. Any number of touch technologies may be possible. The operations of the touch sensitive screen may be configured to operate in parallel with the sensing system;
(c) when the touch sensitive screen detects a touch event, it may be configured to trigger a conversion, feature extraction, and classification process;
(d) the data from the vibro-acoustic buffer is retrieved. Because the touch screens typically have some latency, it may be necessary to look backwards in the buffer to find the vibro-acoustic waveform that corresponds to the touch impact (e.g., if the touch screen has a 20 ms latency, it may be necessary to look back in the buffer 20 ms to find the corresponding vibro-acoustic event). All or part of the buffer may be saved and passed to the next operations;
(e) conversion operations may be performed next. The waveform from the sensor is a time-domain representation of the vibro-acoustic signal. In addition to saving the waveform, the signal is converted into other forms. This includes filtering the waveform and transforming into other forms, including frequency domain representations;
(f) feature extraction operations may be performed next, where touch screen controller data and vibro-acoustic data are analyzed to extract features that characterize different users. For the vibro-acoustic data, features are computed for all representations of the signal;
(g) these features are then passed to a classification unit, which uses the information to label the touch event with a user (in addition to whatever the touch sensitive screen reports, e.g., X/Y position, major/minor axes, pressure, etc.);
(h) the augmented touch event is then passed to the OS or end user applications, to associate a use based on the touch event.
For some embodiments, a classifier may be configured to use one or more of the following features to perform its operations: location of touch contact (2D, or 3D in the case of curved glass or other non-planar geometry), size of touch contact (some touch technologies provide an ellipse of the touch contact with major and minor axes), rotation of the touch contact, surface area of the touch contact (e.g., in squared mm or pixels), pressure of touch (available on some touch systems), shear of touch (“shear stress”, also called “tangential force” in the literature, arises from a force vector perpendicular to the surface normal of a touch screen. This is similar to normal stress—what is commonly called pressure—which arises from a force vector parallel to the surface normal.”), number of touch contacts, capacitance of touch (if using a capacitive touch screen), swept frequency capacitance of touch (if using a swept frequency6 capacitive touch screen), and swept frequency impedance of touch (if using a swept frequency capacitive touch screen). The computation phase may also compute the derivative of the above features over a short period of time, for example, touch velocity and pressure velocity. Other features that the classifier may also use include shape of touch (some touch technologies can provide the actual shape of the touch, and not just a circle or ellipse), and image of the hand pose (as imaged by e.g., an optical sensor, diffuse illuminated surface with camera, near-range capacitive sensing).
The classification engine may use any number of approaches, including but not limited to basic heuristics, decision trees, Support Vector Machine, Random Forest, Naïve bayes, elastic matching, dynamic time warping, template matching, k-means clustering, K-nearest neighbors algorithm, neural network, Multilayer perceptron, multinomial logistic regression, Gaussian mixture models, and AdaBoost. Additionally, the results from several different classifiers may be combined through, for example, a voting scheme.
For some embodiments, it may be possible to use different classifiers based on one or more features. For example, two classifiers could be employed, one for processing sensor waveforms with a high Standard Deviation, and another classifier for waveforms with low Standard Deviation.
Referring to
When an object strikes a certain material, vibro-acoustic waves propagate outward through the material or along the surface of the material. Typically, interactive surfaces use rigid materials, such as plastic or glass, which both quickly distribute and faithfully preserve the signal. As such, when one or more fingers touch or contact the surface of the touch screen 100, vibro-acoustic responses are produced. The vibro-acoustic characteristics of the respective user fingers and their respective unique anatomical characteristics produce unique responses for each user.
Referring back to
The OS 130 runs the computing system so that the function can be activated in line with the classification of the vibro-acoustic signals and the corresponding user. The vibro-acoustic classifier 120 includes a segmentation unit 122 to segment the vibro-acoustic signal into a digital representation; a conversion unit 124 to convert the digitized vibro-acoustic signal into an electrical signal; a feature extraction unit 126 to derive a series of features from the electrical signal; and two classification units 128 and 129 to classify each user using the above-described features to analyze the touch event depending upon whether it was a full screen touch event or an edge touch event as will be further described below.
The segmentation unit 122 samples the vibro-acoustic signal, for example, at a sampling rate of 96 kHz, using a sliding window of 4096 samples of the vibro-acoustic signal. The conversion unit 124 then performs, for example, a Fourier Transform on sampled time-dependent vibro-acoustic signal to produce an electrical signal having frequency domain representation. For example, the Fourier Transform of this window may produce 2048 bands of frequency power.
The vibro-acoustic classifier 120 may further down-sample this data into additional vectors (i.e., buckets of ten), providing a different aliasing. In addition, additional time-domain features may be calculated from the vibro-acoustic signal, such as the average absolute amplitude, total absolute amplitude, standard deviation of the absolute amplitude, the center of mass for both the segmented input signal and the Fourier Transformed signal, and zero crossings.
The feature extraction unit 126 may also calculate a series of features from the frequency domain representation of the vibro-acoustic signals, such as the fundamental frequency of the impact waveform. The classification units 128 and 129 classify the vibro-acoustic signal using the features to for example distinguish what user generated the touch event, so that the computing system may selectively activate a function related to the identified user depending on the classified vibro-acoustic signals.
Referring to
One way of determining whether to enable the first classifier 128 or instead to enable the second classifier 129 for each touch event, is illustrated in
To aid classification, the user can provide supplemental training samples to the vibro-acoustic classifier 120. In one exemplary embodiment, the classification units 128 and 129 may be implemented with a support vector machine (SVM) for feature classification. The SVM is a supervised learning model with associated learning algorithms that analyze data and recognize patterns, used for classification and regression analysis.
Thus it will be understood that what has been disclosed herein are an apparatus and method for differentiating between full touch events and edge touch events in a touch sensitive device. At least one sensor may be employed to detect and capture a waveform signal or acoustical or mechanical effect resulting from a touch event. The waveform signal is converted, and distinguishing features are extracted for use in two distinct classification units to associate such features with the user. A full touch event classifier is trained for such full events, while an edge touch event classifier is trained for analyzing only edge touch events. The appropriate classifier is enabled by a touch screen having two distinct regions, full and edge and in response to the location of the touch event relative to these two regions.
These and other aspects of the disclosure may be implemented by various types of hardware, software, firmware, etc. For example, some features of the disclosure may be implemented, at least in part, by machine-readable media that include program instructions, state information, etc., for performing various operations described herein. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher-level code that may be executed by the computer using an interpreter. Examples of machine-readable media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (“ROM”) and random access memory (“RAM”).
Any of the above embodiments may be used alone or together with one another in any combination. Although various embodiments may have been motivated by various deficiencies with the prior art, which may be discussed or alluded to in one or more places in the specification, the embodiments do not necessarily address any of these deficiencies. In other words, different embodiments may address different deficiencies that may be discussed in the specification. Some embodiments may only partially address some deficiencies or just one deficiency that may be discussed in the specification, and some embodiments may not address any of these deficiencies.
While various embodiments have been described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present application should not be limited by any of the embodiments described herein, but should be defined only in accordance with the following and later-submitted claims and their equivalents.
Number | Name | Date | Kind |
---|---|---|---|
9013452 | Harrison et al. | Apr 2015 | B2 |
9019244 | Harrison | Apr 2015 | B2 |
9329688 | Harrison | May 2016 | B2 |
9329715 | Schwarz et al. | May 2016 | B2 |
9612689 | Harrison et al. | Apr 2017 | B2 |
20050083313 | Hardie-Bick et al. | Apr 2005 | A1 |
20080319932 | Yih | Dec 2008 | A1 |
20090174679 | Westerman | Jul 2009 | A1 |
20110018825 | Kondo et al. | Jan 2011 | A1 |
20110248948 | Griffin | Oct 2011 | A1 |
20120158629 | Hinckley | Jun 2012 | A1 |
20130234982 | Kang | Sep 2013 | A1 |
20140071095 | Godsill | Mar 2014 | A1 |
20140210788 | Harrison et al. | Jul 2014 | A1 |
20140240295 | Harrison | Aug 2014 | A1 |
20140289659 | Harrison et al. | Sep 2014 | A1 |
20140300559 | Tanimoto | Oct 2014 | A1 |
20140327626 | Harrison et al. | Nov 2014 | A1 |
20140368436 | Abzarian | Dec 2014 | A1 |
20150002405 | Kuan | Jan 2015 | A1 |
20150035759 | Harrison et al. | Feb 2015 | A1 |
20150242009 | Xiao et al. | Aug 2015 | A1 |
20160012348 | Johnson | Jan 2016 | A1 |
20160062545 | Lai | Mar 2016 | A1 |
20160077615 | Schwarz et al. | Mar 2016 | A1 |
20160077650 | Durojaiye | Mar 2016 | A1 |
20160077664 | Harrison et al. | Mar 2016 | A1 |
20160085324 | Schwarz et al. | Mar 2016 | A1 |
20160085333 | Christopher | Mar 2016 | A1 |
20160098185 | Xiao et al. | Apr 2016 | A1 |
20160224145 | Harrison et al. | Aug 2016 | A1 |
20170024892 | Harrison et al. | Jan 2017 | A1 |
20170060279 | Harrison | Mar 2017 | A1 |
Number | Date | Country |
---|---|---|
2004-213312 | Jul 2004 | JP |
10-2002-0075283 | Oct 2002 | KR |
2006-070044 | Jul 2006 | WO |
WO2013059488 | Apr 2013 | WO |
Entry |
---|
Non-Final Office Action—dated Jun. 24, 2014—U.S. Appl. No. 13/849,698, filed Mar. 23, 2013, titled: “Method and System for Activating Different Interactive Functions Using Different Types of Finger Contacts.” |
Non-Final Office Action—dated Oct. 16, 2014—U.S. Appl. No. 13/780,494, filed Feb. 28, 2013, titled: “Input Tools Having Viobro-Acoustically Distinct Regions and Computing Device for Use With the Same.” |
U.S. Appl. No. 13/958,427, filed Aug. 2, 2013, titled: “Capture of Vibro-Acoustic Data Used to Touch Types.” |
U.S. Appl. No. 14/191,329, filed Feb. 26, 2014, titled: “Using Capacitive Images for Touch Type Classification.” |
U.S. Appl. No. 13/887,711, filed May 6, 2013, titled: “Using Finger Touch Types to Interact with Electronic Devices.” |
U.S. Appl. No. 14/483,150, filed Sep. 11, 2014, titled: “Method and Apparatus for Differentiating Touch Screen Users Based on Touch Event Analysis.” |
U.S. Appl. No. 14/495,041, filed Sep. 24, 2014, titled: “Method for Improving Accuracy of Touch Screen Event Analysis by Use of Spatiotemporal Touch Patterns.” |
U.S. Appl. No. 14/242,127, filed Apr. 1, 2014, titled: “Method and Apparatus for Classifying Touch Events on a Touch Sensitive Surface.” |
International Search Report and Written Opinion received for PCT Application No. PCT/US2014/049485 dated Nov. 17, 2014, 13 pages. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2014/34977 dated Sep. 18, 2014, 12 pages. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2012/060865 dated Mar. 29, 2013, 10 pages. |
Asano, Futoshi, Goto, Masataka, Itou, Katunobu, Asoh, Hideki; Real-Time Sound Source Localization and Separation System and Its Application to Automatic Speech Recognition; Proceedings of Eurospeech, 2001; p. 1013-1016; 2001. |
Benko, Hrvoje, Wilson, Andrew, Balakrishnan, Ravin; Sphere: Multi-Touch Interactions on a Spherical Display; Proceedings of UIST, 2008; pp. 77-86; 2008. |
Burges, Christopher; A Tutorial on Support Vector Machines for Pattern Recognition; Data Mining and Knowledge Discovery, 2; pp. 121-167; 1998. |
Cao, Xiang, Wilson, Andrew, Balakrishnan, Ravin, Hinckley, Ken, Hudson, Scott; ShapeTouch: Leveraging Contact Shape on Interactive Surfaces; IEEE International Workshop on Horizontal Interactive Human Computer System (Tabletop); pp. 139-146; 2008. |
Deyle, Travis, Palinko, Szabolcs, Poole, Erika Shehan, Starner, Thad; Hambone: A Bio-Acoustic Gesture Interface; Proceedings of ISWC, 2007; pp. 1-8; 2007. |
Dietz, Paul, Harsham, Bret, Forlines, Clifton, Leigh, Darren, Yerazunis, William, Shipman, Sam, Schmidt-Nielsen, Bent, Ryall, Kathy; DT Controls: Adding Identity to Physical Interfaces; ACM Symposium on User Interface Software & Technology (UIST); pp. 245-252; 2005. |
Dietz, Paul, Leigh, Darren; DiamondTouch: A Multi-User Touch Technology; ACM Symposium on User Interface Software & Technology (UIST); pp. 219-226; 2001. |
Gutwin, Carl, Greenberg, Saul, Blum, Roger, Dyck, Jeff, Tee, Kimberly, McEwan, Gregor; Supporting Informal Collaboration in Shared-Workspace Groupware; Journal of Universal Computer Science, 14(9); pp. 1411-1434; 2008. |
Hall, Mark, Frank, Eibe, Holmes, Geoffrey, Pfahringer, Bernhard, Reutemann, Peter, Witten, Ian; The WEKA Data Mining Software: An Update; SIGKDD Explorations, 11(1); pp. 10-18; 2009. |
Harrison, Chris, Tan, Desney, Morris, Dan; Skinput: Appropriating the Body as an Input Surface; Proceedings of CHI, 2010; pp. 453-462; 2010. |
Harrison, Chris, Hudson, Scott; Scratch Input: Creating Large, Inexpensive, Unpowered and Mobile Finger Input Surfaces; Proceedings of UIST, 2008; pp. 205-208; 2008. |
Hartmann, Bjorn, Ringel Morris, Meredith, Benko, Hrvoje, Wilson, Andrew; Augmenting Interactive Tables with Mice & Keyboards; Proceedings of UIST, 2009; pp. 149-152; 2009. |
Hinckley, Ken, Song, Hyunyoung; Sensor Synaesthesia: Touch in Motion, and Motion in Touch; Proceedings of CHI, 2011; pp. 801-810; 2011. |
Hinckley, Ken, Yatani, Koji, Pahud, Michel, Coddington, Nicole, Rodenhouse, Jenny, Wilson, Andy, Benko, Hrvoje, Buxton, Bill; Pen + Touch = New Tools; Proceedings of UIST, 2010; pp. 27-36; 2010. |
Hinkley, Ken, Yatani, Koji, Pahud, Michel, Coddington, Nicole, Rodenhouse, Jenny, Wilson, Andy, Benko, Hrvoje, Buxton, Bill; Manual Deskterity: An Exploration of Simultaneous Pen + Touch Direct Input; Proceedings of CHI, 2010; pp. 2793-2802; 2010. |
Holz, Christian, Baudisch, Patrick; The Generalized Perceived Input Point Model and How to Double Touch Accuracy by Extracting Fingerprints; Proceedings of CHI, 2010; pp. 581-590; 2010. |
Kaltenbrunner, Martin, Bencina, Ross; reacTIVision: A Computer-Vision Framework for Table-Based Tangible Interaction; Proceedings of TEI, 2007; pp. 69-74; 2007. |
Matsushita, Nobuyuki, Rekimoto, Jun; HoloWall: Designing a Finger, Hand, Body, and Object Sensitive Wall; Proceedings of UIST, 1997; pp. 209-210; 1997. |
Mimio; http://www.mimio.com. |
Olwal, Alex, Wilson, Andrew; SurfaceFusion: Unobtrusive Tracking of Everyday Objects in Tangible User Interfaces; Proceedings of GI, 2008; pp. 235-242; 2008. |
Paradiso, Joseph, Leo, Che King; Tracking and Characterizing Knocks Atop Large Interactive Displays; Sensor Review, 25(2); pp. 134-143; 2005. |
Paradiso, Joseph, Hsiao, Kai-yuh, Strickon, Joshua, Lifton, Joshua, Adler, Ari; Sensor Systems for Interactive Surfaces; IBM Systems Journal, 39(3-4); pp. 892-914; 2000. |
Patten, James, Ishii, Hiroshi, Hines, Jim, Pangaro, Gian; Sensetable: A Wireless Object Tracking Platform for Tangible User Interfaces; Proceedings of CHI, 2001; pp. 253-260; 2001. |
Rekimoto, Jun, Saitoh, Masanori; Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments; Proceedings of CHI, 1999; pp. 378-385; 1999. |
Rekimoto, Jun, Sciammarella, Eduardo; ToolStone: Effective use of the Physical Manipulation Vocabularies of Input Devices; Proceedings of UIST, 2000; pp. 109-117; 2000. |
Rekimoto, Jun; SmartSkin: An Infrastructure for Freehand Manipulation on Interactive Surfaces; Proceedings of CHI, 2002; pp. 113-120; 2002. |
Vandoren, Peter, Van Laerhoven, Tom, Claesen, Luc, Taelman, Johannes, Di Fiore, Fabian, Van Reeth, Frank, Flerackers, Eddy; DIP-IT: Digital Infrared Painting on an Interactive Table; Proceedings of CHI, 2008; pp. 2901-2906; 2008. |
Wang, Feng, Ren, Xiangshi; Empirical Evaluation for Finger Input Properties in Multi-Touch Interaction; Proceedings of CHI, 2009; pp. 1063-1072; 2009. |
International Search Report and Written Opinion for PCT/US2012/060865; dated Mar. 29, 2013. |
Non-Final Office Action—dated Mar. 13, 2015—U.S. Appl. No. 13/958,427, filed Mar. 23, 2013, titled: “Capture of Vibro-Acoustic Data Used to Determine Touch Types”. |
Final Office Action—dated Jun. 19, 2015—U.S. Appl. No. 13/958,427, filed Mar. 23, 2013, titled: “Capture of Vibro-Acoustic Data Used to Determine Touch Types”. |
Non-Final Office Action—dated Apr. 6, 2015—U.S. Appl. No. 13/887,711, filed May 6, 2013, title: “Using Finger Touch Types to Interact With Electronic Devices”. |
Final Office Action—dated Aug. 7, 2015—U.S. Appl. No. 14/191,329, filed Feb. 26, 2014, titled: “Using Capacitive Images for Touch Type Classification”. |
Non-Final Office Action—dated Oct. 1, 2015—U.S. Appl. No. 14/503,894, filed Dec. 30, 2015, titled: “Method and Apparatus for Addressing Touch Discontinuities”. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2014/049485 dated Nov. 17, 2014. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2014/033380 dated Mar. 13, 2015. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2014/034977 dated Sep. 18, 2014. |
Non-Final Office Action—dated Oct. 7, 2015—U.S. Appl. No. 14/495,041, filed Sep. 24, 2014, titled: “Method for Improving Accuracy of Touch Screen Event Analysis by Use of Spatiotemporal Touch Patterns”. |
Non-Final Office Action—dated Oct. 2, 2015—U.S. Appl. No. 14/486,800, filed Sep. 15, 2014, titled: “Method and Apparatus for Resolving Touch Screen Ambiguities”. |
Non-Final Office Action—dated Dec. 18, 2015—U.S. Appl. No. 14/483,150, filed Sep. 11, 2014, titled: “Method and Apparatus for Differentiating Touch Screen Users Based on Touch Event Analysis”. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2015/051355 dated Dec. 15, 2015. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2015/050570 dated Dec. 17, 2015. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2015/047428 dated Nov. 27, 2015. |
International Search Report and Written Opinion received for PCT Application No. PCT/US2015/014581 dated May 14, 2015. |
Non-Final Office Action—dated May 31, 2017—U.S. Appl. No. 14/612,089, filed Feb. 2, 2015, titled: “Method and System for Activating Different Interactive Functions Using Different Types of Finger Contacts.” |
Final Office Action—dated Dec. 20, 2016—U.S. Appl. No. 15/073,407, filed Mar. 17, 2016, titled: “Input Tools Having Viobro-Acoustically Distinct Regions and Computing Device for Use With the Same.” |
Non-Final Office Action—dated Nov. 10, 2016—U.S. Appl. No. 13/958,427, filed Aug. 2, 2013, titled: “Capture of Vibro-Acoustic Data Used to Determine Touch Types.” |
Final Office Action—dated Aug. 26, 2016—U.S. Appl. No. 14/219,919, filed Mar. 19, 2014, titled: “Method and Device for Sensing Touch Input.” |
Non-Final Office Action—dated Feb. 2, 2017—U.S. Appl. No. 14/191,329, filed Feb. 26, 2014, titled: “Using Capacitive Images for Touch Type Classification.” |
Final Office Action—dated Jun. 8, 2017—U.S. Appl. No. 13/887,711, filed May 6, 2013, titled: “Using Finger Touch Types to Interact With Electronic Devices.” |
Non-Final Office Action—dated Apr. 21, 2017—U.S. Appl. No. 15/075,648, filed Mar. 21, 2016, titled: “Method and Apparatus for Differentiating Touch Screen Users Based on Touch Event Analysis.” |
Non-Final Office Action—dated Dec. 1, 2016—U.S. Appl. No. 14/486,800, filed Sep. 15, 2014, titled: “Method and Apparatus for Resolving Touch Screen Ambiguities.” |
Non-Final Office Action—dated Nov. 25, 2016—U.S. Appl. No. 14/495,041, filed Sep. 24, 2014, titled: “Imethod for Improving Accuracy of Touch Screen Event Analysis by Use of Spatiotemporal Touch Patterns.” |
Non-Final Office Action—dated May 16, 2017—U.S. Appl. No. 14/503,894, filed Oct. 1, 2014, titled: “Method and Apparatus for Addressing Touch Discontinuities.” |
Final Office Action—dated Jan. 18, 2017—U.S. Appl. No. 14/684,407, filed Apr. 12, 2015, titled: “Method and Apparatus for Classifying Contacts With a Touch Sensitive Device.” |
Final Office Action—dated May 1, 2017—U.S. Appl. No. 14/834,434, filed Aug. 24, 2015, titled: “Touch Sensitive DeviceWith Multi-Sensor Stream Synchronized Data.” |
Non-Final Office Action—dated Jun. 13, 2016—U.S. Appl. No. 14/751,589, filed Jun. 26, 2015, titled: “Method and Apparatus for Classifying a Touch Event on a Touch Screen as Related to One of Multiple Function Generating Interaction Layers and Activating a Function in the Selected Interaction Layer.” |
International Search Report and Written Opinion for PCT/US2016/044552; dated Oct. 17, 2016. |
International Search Report and Written Opinion for PCT/US2016/040194; dated Sep. 19, 2016. |
International Search Report and Written Opinion for PCT/US2015/051582; dated Feb. 26, 2016. |
International Search Report and Written Opinion for PCT/US2015/051106; dated Jan. 28, 2016. |
International Search Report and Written Opinion for PCT/US2015/047616; dated Jul. 1, 2016. |
European Patent Office Extended Search Report for EP 14 83 2247; dated Feb. 23, 2017. |
European Patent Office Extended Search Report for EP 14 79 4212; dated Nov. 9, 2016. |
International Search Report and Written Opinion for PCT/US2015/050570; dated Dec. 17, 2015. |
International Search Report and Written Opinion for PCT/US2015/05135; dated Dec. 15, 2015. |
Number | Date | Country | |
---|---|---|---|
20160085372 A1 | Mar 2016 | US |