1. Technical Field
The invention relates to interactive displays. More particularly, the invention relates to touch detecting, multi-user, interactive displays.
2. Description of the Prior Art
There are many situations in which one or more individuals interactively explore image based data. For example, a team of paleontologists may wish to discuss an excavation plan for a remote dig site.
To do so, they wish to explore in detail the geographic characteristics of the site as represented on digitized maps. In most laboratories, this would require the team to either huddle around a single workstation and view maps and images on a small display, or sit at separate workstations and converse by telephone.
One approach to addressing this shortcoming is a touch detecting interactive display, such as that disclosed in the referenced patent filing “Touch Detecting Interactive Display.” In such a system, an image is produced on a touch detecting display surface. The locations at which a user contacts the surface are, determined and, based on the position of the motions of these locations, user gestures are determined. The display is then updated based on the determined user gestures.
Corresponding with the display surface is a touch sensor 155 that is capable of detecting when and where a user touches the display surface. Based upon the contact information provided by the touch sensor user gestures are identified and a command associated with the user gesture is determined. The command is executed, altering the displayed imagery in the manner requested by the user via the gesture. For example, in
Many touch sensors used n displays, such as that shows in
Typically, the contact information is updated over time at discrete intervals, and based upon the motion of the contact locations, user gestures are identified. Determining gestures from the contact information alone, however, provides considerable challenge. Gesture Identification schemes often fail to correctly address imperfections in
However, during the dragging motion, the user inadvertently decreases the inclination of his finger, and the user3 s knuckles initiate a second contact. As the second contact is separated both temporally and spatially from the initial contact, many gesture identification schemes erroneously determine that the second contact is associated with a new and distinct gesture.
In each of these cases, the gesture identification scheme has failed in that the intent of the user is not faithfully discerned.
Systems addressing the above deficiencies have been proposed. For example, in U.S. Pat. No. 5,543,591 to Gillespie et al a touch sensor provides, on a provisional basis, all motions of a detected contact to a host computer, to be interpreted as cursor movements. If, however, the contact is terminated within a short period of time after initiation of the contact and the distance moved since initiation of the contact is small, the cursor motions are reversed and the contact is interpreted as a mouse click. However, while this approach may be suitable for control of a cursor, it is not suitable for control of imagery, where undoing motions may lead to significant user confusion. Thus, despite such improvements, it would be advantageous to provide a more reliable method of classifying user gestures from contact information that more accurately discerns the intent of a user in performing the gesture.
A method and apparatus for identifying user gesture includes a touch sensor for determining contact information that describes locations at which a user contacts a touch sensitive surface corresponding to a display. The touch sensor provides the contact information to a gesture identification module which uses state information to identify a user gesture and, responsive thereto issues an associated display command to a display control module. The display control module updates the display based on display commands received from the gesture identification module.
To address the above noted deficiencies, a novel state-based approach to identifying user gestures is proposed. Gestures are identified in a manner that more accurately reflects user intent, thereby facilitating more natural interaction with the display.
In the preferred embodiment of the invention the touch sensor is physically coincident with the display, as shown in
The touch sensor of
Alternatively, a resistive touch pad, such as those commonly used in laptop computers, may be placed beneath a flexible display surface. The resistive touch pad comprises two layers of plastic that are separated by a compressible insulator, such as air, with a voltage differential maintained across the separated layers. When the upper layer is touched with sufficient pressure, it is deflected until it contacts the lower layer, changing the resistive characteristics of the upper to lower layer current pathway. By considering these changes in resistive characteristics, the location of the contact can be determined. Capacitive touch pads may also be used, such as the Synaptics TouchPad™ (www.synaptics.com/products/touchpad.cfm).
As shown in
Preferably, the touch sensor explicitly indicates within the contact information that a contact has been initiated or terminated. Alternatively, the gesture identification module may infer an initiation or termination of a contact from the inception, continuation, and ceasing of position information for a particular contact. Similarly, some touch sensors may explicitly report the motion of a contact point within the contact information. Alternatively, the gesture identification module may store the contact information reported by the touch sensor at successive updates. By comparing the position for each contact point over two or more updates, motion may be detected. More specifically, a simple difference between two consecutive updates may be computed, or a more complicated difference scheme incorporating several consecutive updates, e.g. a moving average, may be used. The later approach may be desirable contact positions reported by touch sensor exhibit a high level of noise. In this case, a motion threshold may also be employed, below which motion is not detected.
Herein, the first and second contact are referred to as C1 and C2. The initiation of the first contact, as either reported by the sensor or determined by the gesture identification module, is referred to as D1 (“Down-1”), and the initiation of a second contact is referred to as D2. Similarly, the termination of the first and second contact is referred to as U1 (“Up-1”) and U2, respectively. The presence of motion of the first and second contacts is termed M1 and M2, respectively. More specifically, M1 and M2 are computed as the difference between the position of C1 and C2 at the current update and the position of C1 and C2 at the previous update.
Often, a user may briefly lose contact with the touch sensor, or the touch sensor itself may briefly fail to register a persistent contact. In either case, the software monitoring the contact information registers the termination of one contact and the initiation of a new contact, despite the fact that the user very likely considers the action as a continued motion of a single contact. Thus, in some embodiments of the invention, a smoothing capability may be added to address intermittent loss of contact. Specifically, a minimum time may be required before a termination of a contact is acknowledged. That is, if the touch sensor reports that position information is no longer available for contact C1 or C2, and then shortly thereafter reports a new contact in the immediate vicinity, the new contact may be considered a continuation of the prior contact. Appropriate thresholds of time and distance may be used to ascertain if the new contact is, in fact, merely a continuation of the previous contact.
Upon initialization, the gesture identification module enters the Idle state (3000). In the Idle state, the gesture identification module identifies no gesture and issues no display command to the display control module. The gesture identification module remains in the Idle state until the initiation D1 of a first contact C1. Upon initiation D1 of a first contact C1, the gesture identification module enters the Tracking One state (3010).
In the Tracking One state, the gesture identification module identifies no gesture and issues no display command to the display control module. However, the gesture identification module continues to monitor the contact C1. If the first contact is terminated U1, the gesture identification module enters the Clicking state (3020). If motion M1 of the first contact is detected, the gesture identification module enters the Awaiting Click state (3030). If the initiation of a second contact D2 is detected, the gesture identification module enters the Tracking Two state (3060). Otherwise, the gesture identification module remains in the Tracking One state.
In the Awaiting Click state, the gesture identification module identifies no gesture and issues no display command to the display control module. However, the gesture identification module continues to monitor the behavior of the first contact and awaits a possible second contact. If the first contact is terminated U1 within a predetermined time period 66 tc, the gesture identification module enters the Clicking state. If a second contact is initiated D2 within the predetermined time period Δtc, the gesture identification module enters the Tracking Two state. If the first contact is not terminated and a second contact is not initiated within the predetermined time period Δtc, the gesture identification module enters the Assume Panning state (3040).
In the Clicking state, the gesture identification module identifies a clicking gesture and issues a click command to the display control module, that, when executed by the display control module, provides a visual confirmation that a location or object on the display has been designated.
In the Assume Panning state, the gesture identification module identifies no gesture and issues no display command to the display control module. However, the gesture identification module continues to monitor the behavior of the first contact and awaits a possible second contact. If the first contact is terminated U1 within a predetermined time period Δtp, the gesture identification module returns to the Idle state. If a second contact is initiated D2 within the predetermined time period Δtp, the gesture identification module enters the Tracking Two state. If the first contact is not terminated, and a second contact is not initiated within the predetermined time period Δtp, the gesture identification module determines that neither a click nor a gesture requiring two contacts is forthcoming and enters the Panning state (3050).
In the Panning state, the gesture identification module identifies a panning gesture and issues a pan command to the display control module that, when executed by the display control module, translates the displayed imagery. Generally, the pan command specifies that the imagery be translated a distance proportional to the distance the first contact has moved M1 between the previous and current updates of the first contact position C1. Preferably, the translation of the imagery, measured in pixels, is equal to the movement of the first contact, measured in pixels. This one-to-one correspondence provides the user with a natural sense of sliding the imagery as if fixed to the moving contact location. If the first contact is terminated U1, the gesture identification module returns to the Idle state. If the first contact continues to move M1, the gesture identification module remains in the Panning state to identify another panning gesture and issue another pan command to the display control module. Panning thus continues until one of the contacts is terminated.
In the Tracking Two state, the gesture identification module identifies no gesture and issues no display command to the display control module. However, the gesture identification module continues to monitor the behavior of the first and second contacts. If either the first or second contact is terminated, U1 or U2, the gesture identification module enters the Was Tracking Two state. Otherwise, the gesture identification module determines if the motions of the first and second contact points M1 and M2 are aligned or opposed. If the contact points exhibit Opposed Motion, the gesture identification module enters the Zooming state (3070). If the contact points exhibit Aligned Motion, the gesture identification module enters the Panning state. Aligned Motion thus results in two contacts being treated as one in that the behavior of the second contact is ignored in the Panning state. This greatly alleviates the problems encountered when a user attempts to gestures with his entire hand. As noted previously, a user often believes he is contacting the touch sensor at a single, hand sized region but, in fact, establishes two separate contact points as determined by the touch sensor.
|φ1−φ2|<θa (1)
Similarly, the motions are considered opposed if
|φ1−φ2|>θo (2)
In the preferred embodiment of the invention, θa=θo, That is, any pair of motions M1 and M2 is classified as either aligned or opposed. In this instance, only one of the two tests described in Equations 1 and 2 need be performed. If the test for aligned motion is performed and the criterion is not satisfied, the motions are considered opposed. Conversely, if the test for opposed motion is performed and the criterion is not satisfied, the motions are considered aligned.
In an alternative embodiment of the invention, θa≠θo, providing an angular region of dead space (θa≧φ≦θo) within which the motions are neither aligned nor opposed. In this embodiment, both tests described in Equations 1 and 2 must be performed. If neither criterion is satisfied, the gesture identification module remains in the Tracking Two state.
In the Zooming state, the gesture identification module identifies a zooming gesture and issues a zoom command to the display control module that, when executed by the display control module, alters the magnification of the displayed imagery. Specifically, with each update of contact information, the magnification of the screen is scaled by the factor
where do is the distance between C1 and C2 prior to the most recent update, and d is the distance 330 between C1 and C2 after the most recent update If either the first or second contact is terminated, U1 or U2, the gesture identification module enters the Was Tracking Two state (
In the Was Tracking Two state, the gesture identification module identifies no gesture and issues no display command to the display control module. The gesture identification module awaits the termination of the remaining contact, U2 or U1. Upon termination of the remaining contact, the gesture identification module returns to the Idle state.
Although the invention is described herein with reference to the preferred embodiment, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the present invention. Accordingly, the invention should only be limited by the Claims included below.
This application is related to U.S. utility patent application Ser. No. 10/913,105 entitled “Touch Detecting Interactive Display”, attorney docket number APPL0053, filed Aug. 6, 2003, and U.S. provisional patent application Ser. No. 60/647,343 entitled “Touch Table Touch Detection and Gesture Recognition Technologies”, attorney docket number APPL0058PR, filed Jan. 25, 2005, and U.S. provisional patent application Ser. No. 60/701,892 entitled “Interactive Display Technologies”, attorney docket number APPL0065PR, filed Jul. 22, 2005, which applications are incorporated herein in their entirety by this reference thereto.
Number | Date | Country | |
---|---|---|---|
60701892 | Jul 2005 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 10913105 | Aug 2004 | US |
Child | 11458956 | Jul 2006 | US |