Patent Application
20110221666
This application claims priority to Australian Provisional Application No. 2009905747, filed Nov. 24, 2009 and titled “An apparatus and method for performing command movements in an imaging area,” which is incorporated by reference herein in its entirety.
Touch-enabled computing devices continue to increase in popularity. For example, touch-sensitive surfaces that react to pressure by a finger or stylus may be used atop a display or in a separate input device. As another example, a resistive or capacitive layer may be used. As a further example, one or more imaging devices may be positioned on a display or input device and used to identify touched locations based on interference with light.
Regardless of the underlying technology, touch sensitive displays are typically used to receive input provided by pointing and touching, such as touching a button displayed in a graphical user interface. This may become inconvenient to users, who often need to reach toward a screen to perform a movement or command.
Embodiments include computing devices comprising a processor and an imaging device. The processor can be configured to support a mode where gestures in space are recognized, such as through the use of image processing to track the position, identity, and/or orientation of objects to recognize patterns of movement. To allow for reliable use of other types of input, the processor can further support one or more other modes during which the computing device operates but does not recognize some or all available gestures. In operation, the processor can determine whether a gesture recognition mode is activated, use image data from the imaging device to identify a pattern of movement of an object in the space, and execute a command corresponding to the identified pattern of movement if the gesture recognition mode is activated. The processor can also be configured to enter or exit the gesture recognition mode based on various input events.
This illustrative embodiment is discussed not to limit the present subject matter, but to provide a brief introduction. Additional embodiments include computer-readable media embodying an application configured in accordance with aspects of the present subject matter and computer-implemented methods configured in accordance with the present subject matter. These and other embodiments are described below in the Detailed Description. Objects and advantages of the present subject matter can be determined upon review of the specification and/or practice of an embodiment configured in accordance with one or more aspects taught herein.
Reference will now be made in detail to various and alternative exemplary embodiments and to the accompanying drawings. Each example is provided by way of explanation, and not as a limitation. It will be apparent to those skilled in the art that modifications and variations can be made. For instance, features illustrated or described as part of one embodiment may be used on another embodiment to yield a still further embodiment.
In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the subject matter. However, it will be understood by those skilled in the art that the subject matter may be practiced without these specific details. In other instances, methods, apparatuses or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure the subject matter.
In this example, system 102 features an optical system 104, which can include one or more imaging devices such as line scan cameras or area sensors. Optical system 104 may also include an illumination system, such as infrared (IR) or other source or sources. System 102 also includes one or more processors 106 connected to memory 108 via one or more busses, interconnects, and/or other internal hardware indicated at 110. Memory 108 represents a computer-readable medium such as RAM, ROM, or other memory.
I/O component(s) 112 represents hardware that facilitates connections to external resources. For example, the connections can be made via universal serial bus (USB), VGA, HDMI, serial, and other I/O connections to other computing hardware and/or other computing devices. It will be understood that computing device 102 could include other components, such as storage devices, communications devices (e.g., Ethernet, radio components for cellular communications, wireless internet, Bluetooth, etc.), and other I/O components such as speakers, a microphone, or the like. Display(s) 114 represent any suitable display technology, such as liquid crystal diode (LCD), light emitting diode (LED, e.g., OLED), plasma, or some other display technology.
Program component(s) 116 are embodied in memory 108 and configure computing device 102 via program code executed by processor 106. The program code includes code that configures processor 106 to determine whether a gesture recognition mode is activated, use image data from the imaging device(s) of optical system 104 to identify a pattern of movement of an object in the space, and program code that configures processor 106 to execute a command corresponding to the identified pattern of movement if the gesture recognition mode is activated.
For example, component(s) 116 may be included in a device driver, a library used by an operating system, or in another application. Although examples are provided below, any suitable input gestures can be recognized, with a “gesture” referring to a pattern of movement through space. The gesture may include touch or contact with display 114, a keyboard, or some other surface, or may occur entirely in free space.
As shown by the dashed lines, light from object 118 can be detected by one or more imaging devices 104A based on light emitted from source 104B. Although a separate light source is shown in these examples, some implementations rely on ambient light, or even light emitted from a source on object 118. Object 118 can be moved in the space adjacent display 114 and in view of imaging devices 104A in order to set zoom levels, scroll pages, resize objects, and delete, insert, or otherwise manipulate text and other content, for example. Gestures may involve movement of multiple objects 118—for example, pinches, rotations, and other movements of fingers (or other objects) relative to one another.
Because use of computing device 102 will likely entail contact-based input or other non-gesture input, the support of at least a gesture input mode when gestures are recognized and at least one second mode during which some or all gestures are not recognized is advantageous. For example, in the second mode, optical system 104 can be used to determine touch or near-touch events with respect to surface 120. As another example, when the gesture recognition mode is not active, optical system 104 could be used to identify contact-based inputs, such as keyboard inputs determined based on contact locations in addition to or instead of actuation of hardware keys. As a further example, when gesture recognition mode is not active, device 102 could continue operating using hardware-based input.
In some implementations, the gesture recognition mode is activated or deactivated based on one or more hardware inputs, such as actuation of a button or a switch. For example, a key or key combination from keyboard 122 can be used to enter and exit gesture recognition mode. As another example, software input indicating that the gesture recognition mode is to be activated can be used—for example, an event can be received from an application indicating that the gesture recognition mode is to be activated. The event may vary on the application—for instance, a configuration change in the application may enable gesture inputs and/or the application may switch into gesture recognition mode in response to other events. However, in some implementations gesture recognition mode is activated and/or deactivated based on recognizing a pattern of movement.
For example, returning to
As a particular example, the code may configure processor 106 to search the image data for a finger or another object 118 and, if the finger/object remains stationary in the image data for a set period of time, to activate gesture recognition capabilities. For instance, a user may type on keyboard 122 and then lift a finger and hold it in place to activate gesture recognition capability. As another example, the code may configure processor 106 to search image data to identify a finger proximate surface 120 of screen 114 and, if the finger is proximate to surface 120, to switch into gesture recognition mode.
As noted above, gestures may be used to deactivate the gesture recognition mode as well. For example, one or more patterns of movement may correspond to a deactivation pattern. Executing the command can comprise storing data that the gesture recognition mode is no longer activated. For example, a user may trace a path corresponding to an alphanumeric character or along some other path that is recognized and then a flag set in memory to indicate that no further gestures are to be recognized until the gesture recognition mode is again activated.
Block 402 represents activating the gesture recognition mode in response to a user event indicating that the gesture recognition mode is to be activated. The event may be hardware-based, such as input from a key press, key combination, or even a dedicated switch. As also noted above, the event may be software based. As another example, one or more touch-based input commands may be recognized, such as touches at portions of a display or elsewhere on the device that correspond to activating the gesture recognition mode. As a further example, the event may be based on image data using the imaging hardware used to recognize gestures and/or other imaging hardware.
For example, as noted below, presence of an object beyond a threshold period of time in the imaged space can trigger the gesture recognition mode. As another example, prior to activation of the gesture recognition mode, the system may be configured to recognize a limited subset of one or more gestures that activate the full gesture recognition mode, but not to respond to other gestures until the gesture recognition mode is activated.
Block 404 represents detecting input once the gesture recognition mode is activated. For example, one or more imaging devices can be used to obtain image data representing a space, such as a space adjacent a display, above a keyboard, or elsewhere, with image processing techniques used to identify one or more objects and motion thereof. For example, in some implementations, two imaging devices can be used along with data representing the relative position of the devices to the imaged space. Based on a projection of points from imaging device coordinates, one or more space coordinates of object(s) in the space can be detected. By obtaining multiple images over time, the coordinates can be used to identify a pattern of movement of the object(s) in the space. The coordinates may be used to identify the object as well, such as by using shape recognition algorithms.
The pattern of movement can correspond to a gesture. For example, a series of coordinates of the object can be analyzed according to one or more heuristics to identify a likely intended gesture. For example, when a likely intended gesture is identified, a dataset correlating gestures to commands can be accessed to select a command that corresponds to the gesture. Then, the command can be carried out, and block 406 represents carrying out that command, either directly by the application analyzing the input or by another application that receives data identifying the command. Several examples of gestures and corresponding commands are set forth later below.
In some implementations, identifying the pattern of movement of the object comprises identifying a first pattern of movement followed by a second pattern of movement. In such a case, determining the command to be carried out can comprise selecting one of a plurality of commands based on the first pattern of movement and determining a parameter value based on the second pattern of movement. For example, a first gesture can be used to determine a zoom command is desired and a second gesture can be used to determine the desired degree of zoom and/or direction (i.e., zoom-in or zoom-out). Numerous patterns of movement may be chained together (e.g., a first pattern of movement, second pattern of movement, third pattern of movement, etc.).
Block 408 represents deactivating the gesture recognition mode in response to any desired input event. For example, actuation of a hardware element (e.g., a key or switch) may deactivate the gesture recognition mode. As another example, the dataset of commands may include one or more “deactivation” gestures that correspond to a command to exit/deactivate the gesture recognition mode. As a further example, the event may simply comprise absence of a gesture for a threshold period of time, or absence of the object from the imaged space for a threshold period of time.
Block 502 represents monitoring the area imaged by the optical system of the computing device. As mentioned above, one or more imaging devices can be sampled and the resulting image data representing the space can be analyzed for the presence or absence of one or more objects of interest. In this example, a finger is the object of interest, and so block 504 represents evaluating whether a finger is detected. Other objects, of course, could be searched for in addition to or instead of a finger.
Block 506 represents determining whether the object of interest (e.g., the finger) is in the space for a threshold period of time. As shown in
In
For example, in
As another example, the flow could proceed from
In
Optionally, the user can then rotate his or her hand as shown at G8 in
In some implementations, the rotational portion of the gesture shown at G8 need not be performed. Namely, the user can extend the thumb as shown at G7 and then complete a “sideways shooting” gesture by bringing his/her thumb into contact with the remainder of his/her hand.
In some instances, the single-finger click gesture can cause difficulty, particularly if the gesture recognition system uses a finger to control cursor position. Accordingly,
As shown in
Another example of a gesture is a knob-rotation gesture in which a plurality of fingers are arranged as if gripping a knob. For example, the gesture recognition can recognize placement of two fingers as if the user is gripping a knob or dial, followed by rotation of the user's hand such as shown at 118A in
Yet another example of a gesture is a flat hand panning gesture. For example, a user may place an open and in view of the gesture recognition system and move the hand left, right, up, or down to move an object, pan an onscreen image, or invoke another command.
A further gesture is a closed-hand rotation gesture. For example, a user may close a fist and then rotate the closed fist. This gesture can be recognized, for example, by tracking the orientation of the user's fingers and/or by recognizing the closed fist or closing of the hand, followed by rotation thereof. The closed fist gesture can be used, for example, in 3D modeling software to rotate an object about an axis.
Other gestures can be defined, of course. As another example, a pattern of movement may correspond to a line in space, such as tracing a line parallel to an edge of the display to provide a vertical or horizontal scroll command. As another example, a line in the space can extend toward the display or another device component, with the corresponding command being a zoom command.
Although specific examples were noted above for the “R”, “Z”, and “X” alphanumeric characters, the path could correspond to any alphanumeric character in any language. In some implementations, the path traced by the alphanumeric gesture is stored in memory and then a character recognition process is performed to identify the character (i.e., in a manner similar to optical character recognition, though in this case rather than pixels defined on a page, the character's pixels are defined by the gesture path). Then, an appropriate command can be determined from the character. For example, computer applications can be indexed to various letters (e.g., “N” for Notepad.exe, “W” for Microsoft(R) Word(R), etc.). Recognition of alphanumeric gestures could also be used to sort lists, select items from a menu, etc.
As another example, the path could correspond to some other shape, such as a polygon, circle, or an arbitrary shape or pattern. The system may identify a corresponding character, pattern, or shape in any suitable manner. Additionally, in identifying any gesture, the system can allow for variations in the path (e.g., to accommodate imprecise motion by users).
Any one of the gestures discussed herein can be recognized alone by a gesture recognition system, or may be recognized as part of a suite of gestures, the suite of gestures including any one or more of the others discussed herein, and/or still further gestures. Additionally, the gestures presented in the examples above were presented with examples of commands. One of skill in the art will recognize that that particular pairings of gestures and commands are for purposes of example only, and that any gesture or pattern of movement described herein can be used as part of another gesture, and/or may be correlated to any one of the commands described herein or to one or more other commands.
The various systems discussed herein are not limited to any particular computing hardware architecture or configuration. A computing device can include any suitable arrangement of components that provide a result conditioned on one or more inputs.
Suitable computing devices include microprocessor-based computer systems accessing stored software from a non-transitory computer-readable medium (or media), the software comprising instructions that program or configure the general-purpose computing apparatus to act as a specialized computing apparatus implementing one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device. When software is utilized, the software may comprise one or more components, processes, and/or applications. Additionally or alternatively to software, the computing device(s) may comprise circuitry that renders the device(s) operative to implement one or more of the methods of the present subject matter. For example, an application-specific integrated circuit (ASIC) or programmable logic array may be used.
Examples of computing devices include, but are not limited to, servers, personal computers, mobile devices (e.g., tablets, smartphones, personal digital assistants (PDAs), etc.) televisions, television set-top boxes, portable music players, and consumer electronic devices such as cameras, camcorders, and mobile devices. Computing devices may be integrated into other devices, e.g. “smart” appliances, automobiles, kiosks, and the like.
Embodiments of the methods disclosed herein may be performed in the operation of computing devices. The order of the blocks presented in the examples above can be varied—for example, blocks can be re-ordered, combined, and/or broken into sub-blocks. Certain blocks or processes can be performed in parallel.
Any suitable non-transitory computer-readable medium or media may be used to implement or practice the presently-disclosed subject matter, including, but not limited to, diskettes, drives, magnetic-based storage media, optical storage media (e.g., CD-ROMS, DVD-ROMS, and variants thereof), flash, RAM, ROM, and other memory devices, along with programmable logic as noted above.
The use of “adapted to” or “configured to” herein is meant as open and inclusive language that does not foreclose devices adapted to or configured to perform additional tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or values beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting.
While the present subject matter has been described in detail with respect to specific embodiments thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing may readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, it should be understood that the present disclosure has been presented for purposes of example rather than limitation, and does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2009905747 | Nov 2009 | AU | national |
