Patent Application
20110119216
Computing devices may be configured to accept input from different types of input devices. For example, some computing devices utilize a pointer based approach in which graphics, such as buttons, scroll bars, etc., may be manipulated via a mouse, touch-pad, or other such input device, to trigger computing functions. More recent advances in natural user interfaces have permitted the development of computing devices that detect touch inputs.
However, in some use environments, the number of touch inputs may be significant and require a user to commit a large amount of time to learning the extensive set of touch inputs. Therefore, infrequent or novice users may experience frustration and difficulty when attempting to operate a computing device utilizing touch inputs.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
A computing device that detects precursory user-input preactions executed in an instructive region and user-input action gestures executed in a functionally-active region is provided. The computing device includes a natural input trainer to present a predictive input cue on a display in response to detecting a precursory user-input preaction performed in the instructive region. The computing device also includes an interface engine to execute a computing function in response to detecting a successive user-input action gesture performed in the functionally-active region subsequent to detection of the precursory user-input preaction.
The present disclosure is directed to a computing device that a user can control with natural inputs, including touch inputs, postural inputs, and gestural inputs. Predictive input cues are presented on a display of the computing device to provide the user with instructive input training, allowing a user to quickly learn gestural inputs as the user works with the device. A separate training mode is not needed. The predictive input cues may include various graphical representations of proposed user-input gestures having associated computing functions. Additionally, the predictive input cues may include a contextual function preview graphically representing a foreshadowed implementation of the computing function. In this way, instructions pertaining to the implementation of a predicted user-input gesture as well as a preview of the computing function associated with the predicted user-input gesture may be provided to the user.
The computing device 10 includes an input sensing subsystem 14. Suitable input sensing subsystems may include an optical sensing subsystem, a capacitive sensing subsystem, a resistive sensing subsystem, or a combination thereof. It will be appreciated that the aforementioned input sensing subsystems are exemplary in nature and alternative or additional input sensing subsystems may be utilized in some embodiments.
The input sensing subsystem 14 may be configured to detect user-input of various types. As explained in detail below, user input can be conceptually divided into two types—precursory preactions and action gestures. Precursory preactions refer to, for example, the posture of a user's hand immediately before initiating an action gesture. A precursory preaction effectively serves as an indication of what action gesture is likely to come next. An action gesture, on the other hand, refers to the completed touch input that a user carries out to control the computing device.
The input sensing subsystem 14 may be configured to detect both precursory user-input preactions executed in an instructive region and user-input action gestures executed in a functionally-active region. In the embodiment depicted in
The computing device 10, depicted in
The computing device 10 may additionally include an interface engine 22 to execute a computing function in response to the input sensing subsystem 14 detecting a successive action gesture performed in the functionally-active region subsequent to detection of the precursory posture. The natural input trainer 20 and the interface engine 22 are discussed in greater detail herein with reference to
At t1, an input sequence is initiated by a user. The initiation is executed through implementation of a precursory posture 214. In the depicted scenario, the precursory posture 214 is a hover input performed by the user staged away from the display in an instructive region (i.e., the space immediately above the display 12). However, it will be appreciated that the precursory posture may be another type of input. As previously discussed, a user-input hover may include an input in which one or more hands are positioned in an instructive region adjacent to the display. In some examples, the relative position of the fingers, palm, etc., may remain substantially stationary, and in other examples the posture can dynamically change.
An input sensing subsystem (e.g., input sensing subsystem 14 of
If a correspondence is drawn between the user-input hover and the recognized posture, a predictive input cue may be presented on the display by a natural input trainer (e.g., natural input trainer 20 of
The graphical representation 216 of the proposed user-input action gesture may include various icons such as arrows 218 illustrating the general direction of the proposed input as well as a path 220 depicting the proposed course of the input. Such graphical representations provide the user with a graphical tutorial of a user-input action gesture. In some examples, the graphical representation may be at least partially transparent so as not to fully obstruct other objects presented on the display. It will be appreciated that the aforementioned graphical representation of the proposed user-input action gesture is exemplary in nature and that additional or alternate graphical elements may be included in the graphical representation. For example, alternate or additional icons may be provided, shading and/or coloring techniques may be used to enhance the graphical depiction, etc. Furthermore, audio content may be used to supplement the graphical representation.
The graphical representation 216 of the proposed user-input action gesture may be associated with a computing function. In other words, execution of the proposed user-input action gesture by a user may trigger a computing function. In this example, the computing function is a resize function. In other examples, alternate computing functions may be used. Exemplary computing functions may include, but are not limited to, rotating, dragging and dropping, opening, expanding, graphical adjustments such as color augmentation, etc.
Continuing with
A user may choose to implement the proposed user-input action gesture in the functionally-active region, as depicted at t3 and t4 of
In some embodiments, a natural input trainer may further be configured to present the predictive input cue after the user-input hover remains substantially stationary for a predetermined period of time. In this way, a user may quickly implement a user-input action gesture (e.g., user-input touch) without assistance and avoid an extraneous presentation of the predictive input cue when such a cue is not needed. Likewise, a user may implement a user-input hover by pausing for a predetermined amount of time to initiate the presentation of the predictive input cue. Alternatively, the predictive input cue may be presented directly after the user-input hover is detected.
A user-input hover that remains stationary for an extended amount of time after a first predictive input cue is presented may indicate that a, user needs further assistance. Therefore, the natural input trainer may be configured to present a second predictive input cue after the user-input hover remains substantially stationary for a predetermined period of time. The second cue can be presented in place of the first cue or in addition to the first cue. The second cue, and subsequent cues, can be presented to the user in an attempt to offer the user a desired gesture and resulting computing function when the natural input trainer determines the user is not satisfied with the options that have been offered.
Times t1 and t2 of
If a the natural input trainer determines that the second user-input hover 310 corresponds to a recognized posture, the natural input trainer may present a second predictive input cue on the display in response to the input sensing subsystem detecting the second user-input hover. The second gestural cue may include a graphical representation 312 of a second proposed user-input action gesture executable in the functionally-active region (e.g., against the display) and associated with a second computing function. As shown, the second predictive input cue is different from the first predictive input cue. The predictive input cue may further include a contextual function preview 314 graphically representing a foreshadowed implementation of the second computing function. A user may then choose to execute the second proposed user-input action gesture against the display. In response to the execution and subsequent detection of the gesture by the input sensing subsystem, the interface engine may implement a computing function (e.g., drag), as shown at t4.
At t2, a predictive input cue including a graphical representation 412 of a proposed precursory user-input preaction is presented on the display. The proposed input precursory posture may include various graphical elements 414 indicating the configuration and location of a recognized user-input posture so that a user may adjust the unrecognized hover into a recognized posture. The proposed input precursory posture may be selected based on the characteristics of the user-input hover 410. In this way, a user may be instructed to perform a recognized user-input hover subsequent to detection of an unrecognizable user-input hover. Additionally, the proposed user-input hover may be associated with at least one input gesture and corresponding computing function.
The form of the user-input hover may be assessed based on various characteristics of the user-input hover, such as the input hand (i.e., right hand, left hand), the digits used for input, the location of the input(s), etc. In some examples a conventional form may be bio-mechanically effective. That is to say that the user may complete an input gesture initiated with an input posture without undue strain or stress on their body (e.g., fingers, hands, and arms). For example, the distance a user can spread two digits on a single hand is limited due to the configuration of the joints in their fingers. Thus, a spreading input performed with two digits on a single hand may not be a bio-mechanically effective form. However, a spreading input performed via bi-manual input may be a bio-mechanically effective form. Thus, a predictive input cue including a graphically representation 512 of a proposed user-input hover suggesting a bi-manual input may be presented on the display. In the depicted embodiment, the predictive input cue includes text. However, in other examples additional or alternate graphical elements or auditory elements may be used to train the user.
In response to detection of the-user input hover, a predictive input cue is presented on the display at t2. In the depicted embodiment, the predictive input cue includes a graphical representation 612 of a first proposed user-input action gesture (e.g., user-input touch) executable in the functionally-active region and associated with a first computing function and a graphical representation 614 of a second proposed user-input action gesture executable in the functionally-active region and associated with a second computing function. The predictive input cue may further include a first contextual function preview 616 graphically representing a foreshadowed implementation of the first computing function and a second contextual function preview 618 graphically representing a foreshadowed implementation of the second computing function. In this way, a number of proposed user-input action gestures may be presented to the user at one time, allowing the user to quickly expand their gestural repertoire. Different predictive input cues may be presented with visually distinguishable features (e.g., coloring, shading, etc.) so that a user may intuitively deduce which cues are associated with which gestures.
A predictive input cue may be presented on the display 710 in response to the input sensing subsystem detecting a precursory posture performed in the instructive region 714. As previously discussed, the predictive input cue may include a graphical representation 718 of a proposed user-input action gesture executable in the functionally-active region and associated with a computing function if the precursory posture corresponds to a recognized posture. The predictive input cue may further include a contextual function preview 720 graphically representing a foreshadowed implementation of the computing function.
The capture device 722 may be used to recognize and analyze movement of the user in the instructive region as well as the functionally-active region. The capture device may be configured to capture video with depth information via any suitable technique (e.g., time-of-flight, structured light, stereo image, etc.). As such, the capture device may include a depth camera, a video camera, stereo cameras, and/or other suitable capture devices.
For example, in time-of-flight analysis, the capture device 722 may emit infrared light to the target and may then use sensors to detect the backscattered light from the surface of the target. In some cases, pulsed infrared light may be used, wherein the time between an outgoing light pulse and a corresponding incoming light pulse may be measured and used to determine a physical distance from the capture device to a particular location on the target. In some cases, the phase of the outgoing light wave may be compared to the phase of the incoming light wave to determine a phase shift, and the phase shift may be used to determine a physical distance from the capture device to a particular location on the target.
In another example, time-of-flight analysis may be used to indirectly determine a physical distance from the capture device to a particular location on the target by analyzing the intensity of the reflected beam of light over time via a technique such as shuttered light pulse imaging.
In another example, structured light analysis may be utilized by capture device to capture depth information. In such an analysis, patterned light (i.e., light displayed as a known pattern such as a grid pattern or a stripe pattern) may be projected onto the target. On the surface of the target, the pattern may become deformed, and this deformation of the pattern may be studied to determine a physical distance from the capture device to a particular location on the target.
In another example, the capture device may include two or more physically separated cameras that view a target from different angles, to obtain visual stereo data. In such cases, the visual stereo data may be resolved to generate a depth image.
In the depicted embodiment the input-sensing subsystem includes an input device 802 spaced away from a display 804. As such, input device 802 is capable of detecting user input hovers staged away from display 804. As shown the input device and the display are enclosed by separate housings. However, in other embodiments the input device and the display may reside in a single housing. It will be appreciated that the input device may include an optical sensing subsystem, a capacitive sensing subsystem, a resistive sensing subsystem, and/or a any other suitable sensing subsystem. Furthermore, the functionally-active region is a sensing surface 806 on the input device and the instructive region is located directly above the sensing surface. Therefore a user may implement various inputs, such as a user-input touch and a user-input hover, through the input device 802.
A predictive input cue may be presented on the display 804 in response to the input sensing subsystem detecting a precursory posture performed in the instructive region. As previously discussed, the predictive input cue may include a graphical representation 808 of a proposed user-input action gesture executable in the functionally-active region and associated with a computing function if the precursory posture corresponds to a recognized posture. The predictive input cue may further include a contextual function preview 810 graphically representing a foreshadowed implementation of the computing function.
At 902, method 900 includes detecting a precursory user-input preaction staged away from a display in an instructive region. The instructive region may be adjacent to a sensing surface of the display or in a three-dimensional space away from the display. At 904, method 900 includes determining if the precursory user-input preaction corresponds to a recognized posture. Various techniques may be used to determine if the precursory user-input preaction corresponds to a recognized posture, as previously discussed.
If the precursory user-input preaction corresponds to a recognized posture (i.e., YES at 904), the method proceeds to 906 where it is determined if the recognized posture has a preferred form. The form of the posture may be determined by various characteristics of the posture, such as hand(s) used to implement the posture, the digits used for input, the location of the input, etc. It will be appreciated that in some examples, the preferred form may be a bio-mechanically effective form.
If the precursory user-input preaction does not correspond to a recognized posture (i.e., NO at 904), or if it the recognized posture does not have a preferred form (i.e., NO at 906), at 908, method 900 includes presenting on a display a graphical representation of a proposed precursory user-input preaction stageable in the instructive region, as described above.
However, if the recognized posture has a preferred form (i.e., YES at 906), at 910, method 900 includes presenting on the display a graphical representation of a proposed user-input action gesture executable in a functionally-active region and associated with a computing function. In this way, the user may be provided with a tutorial, allowing a user to easily learn the input gesture. It will be appreciated that in some embodiments a plurality of graphical representations of proposed user-input action gestures may be presented on the display.
At 912, the method includes presenting on the display a contextual function preview graphically representing a foreshadowed implementation of the computing function on the display. This allows a user to view the implementation of the computing function associated with the proposed user-input action gesture before a user-input action gesture is carried out. Therefore, a user may alter subsequent gestural input based on the contextual function preview, in some situations.
At 914, the method includes determining if a change in the posture of the precursory user-input preaction has occurred. In this way, a user may alter the posture of the precursory user-input preaction based on the predictive input cue. In other words, a user may view the predictive input cue, determine that the suggested input is not intended, and alter the precursory user-input preaction accordingly.
If it is determined that a change in the posture of the precursory user-input preaction has occurred (i.e., YES at 914) the method returns to 902. However, if it is determined that a change in the posture of the precursory user-input preaction has not occurred (i.e., NO at 914) the method includes, at 916, detecting a successive user-input action gesture executed in the functionally-active region.
At 918, the method includes executing a computing function in response to detecting the successive user-input action gesture, the computing function corresponding to the successive user-input action gesture. After 918 the method 900 ends.
The systems and methods for gestural recognition described above allows novice or infrequent users to quickly learn various user-input action gestures through graphical input cues, thereby easing the learning curve corresponding to gestural input and decreasing user frustration.
As described with reference to
Logic subsystem 24 may include one or more physical devices configured to execute one or more instructions. For example, the logic subsystem may be configured to execute one or more instructions that are part of one or more programs, routines, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise arrive at a desired result. The logic subsystem may include one or more processors that are configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The logic subsystem may optionally include individual components that are distributed throughout two or more devices, which may be remotely located in some embodiments. Furthermore the logic subsystem 24 may be in operative communication with the display 12 and the input sensing subsystem 14.
Data-holding subsystem 26 may include one or more physical devices configured to hold data and/or instructions executable by the logic subsystem to implement the herein described methods and processes. When such methods and processes are implemented, the state of Data-holding subsystem 26 may be transformed (e.g., to hold different data). Data-holding subsystem 26 may include removable media and/or built-in devices. Data-holding subsystem 26 may include optical memory devices, semiconductor memory devices, and/or magnetic memory devices, among others. Data-holding subsystem 26 may include devices with one or more of the following characteristics: volatile, nonvolatile, dynamic, static, read/write, read-only, random access, sequential access, location addressable, file addressable, and content addressable. In some embodiments, Logic subsystem 24 and Data-holding subsystem 26 may be integrated into one or more common devices, such as an application specific integrated circuit or a system on a chip.
It is to be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated may be performed in the sequence illustrated, in other sequences, in parallel, or in some cases omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
