This disclosure relates to controlling virtual objects.
Manipulating virtual objects displayed on a display device may include using both stylus and touch inputs. Different types of virtual objects may be displayed and manipulated over an entire canvas of the display device. Accordingly, certain ones of the different types of virtual objects may be inadvertently manipulated. For example, an inadvertent touch input upon the display device, such as a user resting their palm upon the display device or a user lifting the stylus from the display device in a hovering position just barely above the display device, may cause an undesired selection and/or manipulation of virtual objects displayed upon the display device. Similarly, an inadvertent touch input upon the display device, such as a user resting a palm or elbow making contact with the display device, may cause an unintended selection and/or manipulation of virtual objects displayed upon the display device.
Techniques are described for controlling the manipulation of displayed virtual objects.
In some implementations, controlling virtual objects displayed on a display device comprises controlling display, on a display device, of multiple virtual objects, each of the multiple virtual objects being capable of movement based on a first type of input and being capable of alteration based on a second type of input that is different than the first type of input, the alteration being different from movement. User interaction relative to the display device on which the multiple virtual objects are displayed is sensed. Positions of the multiple virtual objects on the display device at a time corresponding to the sensed user interaction is determined. A subset of the multiple virtual objects as candidates for restriction based on the sensed user interaction and the determined positions of the multiple virtual objects on the display device at the time corresponding to the sensed user interaction is determined. An operation related to restricting movement of the determined subset of virtual objects based on the first type of input is performed.
In some implementations, performing the operation related to restricting movement of the determined subset of virtual objects based on the first type of input comprises restricting, for the determined subset of virtual objects, movement that occurs based on the first type of input while enabling alteration of the determined subset of virtual objects based on the second type of input, the movement that occurs based on the first type of input for the determined subset of virtual objects being less than movement that occurs based on the first type of input for the multiple virtual objects that are excluded from the determined subset of virtual objects.
In some implementations, sensing user interaction relative to the display device on which the multiple virtual objects are displayed includes sensing a stylus touching the display device and determining a position of the stylus touching the display device.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes comparing the position of the stylus with the determined positions of the multiple virtual objects on the display device, determining that the position of the stylus contacts a first virtual object included in the multiple virtual objects on the display device, based on the comparison, and including the first virtual object in the subset of virtual objects based on the determination that the position of the stylus contacts the first virtual object.
In some implementations, sensing user interaction relative to the display device on which the multiple virtual objects are displayed includes sensing a stylus touching the display device and determining a position of the stylus touching the display device.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes comparing the position of the stylus with the determined positions of the multiple virtual objects on the display device, determining, based on the comparison that the position of the stylus is within a threshold distance of a first virtual object included in the multiple virtual objects on the display device, and including the first virtual object in the subset of virtual objects based on the determination that the position of the stylus is within the threshold distance of the first virtual object.
In some implementations, sensing user interaction relative to the display device on which the multiple virtual objects are displayed includes sensing a stylus hovering over the display device and determining a position of the stylus hovering over the display device.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes comparing the position of the stylus with the determined positions of the multiple virtual objects on the display device, determining, based on the comparison, that the position of the stylus is directly over a first virtual object included in the multiple virtual objects on the display device, and including the first virtual object in the subset of virtual objects based on the determination that the position of the stylus is directly over the first virtual object.
In some implementations, sensing user interaction relative to the display device on which the multiple virtual objects are displayed includes sensing a stylus hovering over the display device and determining a position of the stylus hovering over the display device.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes comparing the position of the stylus with the determined positions of the multiple virtual objects on the display device, determining, based on the comparison, that the position of the stylus is within a threshold distance of a first virtual object included in the multiple virtual objects on the display device, and including the first virtual object in the subset of virtual objects based on the determination that the position of the stylus is within the threshold distance of the first virtual object.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes determining a location of a designated alteration area defined over a portion of less than all of the display device, comparing the location of the designated alteration area with the determined positions of the multiple virtual objects on the display device, determining, based on the comparison, that a first virtual object included in the multiple virtual objects is positioned within the designated alteration area, and including the first virtual object in the subset of virtual objects based on the determination that the first virtual object is positioned within the designated alteration area.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes determining that a first virtual object included in the multiple virtual objects has a hierarchical relationship with a second virtual object included in the determined subset of the multiple virtual objects identified as candidates for restriction, and including the first virtual object in the subset of virtual objects based on the determination that the first virtual object has the hierarchical relationship with the second virtual object.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes determining that movement of a first virtual object included in the multiple virtual objects results in movement of a second virtual object included in the determined subset of the multiple virtual objects identified as candidates for restriction, an including the first virtual object in the subset of virtual objects based on the determination that movement of the first virtual object results in movement of the second virtual object.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes determining positions of the multiple virtual objects relative to the sensed user interaction.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes determining that a user is in an alteration position based on the sensed user interaction, and determining a subset of the multiple virtual objects as candidates for restriction based on the determination that the user is in the alteration position.
In some implementations, determining a subset of the multiple virtual objects as candidates for restriction includes determining an object type of at least one of the multiple virtual objects, and determining whether to include the at least one of the multiple virtual objects in the subset of the multiple virtual objects based on the object type of the at least one of the multiple virtual objects.
In some implementations, determining the subset of the multiple virtual objects as candidates for restriction comprises computing, for the multiple virtual objects, scores based on the sensed user interaction and the determined positions, accessing one or more thresholds, comparing the computed scores to the one or more thresholds, and determining the subset of the multiple virtual objects as candidates for restriction based on the comparison of the computed scores to the one or more thresholds.
In some implementations, computing, for the multiple virtual objects, scores based on the sensed user interaction and the determined positions comprises computing a score for a particular virtual object included in the multiple virtual objects by determining a location of the particular virtual object on the display device relative to a designated alteration area that spans a portion of less than all of the display device. determining a location of the particular virtual object on the display device relative to a stylus, determining a location of the particular virtual object on the display device relative to touch input sensed by a detection unit associated with the display device, determining whether the touch input sensed by the detection unit associated with the display device indicates that a user is in a position for performing alteration based on the second type of input, determining a type of the particular virtual object, determining a relationship of the particular virtual object to any other virtual objects identified as candidates for restriction, and computing, for the particular virtual object, the score based on the location of the particular virtual object on the display device relative to the designated alteration area, the location of the particular virtual object on the display device relative to the stylus, the location of the particular virtual object on the display device relative to the touch input, the determination of whether the touch input indicates that the user is in the position for performing alteration, the type of the particular virtual object, and the relationship of the particular virtual object to any other virtual objects identified as candidates for restriction.
In some implementations, accessing one or more thresholds comprises accessing multiple thresholds that define at least three levels of restriction.
In some implementations, comparing the computed scores to the one or more thresholds comprises comparing the computed scores to the multiple thresholds that define at least three levels of restriction.
In some implementations, determining the subset of the multiple virtual objects as candidates for restriction based on the comparison of the computed scores to the one or more thresholds comprises setting, for each of the multiple virtual objects and from among the at least three levels of restriction, a restriction level based on the comparison of the computed scores to the multiple thresholds.
In some implementations, performing the operation related to restricting movement of the determined subset of virtual objects based on the first type of input includes performing different restriction actions for different restriction levels.
In some implementations, the first type of input is touch input from a user's hand and the second type of input is input from stylus.
In some implementations, the multiple virtual objects include virtual pages that are altered by being written upon based on the second type of user input.
In some implementations, a system, comprises one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising controlling display, on a display device, of multiple virtual objects, each of the multiple virtual objects being capable of movement based on a first type of input and being capable of alteration based on a second type of input that is different than the first type of input, the alteration being different from movement. User interaction relative to the display device on which the multiple virtual objects are displayed is sensed. Positions of the multiple virtual objects on the display device at a time corresponding to the sensed user interaction is determined. A subset of the multiple virtual objects as candidates for restriction based on the sensed user interaction and the determined positions of the multiple virtual objects on the display device at the time corresponding to the sensed user interaction is determined. An operation related to restricting movement of the determined subset of virtual objects based on the first type of input is performed.
In some implementations, a non-transitory computer-readable storage medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising controlling display, on a display device, of multiple virtual objects, each of the multiple virtual objects being capable of movement based on a first type of input and being capable of alteration based on a second type of input that is different than the first type of input, the alteration being different from movement. User interaction relative to the display device on which the multiple virtual objects are displayed is sensed. Positions of the multiple virtual objects on the display device at a time corresponding to the sensed user interaction is determined. A subset of the multiple virtual objects as candidates for restriction based on the sensed user interaction and the determined positions of the multiple virtual objects on the display device at the time corresponding to the sensed user interaction is determined. An operation related to restricting movement of the determined subset of virtual objects based on the first type of input is performed.
Display devices, such as contact-sensitive display devices, make use of contact input in order to manipulate objects, e.g., virtual objects, displayed on the display devices. In some implementations, a first type of input, such as input from a stylus, is used to provide the contact input to the display device for altering the virtual objects (e.g., writing on, reshaping, etc.). In these implementations, a second type of input, such as a user's touch, is used to provide the contact input to the display device for moving (e.g., translating, rotating, etc.) or updating a view (e.g., zooming, scrolling, etc.) of the virtual objects. Techniques are described throughout this disclosure for controlling virtual objects to appropriately respond to both the first type of input and the second type of input.
As shown in
In addition to altering the first virtual object 160 based on the stylus input, the display system 100 restricts movement of (e.g., locks) the first virtual object 160 based on detection of the stylus input corresponding to a location of the first virtual object 160 at the display device using the stylus held in the user's right hand 140. By restricting movement of the first virtual object 160 within the canvas 130, the display system 100 limits the ability of the first virtual object 160 to be manipulated (e.g., translated, rotated, scaled, etc.) in response to touch input. For instance, the display system 100 may lock the first virtual object 160 such that, while the stylus input is detected, the first virtual object 160 does not respond to touch input and remains stationary regardless of the touch input provided. By locking the first virtual object 160, the display system 100 may enhance the ability of the user to alter the first virtual object 160 using the stylus because the first virtual object 160 remains stationary for inadvertent or incidental touch contacts on the first virtual object 160 that otherwise would result in movement of the first virtual object 160 and, potentially, inadvertent or stray marks from the stylus.
In typical operation, the display system 100 would move both the first virtual object 160 and the second virtual object 170 based on detecting the swiping motion. However, because the display system 100 previously detected the stylus input on the first virtual object 160 and restricted movement of the first virtual object 160 (e.g., locked the first virtual object 160) based on detection of the stylus input, the display system 100 handles the detected swiping motion by leaving the first virtual object 160 stationary while moving the second virtual object 170 away from the first virtual object 160. In this regard, the display system 100 does not disrupt the user's ability to alter the first virtual object 160 using the stylus and allows the user to provide touch input to perform the desired action of sliding the second virtual object 170 away from the first virtual object 160. Accordingly, the display system 100 may intelligently handle the combined stylus and touch inputs shown in
Specifically and as described in more detail below, the detector 220 of the system 200 includes a touch detector module 224 configured to identify touch inputs based on the engagement data produced by the sensor 210. Touch inputs are regions of contiguous user contact (or engagement) on the detector 212. In some implementations, a touch input may be represented by data identifying a subset of the detection points of the detector 212, where each of the detection points in the subset is a point that is in contact (or engaged) with a user input mechanism (i.e., user contact) and is positioned in the detector 212 adjacent to another such contacted point in the subset (i.e., contiguous).
The system 200 includes a candidate object identification (ID) module 230 configured to identify virtual objects as being possible candidates for restriction. In some implementations, virtual objects are identified as candidates for restriction (from manipulation, or movement) based upon the output from the detector module 220 indicating that the contact input received by the sensor 210 was from a stylus. Conversely, virtual objects may not be identified as candidates for restriction based upon the output from the detector module 220 indicating that the contact input received by the sensor 210 was from a touch. In some implementations, the candidate object ID module 230 may be selectively configured to identify virtual objects as being candidates for restriction based upon determining, by the detector module 220, that the contact input received by the sensor 210 was from a stylus or from a touch.
The candidate object ID module 230 is further configured to access the location of stylus inputs (either contact input or hover input), access the location of touch inputs, and determine positions of the virtual objects displayed on the display device. The candidate object ID module 230 uses the location of the stylus inputs, the location of the touch inputs, and the positions of the virtual objects to identify one or more (e.g., a subset) of the virtual objects as candidates for restriction. In some implementations, the candidate object ID module 230 may determine (or access) the positions of the virtual objects from the application generating the virtual objects displayed on the display device.
The movement restriction module 240 is configured to receive output from the candidate object ID module 230 and to restrict movement of the one or more candidate objects identified by the candidate object ID module 230. In some implementations, as detailed below, the restriction upon the one or more candidate objects identified by the candidate object ID module 230 may be variable.
In some examples, the restriction upon movement of the candidate object identified by the candidate object ID module 230 is relatively high, e.g., no substantial movement. Conversely, the restriction upon movement of the candidate object identified by the candidate object ID module 230 is relatively low, e.g., substantial movement still allowed. In addition, the restriction upon movement of the candidate object identified by the candidate object ID module 230 may be relatively medium, e.g., the restriction upon movement is between relatively high restriction upon movement and relatively low restriction upon movement.
Each of the components of system 200 will now be described in more detail. As noted previously, the sensor 210 may receive input from the user through the user contacting, touching, or, more generally, engaging the detector 212.
The detector 212 may be made up of an array of detection points distributed over a two-dimensional sensing area. In some implementations, the detection points are evenly distributed in two dimensions over the sensing area. In this instance, the detector 212 may be shaped like a square or a rectangle, depending on whether the number of horizontal detection points equals the number of vertical detection points. In other implementations, the detector 212 may be shaped like a parallelogram. Each detection point in the detector 212 may receive input directly from one or more user input mechanisms (e.g., a user finger and/or a stylus) interacting with the multi-touch display. Detection points can be, for example, capacitive-based or FTIR-based. Measurements may be generated for each detection point, and, in some implementations, a measurement value may be produced for each detection point in the detector 212, irrespective of whether that detection point is or is not currently being engaged by the user. Notably, when there is no user engagement with a detection point, the measurement value for the detection point may be set to a zero value or to whatever value has been assigned by the system 200 as corresponding to no touch (i.e., a baseline value). As previously stated, in some implementations, direct physical contact between an input mechanism and the detector 212 may not be required to generate a touch. For example, in some implementations, if an input mechanism is hovering over the detector 212 (e.g., within 10 mm of the surface of the detector 212), the sensor 210 may detect a small change in capacitance or other measurable characteristic of the detector 212 at the detection points under the hovering input mechanism and may identify the existence of a touch by the input mechanism at the location of these detection points based on this detected small change.
In one implementation, the sensor 210 may be a capacitive touch sensor that includes circuitry to determine the locations of user inputs on the detector 212 by detecting changes in conductor capacitances caused by the one or more input mechanisms used by the user to engage the detector 212. The detector 212 may, for example, be a sensor matrix made up of a first group of conductors that are positioned substantially parallel to each other and a second group of conductors that are positioned substantially parallel to each other, where each of the conductors in the first array is positioned to intersect with every conductor in the second array and, vice versa, to thereby form a matrix. Each of the intersections between a conductor in the first array and a conductor in the second array may be a detection point of the detector 212. Through use of interpolation techniques, the number of detection points in the detector 212 may exceed the number of conductor intersections and the pitch of the detection points may be greater than the pitch of the conductor intersections.
The one or more user input mechanisms may, for example, be fingers or styluses and may engage the detector 212 by directly physically contacting one or more of the conductors in the sensor matrix or by otherwise hovering in close physical proximity to the one or more conductors (e.g., within 10 mm from the surface of the one or more conductors) such that a detectable change in capacitance at the detection points of the detector 212 occurs. For example, a user may engage the detector 212 by positioning a finger in close physical proximity to the detector 212, thereby causing the capacitance at the detection points of the detector 212 that are located at the center of the user's finger to deviate from a baseline value by, for example, 10 femtofarads for the detection points located at the center of the user's finger and 1 femtofarad for the detection points located at the edge of the user's finger.
The sensor 210 may include circuitry configured to generate and output a raw bitmap of measurement values associated with the user's input. In some implementations, the raw bitmap includes a measurement value (e.g., a value indicative of a measured capacitance) for each detection point in the detector 212. As noted above, a baseline measurement value (e.g., zero or a predetermined number greater than zero) may be included in the raw bitmap for each of the detection points that is not currently being engaged/touched by the user. Measurement values deviating from this baseline value by a predetermined amount may indicate detection of a user engagement at the corresponding detection point. For example, the sensor 210 may detect user engagement in the middle of the detector 212, and may output a corresponding bitmap array that includes baseline measurement values for the detection points located at the edge of the detector 212 that are not currently being engaged by the user and additionally includes values that deviate from the baseline values for the detection points located at the middle of the detector 212 that are currently being engaged by the user. In other implementations, the sensor 210 only provides data for those detection points having measurement values that deviate from the baseline values by the predetermined amount. As described in further detail below, the raw bitmap data may be used to identify the location or locations in the detector 212 where the user has touched or engaged the multi-touch display purposefully (i.e., with the intention of using the engagement or touch to control the multi-touch display application) and the location or locations in the detector 212 where the user has touched or engaged the multi-touch display incidentally (i.e., without the intention of using the engagement or touch to control the multi-touch display application).
In some implementations, the sensor 210 may include multiple detectors 212. For example, the sensor 210 may include a first detection grid to detect capacitive objects which contact or hover near the sensor 210 and may further include a second detection grid to detect an active (or passive) stylus. In this instance, the second detection grid may be designed to sense the active (or passive) stylus and thus output the location of the active (or passive) stylus. The first detection grid may operate independently of the second grid such that its operation remains identical irrespective of whether the second grid is included or is not included in the sensor 210. The first grid may sense touches, for example, with human skin and other conductive items when using capacitive sensing and with any physical object when using FTIR sensing.
The stylus detector module 222 is configured to detect the location of a stylus (or, more particularly, a stylus tip) on the detector 212. As previously discussed, this sensing of the stylus can either be accomplished through use of a second detection grid that is dedicated to identifying an active stylus or can be accomplished through use of a single detection grid that detects a passive stylus. In the implementation involving the passive stylus, the stylus detector analyzes the bitmap array output from the sensor 210 and identifies the location of the stylus by identifying a blob indicated by the bitmap array that closely matches the precise shape of the stylus tip. For example, in the case of a capacitive sensing sensor 210, the passive stylus may produce a blob having a small, round shape having dimensions roughly corresponding to the metal tip of the stylus. Notably, because the shape of the tip of the stylus is fixed and the material used to form the tip of the stylus is also fixed, the stylus produces a predictable change in capacitance when engaging the detector 212 and, therefore, is more easily detected and distinguished from other types of touches/contacts on the detector 212. The stylus detector module 222 may analyze the bitmap array and may identify one or more blobs that exhibit the characteristics known to correspond to a stylus touch. In the case of an FTIR sensor 210, the passive stylus also likely has a small round tip. The stylus detector module 212 may analyze the bitmap array and may determine the location of the stylus by identifying a blob having a shape that matches the shape of the small round tip. In the implementation that uses an active stylus, the second detection grid may output its own bitmap array (or portion of a bitmap array), and the stylus detector module 222 may analyze the second bitmap array to identify a touch that corresponds to the location of the tip of the active stylus. In some implementations, the bitmap array generated by the second detection grid does not identify or otherwise include data corresponding to touches other than the active stylus touch.
The touch detector module 224 is configured to identify touches, i.e., contiguous area of contact (or, more generally, engagement), based on the data in the bitmap array produced by the sensor 210. One technique that the touch detector module 224 may use to identify contiguous areas of contact (or, more generally, engagement) is touch tracking. For example, if a user places a hand, one fingertip, and a forearm on the sensor 210, then the touch detector module 224 may identify three different contiguous contact areas or touches, one for the fingertip, one for the palm region of the hand, and one for the forearm. In implementations in which a passive stylus is used, the touch detector module 224 also may identify a contiguous contact area as corresponding to the passive stylus. The touch detector module 224 may identify the contiguous contact areas by analyzing the bitmap array to identify detection points having measurements above a certain threshold. In some implementations, the threshold is zero. In other implementations, the threshold may be a non-zero baseline value. Each contiguous area or touch may be given a distinguishing label and may be packaged as data that identifies a particular subset of the detection points of the detector 212, where each detection point corresponds to a point of contact and is adjacent to each other detection point within the subset. For example, if there are three contiguous areas identified by the touch detector module 224, then one may include 153 detection points and may be labeled “A,” the second may include 53 detection points and may be labeled “B,” and the third may include 640 detection points and may be labeled “C.” By processing the touches further using the touch detector module 224, the system 200 may determine which touches likely correspond to resting body part inputs and which touches likely correspond to pointing inputs.
In some implementations, the system 200 may analyze the values in the bitmap array received from the sensor 210 to distinguish a relatively large and irregular contiguous contact area that results from one or more fingertips engaging the detector 212 from a relatively large and irregular contiguous contact area that instead results from a resting arm or palm engaging the detector 212. For example, the touch detector module 224 may conclude that a large and irregularly shaped touch is not a resting body part input (e.g., an input arising from the resting of a palm, a forearm, an elbow or another relatively large body part) by comparing the values in the bitmap array corresponding to the area to a threshold value. Since the purposeful contact made with a pointing input (e.g., an input made by a fingertip) on the detector 212 is likely firmer (e.g., exhibits greater pressure or weight over more of its contact area) than that made by a resting body part input, the capacitance change generated by the pointing input may be greater than that generated by the unpurposeful resting body part input. As such, if the values corresponding to the blob are greater than a threshold value, for example, the touch detector module 224 may conclude that the touch does not correspond to a resting body part input, despite its irregular and larger shape. Conversely, if the values corresponding to the touch are less than a threshold value, then the touch detector module 224 may conclude that the touch corresponds to a resting body part input. In some implementations, a total capacitance corresponding to the blob is sensed by the detector 212. In these implementations, the total capacitance over an entire contact area of the blob may be generally higher for larger, but weaker, areas of the resting body part input. For example, measuring the total capacitance corresponding to the blob provides for a pressure threshold versus force threshold and may assist in identifying the resting body part input.
In some implementations, the system 200 is configured to restrict movement of virtual objects based upon a type of input received at the sensor 210. For example, the system 200 may be configured to identify and restrict virtual objects based upon determining, by the detector module 220, whether the input (either contact or hovering) is from a stylus. If the input is determined, by the detector module 220, to be from a stylus (either contacting or hovering), then the candidate object ID module 230 can identify that the virtual object located beneath the stylus input to be included in a subset of the virtual objects as candidates for movement restriction. In addition, the detector module 220 can detect a time-based window, such as less than one second, in which the stylus has ceased to provide the contact/hovering input. In this way, the candidate object ID module 230 can identify that the virtual object located beneath the stylus during the time-based window to be included in a subset of the virtual objects as candidates for movement restriction.
In some examples, the system 200 is configured to restrict movement of virtual objects based upon a location of the stylus input (either contact or hovering) with respect to locations of the virtual objects. In these examples, distances from the stylus input to the multiple virtual objects displayed within the canvas are determined and compared to a threshold. Then, based upon the comparison, a subset of the multiple virtual objects that are located within the threshold of the stylus are determined as candidates for movement restriction.
In some implementations, the system 200 is configured to determine a designated alteration area corresponding to the subset of the multiple virtual objects selected as candidates for movement restriction. For example, upon detection of a stylus, the subset of virtual objects displayed within the designated alteration area may be subject to locking (e.g., high-level movement restrictions), whereas objects outside of the designated alteration area may be subject to no movement restrictions. In addition, some or all of the objects outside of the designated alteration area may be subject to the same movement restrictions as those of the subset of objects displayed within the designated alteration area. Furthermore, some of the subset of virtual objects display within the designated alteration area may be subject to limited movement restrictions (e.g., mid-level movement restrictions), as compared to some other of the subset of virtual objects displayed within the designated alteration area. Accordingly, when a user is interacting with a virtual object using a stylus, such as editing a word processing document, inadvertent touch contact on the virtual object by the user, such as contact made by a palm, does not affect undesired changes to the word processing document. Similarly, inadvertent touch contact by the user on other virtual objects different from than the virtual object being interacted with using the stylus do not affect undesired changes to the other virtual objects.
In some implementations, the system 200 is configured to determine a hierarchical relationship of the subset of the multiple virtual objects corresponding to the subset of the multiple virtual objects selected as candidates for movement restriction. For example, a first grouping of virtual objects of the subset of multiple virtual objects selected as candidates for movement restriction may be subject to a first level of movement restriction based upon a type of virtual object each of the first grouping includes, whereas a second grouping of virtual objects of the subset of multiple virtual objects selected as candidates for movement restriction may be subject to a second level of movement restriction based upon a type of virtual object each of the second grouping includes, which differs from the first level of movement restriction.
Consider a situation in which a virtual page and another virtual object (e.g., a virtual ruler) are positioned on a canvas. In this situation, the canvas has a hierarchical relationship to both the virtual page and the other object (e.g., the virtual ruler) because movement of the canvas necessarily and directly causes movement of both the virtual page and the other object (e.g., the virtual ruler). The other object (e.g., the virtual ruler) has a spatial relationship to the virtual page because movement of the other object (e.g., the virtual ruler) only causes movement of the virtual page when the movement of the other object (e.g., the virtual ruler) causes the other object (e.g., the virtual ruler) to contact the virtual page. Based on the hierarchical relationship of the canvas to the virtual page and the spatial relationship of the other object (e.g., the virtual ruler) to the virtual page, the system 200 may perform a first type of movement restriction to the canvas and a second type of movement restriction to the other object (e.g., the virtual ruler) when the system 200 identifies the virtual page as a candidate for restriction. For example, the system 200 may determine to lock the virtual page based on detection of a stylus writing on the virtual page. In this example, the system 200 also may detect the hierarchical relationship of the canvas to the virtual page and also lock the canvas based on the hierarchical relationship. In addition, the system 200 may detect the spatial relationship of the other object (e.g., the virtual ruler) to the virtual page and, when the other object (e.g., the virtual ruler) is within a threshold distance of the virtual page, increase inertia (e.g., reduce the responsiveness to touch input without fully restricting movement) of the other object (e.g., the virtual ruler) based on the spatial relationship.
In some implementations, although the subset of multiple virtual objects have been selected as candidates for movement restriction, one or more of the subset of multiple virtual objects may have different movement restrictions based upon a relative location from an input to the display device. For example, one or more of the subset of multiple virtual objects may have a first movement restriction based upon a first relative location from a stylus input, whereas one or more of the subset of multiple virtual objects may have a second movement restriction based upon a second relative location from a touch input.
In some implementations, the system 200 is configured to determine positions of each of the multiple virtual objects relative to the stylus input corresponding to the subset of the multiple virtual objects selected as candidates for movement restriction. For example, when a user provides a stylus input in one region of a designated alteration area on the display device, virtual objects in another region of the designated alteration area can be restricted from movement. By determining positions of each of the multiple virtual objects relative to the stylus input, movement restrictions to the virtual objects can be determined based upon a threshold distance from the stylus input.
In some implementations, the system 200 is configured to determine object types of the multiple virtual objects corresponding to the subset of the multiple virtual objects selected as candidates for movement restriction. For example, movement of certain virtual objects, such as word processing documents, can be restricted at a first level, whereas movement of other virtual objects, such as digital photographs, can be restricted at a second level. By doing so, object types of the virtual objects can be used to selectively restrict movement.
The system 200 identifies a subset of the multiple virtual objects as candidates for restriction (320). In some implementations, the system 200 identifies the candidates for restriction based on explicit user input to select the candidates. In addition, the system 200 may identify the subset as candidates for restrictions based on a detected user input and a state of the system.
The system 200 senses a user's interaction relative to a display device (410). In some implementations, the user's interaction includes providing input using a stylus or body part (either contact or hovering) at a location on the display device corresponding to one or more virtual objects. Next, the system 200 determines positions of multiple virtual objects displayed on the display device (420). In some implementations, the system 200 determines positions of the multiple virtual objects displayed on the display device by accessing location information from the application which generates the multiple virtual objects. Then, based upon the sensed user's interaction relative to the display device and the determined positions of the multiple virtual objects, the system 200 determines a subset of the multiple virtual objects as candidates for restrictions (430). For example, the system 200 may compare a location the sensed user interaction to the positions of the multiple virtual objects and determine the subset of the multiple virtual objects based on the comparison. In this example, the sensed user interaction may be a stylus input and the system 200 may identify a virtual object that is being contacted by the stylus or that is directly under a hovering stylus as a candidate for restriction. In addition, the system 200 may evaluate the positions of the multiple virtual objects relative to the position of the stylus input and identify those virtual objects that are positioned within a threshold distance of the stylus input as candidates for restriction.
In some examples, the sensed user interaction may be touch input. In these examples, the system 200 may evaluate the type of touch input provided and determine whether the touch input indicates that the user is entering a position for altering a virtual object. The system 200 may detect the user entering a position for altering a virtual object based on detecting the user resting forearms on the sensor 210 in an orientation that is typical of a user that is altering a virtual object (e.g., detecting the user entering a writing pose). Based on a determination that the touch input indicates that the user is not entering a position for altering a virtual object, the system 200 may not restrict movement of the multiple virtual objects and may continue monitoring sensed user interactions relative to positions of the multiple virtual objects. Based on a determination that the touch input indicates that the user is entering a position for altering a virtual object, the system 200 may evaluate the positions of the multiple virtual objects relative to the position of the touch input and identify virtual objects as candidates for restriction based on the evaluation. For instance, the system 200 may identify those virtual objects that are located in an area where a user would typically alter a virtual object when oriented in the detected position of the touch input. When the system 200 detects the user entering a writing pose, the system 200 may identify a virtual object as a candidate for restriction when the virtual object is located centrally between two resting forearms.
In
In some implementations, locking of the subset (the virtual objects 540 and 544 and the canvas 520) of the multiple virtual objects 520, 540, 544, 550, 560, and 570 allows for movement of the virtual objects 550, 560, and 570, but does not allow movement (or greatly restricts movement) of the subset of the multiple virtual objects. For example, while the user's interaction 530 is maintained with respect to the virtual object 540, touch input provided to the virtual objects 550, 560, and 570 causes movement of the virtual objects 550, 560, and 570, but touch input provided to the virtual objects 540 and 544 and the canvas 520 does not cause movement of the virtual objects 540 and 544 and the canvas 520.
In
In some examples, the system 600 detects that the user's interaction 630 is located between virtual objects 640 and 670 and each could potentially be candidates for restriction. However, because the system 600 detects that the virtual object 640 is located within a designated writing area 660 and that the virtual object 670 is outside of the designated writing area 660, the system 600 highlights the virtual object 640 with a highlighted display element 650 indicating that the user is preparing to modify the virtual object 640 and does not highlight the virtual object 670. In some implementations, the designated writing area 660 is determined by an application which is generating the virtual objects 640 and 670. For example, a graphical document application generating the virtual objects 640 and 670 designates a central region of the canvas 620 as the designated writing area 660.
In the example shown in
In this instance, the detector module 220 of the system 200 (in
In
The system 200 makes a determination regarding the location of the particular virtual object displayed on the display device relative to a location of a stylus input (720). In some implementations, the stylus input includes a contact stylus input, and in other implementations, the stylus input includes a hovering stylus input. The system 200 may determine whether the particular virtual object is located directly under the stylus input or within a threshold distance of the stylus input.
The system 200 makes a determination regarding the location of the particular virtual object displayed on the display device relative to a touch input sensed by a detection unit associated with the display device (730). In some implementations, the touch input is provided as an input distinct from a single fingertip touch input. For example, a grouped three- or four-fingertip contact input can be provided adjacent to (or at a predetermined distance from) a particular virtual object. In this example, the system 200 determines whether the position of the particular virtual object relative to the grouped three- or four-fingertip contact input suggests that the particular virtual object is about to be altered.
The system 200 makes a determination whether the touch input sensed by the detection unit associated with the display device indicates that a user is in a position for performing alterations to the particular virtual object based on a second type of input (740). In some implementations, the determination is made that touch input is provided as a result of the user resting his or her forearm or elbow adjacent to (or at a predetermined distance from) the particular virtual object, e.g., in a position to write (or modify) the virtual object.
The system 200 makes a determination regarding the type of the particular virtual object (750). In some implementations, the system 200 determines that the particular virtual object is an editable word processing document, which is frequently altered, or determines that the particular virtual object is a read only word processing document, which is infrequently altered. In some examples, the system 200 determines that the particular virtual object is a digital photograph or a legal document, which is unalterable.
The system 200 makes a determination regarding a relationship of the particular virtual object with respect to other virtual objects that have been identified as candidates for restriction (760). In some implementations, determines whether the particular virtual object is largely related to the other virtual objects that have been identified as candidates for restriction, largely unrelated to the other virtual objects that have been identified as candidates for restriction, or slightly related to the other virtual objects that have been identified as candidates for restriction. The system 200 may determine the relationship in terms of type, kind, spacing, and/or format as the other virtual objects that have been identified as candidates for restriction. The system 200 also may determine whether the particular virtual object has a hierarchical relationship (e.g., parent or child) to another virtual object identified as a candidate for restriction. The system 200 further may determine the relationship based on how movement of the particular virtual object results in movement of the other objects identified as candidates for restriction.
The system 200 determines a score of the particular virtual object (770). In some implementations, scoring of the particular virtual object may be based upon different objectives. For example, some objectives for scoring the particular virtual objects includes object type, a location of the particular virtual object displayed on the display device relative to the designated alteration area, a location of the particular virtual object displayed on the display device with respect to the stylus, a location of the particular virtual object displayed on the display device relative to the touch input, the determination of whether the touch input indicates that the user is in a position for performing alteration, the type of the particular virtual object, and the relationship of the particular virtual object to any other virtual objects identified as candidates for restriction.
The system 200 makes a comparison between the scoring of the particular virtual object and a scoring threshold (780). In some implementations, the system 200 accesses multiple thresholds that define at least three levels of restriction. For example, three distinct scoring threshold ranges may be established in which each scoring threshold range is equally distributed. In another example, three distinct scoring threshold ranges may be established in which each scoring threshold range is weighted one more than another.
In some implementations, the scoring threshold comprises only a single, absolute range. For example, a single scoring threshold range may be established in which the scoring of the virtual object is either above or below the scoring threshold range. In some examples, comparing the computed scores to the thresholds includes comparing the computed scores to multiple thresholds that define at least three levels of restriction. In some implementations, determining the subset of the multiple virtual objects as candidates for restriction based on the comparison of the computed scores to the thresholds includes setting, for each of the multiple virtual objects and from among the at least three levels of restriction, a restriction level based on the comparison of the computed scores to the thresholds.
The system 200 sets a restriction level for the virtual object (790). In some implementations, the restriction level is set to indicate that a high level of restriction is placed upon the virtual object. For example, restriction levels of certain virtual objects can be changed en masse without the need to rescore individual virtual objects. In addition, the restriction level may be user selectable based upon the different types of virtual objects that are being altered. In some implementations, different restriction actions are taken for different restriction levels.
In
In some implementations, scorings is related to input type and input position. For example, scorings range from +0.2 (corresponding to touch inputs indicating that a user is in an alteration position) to −0.1 (corresponding to touch inputs that do not indicate that a user is in an alteration position). In addition, scorings is related to object location with respect to a stylus. For example, scorings range from +10 (corresponding to virtual objects touched by a stylus) to +0.8 (corresponding to virtual objects located directly beneath a hovering stylus) to +0.4 (corresponding to virtual objects within a threshold distance of a hovering stylus) to +0.3 (corresponding to virtual objects within a threshold distance of a touching stylus).
In some examples, scorings is based on hierarchical relationship to candidates for restriction. For example, scoring of a virtual object hierarchically related to a restriction candidate may be substantially the same as the score of the restriction candidate. Further, scoring of a virtual object unrelated to a restriction candidate is 0.
In some implementations, scorings are related to positions of virtual objects relative to touch inputs. For example, scoring of location of a virtual object relative to touch inputs suggesting that the user is in a position for altering the virtual object is +0.2, where the location of the virtual is centered between adjacent touch inputs, e.g., the virtual is located between touch contacts corresponding to the user resting his or her forearms or elbows on opposing sides of the virtual object. In another example, scoring of location of a virtual object relative to touch inputs suggesting that the user is not in a position for altering the virtual object is −0.2, where the location of the virtual is not between adjacent touch inputs, e.g., the virtual object is not located between touch contacts corresponding to the user resting his or her forearms or elbows on opposing sides of the virtual object, or the virtual object is very close to the one or more of the touch inputs. In actual implementations, the scoring system described may make use of machine learning based classifiers where various parameters would be determined by training the system with examples.
In
In using the example scoring weights and threshold scores shown in
In
In
In
In the example shown in
In some implementations, the virtual object 972 is determined to have a medium restriction level. Like the virtual object 971, the virtual object 972 is within the threshold distance 950 from the stylus 940 and is an object, such as a word processing document, that is frequently altered. However, the virtual object 972 is outside of the alteration area 920. Accordingly, the virtual object 972 is considered to have a medium restriction score that meets a medium restriction threshold.
The virtual object 973 is determined to have a low (e.g., no) restriction level. Although the virtual object 973 is positioned in the alteration area 920, the virtual object 973 is considered to have a low restriction score that meets a low restriction threshold because the virtual object 973 is a read-only type of virtual object, such as a legal document, that cannot be altered.
The virtual object 974 is determined to have a medium restriction level. Although the virtual object 974 is outside of the threshold distance 950 of the stylus 940, the virtual object 974 is within the alteration area 920 and is an object, such as a word processing document, that is frequently altered. Accordingly, the virtual object 974 is considered to have a medium restriction score that meets a medium restriction threshold.
The virtual object 975 is determined to have a low (e.g., no) restriction level. The virtual object 975 is outside of the threshold distance 950 of the stylus 940, is outside of the alteration area 920, and away from the alteration positions 960a and 960b. Accordingly, although the virtual object 975 is an object, such as a word processing document, that is frequently altered, the virtual object 975 is considered to have a low restriction score.
The virtual object 976 is determined to have a low (e.g., no) restriction level. Although the virtual object 976 is positioned in the alteration area 920 and is an object, such as a word processing document, that is frequently altered, the virtual object 976 is outside of the threshold distance 950 of the stylus 940, and is located at a position relative to the alteration positions 960a and 960b that suggests the virtual object 976 is unlikely to be written upon by the stylus 940 and unlikely to be dragged to a position to be written upon by the stylus 940 (e.g., outside of the alteration positions 960a and 960b with a direct path to the center of the alteration positions 960a and 960b that is blocked by the alteration position 960b). Accordingly, the virtual object 976 is considered to have a low restriction score that meets a low restriction threshold.
Referring again to
The system 200 detects the first type of input applied to a first virtual object included in the subset and a second virtual object excluded from the subset of virtual objects (1020). In some implementations, detecting the first type of input includes use of a sensor and a detector module associated with the display device to detect touch input that is applied across the first virtual object and the second virtual object.
The system 200 controls movement of the first virtual object in accordance with a restricted responsiveness to the first type of input (1030). In some implementations, restricting, for the subset of virtual objects, responsiveness to the first type of input (e.g., touch input) includes locking the subset of virtual objects such that the subset of virtual objects are unresponsive to the first type of input. In these implementations, the system 200 ignores the first type of input applied to the first virtual object and maintains a position of the first virtual object despite the first type of input applied to the first virtual object. In other implementations, the system 200 may increase the inertia of the first virtual object such that the first virtual object still responds to the first type of input (e.g., touch input), but does not move as much in response to the first type of input.
The system 200 controls movement of the second virtual object without restriction (1040). For instance, because the second virtual object is excluded from the subset of virtual objects identified as candidates for restriction, the system 200 has not placed a restriction on movement of the second virtual object and the second virtual object moves normally (e.g., has a higher responsiveness to touch input than the first virtual object). Movement of the second virtual object may include translation, rotation, and scaling.
In
The system 200 determines a type of application which is generating display of virtual objects on the display device (1210). In some implementations, applications may generate specific types of virtual objects that are more conducive for alteration than other types of virtual objects. For example, graphical applications generate virtual objects that are frequently altered. By contrast, archival applications, such as genealogical record storage applications, generate virtual objects that may be read-only objects, wherein alterations are not allowed. Accordingly, the type of application may be useful in determining how to restrict movement of virtual objects under certain circumstances.
The system 200 determines a type of virtual object displayed on the display device (1220). In some implementations, the virtual object is an alterable or a non-alterable virtual object. For example, if the virtual object is alterable, then the type of virtual object is of a first type of virtual object. Conversely, if the virtual object is unalterable, then the type of virtual object is of a second type of virtual object. Similarly, if the virtual object is alterable, but infrequently altered, then the type of virtual object is of a third type of virtual object. The type of virtual object may be useful in determining how to restrict movement of virtual objects under certain circumstances.
The system 200 determines a location of the virtual object relative to a designated alteration area and to other virtual objects (1230). In some implementations, an area is designated in which virtual objects are altered. For example, a display device has a designated region in which virtual objects are brought into for the purpose of inputting, modifying, outputting, or removing information. In some instances, virtual objects within the designated alteration area are restricted more severely than virtual objects outside of the designated alteration area.
In addition, a location of the virtual object relative to other virtual objects may be useful in determining how to restrict movement of the virtual object. For instance, if the virtual object is positioned adjacent to another object that is being written on and contact between the virtual object and the other object causes movement of the other object, the system 200 may determine to restrict movement of the virtual object in the direction toward the other object, but not in other directions.
The system 200 determines a location of the virtual object relative to a stylus (1240). In some implementations, location of the virtual object relative to a stylus includes a location of a contacting stylus or a location of a hovering stylus. For example, the contacting stylus provides contacting input to a detector associated with the display device or the hovering stylus provides indirect input to the detector associated with the display device. The location of the virtual object relative to the stylus may be useful in determining how to restrict movement of the virtual object in terms of severity and types of motion constraints applied.
The system 200 determines location of the virtual object relative to a touch input (1250). The touch input may be a point touch provided by a user's finger or may be an incidental touch provided by a resting body part of the user (e.g., a palm or forearm). In some examples, the location of the virtual object relative to a stylus is determined prior to determining the location of the virtual object relative to a touch input. For instance, the stylus is used first by a user to provide modifications to the virtual object, and then the touch input is provided by the user, either intentionally or unintentionally. In other examples, the system 200 determines the location of the virtual object relative to the touch input without regard for the location of the virtual object relative to a stylus. The location of the virtual object relative to the touch input may be useful in determining how to restrict movement of the virtual object in terms of severity and types of motion constraints applied.
The system 200 determines whether the touch input indicates that the user is in a position for performing alteration (1260). In some implementations, the touch input results in providing an inadvertent touch input by the user while the user is attempting to alter the virtual object located within the alteration area. For example, the user accidentally makes contact with the detector 212 while altering the virtual object using, for example, the stylus. When a user is in an alteration position, the system 200 may determine to apply a more severe movement restriction, especially for virtual objects located near the alteration position.
The system 200 determines one or more constraints to place on motion of the virtual object for each type of motion that occurs based on the first type of input (1270). In some implementations, the one or more constraints on motion include motion related to translation of the virtual object, rotation of the virtual object, scaling (either up or down) of the virtual object, or combinations of translation, rotation, and/or scaling. For example, constraints on translational motion of the virtual object limit how the virtual object responds to inputs that would result in translation of the virtual object. In this example, the system 200 may damp translational motion of the virtual object, completely suppress translational motion of the virtual object, and apply different constraints for different directions of translational motion (e.g., allow unrestricted translational motion in a first direction, damp translational motion in a second direction, and completely suppress translational motion in a third direction).
In some implementations, constraints on rotational motion of the virtual object limit how the virtual object responds to inputs that would result in rotation of the virtual object. In these implementations, the system 200 may damp rotational motion of the virtual object, completely suppress rotational motion of the virtual object, and apply different constraints for different directions of rotational motion (e.g., damp rotational motion in a first direction and completely suppress rotational motion in a second direction).
In some examples, constraints on scaling motion of the virtual object limit how the virtual object responds to inputs that would result in scaling of the virtual object. In these implementations, the system 200 may damp scaling motion of the virtual object, completely suppress scaling motion of the virtual object, and apply different constraints for different directions of scaling motion (e.g., damp scaling motion in a first direction and completely suppress scaling motion in a second direction).
In this example, the system 1300 has detected a pattern of a user moving a virtual object from the first grouping 1340 to the region X, altering (e.g., signing) the virtual object at the region X, and then moving the virtual object from the region X to the second grouping 1344. Based on the detected pattern, the system 1300 constrains motion of virtual objects of the first grouping of virtual objects 1340 to allow movement only along a leftward direction L1. In addition, the motion of virtual objects of the first grouping of virtual objects 1340 are constrained to be substantially dampened for translational movement along a rightward direction R1 and/or an upward direction U1, and may be slightly dampened for scaling S1 up or down (in a situation where the user may want to review the virtual object in an enlarged format view).
In some implementations, a user selects a topmost one of the virtual objects of the first grouping of virtual objects 1340 by providing input corresponding to a location of a topmost one of the virtual objects of the first grouping of virtual objects 1340 along the leftward direction L1. The topmost one of the virtual objects of the first grouping of virtual objects 1340 is translated along the leftward direction L1 to the region X between the first and second groupings of virtual objects 1340 and 1344, and identified as virtual object 1342.
In some examples, all (or some) motion of the first and second groupings of virtual objects 1340 and 1344 are damped until a threshold condition is met. For example, motion of the first grouping of virtual objects 1340 is substantially damped for directions other than the leftward direction L1. However, once the topmost one of the virtual objects of the first grouping of virtual objects 1340 is translated along the leftward direction L1 into the region X between the first and second groupings of virtual objects 1340 and 1344, thereby meeting a threshold, the virtual object 1342 is not be substantially restricted for modification and/or is not be damped. For example, the virtual object 1342 is signed or edited using a stylus 1330. However, there are constraints to allow movement only along a leftward direction L1 for the virtual object 1342. In addition, the motion of the virtual object 1342 is constrained to be substantially dampened for translational movement along a rightward direction R1 and/or an upward direction U1, is slightly dampened for scaling S1 up or down (in situation where the user wants to review the virtual object in an enlarged format view), and is slightly damped for rotation along first and/or second rotational directions (in situations where the user is more comfortable signing their name at a substantial angle).
In some implementations, the virtual object 1342 is translated along the leftward direction L2. Upon being translated out of the region X, the virtual object 1342 is significantly restricted for modification. For example, the virtual object 1342 is locked for further editing. In addition, there are constraints to restrict movement along a further leftward direction past the leftward direction L2 for the virtual object 1342 once the virtual object 1342 is translated out of the region X. Moreover, the motion of the virtual object 1342 is constrained to be substantially dampened for translational, rotational, and/or scaling movement once the virtual object 1342 is translated out of the region X and onto the second grouping 1344.
The system 200 determines a time window (1420). In some implementations, a time window is determined based upon detecting an alteration event of a virtual object, as well as the history of alteration activity. The time window may be an amount of time that the system 200 waits prior to releasing the virtual object from a restricted state that was entered due to the alteration event. The system 200 may analyze the history of alteration activity to determine the time window. For instance, to the extent relatively long gaps exists between stylus detections for the alteration event, the system may set a relatively long time window. To the extent relatively long gaps do not exist between stylus detections for the alteration event, the system may set a relatively short time window.
The system detects a potential end of the alteration event (1430). For instance, the system 200 may detects a potential end of the alteration event based on ceasing to detect a stylus (either contacting or hovering) that was performing the alteration event. In some implementations, the potential end of the alteration event is detected based upon the history of alteration activity. For example, if a user's interaction with a virtual object (or similar type of virtual object) is predictable, such as reviewing and signing documents, then the potential end of the alteration event is detected based on the predictable interaction.
The system 200 continues to restrict movement of the virtual object for the time window after detecting the potential end of the alteration event (1440). For example, even though the system 200 ceases to detect a stylus that was performing the alteration event, the system 200 continues to restrict movement of the virtual object to allow time for the user to return to the alteration event. In this example, if the user does desire to return to the alteration event, the continued restriction reduces the likelihood of an inadvertent or incidental contact causing movement of the virtual object that would disrupt the alteration event.
In some implementations, the time window is relatively long based upon a time required to detect an end of the alteration event. For instance, alterations made by a user of the virtual object are significant and require a substantially amount of time to complete.
In some implementations, the time window is relatively short based upon a time required to detect an end of the alteration event. For instance, alterations made by a user of the virtual object are minor and require a short amount of time to complete.
The system 200 monitors for input indicating a return to the alteration event during the time window (1450). In some implementations, a user periodically ceases providing input to the virtual object, but continues providing input to the virtual object during a relatively short period of time from the pause in providing input. For example, a user takes a few moments to review features of the virtual object and then continues to edit, remove, or add to the features of the virtual object. The system 200 monitors for inputs, such as stylus inputs, that suggest that the user has returned to the alteration event.
The system releases the restriction on movement of the virtual object based on the time window expiring without the monitoring detecting input indicating a return to the alteration event (1460). For instance, to the extent that the time window expires without detecting another stylus input at the virtual object, the system 200 releases the restriction on movement of the virtual object and the virtual object is once again fully responsive to touch inputs.
In some implementations, the time window is relatively short, wherein the release on the restriction on movement of the virtual object occurs shortly after no further input indicating return to the alteration event is received. For example, when a user is simply reviewing and signing standard documents, such as employee paychecks, the time window expires rather quickly, as compared to when a user is reviewing and editing a contract agreement.
In some implementations, the user's input is relatively constant over time. For example, the input provided by the user 1530 includes editing the virtual object 1540, such as adding a narrative writing to the virtual object 1540. Accordingly, the input provided by the user 1530 does not necessarily include significant pauses (or lapses in time) during writing. However, there are periods of time when the user 1530 pauses when adding the narrative writing to the virtual object 1540.
In
In some implementations, movement of the virtual object 1540 is restricted within a threshold period of time from when the stylus was last detected. For example, since the system 1500 no longer detects input provided by the stylus, but now instead receives the additional input 1550 provided by the user's hand, the additional input 1550 may cause movement of the virtual object 1540. However, because the virtual object 1540 remains restricted for the time window (e.g., a time period in which input from the stylus is not detected), the additional input 1550 does not cause movement of the virtual object 1540 and the user may resume writing without an interruption caused by movement of the virtual object 1540, as long as the user resumes writing within the time window in a manner that enables the system 1500 to detect the stylus prior to expiration of the time window.
The system 200 updates the display of the virtual objects to indicate a manner in which the one or more of the virtual objects have been restricted (1620). In some implementations, the display of the virtual objects includes highlighting (or differentiating) the one or more of the virtual objects that have been identified as being candidates for restriction from the virtual objects that have not been identified as being candidates for restriction. In addition, one or more of the virtual objects that have been identified as being candidates for restriction are displayed on the display device having an indication, such as a ghosted or anchored representation, to indicate that the one or more of the virtual objects are completely restricted from translation, rotational, and scaling movement. Further, one or more of the virtual objects that have been identified as being candidates for restriction are displayed on the display device having unique indications related to each of translation, rotational, and/or scaling restrictions.
The system 200 updates the display of the virtual objects to indicate the first type of input has been received for the first virtual object included in the subset of virtual objects, but movement of the first virtual object is restricted (1630). In some implementations, the display of the first virtual object includes highlighting (or differentiating) the first virtual object based upon receiving the first type of input from other virtual objects displayed on the display device. For example, when the first type of input is provided to the first virtual object, the first object is displayed on the display device having a ghosted representation, whereas other virtual objects remain being displayed in their original representation, i.e., not in a ghosted representation. In addition, when the first type of input is provided to the first virtual object, the first object may be shown as being stretched a point of contact, but not completely moved until the pulling reaches a threshold amount.
In some implementations, restrictions for the virtual objects 1740 and 1744 include damping of particular movements. For example, translational movements of the virtual objects 1740 and 1744 are damped at a high level (meaning no translational movement), whereas rotational movements of the virtual objects 1740 and 1744 are damped at a medium level (meaning some rotational movement is allowed). Accordingly, the virtual objects 1740 and 1744 are substantially limited with respect to translational movement and less limited in rotational movement, but have low (or no) limit to scaling up or down.
In
In some implementations, providing the input on the virtual object 1740 substantially damps (or locks) the virtual object 1740 from substantially all movement until a threshold level of input is supplied to the virtual object 1740. For example, while a user is writing on the virtual object 1740, no other input (either purposeful or inadvertent) would allow for movement of the virtual object 1740 due to the completely damped (or locked) state until the threshold level of input is supplied. Accordingly, as long as the user is writing on the virtual object 1740, the virtual object 1740 remains stationary on the canvas 1720.
In
In
In some implementations, touch inputs can include other dragging motions in order to sufficiently change the damping condition of the virtual object 1740. For example, dragging motions can include rotational and/or scaling motion inputs. When the dragging motions include rotational motion inputs, the highlighting 1742′ (in
The state information may include, for each frame, location, position, and restrictions placed on virtual objects displayed on the display device. The system 200 may store the state information for either a particular number of seconds or frames. For example, the system 200 may save the previous thirty seconds worth of inputs and display manipulations. Once the system 200 fills the particular number of state information entries, then the system 100 may replace the oldest state information frame with the most recent state information frame.
The system 200 detects that a restriction applied to a virtual object to be incorrect (1820). In some implementations, the incorrectly applied restriction conflicts with a current restriction of the virtual object. For example, if the virtual object was identified as being a candidate for restriction, the system 200 may later determine that the virtual object should not have been identified as being a candidate for restriction based on additional input received. The additional input may indicate that the virtual object should not have been restricted or at least result in the system 200 being more confident than not that the virtual object should not have been restricted.
In another example, if the virtual object was not identified as being a candidate for restriction, the system 200 may later determine that the virtual object should have been identified as being a candidate for restriction based on additional input received. The additional input may indicate that the virtual object should have been restricted or at least result in the system 200 being more confident than not that the virtual object should have been restricted.
The system 200 updates the restriction of the virtual object to correct the previously incorrect restriction classification (1830). In some implementations, a memory portion of a display system displaying the virtual object is updated to reflect the appropriate restriction classification of the virtual object.
The system 200 rewinds the state of the display system based upon the history of the prior state information and the updated restriction of the virtual object (1840). In some implementations, the display of the virtual object reverts back to a condition prior to detecting the incorrect restriction on the virtual object. For example, the display device will re-display the virtual object in a manner that is consistent with the restriction classification now deemed appropriate. When the virtual object was incorrectly left unrestricted, the system 200 may reverse any actions performed on the virtual object based on input received when the virtual object should have been restricted. When the virtual object was incorrectly restricted, the system 200 may perform any actions that would have occurred based on input that was received when the virtual object was incorrectly restricted and, thus, suppressed.
In
In
In