Patent Application
20110202834
The design of an effective user interface poses many challenges. One challenge is how to provide a user with an optimal amount of visual information or functionality, given the space limitations of a display and the needs of a particular user. This challenge can be especially acute for devices with small displays, such as smartphones or other mobile computing devices. This is because there is often more information available to a user performing a particular activity (e.g., browsing for audio or video files in a library of files) than can fit on the display. A user can easily become lost unless careful attention is paid to how information is presented on the limited amount available display space. Visual cues are useful for indicating, for example, a user's location when browsing a list or other collection of data, since it is often not possible to show an entire collection (e.g., a list of contacts stored in a smartphone) on a small display.
Another challenge is how to provide a high level of functionality while maintaining a satisfying and consistent user experience. As devices have become more complex, and as consumers have become more demanding, it has become increasingly difficult to design user interfaces that are convenient and pleasing for users, without sacrificing reliability, flexibility, functionality or performance.
Whatever the benefits of previous techniques, they do not have the advantages of the techniques and tools presented below.
Techniques and tools are described that relate to different aspects of a user interface that provides visual feedback in response to user input. For example, boundary effects are presented to provide visual cues to a user to indicate that a boundary in a movable user interface element (e.g., the end of a scrollable list) has been reached. As another example, parallax effects are presented in which multiple parallel or substantially parallel layers in a multi-layer user interface move at different rates, in response to user input. As another example, simulated inertia motion of UI elements is used to provide a more natural feel for touch input. Various combinations of features are described. For example, simulated inertia motion can be used in combination with parallax effects, boundary effects, or other types of visual feedback.
In one aspect, a user interface (UI) system receives gesture information corresponding to a gesture on a touch input device. The UI system calculates simulated inertia motion for a movable user interface element based at least in part on the gesture information, and potentially on other inertia information such as a friction coefficient or a parking speed coefficient. Based at least in part on the gesture information and on the simulated inertia motion, the UI system calculates a post-gesture position of the movable user interface element. The UI system determines that the post-gesture position exceeds a gesture boundary of the movable user interface element, and calculates a distortion effect (e.g., a squeeze, compression or squish effect) in the movable user interface element to indicate that the gesture boundary has been exceeded. Calculating the distortion effect can include, for example, determining an extent by which the gesture boundary has been exceeded, determining a compressible area of the movable user interface element, determining a scale factor for the distortion effect based at least in part on the compressible area and the extent by which the gesture boundary has been exceeded, and scaling the compressible area according to the scale factor. The distortion effect can be calculated based on a distortion point (which, for compression, can be referred to as a compression point or squeeze point), which can indicate the part of the UI element to be distorted.
In another aspect, user input (e.g., a gesture on a touch screen) indicates movement in a graphical user interface element having plural movable layers. Based at least in part on inertia information and the user input, a UI system calculates a first motion having a first movement rate in a first layer of the plural movable layers, and calculates a parallax motion in a second layer of the plural movable layers. The parallax motion is based at least in part on the first motion (and potentially simulated inertia motion), and the parallax motion comprises a movement of the second layer at a second movement rate that differs from the first movement rate. The parallax motion can be calculated based on, for example, a parallax constant for the second layer, or an amount of displayable data in the second layer.
In another aspect, a UI system receives gesture information corresponding to a gesture on a touch input device, the gesture information indicating a movement of a user interface element having a movement boundary. Based at least in part on the gesture information, the UI system computes a new position of the user interface element. Based at least in part on the new position, the UI system determines that the movement boundary has been exceeded. The UI system determines an extent by which the movement boundary has been exceeded, determines a compressible area of the user interface element, determines a scale factor for a distortion effect based at least in part on the compressible area and the extent by which the movement boundary has been exceeded, and presents a distortion effect in the user interface element. The distortion effect comprises a visual compression of content in the compressible area (e.g., text, images, graphics, video or other displayable content) according to the scale factor. Depending, for example, on the size of the compressible area and the size of the display area, some parts of the compressible area may not be visible on a display, so the distortion can be virtual (e.g., in areas that are not visible on a display) or the distortion can be actually displayed, or some parts of the distorted content can be displayed while other parts of the distorted content are not displayed. The visual compression is in a dimension that corresponds to the movement of the user interface element. For example, a vertical movement in a UI element that exceeds a movement boundary can cause content in the UI element to be vertically compressed or squeezed.
The foregoing and other objects, features, and advantages of the invention will become more apparent from the following detailed description, which proceeds with reference to the accompanying figures.
FIGS. 5 and 6A-6E are diagrams showing parallax effects and boundary effects in a user interface having parallel layers, according to one or more described embodiments.
Techniques and tools are described that relate to different aspects of a user interface that provides visual feedback in response to user input. For example, boundary effects are presented to provide visual cues to a user to indicate that a boundary in a movable user interface element (e.g., the end of a scrollable list) has been reached. As another example, parallax effects are presented in which multiple parallel or substantially parallel layers in a multi-layer user interface move at different rates, in response to user input. As another example, simulated inertia motion of UI elements is used to provide a more natural feel for touch input. Various combinations of features are described. In one implementation, a UI system that accepts touch input includes detailed motion rules (e.g., rules for interpreting different kinds of touch input, rules for presenting inertia motion in UI elements in response to touch input, rules for determining boundaries in UI elements, etc.). The motion rules can be combined with various combinations of optional motion features such as parallax effects, boundary effects, and other visual feedback. The visual feedback that is presented according to motion rules and optional motion features in a UI element can depend on many factors, such as the type of the UI element and the content of the UI element.
Various alternatives to the implementations described herein are possible. For example, techniques described with reference to flowchart diagrams can be altered by changing the ordering of stages shown in the flowcharts, by repeating or omitting certain stages, etc. As another example, systems described with reference to system diagrams can be altered by changing the ordering of processing stages shown in the diagrams, by repeating or omitting certain stages, etc. As another example, user interfaces described with reference to diagrams can be altered by changing the content or arrangement of user interface features shown in the diagrams, by omitting certain features, etc. As another example, although some implementations are described with reference to specific devices and user input mechanisms (e.g., mobile devices with a touchscreen interface), described techniques and tools can be used with other devices and/or user input mechanisms.
The various techniques and tools can be used in combination or independently. Different embodiments implement one or more of the described techniques and tools.
As devices have become more complex, and as consumers have become snore demanding, it has become increasingly difficult to design user interfaces that are convenient and pleasing for users, without sacrificing reliability, flexibility, functionality or performance. The feel of a user interface (UI) is becoming increasingly important to distinguish the underlying product from its competitors. An important contributor to the feel of a UI is how it reacts when a user interacts with it. This is especially true for touch-based interfaces.
Accordingly, techniques and tools are described for providing feedback (e.g., visual cues such as parallax effects, boundary effects, etc.) to users in response to user input (e.g., touch input). In some embodiments, movements in elements (also referred to as “controls”) are based at least in part on user input (e.g., gestures on a touchscreen) and an inertia model. For example, a movement in a UI element can be extended beyond the actual size of a gesture on a touchscreen by applying inertia to the movement. Applying inertia to a movement in a UI element typically involves performing one more calculations using gesture information (e.g., a gesture start position, a gesture end position, gesture velocity and/or other information) and one or more inertia motion values (e.g., friction coefficients) to determine a post-gesture state (e.g., a new position) for the UI element. Simulated inertia motion can be used in combination with other effects (e.g., parallax effects, boundary effects, etc.) to provide feedback to a user. In any of the examples herein, movements in UI elements can be rendered for display (e.g., depicting calculated distortion, parallax, or other effects, if any).
Movement in UI elements typically depends to some extent on user interaction. For example, a user that wishes to navigate from one part of a UI element to another (e.g., from the beginning of a scrollable list to the end of the list) provides user input to indicate a desired movement. The user input can then cause movement in the UI element and potentially other elements in the user interface. In some embodiments, a user causes movement in a display area of a device by interacting with a touchscreen. The interaction can include, for example, contacting the touchscreen with a fingertip, stylus or other object and moving it (e.g., with a flicking or sweeping motion) across the surface of the touchscreen to cause movement in a desired direction. Alternatively, a user can interact with a user interface in some other way, such as by pressing buttons (e.g., directional buttons) on a keypad or keyboard, moving a trackball, pointing and clicking with a mouse, making a voice command, etc.
The actual amount and direction of the user's motion that is necessary to produce particular movements can vary depending on implementation or user preferences. For example, a user interface system can include a default setting that is used to calculate the amount of motion (e.g., in terms of pixels) as a function of the size or rate of a user movement. As another example, a user can adjust a touchscreen sensitivity control, such that the same motion of a fingertip or stylus on a touchscreen will produce smaller or larger movements, depending on the setting of the control. Gestures can be made in various directions to cause movement in UI elements. For example, upward and downward gestures can cause upward or downward movements, respectively, while rightward and leftward movements can cause rightward and leftward movements, respectively. Upward/downward motion can even be combined with left/right motion for diagonal movements. Other kinds of motion, such as non-linear motion (e.g., curves) or bi-directional motion (e.g., pinch or stretch motions made with multiple contact points on a touchscreen) also can be used to cause movement in UI elements.
In any of the above techniques, any combination of the inertia, boundary, parallax, distortion, and other effects described herein can be applied. Depending on implementation and the type of processing desired, processing stages shown in example techniques 100, 110, 120 can be rearranged, added, omitted, split into multiple stages, combined with other stages, and/or replaced with like stages.
At 210, a UI system on a device receives touch input information in a touch input stream. For example, the touch input stream comprises data corresponding to a gesture on a touchscreen of a mobile device. Data received from the touch input stream can include, for example, gesture information such as a gesture start position, a gesture end position, and timestamps for the gesture. The touch input stream is typically received from a device operating system, which converts raw data received from a touch input device (e.g., a touchscreen) into gesture information. Alternatively, data received from the touch input stream can include other information, or gesture information can be received from some other source.
At 212, filtering is applied to the touch input stream. In the filtering stage, one or more algorithms are applied to the touch input stream coming from the OS to filter out or correct anomalous data. For example, the filtering stage can correct misaligned touch data caused by jittering (e.g., values that are not aligned with previous inputs) or filter out spurious touch contact points (e.g., incorrect interpretation of a single touch point as multiple touch points that are close together), etc. As another example, if only single-touch-point gestures are allowed, the filtering stage can convert any multi-touch input into a single-touch input. Alternatively, touch input filtering can be performed during generating of the touch input stream (e.g., at the device OS). As another alternative, touch input filtering can be performed during a coordinate space transform stage (e.g., coordinate space transform 220). As another alternative, touch input filtering can be omitted.
At 220, the UI system applies a coordinate space transform to data in the touch input stream corresponding to the gesture. For example, a coordinate space transform is applied to the data from the touch input stream in order to account for possible rotations of the device, scale changes, influence from other animations, etc., in order to properly interpret the original input stream. For example, if a UI element is rotated 90 degrees such that vertical movement in the UI element becomes horizontal movement (or vice versa), a vertical gesture can be transformed to a horizontal gesture (or vice versa) to account for the rotation of the device. If no adjustments are necessary, the coordinate space transform can leave gesture information unchanged. Alternatively the coordinate space transform state can be omitted.
At 230, the UI system calculates the velocity at the end of the gesture. For example, the velocity is calculated by determining a first position near the end of the gesture and an end position of the gesture, and dividing by the time elapsed during the movement from the first position near the end of the gesture to the end position. In one implementation, the first position is determined by finding the gesture position at approximately 100 ms prior to the end of the gesture. Measuring velocity near the end of the gesture can help to provide a more accurate motion resulting from the gesture than measuring velocity over the entire course of the gesture. For example, if a gesture starts slowly and ends with a higher velocity, measuring the velocity at the end of the gesture can help to more accurately reflect the user's intended gesture (e.g., a strong flick). Alternatively, the velocity is calculated by determining the distance (e.g., in pixel units) between the start position for the gesture and the end position of the gesture, and dividing by the time elapsed during the movement from the start position to the end position. The time elapsed can be calculated, for example, by computing the difference between a timestamp associated with the start position and a timestamp associated with the end position.
At 240, the UI system determines whether the gesture is an inertia gesture. As used herein, an inertia gesture refers to a gesture, such as a flick gesture, capable of causing movement in one or more user interface elements to which inertia can be applied. The UI system can distinguish between a non-inertia gesture and an inertia gesture by determining how quickly the user's finger, stylus, etc., was moving when it broke contact with the touchscreen, and whether the velocity exceeds a threshold. If the gesture ends with a velocity above the threshold, the gesture can be interpreted as an inertia gesture. For example, a gesture that starts with panning motion at a velocity below the threshold and ends with a velocity above the threshold can be interpreted as ending with a flick that causes movement to which inertia can be applied. If the gesture ends with a velocity below the threshold, the gesture can be interpreted as a non-inertia gesture. Exemplary techniques and tools used in some implementations for gesture interpretation are described in detail below.
If the gesture is an inertia gesture (e.g., a flick gesture), at 250 the UI system determines whether inertia will be applied to the motion indicated by the gesture. For example, the UI system determines based on gesture information (e.g., end-of-gesture velocity) and/or other information (e.g., user preferences) whether to apply inertia to the motion in the UI element. Despite being considered an inertia gesture, a gesture such as a flick may still not have inertia applied to its resulting movements, such as when a flick gesture is received for a UI element that does not support inertia movements, or for a UI element for which inertia movement has been deactivated (e.g., according to user preference).
If inertia is not to be applied (e.g., when the gesture is not an inertia gesture), at 254 the UI system computes a new position for the UI element based on gesture information (e.g., end-of-gesture position coordinates). If inertia is to be applied, at 252 the UI system computes a new position based on the gesture information (e.g., end-of-gesture position coordinates) and simulated inertia. For example, the simulated inertia involves treating a UI element, or part of a UI element, as a physical object of non-zero mass that moves according to an approximation of Newtonian physics. The approximation can include, for example, a friction coefficient and/or other parameters that control how the movement is calculated and/or rendered.
When the new position of the UI element has been computed (with or without simulated inertia), the UI system determines at 260 whether boundary feedback will be presented. Determining whether boundary feedback will be presented involves determining whether the new position is within boundaries (if any) of the UI element. For example, in a scrollable list, the UI system can determine whether the new position is calculated to be outside the boundaries of the scrollable list (e.g., below the end of a vertically scrollable list). Some UI elements may not have boundaries that can be exceeded by any permitted motion. For example, a UI element may take the form of a wrappable list, which may have a default entry position but no beginning or end. If the wrappable list is axis-locked (e.g., if movement is only permitted along a vertical axis for a vertically scrolling list), the list may have no boundaries that can be exceeded by any permitted motion. For UI elements without any boundaries, or without boundaries that can be exceeded by permitted motion, the determination of whether the new position is within boundaries can be skipped. Axis locking is described in more detail below.
If boundary feedback is to be presented, at 262 the UI system applies a boundary effect to the UI element. For example, the UI system can apply a visual distortion effect such as a “squish” or compression of text, images or other visual information in the UI element, to provide a visual cue that a boundary of the UI element has been reached. Boundary effects are described in more detail below.
The UI system determines at 270 whether parallax feedback will be presented. Determining whether parallax feedback will be presented involves determining whether the UI element has multiple parallel layers or substantially parallel layers that can be moved at different rates based on the same gesture. If parallax feedback is to be presented, at 272 the UI system applies a parallax effect to the UI element. In general, a parallax effect involves movement of multiple parallel layers, or substantially parallel layers, at different rates. Example parallax effects are described in more detail below.
The processing stages in example technique 200 indicate example flows of information in a UI system. Depending on implementation and the type of processing desired, processing stages can be rearranged, added, omitted, split into multiple stages, combined with other stages, and/or replaced with like stages.
For example, although example technique 200 shows stages of receiving data from a touch input stream, applying touch input filtering, applying a coordinate space transform, calculating a velocity at the end of a gesture, and determining whether the gesture is an inertia gesture, such processing stages are only exemplary. Gesture information (e.g., gesture velocity, position, whether the gesture is a candidate for simulated inertia, etc.) can be obtained in other ways. As an example, a module that determines whether to apply inertia motion and determines whether to apply boundary feedback or parallax effects can obtain gesture data from another source, such as another module that accepts touch input and makes calculations to obtain gesture information (e.g., gesture velocity, end-of-gesture position).
As another example, although example technique 200 shows a determination of whether to present boundary feedback occurring before a determination of whether to present parallax feedback, such an arrangement is only exemplary. A determination of whether to present boundary feedback and/or parallax feedback can be performed in other ways. As examples, once a new position has been calculated, determinations of whether to present boundary feedback and/or parallax feedback can occur in parallel, or the determination of whether to present a parallax effect can occur before the determination of whether to present a boundary effect. Such arrangements can be useful, for example, where a gesture may cause movements in multiple parallel layers of a UI element prior to reaching a boundary of the element. A UI system also can determine (e.g., based on characteristics of a current UI element) whether boundary effects and/or parallax effects are not available (e.g., for UI elements that do not have multiple layers or boundaries), and skip processing stages that are not relevant.
Boundary feedback can be used to provide visual cues to a user to indicate that a boundary (e.g., a boundary at the end, beginning, or other location) in a UI element (e.g., a data collection such as a list) has been reached. In described implementations, a UI system presents a boundary effect in a UI element (or a portion of a UI element) by causing the UI element to be displayed in a visually distorted state, such as a squeezed or compressed state (i.e., a state in which text, images or other content is shown to be smaller than normal in one or more dimensions), to indicate that a boundary of the UI element has been reached.
Described techniques and tools for presenting boundary feedback can be applied to any UI element with one or more boundaries that can be manipulated by moving the element. For example, described techniques and tools can be used in an email viewer, such that text in a scrollable email message is distorted (e.g., squeezed or compressed) to indicate that the end of the email message has been reached.
Boundary effects (e.g., distortion effects) can be presented in different ways. For example, a boundary effect can be held in place for different lengths of time depending on user input and/or design choice. A boundary effect can end, for example, by returning the UI element to a normal (e.g., undistorted) state when a user lifts a finger, stylus or other object to end an interaction with a touchscreen after reaching a boundary, or when an inertia motion has completed. As another example, distortion effects other than a squish, squeeze or compression can be used. One alternative distortion effect is a visual stretch. A stretch effect can be used, for example, in combination with a snap-back animation to indicate that boundary has been reached.
Boundary effects can be presented even when it is possible to continue a movement beyond a boundary. For example, if a user scrolls to the end of a vertically-oriented list, causing a distortion of text or images at the end of the list, further motion can cause the list to wrap past the boundary and back to the beginning of the list. The UI also can show an element (or part of an element) at the top of the list to indicate that further movement can allow the user to wrap back to the beginning of the list.
From state 390, the upward motion causes a distortion effect shown in state 392. In this example, the upward motion is finger-tracking motion caused by a drag gesture, but distortion effects also can be caused by other motion resulting from other kinds of gestures, such as inertia motion caused by a flick gesture. The distortion effect indicates that a boundary in the list has been reached. In the example shown in state
In the example shown in state 392, the squeeze point 396 is indicated at the top of a list, outside the display area 300. Other squeeze points are also possible. For example, the squeeze point could be at the center of a list (e.g., at item 50 in a 100 item list) or at the top of a visible portion of a list. In this example, the list can be considered as having two parts—one part above the squeeze point, and one part below the squeeze point—where only one part of the list is squeezed. The squeeze point can change dynamically, depending on the state of the list and/or display. For example, a squeeze point can move up or down (e.g., in response to where the center of the list is) as elements are added to or removed from the list, or a squeeze point can update automatically (e.g., when the end of the list has been reached) to be at the top of a visible portion of the list. As another example, a squeeze point can be placed outside of a list. This can be useful to provide more consistent visual feedback, such as when a UI element does not fill the visible area.
In state 394, the list has returned to the undistorted state shown in state 390. For example, the list can return to the undistorted state after the gesture shown in state 390 is ended (e.g., when the user breaks contact with the touchscreen).
The upward motion shown in
States 390-394 are only examples of possible states. In practice, a UI element can exist in any number of states (e.g., in intermediate states between example states 390-394, etc.) in addition to, or as alternatives to, the example states 390-394. For example, it is preferable to show a gradual transition between an undistorted state (e.g., state 390) and a distorted state (e.g., state 392), or from a distorted state to an undistorted state, to provide a more natural feel and avoid the appearance of abrupt changes in the display. Intermediate states, such as states that may occur between state 390 and state 392, or between state 392 and state 394 can show gradually increasing or decreasing degrees of distortion, as appropriate.
In described embodiments, a UI system can present parallel, or substantially parallel, movable layers. The UI system can present a parallax effect, in which layers move at different speeds relative to one another. The effect is referred to as a parallax effect because, in a typical example, a layer that is of interest to a user moves at a faster rate than other layers, as though the layer of interest were closer to the user than the other, slower-moving layers. However, the term “parallax effect” as used herein refers more generally to effects in which layers move at different rates relative to one another.
The rate of movement in each layer can depend on several factors, including the amount of data to be presented visually (e.g., text or graphics) in the layers, or the arrangement of the layers relative to one another. The amount of data to be presented visually in a layer can measured by, for example, determining the length as measured in a horizontal direction of the data as rendered on a display or as laid out for possible rendering on the display. Length can be measured in pixels or by some other suitable measure (e.g., the number of characters in a string of text). A layer with a larger amount of data and moving at a faster rate can advance by a number of pixels that is greater than a layer with a smaller amount of data moving at a slower rate. Layer movement rates can be determined in different ways. For example, movement rates in slower layers can be derived from movement rates in faster layers, or vice versa. Or, layer movement rates can be determined independently of one another. Layers that exhibit parallax effects can be overlapping layers or non-overlapping layers.
When user interaction causes movement in layers, the movement of the layers is a typically a function of the length of the layers and the size and direction of the motion made by the user. For example, a leftward flicking motion on a touchscreen produces a leftward movement of the layers relative to the display area. Depending on implementation and/or user preferences, user input can be interpreted in different ways to produce different kinds of movement in the layers. For example, a UI system can interpret any movement to the left or right, even diagonal movements extending well above or below the horizontal plane, as a valid leftward or rightward motion of a layer, or the system can require more precise movements. As another example, a UI system can require that a user interact with a part of a touchscreen corresponding to the display area occupied by a layer before moving that layer, or the system can allow interaction with other parts of the touchscreen to cause movement in a layer. As another example, a user can use an upward or downward motion to scroll up or down in a part of the content layer that does not appear on the display all at once.
In some embodiments, lock points indicate corresponding positions in layers with which a display area of a device will be aligned. For example, when a user navigates to a position on a content layer such that the left edge of the display area is at a left-edge lock point “A,” the left edge of display area will also be aligned at a corresponding left-edge lock point “A” in each of the other layers. Lock points also can indicate alignment of a right edge of a display area (right-edge lock points), or other types of alignment (e.g., center lock points). Typically, corresponding lock points in each layer are positioned to account for the fact that layers will move at different speeds. For example, if the distance between a first lock point and a second lock point in a content layer is twice as great as the distance between corresponding first and second lock points in a background layer, the background layer moves at half the rate of the content layer when transitioning between the two lock points.
In addition to indicating corresponding positions in layers, lock points can exhibit other behavior. For example, lock points can indicate positions in a content layer to which the layer will move when the part of the layer corresponding to the lock point comes into view on the display. This can be useful, for example, when an image, list or other content element comes partially into view near an edge of the display area—the content layer can automatically bring the content element completely into view by moving the layer such that an edge of the display area aligns with an appropriate lock point. A lock animation can be performed at the end of a gesture, such as a flick or pan gesture, to align the layers with a particular lock point. As an example, a lock animation can be performed at the end of a gesture that causes movement of a content layer to a position between two elements in a content layer (e.g., where portions of two images in a content layer are visible in a display area). A UI system can select an element (e.g., by checking which element occupies more space in the display area) and transition to focus on that element using the lock animations. This can improve the overall look of the layers and can be effective in bringing information or functional elements into view in a display area. A lock animation also can be used together with simulated inertia motion. For example, a lock animation can be presented after inertia motion stops, or a lock animation can be blended with inertia motion (such as by extending inertia motion to a lock point, or ending inertia motion early by gradually coming to a stop at a lock point) to present a smooth transition to a lock point.
The amounts and rates of movements presented in parallax effects can be calculated and presented in different ways. In a detailed example described below, equations are described for calculating parallax effect movements in which a parallax constant is used to determine anew position for a layer after a gesture. As another example, motion in layers and/or other elements, such as lists, can be calculated based on motion ratios. For example, a UI system can calculate motion ratios for a background layer and a title layer by dividing the width of the background layer and the width of the title layer, respectively, by a maximum width of the content layer. Taking into account the widths of the background layer and the title layer, a system can map locations of lock points in the background layer and the title layer, respectively, based on the locations of corresponding lock points in the content layer.
Movement of various layers can differ depending on context. For example, a user can navigate left from the beginning of a content layer to reach the end of a content layer, and can navigate right from the end of the content layer to reach the beginning of a content layer. This wrapping feature provides more flexibility when navigating through the content layer. Wrapping can be handled by the UI system in different ways. For example, wrapping can be handled by producing an animation that shows a rapid transition from the end of layers such as title layers or background layers back to the beginning of such layers, or vice-versa. Such animations can be combined with ordinary panning movements in the content layer, or with other animations in the content layer, such as locking animations. However, wrapping functionality is not required.
Background layer 450 floats behind the other layers. Data to be presented visually in background layer 450 can include, for example, an image that extends beyond the boundaries of display area 300. The content layer 414 includes content elements (e.g., images) 430A-H. Layers 410, 412 include text information (“Category” and “Selected Subcategory,” respectively). The length of content layer 414 is indicated to be approximately twice the length of layer 412, which is in turn indicated to be approximately twice the length of layer 410. The length of background layer 450 is indicated to be slightly less than the length of layer 412.
In
When user input indicates a motion to the right or left, the system produces a rightward or leftward movement of the layers 410, 412, 414, 450 relative to display area 300. The amount of movement of layers 410, 412, 414, 450 is a function of the data in the layers and the size or rate of the motion made by the user.
In
The lock points shown in
In this example, layers 410, 412, 414, 450 move according to the following rules, except during wrapping animations:
Movement of layers 410, 412, 414, 450 may differ from the rules described above in some circumstances. In this example, wrapping is permitted. User 302 can navigate left from the beginning of content layer 414 (the position shown in
Parallax effects can be used in combination with boundary effects and inertia motion. For example, boundary effects can be used to indicate when a user has reached a boundary of a layer, or a boundary of an element within a layer. As another example, inertia motion can be used to extend motion of UI elements caused by some gestures (e.g., flick gestures). If inertia motion causes movement of a UI element (e.g., a layer) to extend beyond a boundary, a UI system can present a boundary effect.
In response to the flick gesture 510, the UI system produces leftward movement of the layers 530, 532 relative to the display area 300. For example, from display state 590, the flick gesture 510 causes a leftward movement in the layers and leads to display state 592, in which element 540 is no longer visible, and elements 542 and 544 have moved to the left. The text string (“Albums”) in the list header layer 530 also has moved to the left, but at a slower rate (in terms of pixels) than the content layer 532. The movement of the layers 530, 532 is a function of the data in the layers and the velocity of the flick gesture 510.
From display state 592, the inertia motion causes continued leftward movement of the layers 530, 532 without further input from the user 302, and leads to display state 594 in which element 542 is no longer visible. The inertia motion causes the content layer to extend beyond a boundary (not shown) to the right of the element 544 in the content layer 532, which results in a distortion effect in which an image and text in element 544 is squeezed or compressed in a horizontal dimension. The compression is indicated by the reduced length of the dimension lines above the image and text (“Rock & Roll Part in”) of element 544, respectively. The text string (“Albums”) in the list header layer 530 also has moved to the left, but at a slower rate (in terms of pixels) than the content layer 532. The text in list header layer 530 is uncompressed. The distortion effect gives user 302 an indication that the end of the content layer 532 has been reached.
Although a motion that is calculated to extend beyond a boundary may result in a distortion effect, the boundary need not prevent further movement in the direction of the motion. For example, if wrapping functionality is available, further movement beyond the boundary can cause the content layer 530 to wrap back to the beginning (e.g., back to display state 590). In state 594, element 540 at the beginning of the collection is partially visible, indicating that wrapping is available.
The display can return from display state 594 to display state 592, transitioning from a display state with a distortion effect to an undistorted display state. This can occur, for example, without any additional input by the user. The length of time that it takes to transition between states can vary depending on implementation.
Flick gesture 510 is only an example of a possible user interaction. The same gesture 510 and/or other user interactions (e.g., motions having different sizes, directions, or velocities) can cause different effects, different display states, different transitions between display states, etc. Some display states (e.g., display state 594) may occur only if a gesture results in a post-gesture position that is calculated to go beyond a boundary for the layer.
Display states 590-594 are only examples of possible display states. In practice, a display can exist in any number of states (e.g., in intermediate states between example states 590-594, in states with different visible UI elements, etc.) in addition to, or as alternatives to, the example display states 590-594. For example, it is preferable to show a gradual transition between an undistorted state (e.g., state 592) and a distorted state (e.g., state 494), or from a distorted state to an undistorted state, to provide a more natural feel and avoid the appearance of abrupt changes in the display. Intermediate states, such as states that may occur between state 592 and state 594, can show gradually increasing or decreasing degrees of distortion, as appropriate. As another example, a UI system can provide a boundary effect by compressing the elements 542 and 544 shown in display state 592 without moving the elements 542 and 544 to the left in the display area 300.
Described techniques and tools can be used on display screens in different orientations, such as landscape orientation. Changes in display orientation can occur, for example, where a UI has been configured (e.g., by user preference) to be oriented in landscape fashion, or where a user has physically rotated a device. One or more sensors (e.g., an accelerometer) in the device can be used to detect when a device has been rotated, and adjust the display orientation accordingly.
In the example shown in
However, such adjustments are not required. For example, if a display area has equal height and width, reorientation of the display area to a landscape orientation will not change the effective width of the display area.
The direction of motion that can be caused by user 302 is indicated by a left-pointing arrow and a right-pointing arrow in
The up-pointing and down-pointing arrows indicate possible movements of the list 650 in response to user movements. The amount of movement of list 650 can be a function of the size or rate of the motion made by user 302, and the data in list 650. Thus, scrolling of the list 650 can be element-by-element, page-by-page of elements, or something in between that depends on size or rate of the motion. In this example, list 650 includes only one element that is not visible in the display area 300, as shown in
In this example, the amount of movement in layers 610, 612, 614 is a function of the data in the layers and the size or rate of the motion made by the user. Horizontal movement in layers 610, 612, 614 proceeds according to the following rules, except during wrapping animations:
Movement in the layers 610, 612, 614 may differ from the rules described above in some circumstances. In the example shown in
In
The lock points shown in
User 302 can move left or right in content layer 614 after making an up or down movement in list 650. The current position of list 650 can be saved, or the system can revert to a default position (e.g., the top-of-list position indicated in
The system can interpret user movements to the left or right, even diagonal movements extending above or below the horizontal plane, as a valid leftward or rightward motion. Similarly, the system can interpret upward or downward movements, even diagonal movement extending to the left or right of the vertical plane, as a valid upward or downward motion. Although
In
In
Alternatively, avatar 630 can move in different ways, or exhibit other functionality. For example, a UI system can present a distortion effect in avatar 630 to indicate a user's location in a data collection with which the avatar is associated. Avatar 630 also can be locked to particular position (e.g., a lock point) in content layer 614 or in some other layer, such that avatar 630 moves at the same horizontal rate as the layer to which it is locked. As another alternative, avatar 630 can be associated with a list that can be scrolled up or down, such as list 650, and move up or down as the associated list is scrolled up or down.
In this section, a detailed implementation is described comprising aspects of motion feedback including boundary effects and parallax effects, with reference to the following detailed example.
In this detailed example, a set of equations, coefficients and rules are described that can allow a UI system (e.g., a UI system provided as part of a mobile device operating system) to interpret user input such as touch gestures (including multi-touch gestures with more than one touch contact point) and generate motion feedback in response to user input. Features described in this detailed example include inertia movement, panning and zooming operations, boundary effects, parallax effects, and combinations thereof. Described features can help to provide natural-looking, smooth motion in response to user input (e.g., touch gestures).
In this detailed example, processing tasks can be handled by different software modules. For example, a module called “ITouchSession” provides coefficients, gesture positions, and gesture velocity information, and a dynamic motion module in a mobile device operating system uses information provided by ITouchSession to generate motion feedback (e.g., parallax effects, boundary effects, etc.). Preferably, gesture information provided to the dynamic motion module is accurate (e.g., with minimal jitter in position information), detailed (e.g., with time stamps on touch input), and low-latency (e.g., under 30 ms). The information (e.g., motion feedback information) generated by the dynamic motion module can be used by other modules, as well. For example, web browsers or other applications that run on the mobile device operating system can use information generated by the dynamic motion module.
In this detailed example, the dynamic motion resulting from user interaction is defined by a set of motion rules. The motion rules define how different visual elements react on screen in response to different gestures. For example, some rules apply to finger-tracking gestures such as panning or dragging gestures, some rules apply to flick or toss gestures, and some rules apply to pinch or stretch gestures. Additionally, some rules, such as inertia rules, may apply to more than one type of gesture. The specific motion rules that apply to different UI elements (or “controls”) are determined by factors such as the control type and control content; not all motion rules will apply to all UI elements. For example, rules for pinch and stretch gestures do not apply to UI elements where pinch and stretch gestures are not recognized. The motion resulting from the application of motion rules to the input stream generated by the user can be further refined by an optional set of modifiers, which are collectively called “optional motion features.”
In this detailed example, touch input interactions that result in dynamic motion comply with the motion rules. Additionally, different UI elements (or “controls”) can apply zero or more optional motion features, which can be determined by factors such as the desired motion, control type and control content. For example, a list control may opt to enhance motion feedback with boundary effects, while a panorama control may apply a parallax feature to some of its layers.
In addition, when a user interacts with a UI element, it can be helpful to provide some immediate (or substantially immediate) visual feedback to the user (e.g., a change in movement in the UI element, or some other effect such as a tilt or highlight). Immediate or substantially immediate feedback helps the user to know that the user interface is responsive to the user's actions.
In this detailed example, the following motion rules apply in UI elements where the rules (e.g., rules relating to finger-tracking gestures, inertia, boundaries, pinch/stretch gestures) are relevant to the types of motion that are permitted in the respective UI elements. The motion rules can be modified for some UI elements, such as where optional motion features apply to a UI element.
For finger tracking movements (e.g., movements caused by dragging or panning gestures), the content at the initial gesture point moves in direct correspondence to the gesture. For example, content under the user's finger at an initial touch point moves with the user's finger during the gesture. The current position of a visual element is given by the following equation:
p=p
0+(q−q0) (Eq. 1)
where p is the (x, y) vector that represents the current position of the visual element, p0 is the (x0, y0) vector that represents the visual element position at the beginning of the gesture, q is the (x, y) vector that represents the current touch contact position, and q0 is the (x0, y0) vector that represents the touch contact position at the beginning of the gesture.
In a UI element that allows inertia movement (e.g., a scrolling list), when the user finishes a gesture (e.g., by lifting a finger or other object to end the interaction with the touchscreen), a velocity and direction for that movement is identified, and the motion initially continues in the same direction and speed as the gesture, as if the visual element was a real, physical object with a non-zero mass. If the motion is not stopped for some other, permissible reason (e.g., where the UI element reaches a boundary or is stopped by another user gesture), the motion gradually decelerates over time, eventually coming to a stop. The deceleration proceeds according to a combination of equations and coefficients, which can vary depending on implementation. Default system-wide coefficient values can be made available. Default system-wide coefficients can help to maintain a consistent feeling across all controls. Alternatively, different equations or coefficients can be used, such as where a particular control has its own friction coefficient for modeling different kinds of motion.
The velocity (e.g., in pixels/second) at the end of the gesture is computed by the following equation:
v=(q−q0)/(t−t0), (Eq. 2)
where v is the (vx, vy) velocity vector that represents the inertia velocity at the end of the gesture, q is the (x, y) vector that represents the touch contact position at the end of the gesture, q0 is the (x0, y0) vector that represents the touch contact position at the time t0, t is the timestamp of the last touch input of the gesture, and t0 is the timestamp of the least recent touch input that happened within some fixed period of time from the last touch input. Alternatively, the velocity can be calculated in another way. For example, a weighted sum of velocities at different time instances can be calculated, with greater weighting for velocities at the end of the gesture. In this detailed example, calculating the velocity is the responsibility of ITouchSession. However, velocity calculations can be handled by other modules.
The duration of the inertia motion can be computed according to the following equation:
where tmax is the duration of the inertia motion, |v0| is the magnitude of the initial velocity vector (|v0|>γ), μ is a friction coefficient (e.g., MotionParameter_Friction, 0<μ<1), and y is a parking speed coefficient (e.g., MotionParameter_ParkingSpeed, 0<γ<|v0|) that is used to indicate a threshold velocity, below which inertia motion will stop. In one implementation, the friction coefficient is 0.4, and the parking speed coefficient is 60.0. The duration is computed at the start of the inertia motion, and need not be computed again.
The following equation will compute the current velocity vector v at any given time t:
v=V
0·μt (Eq. 4).
The new position p′ for the visual element can be computed based on its last known position p and the time elapsed since the last position update (Δt), as shown in the following equation:
p′=p+v·Δt (Eq. 5).
The motion stops once the velocity reaches a value smaller than γ.
The actual calculation of values relating to inertia motion (e.g., velocity, etc.) can differ depending on implementation.
Motion Rule: Interacting with an Element in Inertia Motion
If a new gesture begins while a UI element is in inertia motion, the inertia motion is immediately interrupted. Depending on the new gesture, the motion in the UI element may be stopped, or a new motion may start. If the new gesture causes a new motion in the UI element, the new gesture controls the UI element's motion. The previous gesture and any consequent inertia do not affect the motion generated by the new gesture. Handling of new gestures during inertia motion can be different depending on implementation. For example, new gestures can be ignored during inertia motion or can have different effects on inertia motion.
The motion of some UI elements is limited by gesture boundaries. The dimensions of gesture boundaries and the effects of exceeding gesture boundaries can differ depending on several factors, such as the content of a UI element and/or a minimum visible area of the UI element. For example, lists which don't wrap around indefinitely may only be able to scroll a certain distance based on the number of items in the list and a minimum amount of visible items (e.g., an amount of items that occupies most or all of a display area).
In this detailed example, for an element A having a width wA, height hA, total area SA and position pA (xA, yA), with a minimum visible area SVmin (width wVmin, height hVmin) currently at position pVmin (xVmin, yVmin), a gesture that begins at an initial position q (xq, yq) has a rectangular gesture boundary area ST (width wT, height hT) at position pT=(xT, yT). The minimum visible area indicates a minimum visible amount of the control (e.g., a minimum number of list items in a scrollable list), but does not require any particular part of the control to be visible. Therefore, the content of the minimum visible area for a particular control can vary depending on, for example, the control's current state (e.g., whether the end or beginning of a scrollable list is currently visible).
Conceptually, the position pT of the gesture boundary area can be defined according to the following equation:
p
T
=q+(pVmin+SVmin)−(pA+SA) (Eq. 6).
The x and y coordinates of the position pT of the gesture boundary area are defined according to the following equations:
x
T
=x
q+(xVmin+wVmin)−(xA+wA) (Eq. 7)
y
T
=y
q+(yVmin+hVmin)−(yA+hA) (Eq. 8).
The dimensions of the gesture boundary area are defined according to the following equation:
S
T
=S
A
−S
Vmin=(hA−hVmin,wA−wVmin) (Eq. 9).
If the new position of the UI element, resulting from user interaction or simulated inertia or some combination, falls outside the area defined by pT+ST, a boundary has been exceeded. An appropriate boundary feedback modifier can be applied or the new position can be clamped (i.e., kept within the allowed boundaries).
In
In
The actual calculation of values relating to motion boundaries and the effects of motion boundaries can differ depending on implementation. For example, if a wrapping feature is available in a UI element, a boundary can indicate a position at which a boundary effect will be presented (e.g., to indicate that the end of a list has been reached) without preventing further movement beyond the boundary (e.g., wrapping movement from the end of the list back to the beginning of the list).
Pinch gestures and stretch gestures are gestures that can change the scale (zoom) of the subject area of a control (e.g., a map or image with zoom capability). Pinch gestures and stretch gestures are considered to be multi-touch gestures because they typically have multiple points of interaction. In a typical pinch or stretch gesture scenario, a user places two fingers some distance apart from each other on a touchscreen, and either increases (for a stretch gesture) or decreases (for a pinch gesture) the distance between them.
The scale adjustment caused by a pinch or stretch gesture can be represented as follows. Let qA0=(xA0, yA0) and qB0=(xB0, yB0) be the positions for initial touch points A and B. The distance d0 between the two points represents a 100% scale factor, and can be calculated according to the following equation:
d
0
=|qA
0
−qB
0| (Eq. 10).
The distance d0 includes a horizontal component xd0 and a vertical component yd0.
Let qA and qB be updated positions for touch points A and B, and let d=(xd, yd) be the distance between them, calculated in a similar manner. The distance d also includes a horizontal component xd and a vertical component yd. The scale factor szoom to apply to the UI element can be calculated according to the following equation:
Note that in Equation 11, the scale s is not isometric, i.e., the X and Y axes will be scaled differently. For isometric scaling, the following equation can be used instead:
In this case, szoom is a scalar, so the same factor is applied to both X and Y components.
Alternatively, a scale factor can be calculated in different ways. For example, inertia can be applied to a pinch or stretch gesture (such as when the gesture ends with a velocity above a threshold), and the scale factor can be based at least in part on the inertia of the gesture (e.g., increasing the scale of the zoom when a stretch gesture ends with a velocity above a threshold).
To make zooming feel natural, the scale factor can be applied to a zooming point (e.g., a center point between touch points qA and qB). The zooming point cz=(xcz, ycz) can be calculated by averaging the two touch contact positions, as shown in the following equation:
Alternatively, a zooming point can be calculated in a different way, or a calculation of a zooming point can be omitted.
A pinch/stretch gesture can also produce position changes (panning) in addition to scale changes. Panning position changes can occur simultaneously with scale changes. The zooming point calculation in Equation 13 is used when simultaneous panning is not allowed. If simultaneous panning is allowed, the zooming point is calculated using the initial touch contact positions qA0 and qB0 rather than the updated touch contact positions qA and qB. If cz0 is the initial zooming point and cz is the updated zooming point, the distance dpan=(xdpan, ydpan) between the two zooming points represents a panning offset to be applied to the UI element, as shown in the following equation:
d
pan
=c
z
−c
z0=(xcz−xcz0,ycz−ycz0) (Eq. 14).
Alternatively, a panning offset can be calculated in a different way, or a panning offset can be omitted.
Optional motion features can be used (e.g., when requested by a control) to refine or add visual feedback to motion generated by gestures. Optional motion features can depend on control type and content. For example, some controls (e.g., a scrolling list) may use an optional axis locking feature that is appropriate for the orientation of the control (e.g., allowing only vertical movements in a vertically scrolling list). Optional motion features can be used in combination with each other and with various motion rules. For example, a vertically scrolling list can use an axis locking feature and a boundary effect feature, while following rules for inertia motion and finger tracking motion. Different UI elements can use different combinations of rules and optional motion features, even when the different UI elements are visible at the same time. For example, a movable layer can use parallax effects but omit boundary effects, while a vertically scrolling list in the movable layer can use boundary effects but omit parallax effects. UI elements of the same basic type can use different sets of optional motion features. For example, a first pair of movable layers can use parallax effects and move at different rates relative to one another, while a third layer parallel to the first pair remains stationary.
When present, optional motion features act like filters, modifying the values generated according to other motion rules, such as the motion rules described above.
For some controls, it may make sense to permit movement only along a particular axis. For example, it can be useful to restrict movement of a movable, horizontal UI layer (sometimes referred to as a panorama control) to movements along the X axis, or to restrict movement of a vertically scrolling list to movements along the Y axis. In such cases, axis locking can be used as an optional motion feature.
In this detailed example, axis locking is applied to a UI element by using the relevant equations in the motion rules described above, but only applying an X or Y component (as appropriate) to the motion of the axis-locked UI element. Changes to the other component are ignored and not applied to the UI element's motion.
Alternatively, axis locking can be performed in another way. For example, in a UI element such as a wheel element that moves about an axis such as a Z axis, axis locking can be used to permit only rotational motion about the axis. As another alternative, axis locking can be omitted.
Parallax effects can be applied to controls that present multiple layers of content. In a parallax effect, multiple layers are animated differently (e.g., moving at different speeds), but the movements of the layers are based on the same input stream generated by the user.
In a parallax effect, layers that are animated in response to a gesture move at different speeds relative to one another. The layer that the user is interacting with directly (e.g., a content layer) is considered to be the top layer on a Z axis, that is, the layer that is closest to the user. Other layers are considered to be lower layers on a Z axis, that is, further away from the user. Examples of a parallax effects can be seen in
In this detailed example, a top layer reacts directly to the gesture, and the other layers move at increasingly lower speeds the further they are from the top layer along the Z axis. Mathematically speaking, that can be accomplished by applying a scaling factor to the delta between an initial gesture position and an updated gesture position. The updated gesture position can be obtained directly from user interaction (e.g., in a finger tracking gesture such as a panning gesture) or from a gesture with simulated inertia (e.g., a flick gesture). If kP is the constant parallax scaling factor to be applied for a particular layer L at initial position pL=(xL, yL), then the parallaxed position pP=(xP, yP) can be computed according to the following equation:
p
P
=p
L
+k
P(q−q0) (Eq. 15)
where q is the (x, y) vector that represents the current, post-gesture position (e.g., after the gesture and application of any simulated inertia), and q0 is the (x0, y0) vector that represents the touch contact position at the beginning of the gesture. The parallax constant kP can vary depending on the application, scenario and/or content of the control. For example, layers with different lengths can have different parallax constants.
Alternatively, parallax effects can be presented in different ways. For example, parallel layers can move according to the model shown in Equation 18 for some movements or parts of a movement and move according to other models in other movements or parts of a movement. Referring again to
When the boundary feedback motion feature is applied, a boundary feedback effect can be applied whenever a gesture would move the UI element past a boundary, either directly (e.g., by a dragging or panning gesture) or indirectly (e.g., by inertia motion generated by a flick gesture). In this first example boundary feedback model, once the UI element hits a boundary the content is compressed in the direction of the motion (e.g., a vertical compression for a vertical motion) up to a certain threshold. If the compression is caused by inertia, the content compresses up to a certain amount based on the velocity at the time the boundary is hit, then decompresses to the original size. If the compression is caused directly (e.g., by dragging), the compression can be held as long as the last touch contact point is held and decompress when the user breaks contact, or decompress after a fixed length of time.
In this first example boundary feedback model, the compression effect is achieved by applying a scale factor and dynamically placing a compression point to ensure that the effect looks the same regardless of the size of the list. In order to properly compute the motion and scale for the boundary effect, the first step is to identify that a boundary has been crossed and by how much. The boundary motion rule described above illustrates how to compute a boundary position in this first example boundary feedback model, and in the second example boundary feedback model described below.
Let q=(xq, yq) be the unmodified, post-gesture position resulting from an active finger tracking gesture (e.g., a dragging gesture) or from simulated inertia (e.g., from a flick gesture), let xL be the left boundary, let xR be the right boundary, let yT be the top boundary, and let yB be the bottom boundary. Let r=(rx, ry) represent how far the post-gesture position exceeds the boundaries with respect to {xL, xR, yT, yB}:
r
x=max(xL−xq,0,xq−xR) (Eq. 16)
r
y=max(yT−yq,0,yq−yB) (Eq. 17)
In cases where only a vertical or horizontal boundary applies (e.g., in axis-locked elements), r may be calculated in only a vertical or horizontal dimension, as appropriate, while omitting a calculation of the other dimension of r.
Let Sc be the compressible area with dimensions (wc, hc), which is some area equal to or greater than the visible area, depending on the value of coefficients k % and k+, where k% is the compression percentage coefficient (e.g., MotionParameter_CompressPercent (k%≧0)), and k+ is the compression offset coefficient (e.g., MotionParameter_CompressOffset{X,Y} (k+≧0)). If k%=0, then the compressible area matches the size of the visible area and the visual result is that only the visible part of the control is being compressed. If k%=1, the compressible area matches the entire control area. k+=(w+, h+) allows an increase in the compressible area by a fixed amount, regardless of the control area size. In this detailed example, the compressible area can be calculated according to the following equation:
S
c
=S
v
+k
%·(SA−SV)+k+ (Eq. 18),
where SA is the control area with dimensions (WA, hA), and SV, is the visible area with dimensions (wV, hV). In one implementation, the compression percentage coefficient is 0.0 and the compression offset coefficient is 0.5*SV.
If the user is actively dragging the content, the compression scale factor scomp=(scompx, scompy) to apply to the target UI element can be computed according to the following equations:
where ks is the compression factor coefficient (e.g., MotionParameter_CompressFactor (0<ks≦1)), and r≦Sv. In one implementation, the compression factor coefficient is 0.2. Alternatively, the scale factor and/or the compressible area can be calculated in different ways. For example, different ranges of compression coefficients can be used.
In words, what is being done here is to find the difference between the compressible area (e.g., in the horizontal or vertical dimensions) and the amount by which the gesture is compressing the compressible area, then calculating the scale factor based on that difference. The compression factor ks, if it is less than 1, limits how much the value of r (the amount by which the post-gesture position has exceeded the boundary) will cause the compressible area to be compressed. A UI system can then place a distortion point (which can also be referred to as a “squeeze point” or “compression point” when applying compression effects) at the other side of the compressible area (i.e., the side of the compressible area opposite the side where the gesture is being made) and apply that scale factor, resulting in a compression effect.
Once the user ends the dragging gesture (e.g., by lifting a finger from the touchscreen), and if no wrap-around functionality is available or if the threshold for wrap-around hasn't been reached, the content in the compressible area returns to a decompressed state. In this first example boundary effects model, decompression proceeds according to the appropriate equations set forth below.
In this first example boundary effects model, if a boundary is exceeded during inertia motion, the following equations are used to compute how far off the boundaries the current position is (r) over time, based on the velocity at the time the boundary was crossed (vh) and how far off the boundary the position is (rI) when the following equations are applied:
If r<0, the motion is complete. Note that rI can come either from inertia or from an active drag, such as when a user drags the content into a compressed state, then flicks, generating inertia.
The compression scale factor sinertiacomp=(sinertiacompx, sinertiacompy) to apply during inertia compression can be computed according to the following equations:
Note that these equations are similar to the case when dragging the content (see Equations 19-21, above), except that the coefficient ks (the compression factor coefficient) has already been applied in this case in Equations 22 and 23. Alternatively, the scale factor can be calculated in a different way. For example, constants such as the compression factor coefficient ks or the value 0.001 in Equation 23 can be replaced with other constants depending on implementation.
In this first example boundary effects model, in addition to computing the scale factor to apply to the target UI element, a compression point Ccomp=(ccompx, ccompy) is calculated in order to generate the expected visual effect. In practice, a compression point can be at different positions in a UI element. For example, a compression point can be located at or near the center of a UI element, such that half (or approximately half) of the content in the UI element will be compressed. As another example, a compression point can be located at or near a border of UI element, such that all (or approximately all) of the content in the UI element will be compressed. The compression point can vary for different UI elements. Using different compression points can be helpful for providing a consistent amount of distortion in the content of UI elements of different sizes. The compression point position can be computed according to the following equations:
Alternatively, compression points can be calculated in a different way, or the calculation of compression points can be omitted.
In this second example boundary feedback model, the appearance of the boundary feedback can be controlled in finer detail by using more coefficients. Also, regardless of whether the compression is caused directly (e.g., by dragging) or by inertia, the same calculations are used for the compression effects
Let q=(xq, yq) be the unmodified, post-gesture position resulting from an active finger tracking gesture (e.g., a dragging gesture) or from simulated inertia (e.g., from a flick gesture), let xL be the left boundary, let xR be the right boundary, let yT be the top boundary, and let yB be the bottom boundary. Let r=(wr, hr) represent how far the post-gesture position exceeds the boundaries with respect to {xL, xR, yT, yB}:
w
r=max(xL−xq,0,xq−xR) (Eq. 30)
h
r=max(yT−yq,0,yq−yB) (Eq. 31)
In cases where only a vertical or horizontal boundary applies (e.g., in axis-locked elements), r may be calculated in only a vertical or horizontal dimension, as appropriate, while omitting a calculation of the other dimension of r.
As in the first example boundary effects model, Sc is the compressible area with dimensions (wc, hc), calculated as shown in Equation 18. However, in this second example boundary effects model, given r=(wr, hr) and a compressible area Sc=(wc, hc), the compression scale factor scomp=(scompx, scompy) to apply to the target UI element is computed according to the following equations:
where ks is a spring factor coefficient (e.g., MotionParameter_SpringFactor (ks>0)), ke is a spring power coefficient (e.g., MotionParameter_SpringPower (ke>0)), kd is a damper factor coefficient (e.g., MotionParameter_DamperFactor (0≦kd≦1)), kL is a compression limit coefficient (e.g., MotionParameter_CompressionLimit (kL>0)), and Δt is the time interval since the last iteration of the simulation (Δt≧0). The equation for r″ imposes limits on the movement in the UI element during boundary feedback. If r″=0, the motion is considered to be complete.
In this second example boundary effects model, the spring factor coefficient ks is a number that specifies how much resistance will counteract the inertia force, and the spring power coefficient ke shapes the curve of the resistance. For example, a spring power coefficient of 1 indicates linear resistance, where resistance increases at a constant rate as compression increases. A spring power coefficient greater than 1 means that the resistance will increase at an increasing rate at higher compression, and less than 1 means that the resistance will increase, but at a decreasing rate, at higher compression. The damper factor coefficient kd represents a percentage of energy absorbed by the system and taken away from the inertia. The damper factor coefficient can be used to smooth out the boundary effect and avoid a repeated cycle of compression and decompression. The time interval Δt can vary depending on the number of frames per second in the animation of the boundary feedback, hardware speed, and other factors. In one implementation, the time interval is about 16 ms between each update. Varying the time interval can alter the effect of the boundary effect. For example, a smaller time interval can result in more fluid motion.
Alternatively, the scale factor and/or the compressible area can be calculated in different ways. For example, different ranges or values of coefficients can be used.
The shape of the position curve 1020 can be modified in different ways, such as by adjusting coefficients. For example, by adjusting the spring power coefficient, the uppermost tip of the boundary effect curve 1020 can be made to go higher (e.g., up to a configurable limit) or lower for a particular initial velocity. A higher tip of the curve can indicate a greater compression effect, and a lower tip can indicate a lesser compression effect. As another example, by adjusting the spring factor coefficient, the duration of the compression can be adjusted to be shorter or longer. In
In this second example boundary effects model, a current inertia velocity vc and a current touch contact position qc can be updated to reflect the physics interaction of the boundary effect. For example, the updated velocity v′c and updated touch contact position q′c are calculated according to the following equations:
Various alternatives to the boundary feedback models described above are possible. For example, if wrapping beyond a boundary (e.g., wrapping back to the beginning of a list after the end of the list has been reached) is permitted, if the compression is caused by dragging, the list can wrap around once a threshold compression has been reached. As another alternative, boundary effects can be omitted.
A UI system can provide programmatic access to system-wide values e.g., (inertia values, boundary effect values). Using system-wide values can help in maintaining consistent UI behavior across components and frameworks, and can allow adjustments to the behavior in multiple UI elements at once. For example, inertia effects in multiple UI elements can be changed by adjusting system-wide inertia values.
In one implementation, in order to provide frameworks with access to the reference values of each coefficient, an API is included the ITouchSession module (HRESULT GetMotionParameterValue(IN MotionParameter ID, OUT float*value)). In one implementation, the identifiers and default values for the coefficients whose values are accessible through the ITouchSession::GetMotionParameterValue( ) API are as follows:
The values that are accessible through the API can vary depending on implementation. For example, a UI system that uses the first example boundary effects model described above can omit values such as spring factor, spring power, and damper factor values. Or, a UI system can use additional values or replace the listed default values with other default values. Values can be fixed or adjustable, and can be updated during operation of the system (e.g., based on system settings or user preferences).
In this example, the system 1100 includes a hub module 1110 that provides a declarative description of a hub page to UI control 1120, which controls display of UI layers. UI control 1120 also can be referred to as a “panorama” or “pano” control in a multi-layer UI system. Such a description can be used, for example, when the UI layers move in a panoramic, or horizontal, fashion. Alternatively, UI control 1120 controls UI layers that move vertically, or in some other fashion. UI control 1120 includes markup generator 1130 and motion module 1140.
The declarative description of the hub page includes information that defines UI elements. In a multi-layer UI system, UI elements can include multiple layers, such as a background layer, a title layer, a section header layer, and a content layer. The declarative description of the hub page is provided to markup generator 1130, along with other information such as style information and/or configuration properties. Markup generator 1130 generates markup that can be used to render the UI layers. Motion module 1140 accepts events (e.g., direct UI manipulation events) generated in response to user input and generates motion commands. The motion commands are provided along with the markup to a UI framework 1150. In the UI framework 1150, the markup and motion commands are received in layout module 1152, which generates UI rendering requests to be sent to device operating system (OS) 1160. The device OS 1160 receives the rendering requests and causes a rendered UI to be output to a display on the device. System components such as hub module 1110, UI control 1120, and UI framework 1150 also can be implemented as part of device OS 1160. In one implementation, the device OS 1160 is a mobile computing device OS.
A user (not shown) can generate user input that affects how the UI is presented. In the example shown in
The system 1100 can distinguish between different gestures on the touchscreen, such as drag gestures, pan gestures and flick gestures. The system 1100 can also detect a tap or touch gesture, such as where the user touches the touchscreen in a particular location, but does not move the finger, stylus, etc. before breaking contact with the touchscreen. As an alternative, some movement is permitted, within a small threshold, before breaking contact with the touchscreen in a tap or touch gesture.
The system 1100 interprets an interaction as a particular gesture depending on the nature of the interaction with the touchscreen. The system 1100 obtains one or more discrete inputs from a user's interaction. A gesture can be determined from a series of inputs. For example, when the user touches the touchscreen and begins a movement in UI element in a horizontal direction while maintaining contact with the touchscreen, the system 1100 can fire a pan input and begin a horizontal movement in the UI element. The system 1100 can continue to tire pan inputs while the user maintains contact with the touchscreen and continues moving. For example, the system 1100 can fire a new pan input each time the user moves N pixels while maintaining contact with the touch screen. In this way, a continuous physical gesture on a touchscreen can be interpreted by the system 1100 as a series of pan inputs. The system 1100 can continuously update the contact position and rate of movement. When the physical gesture ends (e.g., when user breaks contact with the touchscreen), the system 1100 can determine whether to interpret the motion at the end as a flick by determining how quickly the user's finger, stylus, etc., was moving when it broke contact with the touchscreen, and whether the rate of movement exceeds a threshold.
The system 1100 can render motion (e.g., motion in a layer, list, or other UI element) on the display differently depending on the type of gesture. For example, in the case of a horizontal drag gesture (in which the user is currently maintaining contact with the touchscreen) on a content layer in a multi-layer UI system, the system 1100 moves the content layer in a horizontal direction by the same distance as the horizontal distance of the drag. In a parallax effect, the title layer and background layer also move in response to the drag. As another example, in the case of a pan gesture (in which the user has ended the gesture) on the content layer, the system 1100 can move the content layer in the amount of the pan, and determine whether to perform an additional movement in the content layer. For example, the system 1100 can perform a locking animation (i.e., an animation of a movement in the content layer to snap to a lock point) and move the content layer to a left or right lock point associated with an item in the content layer. The system 1100 can determine which lock point associated with the current pane is closer, and transition to the closer lock point. As another example, the system 1100 can move the content layer in order to bring an item in the content layer that is in partial view on the display area into full view. Alternatively, the system 1100 can maintain the current position of the content layer. As another example, in the case of a flick gesture (e.g., where the user was moving more rapidly when the user broke contact with the touchscreen) on the content layer, the system 1100 can use simulated inertia to determine a post-gesture position for the content layer. Alternatively, the system 1100 can present some other kind of motion, such as a wrapping animation or other transition animation. The threshold velocity for a flick to be detected (i.e., to distinguish a flick gesture from a pan gesture) can vary depending on implementation.
The system 1100 also can implement edge tap functionality. In an edge tap, a user can tap within a given margin of edges of the display area to cause a transition (e.g., to a next or previous item in a content layer, a next or previous list element, etc.). This can be useful, for example, where an element is partially in view in the display area. The user can tap near the element to cause the system to bring that element completely into the display area.
Various extensions and alternatives to the embodiments described herein are possible.
For example, described examples show different positions of UI elements (e.g., layers, lists, etc.) that may be of interest to a user. A user can begin navigation of an element at the beginning of an element, or use different entry points. For example, a user can begin interacting in the middle of a content layer, at the end of a content layer, etc. This can be useful, for example, where a user has previously exited at a position other than the beginning of a layer (e.g., the end of a layer), so that the user can return to the prior location (e.g., before and after a user uses an application (such as an audio player) invoked by actuating a content image).
As another example, other models can be used to model inertia and movement. For example, although some equations are provided in some examples that approximate motion according to Newtonian physics, other equations can be used that model other kinds of motion (e.g., non-Newtonian physics).
As another example, although controls can share global parameters, such as a global friction coefficient for inertia motion, parameters can be customized. For example, friction coefficients can be customized for specific controls or content, such as friction coefficients that result in more rapid deceleration of inertia motion for photos or photo slide shows.
As another example, boundary feedback can be applied to pinch and stretch gestures. Such boundary feedback can useful, for example, to indicate that a border of the UI element has been reached.
As another example, additional feedback on gestures can be used. For example, visual feedback such as a distortion effect can be used to alert a user that a UI element with zoom capability (e.g., a map or image) has reached a maximum or minimum zoom level.
As another example, boundary effects such as compression effects can themselves produce inertia movement. For example, when a vertically scrolling list is compressed upon reaching the end of the list, and breaking contact with the touchscreen causes the list decompress, the decompression can be combined with a spring or rebound effect, causing the list to scroll in the opposite direction of the motion that originally caused the compression. In this way, the spring effect could provide boundary feedback to indicate that the end of list had been reached, while also providing an alternative technique for navigating the list. The spring effect could be used to cause a movement in the list similar to a flick in the opposite direction. Inertia motion can applied to motion caused by the spring effect.
With reference to
A computing environment may have additional features. For example, the computing environment 1200 includes storage 1240, one or more input devices 1250, one or more output devices 1260, and one or more communication connections 1270. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing environment 1200. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing environment 1200, and coordinates activities of the components of the computing environment 1200.
The storage 1240 may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, memory cards, or any other medium which can be used to store information and which can be accessed within the computing environment 1200. The storage 1240 stores instructions for the software 1280 implementing described techniques and tools.
The input device(s) 1250 may be a touch input device such as a keyboard, mouse, pen, trackball or touchscreen, an audio input device such as a microphone, a scanning device, a digital camera, or another device that provides input to the computing environment 1200. For video, the input device(s) 1250 may be a video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into the computing environment 1200. The output device(s) 1260 may be a display, printer, speaker, CD-writer, or another device that provides output from the computing environment 1200.
The communication connection(s) 1270 enable communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired or wireless techniques implemented with an electrical, optical, RF, infrared, acoustic, or other carrier.
The techniques and tools can be described in the general context of computer-readable media. Computer-readable media are any available media that can be accessed within a computing environment. By way of example, and not limitation, with the computing environment 1200, computer-readable media include memory 1220, 1225, storage 1240, and combinations thereof.
The techniques and tools can be described in the general context of computer-executable instructions, such as those included in program modules, being executed in a computing environment on a target real or virtual processor. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Computer-executable instructions for program modules may be executed within a local or distributed computing environment. Any of the methods described herein can be implemented by computer-executable instructions encoded on one or more computer-readable media (e.g., computer-readable storage media or other tangible media).
For the sake of presentation, the detailed description uses terms like “interpret” and “squeeze” to describe computer operations in a computing environment. These terms are high-level abstractions for operations performed by a computer, and should not be confused with acts performed by a human being. The actual computer operations corresponding to these terms vary depending on implementation.
In example environment 1300, various types of services (e.g., computing services 1312) are provided by a cloud 1310. For example, the cloud 1310 can comprise a collection of computing devices, which may be located centrally or distributed, that provide cloud-based services to various types of users and devices connected via a network such as the Internet. The cloud computing environment 1300 can be used in different ways to accomplish computing tasks. For example, with reference to described techniques and tools, some tasks, such as processing user input and presenting a user interface, can be performed on a local computing device, while other tasks, such as storage of data to be used in subsequent processing, can be performed elsewhere in the cloud.
In example environment 1300, the cloud 1310 provides services for connected devices with a variety of screen capabilities 1320A-N. Connected device 1320A represents a device with a mid-sized screen. For example, connected device 1320A could be a personal computer such as desktop computer, laptop, notebook, netbook, or the like. Connected device 1320B represents a device with a small-sized screen. For example, connected device 1320B could be a mobile phone, smart phone, personal digital assistant, tablet computer, and the like. Connected device 1320N represents a device with a large screen. For example, connected device 1320N could be a television (e.g., a smart television) or another device connected to a television or projector screen (e.g., a set-top box or gaming console).
A variety of services can be provided by the cloud 1310 through one or more service providers (not shown). For example, the cloud 1310 can provide services related to mobile computing to one or more of the various connected devices 1320A-N. Cloud services can be customized to the screen size, display capability, or other functionality of the particular connected device (e.g., connected devices 1320A-N). For example, cloud services can be customized for mobile devices by taking into account the screen size, input devices, and communication bandwidth limitations typically associated with mobile devices.
The illustrated mobile device can include a controller or processor 1410 (e.g., signal processor, microprocessor, ASIC, or other control and processing logic circuitry) for performing such tasks as signal coding, data processing, input/output processing, power control, and/or other functions. An operating system 1412 can control the allocation and usage of the components 1402 and support for one or more application programs 1414. The application programs can include common mobile computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications), or any other computing application.
The illustrated mobile device can include memory 1420. Memory 1420 can include non-removable memory 1422 and/or removable memory 1424. The non-removable memory 1422 can include RAM, ROM, flash memory, a disk drive, or other well-known memory storage technologies. The removable memory 1424 can include flash memory or a Subscriber Identity Module (SIM) card, which is well known in GSM communication systems, or other well-known memory storage technologies, such as smart cards. The memory 1420 can be used for storing data and/or code for running the operating system 1412 and the applications 1414. Example data can include web pages, text, images, sound files, video data, or other data sets to be sent to and/or received from one or more network servers or other mobile devices via one or more wired or wireless networks. The memory 1420 can be used to store a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and an equipment identifier, such as an International Mobile Equipment Identifier (IMEI). Such identifiers can be transmitted to a network server to identify users and equipment.
The mobile device can support one or more input devices 1430, such as a touchscreen 1432, microphone 1434, camera 1436, physical keyboard 1438 and/or trackball 1440 and one or more output devices 1450, such as a speaker 1452 and a display 1454. Other possible output devices (not shown) can include a piezoelectric or other haptic output device. Some devices can serve more than one input/output function. For example, touchscreen 1432 and display 1454 can be combined in a single input/output device.
Touchscreen 1432 can accept input in different ways. For example, capacitive touchscreens detect touch input when an object (e.g., a fingertip or stylus) distorts or interrupts an electrical current running across the surface. As another example, touchscreens can use optical sensors to detect touch input when beams from the optical sensors are interrupted. Physical contact with the surface of the screen is not necessary for input to be detected by some touchscreens.
A wireless modem 1460 can be coupled to an antenna (not shown) and can support two-way communications between the processor 1410 and external devices, as is well understood in the art. The modem 1460 is shown generically and can include a cellular modem for communicating with the mobile communication network 1404 and/or other radio-based modems (e.g., Bluetooth or Wi-Fi). The wireless modem 1460 is typically configured for communication with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between the mobile device and a public switched telephone network (PSSTN).
The mobile device can further include at least one input/output port 1480, a power supply 1482, a satellite navigation system receiver 1484, such as a Global Positioning System (GPS) receiver, an accelerometer 1486, a transceiver 1488 (for wirelessly transmitting analog or digital signals) and/or a physical connector 1490, which can be a USB port, IEEE 1494 (firewall) port, and/or RS-232 port. The illustrated components 1402 are not required or all-inclusive, as components can be deleted and other components can be added.
In view of the many possible embodiments to which the principles of the disclosed invention may be applied, it should be recognized that the illustrated embodiments are only preferred examples of the invention and should not be taken as limiting the scope of the invention. Rather, the scope of the invention is defined by the following claims. We therefore claim as our invention all that comes within the scope and spirit of these claims.
This application claims the benefit of U.S. Provisional Patent Application No. 61/304,004, filed on Feb. 12, 2010, entitled “MULTI-LAYER USER INTERFACE WITH FLEXIBLE MOVEMENT,” which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61304004 | Feb 2010 | US |
