Patent Application
20120105367
Described herein are novel methods of designing a user interface for electronic devices using proportional force information. The new user interface is more intuitive, easier to use and requires less finger manipulations. These methods are novel in part because reliable input sensors capable of detecting a proportional force, i.e. how hard the user presses a button, have not yet been widely available.
Touch screen technologies have evolved in both cost and functionality that allowed them to expand into new markets such as personal mobile devices for small touch displays and all-in-one large computers that feature touch displays of large size. Newer and faster integrated circuit controllers that form a computing foundation for touch screen capabilities have allowed increasingly complex and novel improvements in user experience as well as development of new applications. One specific improvement that has been extensively developed and broadly adopted by consumers is the ability to provide simultaneous multi-point touch input.
Multi-point touch can be categorized as either a dual-touch or a true multi-point touch. In dual-touch applications the touch screen digitizer is typically configured to calculate the midpoint of two independent simultaneous touch locations. Typically the user places a thumb and pointer finger of the same hand or one finger of each hand on the screen and moves them independently. Usable input parameters provided by dual-touch manipulation are typically the midpoint and distance between the two touch locations. This information provides a new level of input data in the X and Y plane for the user interface developer allowing contemplation of novel applications for user interface design. Similar to dual-touch features, a true multi-point touch provides similar input parameters of midpoint and distance but also includes discrete x,y coordinates for each finger. Multi-point touch input is capable of providing data points for more than 2 input locations and may go up to 10 or more depending on the end use case. For example, a large wall display can be configured to have two or more persons using all their fingers on the same screen for typing on virtual keyboards all at the same time.
These technology improvements as described above have provided increased functionality and have opened a new age of interactive user experience and functionality. However, these improvements try to utilize distance and movement between touch locations as a way to simulate three-dimensional inputs on a two-dimensional sensing platform. Almost all touch screen implementations today provide only X and Y coordinate input and cannot provide a true 3-dimensional input of dynamic X, Y, and Z space. In some ways, this places an artificial restriction on how a user can ultimately interact with touch screen devices. Explained below are a few of the more popular and basic user functions to illustrate an idea of how a true 3-dimensional input capability can transform the user experience to a new level of interaction with the device and its intuitive control.
Accordingly, it is an object of the present invention to overcome these and other drawbacks of the prior art by providing novel methods of operating user input devices configured to provide locations and force of touch measurements.
The methods of the invention allow operating an input device such as a smartphone front panel. The input device itself needs to be configured for detecting at least one location of touch as well as measuring a force of touch at this location, for example as done by a tactile pressure sensor array. Such array is typically adapted to sense capacitance between two electrode layers.
In embodiments, at least two events defining an output of the input device are provided for a particular location. Selection of one event or the other is done based on a force of touch being either above or below a predetermined force of touch threshold. This force of touch threshold is selected to be within the operational range of the touch screen defined as above the initial detection level of force and below a level of force saturation.
In other embodiments, more than one force of touch threshold may be provided for one or more locations. In that case, a corresponding number of events may be provided such that depending on the measured level of force of touch a certain output even is selected.
Yet in other embodiments, additional selection criteria may be used such as duration of time during which the force of touch was above or below a certain threshold.
Subject matter is particularly pointed out and distinctly claimed in the concluding portion of the specification. The foregoing and other features of the present disclosure will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only several embodiments in accordance with the disclosure and are, therefore, not to be considered limiting of its scope, the disclosure will be described with additional specificity and detail through use of the accompanying drawings, in which:
The following description sets forth various examples along with specific details to provide a thorough understanding of claimed subject matter. It will be understood by those skilled in the art, however, that claimed subject matter may be practiced without one or more of the specific details disclosed herein. Further, in some circumstances, well-known methods, procedures, systems, components and/or circuits have not been described in detail in order to avoid unnecessarily obscuring claimed subject matter. In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated and make part of this disclosure.
The main idea of the invention is to provide a method of operating an input device, in which the device is configured for detecting at least one location of touch as well as measuring a force of touch at this location. A tactile pressure sensor array based on sensing capacitance between two electrode layers may be used as an example of such devices. Such a sensor array may be designed as a two-dimensional matrix capable of detecting having an X and a Y coordinate of one or more locations of touch and at the same time the sensor may be configured to independently and simultaneously measure the force of touch at each touch location.
One novel aspect of the invention is measuring the force of touch and categorizing it to be either above or below at least one predetermined force of touch threshold. That force of touch threshold is selected to be within the operational range of the touch-sensitive input device. Depending on whether the level of measured force of touch falls into a first or a second range (defined as above or below that threshold), the device may be configured to select either a first event or a second event as an output of the input device. In embodiments, more than one threshold may be used so that more than two events may be used for selecting the output of the device as illustrated in more detail below.
In yet other embodiments, the absolute level of force may be used as an input parameter if it falls above or below at least one predetermined force of touch threshold. In further embodiments, if the force of touch falls into a predetermined continuous measurement interval, the response of the input device may be selected accordingly. In yet other embodiments, the force of touch may be measured continuously and the change in that force may be used as defining the output as being either a first or the second event. For example, if the force of touch at a particular location is changing to cross over a predefined threshold, this may be used to select the event defining the output of the input device.
The method of the invention in its most general form comprises a step of selecting a first event or a second event as the output of the input device based on detected location of touch and the level of force of touch at this location being above or below at least one predetermined threshold.
The pinch gesture feature provides a way to implement a dynamic depth function for viewing pictures or documents either up close or far away. The present invention provides for a 3rd dynamic data range input to correlate between zooming in or out. The user may be asked to place two fingers on the display to activate the zoom in/out function. Examples of implementing this function according to the prior art are as follows:
According to the present invention, the user experience can be greatly enhanced by using a single finger instead of using multiple fingers to simulate a dynamic depth input parameter. Similar to using a finger in 3D space, force sensing allows the user to “push” the picture away or “pull” the picture in closer. The degree of zoom may be defined by the level of force that is exerted on the touch screen. In addition, the user is capable of using a single finger to simultaneously pan, zoom and rotate, just like one would do when manipulating or pushing an actual object.
The copy/paste feature is essential for editing or preparing documents and emails with content from several sources such as other documents, emails, multi-media content, pictures, internet postings or web pages. This and similar functions require multiple touch events and sometimes complex manipulation of content on the screen. The sequence of events includes highlighting and selecting content, followed by moving selected content, then followed by saving or deleting the content as desired. Several popular user interfaces and touch screen technologies of the prior art allow this function to be accomplished in the following way:
With a true 3D capability afforded by the methods of the present invention, the user experience can be greatly enhanced by using a single finger instead of using multiple fingers or multiple operations to select and manipulate target content. This intuitive operation can only be accomplished with a dynamic depth input parameter in the Z-axis through force input. Similar to using a finger in 3D space, the force of touch sensing allows the user to immediately select and highlight the desired content at one time. One way to accomplish this may be configured using multiple force levels to define different events at the same location of touch. A lighter force of touch may be used to highlight all of the desired content, while a subsequent heavy or quick push after selection may be used to simulate a grab (or final selection) of the target content. The content may then be automatically placed into the device memory to be used later.
A primary example for using this feature is with a multi-media content such as a virtual album of songs and group of videos. Current touch screen technologies only provide a one-dimensional control function and at most a two-dimensional cursor control for scrolling through content. In addition to basic searches, lists can include additional levels, activated through force threshold events, that describe their content in greater detail, which should also be considered. For example, a song can be categorized by genre, artist, year, etc. or additional background information on a song or artist can be linked to each listed item. One essential user enhancement for scrolling may be controlling the rate of scanning through a list. Several methods of the prior art are described as capable of changing a rate of scrolling or other control functionality:
A circular scroll gesture which allows the user to continuously rotate the finger as if on an iPod wheel is better than pressing a button multiple times in a repeated fashion. However, for very large list of songs the number of times the person has to make a complete circular motion becomes burdensome so time-based acceleration is implemented in the prior art—a number of songs scrolled for one revolution increases after several rotations. Similarly, a button interface can implement the same type of acceleration features when holding down the button such that the list steps through the songs with increasing speed. The problem with this implementation is that there is no easy way to decelerate and so often times the user has to concentrate on looking at the fast scrolling information to try and stop as close to the desired location as possible and correct for either an overshoot or an undershoot. The flick gesture for scrolling through a list maybe fun, but it is not a very accurate way to reach the desired song.
The present invention improves the user experience by allowing using a single finger for this function. Acceleration function of the prior art was designed individually depending on the type of list being searched or on how the scroll feature is designed. Having the capability to simulate a dynamic depth input parameter may be used to provide an intuitive experience. The rate of scroll may be either increased or decreased based on the level of force of touch. In embodiments, one can increase the rate of search by gently increasing the pressure exerted on the touch surface. In other embodiments, if there are several identifying levels to a list, such as rock and country songs, one can use different tap thresholds to change type of songs within a specific list. Yet in other embodiments, a combination of force level and duration can enable purchase requests or background information can be displayed for each item on a list.
A typical conventional user input device, such as a mouse or touch pad, uses a switch to provide actionable input. In this case, the switch is either pressed or released, which can be considered as a “binary input”.
The application framework or operating system may generate different events based on these user inputs, which define instances where the application may optionally execute certain functions. Table 1 shows an order in which events may be generated by a mouse or other input device for a typical implementation.
With each event, the application framework or operating system typically provides the x,y location at which the event occurred, as well as optional additional information, such as the state of mouse buttons, keyboard buttons, scroll wheels, etc.
The events “Click” and “Double Click” are a somewhat special case, as they may be placed in different locations within the event order depending on the particular framework or application design. For instance, the “Click” event could be generated right after the “Down” event to respond when a mouse button is pressed, or it could be generated right after the “Up” event to respond when the mouse button is released. Applications may also have different criteria for events such as “Click” or “Double Click,” such as whether or not to respond if the mouse button is released when the input pointer has moved outside the control region, or what sort of time is required between successive clicks to generate a Double Click event—see
A “tactile control” method of the invention includes an input region in which at least one or even several specific force of touch thresholds may be defined to generate events associated with user input, where an “event” is a set of functionality which is executed when its operating conditions are met. By selectively defining different combinations of thresholds and events, a tactile control can be used to implement a wide variety of user input, from the very simple mimicking of a button to very complex force-sensitive gestures.
A “tactile control” is then defined as a region of input space in which one or more force of touch thresholds are defined, each of which is associated with its own set of events. An example of a set of tactile events is given in Table 2.
With each, the application framework or operating system may provide the level of the applied force in addition to the X and Y position and other standard information typically collected by such systems.
This series of events is very similar to those generated by a binary input control, with two key additions. “Move” is identical in concept to the traditional Move event generated when the pointer location changes while over a control. “Force” is analogous and is caused when the applied force of touch to the control changes. “Active” can optionally be called whenever new tactile data is generated, which would typically be on a continuous basis.
One benefit of the “Active” event is that it enables different types of input gestures that may be time-dependent as well as force- and/or location-dependent. For example, many desktop applications feature “tool tips”, which appear when a user holds the pointer over an icon or other control. If the pointer remains still for a sufficient time, the tool tip is displayed to provide additional information, and when the pointer moves, the tool tip disappears. A tactile equivalent may be to hold the force over a threshold for a specific amount of time which may be typically shorter since the force level also helps to identify the desired action, at which point additional information may be displayed, or even different control functionality offered.
As with a typical binary input control, the “Click” and “Double Click” events of the invention may be defined to trigger on either the rising or falling edge of activation, depending on what is most appropriate for a particular application.
To avoid accidental repeated activations, a tactile control may further include a value for “hysteresis” in addition to the activation threshold. The “Positive Edge” event may be triggered when the force of touch rises above the predetermined activation force of touch threshold, while the “Negative Edge” event may be triggered when the force of touch drops below the activation threshold minus the amount of hysteresis. This allows the control to not respond to noise in the force signal generated either by electrical interference or by a non-smooth input from a user.
In addition to registering event handlers and setting the force of touch threshold and hysteresis, there are several other parameters that may be implemented to fit the needs of a particular application. For example, having multiple thresholds may be associated with multiple sets of Move/Force/Active events at each threshold. An implementation would have to define whether such events would be passed on to successive thresholds or not, as well as the order in which to process them.
A tactile control may need to implement a standard pushbutton operation, which is easily accommodated by the described framework. This may be done simply by defining a single threshold without using Force or Active events. The threshold would be chosen based on how much force was desired to activate the control. As with a traditional input device button, the application framework would determine whether the “click” event was activated on the rising or falling edge—see
To move a physical item (for example a brochure located on a desk) with a finger, the user needs to apply sufficient force to overcome friction between the brochure and the desk. This means that a light touch will cause the finger to slide over the brochure, but pressing harder will cause the brochure itself to move across the desk. This natural behavior of real-world objects may be simulated in a user interface by measuring the force that the user applies onto the input device surface. While an ability to move the cursor is not as widely applicable in touch screen applications since the pointer location is taken directly from the screen rather than from a pointing device, this ability could prove important for touchpads and for applications where the cursor location may change the way in which an object behaves.
In embodiments, tactile controls may be overlaid to provide multiple functions in the same screen area by defining different activation thresholds. For example, an intuitive drawing application could allow drawing lines in which thickness may be based on the amount of pressure applied over the entire touch screen area. At the same time, different controls for changing the color or style of the line may be located within the drawing area and associated with higher activation thresholds than the drawing area itself so that the entire screen could be used without accidentally activating any of the controls—see
Yet in other embodiments, a virtual stack of items on a display screen can be manipulated individually or together depending on the level of force applied. For example, if a stack of virtual playing cards is displayed on a screen and a player has a choice of picking one, three, or five cards from the stack, three force levels may be used to simulate the increased friction between the cards so the appropriate number of cards are chosen from a single movement. Another embodiment could be in an e-Reader device, where the amount of force applied in a “swipe” motion may determine how many pages would be turned, a gesture directly mimicking the type of physical gesture used when browsing the pages of an actual book.
In embodiments, the function of select, copy/cut, and paste text may be realized by using light pressure to control the cursor location at the start of the selection and then using harder pressure for text selection such that the end location can be determined by the user. Once the text block has been highlighted, higher pressure applied to the selected text itself may allow the text to be moved to the desired location. A double-click may be used to implement copy or delete functionality as required by the application.
To implement this functionality in a tactile control, two thresholds may be defined over the entire text area, one for a Select and one for a Copy/Cut action.
For the a cut/copy threshold, a time duration parameter may be specified to determine whether a Double Click action had been performed to change from a Cut mode to a Copy mode. A Double Click in this case would be defined as two rising edges within the specified time duration.
For applications where clipboard-like functionality is needed to store cut or copied data, the applied force may be used to indicate the “paste” location, for instance by detecting a higher-force “click” gesture within a body of text where nothing was currently selected—see
Another useful user interface that can be implemented with proportional force sensing is the ability to scroll through long lists such as a phone list or songs in a precise manner. Two buttons are used to determine the direction of scroll, but because the level of force is detected, the speed may be determined based on the force that the user applies thereto. This would allow a hard press to scroll very quickly until the approximate region was reached, and then by softening the press, the user could slow down for easier selection of a specific item. The same control button may then be used to select the item in the list. The advantage of this arrangement is that the user works less and reaches the desired selection more quickly.
A tactile control method of the present invention may also be used to vary the scroll speed based on the amount of force applied to the control. To implement this, two thresholds are defined, one for scrolling and one for selecting. For the scrolling threshold, events are defined for Rising Edge, Falling Edge, and Active. A time duration may be further specified which must elapse before the force-sensitive scrolling becomes activated—see
The timer may allow using a gentle “tap” to scroll by one item or a longer press to initiate a force-sensitive scrolling. This timer may be much shorter than the timers used to change the scroll speed purely based on time. At the same time, it allows other functions such as tap for advancing one song at a time or a quick hard press that selects the desired song. The degree of sensitivity of the scrolling speed may be mapped to the applied force and may be adjusted based on the needs of the application.
For the selecting threshold, the level higher than the scrolling threshold may be set which defines a single event for Rising Edge. Also, while the user is scrolling, the action of pressing harder and exceeding the selected force of touch threshold levels would not activate selection since the user should stop and confirm the selection.
While the scrolling threshold would not need any other events, it may prevent any passing along to the Scrolling Threshold event handlers to avoid inadvertent scrolling when trying to select. An alternative implementation would be to implement the “Click” event for the Select Threshold.
Mobile and other devices are frequently used to examine very large images at varying degrees of zoom, such as in the case of navigation, where a map may need to be zoomed in or out and panned in various combinations to achieve the desired view. One of the most successful user interface gestures is the pinch gesture which allows a graphical object to be zoomed in or out based on two fingers coming together for zooming in and spreading apart for zooming out. While this gesture has enabled much-enhanced abilities for the user to manipulate a map or a photo, it does require two hands to operate on a mobile device since one hand is used to hold the device while the other hand makes the gesture. This can be a problem in situations where both hands are not available, such as when carrying luggage or driving a car. Another limitation of the pinch gesture is that it requires multiple repeated gestures to zoom in from a very large area of the map to a very detailed region.
The present invention uses the ability to measure the force that the operator applies to zoom in or out of an image such as a map or photo. It requires a one-finger contact and thus allows a one-handed operation, for example when the mobile device is held by the fingers and the thumb makes the contact with the surface. A low-level force is used to locate the contact of the finger to the graphical image (thus allowing the user to pan the image) and a high-level force controls the zoom function. In addition to advantageous one-handed operation, this gesture does not require multiple gestures to zoom in from a large area to a detailed region thus saving the user effort. The proportional control of the zoom function further allows the user to control the speed of zoom allowing to precisely selecting the desired view.
To zoom out, a separate region on the screen may be designated as a zoom-out button or a simple tap and press changes the zoom direction from zooming in to zooming out.
With tactile control methods of the invention, the different gestures may be combined together so that a single finger may be used to do both panning and zooming simultaneously. To implement this on a tactile control of the invention, two force thresholds may be used, one for panning, and another for zooming. A light touch may initiate pan only. A harder touch may activate zooming and panning, with the degree of zoom determined by the level of applied force.
The Pan Threshold control may only need a single event to detect changes in location while the applied force was at or above a predetermined force of touch threshold.
For the Zoom Threshold control, a time duration parameter may be specified to determine whether a tap action had been performed to change the zoom direction. A Double-Click in this case may be defined as two rising edges within the specified time duration.
The Double-Click event in this case may be activated after the Rising Edge event, so that the default behavior may always be to zoom in with increasing force. Double-Clicking may switch to zooming out until the control is released, at which point the zoom direction would revert back to zooming in—see
Different activation thresholds of tactile controls may be adjusted over time according to the present invention allowing the overall force-sensitivity of a tactile input device to accommodate different users' grasp and input capabilities. Such adaptive functionality may increase usability of touch-enabled devices.
Determining the appropriate force thresholds for the various types of input gestures available in a tactile control may be based on one or a combination of four different sources:
Default values may be determined by having a large population of users perform a set of gestures on a tactile input device, and then selecting thresholds that would accommodate the largest percentage of users.
A “gesture training” program may be used on a particular device to configure input thresholds for a specific combination of device and user. Optionally, the resulting data may be anonymously uploaded to the application developer to provide increased population data for determining default values, which may be “pushed” to other devices.
One limitation of calibration-type training programs is that users may not use the same motion or gesture as they may use when just using the device naturally. To help compensating for this limitation, a tactile input device of the invention may continually monitor the force inputs used on various types of tactile controls to determine automatically when adjustments may be necessary.
One method of adjusting thresholds according to the invention may include analyzing the actual applied force of touch for all inputs of a specific type. For example, if the “button” activation force is set initially at 300 g, but the device consistently measured that the user always used an actual force of 500 g or more when using button inputs, the force of touch threshold may be appropriately increased—see
Another method may involve analyzing repeated gestures to detect errors. For example, if the activation force for tactile buttons is set initially at 400 g and the device detected that many button “clicks” were preceded by a peak force of 350 g on the same button, it may determine that for that user a force of touch threshold need to be reduced to below 350 g instead—see
A particular tactile input device may implement one or all of these different methods of setting the event activation thresholds so as to better suit a particular user's needs.
Advantages of the above described methods include:
Smartphones may eventually replace hand-held gaming controls and consoles. One limitation of the contemporary smartphone is lack of accelerometer and in some cases lack of touchscreen when compared with hand-held gaming controls. In embodiments of the present invention, force-measuring sensors may be adapted to provide desired accelerometer function by measuring the level of the force of touch. In one example, a racing game may be improved by providing an acceleration control button responsive to the actually measured level of touch. Pushing harder on this button may activate a car or another object to move faster on the screen.
The advantage of this approach is achieving expanded functionality to facilitate rich gaming experience—but without increasing the physical size of the device or requiring other control hardware to be used concurrently with the main control panel. This makes using smartphones advantageous for gaming purposes.
A gloved finger presents a challenge to the present touch-screen input devices as capacitance measurement becomes problematic. The present invention provides for at least basic control function using a gloved hand by measuring a force of touch at a particular location. Using force sensors at the corners of the screen may allow calculating the location of touch by knowing the forces at all four corners—even without measuring such location using capacitance principles.
Tactile controls and methods of the invention may further be helpful in improving handwriting while in the drawing mode. In comparison to a fixed line width, adding the force allows the line width to vary similar to how a paint brush stroke, pencil or a pen works on paper thus providing more realistic signature and creative freedom.
The invention describes a very general approach to interpreting tactile input data and may accommodate a wide variety of different types of input hardware and system platforms.
In its most general sense, the force gesture method described herein may be used with any hardware whose sensor data may be used to detect and collect one or more data points, each such data point including information about location and force of touch. This may be viewed as somewhat analogous to a computer mouse configured for generating location data plus the state of each of its buttons. The translation of the sensor inputs to location and force of touch may be provided by the hardware driver for a particular operating system, or it may be done by the GUI framework or even the application itself if the raw sensor data is made available. A sample implementation architecture concept is shown in
Many different types of hardware may be used to generate the location and force of touch data, including but not limited to the following examples:
The herein described subject matter sometimes illustrates different components or elements contained within, or connected with, different other components or elements. It is to be understood that such depicted architectures are merely examples, and that in fact many other architectures may be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality may be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated may also be viewed as being “operably connected”, or “operably coupled”, to each other to achieve the desired functionality, and any two components capable of being so associated may also be viewed as being “operably couplable”, to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable and/or physically interacting components and/or wirelessly interactable and/or wirelessly interacting components and/or logically interacting and/or logically interactable components.
Although the invention herein has been described with respect to particular embodiments, it is understood that these embodiments are merely illustrative of the principles and applications of the present invention. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims.
This application claims priority benefit from a provisional application No. 61/408,737 filed 1 Nov. 2011 with the same title, which is incorporated herein in its entirety by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61408737 | Nov 2010 | US |
