Patent Application
20190317620
The present disclosure relates generally to a user interface and in particular to a user interface element that dynamically adapts to different user input tools.
Recent years have seen a proliferation of touchscreen-based computing devices, including mobile phones, tablets, and hybrid computers operable as a tablet or a laptop. These devices generally incorporate a display area overlaid with a touch-sensitive overlay (e.g., a resistive or capacitive overlay). The display can present a user interface incorporating various input and/or output elements, such as virtual buttons and video, still-image, or text presentations. The overlay can detect contact by an object such as a finger or stylus or the like and can determine the portion of the display that was contacted. Based on the contacted portion of the display and the currently presented interface, the device can infer a user request or instruction.
A graphical user interface element displayed by a computing device can be dynamically adapted to different user input tools, for example a fingertip or a stylus.
The following detailed description together with the accompanying drawings will provide a better understanding of the nature and advantages of the present invention.
Certain embodiments of the present invention relate to computing devices, such as mobile computing devices, that have touchscreen user interfaces operable by different input tools, such as a user's finger or a stylus held by the user. These different input tools can contact the touchscreen with different-sized contact areas. For example, a stylus can have a fine point comparable to a pen or pencil, while a fingertip is comparatively blunt. Consequently, the stylus can contact a smaller area of the touchscreen than a finger. Further, a user may find different gestures more or less comfortable depending on the input tool. For example, while twisting or rotating a finger while keeping the fingertip in contact with the screen may be uncomfortable, twisting or rotating a stylus may feel more natural and comfortable. For reasons such as these, a user input interface that is optimized for fingertip contact may not be optimal for other input tools such as a stylus, and vice versa.
Accordingly, certain embodiments provide graphical user interfaces and/or graphical user interface elements that can be dynamically and automatically altered depending on the type of input tool in use. The type of input tool can be automatically detected, and the detection process can be transparent to the user. For example, a stylus can have a touch sensor built into its body, covering an area where a user's hand is expected to contact the stylus while in use. In response to detecting contact with the touch sensor, the stylus can communicate to the computing device that the stylus is in an “active” state, and the computing device can present a graphical user interface with input elements adapted for stylus input. When the touch sensor of the stylus ceases to detect contact, the stylus can communicate to the computing device that the stylus is in an “idle” state, and the computing device can present a graphical user interface with input elements adapted for fingertip input.
Examples of adaptations for a particular input tool can include changing the number of input elements displayed on the screen (e.g., fewer and larger input elements for fingertip input than for stylus input), changing the arrangement of input elements (e.g., elements placed closer together for stylus input), changing a control-adjustment paradigm (e.g., slider buttons for fingertip input and rotation-based controls such as a virtual set-screw for stylus input), changing the manner of input (e.g., stroke-based input for stylus versus virtual keyboard for fingertip input), and so on. Additional examples are described below.
Other sensors disposed on stylus 102 can include contact sensors 116, 118 disposed at the tip end 120 and/or the reverse end 122. Sensors 116 and 118 can be configured to detect proximity to and/or contact with computing device 100. In some embodiments, tip end 120 can be tapered as shown and can come to a point or nearly to a point (with a small cross sectional area relative to a finger), while reverse end 122 can be blunt or slightly rounded, making the two ends visually distinguishable. The particular size and shape of tip end 120 and/or reverse end 122 can be modified as desired. In embodiments described herein, it is assumed that tip end 120 provides a contact area with touchscreen display 104 that is measurably smaller than an average human finger; however, this is not required. Sensors 116, 118 can be any sensors usable to detect or confirm contact with touchscreen display 104. In some embodiments, sensors 116 and 118 can modify a surface capacitance of tip end 120 or rear end 122 on contact; if touchscreen display 104 is a capacitive touchscreen, this change can be sensed directly by computing device 100.
Stylus 102 can be an active electronic device capable of communicating signals from its sensors (e.g., sensors 114, 116, 118) to computing device 100. Wired or wireless communication channels can be used; specific examples are described below with reference to
It will be appreciated that the computing device and stylus of
A stylus can have any size and form factor. In some embodiments, a stylus can be suitably sized and shaped to be held comfortably in a human hand; however, it is not required that the stylus be handheld or even that a user of the stylus have hands. As long the stylus has sensors able to detect when it is or is not being held or actively used by a user, active and idle states as described herein can be distinguished. For example, in some embodiments, a stylus can include internal accelerometers (or other motion sensors) in addition to or instead of touch sensors; whether the stylus is being held by a user can be determined by comparing accelerometer data to patterns of motions expected when a stylus is being held by a person (including, e.g., small vibration due to the natural unsteadiness of human hands).
Storage device 212 can be implemented, e.g., using disk, flash memory, or any other non-transitory storage medium, or a combination of media, and can include volatile and/or non-volatile media. In some embodiments, storage device 212 can store data objects such as audio files, video files, image or artwork files, information about a user's contacts (names, addresses, phone numbers, etc.), information about a user's scheduled appointments and events, notes, and/or other types of information or data files. In some embodiments, storage device 212 can also store one or more application programs to be executed by processing subsystem 210 (e.g., video game programs, personal information management programs, media playback programs, etc.) and/or one or more operating system programs or other firmware to implement and support various device-level capabilities including generation of graphical user interfaces and processing of user input.
Touchscreen interface 214 can be implemented, e.g., as an LCD display with a touch-sensitive overlay such as a resistive or capacitive overlay and can be operable using various input tools such as fingertip, stylus, and so on. In some embodiments, computing device 202 can also include other user input devices such as a touch pad, scroll wheel, click wheel, dial, button, switch, keyboard, keypad, microphone, or the like, and/or output devices such as indicator lights, speakers, headphone jacks, or the like, together with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors, or the like). A user can operate touchscreen 214 (and any other input devices that may be present) to interact with computing device 202.
Processing subsystem 210 can be implemented as one or more integrated circuits, e.g., one or more single-core or multi-core microprocessors or microcontrollers, examples of which are known in the art. In operation, processing subsystem 210 can control the operation of computing device 202. In various embodiments, processing subsystem 210 can execute a variety of programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can be resident in processing subsystem 210 and/or in storage media such as storage device 212.
Network interface 216 can provide voice and/or data communication capability for computing device 202. In some embodiments network interface 216 can include radio frequency (RF) transceiver components for accessing wireless voice and/or data networks (e.g., using cellular telephone technology, advanced data network technology such as 3G or EDGE, Wi-Fi (IEEE 802.11 family standards), or other mobile communication technologies, or any combination thereof), components for short-range wireless networking (e.g., using Bluetooth standards), GPS receiver components, and/or other components. In some embodiments network interface 216 can provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface. Network interface 216 can be implemented using a combination of hardware (e.g., driver circuits, antennas, modulators/demodulators, encoders/decoders, and other analog and/or digital signal processing circuits) and software components.
Stylus interface 218 can receive communications from stylus 204 and/or send communications to stylus 204. For example, stylus interface 218 can use a wireless communication protocol such as Bluetooth or Bluetooth LE that provides for serial data transfer. A serial protocol with a universe of messages and message formats can be defined. For instance, depending on implementation, stylus 204 can send any or all of the following messages: a “stylus active” message indicating that stylus 204 is in the active state; a “stylus idle” message indicating that stylus 204 is in the idle state; and/or a message containing sensor data from one or more sensors of stylus 204. Other messages or combinations of messages can also be supported.
Accessory I/O interface 220 can allow computing device 202 to communicate with various accessories. For example, accessory I/O interface 220 can support connections to a computer, an external keyboard, a speaker dock or media playback station, a digital camera, a radio tuner, an in-vehicle entertainment system or head unit, an external video device, a memory card reader, and so on. In some embodiments, accessory I/O interface 220 can support wireless communication (e.g., via Wi-Fi, Bluetooth, or other wireless transports). The same wireless transceiver hardware as network interface 216 and/or stylus interface 218 can be used for both networking and accessory communication. Additionally, in some embodiments, accessory I/O interface 220 can include a connector as well as supporting circuitry. In some instances, stylus 204 can connect via accessory I/O interface 220, and a separate stylus interface is not required.
Through suitable programming, processing subsystem 210 can provide various functionality for computing device 202. For example, processing subsystem 210 can execute operating system code and/or application code to render a graphical user interface (“GUI”) image that includes one or more user-operable input elements (e.g., virtual buttons, knobs, sliders, set-screws, data-entry boxes, etc.). Processing subsystem 210 can also receive and interpret messages from stylus 204 via stylus interface 208 and can use the messages to modify the rendering of the GUI image based on the received messages. For instance, as described below, the number, size, arrangement, and/or control type of input elements can be modified. Processing subsystem 210 can also execute other programs to control other functions of computing device 202, including application programs that may be stored in storage device 212.
Stylus 204 can include various sensors such as contact sensors 232, touch sensors 234, and accelerometer(s) 236; a signal processor 238; and a transmitter (or transceiver) 240. Stylus 204 can also include other components (not explicitly shown) such as a battery, power controllers, and other components operable to provide various enhanced capabilities.
Contact sensor(s) 232 (e.g., implementing sensors 116, 118 of
Touch sensor(s) 234 (e.g., implementing sensors 114 of
Accelerometer(s) 236 can be configured to detect motion of stylus 204 and generate corresponding electronic signals. A three-axis MEMS-based accelerometer or the like can be used.
Transmitter 240 can be configured to send messages from stylus 204 to stylus interface 218 of computing device 202. In the example shown, transmitter 240 and stylus interface 218 support wireless communication, and transmitter 240 can include a combination of hardware (e.g., driver circuits, antennas, modulators/demodulators, encoders/decoders, and other analog and/or digital signal processing circuits) and software components. In addition or instead, transmitter 240 can provide a physical connection to computing device 202 and can include a connector, a cable, drivers, and other analog and/or digital signal processing circuits. In some embodiments, transmitter 240 can provide one-way communication from stylus 204 to computing device 202; in other embodiments, a two-way transceiver can be used, and stylus 204 can both receive messages from computing device 202 and send messages to computing device 202.
Signal processor 238 can detect electronic signals from the various sensors including contact sensor(s) 232, touch sensor(s) 234, and/or accelerometer 236 and can generate messages to be communicated to computing device 202 via transmitter 240. In various embodiments, the messages can include a representation of the electronic sensor signals and/or other information derived from the electronic sensor signals such as whether the stylus is in the active or idle state, whether (and what portion of) the stylus is in contact with the touchscreen interface of computing device 202, and so on. Status messages such as battery status information, stylus identifying information, and the like, can also be communicated using transmitter 240.
In some embodiments, stylus 204 can implement a wireless communication protocol (e.g., using transmitter 240) that provides proximity detection based on signal strength. For example, using Bluetooth LE, stylus 204 and computing device 202 can establish a pairing, and stylus 204 (or computing device 202) can use the strength of Bluetooth LE signals from computing device 202 (or stylus 204) to estimate proximity between the devices. Signals from contact sensor(s) 232 in combination with the proximity estimate can be used to determine whether contact with the touchscreen has occurred.
It will be appreciated that the system configurations and components described herein are illustrative and that variations and modifications are possible. The computing device and/or stylus may have other capabilities not specifically described herein (e.g., mobile phone, global positioning system (GPS), broadband data communication, Internet connectivity, etc.).
Further, while the computing device and stylus are described herein with reference to particular blocks, it is to be understood that these blocks are defined for convenience of description and are not intended to imply a particular physical arrangement of component parts. Further, the blocks need not correspond to physically distinct components, and the same physical components can be used to implement aspects of multiple blocks. Blocks can be configured to perform various operations, e.g., by programming a processor or providing appropriate control circuitry, and various blocks might or might not be reconfigurable depending on how the initial configuration is obtained. Embodiments of the present invention can be realized in a variety of apparatus including electronic devices implemented using any combination of circuitry and software.
It should be noted that menu 402 can be displayed when the stylus is in the idle state, and the various control elements presented in menu 402 can be adapted to be easily operated by a user's finger making gestures on a touchscreen, e.g., by making the controls of appropriate size for typical fingertips to unambiguously touch one control and by making the operational gestures correspond to actions a user's fingers can comfortably perform. For instance, a user can slide a finger across slider 410 to adjust its position to a desired location. A user can tap a finger on the appropriate location to select a shadow option from section 406.
A different user interface can be deployed if the stylus is in the active state. For example,
In some embodiments, user interface screen 500 can be presented when the stylus is in active mode, and controls in section 508 and/or menus that are displayed can be adapted for stylus operation. For instance, if a stylus is expected to have a smaller contact area than a finger, control elements in section 508 can be smaller and/or more closely spaced than controls in section 308 of
In addition, control handles 512 can be used to invoke image properties menus. By way of example,
A virtual set-screw (e.g., set-screw 604 or set-screw 606) can be operated similarly to a real set-screw that is adjustable by inserting and twisting a screwdriver. In the case of a virtual set-screw, stylus 102 can act as the “screwdriver.” While the use of a set-screw as a control paradigm may visually cue the user that the control element is operable by rotation, other rotational control elements can be displayed, and in some instances, the option of rotational control need not be visually indicated (examples of implied rotational controls are described below).
Rotational control can be implemented in any user interface, provided that rotational movement of stylus 102 about its longitudinal axis can be detected. For example, as described above with reference to
In some embodiments, the adjustment can be applied in real time in response to the rotational movement of the stylus. For example, there can be a mapping between the change in the setting and the amount of rotation detected. In some instances, the mapping can also take into account the speed of rotation (e.g., adjusting a value by larger amounts for a given amount of rotation if the rotation is made at high speed and by smaller amounts if the rotation is made at low speed).
In addition, value indicators 608, 610 (which indicate the current value for the corresponding setting) can be updated in real time. In some embodiments, a set-screw or other rotational control element can provide a fine control over the values of various settings.
In some embodiments, when the stylus is in the active state and in contact with the surface, concurrent rotational and translational motions of the stylus can be detected and used to concurrently control or adjust different properties of a displayed object. For example, referring to interface screen 500 of
As another example, a user can use simultaneous translation and rotation of a stylus to fine-tune a drawn object.
In this example, object 904 is a curved object with a path defined using Bezier curves. Bezier curves are defined by a series of control points 910, each of which has an associated tangent line (not shown in
In certain embodiments of the present invention, translational motion of a stylus can be used to adjust the position of a control point of a Bezier curve while rotational motion can be used to adjust the orientation angle of the tangent line.
For instance,
As the examples of
It will be appreciated that the user interface screens of
The same application can present different user interface screens, e.g., depending on whether the stylus is in the active state or the idle state. In some instances, an application can dynamically switch to a different interface screen if the state of the stylus changes. For instance, a presentation editing application can present interface screen 300 for as long as the stylus is in an idle state and switch to interface screen 500 if the stylus enters the active state.
At block 1202, process 1200 can obtain stylus data. For example, as described above, stylus 102 can send a signal to computing device 100, and process 1200 can receive this signal. At block 1204, process 1200 can determine the state of the stylus (e.g., active or idle). For example, as described above, the signal sent by stylus 102 can include a message indicating the stylus state, which stylus 102 can determine based on sensor readings. As another example, stylus 102 can send a signal that includes the sensor data, and process 1200 or another process on computing device 100 (e.g., an operating system process) can determine the state of the stylus based on the received signal.
At block 1206, a determination is made as to whether stylus 102 is in the active state. If so, then at block 1208, the user interface can be rendered in a “stylus mode.” For example, the code for rendering the user interface can include conditional statements that can cause certain input elements to be rendered in a different manner (e.g., slider versus set-screw) or to be rendered or not rendered (e.g., control handles 512 of
If, at block 1206, stylus 102 is not in the active state (e.g., if stylus 102 is in the idle state), then at block 1210, the user interface can be rendered in a “finger mode.” For example, as noted above, the code for rendering the user interface can include conditional statements governing how input elements are rendered, and block 1210 can include executing the same code as block 1208, with the difference in condition resulting in a different rendering.
The difference between user interfaces rendered at blocks 1208 and 1210 can include a difference in the rendering of at least one user-operable input element of the user interface. For instance, the stylus mode (block 1208) can provide a rendering of the element that is based on an expectation that a stylus will be used to operate the element, while the finger mode (block 1210) can provide a rendering of the element that is based on an expectation that a finger will be used to operate the element. Thus, for example, an element can be rendered at a smaller size in the stylus mode than in the finger mode. An element can be rendered for a different type of operation in the stylus mode as opposed to the finger mode (e.g., an adjustable control can be rendered as a set-screw in the stylus mode and as a slider in the finger mode). The renderings can also differ in the number and placement of user-operable input elements.
It is to be understood that rendering a user interface or user interface element in a mode associated with a particular input tool (e.g., a stylus mode associated with a stylus) need not preclude the use of other input tools (e.g., a finger) to interact with the interface or element. Accordingly, a user might hold the stylus in one hand and touch the screen with fingers of the other hand, or the user might hold the stylus in a manner that leaves one or more fingers available to touch the screen, or one user might be holding the stylus while another user touches the screen with fingers. In such instances, the graphical user interface can remain in stylus mode for as long as the user continues to hold the stylus. Alternatively, depending on how the touch sensors of the stylus are arranged, it can be possible to determine whether the contact with the user's hand corresponds to holding the stylus in an orientation associated with actual use (e.g., a contact pattern associated with holding a pencil or pen) or a non-use orientation (e.g., wedged between two fingers), and the mode to be used for rendering the interface can be selected accordingly.
In some instances, a stylus might not be present. For instance, in embodiments described above, a computing device can detect the presence of a stylus based on transmissions from the stylus; if no transmissions are received, then the stylus can be considered to be absent. This case can be handled, e.g., by assuming the idle state for the stylus in the absence of transmissions. As another example, in some embodiments, the stylus and the computing device can establish a pairing (e.g., a Bluetooth pairing), and the computing device can ignore any status-related messages from any stylus with which it is not paired. Thus, a user's experience need not be affected by the mere presence of a stylus in the vicinity; a particular device would enter the stylus mode only if it is paired with a stylus that is in the active state.
It will be appreciated that process 1200 is illustrative and that variations and modifications are possible. Steps described as sequential may be executed in parallel, order of steps may be varied, and steps may be modified, combined, added or omitted. For instance, the determination of the state of the stylus can be made by the computing device or the stylus.
At block 1302, the control logic of stylus 102 can receive sensor data (e.g., from touch sensors 114 and/or internal motion sensors). At block 1304, the sensor data can be analyzed to determine whether stylus 102 is in a user's hand. For example, touch sensor data can be analyzed to detect a pattern of contact areas consistent with a human hand. In some embodiments, the analysis can include comparing the pattern of contact areas to patterns associated with a human hand in a particular pose (e.g., holding a pencil or pen in order to write). As another example, motion sensor data can be used to detect movement consistent with being held by a person as opposed to being at rest on a surface. Such distinctions can be based on amplitude and/or frequency analysis of the accelerometer data. In some embodiments, a combination of touch sensor and motion sensor data can be used.
At block 1306, if it is determined that stylus 102 is in a user's hand, then at block 1308, stylus 102 can transmit a signal to computing device 100 indicating that it is in the active state. If, at block 1306, stylus 102 is not determined to be in a user's hand, then at block 1310, stylus 102 can transmit a signal to computing device 100 indicating that it is in the idle state.
Process 1300 can be performed iteratively, e.g., at regular intervals such as every 0.1 seconds or every 0.01 seconds, or every 0.001 seconds, and stylus 102 can periodically report its state. In some embodiments, transmission of a signal at block 1308 or 1310 occurs only if the state has changed since the last execution of process 1300. Where transmission occurs on a state change, computing device 100 can assume that the state is unchanged until the next transmission. In some embodiments, computing device 100 can send a message to stylus 102 requesting the current state, and process 1300 can be executed in response to such a request. For example, computing device 100 might send a state-requesting message if it has not received a state message from stylus 102 within some “heartbeat” period, or computing device 100 might send a state-requesting message if it is preparing to render a new user interface or otherwise update the displayed image.
It is to be understood that process 1300 can be but need not be performed by the stylus. In some embodiments, stylus 102 can transmit its sensor data to computing device 100 without analysis, and computing device 100 can perform the analysis and determine whether the stylus is in the active or idle state.
As noted above, stylus state can be determined based on sensors that respond to touching of particular surfaces of the stylus (e.g., a barrel portion in the case of an elongate stylus as shown in
While the invention has been described with respect to specific embodiments, one skilled in the art will recognize that numerous modifications are possible. The computing device and the stylus can vary as to form factor, size and components. Any computing device having a touchscreen display can be used. A stylus can include any tool that can operate a touchscreen display of a computing device by direct contact or near-contact and that is capable of communicating sensor data and/or state information to the computing device. In the examples shown above, a stylus communicates wirelessly with the computing device, e.g., using Bluetooth or Bluetooth LE or the like. However, a wired connection can be implemented. For example, a stylus can be connected into a cable that plugs into a receptacle connector in the housing of a computing device; the cable can be sufficiently long and flexible to avoid interfering with use of the computing device. Where a wired connection is used, the stylus can draw power from the computing device or from its own internal power source. In the case of a wireless connection, the stylus can have its own internal power source; in some instances, it may be possible to physically connect the stylus to the computing device for recharging of the stylus, and in some instances, the stylus may support both wired and wireless communication interfaces.
In some embodiments, the stylus can include one or more user-operable input controls such as buttons, switches, dials or the like, and communication from the stylus to the computing device can include messages indicating the status and/or operation of any input controls that may be present on the stylus.
In embodiments described above, a computing device can use the active or idle state of a stylus to affect rendering of a user interface. This state information can also be used for other purposes. For example, if the stylus is in the idle state, the stylus can also reduce its power consumption, e.g., by powering down some of its sensors and discontinuing or reducing transmissions to the computing device. At least one sensor can remain operational in order to detect a possible transition to the active state, and this event can result in waking up other sensors that can confirm the transition. As another example, if the user picks up the stylus while the computing device is in a low-power state (e.g., with its touchscreen display turned off), the stylus can notify the computing device, and the computing device can transition to a ready-to-use state, e.g., turning on its touchscreen display.
Embodiments of the present invention can be realized using any combination of dedicated components and/or programmable processors and/or other programmable devices. The various processes described herein can be implemented on the same processor or different processors in any combination. Where components are described as being configured to perform certain operations, such configuration can be accomplished, e.g., by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation, or any combination thereof. Further, while the embodiments described above may make reference to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and/or software components may also be used and that particular operations described as being implemented in hardware might also be implemented in software or vice versa.
Computer programs incorporating various features of the present invention may be encoded and stored on various computer readable storage media; suitable media include magnetic disk or tape, optical storage media such as compact disk (CD) or DVD (digital versatile disk), flash memory, and other non-transitory media. (It is understood that “storage” of data is distinct from propagation of data using transitory media such as carrier waves.) Computer readable storage media encoded with the program code may be packaged with a compatible electronic device, or the program code may be provided separately from electronic devices (e.g., via Internet download or as a separately packaged computer-readable storage medium).
Thus, although the invention has been described with respect to specific embodiments, it will be appreciated that the invention is intended to cover all modifications and equivalents within the scope of the following claims.
This application is a continuation of U.S. application Ser. No. 15/607,186, filed May 26, 2017, which is a continuation of U.S. application Ser. No. 14/616,532, filed Feb. 6, 2015, now U.S. Pat. No. 9,665,206, which is a continuation of U.S. application Ser. No. 14/030,682, filed Sep. 18, 2013, each of which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15607186 | May 2017 | US |
| Child | 16445108 | US | |
| Parent | 14616532 | Feb 2015 | US |
| Child | 15607186 | US | |
| Parent | 14030682 | Sep 2013 | US |
| Child | 14616532 | US |
