Patent Application
20100138766
The present disclosure relates to methods, systems, and techniques for user interface improvements and, in particular, to user interfaces for using accelerometer data.
User interfaces on mobile devices such as cell phones, smart phones, PDAs, etc. are faced with limitations not necessarily present on larger more stationary devices such as personal computers. For example, the extremely small display screen size inherently limits how much a user can view on the display screen at any one time. Such limitations may be due in part to the scarceness of resources of the device such as battery life. Programmers of device software for such mobile devices may be encouraged to refrain from using system resources (for example, compute power behind a user interface) for too long.
In addition, sound user interface design principles caution from displaying too much at once to a user to limit the perceived crowdedness and visual noise caused by presenting too many objects at once to a user. As a result, user interfaces for such devices commonly present commands through multiple layers of menus, which require a user to learn them and possibly invoke many input “strokes” to accomplish a task.
Recent designs in user interfaces have made mobile devices such as Apple's® iPhone™ more user friendly by displaying content in portrait or in landscape mode in response to a user changing the orientation of a device from portrait to landscape and visa versa. While such features are useful, they do not address the problems of mobile devices described.
Embodiments described herein provide enhanced computing-based methods, systems, and techniques for implementing user interfaces on computing devices typically with small display screens, such as mobile devices. Example embodiments provide a gravity-based user interface mechanism (a “GBUIM”), which enables users to invoke user interface (UI) controls and capabilities using a “tilt” mechanism without having to display most, or even all, of the user interface controls on the display screen prior to making them available for use. This allows, for example, applications written for mobile devices to utilize the full display screen real estate for content that relates to their respective primary purposes without the clutter of user interface controls for manipulating such content.
Example embodiments operate in conjunction with accelerometer information, which provides substantially real-time or near real-time orientation information, to offer enhanced UI functionality. More specifically, according to one example embodiment, when the user tilts the mobile device up or down at varying levels (e.g., the top of the device viewed in portrait mode is rotated forward or backward), user interface controls may be presented, such as by overlaying or replacing the content currently displayed on the mobile device display screen. (This rotation may be thought of as rotation along a transverse axis, such as “pitch” in flight dynamics terms.) According to another example embodiment, when the user tilts the side of the mobile device in one direction at varying levels (e.g., left side down) or in the opposite direction (e.g., right side down), an undo or redo operation, respectively, may be performed. (This rotation may be thought of as rotation along a longitudinal axis, such as “roll” in flight dynamics terms.) In some embodiments, different operations may be invoked based upon the level of tilt. For example, a greater level of tilt may result in a repeated undo/redo. Levels of tilt may be expressed, for example, as degrees or percentage tilt (or using any other similar measurement of tilt) from a horizontal orientation to a vertical orientation. Note that in other embodiments, different UI functions may be invoked as a result of these tilting operations.
Although any mechanism for providing near real-time orientation information may be used with the techniques described here to present a GBUIM, example embodiments are described as obtaining orientation data from an accelerometer device, which measures acceleration and gravity induced reaction forces typically in units of gravity (“g”s). Near real-time orientation (e.g., inclination) data can be extracted from the acceleration data. Accelerometers are increasingly available on mobile devices to provide data that can be incorporated into mobile applications. For example, they have been used in devices that implement game controllers or other portable electronic devices. Accelerometers have also been incorporated as part of cellular phones and smart phones, in order to provide enhanced location and/or orientation data to applications developed for such phones. One such accelerometer is present in iPhone™ devices manufactured by Apple Corporation, and its data is accessible using Apple's standard SDK (software development kit) available from Apple Corp. Other known and available accelerometers (such as an LIS302DL from STMicroelectionics) may be used.
Also, although the examples described herein often refer to a mobile device such as a smart phone, the techniques described herein can also be used by other types of mobile devices. Accordingly, for the purposes herein, mobile devices may include devices such as cellular telephones, smart phones, personal digital assistants (“PDAs”), gaming consoles, portable electronic devices, other mobile devices with integrated display screens, standalone display screens controlled by remote mobile devices, etc. Also, the examples described herein describe the presentation of user interfaces having user interface controls (UI controls), although interfaces having different sorts of interaction mechanisms (e.g., such as voice commands) may also be invoked or caused to be presented using the GBUIM techniques described here.
A representation of the abstraction of vertical levels 502 shows how, in illustration 530, as the device tilts up from initial level 503, the device movement crosses close threshold 504 past open threshold 505, to cause a menu to be opened (such as shown in
Other embodiments may define different threshold levels or different behavior between them. A range of movement that does not produce any action, or that produces a different action, may be similarly applied to other behaviors assigned to tilt up/down motions. In addition, such level measurements may be appropriately defined for the type of accelerometer data received. For example, illustration 550 demonstrates a different example of a tilt down operation, where, the close threshold level 526 is lower relative to horizontal level 525. In this case, in order to close an open menu (or invoke the behavior assigned to a tilt down motion), the device “top” is tilted down (backward) below the horizontal position.
Similarly, the user can tilt the right side of the mobile device down (the opposite rotation) to cause a “redo” operation to again yield string “abd” instead of string “abc.”
Although not shown, in some embodiments a secondary undo or redo threshold (e.g., a multi-undo or multi-redo threshold) is defined that allows an application to implement a multiple stroke, character (or other unit) undo/redo operation. The secondary undo/redo may provide a repeated stroke undo/redo, thereby alleviating a user needing to engage in multiple tilt operations to undo/redo several strokes at a time. In the example shown in
A representation of the abstraction of vertical levels 912 next to the tilt left illustration 900 indicates that a tilt left operation may trigger an undo operation when the position of the device falls within the undo area (e.g., within undo range) 910. Further, in some embodiments, when the tilt movement position exceeds the multi-undo threshold 911, the tilt left operation may trigger a repeated stroke/character/unit undo operation as described above with reference to
Example embodiments described herein provide applications, tools, data structures and other support to implement a gravity-based user interface mechanism to be used for enhancing the usability of mobile devices, especially those with limited screen real estate or having small profiles. Other embodiments of the described techniques may be used for other purposes, including for user interfaces for gaming consoles that may or may not be associated with smaller display screens. In the following description, numerous specific details are set forth, such as data formats and code sequences, etc., in order to provide a thorough understanding of the described techniques. The embodiments described also can be practiced without some of the specific details described herein, or with other specific details, such as changes with respect to the ordering of the code flow, different code flows, etc. Thus, the scope of the techniques and/or functions described are not limited by the particular order, selection, or decomposition of steps described with reference to any particular routine.
Also, although certain terms are used primarily herein, other terms could be used interchangeably to yield equivalent embodiments and examples. For example, it is well-known that equivalent terms could be substituted for such terms as “tilt,” “rotation,” “display,” etc. In addition, terms may have alternate spellings which may or may not be explicitly mentioned, and all such variations of terms are intended to be included.
In a typical implementation, mobile device/computing system 1000 is a standalone mobile device, e.g., a client device, that communicates over a network to one or more other devices, carriers, servers, etc. However, in some embodiments computing system 1000 may comprise one or more computing systems and may span distributed locations. In addition, each block shown in mobile device/computing system 1000 may represent one or more such blocks as appropriate to a specific embodiment or may be combined with other blocks. The various blocks may use standard (e.g., TCP/IP) or proprietary interprocess communication mechanisms to communicate with each other.
In the embodiment illustrated and described, mobile device 1000 comprises a computer memory (“memory”) 1001, a display 1002, one or more Central Processing Units (“CPU”) 1003, other Input/Output devices 1004 (e.g., keyboard, mouse, CRT or LCD display, etc.), other computer-readable media 1005, one or more network connections 1006, and one or more orientation sensors 1007. The GBUIM embodied as a gravity user interface (UI) support module 1010 is shown residing in memory 1001. In other embodiments, some portion of the contents or some of or all of the components/capabilities of the gravity UI support module 1010 may be stored on and/or transmitted over the other computer-readable media 1005. In addition, it will be appreciated that memory 1001 is one type of storage media, and may include many different forms of memory. The gravity UI support module 1010 preferably executes on one or more CPUs 1003 and manages the handling of tilt operations, in response to tilt movements detected by orientation sensors 1007, with respect to the UI preferences and application data 1015, as described herein. Other code or programs 1030 and potentially other data repositories, such as data repository 1020, also reside in the memory 1010, and preferably execute on one or more CPUs 1003. Of note, one or more of the components in
In a typical embodiment, the gravity UI support module 1010 interacts with data provided by the data repository 1015, which may include, for example, data representing user preferences, and manages events triggered by the orientation sensors 1007, such as an accelerometer. In at least some embodiments, the user preference data 1015 is provided external to the gravity UI support module 1010 and is available, potentially, over one or more networks 1050 or via other systems communicatively coupled to the mobile device 1000. Other modules also may be present to interact with gravity UI support module 1010. In addition, the gravity UI support module may interact via a network 1050 with other client devices 1055 such as other mobile devices, one or more mobile device application providers 1065, and/or one or more carrier systems 1060. Network 1050 may be wireless network such as a telecommunications network and/or comprise a connection to a local or wide area network such as the Internet. In other embodiments not described here, network 1050 may comprise wired data transmissions.
In an example embodiment, the gravity user interface support module 1010 is implemented using standard programming techniques. However, a range of programming languages known in the art may be employed for implementing such example embodiments, including representative implementations of various programming language paradigms, including but not limited to, object-oriented (e.g., Java, C++, C#, Smalltalk, etc.), functional (e.g., ML, Lisp, Scheme, etc.), procedural (e.g., C, Pascal, Ada, Modula, etc.), scripting (e.g., Perl, Ruby, Python, JavaScript, VBScript, etc.), declarative (e.g., SQL, Prolog, etc.), etc.
The embodiments described above may also use well-known or proprietary synchronous or asynchronous computing techniques, or alternately decomposed using a variety of structuring techniques known in the art, including but not limited to, multiprogramming, multithreading, etc. Some embodiments are illustrated as executing concurrently and asynchronously and communicating using message passing techniques. Equivalent synchronous embodiments are also supported. In addition, programming interfaces to the data stored as part of the gravity UI support module 1010 (e.g., the user preference data in the data repositories 1015) can be made available by standard means such as through C, C++, C#, and Java APIs; libraries for accessing files, databases, or other data repositories; through scripting languages such as XML; or through Web servers, FTP servers, or other types of servers providing access to stored data. The data repository 1015 may be implemented as one or more database systems, file systems, XML, or any other method known in the art for storing such information, or any combination of the above, including implementation using distributed computing techniques.
Furthermore, in some embodiments, some or all of the components/functionality of the gravity UI support module 1010 may be implemented or provided in other manners, such as at least partially in firmware and/or hardware, including, but not limited to one ore more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and/or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, functionality, and/or data structures may also be stored as contents (e.g., as executable or other machine-readable software instructions or structured data) on a computer-readable medium (e.g., as a hard disk; a memory; a computer network or cellular wireless network or other data transmission medium; or a portable media article to be read by an appropriate drive or via an appropriate connection, such as a DVD or flash memory device) so as to enable or configure the computer-readable medium and/or one or more associated mobile devices to execute or otherwise use or provide the contents to perform at least some of the described techniques. Some or all of the components, functionality, and data structures may also be transmitted as contents of generated data signals (e.g., by being encoded as part of a carrier wave or otherwise included as part of an analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, embodiments of this disclosure may be practiced with other device configurations.
As described in
All of the above U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and/or listed in the Application Data Sheet are incorporated herein by reference, in its entirety.
From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the present disclosure. For example, the methods, systems, and techniques for processing tilt operations discussed herein are applicable to other architectures other than an Apple iPhone architecture. Also, the methods, systems, and techniques discussed herein are applicable to differing protocols, communication media (optical, wireless, cable, etc.) and devices (such as wireless handsets, electronic organizers, personal digital assistants, portable email machines, game machines, pagers, navigation devices such as GPS receivers, etc.).
