This application is submitted in the name of the following inventors:
1. Field of the Invention
The invention relates to a touch panel user interface; for example, not intended to be limiting in any way, in one embodiment, the user interface might be used to control presentation of media streams in a home entertainment system.
2. Related Art
In presentation systems for media streams and other information (such as for example, home entertainment systems), it is advantageous to provide users with a user interface that is simultaneously natural, easy to use, and powerful in its ability to command the system. In the case of a home entertainment system, or any other system involving substantial attention to be paid by the user to the system and not to the controls, it would be particularly advantageous if the controls themselves were to be similar to the display provided by the home entertainment system.
Known user interfaces on touch panels are similar to user interfaces on computers: when a set of objects is presented, the user must first press on the object of interest, which has the effect of highlighting the object, then press another object (such as a button) to perform some action on the highlighted object (such as showing more information about that object). There are at least two disadvantages to this paradigm on a touch interface: the interface requires a separate mechanism to navigate through the set of objects; and when the user wants to show interest in an object, he or she must press a different object (the button).
This invention provides a solution to both problems by eliminating both the navigation objects (such as arrow buttons) and the object required to perform an action. In one aspect, the invention provides a system that works as described below. Although a preferred embodiment is described below, alternate embodiments need include only one of these two solutions. For a first example, pressing on a peripheral object centers and highlights it, but the user must press a separate object (such as a button) to perform an action on the highlighted object. For a second example, the user has buttons to highlight various objects and navigate through the set of objects, but pressing on the highlighted object performs the default action.
The direct and natural selection of objects can be extended to manipulation. For example, the columns of a table can be reordered by dragging them. For a different example, a list of favorite movies can be created by dragging covers from a mosaic of covers into a box representing the user's favorite movies. For a different example, a scrollable list of movies can be shifted up or down by dragging up or down across the touch panel, simulating the motion of a real piece of paper. For another example, a picture taken by a digital camera can be rotated go degrees (if it was taken in portrait mode but saved in landscape mode) by putting two index fingers on opposite corners of the picture and rotating it in the proper direction.
Accordingly, it would be advantageous to provide a user interface not subject to drawbacks of the known art.
The invention provides techniques, embodied in methods and systems, regarding user interfaces that are simultaneously natural, easy to use, and powerful in their ability to command the system.
In a first aspect, the invention is embodied in a system that includes a touch panel, capable of both providing relatively high quality graphic output and of receiving input signals from a pointing device with substantially direct physical contact with the touch panel display, such as a finger or stylus, with the effect that the user has the natural feeling of identifying or controlling screen elements that are presented. In the context of the invention, there is no particular requirement of involving actual touching. For an example, not intended to be limiting in any way, the system might be responsive to movement by a user (such as typing in a projected 3D field looking like a keyboard), such as available with some PDA devices.
In one embodiment, the touch panel might present its information in a “frame” mode, in which the main home entertainment screen is duplicated within a frame on the touch panel, with additional information (either for control or for elucidation) being presented at positions other than within the frame.
For an example, not intended to be limiting in any way, the touch panel might show a frame including within it a presentation of a substantial duplicate of the graphic output being made on the home entertainment screen. However, because the presentation is limited to a frame smaller than the full screen size of the touch panel, the touch panel might also present a set of control elements (including channel and volume control), as well as other and further control elements. These other and further control elements might include (a) a list of bookmarks or watchpoints available in the ongoing presentation, (b) a progress bar, enhanced as described below, showing the amount of the media stream that has been presented so far, (c) other and further metadata, such as for example a set of alternative titles or alternative presentations that are concurrently available (for channel surfers), might also be made available outside the presentation frame on the touch panel.
Examples of such other and further metadata might include (a) descriptions or identifications of individual sub-streams within the media stream. For example, not intended to be limiting in any way, sub-streams (and sub-sub-streams, and the like), might be presented in one or more of the following ways.
In the event that the media stream includes a sequence of “scenes” (possibly including a moving picture or a still picture of a 3D collection of objects), the metadata might include various information, such as for example: when that scene begins, when it ends, and what “sub-scenes” (for scenes for which this can be defined) are included within the scene. Such scenes in a moving picture might be detected by similar analysis as described above. Such scenes in a moving picture might be presented in the “progress bar” as separate (or separable) elements, such as for example: showing scenes at right angles to the regular progress bar, showing scenes at another angle to the regular progress bar, or linking scenes together using some form of arrow or indicator of continuation. The user might use that metadata to select a particular scene for presentation, to set (either deliberately or automatically) a bookmark or watchpoint within that particular scene, or to expand upon the scene to show its sub-scenes. In one embodiment, in the event that sub-scenes can be defined at a sufficiently short time resolution, the sub-scene might be presented as a direct sequence of still pictures. In alternative embodiments, thinking the scene itself might be presented at the speed with which the user moves his or her finger along the touch panel. In alternative embodiments, if the user moves his or her finger along a sequence of scenes, the system might present short clips from each of those scenes, thus presenting a “flavor” of the media stream without having to see the whole thing, and the like.
Examples of such other and further metadata might include (a) alternative or enhanced versions of scenes in the media stream, (b) alternative or enhanced versions of characters in the media stream, or (c) other modifications to the media stream capable of being computed in response to the media stream and in response to user input. For example, such other and further metadata might be used to present the media stream in one or more of the following ways.
In the event that the media stream includes alternative versions of a “character” (possibly including an animated character, a wire frame description of a 3D model, or a set of moving pictures or still pictures of a human actor), the metadata might include various information such as what locations that character fits into the media stream, what actions or emotions that character is displaying, and the like. Such scenes in a might be detected in similar manner as metadata for scenes, as described above. Once detectable, the user might use that metadata to select among multiple characters for presentation, such as for example substituting the character of Marilyn Monroe for Nicole Kidman in the movie “Moulin Rouge”, substituting actual newsreel footage of Josef Stalin for a foreign leader in a fictional work, substituting photographs of a selected fashion model for Evita Peron in newsreel footage, and the like.
Particular value for the progress bar is that it can be integrated with a set of bookmarks or watchpoints (already supplied with the media stream, supplied from an alternative source as the media stream, or generated dynamically such as in response to other bookmarks or watchpoints or metadata as described above). For an example, not intended to be limiting in any way, very many (probably millions) of new bookmarks might be generated in response to linear interpolation between those bookmarks already supplied with the media stream. The progress bar can also be integrated with a set of chapter titles, such as for example in a feature movie or a sequence of television episodes in a television show season; such chapter titles might be treated as selectable “scenes” as described above. In the context of the invention, there is no particular requirement that the progress bar is linear: for example, a season of N episodes might be presented as one linear progress bar with N parts, or as N separate progress bars (possibly varying by their presentation length, or by some other parameter, and possibly joined end-to-end, or with explicit spacing in-between), or any other presentation appealing to the user. The controlling user at the touch panel can see at a glance when the progress of the presentation approaches a chapter (or episode or scene) ending, or a scene marked using metadata for the attention of that controlling user, and can also see how far forward or back must be skipped to return to a preferred bookmark.
For an example, not intended to be limiting in any way, the touch panel might show (a) a still picture associated with each bookmark or watchpoint, (b) a still picture or short loop associated with wherever the user selects on the progress bar, (c) a still picture associated with a selected offset, plus or minus, from wherever the user selects on the progress bar, and the like.
For an example, not intended to be limiting in any way, the touch panel might allow the user to (a) focus on a particular segment of an entire media stream, such as for example one defined as a 10-minute segment of a 120-minute movie, (b) focus on a particular set of features of an entire media stream, such as for example those scenes in a 120-minute movie in which Kate Beckinsale appears, either shown as a sequence of still pictures, short loops, or as a form of markup on the progress bar, (c) focus on a subsegment of an already-focused on segment, with the effect of focusing on shorter and shorter segments, or shorter segments with particular actors, or segments with selected multiple actors, and the like, (d) focus on other Boolean combinations thereof, such as for example AND and OR operations applied to the examples above, (e) and other focusing techniques such as hyperlinks, overlays, and the like.
For an example, not intended to be limiting in any way, the touch panel might show a frame including a presentation of a substantially similar graphic output as being made on the home entertainment screen, as described above, with metadata associated with the presentation (more precisely, with the media stream associated with that presentation) being optionally presented in other regions outside the frame, such as for example, to the left, right, bottom, or top of that frame, and the like.
For an example, not intended to be limiting in any way, the touch panel might show a frame including a presentation of a substantially similar graphic output as being made on the home entertainment screen, as described above, with metadata associated with the presentation (more precisely, with the media stream associated with that presentation) being optionally presented in the same region as the frame, such as for example as a subtitle, supertitle, balloon statement by a character, line pointing into the frame with commentary either inside or outside the frame, and the like.
For an example, not intended to be limiting in any way, the touch panel might show a frame including a presentation of a substantially similar graphic output as being made on the home entertainment screen, as described above, with metadata including a progress bar showing the amount of the media stream having so far been presented, yet to be presented, or both, or some other selection of the media stream, such as those focused-upon sections described above.
Although described as a “progress” bar herein, in the context of the invention, there is no particular requirement that the progress bar actually designates or is responsive to actual “progress” on the part of the system in presenting the media stream. For an example, not intended to be limiting in any way, in embodiments of the invention where the media stream instead represents a database or a set of other data, the “progress” bar might represent an index into that database or that set of other data, rather than (or in addition to) a measure of the amount of the data available in association with that title.
The progress bar might be augmented with significant additional metadata, such as one or more of the following.
In a second aspect, the invention is embodied in a system that includes features of the guide and mosaic user interfaces, as described in the incorporated disclosure. As described in the incorporated disclosure, the guide user interface allows the user to review metadata about each title, with the effect that the user might sort a list of thousands of possible titles with the effect of restricting the titles visible in the window to only a group of those that user is interested in, from which selection of the title the user wishes to view is presumably significantly easier. As described in the incorporated disclosure, the mosaic user interface allows the user to visually picture the relative “closeness” (according to some measure) of multiple titles, with the effect that the user might determine which titles are similar to those the user has liked, again from which selection of the title the user wishes to view is presumably significantly easier. In one embodiment the touch panel includes a presentation of individually selected titles by their cover art, with the associated metadata for those titles also presented on the touch panel. This has the effect that the user can view textual metadata for titles concurrently with viewing cover art (even possibly animated cover art) for those titles.
In a second embodiment, the touch panel might present its information in “mosaic” mode, in which the screen space available to the touch panel represents a substantial duplicate of what is presented on the main home entertainment screen, with the effect that the user does not lose focus on any visual action while using the touch panel.
For an example not intended to be limiting in any way, when the user touches the cover art for a particular title, action is taken depending on whether that title is the focused-upon title (which in one embodiment is shown using highlighting or other emphasis). For example, in one embodiment, the following actions might be taken.
The inventors have found that the described behavior is superior to known user interfaces. In some known user interfaces, touching (or “clicking on”) an object just causes that object to become highlighted, whether or not it was highlighted already, and a different action entirely (such as for example double-clicking or pressing a different button) to perform a default action for that object. These known techniques would not be as suitable for a touch panel interface, since at least (a) double-clicking is difficult to perform reliably by a user with a touch panel, (b) a relatively small touch panel display might not have sufficient room for another button to touch.
More generally, the user interface provided by the invention has the property that the user need only “press what's interesting.” If the “interesting” object is something newly-interesting, the system presents it as the focused-upon item. If the “interesting” object was already indicated as interesting, the system performs a default action for that “interesting” object. In alternative embodiments, the user might set parameters to alter the default action, either temporarily or until changed again.
In alternative embodiments, the touch panel might be used to present a relatively large collection of data in a natural way that the user might navigate.
For an example, not intended to be limiting in any way, when the user touches a selected location on the screen, the screen is redrawn in response to the selected location. This might or might not cause the screen to be rearranged for cover art associated with that location, but might (a) move the screen in an X or Y direction, (b) change the scale of the display, with the effect of “zooming in” or “zooming out”, with the effect of moving the screen in a Z direction, (c) perform a focusing operation as described above, thus effecting a selection much like querying a database.
More generally, if titles are related to each other by some measure of “closeness”, the system might respond to distinct actions by the user by redrawing the screen to reflect responses to distinct commands by that user. For example, a right sweep might indicate to start panning the screen dynamically to the right, and other gestures (left sweep, circle, squiggle, “graffiti”-like symbols, and the like) might indicate other commands by that user.
More generally, in the context of the invention, there is no particular requirement that the titles chosen for display on the screen must be static. For example, the system might periodically redraw at least some of those titles, in response to time. For an example, not intended to be limiting in any way, the system might redraw each title on the screen with a new one periodically (or with a probabilistic parameter such as a Poisson distribution), with the effect that the screen is redrawn nearly entirely after a period of about 5-10 minutes. For an example, the probabilistic parameter might be responsive to active or passive metadata about users in the region with the presentation element, with the effect that the screen is redrawn more slowly when there are few (or no) users present, and more quickly when there are more (or at least some) users present.
More generally, the invention might be used to navigate a 2D representation of any kind. In one embodiment, the 2D representation of titles gives the impression of navigating an infinite plane (R2), where duplicates of titles are presented to prevent the user from reaching an edge of that plane. In alternative embodiments, there need not be duplicates of any titles, and the user might reach an edge where the titles to be found are farthest away, by some measure of closeness, from the focused-upon title. In one embodiment, the user might use the touch panel to “walk” substantially randomly (either intentionally or by caprice) along the infinite plane. In the context of the invention, there is no particular requirement that the 2D surface to be presented is like a plane. For an example, not intended to be limiting in any way, in one embodiment, the 2D representation of titles gives the impression of navigating a finite torus with a finite number of holes, where duplicates of titles are presented as a natural result of circling the torus along one or more of its axes.
For an example, not intended to be limiting in any way, the cover art is typically arranged in a rectilinear pattern, with offsets for each row so that scrolling either horizontally or vertically will eventually show all available titles. In one embodiment, the cover art might be arranged in another pattern, such as a hexagonal pattern, or a rectilinear pattern with some overlap of cover art, so as to either space the cover art out more and allow easier selection of moving cover art, or to cluster the cover art together and allow the viewer to see more selections.
For an example, not intended to be limiting in any way, the system might be responsive to the user touching the touch panel in more than one location, either substantially simultaneously, with very brief delay, or within a delay associated with a time parameter. The user might touch (1) two or more separate titles, such as with two or more fingers, (2) a curved or straight line on which lie multiple titles, (3) “doubleclick” on a title by touching it twice, (4) “clicking and dragging” a title by touching it and drawing a line to another region of the touch panel, and the like. In one embodiment, the system will attempt to determine a preference the user is expressing, and to act upon it.
After reading this application, those skilled in the art would recognize that the invention provides an enabling technology by which substantial advance is made in the art of user interfaces and human-machine control systems.
For example, the invention might be used to provide one or more of, or some combination or extension or mixture of, any of the following.
After reading this application, these and other and further uses of the invention would be clear to those skilled in the art.
This application incorporates by reference and claims priority of at least the following documents.
These documents are hereby incorporated by reference as if fully set forth herein, and are sometimes referred to herein as the “incorporated disclosure”. Inventions described herein can be used in combination or conjunction with technology described in the incorporated disclosure.
In the description herein, a preferred embodiment of the invention is described, including preferred process steps and data structures. Those skilled in the art would realize, after perusal of this application, that embodiments of the invention might be implemented using a variety of other techniques not specifically described, without undue experimentation or further invention, and that such other techniques would be within the scope and spirit of the invention.
Definitions
The general meaning of each of these following terms is intended to be illustrative and in no way limiting.
The scope and spirit of the invention is not limited to any of these definitions, or to specific examples mentioned therein, but is intended to include the most general concepts embodied by these and other terms.
System Elements
A system 100 includes elements as shown in
The local server 110 includes elements as shown in
In one embodiment, the content database 111 includes at least the following information. The content database ill includes elements as shown in
The touch panel 130 includes elements as shown in
Using this format, the user 140 might browse the selection of available titles, view cover art and (other) metadata for those titles, and select one or more of those titles for presentation. In one embodiment, presentation occurs on both the presentation element 120 and on the touch panel 130.
In one format, these elements include at least the following information.
Using this format, the user 140 might view a presentation of one particular available title, while concurrently maintaining control over features of that presentation which the user 140 might desire to skip or repeat. For an example, not intended to be limiting in any way, the touchscreen 130 might provide control elements 131g with which the user might indicate the desire to skip or repeat the next one of, or a class of, scenes in the media stream. For an example, not intended to be limiting in any way, these control elements 131g might be dynamically generated, with the effect that (say) as the media stream nears content marked for parental control for violence, a control element 131g would appear on the screen asking if the user desires to skip that violent scene.
Method of Operation
Although described serially, the flow points and method steps of the method 200 can be performed by separate elements in conjunction or in parallel, whether asynchronously or synchronously, in a pipelined manner, or otherwise. In the context of the invention, there is no particular requirement that the method must be performed in the same order in which this description lists flow points or method steps, except where explicitly so stated.
The method 200 includes steps as shown in
At a step 210, the mosaic-like user interface is generated, and sent by the local server 110 to the presentation element 120 and on the touch panel 130. In one embodiment, the local server 110 might generate the mosaic-like user interface, while in alternative embodiments, the mosaic-like user interface may be generated relatively remotely and sent to the local server 110. The mosaic-like user interface includes at least one focused-upon title 211.
At a step 220, the metadata associated with the focused-upon title 211 is retrieved from storage (such as for example, a database), and presented on the touch panel 130. In one embodiment, the local server 110 might generate the mosaic-like user interface, while in alternative embodiments, the mosaic-like user interface may be generated relatively remotely and sent to the local server 110.
At a step 230, the command or request is received from the user 140 by sensing a touch by the user 140 on the touch panel 130. However, after reading this application, those skilled in the art would recognize that actual touching is not required. For an example, not intended to be limiting in any way, the touch panel 130 might include a proximity sensor for a stylus or other object, the touch panel 130 or other sensor might detect nearness of an element broadcasting in RF or other frequencies, the touch panel 130 or other sensor might include an IR or visible light sensor combined with a processing element for detecting when a “touch” should be noted, the touch panel 130 might be sensitive to noise (including possibly spoken commands from the user 140) or to chemical compounds on a stylus, and the like.
At a step 240, the selected media stream 241 (as selected by the user 140 at the step 230 with a “present media stream” or “present title” command) is presented on the presentation element 120 and on the touch panel 130. The local server 110 sends the same selected media stream 241 to both the presentation element 120 and the touch panel 130.
At a step 250, metadata regarding the selected media stream 241 is retrieved from storage (such as for example, a database), and presented on the touch panel 130. In one embodiment, the metadata might be similar to the metadata presented in the step 220.
At a step 260, the touch panel 130 maintains the control button region 131b, including a set of control elements 131g which the user 140 might use with presentation of the selected media stream 241. In one embodiment, the control elements 131g are maintained by the touch panel 130 using a processor and memory, while in alternative embodiments, the control elements 131g might be maintained relatively remotely and sent to the touch panel 130 for presentation to the user 140.
At a step 270, the touch panel 130 maintains the progress-bar region 131c, including an enhanced progress bar regarding presentation of the selected media stream 241. In one embodiment, the enhanced progress bar includes at least some of the following.
Although preferred embodiments are disclosed herein, many variations are possible which remain within the concept, scope, and spirit of the invention. These variations would become clear to those skilled in the art after perusal of this application.
After reading this application, those skilled in the art will recognize that these alternative embodiments and variations are illustrative and are intended to be in no way limiting. After reading this application, those skilled in the art would recognize that the techniques described herein provide an enabling technology, with the effect that heretofore advantageous features can be provided that heretofore were substantially infeasible.
The set of inventive techniques are further described in the Technical Appendix. After reading this application and its Technical Appendix, those skilled in the art would recognize how to make and use the invention. All reasonable generalizations of techniques shown in this application and its Technical Appendix are within the scope and spirit of the invention, and would be workable, without further invention or undue experimentation.
At least the following documents are part of the technical appendix.
The Technical Appendix is submitted with this application and hereby made a part of this application. The Technical Appendix, and all references cited therein, are hereby incorporated by reference as if fully set forth herein.
This Technical Appendix is intended to be explanatory and illustrative only, and not to limit the invention in any way, even if few (or only one) embodiment is shown.
Number | Date | Country | |
---|---|---|---|
Parent | 10418949 | Apr 2003 | US |
Child | 10936789 | Sep 2004 | US |
Parent | 10418739 | Apr 2003 | US |
Child | 10936789 | Sep 2004 | US |
Parent | 10655496 | Sep 2003 | US |
Child | 10936789 | Sep 2004 | US |