This background description is provided for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, material described in this section is neither expressly nor impliedly admitted to be prior art to the present disclosure or the appended claims.
Current storytelling comes in many forms, including novels, short stories, audio books, movies, television programs, and live theater. Some less-visual forms of storytelling tell stories well using mobile devices, such as through text for novels and audio for audio books. Visual storytelling, however, suffers from the size of a mobile device's display. These smaller displays often make the presentation of sufficient detail or sufficient context mutually exclusive. While sufficient detail can be provided by zooming in to see a character's expression on a small screen, the surrounding context of what is around or near that character is lost. Alternately, while sufficient context is provided by showing most of a scene, much of the detail needed to effectively tell the story is lost when showing a large scene on a small screen.
Techniques and apparatuses for visual storytelling on a mobile device are described with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:
Overview
This document describes techniques and apparatuses of visual storytelling on a mobile media-consumption device. These techniques and apparatuses enable a user to view events central to the story while also viewing context for the story. By so doing, a user may enjoy the story as the story's author intended without sacrificing a user's ability to engage with the story's context.
Assume, for example, that a user wishes to watch the movie Ratatouille about a rat that becomes a chef in Paris, France. Conventional techniques shrink down the movie, which was intended for a large movie screen, to fit the movie into the user's mobile-device display. Movie screens in theaters vary in size, many averaging about 30 feet by 70 feet, or about 2,100 square feet in viewing area. Larger smartphone displays measure four to five inches diagonally, or about 0.07 square feet. Even taking into account that users are much closer to a mobile-device display that a movie screen, it is readily apparent that substantial detail is lost when using these conventional techniques.
The techniques described in this document, however, enable a user to enjoy the story in sufficient detail as well as the story's context. For this same movie, the techniques present the movie as a visual experience through which a user's mobile-device display is a view into that visual experience. The user may select to watch, in sufficient visual detail, the events central to the story, and deviate from those events to view the surrounding context.
Assume that the rat in the movie, named Remy, is floating rapidly down a sewer pipe. The user can view Remy by orienting his or her display to see Remy in detail. The user can also, however, tilt the display up to see the ceiling of the sewer pipe, left to view the pipe that Remy has already floated down, right to view the pipe Remy is heading toward, and so forth. Further, when the user deviates from the events central to the story—here Remy traveling down the pipe—to look at trash in the sewer water or grime on the pipe's ceiling, the techniques pause or delay the events central to the story so that the story still can be fully told to the user. The user may look away, focus on the trash and so forth, and when the user moves the display to see Remy once again, the story continues. By so doing, a user can enjoy the story as the author intended and with context, all through a mobile-device screen.
This is but one example of the many ways in which the techniques enable visual storytelling on a mobile device. The following discussion turns to an operating environment, techniques that may be employed in the operating environment, and a device in which component(s) of the operating environment can be embodied. In the discussion below, reference is made to the operating environment by way of example only.
Example Environment
Visual experience 106 is capable of telling a story and providing context for that story, though some context within visual experience 106 can be unrelated to the story. Each story is an authored series of events previously written by a storyteller (e.g., a human writer) and presentable in a moving visual form, such as a theatrical production of Leo Tolstoy's novel War and Peace, Orson Welles's film Citizen Kane, and the computer-animated movie Toy Story by John Lasseter, Pete Docter, Andrew Stanton, and Joe Ranft.
Which portions of visual experience 106 are presented on display 104 depend on view selections made by a user of mobile device 102. Thus, mobile device 102 is capable of receiving view selections and, in response to these view selections, presenting views of visual experience 106 on display 104. View 108 is one example view showing a portion of visual experience 106, here a dog staring at a cat, which is central to the story being told.
Because view 108 shows characters central to the story, the techniques progress through the authored series of events making up the story. When the view selected is contextual, mobile device 102 ceases to tell the story. Thus, when a user selects to view various contextual elements, such as a bird, a flower, or a mouse, the story ceases to progress. When the user again selects to view the dog and cat, the techniques continue the story. By so doing, a story can be told in sufficient detail while still permitting a user to view context for that story.
More specifically, consider
Mobile device 102 includes or has access to computer processor(s) 202, computer-readable storage media 204 (media 204), one or more displays 104, and view-selection mechanism 206. Media 204 includes an operating system 208 and storytelling module 210. Although not shown, media 204 may also include applications and user data, such as digital images, audio files, or documents.
View-selection mechanism 206 is capable of sensing a user's view selections through various input manners and devices, which range from buttons, capacitive sensors, and touch screens to orientation sensors and cameras capable of determining an orientation or orientation change of mobile device 102. Buttons and capacitive sensors enable a user to select views through, for example, four buttons or sensors, one each to select to move the view right, left, up, or down. Touch screens or pads enable a user to select views with visual controls similar to the buttons but also through zooming gestures, such as a pinch gesture to zoom out or a spread gesture to zoom in. Cameras and orientation sensors can determine selections that tilt, turn, move in, move out, move up, move left, move right, and move down display 104, to name just a few. Orientations sensors, in some cases, include micromachined accelerometers. These micromachined accelerometers, depending on the types, are configured to measure in multiple axes magnitude and direction of proper acceleration (e.g., G-force) as a vector quantity. By so doing, the micromachined accelerometers can sense orientation, coordinate acceleration, vibration, shock, and falling. For use as orientation sensors, these micromachined accelerometers can sense six degrees of freedom of mobile device 102, including three degrees of freedom in translation (X, Y, and Z) and three in rotation (pitch, yaw, and roll). Cameras can be used to track a device's location, such as relative to a user viewing the display, by tracking imaged objects (e.g., a book pictured by the camera can be used to determine, based on the book changing size or location in an image captured by the camera, an orientation or location in three dimensions of the display) or objects that relate to the viewer, such as by tracking a user's facial features (e.g., eyes, cornea, irises).
Furthermore, view selections received through these various input manners can be mapped to a particular desired view. For example, if a user moves a display away from herself, this movement can be sensed by an accelerometer or camera of view-selection mechanism 206, and sensor data from the accelerometer or camera passed directly to storytelling module 210 or processed by view-selection mechanism 206 to analyze the sensor data to determine that the display is being moved away from the user. This determination can map to desired views, such as a zooming in or out of the currently viewed content. Other examples include selecting a right button or touch control, which is then sensed by (or passed as selection data to) view-selection mechanism 206, which in turn determines that display 104 has moved right relative to content being displayed. This determination can be passed to storytelling module 210, which then presents a different view (to the right of the prior view) of visual experience 106.
Note that these various input manners can be received and responded to in real time—a user's selection to zoom in by pulling a display in, look up by moving a display up or tilting up, or look down by tilting down, can result in the view changing as fast or nearly as fast as the user makes the selection. The techniques enable a user to view a story, movie, or other content as fast or almost as fast as the user would view his or her own world. Furthermore, these various inputs can be combined. Assume for example that a user is viewing both a man and woman sitting at an outdoor café. Assume that the user wants to see the brand of soft drink being consumed by the woman. The user may simply move the display from being centered between the couple to at the soft drink, and then at the same time or after, move the display toward the user. By so doing, view-selection mechanism 206 senses the orientation change, passes the determined orientation to storytelling module 210, which then zooms in and centers on the soft drink, thereby showing it in detail.
This mapping of view selections to portions of visual experience 106 can be conceptualized as a direct mapping from a view selection to a two or three dimensional space representing visual experience 106. Thus, for a two-dimensional movie, view selections can be mapped to point, square, or rectangle on a planar surface as a representation of the two-dimensional movie. For a three-dimensional space, such as some computer-generated shorts, television shows, or movies, or some live-action movies that are filmed with additional cameras, a view selection can be mapped to a non-planar surface representing possible views of visual experience 106. This can be conceptualized as a sphere, for example, with each new frame of the experience mapped to an interior of the sphere with a user viewing from within the interior. Thus, a view selection tilting a display down can view grass on the ground, a view selection tilting a display up can view clouds in the sky, and a view selection moving left or right can view content to the left or right.
Returning to
Storyline 212 includes an authored series of events 218 (events 218) that, when presented in the authored series, tell a story. Events 218 can be authored by one or many human authors, alone or with computer assistance. Further, events 218 can be of a fixed order and/or number, though each may vary in an amount of time to present each event. A fast-moving music video, for example, may present new scenes and actions in less than one second. A three-hour drama, however, may have few scene changes and long stretches of dialog, which can therefore include events that may be many minutes long. Events can be part of respective sets of events making up scenes or acts, which an author may arrange such that the techniques, when returning to an incomplete event, repeat sets of events, though this is not required.
By way of example, consider
Story views 214 present at least a portion of one of events 218, which can be previously determined to enable a key element of the event to be seen or a substantial portion of the event to be seen. This previous determination is not required, as storytelling module 210, in some cases, can determine which views present events 218 of storyline 212. Whether predetermined or not, various manners can be used to determine whether a view is a story view or a context view.
By way of example, consider
Here assume that the author assigned conditions for determining whether a view is a story view or context view such that story views are those that include at least the dog's eyes and the cat's eyes but do not require both of their bodies to be within the view.
Various other manners can be used to assign or determined which views of visual experience 106 are story views 214 and which are context views 216, however. Examples include a threshold portion (e.g., 80%) of an ideal story view, a point on an image of the visual experience being within the view (e.g., all of the cat or dog but not both), a set of points or subset of the set (e.g., 4 of 5 points assigned to image 402) within the view, and so forth.
Methods
The following discussion describes techniques of visual storytelling on a mobile media-consumption device. These techniques can be implemented utilizing the previously described environment, such as display 104, view-selection mechanism 206, and/or story telling module 210 of
At 702, a visual experience capable of telling a story is presented. As noted in part above, the story is an authored series of events and the visual experience includes story views and context views. The story views present at least a portion of an event of the authored series of events. The context views present visual context of the visual experience.
At 704, view selections from a mobile media-consumption device on which the visual experience is presented are received. These view selections can alter or maintain a view into the visual experience from a prior view to a current view. Altering or maintaining the view can be effective to switch, in some cases, from a story view to a context view and vice-versa. Note that view selections, even if maintaining an orientation for the device, may still result in a change between a story view and a context view, as events of the story may move within the visual experience.
When the view selections result in story views, the techniques proceed along a Story View path to operation 706, where the story is told by progressing through the authored series of events. When the view selections result in context views, the techniques proceed along the Context View path to operation 708, where the story ceases to be told by ceasing to progress through the authored series of events. Methods 700 may repeat this operation, as shown at dashed lines from operation 706 to 704 and operation 708 to 704, as view selections may alternate from story view to context view and vice-versa until the story is fully told.
At 708, storytelling module 210 may cease to progress through the authored series of events by pausing an event of a last-presented story view. Alternately or additionally, storytelling module 210 may cease progress through the authored series of events by repeating the event, a set of events of which the event is a part, or a portion of the event. Examples of this repetition are illustrated in
For example, storytelling module 210 may repeat the last-presented event, such as through repeating a seamless portion of an event, like repeating a loop of a half-second clip of the dog staring at the cat. This is illustrated in
Consider the following example of methods 700, illustrated in
As shown by table 802, the user spent 8:42 viewing context and 20:11 viewing the story (which can take as little as 20 minutes), for a total time spent 806 of 28:53. The additional 11 seconds to complete the story is for those events or event portions looped or otherwise repeated, with four seconds repeated from 11:03 to 11:07 and seven seconds looped with event 12 (a portion of event 12 from 17:33 to 17:40 to complete event 12 at 18:14). Not only may each user's time spent to view a story be different, but each user's view of story may also vary, based on each person likely selecting different views, even of the story itself.
In this example, the views experienced by the user include images presenting the story and images presenting the context, with the actual images seen through the user's display depending solely on the user's view selections. This can be represented mathematically as:
E(t)=F(Si(Ti),v(t))
E(t) is the experienced views as a function (F) of time (t), where Si is the multiple authored events, Ti is the view changes over time, and v(t) contextual views over time.
While methods 700, and table 802 illustrated in
At 902, a story is told through a visual experience presented on a mobile media-consumption device. As noted above, the context can be for the story or be part of visual experience 106 with little or no relation to the story.
At 904, a selection to view context is received. Storytelling module 210 may determine that this selection is to view context, such as by determining that the view selected no longer presents the story. In some cases, context includes a subordinate story, which is an authored series of subordinate events similar to the story for the visual experience, but not required for the story to progress.
Consider again the dog-chases-cat story provided above. In that example, assume that a user selects to move from viewing the dog and cat to view a bird flying in the sky, which is neither related to nor provides context about the dog and the cat. This is illustrated in
Context deviation 1012 includes subordinate story 1020 without time spent to move to and from the subordinate story and includes two events (not shown). Subordinate stories can include unrelated or related stories, such as showing mouse 410 jump onto flower 409 (both of
For example, a user may select to view only the cat and, in so doing, storytelling module 210 provides a subordinate story showing the cat sit on the branch, scratch her ears, lick her paws, and tauntingly ignore the dog. While subordinate story 1016 provides character development and perhaps some understanding of why the dog is chasing the cat, the author can establish that this character development is not an event of the story, and thus is not needed for the story to progress.
At 906, the context is presented and the story is paused, both responsive to the selection to view context. In some cases presenting the context for the story is performed during presentation of the event, such with a looping of the event. Thus, if a portion of the event is shown, but is insufficient to cause the story to progress, the portion of the event may repeat or loop, but on returning the event is fully told. If a context view includes the left half of the dog, for example, a user may see the dog's body shake from barking and hear the bark. Further, when presenting context, audio for the story can continue to be heard, such as in a loop of a current event. In such cases, volume for the event's sounds may be reduced, such as based on a visual distance from the context view to the event.
Returning to the flying-bird example above, assume that storytelling module 210 receives a selection to view the flying bird from orientation sensors of tablet computing device 102-2 of
At 908, a selection to view the story is received. Storytelling module 210 may determine that this selection is to view the story, such as by determining that the view selected presents minimum elements of a story view. For example, the selection may be of a view that includes a minimum event range in which elements of characters of the story are located. In the context of the present example, assume that storytelling module 210 receives selection to view one of story views 502, all of which include minimum event range 604, which indicates selection of at least one valid story view.
At 910, an event at which the story was paused responsive to the selection to view context is presented. As described above, the event, or a portion thereof, may begin looping at or near a time of being paused. In such cases, the event of the story is presented by re-entering a current loop of the event and then progressing to a subsequent event. By so doing, the story progresses, though one or more events or portions of events may be repeated, as noted above.
In some cases, storytelling module 210 may indicate to the user whether or not a current view is or is not progressing the story (or subordinate story) by presenting a visual indicator in the view. Examples include items that are noticeable but unobtrusive, such as a moth fly near the dog and cat, items of clothing of a same or similar color, even if on different characters, and so forth. More-obvious indicator can also be used, such as reduced resolution when in contextual views, zooming out or zooming in response to contextual views, a flashing light, and so forth. When the last event of the story is presented, storytelling module 210 may end the visual experience.
Example Device
Example device 1100 can be implemented in a fixed or mobile device being one or a combination of a media device, computing device (e.g., mobile device 102 of
Example device 1100 can be integrated with electronic circuitry, a microprocessor, memory, input-output (I/O) logic control, communication interfaces and components, other hardware, firmware, and/or software needed to run an entire device. Example device 1100 can also include an integrated data bus (not shown) that couples the various components of the computing device for data communication between the components.
Example device 1100 includes various components such as an input-output (I/O) logic control 1102 (e.g., to include electronic circuitry) and microprocessor(s) 1104 (e.g., microcontroller or digital signal processor). Example device 1100 also includes a memory 1106, which can be any type of random access memory (RAM), a low-latency nonvolatile memory (e.g., flash memory), read only memory (ROM), and/or other suitable electronic data storage.
Example device 1100 can also include various firmware and/or software, such as an operating system 1108, which can be computer-executable instructions maintained by memory 1106 and executed by microprocessor 1104. Example device 1100 can also include other various communication interfaces and components, wireless LAN (WLAN) or wireless PAN (WPAN) components, other hardware, firmware, and/or software.
Other examples capabilities and functions of these components are described with reference to elements shown in
Although the invention has been described in language specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed invention.
This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 61/856,445, entitled “Visual Storytelling on a Mobile Media-Consumption Device” and filed on Jul. 19, 2013, the disclosure of which is incorporated in its entirety by reference herein.
Number | Name | Date | Kind |
---|---|---|---|
5607356 | Schwartz | Mar 1997 | A |
5892507 | Moorby | Apr 1999 | A |
6714909 | Gibbon | Mar 2004 | B1 |
7184574 | Zahavi | Feb 2007 | B1 |
7467218 | Gould | Dec 2008 | B2 |
7860309 | Bodnar et al. | Dec 2010 | B1 |
7904812 | Atlas | Mar 2011 | B2 |
9086724 | Sakaguchi et al. | Jul 2015 | B2 |
9087403 | Keating et al. | Jul 2015 | B2 |
9183560 | Abelow | Nov 2015 | B2 |
9589597 | Austin et al. | Mar 2017 | B2 |
9607424 | Ng | Mar 2017 | B2 |
20030145331 | Escobar | Jul 2003 | A1 |
20030232318 | Altenhofen et al. | Dec 2003 | A1 |
20040021668 | Chevallier et al. | Feb 2004 | A1 |
20050025465 | Danieli | Feb 2005 | A1 |
20060256133 | Rosenberg | Nov 2006 | A1 |
20070028270 | Ostojic | Feb 2007 | A1 |
20070061862 | Berger et al. | Mar 2007 | A1 |
20080005669 | Eilertsen et al. | Jan 2008 | A1 |
20080215172 | Digon | Sep 2008 | A1 |
20080267451 | Karazi | Oct 2008 | A1 |
20090113278 | Denoue et al. | Apr 2009 | A1 |
20090181736 | Haigh-Hutchinson et al. | Jul 2009 | A1 |
20090198719 | DeWitt | Aug 2009 | A1 |
20090201313 | Thorn | Aug 2009 | A1 |
20090249404 | Chen | Oct 2009 | A1 |
20090325660 | Langridge | Dec 2009 | A1 |
20100153520 | Daun et al. | Jun 2010 | A1 |
20100157064 | Cheng et al. | Jun 2010 | A1 |
20100174421 | Tsai et al. | Jul 2010 | A1 |
20100208138 | Mohri | Aug 2010 | A1 |
20100251101 | Haussecker et al. | Sep 2010 | A1 |
20110126106 | Ben Shaul et al. | May 2011 | A1 |
20110299832 | Butcher | Dec 2011 | A1 |
20110304627 | Kegel | Dec 2011 | A1 |
20120092348 | McCutchen | Apr 2012 | A1 |
20120105473 | Bar-Zeev et al. | May 2012 | A1 |
20120169882 | Millar et al. | Jul 2012 | A1 |
20120249424 | Bove et al. | Oct 2012 | A1 |
20140002580 | Bear et al. | Jan 2014 | A1 |
20140133834 | Shannon | May 2014 | A1 |
20140282013 | Amijee | Sep 2014 | A1 |
20150012827 | Elmeih | Jan 2015 | A1 |
20150022557 | Austin et al. | Jan 2015 | A1 |
20150023650 | Austin et al. | Jan 2015 | A1 |
20150042795 | Tsuria et al. | Feb 2015 | A1 |
20160054863 | Elmieh et al. | Feb 2016 | A1 |
Number | Date | Country |
---|---|---|
1087618 | Mar 2001 | EP |
2209306 | Jul 2010 | EP |
WO-2006056311 | Jun 2006 | WO |
Entry |
---|
“International Search Report and Written Opinion”, Application No. PCT/US2015/041339, Oct. 30, 2015, 13 pages. |
“SightLine: The Chair—Virtual Surreality”, Retrieved from: http://sightlinevr.com/index.html on Nov. 30, 2015, 2 pages. |
“Restriction Requirement”, U.S. Appl. No. 14/335,821, Nov. 4, 2015, 5 pages. |
“International Preliminary Report on Patentability”, Application No. PCT/US2014/046907, Jan. 19, 2016, 8 pages. |
“International Preliminary Report on Patentability”, Application No. PCT/US2014/047302, Jan. 19, 2016, 8 pages. |
“Non-Final Office Action”, U.S. Appl. No. 14/209,432, Mar. 10, 2016, 21 pages. |
“Non-Final Office Action”, U.S. Appl. No. 14/335,821, Feb. 25, 2016, 9 pages. |
“International Search Report and Written Opinion”, Application No. PCT/US2014/046907, Nov. 6, 2014, 11 Pages. |
“International Search Report and Written Opinion”, Application No. PCT/US2014/047302, Jan. 22, 2015, 13 Pages. |
“International Search Report and Written Opinion”, Application No. PCT/US2014/047269, Nov. 10, 2014, 13 pages. |
Heymann, et al., “Representation, Coding and Interactive Rendering of High-Resolution Panoramic Images and Video Using MPEG-4”, Panoramic Photogrammetry Workshop, Berlin, Germany, Feb. 28, 2005, 5 Pages. |
“360 degree Interactive Storytelling”, University of Glasgow Retrieved from the Internet: URL:http://www.kwiatek.krakow.pl/publications/28128927—360storytelling—University—of—Glasgow—Karol—Kwiatek—PDF.pdf, Sep. 27, 2012, 80 Pages. |
“Final Office Action”, U.S. Appl. No. 14/209,432, Jul. 7, 2016, 18 pages. |
“Non-Final Office Action”, U.S. Appl. No. 14/513,761, Oct. 4, 2016, 12 pages. |
“Notice of Allowance”, U.S. Appl. No. 14/335,821, Oct. 25, 2016, 5 pages. |
“Corrected Notice of Allowance”, U.S. Appl. No. 14/335,821, Jan. 12, 2017, 2 pages. |
“First Action Interview Office Action”, U.S. Appl. No. 14/335,833, Mar. 6, 2017, 3 pages. |
“First Action Interview Office Action”, U.S. Appl. No. 14/526,311, Mar. 9, 2017, 3 pages. |
“International Preliminary Report on Patentability”, Application No. PCT/US2015/041339, Feb. 2, 2017, 10 pages. |
“Pre-Interview Communication”, U.S. Appl. No. 14/335,833, Dec. 29, 2016, 3 pages. |
“Pre-Interview Communication”, U.S. Appl. No. 14/526,311, Feb. 10, 2017, 7 pages. |
“Final Office Action”, U.S. Appl. No. 14/513,761, dated May 3, 2017, 13 pages. |
“Foreign Office Action”, EP Application No. 14752459.9, dated Mar. 6, 2017, 8 pages. |
“Notice of Allowance”, U.S. Appl. No. 14/335,833, dated May 24, 2017, 8 pages. |
Number | Date | Country | |
---|---|---|---|
20150026576 A1 | Jan 2015 | US |
Number | Date | Country | |
---|---|---|---|
61856445 | Jul 2013 | US |