Patent Application
20150261747
Generally described, user computing devices may facilitate the playback or display of items of content, such as audiobooks, electronic books, songs, videos, television programs, computer and video games, multi-media content, and the like. For example, an e-reader user computing device may display an electronic book on a screen or play an audiobook through speakers or headphones. In some instances, the publisher or author of an item of content may provide supplemental content to accompany the item of content, such as a soundtrack, video clip, multi-media content, etc. This supplemental content may be played by the user computing device as the user consumes the item of content.
In some current approaches, this supplemental content is created specifically for an item of content. For example, a soundtrack for a particular electronic book or audiobook may be recorded and synchronized to that electronic book or audiobook. However, this approach can be time-consuming and expensive, possibly requiring the generation of new supplemental content for each item of content to be enhanced. This purpose-generated supplemental content may become “stale” for a user of the content, especially in cases where only a small amount of supplemental content is generated for the item of content. Moreover, if the user dislikes the supplemental content provided with an item of content, he or she may have no recourse other than to disable the supplemental content. Likewise, users that wish to customize the supplemental content may not have the opportunity to do so under these approaches.
The foregoing aspects and many of the attendant advantages will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:
Generally described, aspects of the present disclosure relate to enhancing digital content. In this regard, a computer-implemented content enhancement service is disclosed. A base item of content, such as an audiobook, electronic book, song, video or computer game, television program, movie, multi-media clip, etc. may be enhanced with one or more items of audio, visual, or audiovisual enhancement content, such as sounds, songs, video clips, images, text, animations, computer or video mini-games, etc. selected by the content enhancement service and/or by a user. Accordingly, in some embodiments, a user computing device conveys enhancement content (e.g., by displaying visual enhancement on a screen of the user computing device and/or playing audio enhancement content over speakers or headphones connected to the user computing device) while conveying the base item of content for which the enhancement content was selected. The enhancement content may be created specifically to accompany a base item of content, or it may be pre-existing content that is unrelated to the base item of content.
The content enhancement service may analyze a portion of the base item of content to identify one or more characteristics of the portion of the base item of content. The content enhancement service may determine the characteristics of a portion of the base item of content by machine content analysis techniques, by human input, or by obtaining information about the portion from an external resource that maintains information about the base item of content. These characteristics may be used to guide the selection of enhancement content for the portion of the base item of content. Characteristics of a portion of a base item of content may include, for example, a genre of the base item of content; a mood of the portion of the base item of content; a geographical setting of the portion of the base item of content; a historical setting of the portion of the base item of content; and a time of day of the portion of the base item of content. For instance, for a portion of a base item of content set on a beach, the content enhancement service may select enhancement content such as sounds for waves crashing or seagulls squawking, or an image of a beach. In another example, for a portion of a base item of content set in a kitchen during the morning, the content enhancement service may select enhancement content such as the sound of a toaster popping or bacon sizzling. Characteristics of a base item of content may also be referred to herein as “labels.”
The content enhancement service may also analyze the portion of the base item of content for enhancement objects (sometimes referred to herein simply as “objects”) present in the portion of the base item of content. The presence of an enhancement object may suggest or signify a specific item of enhancement content, such as a specific sound, image, video clip, etc. to be conveyed. In some embodiments, an enhancement object includes at least one word, phrase, or sentence present in the portion of the base item of content. An enhancement object may also appear as an image on a page of an electronic book, or may be recited (e.g., spoken, sung, shouted, etc.) during an audiobook. In some embodiments, a specific type of enhancement content is identified or suggested by the enhancement object. An enhancement object may be a sound object, an image object, or another type of enhancement object, depending on which specific type or types of enhancement content may be called for by the enhancement object. For example, the word “cow” may be a sound object associated with a mooing sound, an image object associated with a picture of a cow, or both. Onomatopoeia may also be sound objects (e.g., “the wind howled”).
In some embodiments, the characteristics of a portion of the base item may influence the selection of an item of enhancement content corresponding to an enhancement object present in the portion of the base item of content. For example, the sound object “birds” may be recited in a horror-themed audiobook. Accordingly, the sound of a bird that fits the horror theme, such as a raven's caw, may be selected as an item of enhancement content. In another example, the sound object “boombox” may appear in an electronic book set in the 1980s. Accordingly, a song popular on the radio in the 1980s may be selected as an item of enhancement content.
In some base items of content, no enhancement object is present. Rather, an item of enhancement content may be selected if its characteristics or labels are similar to a portion of the base item of content with which the item of enhancement content is to be conveyed. Characteristics of an item of enhancement content may also be referred to herein as “labels.” In one example, a portion of the base item of content may be analyzed using natural language techniques or other content analysis techniques to determine an event or mood of the portion. For example, a portion of a base item of content may include the words “party” and “celebrate,” indicating a cheerful mood. Enhancement content corresponding to the cheerful mood, such as a fast song in a major key, may be selected to accompany the cheerful portion. The characteristics of an item of enhancement content may be either pre-assigned by the content enhancement service, or characteristics may be determined by analyzing the item of enhancement content. For example, the content enhancement service may maintain one or more rules or graphs to determine the characteristics of an item of enhancement content. An example rule might be, “Fast songs in major keys have a cheerful characteristic.”
Those skilled in the art will recognize that items of enhancement content need not be stored on the computing device that conveys the base item of content. For example, items of enhancement content may be streamed over a network to the conveying computing device (e.g., a user computing device) from external resources, such as a content enhancement server or a network content library remote from the conveying computing device. Items of enhancement content may also be relayed among multiple external resources before being transmitted to the conveying computing device. For example, a content enhancement server may request enhancement content from a network content library, obtain that enhancement content, and then transmit the obtained enhancement content to the conveying computing device.
The selections of which items of enhancement content accompany which portions of the base item of content may be made and/or changed dynamically by users or by the content enhancement service. These selections may be collected and stored as an enhancement content settings file or content pack for use with the base item of content. Enhancement content settings files or content packs may be transmitted over a network from a content enhancement server to a user computing device, or vice versa. A user computing device may also transmit an enhancement content settings file or content pack to another user computing device.
Turning to
The user computing device 102 may be any computing device capable of communicating over the network 106, such as a laptop or tablet computer, personal computer, personal digital assistant (PDA), hybrid PDA/mobile phone, mobile phone, electronic book reader, set-top box, camera, audiobook player, digital media player, video game console, in-store kiosk, television, one or more processors, integrated components for inclusion in computing devices, appliances, electronic devices for inclusion in vehicles or machinery, gaming devices, or the like. The user computing device 102 may generally be capable of conveying content to a user of the user computing device 102. For example, the user computing device 102 may be capable of playing audio content by directing audible output through speakers, headphones, or the like. The user computing device 102 may also be capable of displaying images or video content on a display screen. The user computing device 102 may also be capable of communicating over the network 106, for example, to request items of enhancement content from the network content library 104 or the content enhancement server 110.
The user computing device 102 may be in communication with a user content library 112. The user content library 112 may electronically store items of enhancement content, such as sounds, songs, video clips, images, animations, computer or video mini-games, etc. These items of enhancement content may have characteristics or labels associated with them. In some embodiments, items of enhancement content may be organized into groups (e.g., playlists or queues) that are designated by the user as having certain characteristics. For example, the user may have a “party” playlist. The content enhancement service may treat the items of enhancement content in the “party” playlist as having a “party” characteristic. The items of enhancement content in the playlist may be selected to be conveyed with a portion of a base item of content describing a party. The user content library 112 may additionally store one or more graphs or rules used to select items of enhancement content, as discussed below with respect to
The network content library 104 may host or maintain items of enhancement content that may be accessed over the network 106 by a user computing device 102 or by the content enhancement server 110. The network content library may include one or more electronic data stores to host or maintain items of enhancement content, and may further store one or more graphs or rules used to select items of enhancement content, as discussed below with respect to
The network 106 may be any wired network, wireless network, or combination thereof. In addition, the network 106 may be a personal area network, local area network, wide area network, cable network, satellite network, cellular telephone network, or combination thereof. Protocols and components for communicating via the Internet or any of the other aforementioned types of communication networks are well known to those skilled in the art of computer communications and thus, need not be described in more detail herein.
A human interaction task system 108 may be included in the network environment to assist the content enhancement server 110. Generally described, the human interaction task system 108 is a computerized system that electronically processes human interaction tasks (HITs). A HIT may be a difficult, time-consuming, or expensive task for a computing device to perform. However, it might be relatively easy and quick for a human to perform a HIT. Accordingly, the human interaction task system 108 might request a human worker to perform a HIT, e.g., for gathering information or answering a query, and to return the results or answers to the human interaction task system 108 for further processing and/or presentation to the requestor. Thus, in some embodiments, the content enhancement service directs the human interaction task system 108 to pose one or more queries about an item of content to a human worker of the human interaction task system 108. For example, a human worker may be asked to identify characteristics of a base item of content or of an enhancement item of content, to identify enhancement objects in the base item of content; or to suggest enhancement content for use with a base item of content. The human interaction task system 108 may receive answers to these queries and transmit them to the user computing device 102 or the content enhancement server 110 to guide the operation of the content enhancement service. The human worker of the human interaction task system 108 may volunteer to respond to these and other tasks and to communicate other information about the item of content to the content enhancement server 110.
The content enhancement server 110 is a computing device that may perform a variety of tasks to enhance base items of content. For example, the content enhancement server 110 may receive a request to enhance content from the user computing device 102 over the network 106, and, in response to the request, transmit items of enhancement content to the user computing device 102. Additional operations of the content enhancement server 110, such as aligning enhancement content to base items of content and analyzing characteristics of content, are described with respect to
The content enhancement server 110 may be in communication with a content data store 114. Like the user content library 112, the content data store 114 may electronically store items of enhancement content, such as sounds, songs, video clips, images, animations, computer or video mini-games, etc. In some embodiments, the content data store 114 also stores one or more base items of content. The content data store 114 may additionally store one or more graphs or rules used to select items of enhancement content, as discussed below with respect to
The user computing device 102 and content enhancement server 110 may each be embodied in a plurality of components, each executing an instance of the respective content user computing device 102 and content enhancement server 110. A server or other computing system implementing the user computing device 102 and content enhancement server 110 may include a network interface, memory, processing unit, and computer readable medium drive, all of which may communicate with each other by way of a communication bus. Moreover, a processing unit may itself be referred to as a computing device. The network interface may provide connectivity over the network 106 and/or other networks or computer systems. The processing unit may communicate to and from memory containing program instructions that the processing unit executes in order to operate the user computing device 102 and content enhancement server 110. The memory generally includes RAM, ROM, and/or other persistent and/or auxiliary non-transitory computer-readable storage media.
It will be recognized that many of the devices described above are optional and that embodiments of the environment 100 may or may not combine devices. Furthermore, devices need not be distinct or discrete. Devices may also be reorganized in the environment 100. For example, the content enhancement server 110 may be represented in a single physical server or, alternatively, may be split into multiple physical servers. The entire content enhancement service may be represented in a single user computing device 102 as well. Additionally, it should be noted that in some embodiments, the content enhancement service may be executed by one more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly provisioned and released computing resources, which computing resources may include computing, networking and/or storage devices. A hosted computing environment may also be referred to as a cloud computing environment.
The content enhancement server 110 includes a processing unit 202, a network interface 204, a non-transitory computer-readable medium drive 206, and an input/output device interface 208, all of which may communicate with one another by way of a communication bus. As illustrated, the content enhancement server 110 is optionally associated with, or in communication with, an optional display 218 and an optional input device 220. The display 218 and input device 220 may be used in embodiments in which users interact directly with the content enhancement server 110, such as an integrated in-store kiosk, for example. In other embodiments, the display 218 and input device 220 may be included in a user computing device 102 shown in
The memory 210 contains computer program instructions that the processing unit 202 executes in order to implement one or more embodiments of the content enhancement service. The memory 210 generally includes RAM, ROM, and/or other persistent or non-transitory computer-readable storage media. The memory 210 may store an operating system 214 that provides computer program instructions for use by the processing unit 202 in the general administration and operation of the content enhancement server 110. The memory 210 may further include other information for implementing aspects of the content enhancement service. For example, in one embodiment, the memory 210 includes a user interface module 212 that facilitates generation of user interfaces (such as by providing instructions therefor) for display upon a computing device such as user computing device 102. For example, a user interface may be displayed via a navigation interface such as a web browser installed on the user computing device 102. In addition, memory 210 may include or communicate with the content data store 114. Content stored in the content data store 114 may include base items of content and items of enhancement content in audio, visual, or multi-media formats, as described in
In addition to the user interface module 212, the memory 210 may include a content enhancement module 216 that may be executed by the processing unit 202. In one embodiment, the content enhancement module 216 may be used to implement the content enhancement service. For example, the content enhancement module 216 may be used to select items of enhancement content and assign them to portions of a base item of content; to direct the content enhancement server 110 to obtain items of enhancement content from the network content library 104 and to store them in content data store 114; or to direct the content enhancement server 110 to transmit items of enhancement content to a user computing device 102 over a network 106, among other tasks.
Those skilled in the art will recognize that in some embodiments, the content enhancement service is implemented partially or entirely by the user computing device 102. Accordingly, the user computing device 102 may include a content enhancement module 216 and other components that operate similarly to the components illustrated as part of the content enhancement server 110, including a processing unit 202, network interface 204, non-transitory computer readable medium drive 206, input/output interface 208, memory 210, user interface module 212, and so forth.
In block 302, the content enhancement service may locate enhancement objects present in a portion of the base item of content. An enhancement object may appear in textual or graphical form, or may be recited in an item of audio content. For example, Enhancement objects may also be identified in textual representations of items of content, such as transcripts of audiobooks, videos, television episodes, etc. Electronic books may also be a textual representation of an item of content. As discussed above, enhancement objects may appear in a portion of a base item of content and may suggest, specify, or signify an item of enhancement content to be conveyed while the portion of the base item of content is conveyed. Examples of enhancement objects may include words that correspond to sounds (such as onomatopoeia, or such as objects that generate sounds, like animals, cars, radios, etc.); images that correspond to sounds; words that correspond to images, video clips, or animations; and the like.
Enhancement objects may be located in the base item of content using a variety of techniques. In one embodiment, machine techniques known in the art may be used to identify enhancement objects. For example, the content enhancement service may employ natural language techniques, semantic analysis, named entity extraction, or other content analysis techniques. Further information on the use of machine techniques to locate objects in an item of content may be found in U.S. patent application Ser. No. 12/972,058, entitled “GRAPHICALLY REPRESENTING ASSOCIATIONS BETWEEN REFERENTS AND STORIES,” filed on Dec. 17, 2010; and in U.S. patent application Ser. No. 12/881,021, entitled “SYSTEMS AND METHODS FOR ASSOCIATING STORIES WITH RELATED REFERENTS,” filed on Sep. 13, 2010. The disclosures of both of these applications are hereby incorporated by reference in their entireties. In another embodiment, the content enhancement service may incorporate input from a human interaction task system 108 to locate enhancement objects in a base item of content. For example, the human interaction task system 108 may convey a portion of a base item of content (such as a passage of text in an electronic book or an excerpt of an audiobook) to a human worker of the human interaction task system 108, and ask the human worker to identify one or more enhancement objects in the portion of the base item of content. The human worker may also be asked to identify or provide enhancement content that may associated with the enhancement object. In still another embodiment, the content enhancement service may rely on a list of enhancement objects known to be present in the portion of the base item of content. This list may be generated by, for example, one or more experts associated with the content enhancement service or by user input. A list may be obtained from a network content library, network-based knowledge base, or network-based encyclopedia, among other examples.
In block 304, the content enhancement service may identify the characteristics of the enhancement objects identified in block 302. Characteristics of an enhancement object may include, for example, what type of enhancement content would be appropriate for the enhancement object (sound, song, video, image, etc.); how long the enhancement content should be conveyed; or other attributes of the enhancement content specified by the enhancement object (e.g., the volume of audio enhancement content; the genre of audio or video enhancement content; etc.). For example, the sentence, “The car honked for a full ten seconds,” contains the onomatopoeia “honked,” indicating that honked is a sound object, as well as information that suggests how long the car honked: “a full ten seconds.” Having identified “honked” as an enhancement object, the content enhancement service may determine that the characteristics of the enhancement object “honked” include the fact that the word “honked” indicates a sound and that the duration of the sound should be ten seconds. In another example, the base item of content may include the sentence “Cameron gazed at the Pointillist painting.” Having identified “painting” as an enhancement object, the content enhancement service may determine that the characteristics of the enhancement object “painting” include the fact the word “painting” indicates an image and that the image should be of a painting of the Pointillist style.
Characteristics of an enhancement object may be identified in a variety of ways. In one embodiment, machine techniques known in the art may be used to identify the characteristics of enhancement objects. For example, the content enhancement service may employ natural language techniques, semantic analysis, named entity extraction, or other content analysis techniques known in the art to determine the characteristics of an enhancement object, e.g., by analyzing any words, phrases, or sentences in the vicinity of the enhancement object. In another embodiment, the content enhancement service may incorporate input from a human interaction task system to identify the characteristics of an enhancement object. For example, the human interaction task system may convey a portion of a base item of content (such as a passage of text in an electronic book or an excerpt of an audiobook) to a human worker of the human interaction task system and indicate which enhancement objects are present. The human interaction task system may then ask the human worker to identify any characteristics of the enhancement object, such as what type of enhancement content is specified by the enhancement object, how long an item of video or audio enhancement content should last, etc. In still another embodiment, the content enhancement service may rely on a list of characteristics of the enhancement objects identified in the base item of content. This list may be generated by, for example, one or more experts associated with the content enhancement service or by user input; or a list may be obtained from, for example, a network content library, network-based knowledge base, or network-based encyclopedia.
Referring again to
In some embodiments, the content enhancement service identifies characteristics of a portion of the base item of content by analyzing words present in the content. For example, words, phrases, sentences, images, videos or the like conveyed in the base item of content may indicate characteristics of the portion. For example, words in a portion such as “smile,” “laugh,” or “celebrate” might prompt the content enhancement service to assign a “cheerful” mood characteristic to that portion, while words such as “sand” and “ocean” might prompt the content enhancement service to assign a “beach” location characteristic to the portion. Words, phrases, sentences, or the like may be associated with images or videos conveyed by the base item of content so that the content enhancement service can identify characteristics of the portion in which those images or videos are conveyed. Further information on the use of machine techniques to identify characteristics of an item of content may be found in U.S. patent application Ser. No. 12/972,058, entitled “GRAPHICALLY REPRESENTING ASSOCIATIONS BETWEEN REFERENTS AND STORIES,” filed on Dec. 17, 2010; and in U.S. patent application Ser. No. 12/881,021, entitled “SYSTEMS AND METHODS FOR ASSOCIATING STORIES WITH RELATED REFERENTS,” filed on Sep. 13, 2010. The disclosures of both of these applications have been incorporated by reference above.
In other embodiments, the content enhancement service may rely on input from a human interaction task system 108 to identify one or more characteristics of a portion of a base item of content. For example, the human interaction task system 108 may convey a portion of the base item of content to a human worker along with a query such as “In what location or time period is this portion set?” or “What is the mood of this portion?” In still other embodiments, the content enhancement service may obtain information about the base item of content from a network resource such as a network-based encyclopedia or knowledge base. For example, a network-based encyclopedia may maintain summaries of one or more portions of a base item of content that may be accessed by the content enhancement service.
In block 308, the content enhancement service may access enhancement content fitting the characteristics of the portion of the base item of content and/or matching an enhancement object present in the portion of the base item of content. The enhancement content may be accessed remotely over the network 106 (e.g., the user computing device 102 may access remote content from the network content library 104 or the content data store 114, or the content enhancement server 110 may access an item of enhancement content from the network content library 104 and store it in the content data store 114). In embodiments where items of enhancement content are accessed remotely by a computing device, the items of enhancement content may be transmitted by streaming, by torrent, or by any other network data transfer technique or protocol known in the art. The enhancement content may also be accessed locally from an electronic data store associated with a computing device (e.g., by the user computing device 102 locally accessing the user content library 112 or by the content enhancement server 110 locally accessing the content data store 114). In some embodiments, all of the items of enhancement content for use with a base item of content are organized into a content pack and transmitted to or accessed by a user computing device 102.
In some embodiments, items of enhancement content may have characteristics, which may include, for example, the type of enhancement content; the mood of the enhancement content; the duration of the enhancement content; an enhancement object to which the item of enhancement content corresponds, and so forth. Generally, an item of enhancement content may have characteristics of the sort that may also be associated with portions of a base item of content. For example, an item of enhancement content may be a sound file corresponding to a raven's cry. The sound file might be marked to indicate that it corresponds to a sound object such as a “raven” or “bird,” and that it might have characteristics such as “horror” or “fear,” as ravens often appear in items of content of the horror genre. If an item of enhancement content matches one, some, or all of the characteristics of the portion of the base item of content, and/or includes a sound or image associated with the object identified in the portion of the base item of content, it may be selected as an item of enhancement content for use with the portion of the base item of content.
These characteristic or object matches need not be exact or verbatim matches. As discussed above, the content enhancement service may have access to one or more rules or graphs stored in an electronic data store to guide the selection of enhancement content. Generally described, rules or graphs may be used to translate characteristics or enhancement objects of a base item of content into characteristics likely to be present in or enhancement objects likely to correspond to items of enhancement content. For example, a portion of the base item of content may be set in a kitchen, which can be a location characteristic. A rule or graph stored in an electronic data store associated with the content enhancement service (e.g., stored on user content library 112, network content library 104, or content data store 114) may indicate the characteristic “kitchen” is related in a graph to a number of items of enhancement content, such as the sound of bacon sizzling or a toaster popping. In another example, a portion of a base item of content may have a sad mood. A rule that might be associated with sad moods might be, “Pick only songs with slow tempos and in minor keys to accompany this portion,” as slow songs with minor keys are typically associated with sadness.
In block 310, items of enhancement content may optionally be prioritized for a particular portion of the base item of content. In some embodiments, the content enhancement service sets a cap or quota on the number of items of enhancement content to be conveyed with the portion of the base item of content, for example, so as not to overwhelm the user or distract the user from the base item of content. Because a limited number of items of enhancement content may be conveyed with a portion of a base item of content, it may desirable to prioritize relatively more important characteristics of the portion of the base item of content or enhancement objects in the portion, and to select items of enhancement content accordingly. Thus, in one embodiment, only a subset of the characteristics of a portion of base item of content (or a subset of the characteristics of an item of enhancement content) are used to select an item of enhancement content. Ways to prioritize characteristics of a base item of content are discussed below with respect to
The cap or quota may vary based on the type of the item of base content and the type(s) of any items of enhancement content that may be selected. For example, a cap or quota for a page of an electronic book might be one or two sounds and a song, and no other items of enhancement content. Accordingly, if items of enhancement content including an image, a video clip, five sounds, and two songs fit the characteristics or enhancement objects present in the page of the electronic book, the content enhancement service may select only two of the five sounds and one of the two songs to be conveyed, and ignore the remaining items of enhancement content. Additionally, in some embodiments, the cap or quota for a type of enhancement content is zero. For example, the cap or quota for items of video enhancement content may be zero for a television program base item of content, because it may not be desirable or practical to convey a video clip while a television program is conveyed. In other embodiments, the cap or quota is based not on a number of items of enhancement content, but rather the size of an item of enhancement content (e.g., duration or bytes).
In block 312, the content enhancement service may align the selected items of enhancement content with the portions of the base item of content to be enhanced. For example, the content enhancement service may map each item of enhancement content accessed (or designated to be accessed by streaming) by the user computing device 102 or the content enhancement server 110 to one or more portions of a base item of content. These mappings may be collected into an enhancement settings content file, which may be shared over the network 106 with other user computing devices 102 or with the content enhancement server 110. The content enhancement service may also convey the mapped items of enhancement content in block 314, e.g., by causing the user computing device 102 to convey the mapped item of enhancement content.
For base items of content that are conveyed continuously (e.g., audiobooks, videos, television programs, etc.), items of enhancement content may be mapped to timestamps of the base item of content and conveyed when the conveying computing device (such as the user computing device 102) reaches that timestamp in the base item of content.
In other embodiments, however, base items of content are conveyed in discrete portions (e.g., electronic books). Accordingly, the content enhancement service may not be able to rely on timestamps as may be present in continuously conveyed items of content. The content enhancement service may estimate a speed of the user's consumption of the base item of content, for example, by determining how many words the user can read in an electronic book per unit time. The user's reading speed may be determined by, for example, how quickly a user turns the pages of an electronic book. In embodiments in which the user computing device 102 includes a touchscreen, the user's reading speed may be determined by, for example, how quickly a user's finger or stylus passes over the words of the electronic book. The content enhancement service may be able to estimate the time at which a user may reach an enhancement object by determining the distance, in words or pages, a user is from reaching an enhancement object, and dividing that distance by the calculated reader speed. For example, the content enhancement service may determine that a reader reads at approximately 200 words per minute (wpm). A user computing device 102 may convey a page of an electronic book that contains 100 words, with an enhancement object 50 words down the page. Accordingly, when the user turns to the page containing the enhancement object, the content enhancement service may direct the user computing device 102 to convey an item of enhancement content associated with the enhancement object 15 seconds after the user turns to that page (200 words/50 wpm=¼minute, or 15 seconds). The content enhancement service may also include a delay (e.g., a delay of a fixed time, such as five seconds, or a delay factor; such as increasing the estimated time by 10%-100% or by some other factor) so that enhancement content is not conveyed before a user of a user computing device 102 reaches it in the portion of the base item of content. This delay may be advantageously incorporated so as not to startle or confuse a user by conveying an item of enhancement content early.
The content enhancement service may also (or instead) align items of enhancement content with the discrete portions themselves. For example, pages 5-10 of an electronic book might be set during a sunny day, while pages 11-20 might be set during a thunderstorm, with page 16 containing an enhancement object such as a lightning bolt. Accordingly, enhancement content corresponding to a sunny day (e.g., the sound of birds singing or an image of a sunny day) might be conveyed while any one of pages 5-10 is conveyed by the user computing device 102. While the user computing device 102 conveys any one of pages 11-20, however, enhancement content pertaining to a thunderstorm, such as the sound of high winds or heavy rain, may be conveyed instead. Additionally, when the user computing device conveys page 16, enhancement content corresponding to the lightning bolt enhancement object, such as an image of a lightning bolt or the sound of thunder, may also be conveyed.
In some instances, an item of enhancement content may be conveyed in its entirety before the user finishes the portion of the base item of enhancement content to which the item of enhancement content is assigned. For example, the user may be reading a sad portion of an electronic book to which a sad four-minute song is mapped. It may take the user ten minutes to read the sad portion of the electronic book. In one embodiment, the item of enhancement content is only conveyed one time while the portion of the base item of content to which the enhancement content is mapped is conveyed. In another embodiment, the item of enhancement content may be repeated while the portion of the base item of content to which the enhancement content is mapped is conveyed. In still a further embodiment, the content enhancement service may select, access, and convey a second item of enhancement content after the first item of enhancement content has been conveyed. This second item of enhancement content may be automatically selected as described above with respect to block 308.
As discussed above, in some embodiments, one or more of the items of enhancement content may be organized (by a user or by the content enhancement service) into an enhancement content pack that provides enhancement content for one or more portions of a base item of content. The enhancement content pack may be a single file or package of files that includes one or more items of enhancement content that are conveyed when the portion of the base item of content to which they are mapped is conveyed. An enhancement content pack may include enhancement content for every portion of the base item of content, or for fewer than all of the base items of content. An enhancement content pack may be a companion to the base item of content and synchronized to the base item of content, as discussed in U.S. patent application Ser. No. 13/070,313, filed on Mar. 23, 2011 and titled “SYNCHRONIZING DIGITAL CONTENT,” the disclosure of which is hereby incorporated by reference in its entirety.
Those skilled in the art will recognize that in some embodiments, enhancement content is only conveyed by a user computing device 102 when the user interacts with a user interface on the user computing device. For example, a user interface may display a portion of a base item of content, such as text, images, or the like. When the user interacts with the user interface (e.g., by tapping a word or image that is an enhancement object in a portion of an electronic book displayed by the user interface), the enhancement content may be conveyed.
Those skilled in the art will also appreciate that the above-described illustrative routine 300 may be carried out substantially while the base item of content is conveyed, e.g., in real time. For example, while a first portion of the base item of content is conveyed, a second portion immediately after the first portion may be analyzed by the content enhancement service for characteristics and enhancement objects. In another example, a portion of the base item may be analyzed for characteristics and enhancement objects while it is being conveyed by the user computing device 102. Enhancement content may be selected substantially while the portion of the base item of content is being conveyed as well. Accordingly, enhancement content may be selected dynamically without the user or the content enhancement service needing to select and map enhancement content in advance.
Upon receiving a request for enhancement content, the content enhancement service may determine whether appropriate local enhancement content (e.g., enhancement content corresponding to characteristics or an enhancement object present in a portion of a base item of content) is available locally in block 404. For example, local enhancement content for a user computing device 102 may be stored in user content library 112.
If the content enhancement service determines that appropriate enhancement content is locally available, in block 406, an item of enhancement content may be selected from the user content library 112 local to the user computing device 102. However, if no appropriate enhancement content is locally available or if insufficient appropriate enhancement content is locally available, the user's request for enhancement content may be transmitted over a network to the content enhancement server 110 in block 408.
In block 410, the content enhancement service may determine whether appropriate enhancement content is available through the content enhancement server 110, for example, whether appropriate enhancement content is stored in content data store 114. If there is no appropriate enhancement content available in the content data store 114, the content enhancement service may proceed to block 412 and attempt to obtain enhancement content from a network content library 104.
As discussed above with respect to
In some instances, however, the content data store 114 associated with the content enhancement server 110 may store an appropriate item of enhancement content. If that is the case, the content custom enhancement service may proceed from block 410 to block 414 to select the appropriate item of enhancement content. The selected appropriate item of enhancement content may then be provided to the user computing device 102 from the content data store 114, as shown in block 416.
Those skilled in the art will appreciate that items of enhancement content from more than one source may be supplied for or assigned to the same portion of a base item of content. In one example, a song to be played while the base item of content is conveyed on the user computing device 102 is streamed from the network content library 104, while one or more sounds corresponding to sound objects present in the portion of the base item of content are played from the user content library 112. In another example, a user computing device 102 may play one or more songs stored in the user content library 112 while conveying the base item of content, and obtain and play sounds corresponding to the sound objects in the base item of content from a network content library 104 or from a content data store 114 associated with the content enhancement server 110.
Those skilled in the art will also appreciate that in embodiments where enhancement content is obtained from a network content library 104, from a content data store 114, or from any other source remote from the user computing device 102, the enhancement content may not be transmitted to the user computing device 102 until it is to be conveyed by the user computing device 102. This approach may be particularly advantageous for user computing devices 102 with limited data storage capacities. For example, an electronic book may include a reference to an opera sound object. When the user computing device 102 conveys the portion of the electronic book in which the opera sound object appears (e.g., the page, chapter, or other portion in which the word “opera” appears or is implied), the content enhancement service may cause the user computing device 102 to download or stream an opera sound item of enhancement content from, for example, network content library 104 or content enhancement server 110. In other embodiments, enhancement content from remote sources may be obtained by the user computing device 102 before the user computing device conveys a base item of content, e.g., by pre-downloading items of enhancement content from remote electronic data stores and storing them to user content library 112.
Those skilled in the art will recognize that a routine similar to routine 400 may be followed by the content enhancement server 110 to obtain items of enhancement content. For example, the content enhancement server 110 may have a base item of content stored in a content data library 114 that is locally accessible to the content enhancement server. The content enhancement server 110 may determine whether appropriate enhancement content for the base item of content is present in the content data store 114. If no appropriate enhancement content is present in the content data store 114, the content enhancement server may query a remote network content library 104 or even a remote user computing device 102 to determine whether either has an appropriate item of enhancement content stored in an electronic data store. If so, the content enhancement server 110 may request the item of enhancement content, and, if granted permission to access the item of enhancement content, may obtain the item (e.g., by downloading it over network 106) and store it to content data store 114.
The title layer 502 may include information about characteristics that are global to the entire base item of content, e.g., to all portions of the base item of content. Characteristics that may be imparted by the title layer 502 to portions of the base item of content may include, for example, characteristics associated with the genre of the base item of content (e.g., horror, comedy, tragedy, etc.) or a mood global to the entire base item of content. For example, the Shakespeare play Hamlet is a play of the tragedy genre, and has a gloomy mood reflected even in the scenes of comic relief. All portions of Hamlet may accordingly be assigned characteristics such as “tragedy” or “gloomy.”
The unity layer 504 may also impart one or more characteristics to a portion of an item of content. Generally described, a unity (also known as a dramatic unity or Aristotelian unity) may be a location, historical era, event, or character common to multiple portions of a base item of content. The unity layer 504 may thus impart characteristics to multiple portions of a base item of content that share a unity. For example, the unity layer 504 in
The portion layer 506 may indicate the characteristics specific to one or more portions of the base item of content. While the portions shown in
The characteristics imparted to a portion of a base item of content by each layer may be combined to determine a full set of characteristics for the portion of the base item of content, as shown in portions 508A-C. For example, portion 508A has a horror (genre) characteristic from the title layer 502, a city (location) characteristic from unity layer 504, and a morning (time of day) characteristic from portion layer 506. Portion 508B has a horror (genre) characteristic from the title layer 502, a beach (location) characteristic from unity layer 504, and night (time of day) and melancholy (mood) characteristics from portion layer 506. Portion 508C has a horror (genre) characteristic from the title layer 502, a beach (location) characteristic from unity layer 504, and has suspense (mood) and fear (mood) characteristics from portion layer 506.
A portion may include enhancement objects, shown in
If no enhancement objects are present, the content enhancement service may, using one, some, or all of the known characteristics of the portion of the base item of content, select enhancement content based on the known characteristics. For example, as shown in the object block 510A, no enhancement object is present in the portion 508A. However, the portion 508A has a number of characteristics affiliated with it: horror, city, and morning. Based on these characteristics, the content enhancement service may, using one or more content rules or graphs stored in a data store associated with the content enhancement service, select a crow sound in the selection block 512A to reflect these characteristics, as crows are common in horror themes and the sound of birds chirping is common in the morning.
If an enhancement object is present in the portion of the base item of content, the enhancement object may be used to guide the selection of an item of enhancement content. In some embodiments, only the enhancement object is used to guide the selection of an item of enhancement content, and characteristics are not considered. For example, the portion 508B includes in the object block 510B a sound object corresponding to waves. The portion 508B also includes affiliated characteristics: horror, beach, night, and melancholy. The sound of waves may not need to be varied based on the genre, mood, location, or time of day when the waves occur. Accordingly, an item of enhancement content corresponding to the sound of waves may be selected in selection block 512B without necessarily considering the characteristics of the portion 508B.
In some instances, both the characteristics of the portion of the base item of content and the enhancement object are used to guide the selection of an item of enhancement content. For example, the portion 508C has characteristics corresponding to horror, beach, suspense, and fear. The portion 508C also contains an image object corresponding to a ghost in block 510C. An electronic data store containing multiple items of enhancement content may include, for example, multiple images of ghosts. Some images of the ghosts may be better suited to fearful portions of horror-themed content than other images (e.g., an image of a scary ghost as opposed to an image of a friendly ghost or a child dressed as a ghost for Halloween). Accordingly, both the characteristics of the portion 508C (horror, fear) and the image object 510C (a ghost) may be used to select, in selection block 512C, an image of a scary ghost.
Any number of layers containing any number of characteristics may be used, and layers need not specifically align with whole portions of the base item of content. For example, a unity layer may cover part of Chapter 2 and part of Chapter 3, but may not cover either portion entirely. Other layers than those depicted may be used as well. For example, layers higher than the title layer 502 might include a series-level layer (e.g., for a base item of content that is part of a series of base items of content) or an author-level layer (e.g., covering all base items of content written by a particular author). Additionally, the types of characteristics in one layer need not be mutually exclusive with the types of characteristics present another layer. For example, as discussed above, the unity layer 504 may include information about the location of one or more portions of the base item of content. However, information about the location of one or more portions of the base item of content may also be present in the portion layer 506 or the title layer 502.
In some embodiments, layers can take priority over one another, such that the characteristics of one or more layers are used to select an item of enhancement content while the characteristics of one or more other layers are not used. In one embodiment, a higher layer (e.g., a title layer) takes priority over all lower layers (e.g., a unity layer or portion layer). For example, returning to the above example of Hamlet, the title layer of Hamlet may impart characteristics such as “tragedy” or “gloomy” to all portions of the Hamlet base item of content. These prioritized characteristics may be used to select items of enhancement content to the exclusion of characteristics present in lower layers. For example, the “tragedy” and “gloomy” characteristics may displace a “humorous” characteristic present at the “Alas, poor Yorick” comic relief portion layer in Hamlet, such that only tragic, gloomy items of enhancement content are selected to be conveyed with the portion of the base item of content containing the comic relief scene. In another embodiment, lower layers (e.g., a portion layer) may take priority over higher layers (e.g., a title layer). Returning to the example of Hamlet, the prioritized “humorous” characteristic of the comic relief portion layer may displace the “tragedy” and “gloomy” characteristics present in the title layer. Accordingly, only humorous items of enhancement content may be selected to be conveyed with the portion of the base item of content containing the comic relief scene, with the “tragedy” and “gloomy” characteristics not taken into consideration. Still other ways of prioritizing characteristics from layers are possible.
Each portion slot 602 may be associated with an enhancement content slot 604. The enhancement content in each enhancement content slot 604 may be conveyed to a user while the user computing device 102 conveys a portion of the base item of content with which the enhancement content slot 604 is affiliated, as discussed above. For example, the content enhancement service may cause the user computing device 102 to convey “Morning” from the Peer Gynt Suite while the user computing device 102 also conveys Chapter 1 of the base item of content. Likewise, while the user computing device 102 conveys Chapter 2, the content enhancement service may cause “Moonlight Sonata” to be conveyed to the user. If a content enhancement slot 604 is empty, it may be indicated as such in the user interface 600. Content enhancement slot 604 may also indicate the origin of each item of enhancement content assigned to each portion of the base item of content. For example, “Morning” from the Peer Gynt Suite may be indicated as being a local item of enhancement content—that is, an item of enhancement content that may be stored on a data store locally accessible to the user computing device 102, such as user content library 112. Alternately, if the enhancement content is to be transmitted to the user computing device 102 from a remote source, such as from content enhancement server 110 or network content library 104, the enhancement content slot 604 may indicate that the item of enhancement content originates from a remote source. In this way, users of the content enhancement service may be notified that if no connection over network 106 is available to user computing device 102, they may not have access to an item of remote enhancement content (e.g., a user may not be able to stream an item of enhancement content to his or her user computing device 102).
The user interface 600 may also include an enhancement content menu 606. The enhancement content menu 606 may list one or more items of enhancement content as well as characteristics about each item of enhancement content or any enhancement objects to which an item of enhancement content may correspond. For example, for an item of enhancement content that is a song, the name of the song, the composer or artist of the song, the duration of the song, the genre of the song, any characteristics of the song may be displayed in the enhancement content menu. These characteristics may have been previously assigned to the item of enhancement content, or may be identified by the content enhancement service as discussed above with respect to block 306 in
In some embodiments, the enhancement content menu 606 includes all items of enhancement content that could be made available to the user computing device by the content enhancement service. For example, the content enhancement menu 606 may include a list of all items of enhancement content stored in the user content library 112, in the content data store 114 associated with the content enhancement server 110, and any available items of enhancement content that may be stored on the network content library 104, as shown in
In addition to manually assigning items of enhancement content to each portion of the base item of content, the user may interact with recommended button 608 to assign automatically one or more items of enhancement content to each portion of the base item of content. For example, a content enhancement service may compare the characteristics of a portion of the base item of content to the characteristics of one or more items of enhancement content. The item of enhancement content may be assigned to the portion of the base item of content if the characteristics of the portion of the base item of content and the item of enhancement content are an exact or close match. Items of enhancement content matching enhancement objects in the portion of the base item of content may also be assigned automatically to the portion.
In some embodiments, a user's interaction with the recommended button 608 prompts the content enhancement service to assign enhancement content to one or more portions of the base item of content based on recommendations stored by the content enhancement service, for example, on content enhancement server 110. The user may optionally modify these assignments of enhancement content. For example, the author or publisher of the base item of content may have a particular set of items of enhancement content in mind that should be conveyed along with the base item of content. Alternately, the content enhancement service may determine recommendations based on how frequently an item of enhancement content is assigned to a particular portion of an item of content by users of the content enhancement service. If an item of enhancement content is assigned to a portion of base content particularly frequently, the content enhancement service may recommend using that item of enhancement content with the portion of the base item of content. The content enhancement server 110 may also maintain or access a social graph affiliated with a user of the user computing device 102 to determine and recommend which items of enhancement content are being selected by persons in the user's social graph.
In other embodiments, interacting with the recommended button 608 causes the user computing device 102 to transmit a request to the content enhancement service for an enhancement content settings file specifying one or more items of enhancement content for each portion of the base item of content, or for an enhancement content pack. The content enhancement server 110 may maintain, for example, popularity ratings, star ratings, or download counts of a particular enhancement content settings file or enhancement content pack. The content enhancement server 110 may also maintain or access a social graph affiliated with a user of the user computing device 102 to determine and recommend which enhancement content settings files or enhancement content packs have been accessed by persons in the user's social graph.
The recommendations made by the content enhancement service may be modified by user feedback. For example, the content enhancement service may receive an indication from a user that the user did not enjoy a particular item of enhancement content. Accordingly, that item of enhancement content may not be recommended to the user for use with subsequently-conveyed base items of content or portions of a base item of content. The user may also specify which types of items of enhancement content he or she prefers to be conveyed. For example, the user may prefer songs to sounds. Accordingly, the content enhancement service may recommend songs instead of sounds for portions of a base item of content. Other ways for obtaining and applying user feedback to recommendations for items of enhancement content are possible.
In some instances, the enhancement content menu 606 may not contain an item of enhancement content desired by the user to be conveyed while the portion of the base item of content is conveyed. Accordingly, when a user interacts with import button 610, the user computing device 102 may obtain an item of enhancement content and optionally store it to the user content library 112. For example, the user computing device may obtain an item of enhancement content from the network content library 104 or from a content data store 114 associated with the content enhancement server 110.
Those skilled in the art will appreciate that in some embodiments, an imported item of enhancement content is not stored with the user computing device 102. Rather, the import button 610 may be used to obtain a link to an item of enhancement content stored remotely: for example, on the network content library 104 or in the content data store 114 associated with the content enhancement server 110. The user computing device 102 may stream an imported item of enhancement content substantially while conveying the portion of the base item of content on the user computing device 102. An item of enhancement content to be streamed from a remote source may be marked as remote in an enhancement content slot 604, as noted above.
After assigning items of enhancement content to one or more portions of the base item of content, a user may export their settings by interacting with export settings button 612. For example, the settings generated by the user may be collected into an enhancement content settings file, transmitted to the content enhancement server 110, and optionally stored to the content data store 114. The items of enhancement content that the user has assigned to each portion of the base item of content may collected and store as an enhancement content pack, which may be transmitted to another user computing device 102 or to the content enhancement server 110 for optional storage in content data store 114. Settings files and content packs stored on the content enhancement server 110 may then be transmitted by the content enhancement server 110 to another user computing device 102 upon request. For example, a user may generate a request for an enhancement content pack or enhancement content settings file by interacting with import button 614. Alternately, the user may import an enhancement content settings file or enhancement content pack stored or previously generated by the user on his or her user computing device 102.
Once the user has assigned items of enhancement content to portions of the base item of content to his or her content, the user may interact with the save and play button 616. The user's assignments of items of enhancement content may be stored in memory on the user computing device 102, and the base item of content may then be conveyed to the user, along with any items of enhancement content that the user has selected to be conveyed.
If the user is dissatisfied with the assignments of enhancement content to one or more portions of the base item of content, the user may interact with the clear all button 618. If the user interacts with the clear button 618, one or more of the content enhancement slots 604 may be emptied. The user may then assign items of enhancement content to each portion of the base item of content as desired.
It should be appreciated in some embodiments, multiple different types of enhancement content are associated with the same portion of the base item of content. For example, a portion of the base item of content may be associated with a song, with a sound clip, with a video clip and/or with a visual content. A portion of the base item of content may be associated with zero, one, or more than one item of enhancement content. Additionally, the items of enhancement content selected by the user need not match the characteristics of the portion of the base item of content to which the item of enhancement content is assigned.
Those skilled in the art will also appreciate that the user interface 600 may be displayed to a human worker of a human interaction task system 108 in conjunction with one or more queries regarding the base item of content and the items of enhancement content. For example, the human interaction task system 108 may pose a query to the human worker such as, “Please select a song for each chapter of this electronic book,” or, “Please identify any enhancement objects that may be present in this portion of the base item of content based upon the following characteristics of the portion.” For example, for a portion of a base item of content set on a farm (location characteristic), the human worker might indicate that sound objects, such as cows, chickens, or pigs, may be present. Items of enhancement content may be selected or recommended by the content enhancement service accordingly; for example, an audio clip of a cow mooing or a chicken clucking Human workers of the human interaction task system 108 may also be requested to generate enhancement content settings files or enhancement content packs using the user interface 600.
By interacting with the enhancement settings button 710, the user may be taken to a user interface such as user interface 600. The user may have the opportunity to add, change, or remove assignments of enhancement content to the base item of content, for example. The user may also be given the opportunity to modify other settings, such as visual settings (e.g. contrast, brightness, gamma, saturation, etc.) or audio settings (e.g. volume, bass, treble, etc.). Interfaces and instrumentalities for adjusting these settings are well known in the art and need not be described in further detail here.
The user may also have the option to toggle whether the user computing device 102 conveys items of enhancement content by interacting with toggle button 712. If enhancement content is disabled, it is not conveyed by the user computing device 102. If enhancement content is enabled, it may be conveyed by the user computing device 102. Toggle button 712 may read “disable enhancement” while enhancement content is enabled, and may read “enable enhancement” while enhancement content is disabled. The user may also have the option of toggling only certain types of enhancement content, e.g., disabling audio content but leaving visual enhancement content enabled.
Conditional language such as, among others, “can,” “could,” “might” or “may,” unless specifically stated otherwise, are otherwise understood within the context as used in general to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment.
Conjunctive language such as the phrase “at least one of X, Y and Z”□ unless specifically stated otherwise, is otherwise understood with the context as used in general to convey that an item, term, etc. may be either X, Y or Z, or a combination thereof. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y, and at least one of Z to each be present.
Any routine descriptions, elements or blocks in the flow diagrams described herein and/or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or elements in the routine. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted or executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved as would be understood by those skilled in the art.
It should be emphasized that many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.
