Patent Grant
7716054
Various multi-media providers offer diverse mechanisms with which to capture and/or record content. For instance, content can be captured via cameras, video cameras, audio recorders, and devices that can combine features of such recorders. Furthermore, as mechanisms for storing captured content have traversed from analog to digital realms, new paradigms for memorializing content, maintaining integrity of stored content, retrieval of such content and review of stored content have been experienced.
Recent focus has particularly been directed at review of saved content. For example, supply tools are now available for visualizing textual data, summarizing and condensing video and/or photographic data, and the like. Such tools have been useful for many educational and professional pursuits. For example, various professions require summarization and review of recorded media, such as auditory content, including, e.g., speech, conversation, musical works, and video content, including, e.g., live or simulated audio/visual events. For instance, physicians, psychiatrists and psychologists often record patient interviews to preserve information for later reference and to evaluate patient progress. Patent attorneys typically record inventor interviews so as to facilitate review of a disclosed invention while subsequently drafting a patent application. Broadcast news media is often recorded and reviewed to search for and filter conversations related to particular topics of interest. In general, along with a capacity for storing large quantities of distributed media, a need has arisen for reviewing and summarizing recorded media information.
Summarization can refer broadly to a shorter, more condensed version of some original set of information, which can preserve some meaning and context pertinent to the original set of information. Summaries of some types of information have received more attention than others. Particularly, visual and video content have been widely and persistently studied for mechanisms to identify and distinguish segments of interest. To a lesser degree, audio content has also received some attention toward facilitating review and summary of such content.
Review of audio content can be particularly useful as much information exchanged in verbal meetings can be lost. For example, while taking notes on a lecture using pen and paper, vast quantities of detail, not memorialized on such notes can be lost shortly after the lecture. Recording content from a meeting, whether face-to-face or over a remote communication platform (e.g., telephone, computer network, etc.) can be a valuable mechanism for preserving such content. However, difficulties can arise in regard to recordings as well, typically related to review of information. For example, scanning through hours of media recordings can take an amount of time commensurate with capturing the recording in the first place. Consequently, summaries that provide facilitated review of information can enhance efficiencies associated with such review.
The following presents a simplified summary in order to provide a basic understanding of some aspects of the claimed subject matter. This summary is not an extensive overview. It is not intended to identify key/critical elements or to delineate the scope of the claimed subject matter. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
The subject matter disclosed and claimed herein, in various aspects thereof, provides for identifying points of interest within oral conversations and marking file locations related to such points in order to facilitate a conversation's review. Points of interest can be identified based on a level of energy, e.g., excitement, pitch, tone, pace, or the like, associated with one or more speakers. Alternatively, or in addition, speaker and/or reviewer activity can form the basis for identifying points of interest within the conversation. In such a manner, audio content can be succinctly summarized with reference to particular points of interest, inferred or provided by a device user.
In accordance with additional aspects of the subject disclosure, a compilation of identified points of interest within audio content is provided. A content navigation component can retrieve portions of audio content within a threshold proximity to an identified point of interest. In addition, such portions can be extracted from a media file and compiled within a separate file. Furthermore, a user can indicate particular points of interest that can be saved and additional points of interest that can be discarded. As described, the disclosed subject matter can provide for a customized compilation of audio content, pertinent to the needs of a particular individual.
Further aspects of the claimed subject matter provide for analyzing and compiling diverse media types related to identified portions of audio content. A context component can identify a time and location associated with a recording device (e.g., utilizing an electronic clock and global or satellite positioning mechanism). A network search component can access a data network, such as the Internet, and determine information pertinent to the time and/or location. As an example, local weather, news, travel information, sales data, pertinent stock information, or various other data can be compiled by the networks search component and associated with identified points of interest in an audio media file. In addition, such compilation can be directed by stored user preferences. Furthermore, a compilation of various media types, including video, photographic, textual, related to the identified points of interest can be generated, along with the contextual data, to form a diverse composition of audio/visual data related to identified points of interest.
The following description and the annexed drawings set forth in detail certain illustrative aspects of the claimed subject matter. These aspects are indicative, however, of but a few of the various ways in which the principles of the claimed subject matter may be employed and the claimed subject matter is intended to include all such aspects and their equivalents. Other advantages and distinguishing features of the claimed subject matter will become apparent from the following detailed description of the claimed subject matter when considered in conjunction with the drawings.
The claimed subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the claimed subject matter. It may be evident, however, that the claimed subject matter may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the claimed subject matter.
As used in this application, the terms “component,” “module,” “system”, “interface”, or the like are generally intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. As another example, an interface can include I/O components as well as associated processor, application, and/or API components, and can be as simple as a command line or a more complex Integrated Development Environment (IDE).
Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term “article of manufacture” as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. For example, computer readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips . . . ), optical disks (e.g., compact disk (CD), digital versatile disk (DVD) . . . ), smart cards, and flash memory devices (e.g., card, stick, key drive . . . ). Additionally it should be appreciated that a carrier wave can be employed to carry computer-readable electronic data such as those used in transmitting and receiving electronic mail or in accessing a network such as the Internet or a local area network (LAN). Of course, those skilled in the art will recognize many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.
Moreover, the word “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word exemplary is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.
As used herein, the terms to “infer” or “inference” refer generally to the process of reasoning about or inferring states of the system, environment, and/or user from a set of observations as captured via events and/or data. Inference can be employed to identify a specific context or action, or can generate a probability distribution over states, for example. The inference can be probabilistic—that is, the computation of a probability distribution over states of interest based on a consideration of data and events. Inference can also refer to techniques employed for composing higher-level events from a set of events and/or data. Such inference results in the construction of new events or actions from a set of observed events and/or stored event data, whether or not the events are correlated in close temporal proximity, and whether the events and data come from one or several event and data sources.
Referring to
As used herein with respect to audio analysis, the term ‘energy level’ is intended to include, but not be limited to, speaker volume. In addition, energy level can include excitement level, activity level, animation level, or interest level, or the like, measured by speaker(s) volume, pitch, tone, rate of speech, and a rate of change in any of the foregoing or like qualities. For instance, a rate or volume at which a speaker articulates words can indicate excitement level or a desire to express a point, and can be related to energy level. A speaker rapidly articulating words can indicate a high excitement level or a relatively intense desire to express a point. Conversely, a speaker articulating words more slowly can indicate a lower level of excitement. Changes in pitch, volume, tone, etc., and/or a frequency of such changes can also provide an indication of energy level, as used herein. Furthermore, a number of speakers speaking concurrently can also be included within an energy level determination. For example, a single person speaking can demonstrate a relatively low energy level, whereas multiple speakers speaking at once, in effect talking over each other, can demonstrate a higher energy level, as used herein. Consequently, energy level is to be construed as including all of the foregoing, and not limited to just speaker volume.
Specifically, an inference component 104 can determine a point of interest based at least in part upon identification of an energy level in an oral conversation 106. Such point of interest can be represented by an energy level higher than a threshold energy level, for instance. Alternatively, or in addition, the threshold energy level identifying a point of interest can be established by a default within system 100, or a user defined parameter (e.g., a more detailed discussion of user preferences is discussed in regard to
Inference component 104 can determine an energy level within oral conversation 102 at least in part based on measurable auditory indications. For example, a pitch of a speaker's voice, volume of a speaker's voice, voice tone, length and frequency of pauses between words, a number of speakers speaking simultaneously, or like indication can be utilized to determine the energy level in the oral conversation. Non-verbal auditory indications can also be utilized, such as foot tapping, or hand tapping, or a speaker banging a desk, table, or the like.
In addition, inference component 102 can continuously monitor an oral conversation in order to identify points of interest as the oral conversation progresses. Specifically, an energy level can be determined as part of real-time or near real-time measurement and calculation (e.g., of auditory indications, as described above) that can determine contemporaneous energy levels and changes in such levels. As a specific example, measurements of speaker pitch, volume, tone, pause frequency and/or number of speakers can be taken every second. It should be appreciated that equipment for measuring auditory indicators such as that described above can take measurements much more frequently (e.g., thousands of times per second, or more frequent), or much more slowly (e.g., once every 10 or 20 seconds, or less frequent) as suitable. Furthermore, measurement frequency can be dynamically adjusted in accord with aspects of the subject disclosure, based on frequency of change in measured energy level, or like factor, for instance.
System 100 can also identify a point of interest within an oral conversation 102 based on speaker activity. A verbal cue, such as ‘great idea’, or the like, can form the basis for a point of interest. Optionally, a loud or sharp noise can trigger identification of a point of interest, such as banging on a desk, a speaker clapping their hands, or similar event. Predetermined auditory indications can also motivate inference component 104 to identify a point of interest. For example, a predetermined word or phrase such as ‘record point of interest’, or the like, can be utilized. As another example, snapping one's fingers can trigger such a point, or clapping hands, tapping a foot, or any suitable auditory trigger known in the art, or made known to one of skill in the art by way of the embodiments articulated herein.
It should be appreciated that inference component 104 can also identify points of interest for more than a single individual. An energy level associated with one or more particular speakers can trigger identification of points of interest particular to those speakers, for example. Furthermore, one or more distinct activity triggers (e.g., verbal cue, snapping finger, clapping hand, tapping foot, press of a button on a device, a show of hands or other somatic indication of approval or dissent, and/or a number of such hands, etc.) associated with a particular speaker(s) can be utilized to generate a point of interest particular to the speaker(s). As a more specific example, 5 speakers can each record an auditory trigger and associate the trigger with their name, and store their name and associated trigger on one or more components of system 100 (e.g., inference component 104, annotation component 106, media store 108 etc.). If inference component 104 detects a trigger associated with a speaker, a point of interest can be generated and also associated with the speaker.
Inference component 104 can identify points of interest upon review of an audio and/or visual recording as well as during the initial recording. For instance, implicit or explicit activity of one or more reviewers can be utilized to identify points of interest (e.g., particular to the one or more reviewers, and/or common to multiple reviewers) within the reviewed recording. As an example, points of interest can be generated implicitly based on actions such as writing notes, biometric indicators/actions, discussion amongst the reviewers and an energy level, as defined herein, associated therewith, or the like, upon review of content. As a further example, points of interest can be identified explicitly based on a predetermined trigger (e.g., verbal word/phrase, somatic gesture, and or sound, such as snapping finger, tapping foot, etc.) or based on marking, voting, show of hands, or the like.
Furthermore, the inference component 104 can identify points of interest based on implicit or explicit patterns of usage. For instance, one or more implicit activities, as defined herein, typically exhibited by a person during a conversation, discussion, lecture, or the like, or during review of such conversation, etc. Additionally, explicit actions previously detected, or established as a predetermined trigger associated with an individual, can be utilized to identify points of interest with respect to current content.
In addition to the foregoing, inference component 104 can identify a point of interest from implicit or explicit non-auditory cues related to the oral conversation. For instance, a video recording component (not shown) can capture video associated with the conversation or with a review of the conversation (e.g., a review can include studying recorded lesson notes, reviewing a recorded conversation, analyzing an audio recording for content, or the like). If a particular visual event occurs, such as measurable level of movement, a particular somatic gesture, or the like, inference component 104 can identify a point of interest at a correlated point in the oral conversation. Alternatively, or in addition, explicit annotations by means of marking, voting, note taking (e.g., an act of writing commensurate with a point in an audio recording) or the like (e.g., show of hands, pressing a button on a device, such as a cell phone, remote controller, wireless device, or the like) can be utilized to identify points of interest. Furthermore, other implicit measures of engagement via interaction patterns during initial recording or subsequent review can be used to identify points of interest.
After (or, e.g., contemporaneous with) identifying a point of interest, system 100 can label a portion of the conversation 102 where the point of interest occurred. Specifically, an annotation component 106 can mark an audio file at a location associated with the point of interest. The marked audio file can be stored, for example, at a media store 108. A location marked by annotation component 106 can be referenced quickly (e.g., by a playback device or summarization component such as that depicted at FIG., infra) for review at a later time. Moreover, annotation component 108 can provide one or more distinct marks for one or more speakers and/or users of system 100 (e.g., which could include a recording device, such as but not limited to a hand-held audio or audio/visual recorder, a camera, camcorder, or the like). Furthermore, annotation component 106 can mark points of interest inferred by inference component 104 (e.g., from a measured energy level) in a distinct manner as opposed to points of interest identified in relation to speaker activity (e.g., such as a somatic gesture, verbal or non-verbal auditory trigger, or the like).
Annotation component 106 can also mark an audio file in a manner commensurate with user preferences (e.g., as discussed in regard to
In accordance with aspects of the subject disclosure, it should be appreciated that media store 108 can include nonvolatile memory, such as read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory, volatile memory including random access memory (RAM), such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). Additionally, media store 108 can include removable/non-removable, volatile/non-volatile computer storage media, such as devices like a magnetic disk drive, floppy disk drive, tape drive, Jaz drive, Zip drive, LS-100 drive, flash memory card, or memory stick. In addition, media store 108 can include storage media separately or in combination with other storage media including, but not limited to, an optical disk drive such as a compact disk ROM device (CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RW Drive) or a digital versatile disk ROM drive (DVD-ROM).
In addition, system 200 can store context related to an oral conversation 204 at media store 206. The context can be included within the audio media file, discussed above, or in a separate file related to such media file. Additionally, the context can be provided by a context component 208, that can determine a time and/or a location associated with the oral conversation 204. More specifically, context component 208 can reference a time measurement device (e.g., electronic clock) associated with a recording device, such as audio organization component 202, or a data store, such as media store 206, and capture one or more times pertinent to the oral conversation. For instance, a time at which the conversation began and ended can be determined by context component and associated with oral conversation 204 and/or media store 206 (e.g., in a manner related to the audio media file stored thereon). Moreover, a date and time sequence can be recorded by context component 208. Such date and time sequence can be included within/appended to the audio media file, and provide a date and time of recording for oral conversation 204 and various portions thereof. Such data and time can be displayed during playback and/or retrieval of portions of the audio media file. Additionally, context component can record a time that a location of the audio media file is marked by audio organization component 202 and associate such time with the file and/or the marked portion of the file. In such a manner, system 200 can provide time and data information associated with a conversation and/or review of the conversation.
In addition to the foregoing, context component 208 can also provide location information related to an oral conversation 204. For example, a global position system (GPS) or satellite position system (SPS) included, for instance, on a recording device (e.g., audio organization component 202) can be referenced to determine location information related to capture and/or analysis of oral conversation 204. Specifically, a location where a conversation took place can be identified and included within or in a file related to an audio media file stored at media store 206. Additionally, one or more locations associated with an analysis, summarization, and/or review of audio content (204) can also be identified and included/stored with such content. In such a manner, contextual information can be provided to facilitate recollection of events associated and potentially pertinent with the content (204).
As a specific, non-limiting example, system 200 can record and/or analyze a monologue or dialogue (e.g., a discussion) about a particular painting of interest to one or more speakers. Typically, if no context associated with the recording is captured and preserved, a speaker may lose track of where the discussion occurred, and what it was in relation to. For example, a discussion may include topics that were on a speaker's mind at the moment of discussion that provided context and meaning for aspects of the discussion itself. Additionally, such aspects could lose meaning, or become nonsensical, without the context. Moreover, if over time the context is forgotten, and not specifically articulated within the discussion, the meaning provided by the context could be lost.
To help remedy the foregoing problem, system 200 can provide additional information, such as date, time, and location pertaining to a discussion, to enable recollection of non-articulated (and, e.g., non-recorded) context. For instance, system 200 can store a determination that the discussion took place in Seattle at 12:35 pm PST, December 1 of a particular year. Additionally, a place of interest associated with location information determined by context component 208 (e.g., a public museum) can be included. Such information can assist in recalling a context that was on a speaker's mind when a discussion was recorded, and assist the speaker in recalling meaning not specifically articulated in the discussion. In such a manner, system 200 can facilitate summarization and review of audio content, and provide additional context therewith.
System 300 can also obtain information from one or more communication networks (312) to identify additional contextual information associated with audio content (304) and/or recording/analysis thereof. Network search component 310 can access a remote network 312, and perform searching and filtering functions on components of network 312. For instance, one or more data servers (not shown) associated with network 312 can be accessed. Such data servers can include powerful search engines that can rapidly access and filter data in relation to specified search criteria. Specifically, search criteria can be included by network search component 310 related to time and/or location information generated by context component 308. In addition, network search component 310 can access conversation 304 or a related data filed stored at media store 306, to extract additional search criteria associated with the conversation 204.
As a non-limiting example, suppose conversation 304 is related to artwork on display at The Louvre Museum in Paris. Network search component can obtain information from the discussion (e.g., keywords utilized within conversation 304, such as a discussion of particular art objects, or the like) as well as time, date and location information from context component 308. Utilizing such information, network search component 310 can access a data network 312 (or, e.g., components thereof such as search engines, servers, databases, and the like) such as the Internet to identify additional contextual information related to conversation 304. For instance, weather-related information can be identified and retrieved. More specifically, network search component 310 can find the temperature, precipitation level, humidity, degree of sunshine, wind, or other suitable data related to weather, in Paris near The Louvre Museum on a particular day associated with conversation 304. In addition, network search component can determine other data related to particular objects of art (e.g., discussed in conversation 304) appearing in local, regional, or global news, or the like. Similarly, related data can include information about people associated with the conversation. Moreover, travel information, political news, financial news, or any suitable combination thereof or of the like, related to the conversation, a topic discussed, a speaker(s) or reviewer(s) involved, or the like, can be determined by network search component 310 via network 312, in a manner articulated herein or known in the art.
Information determined by network search component 310 can be included within an audio media file containing conversation 304 and/or an analysis/summary thereof (e.g., a summary including marked portions of interest as discussed herein). Such information can be included as metadata, or like type of extended file information, for instance. Alternatively, or in addition, information gathered by network search component 310 can be stored in a separate but related file on media store 306 or an associated storage device. It should be appreciated that not all information pertinent to audio content 304 and available on a network 312 such as the Internet or a large corporate intranet, or the like, can be specifically articulated as embodiments herein. However, such information deemed pertinent to one of skill in the art is incorporated within the subject specification. As described, system 300 can access powerful network search engines in order to identify contextual information associated with audio content, and store such information in conjunction with a summary/review of the content. Consequently, system 300 can facilitate recollection speaker meaning associated with an oral conversation 304 via such contextual information.
In addition to the foregoing system 400 can facilitate customized summarization of audio content in accord with aspects of the subject innovation. More specifically, user input component 404 can store one or more user profiles associated with one or more users of a device (402) or speakers within a conversation (not shown). A user profile can indicate various defaults in conjunction with a particular user's preferences. For instance, a user can specify a preferred mechanism for analyzing audio content and identifying points of interest. More specifically, user input component 402 can store a default tone, pitch, volume, rate of speaking, frequency and duration of pauses, or the like in order to establish a threshold energy level associated with the speaker. Subsequently, audio organization component 402 can utilize such default threshold energy level in order to infer points of interest within audio content involving the speaker (e.g., as discussed in more detail at
User input component 404 can also assist a user in establishing user-specific activities associated with identifying points of interest. As an example, a user can record a particular verbal trigger, such as ‘cool idea’, that audio organization component can utilize to identify an activity-based trigger for the user. The verbal trigger can be something the user typically does when encountered with an interesting idea, thought, or topic of discussion. Additionally, non-verbal triggers can be included, such as a snap of a finger, clapping of hands, clucking of tongue, tapping of fingers, press of a button on a device (e.g., a cell phone in radio frequency (RF) communication with system 400, such as via a Bluetooth-type mechanism) or any suitable combination of these or like triggers. Such non-verbal triggers can also be associated with a user's preferred activity-based point of interest triggers. Consequently, a user/speaker can customize a mechanism in which points of interest can be identified by the user/speaker activity.
In accord with additional aspects of the subject disclosure, system 400 can include user preferences related to bookmarking portions of an audio file. For example, a user can define a particular bookmark to be associated with the user. Any point of interest, either inferred or triggered for instance, related to the user can be labeled on an audio media file utilizing the particular trigger. For instance, the label can include the user/speaker's name, or other information unique to the speaker. Alternatively, or in addition, a summary display device (not depicted) providing a visual review of audio content and bookmarked portions, can display a user's bookmark in a particular color, font, size, shape, or like distinguishing characteristic. In such a manner, system 400 can facilitate customized summary, analysis and review of audio content by incorporating preferences established by a specific user.
In accordance with particular aspects of the claimed subject matter, system 500 can identify portions of diverse content 506 related to points of interest identified by audio organization component 502, and associate such portions of diverse content 506 with an audio media file containing labeled audio content as disclosed herein. Multi-media component 508 can receive and analyze portions of diverse media types (506). For instance, diverse media 506 can include photographic media, textual media, video media, as well as additional types of audio media including music, lyrics, and the like. Additionally, multi media component 508 can determine portions of diverse content related to a point of interest identified by audio organization component 502, and associate such diverse content and audio content (e.g., in a diverse compilation file 510).
For example, if audio content contains a discussion of a song title, multi media component 508 can search diverse content 506 (e.g., obtained and provided by network search component 306, or like component) and identify a sound clip related to the song title (e.g., containing all or a subset of the song). Additionally, multi media component 508 can identify a photograph of performers associated with the song title, and/or a video clip of a television interview, music video, news discussion, or the like associated with the song title. Moreover, multi-media content 508 associate such portions of diverse media types associated with the point of interest. For example, one or more files containing such diverse media (506) can be linked to a bookmarked portion of an audio file containing the audio content (stored at 504 for instance). Additionally, multi-media component 508 can extract associated portions of the diverse media types, including the audio content, and compile a diverse compilation file 510.
Diverse compilation file 510 can be a summary of all points of interest identified with respect to a particular audio content (not depicted), and various forms of media content related to such points of interest. As such, diverse compilation file 510 can be a pseudo-electronic scrapbook of particular audio points of interest, containing related video, photographic, musical etc., media associated with those points of interest and providing rich, diverse, and varied context therewith. In addition, further contextual information can be included within diverse compilation file 510, including time and/or location information (e.g., as provided by context component 208 discussed in more detail at
It should be appreciated that diverse compilation file 510 can be incorporated within a presentation application, suitable for compiling and displaying/playing varied media types. For example, a presentation application, which can include a PowerPoint-type application or like application, for instance, can be employed to compile diverse types of media related to audio content and identified points of interest therein. Furthermore, such compilation (510) can be done via remote network connection, utilizing a plurality of network storage devices, servers, or the like (e.g., in conjunction with network search component 308). Additionally, multi-media component 508 can compile diverse media related to audio content in accord with predetermined, suitable user preferences. In such a manner as described above, system 500 can compile rich and diverse information related to one or more portions of audio content in accord with aspects of the subject disclosure.
In addition to the foregoing, system 600 can search, locate, retrieve and playback stored portions of audio content. Specifically, navigation component 606 can retrieve a portion of an audio file proximate to a marked location. Consequently, any suitable location marked by audio organization component 602 can be retrieved by navigation component 606 for subsequent review. Additionally, navigation component 606 can display (e.g., on a user interface display of a device) an indication of custom bookmarks or custom generated points of interest (e.g., in accord with user specified preferences, discussed in more detail at
For instance, navigation component 606 can retrieve all bookmarked points of interest related to a first speaker. In addition, navigation component 606 can play back portions of the bookmarked content. Optionally, navigation component 606 can also display, play or present associated media types related to various points of interest identified and stored by system 600 (e.g., compiled by a multi-media component 508, as discussed above). Consequently, system 600 can facilitate quick, efficient and customized review of audio content stored in accord with particular embodiments described herein.
Summarization component 702 can access an audio media file stored at media store 704. A point of interest marked on the media file can be identified by summarization component 704. For example, such point of interest can be marked via metadata or like extended information associated with the file by audio organization component 704. Such metadata, or the like, can be recognized by summarization component 702 to identify a marked point of interest. Once marked points of interest are identified, summarization component 702 can compile a list of such points and locations, and extract portions of the audio media file that are within a threshold proximity thereto. For example, portions of the audio media file 15 seconds prior and 15 seconds after a marked point of interest can be extracted. Alternatively, or in addition, portions can be identified by data size (e.g., 30 kilobytes prior and after a marked file location), or can be determined as a function of relevance to the point of interest. It should be appreciated that like mechanisms for identifying an appropriate size portion of an audio file, known in the art or made known to one of skill in the art by way of the context provided herein, are incorporated into the subject disclosure.
As a more specific example, summarization component 702 can automatically determine an appropriate size portion (e.g., to determine how much of a file to extract for summary 708) as a function of relevancy to the marked point of interest. Specifically, summarization component 702 can analyze a first portion of an audio media file within a first proximity to a marked point of interest (e.g., within a certain time, data size, or the like). The first portion can be translated into text, and a first set of keywords, representative of such first translated portion, can be extracted from the translated text. In addition, portions of the audio media file within a smaller, second threshold proximity to the marked point of interest (e.g., a fraction of the size of the first marked portion) can be analyzed and keywords relevant to such smaller portion can be extracted. Subsequently, summarization component 702 can compare relevancy of keywords within the first portion to keywords within the second portion. A determination can then be made as to a point where relevancy of keywords within the second portions drops below a default relevancy factor (e.g., established with respect to keywords of the first portion). A size of the first portion can be limited to the point where extracted keywords fall below the relevancy factor. Content within the limited portion can be extracted from the audio media file and incorporated into the summary 708. Other content can be ignored by system 700, or discarded as discussed below.
In addition, system 700 can discard portions of an audio media file contained within media store 704 in order to preserve free memory space. Parsing component 710 can analyze the audio media file and compare it to the summary 708 generated by summarization component 702. Portions of the audio media file that are not extracted by the summarization component and included within summary 708 can be discarded by parsing component 710. Furthermore, parsing component 710 can be toggled on and/or off by a user preference file, a button on a device, or like mechanism. When toggled off, no data is discarded by system 700. In such a manner, system 700 can automatically free up stored memory that is not related to identified points of interest, if a device user so desires.
The aforementioned systems have been described with respect to interaction between several components. It should be appreciated that such systems and components can include those components or sub-components specified therein, some of the specified components or sub-components, and/or additional components. For example, a system could include inference component 104, annotation component 106, network search component 310, and network 312, or a different combination of these and other components. Sub-components could also be implemented as components communicatively coupled to other components rather than included within parent components. Additionally, it should be noted that one or more components may be combined into a single component providing aggregate functionality. For instance, summarization component 702 can include parsing component 710, or vice versa, to facilitate compilation of a pertinent summary and discarding of non-pertinent information via a single component. The components may also interact with one or more other components not specifically described herein but known by those of skill in the art.
Furthermore, as will be appreciated, various portions of the disclosed systems above and methods below may include or consist of artificial intelligence or knowledge or rule based components, sub-components, processes, means, methodologies, or mechanisms (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, classifiers . . . ). Such components, inter alia, and in addition to that already described herein, can automate certain mechanisms or processes performed thereby to make portions of the systems and methods more adaptive as well as efficient and intelligent.
In view of the exemplary systems described supra, methodologies that may be implemented in accordance with the disclosed subject matter will be better appreciated with reference to the flow charts of
Additionally, it should be further appreciated that the methodologies disclosed hereinafter and throughout this specification are capable of being stored on an article of manufacture to facilitate transporting and transferring such methodologies to computers. The term article of manufacture, as used, is intended to encompass a computer program accessible from any computer-readable device, carrier, or media.
Referring to
The conversation can be captured at reference number 802 by various media recording devices. For instance, an audio recorder, such as a tape recorder, or digital sound recorder can be utilized; also a video recorder, camcorder or similar device capable of recording sound along with other media types can be utilized. At reference number 804, locations related to determined points of interest can be marked. Such points of interest can be inferred from speaker activity level or indicated as a function of speaker activity. In addition, marking can include metadata that is included with an audio media file, distinguishing a portion of the file related to the point of interest from other portions of the file, or at least from non-marked portions. At 806, marked portions of the audio file are associated with each other. For instance, a digital link can be created between the marked locations to facilitate efficient retrieval and navigation of marked portions of an audio file. As described, methodology 800 can enable summarization and review of audio content related to identified points of interest.
At reference number 910, the marked portions of the audio file can be navigated to. At 912, portions of the audio file proximate a particular mark can be played. At 914, diverse content related to the marked portions of the audio file can be associated and/or incorporated into the summary file. For instance, video or photographic media related to portions of the audio file marked at reference number 904 can also be associated with the summary compiled at reference number 908. In such a manner, methodology 900 can provide a summary of recorded content, including diverse media types, related to marked portions of audio content.
At 1006, a point of interest can be inferred from at least an energy level of one or more speakers. For instance, energy level can be determined from auditory indications related to the conversation. A pitch, volume, tone, rate of speech, frequency and length of pauses, or a number of speakers speaking concurrently can all be incorporated into a contemporaneous energy level determination. At 1008, a file location is marked proximate to the identified point(s) of interest. Such markings can be in the form of metadata or other extended information suitable for distinguishing one portion of a file from other portions. In addition, the markings can be particular to a particular speaker, enabling identification of points of interest also particular to the speaker.
At 1010, a particular location in an audio file can be marked retroactively. For example, a speaker can determine that a topic of discussion is of interest two minutes after the topic has begun. In order to capture the beginning of the discussion, the speaker can indicate that a mark should be placed two minutes prior to a contemporaneous point in the file. At 1012, portions of the audio file outside of a threshold proximity to identified points of interest can be deleted. At 1014, a summary of non-deleted portions can be compiled into a single file, preserving only the portions that are of interest to one or more speakers, and preserving memory on a storage device. As described, methodology 1000 can facilitate efficient and pertinent summarization of audio content in accord with aspects of the claimed subject matter.
At 1108, related portions of audio and diverse media can be compiled. For instance, results of a search conducted at reference number 1106 can be downloaded, parsed as necessary, and extracted. The extracted information can be compiled into one or more associated files and linked with the related portions of marked audio content. In such a manner as described, methodology 1100 can facilitate compilation of various points of interest within audio content along with a rich and diverse compilation of content associated thereto.
In order to provide additional context for the various aspects of the disclosed subject matter,
Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks and/or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the inventive methods may be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, mini-computing devices, mainframe computers, as well as personal computers, hand-held computing devices (e.g., personal digital assistant (PDA), phone, watch . . . ), microprocessor-based or programmable consumer or industrial electronics, and the like. The illustrated aspects may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. However, some, if not all aspects of the invention can be practiced on stand-alone computers. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
With reference to
The system bus 1218 can be any of several types of bus structure(s) including the memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any variety of available bus architectures including, but not limited to, 11-bit bus, Industrial Standard Architecture (ISA), Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), and Small Computer Systems Interface (SCSI).
The system memory 1216 includes volatile memory 1220 and nonvolatile memory 1222. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer 1212, such as during start-up, is stored in nonvolatile memory 1222. By way of illustration, and not limitation, nonvolatile memory 1222 can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory 1220 includes random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM).
Computer 1212 also includes removable/non-removable, volatile/non-volatile computer storage media.
It is to be appreciated that
A user enters commands or information into the computer 1212 through input device(s) 1236. Input devices 1236 include, but are not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, web camera, and the like. These and other input devices connect to the processing unit 1214 through the system bus 1218 via interface port(s) 1238. Interface port(s) 1238 include, for example, a serial port, a parallel port, a game port, and a universal serial bus (USB). Output device(s) 1240 use some of the same type of ports as input device(s) 1236. Thus, for example, a USB port may be used to provide input to computer 1212 and to output information from computer 1212 to an output device 1240. Output adapter 1242 is provided to illustrate that there are some output devices 1240 like displays (e.g., flat panel and CRT), speakers, and printers, among other output devices 1240 that require special adapters. The output adapters 1242 include, by way of illustration and not limitation, video and sound cards that provide a means of connection between the output device 1240 and the system bus 1218. It should be noted that other devices and/or systems of devices provide both input and output capabilities such as remote computer(s) 1244.
Computer 1212 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1244. The remote computer(s) 1244 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device or other common network node and the like, and typically includes many or all of the elements described relative to computer 1212. For purposes of brevity, only a memory storage device 1246 is illustrated with remote computer(s) 1244. Remote computer(s) 1244 is logically connected to computer 1212 through a network interface 1248 and then physically connected via communication connection 1250. Network interface 1248 encompasses communication networks such as local-area networks (LAN) and wide-area networks (WAN). LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet/IEEE 802.3, Token Ring/IEEE 802.5 and the like. WAN technologies include, but are not limited to, point-to-point links, circuit-switching networks like Integrated Services Digital Networks (ISDN) and variations thereon, packet switching networks, and Digital Subscriber Lines (DSL).
Communication connection(s) 1250 refers to the hardware/software employed to connect the network interface 1248 to the bus 1218. While communication connection 1250 is shown for illustrative clarity inside computer 1212, it can also be external to computer 1212. The hardware/software necessary for connection to the network interface 1248 includes, for exemplary purposes only, internal and external technologies such as, modems including regular telephone grade modems, cable modems, power modems and DSL modems, ISDN adapters, and Ethernet cards or components.
The system 1300 includes a communication framework 1350 that can be employed to facilitate communications between the client(s) 1310 and the server(s) 1330. The client(s) 1310 are operatively connected to one or more client data store(s) 1360 that can be employed to store information local to the client(s) 1310. Similarly, the server(s) 1330 are operatively connected to one or more server data store(s) 1340 that can be employed to store information local to the servers 1330.
What has been described above includes examples of aspects of the claimed subject matter. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but one of ordinary skill in the art may recognize that many further combinations and permutations of the disclosed subject matter are possible. Accordingly, the disclosed subject matter is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the terms “includes,” “has” or “having” are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6076059 | Glickman et al. | Jun 2000 | A |
| 6480826 | Pertrushin | Nov 2002 | B2 |
| 6594630 | Zlokarnik et al. | Jul 2003 | B1 |
| 6972802 | Bray | Dec 2005 | B2 |
| 7502741 | Finke et al. | Mar 2009 | B2 |
| Number | Date | Country | |
|---|---|---|---|
| 20090006082 A1 | Jan 2009 | US |
