A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the U.S. Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
In the course of producing a picture, such as a movie, it is common for an audio engineer (sound editor) to add audio clips (or segments) as well as audio properties (e.g., equalization, reverberation, voice matching, room sounds) to audio clips. The audio clips and audio properties therefore can form a sound mix (or audio mix) for a corresponding video track. The audio clips can be provided in one or more audio tracks for the video track. The formation of a sound mix takes substantial effort and can be referred to as audio production.
However, while audio production is being performed on a picture (video cut), a picture editor often separately continues to edit the picture. For example, the picture editor might move or edit various sequence of the video track. Thereafter, the edited video can be provided to the audio engineer to add, modify and/or delete audio clips. The audio engineer is then required to manually rearrange the sound mix from the former video to fit with the edited video.
Conventionally, when manually conforming a sound mix to a new video edit, an audio engineer (sound editor) imports the picture editor's newly arranged audio clips and video into an existing multi-track project on new tracks. Next, the audio engineer can walk through the audio edits one by one, comparing the old placement of audio clips to the placement of new audio clips. Audio clips from the former video cut must be adjusted so that they are properly aligned with the new video cut. Audio clips may also need to be added or deleted because a scene was deleted or added.
There are software programs that assist audio engineers with placing and editing audio clips, including configuring audio properties for the video clips. One example of an existing software program for audio editing/production application is “Soundtrack Pro” available from Apple Inc. of Cupertino, Calif. Even so, it is a tedious manual task of adjusting audio clips from a former video sequence to a new video sequence. Hence, there is a need to provide improved approaches to adjust audio clips from one video sequence to another video sequence.
The invention pertains to methods, graphical user interfaces, computer apparatus and computer readable medium for merging different versions of a digital media asset (e.g., movie). For example, a user of a computing device can utilize the methods, graphical user interfaces, computer apparatus, and computer readable medium to merge modifications separately and concurrently made to a digital media asset in production. The digital media assets can be audio, video or graphical.
The invention can be implemented in numerous ways, including as a method, system, device, apparatus (including graphical user interface), or computer readable medium. Several embodiments of the invention are discussed below.
As a computer-implemented method for conforming first and second sequences derived from a base sequence into a resultant sequence, one embodiment of the invention can, for example, include at least: comparing each clip in the base sequence to each clip in the first sequence and each clip in the second sequence to produce comparison information; categorizing each clip of the first sequence and each clip of the second sequence into one of a plurality of predetermined categories based on at least the comparison information; determining probable placement of at least a plurality of clips from the first sequence and the second sequence for the resultant sequence based on at least the categorizations of the clips into the predetermined categories; and presenting the resultant sequence with the plurality of the clips from the first sequence and the second sequence placed therein in accordance with the probable placements.
As a computer readable storage medium including at least executable computer program code tangibly stored thereon for conforming first and second sequences derived from a base sequence into a resultant sequence, one embodiment of the invention can, for example, include at least: computer program code for selecting the first sequence to process; computer program code for categorizing each element of the first sequence into one of a plurality of predetermined categories in relationship to at least one element in the base sequence; computer program code for selecting the second sequence to process; computer program code for categorizing each element of the second sequence into one of a plurality of predetermined categories in relationship to at least one element in the base sequence; computer program code for determining probable placement of at least a plurality of elements from the first sequence and the second sequence for the resultant sequence based on at least the categorizations of the elements into the predetermined categories; and computer program code for presenting the resultant sequence with the plurality of the elements from the first sequence and the second sequence placed therein in accordance with the probable placements.
As a method for merging changes made with respect to a first sequence of digital media elements and a second sequence of digital media elements into a resultant sequence of digital media elements, the first sequence and the second sequence being derived from a base sequence, one embodiment can, for example, include at least: categorizing each digital media element in the first sequence into one of a set of predetermined categories in relationship to the base sequence; categorizing each digital media element in the second sequence into one of a set of predetermined categories in relationship to the base sequence; and determining probable placement of at least a plurality of digital media elements from the first sequence and the second sequence into the resultant sequence.
As a system for merging versions of digital media assets, one embodiment of the invention can, for example, include at least: a video production module configured to produce a first video sequence and a second video sequence, the second video sequence being a later modified version of the first video sequence; and an audio production module configured to perform audio editing to the first video sequence, thereby producing a modified version of the first video sequence having additional audio elements. The audio production module can further include a conform function configured to produce a third video sequence that includes video elements from the second video sequence and the additional audio elements from the modified version of the first video sequence. The video production module and the audio production module can be separate software application programs or can be provided by a single software application program.
As a computer-implemented method for identifying and tracking lineage of media elements, one embodiment of the invention can, for example, include at least: creating or retrieving a media element for use in a media sequence; assigning a unique identifier to the media element; subsequently duplicating or modifying the media element to form another media element; and appending an additional unique identifier to the one or more unique identifiers already associated with the media element, thereby producing a hierarchical identifier that is assigned to the another media element.
As a computer readable storage medium including at least executable computer program code tangibly stored thereon for conforming first and second sequences derived from a base sequence into a resultant sequence, another embodiment of the invention can, for example, include at least: computer program code for analyzing media content corresponding to a clip in the first and second sequences to provide media content information; and computer program code for determining probable placement of the clip in the resultant sequence based on the media content information.
Other aspects and advantages of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings which illustrate, by way of example, the principles of the invention.
The file of this patent contains at least one drawing executed in color. Copies of this patent with color drawing(s) will be provided by the Patent and Trademark Office upon request and payment of the necessary fee.
The invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements, and in which:
The invention pertains to methods, graphical user interfaces, computer apparatus and computer readable medium for merging different versions of a multimedia project (e.g., movie). For example, a user of a computing device can utilize the methods, graphical user interfaces, computer apparatus, and computer readable medium to merge modifications separately and concurrently made to a multimedia project in production. The multimedia project can include audio, video and/or graphical elements.
In one embodiment, the invention can be implemented by a conform tool. A conform tool is able to automatically merge two versions of the same multimedia project in production. The merged version can be reviewed. In particular, the conform tool can also assist with the review of various merge decisions, which can be accepted, declined or altered. The different versions can results from a single program or from multiple programs.
Although the invention is largely described below as merging versions with respect to positioning of media elements (e.g., clips) in media sequences, it should be understood that modifications to different versions are not limited to positioning but can additionally or alternatively pertain to one or more of duration (length), media, and media offset.
Although the invention is primarily described herein as pertaining to merging different versions of a multimedia project, it should be understood that the invention is not limited to multimedia projects or media sequences. For example, the merging of different version of any data can utilized the conform processing discussed below. Besides media data, the data could, for example, be notes, markers, annotations, etc. that are added or associated with different versions.
Embodiments of the invention are discussed below with reference to
The video production program 102 can be used to create an original project that includes at least one video track and may also include various audio tracks. For example, a picture editor can use the video production program 102 to manipulate video clips/tracks to form a desired resulting video (i.e., video sequence), such as for a movie. Often the video clips/tracks contain or are associated with audio clips.
The original project can be provided to an audio production program 104 so that a sound editor (audio editor) can create an original audio mix to be used with the video. Typically, the audio mix is a mixture of audio sounds from a plurality of different audio tracks. However, as the sound editor is creating the audio mix, the picture editor is producing an updated project (e.g., having at least an altered video track) as compared to the original project. As a result, it is not uncommon for an updated project to be provided to the sound editor after the sound editor has created at least parts of the audio mix to be used with the video. Hence, the audio production program 104 can include a conform process that assists the sound editor is moving the original audio mix associated with the original project to a resulting audio mix for the updated project. A resulting project can be formed that uses the video from the updated project and the resulting audio mix for the audio. In one embodiment, the conform process recommends or suggests positions and/or properties for audio elements of the original audio mix. In other words, the conform process can attempt to conform the original audio mix from the original project to match the video from the updated project.
The conform process 200 can receive 200 a first video cut (video sequence). The first video cut can, for example, be provided by a video production program. After receiving 200 the first video cut, audio elements can be added, removed or modified 204 to the first video cut. For example, one or more audio elements in one or more audio tracks can be added with the first video cut.
A decision 206 then determines whether a second video cut has been received. The second video cut is a video cut of a video provided sometime after the first video cut. The second video cut has been modified as compared to the first video cut. When the decision 206 determines that the second video cut has not been received, then the conform process 200 can return to repeat the block 204 so additional audio elements can be added, removed or modified 204. On the other hand, when the decision 206 determines that the second video cut has been received, the conform process 200 can analyze 208 the first video cut and the second video cut to propose positioning and/or properties of the audio elements with respect to the second video cut. The proposed positioning and/or properties of the audio elements with respect to the second video cut can then be accepted, declined or changed 210. In other words, the proposed positioning and/or properties can be reviewed. For example, a user (e.g., sound editor) of the audio production program can review the proposed positioning and/or properties. By reviewing the proposed positioning and/or properties, the user can accept, decline or change the proposals. A graphical user interface can assist the user in reviewing (e.g., accepting, declining or changing) the proposed positioning and/or properties.
The options process 250 can initially select 252 a first audio element of the first video cut. Next, one or more possible options for the selected audio element can be determined 254 with respect to the second video cut. These different options can differ in position and/or properties for the selected audio element. In addition, the likelihood of the one or more possible options can be determined 256. The possible option having the highest likelihood can then be chosen 258. The chosen possible option can also be referred to as the proposed option. Thereafter, the chosen possible option can be presented 260 along with a confidence indication. The confidence indication is a visual indication that can be displayed to inform the user of the confidence the system has in the chosen possible option. As this point, the options process 250 is completed so processing can return to block 210 of
Initially, the conform process 300 produces 302 a first video sequence with the video production program. The first video sequence in the video production program can then be saved 304.
Next, the first video sequence can be modified 306 in the video production program. The modified first video sequence can then be saved 308 as a second video sequence. Concurrent with the modifying 306 and the saving 308 of the second video sequence, the conform process 300 can also open 310 the first video sequence in the audio production program. Then, using the audio production program, audio elements of the first video sequence can be modified 312.
Following the block 312, a decision 314 can determine whether a conform request has been made. Here, a user of the audio production program can interact with a graphical user interface to request that a conform operation be initiated. As an example, the conform operation can be initiated by a menu selection or a button action. When the decision 314 determines that a conform operation has not been requested, the conform process 300 can return to repeat the block 312 so that the user of the audio production program can continue to modify 312 the audio elements within the first video sequence and/or request a conform operation.
On the other hand, when the decision 314 determines that a conform request has been made, the first video sequence and the second video sequence to be conformed are selected 316. Here, the first video sequence is the video sequence resulting from block 312, and the second video sequence is the second video sequence saved in block 308. The conform process 300 can then operate to determine 318 how elements of the first video sequence having subsequently modified audio and the second video sequence have changed as compared to the first video sequence which was saved at block 304. Next, most likely positions and/or properties for the audio elements can be selected 320 to match the second video sequence. A resultant sequence can then be formed 322 based on the second video sequence with audio elements in their selected position and/or properties.
Thereafter, the resultant sequence can be presented 324 within a graphical user interface of the audio production program. The graphical user interface can be referred to as a conform graphical user interface. After the resultant sequence is presented 324, the proposed positions for the audio elements can be reviewed 326. In doing so, the user of the audio production program can accept, decline or change the proposed positions and/or properties for the audio elements. Following the block 328, the resultant sequence can be saved 330. After the block 330, the conform process 300 can end.
The review process 400 can initially display 402 a list of clips in the resultant sequence to be reviewed. In one embodiment, the list of clips includes those one or more clips (e.g., audio elements) that have been added, changed or removed relative to the first sequence. In one implementation, the list of clips can provide a confidence level for the proposed change, and positional reference positions (original, resulting and/or difference) for each of the clips. A decision 404 then determines whether one of the clips in the resultant sequence has been selected. When the decision 404 determines that a clip has not been selected, the review process 400 can await a selection of a clip.
Once the decision 404 determines that a clip has been selected (e.g., by a user of the audio production program), the selected clip can be highlighted 406 in the displayed list of clips. More generally, by highlighting the selected clip, the selected clip is displayed in a visually distinct manner. In other words, the selected clip is distinctively displayed. In addition, clip details pertaining to the selected clip can be displayed 408. The clip details provide detailed information regarding the selected clip, including for example: name, whether position/duration has changed, and/or whether media for the clip has changed. The position of the selected clip can also be distinguishably displayed with respect to a timeline. Alternatively, when the clip is selected using the timeline, the selected clip can be distinctively displayed in the list of clips.
In reviewing the clips within the list of clips, the user can interact with the audio production program. With respect to
Following the block 418, or following the decision 420 when the selected clip has not been accepted, a decision 422 can determine whether the review process 400 should end. When the decision 422 determines that the review process 400 should not, then the review process 400 returns to repeat the decision 404 so that another clip can be processed in a similar manner. Alternatively, when the decision 422 determines that the review process 400 should end, then the review process 400 can end.
According to embodiment of the invention, the conform process can assist users in reviewing and approving proposed position and/or properties for audio clips being conformed from one multimedia project to another multimedia project. Many of these embodiments can utilize graphical user interface components. Various examples of graphical user interface components are discussed below in
The conform window 500 includes a first video timeline 502 that can pertain to an original project, and a second video timeline 504 that can pertain to an updated project. The conform window 500 also includes an audio clip review region 506. The audio clip review region 506 can provide information on proposed positioning of audio elements with respect to the updated project. The audio clip review region 506 displays a list of audio elements. For each of the audio elements, the audio clip review region 506 can display status, clip name, confidence, change (e.g., whether changed or type of change), position change, duration change, offset change, etc. A selected audio element 508 can be highlighted in the audio clip review region 506. The position of the selected audio element 508 can also be visually indicated (e.g., highlighted) in the first video timeline 502 at visual indicator 503 as well as in the second video timeline 504 at visual indicator 505. Also, for the selected audio element 508, the audio clip review region 506 can also provide a detail region 510. The detail region 510 can present detailed information concerning the nature of the modification of the audio element being proposed. For example, the detail region 510 can provide information on position, duration and/or media. The detailed region can also include (or have proximate thereto) a confidence indicator 511 and an approve control 512 (e.g., approve button). The confidence indicator 511 provides an indication of the degree of confidence (or confidence level) with the positioning of the selected audio element. The user can approve a proposed modification by selecting the approve control 512. The conform window 500 can also include a finish control 514 (e.g., finish button) to enable the user to end the review of the conform process.
The audio clip review region 506 can also include a status indicator 516 to visually indicate that the associated proposed modification has been approved. The status indicator 516 can be visually illustrated for each of the audio clips within the audio clip review region 506 that have been already approved by the user. In one embodiment, the status indicator 516 can be a graphical symbol such as a symbol, icon or the like. The approval of an audio clip within the audio clip review region 506 can, for example, be performed using the approve control 512. Advantageously, the user of the project edit window 506 can receive a visual indication of those of the audio clips within the audio clip review region 506 that have already been approved. Approval can, for example, denote acceptance of an associated proposed modification for an audio clip (or a group of audio clips).
The list of audio elements in the audio clip review region 506 can also be filtered using filter controls 518 and 520. The filter control 518 can operate to cause those of the proposed modifications that are “unchanged” to be hidden from being displayed in the list of audio elements in the audio clip review region 506. The filter control 520 can operate to cause those of the proposed modifications that have been “approved” to be hidden from being displayed in the list of audio elements in the audio clip review region 506. More particularly, upon selection of the filter control 520, those previously approved audio clips are removed from the audio clip review region 506. For example, since three of the audio clips are denoted as being approved by the status indicators 516, such particular audio clips would no longer be displayed in the audio clip review region 506 if the filter control 520 were activated. In other words, by selection of the filter control 520, those of the audio clips that have been approved are no longer presented in the audio clip review region 506. As such, the audio clip review region 506 can serve as a list of those remaining audio clips that have yet to be reviewed and approved by the user.
The confidence level 522 assigned to the proposed modifications in the list of audio element in the audio clip review region 506 can assist the user in determining whether to approve the proposed modifications. A group control 524 can be used to influence the extent to which audio clips are grouped together for review.
The editor pane 606 includes a plurality of selectable tabs 626. As illustrated in
The control pane 608 can include a playhead position control, transport controls, and a selection length control to facilitate editing of digital media assets associated with a multimedia project.
The project edit window 640 includes the project edit pane 604 similar to that discussed above with respect to
The project edit window 640 also includes the editor pane 606. In this embodiment, the editor pane 606 illustrates the conform tab being selected from the selectable tabs 626, and a graphical user interface for conform review. Namely, within the editor pane 606, the graphical user interface for conform review includes a project selection control 642 that enables a user to select a project to be displayed. In this embodiment, the project selection control 642 can select one of an original project, an updated project, or a result project.
As illustrated in
The editor pane 606 also includes an audio clip review region 644. The audio clip review region 644 can provide information on proposed positioning and properties of audio elements with respect to the selected project. The audio clip review region 644 displays a list of audio elements. For each of the audio elements, the audio clip review region 644 can display various attributes (e.g., position and/or properties) for clips, including: status, clip name, confidence, change (e.g., whether changed or type of change), position change, duration change, offset change, etc. A selected audio element 646 can be highlighted. Also, for the selected audio element 646, the editor pane 606 can also provide a detail region 648. The detail region 648 can present detailed information concerning the nature of the modification of the audio element being proposed. For example, the detail region 648 can provide information on position, duration and/or media. The detail region 648 can also include (or have proximate thereto) a confidence indicator 649 and an approve control 650 (e.g., approve button). The confidence indicator 649 provides an indication of the degree of confidence (or confidence level) with the positioning of the selected audio element. The user can approve a proposed modification by selecting the approve control 650. The editor pane 606 can also include a finish control 652 (e.g., finish button) to enable the user to end the review of the proposed modifications provided by the conform process.
Also, in the event that the user desires to edit any of the media assets in the project edit pane 604, the user can do so at any time. Hence, the user is not limited to accepting proposed position and/or properties for the selected audio element 646 identified by the conform process but can choose to instead directly edit the corresponding media asset.
Further, the detail region 648 can indicate one or more possible states that the conform process identified for the selected audio element 646. Examples of possible states can include one or more of: added, moved, use original clip, and unchanged. The unchanged state means that the conform process is not suggesting positional change for a selected audio element. The states can also include media states that impact the media of the audio element. The media states can include one or more of: unchanged, slipped, and use original clip. A user can select one of the states to be utilized for the selected audio element 646 in the result project. If the user selects a different state than the current selected state, then the position (and/or properties) of the selected audio element 646 will be changed so that the information in the audio clip review region 644 is updated as needed. The project edit pane 604 can also be updated to indicate the updated position of the selected audio element 646.
In one embodiment, the conform process might in some cases not be able to reconcile changes for an audio clip between the modified original project and the updated project. For example, changes made to the modified original project and the updated project might conflict and require user decision to resolve. As another example, if the backing media for an audio clip is changed to a different media file, the conform process might not be able to resolve which media file (original or replacement) is to be used. The conform process can make its best guess for the desired position, but the audio clip can be visually identified as being conflicting. A graphical user interface can also provide alternative positions for a clip so that a user can elect to decline a default position (or the best guess) and instead select an alternative position.
The list of audio elements in the audio clip review region 644 can also be filtered using filter controls 654 and 656. The filter control 654 can operate to cause those of the proposed modifications that are “unchanged” to be hidden from being displayed in the list of audio elements in the audio clip review region 644. The filter control 656 can operate to cause those of the proposed modifications that have been “approved” to be hidden from being displayed in the list of audio elements in the audio clip review region 644. A confidence level 658 assigned to the proposed modifications in the list of audio element in the audio clip review region 644 can assist the user in determining whether to approve the proposed modifications. The confidence level 658 can indicate the degree of confidence the conform process has in the proposed modification (e.g., proposed position) for a particular audio element. A group control 650 can be used to influence the extent to which audio clips are grouped together for review. In other embodiments, other filter controls can be used based on other conditions, such as confidence level, changes to media file, offset, name/text, etc.
In one embodiment, the selected audio element 646 can also be linked to the project timeline 616. In such case, when the selected audio element 646 is selected, the project timeline 616 can scroll horizontally to bring the corresponding portion into center view within the project edit window 662, and the playhead for playback can also be moved to align with the beginning of the selected audio element 646. Consequently, a user is able to visually appreciate the selected audio element in context of the project. The user can also easily initiate playback of the selected audio element if desired, which can be helpful when seeking to determine whether the proposed position of the selected audio element should be approved. Additionally, the selection of an audio element within the project timeline 616 can also cause the corresponding audio element in the list of audio elements list of audio elements in the audio clip review region 644 to be visually identified. Also, if a different state is selected from the detail region 648 for the selected audio element 646, display of an associated timeline (e.g., project timeline 616, updated timeline 666) can also update to distinctively identify the selected audio element 646 at the position corresponding to the selected state.
The audio clips being clustered into a group depends on the group control 660. The conform tool can, in one embodiment, group the audio clips based on how close together they are on a timeline, and how similarly they were changed between their states between the original project and the updated project. In one embodiment, the farther the group control 660 is moved (slid) to the right, the more the grouping constraints on groups are loosened, and when the group control 660 is moved to the far left, there are no groups.
The audio clips within the selected group 670 are indicated within the original timeline 664 and also within the updated timeline 666. The original timeline 664 can include a visual indication 676 for each of the audio clips within the selected group 670, and the updated timeline 666 can include a visual indication 674 for each of the audio clips within the selected group 670. A visual comparison of the visual indications 676 and the visual indications 674 provides some perspective on the relative position changes that the associated audio clips have undergone.
Upon selection of the group within the audio clip review region 644, the detail region 648 can display a group indication 672. By selection of the approve control 650, the audio clips associated with the selected group 670 can be approved as a group. Hence, using the group control 660, facilitates efficient review of the audio clips that are to be reviewed within the audio clip review region 644.
To assist with locating of clips in different video cuts (or sequences), each clip can be provided with an identifier. The identifiers can thus be used to locate clips that have moved or otherwise altered in the different video cuts. An identifier is, for example, a unique identifier (UID). The VPP 102 or the APP 104 can operate to assign identifiers to the clips. A clip can have a plurality of identifiers so that its history can be understood. Each clip can thus have its lineage in a series of identifiers. New clips are assigned identifiers. New clips can result from adding a clip, splitting a clip, duplicating a clip, or copying & pasting a clip. For example, it is fairly common for one initial clip to be split into two or more clips during editing. A clip that is split into two clips will yield two clips with each having the identifier of the initial clip as well as having a new unique identifier for each new clip. The identifiers can be assigned to clips when the clips are created or when exported by VPP 102 to AAP 104, or when imported by AAP 104 from VPP 102. The identifier history for a multimedia project having the clips can be provided in a markup language (e.g., XML) format.
The clip identification process 700 can begin with a decision 702. The decision 702 can determine whether a video sequence is being provided. For example, a video sequence can be provided by being exported from the VPP 102 and imported by the APP 104. When the decision 702 determines that a video sequence is not being provided, the clip identification process 700 can await until a video sequence is being provided. Once the decision 702 determines that a video sequence is being provided, all identifiers for clips in the video sequence are determined 704. A first of the identifiers is then selected 706. One or more clips having the selected identifier can then be determined 708. Next, a decision 710 can determine whether two or more clips have the selected identifier. For example, clips can have the same identifier due to a split or duplication of the clip. When the decision 710 determines that two or more clips have the selected identifier, an additional identifier (i.e., unique identifier) can be associated 712 to each of the two or more clips having the selected identifier.
Following the block 712, or directly following the decision 710 when there are no new clips, a decision 714 determines whether there are more identifiers. When the decision 714 determines that there are more identifiers, the clip identification process 700 can return to repeat the block 706 and subsequent blocks so that a next identifier can be selected 706 and similarly processed. Once the decision 714 determines that there are no more identifiers to be processed, a decision 716 can determine whether there are any remaining clips, namely new clips, without an identifier. When the decision 716 determines that there are new clips without an identifier, a new identifier can be assigned 718 to each of the new clips. Following the block 718, or directly following the decision 716 when all clips have identifiers, the clip identification process 700 can end.
The clip placement process 800 operates to process or more sequences as discussed below. The clip placement process 800 can initially select a first sequence to be processed. The sequence being selected is typically a media sequence having a plurality of clips. The sequence is typically part of a project which includes one or more different sequences. After the first sequence to be processed has been selected 802, each clip of the selected sequence can be categorized 804 into one of a set of predetermined categories in relationship to at least one clip in a base sequence. Next, a decision 806 can determine whether there are more sequences to be processed. When the decision 806 determines that there are more sequences to be processed, the clip placement process 800 returns to repeat the block 802 so that a next sequence can be selected and processed in a similar fashion.
On the other hand, when the decision 806 determines that there are no more sequences to be processed, probable placement of at least a plurality of clips from the selected sequence in a resultant sequence can be determined 808 based on at least the categorization of the clips. Here, in conforming the sequences, clips are processed so as to be intelligently conformed to the resultant sequence. Next, the resultant sequence with the plurality of clips from the selected sequences placed therein in accordance with the probable placements can be presented in 810. Following the block 810, the clip placement process 800 can end.
The clip placement process 850 can initially select 852 a sequence to process. The selected sequence is processed in comparison to a base sequence. A first clip in the base sequence can be selected 854. An identifier for the selected clip can then be obtained 856. Thereafter, all clips in the selected sequence that include the obtained identifier can be identified 858. Each of the identified clips can then be categorized 860 as same, modified, added or deleted. A decision 862 can then determine whether there are more clips to be processed. When the decision 862 determines that there are more clips to be processed, the clip placement process 850 can return to the block 854 so that a next clip in the base sequence can be selected and similarly processed.
On the other hand, when the decision 862 determines that there are no more clips in the selected sequence to be processed, a decision 864 determines whether there are more sequences to be processed. Typically, two sequences, namely, a first sequence (e.g., first video sequence or first video cut) and a second sequence (e.g., second video sequence or second video cut) are processed. When the decision 864 determines that there are more sequences to be processed, the clip placement process 850 can return to repeat the block 852 so that another sequence to be processed can be selected and similarly processed.
Alternatively, when the decision 864 determines that there are no more sequences to be processed, the clip placement process can identify 866 change region information within at least one of the selected sequences. For example, the change region information can consider were the first and second sequences have contiguous sections that have been moved or deleted. More generally, the first and second sequence can be analyzed to locate regions that have particular meaning,
Probable placements, if any, of the clips from the selected sequences into a resultant sequence can then be determined 868. In the determination 868 of the probable placement of the clips into the resultant sequence, the clip placement process 850 can be based on at least the categorizations and/or the change region information. Thereafter, the resultant sequence with clips from the selected sequences that have been placed at probable placements can then be presented 870. After the resultant sequence has been presented 870, the clip placement process 850 can end.
In one embodiment, the blocks 856 and 858 use a unique identifier (UID) to identify clips in a selected sequence that match clips in a base sequence. Typically, this is performed for two different selected sequences, a first sequence and a second sequence. At block 860, the identified clips in the selected sequence can be categorized into different lists of clips. These different lists of clips can include: added clip list, deleted clip list, same clip list, and modified clip list. For a given selected clip in the base sequence, if there are no identified clips in the selected sequence that match, then the selected clip has been deleted. In this case, the selected clip is added to the deleted clip list.
If the selected clip in the base sequence has a matching clip in the selected sequence, then the properties of the respective clips can be compared to determine whether the clip has been modified. For example, a clip in the selected sequence can be modified as compared to the base sequence by being moved in the sequence, by being re-sized, by changing its underlying media, by changing media offset, or by changing tracks. In any event, if the clip has not been modified, then the matching clip in the selected sequence is added to the same clip list. Alternatively, if the clip has been modified, the matching clip is added to the modified clip list.
When UIDs are used, categorization of the identified clips with respect to the different lists of clips can be performed as follows. If the identified clip has a UID in the base sequence but not the selected sequence, the identified clip is deemed a deleted clip and thus is added to the deleted clip list. If the UID of the identified clip in the selected sequence is not in the base sequence, the identified clip is deemed an added clip and thus added to the added clip list. Further, if the identified clip in the selected sequence does not have a UID, the identified clip can also be deemed an added clip and thus added to the added clip list.
In some cases, there may be more than one matching clip in the selected sequence. These matching clips can be processed in various different ways. In one embodiment, considering properties of the matching clips as compared to the properties of the selected clip in the base sequence, the best matching clip is determined and such matching clip can be added to the modified clip list. The remaining matching clips are treated as newly added clips and are placed in the added clip list. In one implementation, if one or more of the matching clips overlap the length of the selected clip in the base sequence, such clips can be deemed related and all such related matching clips can be added to the modified clip list, with any remaining, non-related matching clips being placed in the added clip list. Multiple matching clips can be deemed related to a selected clip in the base sequence, such can occur, for example, when a clip is split into two parts that roughly stay together.
In another embodiment, when the clips to not have identifiers (e.g., UIDs) that can be used to determine those clips that match, a heuristic approach can be utilized. As an example, a heuristic can consider various properties of clips, such as start time, length, slip (offset) and/ media, to compute a probability of clips matching one another. The properties can be weighted differently. Those one or more properties that have a probability greater than a threshold amount can be considered matching clips. For example, if a clip in the selected sequence does not match any of the identified clips in the base sequence above the threshold amount, then the clip is considered an added clip. As another example, if no clip in the resultant sequence matches an identified clip in the base sequence above the threshold amount, then the clip is considered a deleted clip.
In one implementation, the probability is a number between zero and one can be assigned to each of the identified clips. In one embodiment, the identified clips are considered with respect to the selected clip in the base sequence. For a given identified clip, the identified clip having the highest probability of matching the selected clip is chosen. If the highest probability of all the one or more identified clips with respect to the chosen clip is zero, then the chosen clip is deemed a deleted clip and thus is added to the deleted clip list.
On the other hand, if the highest probability of all the one or more identified clips with respect to the chosen clip is greater than zero, further processing can categorize each of the one or more identified clips as added, modified or same. If the chosen clip has a probability of zero, there is no matching clip, so the chosen clip is treated as a newly added clip and thus placed in the added clip list. If the chosen clip has a probability of one, it is considered a matching clip, so the chosen clip is treated as the same and placed in the same clip list. Otherwise, if the chosen clip has a probability greater than zero but less than one, the chosen clip is probably a matching clip, so the chosen clip is placed in the modified clip list.
In one embodiment, the block 868 determines probable placements of clips from the selected sequences into the resultant sequence using the different lists of clips, namely, added clip list, deleted clip list, same clip list, and modified clip list, for each of the first and second sequences. The probable placements can be configured as default placements. Optionally, the probable placements can include one or more alternative placements. Probable placement of clips from the selected sequence into the resultant sequence can be determined as follows:
A. Those clips in both the same clip list for the first sequence and the same clip list for the second sequence are considered unchanged clips and are therefore added to the resultant sequence.
B. Those clips in the added clip list for the second sequence are added clips and are therefore added to the resultant sequence.
C. Those clips in the deleted clip list for the second sequence are either deleted or provided in the resultant sequence. If such clips are also in the same clip list for the first sequence, then the clips can be deleted (i.e., not added to the resultant sequence). If such clips are not also in the same clip list for the first sequence, then such clips can be added to the resultant sequence.
D. Those clips in the modified clip list for the second sequence are either provided in the resultant sequence, merged into the resultant sequence or deleted. If such clips are also in the same clip list for the first sequence, then the clips can be added to the resultant sequence. If such clips are also in the deleted clip list for the first sequence, then such clips can be deleted (i.e., not added to the resultant sequence). If such clips are also in the modified clip list for the second sequence, then such clips can be added to the resultant sequence in a manner that merges the changes from both the first sequence and the second sequence. For example, an effort is made to preserve changes made to both the first sequence and the second sequence. Multiple options can also be provided particularly when the changes conflict such that changes are not able to be preserved. For example, if the second sequence was modified by moving a video clip thirty seconds forward in time and the first sequence was modified by moving an audio clip one second forward in time, the resultant sequence can include the modified clip from the second sequence at its modified position thirty seconds forward in time and also the modified clip from the first sequence at a modified position thirty-one seconds forward in time. Here, the modified clip from the first sequence is able to be positioned relative to the moved position of the associated clip of the second sequence.
E. Those clips in the same clip list for the second sequence are either deleted or provided in the resultant sequence. If such clips are also in the deleted clip list for the first sequence, then the clips can be deleted (i.e., not added to the resultant sequence). If such clips are in the modified clip list for the first sequence, then such clips can be added to the resultant sequence while retaining the modifications to such clips via the first sequence.
F. Those clips in the added clip list for the first sequence are added to the resultant sequence. If the corresponding portion (i.e., overlapping portion) of the second sequence has moved, the placement of the clips in the resultant sequence can depend on the movement of the second sequence. For example, if a clip (or series of clips) was modified by a positional movement in the second sequence, the clip added in the first sequence can be placed in the resultant sequence by applying the positional movement to the clip being added. Optional placement for these clips can be its original placement which can also be made available as another alternative.
In one embodiment, when it is determined that an original clip is split into a plurality of clips in the first sequence (which still overlap with the position of the original clip), then they can be considered as a group to make the confirm process more manageable and more user friendly. Also, when the original clip is separately and concurrently moved in the second sequence, the plurality of the clips in the first sequence can be treated as a group and all be moved in the same manner in which the original clip was moved in the second sequence.
In one embodiment, the media (or media content) used by a clip (e.g., audio clip) can be used as a factor in determining where to position the clip in the resultant sequence. For example, when the position of a media clip (e.g., video clip) differs between the first sequence and the second sequence, processing can evaluate whether an audio clip being provided in the resultant sequence should utilize a position corresponding to the media clip in the first sequence or in the second sequence. For example, such processing can analyze media content for information (“media content information”), such as time overlap or regions of important media content such as high amplitude or voice-spectrum audio content. The media content information can then be used to assist in making the positional determination (e.g., recommendation). In one implementation, time overlap can consider the duration of time the audio clip overlaps video clip(s) in determining position of the audio clip in the resultant sequence. As an example, if an audio clip overlaps both a first video clip and a second video clip in the first sequence, the audio clip can be considered to be more probable to be positioned with the video clip that most overlaps the audio clip. In one implementation, the audio content of the audio clip can be examined and used in determining position of the audio clip in the resultant sequence. As an example, if an audio clip overlaps both a first video clip and a second video clip in the first sequence, the audio clip can be considered to be more probable to be positioned with the video clip that overlaps with meaningful audio content of the audio clip. For example, if the audio content of the audio clip is more meaningful at the last twenty-five percent of its duration, then the portion of the video clip that best overlaps that more significant portion of the audio content can be considered to be the more probable position for the audio clip. The analysis of the audio content to determine more meaningful regions can, for example, be based on volume level and/or frequency spectrum, since greater volume and/or content with an interesting frequency signature (e.g., voice-spectrum audio indicating dialogue) may signal more important audio.
Additional details on media production are contained in: (i) U.S. patent application Ser. No. 11/735,468, filed Apr. 14, 2007, and entitled “MULTI-TAKE COMPOSITING OF DIGITAL MEDIA ASSETS,” which is hereby incorporated herein by reference; (ii) U.S. Provisional Patent Application No. 60/911,884, filed Apr. 14, 2007, entitled “TECHNIQUES AND TOOLS FOR MANAGING ATTRIBUTES OF MEDIA CONTENT,” which is herein incorporated herein by reference; and (iii) U.S. patent application Ser. No. 11/735,466, filed Apr. 14, 2007, and entitled “MULTI-FRAME VIDEO DISPLAY METHOD AND APPARATUS,” which is hereby incorporated herein by reference.
The various aspects, features, embodiments or implementations of the invention described above can be used alone or in various combinations.
The invention is preferably implemented by software, hardware, or a combination of hardware and software. The invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of the computer readable medium generally include read-only memory and random-access memory. More specific examples of computer readable medium (i.e., computer readable storage medium) include Flash memory, EEPROM memory, memory card, CD-ROM, DVD, hard drive, magnetic tape, and optical data storage device. The computer readable medium can also be distributed over network-coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.
The advantages of the invention are numerous. Different aspects, embodiments or implementations may, but need not, yield one or more of the following advantages. One advantage of the invention is that digital media assets (e.g., audio clips) associated with a first video cut can be automatically processed to be placed on a second video cut. Another advantage of the invention is that placement of audio clips from one video cut to a subsequent video cut can be automatically proposed. Still another advantage of the invention is that proposed placement of digital media assets can be efficiently reviewed for approval or disapproval.
The many features and advantages of the present invention are apparent from the written description. Further, since numerous modifications and changes will readily occur to those skilled in the art, the invention should not be limited to the exact construction and operation as illustrated and described. Hence, all suitable modifications and equivalents may be resorted to as falling within the scope of the invention.
This application claims priority to U.S. Provisional Patent Application No. 60/911,886, filed Apr. 14, 2007, entitled “MULTIPLE VERSION MERGE FOR MEDIA PRODUCTION”, which is herein incorporated herein by reference. This application is also related to U.S. patent application Ser. No. ______,filed Apr. 14, 2008, entitled “MULTIPLE VERSION MERGE FOR MEDIA PRODUCTION”, which is herein incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
60911886 | Apr 2007 | US |