Patent Grant
7030872
This application claims the benefit under 35 U.S.C. §119 of the following co-pending and commonly-assigned patent application, which is incorporated by reference herein:
United Kingdom Patent Application Number 01 09 741.9, filed on Apr. 20, 2001, by Akemi Tazaki, entitled “IMAGE DATA EDITING”.
This application is related to the following commonly-assigned patents and co-pending patent application, which are incorporated by reference herein:
U.S. Pat. No. 5,892,506, filed on Mar. 18, 1996 and issued on Apr. 6, 1999, by David Hermanson, entitled “MULTIRACK ARCHITECTURE FOR COMPUTER-BASED EDITING OF MULTIMEDIA SEQUENCES”,
U.S. Pat. No. 5,818,542, filed on Apr. 10, 1996 and issued on Oct. 6, 1998, by Stephane Robert Harnois, entitled “PROCESSING IMAGE DATA”,
U.S. Pat. No. 6,084,588, filed on Apr. 8, 1997 and issued on Jul. 4, 2000, by Gisbert De Haan, entitled “INTERACTION BETWEEN MOVING OBJECTS AND MATTE DERIVED FROM IMAGE FRAMES”, which application claims priority to United Kingdom Patent Application No. 9607649 filed on Apr. 12, 1996;
U.S. Pat. No. 6,269,180, filed on Apr. 9, 1997 and issued on Jul. 31, 2001, by Benoit Sevigny, entitled “METHOD AND APPARATUS FOR COMPOSITING IMAGES”, which application claims priority to United Kingdom Patent Application 9607633, now British Patent No 2 312 124; and
U.S. patent application Ser. No. 08/843,282, filed on Apr. 14, 1997, by Raju C. Bopardikar et al., entitled “VIDEO STORAGE”, which application claims priority to U.S. Provisional Patent Application Ser. No. 60/015,468 filed on Apr. 15, 1996 and United Kingdom Patent Application 96 19120 filed on Sep. 12, 1996, now British Patent No. 2 312 319.
1. Field of the Invention
The present invention relates to image data editing and in particular to an apparatus having frame storage means, processing means, manually operable input means and display means.
2. Description of the Related Art
Computerised video or digitised film editing environments are known. In some of these environments, edit decisions are made based on a compressed version of the source material. In alternative environments, online editing is performed on the full bandwidth source material itself. An environment of this type is licensed by the present assignee under the Trademark “FIRE”.
To facilitate editing activities, it is known to display timelines to an editor, such as those described in U.S. Pat. No. 5,892,506 assigned to the present Assignee.
Computerised systems and computer programs for performing effects upon clips of image frames are known. For example, a dissolve effect is described in U.S. Pat. No. 5,818,542 assigned to the present Assignee. An effect for the matching of film like grain upon video material is described in British Patent No. 2 312 124 (U.S. Pat. No. 08/827,641) assigned to the present Assignee. An effect for allowing three dimensional particles to interact with two dimensional video material is described in U.S. Pat. No. 6,084,588 assigned to the present Assignee. Many other effects are also implemented in software licensed by the present assignee under the Trademark “FLAME”.
A culture has been established in which computerised editing activities and computerised effects processing activities are perceived as separate activities. Consequently, established platforms provide for effects processing to be implemented or, alternatively, provide for editing activities to be implemented. Consequently, during an editing operation, it is difficult for an editor to make any changes or place any constraints upon effects that have been implemented within particular clips.
According to a first aspect of the present invention, there is provided image data editing apparatus, comprising frame storage means storing a plurality of clips, processing means, manually operable input means and display means, wherein said display means is configured to display: (a) a first region in which a plurality of said clips representing an output sequence are represented by clip regions on a timeline, and transitions between said clips are represented by edit regions; (b) a pointer configured to be moveable over said timeline, (c) a second region representing input controls, wherein said controls relate to transition operations when said pointer is located over an edit region, and said controls relate to other functions when said pointer is not located over an edit region.
In a preferred embodiment the second region represents a portion of the first region and shows individual frames of the output sequence. Preferably, the display means is configured to move the pointer over the timeline when the output sequence is being played, and the pointer is located when the output sequence stops being played, and the output sequence is stopped by inputs received at said manually operable input means.
Preferably, the pointer is moved and located in response to inputs received at the manually operable input means. Preferably the first region comprises a plurality of timelines; the apparatus is configured to receive a manual input identifying a selection of one of said timelines, and the pointer is located by said selection of one of said timelines.
A system for editing image data is illustrated in
The graphical user interface provides the user with several controls and interfaces for controlling the manipulation of image data. The system also includes a graphics tablet 103, to allow the user to interact with a graphical user interface and a keyboard 104 to facilitate alpha numeric input.
The system further comprises a disk based frame storage system 105, referred to herein as a Framestore. In preparation for image editing and manipulation, images from one or more film or video input reels are transferred to the framestore 105 via a digital tape player or film scanning apparatus etc.
Framestore 105 may be of the type supplied by the present assignee under the Trademark “STONE” and includes several high capacity hard disk drives arranged to supply and store image data in parallel across many individual drives at once. The drives are configured as a redundant array of inexpensive disks (RAID). Further details of the RAID system are disclosed in British Patent No 2 312 319 (U.S. Ser. No. 08/843,282) assigned to the present Assignee.
From the framestore 105 it is possible to play back and record high resolution film images or video images at any location in a clip without having to wait for a tape mechanism to rewind to reach a required frame position, thereby facilitating a process known as non linear editing.
In a typical application, film clips are digitised and stored on digital tape for transfer to the framestore 105. The clips include several camera shots that are to be combined into the same scene. In addition, effects are to be performed and, increasingly, it has been appreciated that the nature of effects and their combination within the editing activity forms part of the overall artistic effect. In particular, the integrity of an effect, i.e. the extent to which it is perceived to be real life, may be influenced not only by the nature of the effects process itself but also by the nature in which the material is combined during the editing process and it is appreciated that these two activities of effects generation and editing are artistically related.
In this example, computer 101 is a silicon graphics octane and includes a CD ROM drive 106. Application software, providing a graphical user interfaced and image editing functionality is installed from a CD ROM 107.
Computer system 101 is illustrated in
Switch 206 enables up to seven different non blocking connections to be made between connected circuits. A graphics card 208 receives instructions from CPU 201 or from CPU 202 in order to render image data and graphical user interface components on display monitor 102. A high bandwidth SCSI bridge 209 allows high bandwidth communication to be made with a digital tape player and framestore 105. An input/output bridge 210 provides input/output interface circuitry for peripherals, including the graphics tablet 103, the keyboard 104 and a network. A second SCSI bridge 211 provides interface connections with an internal hard disk drive 212 providing a capacity of thirteen gigabytes. The second SCSI bridge 211 also provides connections to CD ROM 106, to facilitate the installation of instructions to hard disk 212.
Main memory 207 and its data content are illustrated in
Monitor 102 is shown in
A graphical user interface 400 is divided into a first upper tile 401 and a second lower tile 402. The tiles are scalable but are not floating and therefore remain substantially in position thereby ensuring that particular user operable interface commands remain substantially in the same place.
Upper tile 401 is used to display data relating to media management and lower tile 402 is used to display timelines and related information. The interface provided within tile 401 allows media to be selected and moved and identified as belonging to a particular project, by the creation of metadata representing clip libraries etc.
Timelines shown within tile 402 allow edits to be defined. In particular, data is loaded from input reels to define source clips of material. These source clips are established within input timelines with priorities so as to define the actual image frames that would be included within an output sequence.
Input clips may be arranged such that several clips are displayable in parallel. An output is derived exclusively from one of a plurality of clips, or, alternatively, image data from two or more clips is combined so as to generate an effect.
An example of editing data displayed within lower tile 402 is shown in
Timelines present a graphical representation to an editor as to how edit decisions take place. The scale of the timelines is such that a play back duration of several minutes may be made available. Such a play back duration is desirable for viewing edit decisions but does not allow a representation of the whole of the output sequence to be perceived and does not allow modifications to be made at the individual frame level.
Region 503 is a scroll bar having a length which represents the duration of the output sequence as a whole. A cursor 531 is movable over this bar by operation of the touch tablet, identifying therein a window which is then used to select a portion of the timelines displayed within region 502.
Thus, the first region 503 represents output material of a first duration, preferably the whole duration of the output sequence, while the second region 502 represents a portion of the first region and shows regions representing clips and regions representing transitions between clips in the form of a timeline. Several regions representing clips and transitions between clips are shown in
Region 502 includes tape transport controls 555, operable by the graphics tablet, which have “play”, “stop”, “reverse”, “fast forward” and “rewind” buttons. Region 502 also has a timeline cursor 532 which, after selection of a play operation, possibly using tape transport controls 555, traverses across the screen at frame rate. During play operation, while the cursor 531 moves along the timelines, an area 556 of region 501 displays a moving image representing the composite video output. The moving image is such that at any point in time the area 556 displays a frame identified by a displayed time code 557 and represented by the position of cursor 532 on the timelines.
When play back is stopped, the cursor 532 resides at a particular frame location such that a full frame is present on its left side and a full frame is present on its right side. These individual frames, on either side of timeline cursor 532, may be displayed within a third region 501 such that said third region represents a portion of the second region 502. As an alternative to displaying two adjacent frames, other frames may be displayed within region 501, as will be described below, but as such these still represent a portion of the frames displayed within the second region 502.
Therefore, first region 503 shows the representation of the entire output sequence. Second region 502 displays a portion of the first region and places emphasis on the transitions between clips. A portion of the second region is then displayed within the third region 501 which may allow individual frames to be viewed or alternatively, a moving image. Thus, within a relatively small display region it is possible for an editor to be presented with an overall view of the sequence, a view showing clip transition and a view showing effects upon individual frames.
In addition to displaying individual frames within region 501, it is also possible for modifications to be selected and controlled based on the images displayed within region 501.
The user interface 400 provides an environment in which clips located on the timelines may be edited and effects may be applied to the timelines. In order to facilitate these operations, the user is able to specify a particular timeline to the system as being the timeline to which editing and/or effects operations are to be applied using the graphics tablet 103 or the keyboard 104. I.e. the user is able to specify a particular timeline as the active timeline. On receiving an indication that a specific timeline is to be the active timeline, the user interface 400 indicates the specified timeline by a marker. For example, in
Particular operations performed within region 501, are generally with reference to individual frames. These operations are user selectable. However, in addition, default tools that are dependent upon the state of the system are selected by the system for presentation within region 501. Specifically, the default tools that are presented in region 501 are context sensitive in that if the cursor 532 is located, i.e. motionless at a particular position, such that the cursor 532 lies within a clip on the active timeline, it is assumed by the system that an effect is to be modified and region 501 displays images and tools to facilitate effect modification. Similarly, if the cursor 532 is located at a position on the currently active timeline that corresponds to a transition edit, it is assumed that the nature of the transition is to be modified and the system displays default tools relevant to the modification of that transition.
The cursor 532 may be located at a particular position on the active timeline in a number of different ways, each of which will affect the default tool display in region 501. Firstly, during the play-back operation, in which moving images are presented in region 501, the play may be stopped, by, for example, the user activating the stop button of the tape transport controls, and thus the cursor 532 will be stopped and located at a particular position; secondly, the user may move the cursor, using tape transport controls such as “fast forward” or “rewind”, or drag the cursor 532 using the graphics tablet etc. and locate it at a new position; thirdly, the user may select a different timeline as the active timeline and thus the cursor 532 becomes located at a particular position on the newly active timeline.
In terms of modifying editing parameters, a group of operations related to transition edits are identified. In this example, these consist of the following:
Similarly, effects operations in this example may be summarised as follows:
An example of images displayed in region 501 when the timeline cursor is located at a cut is illustrated in
In the event that the timeline cursor 532 is located at a transition edit icon, or the user of the system uses the graphics tablet 103 to identify a particular transition edit icon, then default tools relevant to the transition are presented in region 501. An example, showing default tools displayed within region 501 for a wipe, is illustrated in
Region 501 also displays duration controllers 704, which may be manipulated in order to adjust the duration of the wipe, and an acceleration control curve 705 containing a control point 706 which may be manipulated in order to adjust the varying speed at which the wipe occurs. For example, the wipe may be adjusted so that it starts slowly and gets faster towards its completion.
An example, showing default tools displayed within region 501 for a dissolve is shown in
Tile 402 of
The cursor 532 then proceeds to move rightwards at frame rate while corresponding output video images 902 are displayed within region 501. The output images continue to be displayed until the cursor 532 has moved to a position which is a calculated period after the end of the transition. In the present embodiment, the second calculated period is also the greater of 15% of the transition duration and 1 second. Thus, in the case of transition 554, cursor 532 moves rightwards until it reaches a position indicated by dashed line 903.
On completion of the playing of the output corresponding to the transition, the cursor 532 is moved by the system back to a stationary position over the transition, and region 501 resumes the display of the relevant transition editing images and tools, as shown in
In summary of
The locating of the cursor 532 at a position within a clip calls an effects default interface, as illustrated in
In addition to the effects modification screen shown in
Flow charts illustrating the contextual manner in which the system displays editing and effects tools are shown in
If it is determined at step 1206 that the cursor stopped at a transition edit, and so it is now located at a transition edit, the process enters the transition edit procedure at step 1207. On leaving step 1207 the system determines whether further effects modification has been requested by the user at step 1208. If this is so, then the process returns to step 1201 where the effects modification procedure is re-entered. If step 1207 finds that a requirement for effects modification has not been indicated, then the present session of the effects/transition editing is ended at step 1209.
If the system determines at step 1202 that play-back has not been requested, it is determined at step 1210 whether user inputs have been received which indicate that transition editing is required. If transition editing has been requested, then the process enters step 1207 in which transition editing may take place. If it is determined at step 1210 that editing is not required then the effects/ transition editing session is ended at step 1209.
The effects modification procedure of step 1201 is shown in further detail in the flow chart of
At step 1304 a question is asked as to whether inputs have been received indicating that effects modification procedure should be exited. This indication may take one of several forms including: inputs indicating that the cursor 532 should moved so that it is located at a transition on the active timeline (e.g. by dragging the cursor); inputs indicating the selection of a transition edit icon; inputs indicating that the active timeline should be changed, resulting in the cursor 532 being located at a transition; and inputs indicating that play-back of the output sequence is required. If the question asked at step 1304 is answered “no” then step 1302 is re-entered and effects modification continues. Alternatively, if the question at step 1304 is answered “yes”, then step 1201 is completed and the process enters step 1202.
The transition edit procedure of step 1207 is shown in greater detail in the flow chart of
At step 1405 it is determined whether inputs have been received indicating that the transition edit procedure should be exited. This indication may take one of several forms including: inputs indicating that the cursor should be moved to a position so that it is no longer located on a transition (e.g. moved to the middle of a clip); and inputs indicating that a different timeline should become the active timeline and such that the cursor is no longer located on a transition on the active timeline. If it is determined at step 1405 that exit from the transition edit procedure is not required then step 1406 is entered.
At step 1406 it is determined whether play-back of the output sequence has been requested by the user. If this is so, then the transition edit portion of the output sequence is displayed at step 1407, as described with reference to
If it is determined at step 1405 that exit of the transition edit procedure is required by the user, then step 1207 is completed.
The system therefore provides a graphical user interface which allows a user to perform editing of clips and also apply/ modify effects. Furthermore, it provides a display of tools which is context sensitive, in that transition editing tools or effects modification tools are displayed in dependence upon the location of the timeline cursor.
| Number | Date | Country | Kind |
|---|---|---|---|
| 0109741 | Apr 2001 | GB | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4937685 | Barker et al. | Jun 1990 | A |
| 5353391 | Cohen et al. | Oct 1994 | A |
| 5440348 | Peters et al. | Aug 1995 | A |
| 5519828 | Rayner | May 1996 | A |
| 5675752 | Scott et al. | Oct 1997 | A |
| 5818542 | Harnois | Oct 1998 | A |
| 5892506 | Hermanson | Apr 1999 | A |
| 6084588 | De Haan | Jul 2000 | A |
| 6204840 | Petelycky et al. | Mar 2001 | B1 |
| 6269180 | Sevigny | Jul 2001 | B1 |
| 6473094 | Sheasby et al. | Oct 2002 | B1 |
| 20010036356 | Weaver et al. | Nov 2001 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20020154140 A1 | Oct 2002 | US |
