REAL TIME ONLINE VIDEO EDITING SYSTEM AND METHOD

Abstract

A system and method for online video editing allowing multiple users to simultaneously edit a production in real time. The system includes a method for exchanging asynchronous messages between a client computer and a server. The system further includes an atomic data structure of commands for manipulating multimedia assets. Client computers subscribe to a production on the server. Changes made to the production on the client computer are applied to the local state of the production and are also transmitted to the server via asynchronous messaging. The server checks for conflicts, resolves the conflicts, applies any necessary changes to the state of the production on the server, and pushes any resulting changes to the production to all subscribing client computers via asynchronous messaging.

Description

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to the field of web-based software, and more particularly, to a system and method for creating interactive multimedia widgets for use on web pages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows client computers connected to a server via the internet.

FIG. 2 shows a block diagram of a system for asynchronous messaging over the internet.

FIG. 3 a shows a flowchart for a computer routine for updating data between a client computer and a server using asynchronous messaging.

FIG. 3 b shows a flowchart for a computer routine for performing conflict resolution on a server when the server receives data from a client computer.

FIG. 3 c shows a flowchart for a computer routine for updating the data on a client computer in response to data received from a server.

FIG. 4 shows a sample representation of a data structure.

FIG. 5 shows one possible representation of a graphical user interface as seen on a client computer.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to an online project, referred to as a production, for the creation of a widget (or any other form of a portable chunk of code that can be installed and executed within any separate html-based web page by an end user without requiring additional compilation) or video that includes any media with interactive properties. In some embodiments of the present invention, the widget may display a movie including one or more forms of interactive or non-interactive media.

In some embodiments of the present invention, each production exists as a set of data on the server. Individual users can create external links to a production such that it can be viewed in a web-browser or such that it can be embedded as an object in a webpage. Further, users may run the client version of the present invention in a web-browser and use it to edit a currently existing production or to create an entirely new production to which that user, or other users, may then subscribe. Subscribing to a production shall mean the creation of a channel of communication between the client computer and the server through which the client computer receives real time updates of the current state of the production from the server. Thus, when a user subscribes to a production, the client computer receives constant feedback and updates from the server, as discussed below, so that the client computer always has the most recent version of the production.

Referring to FIG. 1, according to an embodiment, the present invention includes one or more client computers 110 and a server 120 running a connection manager 130 where either the server, the connection manager, or both are in communication with a backend server 180, and where the client computers 110 communicate with the server via the Internet 140. The client computers may be a personal computer, a PDA, a cellular phone, or any other device with a graphical user interface capable of sending and receiving data. The client computer may be connected to the server with a wireless or wireline connection and may communicate with the server over a local area network, a wide area network, or the Internet.

Referring to FIG. 2, according to an embodiment of the invention, backend server 280 is an instance of backend server 180, server 220 is an instance of server 120, client 210 is and instance of client computers 110 and connection manager 230 is an instance of connection manager 130. Server 220 is connected to client 210 by one or more TCP (often HTTP over TCP) connections 240. Server 220 receives queries 260 from the client computers 210 and communicates responses 270 to the client computers 210. Server 220 also runs a connection manager 230 that maintains an open socket connection 250 with the client computer 210 and that uses asynchronous messaging to communicate with the client computer 210. This open socket connection 250 is a lasting, dedicated connection between the client computer 210 and the server 220. The connection manager maintains this open socket connection 250 with the client computer and, by so doing, allows the server 220 to push data to the client computer 210 without having the client computer 210 first send a query 260 as in a traditional web communication. Additionally, the application running on the client computer 210 may transmit packets of data as asynchronous messages via the open socket connection 250 to the connection manager 230 and server 220 and may continue its operations without waiting for a response from the server 220. Further, because of the dedicated nature of the open socket connection 250, the connection manager 230 can push small, piecemeal packets of data to the client computer 210 without having to send the entire production each time the production has been updated.

In many embodiments, there are many client computers 210 and/or many client applications on the same or different client computers (additional client computers not shown) connected to server 220 and connection manager 230 at the same time. In some of these embodiments, more than one client computer 210 connected to server 220 and connection manager access the same production at the same time. In some of these embodiments, any particular connection could also be accessing more than one production simultaneously.

In many embodiments, if it is not possible to create a direct socket connection 250 between the client computer 210 and the connection manager 230, then a virtual connection can be created using the more generic HTTP connections 240. Because there is no direct socket in this case, the client computer 210 must periodically pick up its messages by sending a “check mailbox” query to the server 220. The server 220 then responds by transmitting to the client computer 210 over the TCP connection 240 data pertaining to any updates to the publication, or data pertaining to any other pertinent information.

In some embodiments of the present invention, still referring to FIG. 2, when the client computer 210 wants to start communicating the connection manager 230 creates an open socket connection 250 with the client computer 210. When a user wishes to edit or view a specific production, the client computer 210 software attempts to subscribe to the production by sending a message over the socket 250 which is relayed to the backend servers 280. If the backend server 280 accepts the request to subscribe to the production, the connection manager 230 sends the client computer 210 the current production data. In some embodiments of the present invention, the server 220 first sends a response 270 to the client computer 210 containing data necessary to create the open socket connection 250 with the connection manager. In other embodiments, the client 210 sends the data necessary to create the open socket connection 250 in the query 260. For example, in an Adobe Flash based implementation, a client computer 210 may load a webpage containing HTML code embedding a flash player which knows which production to play. This player will then negotiate its connections to the servers 230 and 220 via the socket connection 250 and/or the TCP/HTTP connection 240 protocols.

When any changes occur to the production (e.g., if that user or another user edits the production), the connection manager 230 may push the update data via the open socket connection 250 to each client computer 210 currently subscribed to the production. Because of the nature of the open socket connection 250, the connection manager 230 need only push the small bit of data pertinent to the update rather than all the data for the entire production. If the subscriber is permitted to edit the production, the client computer 210 may send any data pertinent to changes in the production to the server via asynchronous messaging as the changes occur. Alternatively, the connection manager 230 may wait until the client computer 210 requests an update before pushing data to the client computer 210 over the open socket connection 250. This may be performed by comparing the state of the production on the client computer 210 with the state of the production on the server 220, comparing the most recent changes to the production on the server 220 to those on the client computer 210, or by any other acceptable technique.

Referring to FIG. 3a, in an embodiment of the present invention, the system uses asynchronous messaging to provide real time co-editing of a production amongst multiple users. To accomplish this, the system maintains data pertaining to the current state of the production on the client computers as well as on the server. The data includes a file, whether stored locally on the server or remotely, for any form of media content, including, but not limited to: audio clips, video clips, images, and interactive widgets, such as polls, quizzes, photo books, video transitions, overlays, webcam broadcasts, chat rooms, and the like and is referred to as an asset. The actual data maintained on the client computer may vary, depending on the implementation of the system, from an exact replica of the data on the server to a minimal amount of data that merely indicates where pertinent data is located on the server, and any representation in between. When a single user edits a production, the application running on the client computer accepts 300 and processes user input into data pertinent to changing the current state of a production (e.g., add, move, modify, or delete as asset). The client computer sends 310 an asynchronous message containing the data pertinent to the changes in the production to the server via the dedicated open socket connection (or an equivalent proxy). Without waiting for a response from the server, the client computer changes 320 the state of the local production in accordance with the user input and allows the user to continue to modify the production in this manner. Depending on the embodiment, the steps represented by blocks 310 and 320 may be reversed in order.

Referring to FIG. 3b, when the server receives 330 the asynchronous message from the client computer, the server determines 340 whether the new instructions conflict with the current state of the production as it is maintained on the server. If the new instructions do not conflict the server pushes 350 an asynchronous message back to the client computer indicating that the state of the production was successfully changed. In alternate embodiments, if there is only one subscriber, an asynchronous message is not necessarily pushed back to the client computer. In the case where more than one client computer is subscribed to the production, the server pushes 350 data, in the form of an asynchronous message, containing the changes to the production to all of the other subscribers.

If the new instructions conflict, the server rejects 360 the instructions, does not make any changes to the state of the production, and pushes 370 data to the client indicating that a conflict has occurred. In some embodiments, the server immediately pushes data pertinent to the current state of the production to the client computer. In another embodiment, the server pushes a request for the client computer's current state of the production to the client computer, receives a response, and then only pushes the data necessary to update the current state of the production on the client computer. In yet another embodiment, the server waits for a request from the client computer for an update regarding the current state of the production.

Referring to FIG. 3c, in some embodiments of the present invention, when the client computer receives 380 a data push from the server and determines 382 that submitted changes to the state of the production were rejected, the client computer removes 384 the changes from the state of the production maintained on the client computer. The client computer also determines whether any subsequent changes to the state of the production relied on the rejected changes and removes 388 those changes as well. Any changes that have not been rejected are left as applied. If the data push did not contain a rejection of a previous update made by the client computer, the system applies 390 any update to the production from the server.

The above described use of the connection manager greatly facilitates real-time interaction and co-editing amongst multiple users. Some embodiments of the present invention employ the connection manager such that when multiple users are subscribed to a production, any changes to the production made by one user are subsequently pushed to all of the other subscribing users. In this way, the system does not require individual users to continuously have to request updates to a production. Thus, when one user makes a modification to the current state of the production, once that modification is accepted by the server, it is pushed to all of the other subscribing client computers, which then make the appropriate modification to the current state of the locally maintained production.

In some embodiments of the present invention, production data is stored in an atomic structure. Thus, each production includes a plurality of assets and a plurality of instructions or command elements. Command elements are data dictating the proper manipulation and placement of each asset within a production. For example, in addition to other data, each command element contains an operator indicating the action to be applied to an asset, such as being added, modified, moved, deleted, etc. The command elements may be independently executable, or they may depend on one another. Authoritative command elements are a type of command element that do not depend on other command elements and thus may be processed without reference to another command element. Non-authoritative commands are a type of command element that can only be processed in conjunction with one or more additional command elements.

Referring to FIG. 4, in some embodiments of the present invention, each production includes a block of data 400 in computer memory (not shown) having a series of discrete items of data each representing one editing command element 410. Each editing command element 410 includes a Command Identification Number (“CID”) variable 420, a sequence number 430, a flag variable 440 indicating whether the command is authoritative, and operator data 450 including the actual command execution data as well as any other data required to identify a particular asset and manipulate it within a production.

Still referring to FIG. 4, in some embodiments of the present invention, when a user adds content to a production (e.g., adding a video clip, adding sound, adding a still image, etc.), the client computer software generates a command element 410 with a unique CID 420 and a unique sequence number 430 containing operator data 450 that corresponds to the relevant instruction within the system. The sequence number 430 for each new command element 410 increases sequentially with each additional command element. In some embodiments, the CID 420 for each command element 410 increases sequentially with each newly added asset and corresponds to the particular asset that has been added to the production. In some embodiments, each command element has a CID that corresponds to a particular asset, and the command element shares that CID with all other command elements that manipulate (e.g., move, delete, etc.) that particular asset. Thus, the sequence number 430 indicates the order in which each of the command elements 410 was added to the production and the CID 420 refers to the specific asset the operator data 450 of the command element 410 manipulates.

For example, still referring to FIG. 4, if a user adds n assets to the production, n command elements 411, 412, 413, . . . , 41n will be added to the data representing the production with the CID 420 and the sequence number 430 ranging sequentially from 1 to n and each having operator data 450 indicating, for example, which asset is to be added as well as an operator indicating that the asset should be added to the production and any other data pertinent to the operation. When, however, the user further manipulates an asset that is already part of the production (e.g., deletes the asset from the production or moves the asset to another time index or location within the production), such as the asset represented by the command element 412 having a CID with a value of “2,” the command element 415 containing the operator data 450 to manipulate the asset that was added to the production with command element 412 will contain the next sequential sequence number 430 (n+1), but will have the same CID 420 as the original command element 412 (i.e., “2”).

In some embodiments, the system employs shadowing. Shadowing includes determining whether an asset or a command element becomes unnecessary because of a more recent command element. In some embodiments, the above described system will shadow assets or command elements when the value stored in the CID of one command element is the same as the value stored in a second command element and the operator data in the more recent of the two command elements (i.e., the one with the greater sequence number) indicates an operation that would make the previous command element unnecessary. For example, referring again to FIG. 4, if the user adds two assets to a production, two command elements 411 and 412 will be created having sequential sequence numbers 430 and sequential CIDs 420. Assuming the user continues to add assets to the production until n command elements 413, . . . , 41n exist, if the user then chooses to delete the second asset added, a new command element 415 will be created containing a CID 420 with the value of “2” (to correspond with the CID 420 of the command element 412 that adds the asset to the production 400) and a sequence number 430 with the value of n+1. The new command element 415 will further contain operator data 450 indicating that the asset should be removed from the production. Because this command element 415 indicates that the asset is to be deleted, all other command elements (e.g., command element 412) that manipulate this particular asset (e.g., adding the asset to the production, moving the location of the asset, etc.) become unnecessary and thus are termed shadow assets. To accomplish this shadowing, the system builds and maintains a list of all relevant command elements by only populating the list with command elements that are still relevant in relation to command elements with a greater sequence number 430 and the same CID 420. Thus, in the previous example, when the system populates the list of relevant command elements and when it encounters the first command element 412 with a CID 420 with a value of “2,” it will examine all subsequent command elements 415 with a CID 420 with a value of “2.” When the system determines that the latest command element 415 sharing this CID 420 contains an operator 450 indicating deletion, the system will not populate the list with any of the command elements 412 and 415 with a CID 420 of “2.” Similarly, if the asset is being moved to a different place in the timeline of the production, the system can ignore all previous move command elements and thus only move the asset once when building the production.

In other embodiments of the invention, the CID of each command element is not strictly associated with a particular asset. Instead, the CID is unique to a command element unless a new command element affects a previous command element, in which case, the CID for each of these command elements would have the same value. For example, referring to FIG. 4, if a user adds two assets to a production in the form of video clips, two command elements 411 and 412 with CIDs 420 and sequence numbers 430 with values of “1” and “2” will be generated, each containing operator data 450 for adding each of the assets. If, for example, the user then adds an image overlay to the second asset or otherwise manipulates or edits the image in the second asset, a third command element 413 with a CID 420 having a value of “3” and a sequence number 430 having a value of “3” would be generated. While both command elements affect the same asset, one does not shadow the other, as described above, because it does not make the previous command element 412 irrelevant. If the user subsequently deletes the second asset from the production, then command element 415 is generated with a CID 420 having a value of “2;” a sequence number 430 having a value of “n+1;” and with operator data 450 containing an operator indicating the asset should be deleted from the production because this new command element 415 makes the command element 412 adding the asset unnecessary and irrelevant. Similarly, because command element 413 depends on command element 412, it would also become shadowed. This may be accomplished by checking for dependent command elements when one element is shadowed and then adding command elements to delete the dependent command elements and making this last set of command elements dependent on the original deleting command element. This may also be accomplished by only populating the relevant command elements list with command elements that are not shadowed and that do not depend on command elements that are shadowed.

In another embodiment, the above described shadowing of assets allows the system to create a clean version of the production such that the list of relevant elements replaces the master list of relevant elements. The clean version of the production thus only contains the non-shadowed assets in the production. The system may prepare the clean version at the user's command. Alternatively, the system may be configured to create clean versions of productions at fixed time intervals or after a period where a specific production has not been modified for some fixed time period. Further, a clean version of a production may be generated at a time when a user has completed editing a production and makes it available to the public. Generating a clean version decreases storage requirements by decreasing the overall size of the production. The clean version also makes for a faster download for client computers subscribing to the production due to its decreased size.

In another embodiment of the above described system, referring again to FIG. 4, each command element 410 also contains flag data 440 indicating whether the command element 410 is authoritative. Each non-authoritative command must depend on one authoritative command element; however, an authoritative command can have multiple dependant command elements. The authoritative commands may thus act as undo and redo points within the production. For example, if the user provides input to the system indicating that he or she wishes to undo the last command, the system will remove the command as well as all non-authoritative commands that depend on that authoritative command. Similarly, if the user provides input to the system that indicates he or she wishes to redo a command, the system will redo the authoritative command only and will allow the user to selectively redo each subsequent non-authoritative command following the last authoritative command. In this manner, the authoritative command flag served to indicate legal undo and redo points.

In some embodiments, the system further includes a variable containing the maximum sequence number. This allows the system to keep track of which command element is the most recent while at the same time allowing it to not delete the command elements that the user has chosen to undo. In some embodiments, the above described system further includes a maximum sequence number variable containing the value of the sequence number of the last command with which the system should populate any clean version of the production. Thus, if a user has added a variety of command elements and then later chooses to undo them, rather than permanently remove these command elements from memory, the system instead merely decreases the maximum sequence number such that the system will not process those command elements. Thus, the user can easily “redo” or reapply those command elements. Further, this variable allows for a variety of system optimization techniques.

In some embodiments, each production has a current sequence number and a maximum sequence number. The current sequence number represents the current undo/redo state. When the system receives input indicating that the user wishes to undo a command, the current sequence number decreases, or winds backwards. When the system receives input indicating that the user wishes to redo a command, the system increases the current sequence number. The maximum sequence number represents the total number of commands, including those that the user has chosen to undo but has not chosen to redo. Using the maximum sequence number, the system can track whether there currently exist any legal redo commands. If the system has received input indicating that one or more commands are to be undone and the system has decreased the current sequence number accordingly, when the system receives input from the user indicating a change to the production, the system truncates the maximum sequence number to have a value equal to the current sequence number after the change because the old undone commands would no longer be compatible with the new changes the production.

In some embodiments, there are two versions, sometimes referred to here as contexts, of each production, a published version, sometimes referred to here as an audience version or context, and a private version, sometimes referred to here as an author version or context. Each context has a maximum and a current sequence number, as discussed above. In these embodiments, whenever the system receives input from the user indicating a change to the production the system only applies the changes the author context. When the system receives input from the user indicating that he or she wishes to publish the production by making it viewable by the public, the system will copy any information in the author context to the audience context, which is viewable by the public.

In some embodiments, the system makes use of more than just two contexts and may thus include, for example, semi-private contexts. For example, a semi-private context might only be viewable by a limited audience. Further, such a context might be password protected or require some other appropriate form of user authentication.

In some embodiments, the above described system further includes a conflict resolution system whereby multiple users can simultaneously co-edit the same production. In this embodiment, the user adds, deletes, and otherwise manipulates assets in the production resulting in a series of command elements. The client computer transmits each new command element to the server. In order to provide real time editing, the client computer does not wait for a server response and instead assumes as true and accepted by the server each of the user's edits unless the server indicates otherwise. The system thus allows the user to continue to make additional edits to the production without any lag time. When the server receives a command element from the client computer, the server compares the sequence number of the command element with the last sequence number of the server's version of the production. If the new command element's sequence number indicates that there have been changes made to the server version since the client executed the command, the system checks for a conflict. One indication that changes have been made to the server version since the client executed the command is when the new command element's sequence number is less than or equal to the last sequence number in the server version. One method by which the server may check for conflicts is by checking the previous command elements having CIDs having the same value to see if they conflict with one another. For example, if a previous command element sent by the first user deletes the asset that was added to the production by the command element having a CID having a value of “2” and if the second user then sends a command element containing instructions to move the asset that was added to the production by the command element having a CID having a value of “2” to a different point in the time line, the server will reject this later command element and will then push data to the client version of the program running on the second user's computer. The pushed data would indicate that the proposed command element was rejected and would also contain the command elements added to the production by the other user. Alternatively, more sophisticated conflict checking methods are implemented. For example, the conflict resolution aspect of the invention may instead process all of the command elements received since the second user made an edit and check each one regardless of the CID and then perform the data push. This data push not only indicates that the proposed command element was rejected, but it also contains the command elements added by other users that have been accepted by the server. Once the client computer receives this information, it will determine whether any of the next command elements it submitted relied on this rejected command element and will remove them if necessary.

In another embodiment of the invention, the operator data portion of a command element may include a pointer to another command element or to another array of command elements. In this manner, the invention allows for arbitrary nesting of command elements and of command element arrays. Thus, a command element in a production may refer to another set of command elements, or it may refer to another external production. In this manner, productions may become nested within one another. Further, the use of nested command elements allows the production to incorporate assets that make further use of the atomic command element structure.

FIG. 5 shows one possible representation of the graphical user interface 500 as seen on the client computer. The interface includes a timeline 510, assets 520 on the timeline in the order they will be processed in the production, and a playback window 530. While a production is being edited, any user subscribed to the production may play the current state of the production and view it in the playback window 530. If any user adds a new asset 540 to the publication, all of the client computers subscribed to the production will immediately display the asset 540 being added. Any client computer currently playing the production will continue to play the production and will include the new asset in the playback when it reaches it in the timeline.

Claims

1. A web-based video editing system comprising: a server connected to a network;a client communications device connected to the network, wherein the client communications device is in communication with the server;media storage connected to the server, the media storage including media files that are used in a video production;production storage connected to the server, the production storage including a set of atomic production commands and references to media files related to the atomic production commands, the commands and references defining a video production;wherein the server is configured to send the set of atomic production commands and references to media files to the client communications device;receive a new production command from the client communications device;determine if the new production command is valid for the set of atomic production commands and references to media files;add the new production command to the set of atomic production commands and references to media files and send a production command accepted communication to the client communications device when the new production command is determined to be valid, andsend a production command rejected communication to the client communications device when the new production command is determined not to be valid.

2. The system of claim 1 wherein the server is further configured to: send the set of atomic production commands and references to media files to a plurality of client communications devices;receive a new production command from any of the plurality of client communications devices, wherein the server is configured to send an update to the set of atomic production commands and references to media files to one or more of the plurality of client communications devices; andreceive a plurality of new production commands from more than one of the plurality of client communications devices before the server sends an update to the set of atomic production commands and references to media files for the first of the plurality of new production commands from more than one of the plurality of client communications devices to one or more of the plurality of client communications devices.

3. The system of claim 1 wherein the client communications device is in communication with the server using asynchronous messaging.

4. The system of claim 1 wherein the server includes the media storage.

5. The system of claim 1 wherein the server includes the production storage.

6. The system of claim 1 wherein the server includes the media storage and the production storage

7. The system of claim 1 wherein the server is further configured to send software to the client communications device that displays the video production according to the set of atomic production commands and references to media files and provides a graphical user interface for making modifications to the video production according to the set of atomic production commands and references to media files and transmits a production command to the server in response to a user's interaction with the graphical user interface.

8. The system of claim 1 wherein the client communications device comprises a graphical user interface.

9. The system of claim 8 wherein the client communications device comprises at least one selected from the group consisting of: a personal computer, a laptop computer, a handheld computer, a phone, and a video player.

10. The system of claim 1 wherein the network is the Internet.

11. A web-based video editing method using a server connected to a network, media storage connected to the server, the media storage including media files that are used in a video production, production storage connected to the server, the production storage including a set of atomic production commands and references to media files related to the atomic production commands, the commands and references defining a video production, and a client communications device connected to the network wherein the client communications device is in communication with the server, the method comprising: the server sending the set of atomic production commands and references to media files to the client communications device;the server receiving a new production command from the client communications device;determining if the new production command is valid for the set of atomic production commands and references to media files;adding the new production command to the set of atomic production commands and references to media files and sending a production command accepted communication to the client communications device when the new production command is determined to be valid, andsending a production command rejected communication to the client communications device when the new production command is determined not to be valid.

12. The method of claim 11 further comprising: the server sending the set of atomic production commands and references to media files to a plurality of client communications devices;the server receiving a new production command from any of the plurality of client communications devices and sending an update to the set of atomic production commands and references to media files to one or more of the plurality of client communications devices; andthe server receiving a plurality of new production commands from more than one of the plurality of client communications devices before the sending an update to the set of atomic production commands and references to media files for the first of the plurality of new production commands from more than one of the plurality of client communications devices to one or more of the plurality of client communications devices.

13. The method of claim 11 further comprising the client communications device communicating with the server using asynchronous messaging.

14. The method of claim 11 wherein the server includes the media storage.

15. The method of claim 11 wherein the server includes the production storage.

16. The method of claim 11 wherein the server includes the media storage and the production storage

17. The method of claim 11 further comprising the server sending software to the client communications device that displays the video production according to the set of atomic production commands and references to media files and provides a graphical user interface for making modifications to the video production according to the set of atomic production commands and references to media files and transmits a production command to the server in response to a user's interaction with the graphical user interface.

18. The method of claim 11 wherein the client communications device comprises a graphical user interface.

19. The method of claim 18 wherein the client communications device comprises at least one selected from the group consisting of: a personal computer, a laptop computer, a handheld computer, a phone, and a video player.

20. The method of claim 11 wherein the network is the Internet.