The present invention relates to a method and platform for creating and managing video production workflows from conception to completion.
Multi-camera video production is equipment and labor intensive. Preparing for a broadcast requires extensive setup time and labor. Truckloads of equipment are often filled with dedicated audio and video (A/V) specific hardware along with a team of highly trained personnel who are required to convene on-location in order to create a video broadcast. Most A/V equipment by its nature is designed to be operated hands-on on-site at a live event site or in a dedicated production studio.
Economic and efficiency considerations, including project budgets and skilled labor shortages, are driving a need to facilitate remote collaboration among video crew members in addition to remote operation distant from the actual shoot location. While limited remote control capabilities may exist for some traditional A/V equipment, most implementations are highly constrained in functionality and require numerous intermediary devices to adapt to a remote use case for which the equipment was never intended. These factors make remote production with an offsite crew complex, costly, and difficult to accomplish.
Video editing involves a complex series of steps and processes that starts with the identification of the digital asset, i.e., the digital file that contains the content to be incorporated into the video, and ends with a final production version. This process includes the movement of digital assets from one device to another to finally reach the person that will edit the video file. The assignment to an editor, and subsequently to a project, and finally tracking their progress and providing final acceptance has traditionally required one or more people to create ad-hoc lists of statuses and assignments which tend to be error-prone and inadequate for providing status updates in a timely manner.
Pre-production, production, and editing/post-production are all touched by growing number of digital video assets using a limited number of people to manage the full process. This increased growth in the creation of digital content has created a backlog of content that is difficult to manage and assign to specific projects and personnel—pre-production, production, and editing/post-production are all touched by growing number of digital video assets.
The industry has recently begun to adopt Information Technology (IT) to transport video, audio and control signals over Internet Protocol (IP)-based infrastructure. In addition to being beneficial to remote production use cases, IT based workflows have tremendous potential cost and efficiency benefits. However, widespread adoption is hampered by the inherent limitations of legacy A/V equipment and production methods as well as the technical knowledge required to conform to IP standards. Video technicians are accustomed to plugging A/V cables into dedicated A/V devices to accomplish dedicated A/V tasks. Transitioning to plugging IT devices into IP networks comes with knowledge gaps and hurdles for these personnel.
Existing applications suffer many drawbacks that may hinder the user's ability to have timely and detailed awareness of the status of projects and files. These applications are typically only able to apply media management functions, resulting in awkward movement of digital assets from one system to the next in order to complete the video production workflow process. This results in the need for multiple systems to support a user's need to take content from a recording source and, through a series of file copying between separate systems, end up with a final production version of their digital assets. This weakness exists in current video production workflows, allowing for discontinuity between file names, projects, and associated data. What is required is a system that keeps the relationships intact and applies changes in a cascading manner. This is similar to the data quality concept in modern database management systems called “referential integrity,” where changes to data in one location are automatically reflected in other related records.
All of these disadvantages work together to increase the difficulty of creating and tracking assets, jobs, and statuses. Accordingly, a need exists within the field for novel orchestration layer management capabilities for streamlining the process of video production, for ease of use and to facilitate collaboration. An ecosystem that takes full advantage of IT workflows while being easily navigable by legacy A/V technicians is desperately needed.
The inventive cloud-based video production platform transforms the traditional video production model by converting video, audio, and control signals to IP protocols directly on a camera platform. One example of an appropriate camera platform is described in International Publication No. WO 2021/195641, referred to as the “AIRstation™”, the disclosure of which is incorporated herein by reference. Within the inventive video production platform, which may be referred to as the “AIRcloud™ system”, the IP signals are connected via wired and/or wireless connections to virtual computing resources. This approach allows the entire video production ecosystem to be virtualized. Ubiquitous IP infrastructure can be used to transport the signals, allowing dedicated A/V hardware to be converted into on-demand virtual compute resources. This approach drastically reduces the on-location equipment footprint, the personnel required to set it up, and creates the opportunity for a distributed workforce. In this system, the equipment can be operated locally, remotely, or a combination thereof, by way of a robust internet connection.
In order to effectively deploy a video over IP ecosystem, a comprehensive easy-to-use management system is required. The inventive AIRcloud™ system is an integrated tool for IP-based video production. It allows for operation of A/V and IT resources remotely, defining user roles, enabling permissions, provisioning cameras, compute resources, and storage, managing media, as well as managing accounts and payments. It is an ecosystem that encompasses management and use of the entire pre-production, production, and post-production A/V workflow in a cloud based environment.
The inventive system employs software components that include multiple individual smaller software modules, relational databases and network-based connectivity. User account management, user access permissions, user groups and job assignments are defined in the system and used to create and edit video production workflows. One of the key improvements provided by the inventive platform is that digital asset information, i.e., information about the contents of a digital file and how it was created, which may include file metadata, camera and lens metadata, IMU and encoder metadata, as well as user metadata is automatically captured as it is generated. The modularity of this system provides for automatically directing the uploaded content to externally or internally connected services, such as transcription systems, or computer vision analysis for object identification, which can automatically perform functions on the digital asset prior to, or after, an editor makes changes to the digital asset. Digital assets are assigned to one or more users through the creation of project jobs. During the post-production process, when one or more digital assets are identified as requiring an editor to make specific changes to the asset, the digital asset is assigned to a user-defined project job by a manager. This project job is further assigned to an editor. The editor may change the status of the project job when they begin the editing process and/or when their edits are complete. This will submit the project job for review. A manager may then review the digital asset's edits and take other actions such as accepting the work, changing the status, and downloading the asset. If the work has not been accepted as complete, the manager may provide new instructions or comments, change the status and they may be notified when the editor has completed the revisions. This iterative process can continue across revisions until the digital asset has been accepted.
One of the important innovations of the AIRcloud™ system is that it enables automation of traditionally manual tasks. Traditional broadcast workflows demand most functions be executed manually by human operators. In contrast, the AIRcloud™ system allows for many of these functions to be automated across the pre-production, production, and post-production workflow.
In one aspect of the inventive system, a platform for managing video production workflows includes: an interface in communication with a network, the interface configured for entry of and transmission of instructions to and receiving information from a plurality of modules, each module in communication with the network, the modules comprising: an admin module configured to distribute instructions to other modules and receive status information from other modules; an offsite camera control module; a remote editing module; a remote production module; a client review and collaboration module; and a distribution/broadcast module; and at least one camera in communication with the network; wherein the admin module is used by a project manager to assign jobs, access and distribute digital assets, and to communicate with platform users. The platform users may include one or more administrator, producer, technical director, camera operator, audio technician, graphics operator, and client. In some embodiments, instructions and information may be configured to trigger one or more automated action within one or more of the plurality of modules and the at least one camera. The plurality of modules may further include a machine learning module configured for artificial intelligence (AI)-assisted asset identification and categorization. The camera may be configured for control via a remotely-located camera operator.
In another aspect of the invention, a platform for managing video production workflows includes: an interface in communication with a network, the interface configured for entry of and transmission of instructions to and receiving information from a plurality of modules, each module in communication with the network, the modules comprising: an admin module configured to provide instructions to other modules and receive status information from the other modules; an offsite camera control module; a remote editing module; and at least one digital file source in communication with the network; wherein the admin module comprises a user interface (UI) that is used by a project manager to assign jobs, access and distribute digit assets, and to communicate with platform users, and wherein the interface is further configured to deliver a completed video production to a content distribution network (CDN) in communication with the network. The network may include a combination of wired and wireless connections.
The platform users may include one or more of an administrator, the project manager, a producer, a technical director, a camera operator, an audio operator, on-screen or voice-over talent, and a client. Instructions and information are configured to trigger one or more automated action within one or more of the plurality of modules and the at least one digital file source. The at least one digital file source generates one or more digital file comprising one or a combination of an image, a series of images, an audio, metadata describing the image, the series of images, or the audio. The metadata may include one or a combination of identifiers consisting of name, date, time, format, client, project, subject matter, location, settings under which the one or more digital file was created, editing history, editing permissions, and copying permissions. In some embodiments, the wherein the metadata may include one or more of camera settings, camera lens settings, camera robotic controller settings, camera sensor readings, and audio levels.
In some embodiments, the plurality of modules includes a machine learning module configured for artificial intelligence (AI)-assisted asset identification and categorization. The machine learning module may be configured to extract information from the one or more digital file and execute an analysis comprising one or more of object detection, keyword tagging, face detection, video segment identification, computer vision analysis of human performance, body mechanics, emotion perception, and audio analysis. In other embodiments, the machine learning module may be further configured to extract audio information from the one or more digital file and execute an analysis of a speech component of the audio information for automatic speech to text transcription or to generate a keyword or a command for further action.
The plurality of modules further comprises a remote production module configured for managing broadcast of the video production to the CDN. A client review and collaboration module configured to provide edited content to a client to review and accept or reject the edited content prior to delivering the completed video production.
In some embodiments, the at least one digital file source is a camera configured for control via a remotely-located camera operator, wherein the remotely-located camera operator is in communication with the interface. In other embodiments, the at least one digital file source is an audio recorder, which may be incorporated into a camera. The at least one digital file source may be a plurality of digital file sources, wherein each digital file source is associated with a unique code for establishing a link to a predetermined site plan. The unique code may be configured for scanning by a platform user's mobile device, wherein scanning the unique code causes the predetermined site plan to be displayed on the mobile device, wherein an assigned position of the digital file source associated with the unique code is displayed within the predetermined site plan.
In some embodiments, the admin module may be further configured for the project manager to assign access limits for controlling access to elements within the platform and the digital assets by each user as required for the platform user's assigned job. The admin module may be further configured to collect status information for platform components and generate a master status dashboard (MSD) to display status and settings for one or more of the at least one digital file source, communications, and performance data in real time. In some implementations, the status and settings displayed on the MSD are one or more of camera settings, CPU status, connection conditions, stream health, available storage, recording status indicator, and uptime.
In some embodiments, the remote editing module may be configured to manage a workflow for jobs comprising editing, reviewing and finalizing the project, wherein the remote editing module is further configured to deliver a digital asset to a designated platform user when a job is assigned to the designated platform user.
In still another aspect of the invention, a method for managing video production workflows over a network, the method includes providing an interface configured for entry of and transmission of instructions to and receiving information from a plurality of modules, each module in communication with the network, the modules comprising: an admin module configured to provide instructions to other modules and receive status information from the other modules; an offsite camera control module; a remote editing module; and at least one digital file source in communication with the network; wherein the admin module comprises a user interface (UI) that is used by a project manager to assign jobs, access and distribute digit assets, and to communicate with platform users; and delivering a completed video production to a content distribution network (CDN) in communication with the network. The network may include a combination of wired and wireless connections.
The platform users may include one or more of an administrator, the project manager, a producer, a technical director, a camera operator, an audio operator, and a client. Instructions and information are configured to trigger one or more automated action within one or more of the plurality of modules and the at least one digital file source. The at least one digital file source generates one or more digital file comprising one or a combination of an image, a series of images, an audio, metadata describing the image, the series of images, or the audio. The metadata may include one or a combination of identifiers consisting of name, date, time, format, client, project, subject matter, location, settings under which the one or more digital file was created, editing history, editing permissions, and copying permissions. In some embodiments, the metadata may include one or more of camera settings, camera lens settings, camera robotic controller settings, camera sensor readings, and audio levels.
In some embodiments, the plurality of modules includes a machine learning module configured for artificial intelligence (AI)-assisted asset identification and categorization. The machine learning module may be configured to extract information from the one or more digital file and execute an analysis comprising one or more of object detection, keyword tagging, face detection, video segment identification, computer vision analysis of human performance, body mechanics, emotion perception, and audio analysis. In other embodiments, the machine learning module may be further configured to extract audio information from the one or more digital file and execute an analysis of a speech component of the audio information for one or more of automatic speech to text transcription and generating a keyword or a command for further action.
The plurality of modules further comprises a remote production module configured for managing broadcast of the video production to the CDN. A client review and collaboration module configured to provide edited content to a client to review and accept or reject the edited content prior to delivering the completed video production.
In some embodiments, the at least one digital file source is a camera configured for control via a remotely-located camera operator, wherein the remotely-located camera operator is in communication with the interface. In other embodiments, the at least one digital file source is an audio recorder, which may be incorporated into a camera. The at least one digital file source may be a plurality of digital file sources, wherein each digital file source is associated with a unique code for establishing a link to a predetermined site plan. The unique code may be configured for scanning by a platform user's mobile device, wherein scanning the unique code causes the predetermined site plan to be displayed on the mobile device, wherein an assigned position of the digital file source associated with the unique code is displayed within the predetermined site plan.
In some embodiments, the admin module may be further configured for the project manager to assign access limits for controlling access to elements within the platform and the digital assets by each user as required for the platform user's assigned job. The admin module may be further configured to collect status information for platform components and generate a master status dashboard (MSD) to display status and settings for one or more of the at least one digital file source, communications, and performance data in real time. In some implementations, the status and settings displayed on the MSD are one or more of camera settings, CPU status, connection conditions, stream health, available storage, recording status indicator, and uptime.
In some embodiments, the remote editing module may be configured to manage a workflow for jobs comprising editing, reviewing and finalizing the project, wherein the remote editing module is further configured to deliver a digital asset to a designated platform user when a job is assigned to the designated platform user.
The following detailed description provides examples of embodiments of the inventive system with reference to the accompanying drawings.
As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well as the singular forms, unless the context clearly indicates otherwise. As a clarifying example, when an object is recited, unless that object is expressly described as a single object, “one or more object”, “at least one object”, or “a plurality of objects” also falls within the meaning of that term.
Definitions: Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one having ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. For further clarity, but not limitation, additional definitions are provided for the following terms:
“User” means a person operating in one of more roles on a project. Users may include Project Managers, Administrators, Producers, Technical Directors, camera operators, audio operators and support personnel who may be authorized to interact with the platform.
“Role” means a job assignment for a project, which, in addition to the Users described above may include, for example, Graphic Operator, Replay Operator, Client, Field Crew, Editor.
“Project” means an event or series of events involving video and/or audio capture and production. “Project” may also include post-production and editing.
“Project manager” means one or more person responsible for planning and execution of a Project, including a Administrators, Producers, Technical Directors.
“Instance” means a computer, machine, or piece of equipment. For example, an “instance” may be a virtual machine in the cloud, a machine at a remote location separated from the shoot site, or a machine on location with the cameras.
“Asset” means any person, camera, gear, computer instance, or digital file associated with a project or user.
“Digital file source” means a device that is the source of an image, video and/or audio file (collectively, “digital file”), and data related thereto, including metadata that describes elements and features of the digital file and how it was created For purposes of this description, “digital file source” includes not only a conventional camera for direct image and/or sound detection/collection, e.g., raw video, but also includes a storage device or other source of a digital file that may have been generated separately in time or as part of independent effort using an independent camera and stored in the storage device, or may be a computer-generated digital file in which a conventional camera was used to generate none of, or only a portion of, the digital file. For example, a pre-existing animation may be digital file.
“Permission” means authorization/access to a protected asset.
“Online” means an asset is available for immediate use.
“Offline” means an asset not available for immediate use.
“Remote” in the context of production and editing means work conducted distant from the event location and/or away from the traditional workplace.
“Broadcast” refers to transmission of AV signals to one or more recipients simultaneously, by wired or wireless means, and includes but is not limited to, over the air and live stream transmission.
In describing the invention, it will be understood that a number of techniques and steps are disclosed. Each of these has individual benefit and each can also be used in conjunction with one or more, or in some cases all, of the other disclosed techniques. Accordingly, for the sake of clarity, this description will refrain from repeating every possible combination of the individual steps in an unnecessary fashion. Nevertheless, the specification and claims should be read with the understanding that such combinations are anticipated and, therefore, within the scope of the invention and the claims.
Access to the platform portal begins with a login username and password landing page. The login process may be secured using one or more known authentication methods. Examples of authentication methods include: Single-Sign-On (SSO), Security Assertion Markup Language (SAML), Oauth, and OpenID. Other methods are known to those of skill in the art. This access may be achieved via with the admin controller 105 or any other communication device that is assigned to or associated with a user. In some projects, users may be pre-arranged and already have credentials assigned for logging on to the system. For users who may not yet have an account, a “new user signup” button links to the account creation screen where they enter various information including but not limited to; name, company, email address, nickname, phone number, hometown, etc. The user may upload an image for use as their avatar/icon. If no image is provided, two-character initials can be substituted, or the user may be assigned a user number. Final step is to provide a login password. After approved login, the user will be directed to the platform dashboard, which is diagrammatically shown in
The dashboard 202 is the starting point and central hub for users to navigate the various features of the platform portal, which may be referred to as the “AIRcloud™ Portal”. Once a user is assigned a role on a project, one or more associated project icon 204 will appear on their dashboard. The user will then have access to project details, communications with other team members, as well as any features that are accessible via permissions and roles. Permissions are automatically generated based on the user's defined role in the project and can also be customized by an Administrator or Producer.
Project functions may include, but are not limited to, selecting a multiview interface, viewing the program feed, camera control, virtual workstation control, shoot setup configuration, audio/video routing, media asset management, post-production setup and configuration, communications setup and configuration, project detail summary.
In pre-production, to create a new project, an Administrator or Producer selects “new project” by clicking on “New Project” 206 on the dashboard. They can then upload an icon image, if appropriate, and enter various information about the project including but not limited to, project name, client, project ID, shoot date(s), location, an example of which is provided as project box 210. All projects to which a particular user is connected will appear on the dashboard. Selection of a project icon 204 on the dashboard 202 opens up navigation menu 220 for the project. Each of buttons 221-230 within menu 220 takes the user to a different module or function within the overall system, each of which will be described below in more detail.
The “Shoot Setup” button 226 of menu 220 launches the shoot setup screen
The Shoot Setup feature is a core component of the AIRcloud™ system's provisioning and permissioning automation. This is where cameras, computer instances, connected equipment and personnel are securely provisioned and assigned for the project.
Using the Shoot Setup Screen 300 shown in
Referring to
A powerful feature of the inventive AIR system is the automation that is triggered as a result of assets being provisioned and assigned. A single manual act can trigger multiple automatic actions. In one example, referring to
The result of the automatic routing process for multiple assets can be viewed by selecting the “Routing” button 221 in navigation menu 220. This displays a live video/audio routing screen, an example of which shown in
At this point in the workflow, when a Camera Operator (User) is assigned to Camera #1, permission for control of that camera is automatically configured. When the Camera Operator logs into the system and selects the camera for control, the IP address and appropriate permissions and settings for the camera platform auto-populates within the AIR mobile application.
For other users, their permissions will be similarly auto-populated. For example, a Technical Director controlling a Cloud Switching Instance is automatically assigned permissions for remote control of that instance when they are assigned the Technical Director role for a project. They are additionally automatically assigned permission to route any of the audio/video signals associated with the project.
This automation removes multiple manual steps in the setup process, saving time and reducing the chance of human error. Communications and notifications: If desired, notifications can be automatically sent to appropriate crew members when modifications are made to any aspect of the Project. Similarly, with the communications system, permissions for users are automatically provisioned and users are routed to their appropriate groups for real time communications once they are assigned a role on the project.
“Multiview” refers to multiple camera video feeds being displayed in one predefined space or screen. Multiview is common in a broadcast production truck or broadcast stage control room. Traditionally, each video feed is sent to an individual monitor rather than to an aggregated display of multiple video feeds on a single screen. Hardware, such as Blackmagic Design's video switchers, provide an aggregated view over a single output as a standard feature. However, the multiview feed is often unavailable to technicians outside a control room space.
The ability to view feeds from multiple cameras simultaneously in real time is critical to the production process. Producers and Directors require this capability in order to select the correct shot(s) for the program feed, set proper camera angles, prepare replays, view graphics, and other purposes. Other project technicians can benefit from viewing these real-time feeds, but this is rarely available due the expense and complexity of sending multiple feeds to multiple destinations. Camera operators are sometimes given a “program return” video feed showing the master program output, typically their own camera with graphics superimposed which they use to frame the shot appropriately to accommodate the graphic positioning. Camera operators are often unable to see what other cameras in a multi-camera setup are capturing in real time. The availability of such a feature would assist in adjusting framing, seeing events outside of direct camera view and anticipating framing needs. This is particularly applicable to camera operators who are not positioned directly behind the camera or off-site.
Using the inventive approach, entering the production phase, the user can select the “Multiview” button 230 in navigation menu 220 (
The “Program Feed” button 229 on the navigation menu 220 (
Establishing remote control of camera platforms can be complex. An uninterrupted control connection must be maintained, and latency must be kept to an absolute minimum. Disruption of control signals can have catastrophic results for video quality. Serial control connections are limited in their practical range. Conversely, IP-based protocols are not distance constrained but face challenges in traversing NAT (Network Address Translation) implementations and firewalls. Network admin privileges at the host location are usually required for port forwarding. Most organizations will not permit port forwarding since it opens their networks to security risks. These challenges make it difficult to establish camera platform control connections.
The inventive system's camera control function bypasses these issues with a preconfigured easy-to-use access link for a remote camera controller. Selection of the “Camera Launchpad” button 228 on the navigation menu 220 (
Using the Camera Launchpad function, these complicated NAT transversal techniques are transparent to the user. Users follow three simple steps to gain access to a remote camera platform.
A camera operator logs in to the portal as described above to select the project to which they have been assigned, and then selects the camera for use on the Launchpad screen.
Referring to
The Launchpad interface shows which cameras are authorized for user control by a combination of visual distinctions to indicators for online/offline status for the camera platform. Visual distinctions can be based on color coding alone or in combination with opacity, brightness, or patterns such as dashed lines. For example, units that are available and authorized for user control may appear as an icon with 100% opacity, while units not authorized for this particular user may be displayed with reduced, e.g., 50%, opacity. A green dot may be used to indicate the system is authorized and connected to the main controller. A yellow dot may indicate that the asset (camera) is authorized but not connected. A gray dot may indicate the camera is already under control by another user. A red dot may mean that the system is not authorized for external control, and a blue dot may be used to indicate that the system is connected but not assigned. A user can submit a request for permission to control a camera for which they are not yet authorized and that permission request can be automatically forwarded to an administrator for approval. This feature eliminates the need for manual entry of IP addresses and access codes, thus reducing the chance for human error and permitting fast unfettered access to the camera control hardware.
Clicking or double-clicking on a unit launches the application for operating that camera. In some embodiments, the application is described in International Publication No. WO 2021/195641, which is used in conjunction with the AIRstation™ robotic camera system. To control a camera platform (AIRstation™ or other), the user selects an authorized camera icon for control. The mobile device's browser then launches the AIR camera control mobile application via direct communication between the web browser and the mobile operating system. This may be accomplished through an advanced process known as “deep linking” where an “intent URI” (uniform resource identifier) is used to securely launch the mobile app. URIs are known by those of skill in the art for identifying logical or physical resources used by web technologies. Credentials for control will have been automatically provisioned and access has been pre-approved.
In the example shown in
For the system to function effectively, camera platforms must be placed according to plan. To assist the on-site crew in placing the camera platforms in the correct positions, the system may use QR codes, bar codes, BLE (Bluetooth low-energy), Radio Frequency ID (RFID) tags, or other identification scheme that may be physically or electronically associated with the gear to provide links to display the predetermined site plan. For example, for system implementations utilizing AIRstation™-equipped camera robots, a QR code displayed on the back of the unit, when scanned, will show the technician exactly where the camera platform should be placed on the site. To illustrate, in
Video production often requires video and audio signals to be routed to more than one destination. Routing of audio and video signals is traditionally handled by dedicated A/V specific hardware. See, e.g., U.S. Pat. No. 9,191,721, of Holladay, et al., the disclosure of which is incorporated herein by reference. Companies such as Haivision (Montreal, QC, Canada) have improved upon this idea with products like the Haivision Hub™, a hardware/software product which allows data streams in their SRT (Secure Reliable Transport) format to be routed to various destinations via a simple interface. The drawback to the Haivision Hub™ is that it is designed only to ingest streams transported using the SRT protocol.
Selection of the “Routing” button 221 in the navigation menu 220 takes the user to an AIRhub™ screen, a matrix router for live video and audio assets, an example of which is shown in
To modify or add additional sources and destinations, the user adds an end point (451) from the available destinations (450), positions that end point in the routing area (452) and then drags a connector (453) from the source to that destination. Additional destinations end points can be configured and added to the available list by selecting the “add destination” feature (454). For switcher outputs, the video and audio are similarly routed by drag and connect. This feature can be used to route camera feeds into switcher instances, replay instances, graphics programs, distribute program outputs, and/or raw camera feeds to CDN, route camera feeds to digital storage end points, etc.
The inventive AIRcloud™ system's routing scheme also records these sources and destinations as metadata that can be recalled at a later date for media management, digital rights management, and other purposes.
Selecting of the “Virtual Workstation” button 227 in the navigation menu 220 (
Examples of appropriate computer-based video switching and instant replay software that may be used with the inventive system include vMix™ (from StudioCoast Pty Ltd., Robina, Queensland, AU), OBS (Open Broadcaster Software, from the OBS Project, an open source software distributed under General Public License, v2.), VizVector (from Vizrt), SimplyLive Replay (from Reidel Communications). Editing software examples include Adobe® Premiere®, Apple® Final Cut Pro®.
The inventive AIRcloud™ system enables more complex and sophisticated routing and switching because it is not constrained by traditional video hardware limitations of inputs, outputs, and layers. The system can expand exponentially on demand by daisy-chaining compute instances together. Furthermore, it is not constrained by geography.
The ability to communicate with all team members simultaneously in real time is critical to the production process. Dedicated intercom systems from multiple manufacturers exist for this purpose (Clear-Com, RTS, et al) but they are tied to proprietary hardware, limited in range, and are complicated or impossible to deploy to crew members working remotely. For remote access users there may also be challenges to preserving secure communications.
Selection of the “Comms” button 224 in the navigation menu 220 (
Once principal photography is complete many media projects must go through a post-production process. This can involve assembling an entire program from the beginning using camera original footage and/or working from a live-switched program master modifying and fine tuning the choices that were made in real time. This process can also include extracting highlight clips from the program feed and/or individual camera recordings, adding graphics and animation, modifying the program audio, repurposing the captured content for new purposes, as well as many other use cases.
The editing process typically involves aggregating video, audio, and graphic assets into a media storage device attached to an edit workstation. If multiple editors are working on a project, those assets and edit system configurations must be duplicated across all workstations. This is frequently a time-consuming, laborious, and potentially error-prone manual process. A substantial portion of an editor's labor is spent manually aggregating assets, manually categorizing them, manually managing the media, manually uploading edits for review, and then manually interpreting and organizing the feedback from reviewers for changes. Version management of assets can also be a problem.
EditShare™ (Watertown, MA) and others offer products that improve certain aspects of the edit process by enabling media to be shared across a Local Area Network (LAN) or, in some cases, duplicated across a Wide Area Network (WAN). Frame.io (an Adobe® company, New York, NY), is one example of a source of video review and collaboration platform that provides improved media management as well as the media ingestion process through their C2C (Camera-to-Cloud) product. The main drawback to these products is that they still require significant manual intervention as well as manual verification that the file was transferred securely. They also require that the workflow adhere to the duplication model, distributing the media to multiple endpoints for use.
The inventive AIRcloud™ system's post processing function removes the need to duplicate media by virtualizing the entire post-production process. Instead of sending media to various computer storage endpoints for use, the AIRcloud™ system brings the compute power and the editor to where the media resides. By concentrating the media in cloud-based storage with direct connection to cloud compute instances, any editor that logs into the virtual compute instance(s) will have access to identical assets without the need for duplication. The centralization of the post-process reduces the need for media duplication and helps streamline version management. It also ensures that the edit system configuration and media assets are the same across all editors and contributors to the editing process.
The integration of the AIRstation™ camera platform with the AIRcloud™ storage allows for media uploads from on location to cloud storage instances. It also provides bit-level file integrity verification to ensure the entire file has been uploaded and that it is an exact copy of the original. Support is also provided for asynchronous uploads to allow for a file upload to start, pause and continue uploading at a later time.
Selection of the “Post Setup” button 223 in navigation menu 220 (
Once an edited version of a project is completed, the editor 1002 can designate that file as “ready for review”. Text based notifications can be automatically generated for reviewers. Reviewers (internal 1010 and client 1012) can then log in to the inventive AIRcloud™ system where they will be directed to the file for edit review. A file playback interface allows for time-marked feedback. Timestamped notations are important for clarity. The time-marked feedback is available to all reviewers as well as approved project personnel. Indexed, timestamped, and collated feedback viewable by all contributors and the project editor(s), greatly streamlines the review process.
Aggregation, classification, storage, and retrieval of media assets is one of the more labor intensive portions of the post-production process. These are traditionally manual processes performed by a combination of producers, editors, and assistant editors. While some automation of image classification has been made possible in recent years through improved Machine Learning algorithms, the methods to trigger that process remain cumbersome.
The AIRcloud™ Media Asset Management (“MAM”) system augments existing post-production methods by automating the classification, movement, storage and retrieval of assets. This function can be selected using the “Media” button 222 in navigation menu 220. A traditionally fragmented process is centralized and standardized through a unified process that enables tight classification, tracking, and automated movement of media assets.
The AIRcloud™ MAM feature enables searching for assets by name, date, format, client, project, or other identifiers, tagging assets with keywords and/or additional metadata, manual and automated movement of assets, automated modifications of permissions, copying of assets to the edit system, and moving assets to archive or storage. From the moment an asset enters the system, its movement through routing, switching, storage, and editing can be tracked via this feature. This comprehensive tracking of assets throughout their lifecycle adds an unprecedented level of detail to asset metadata.
Artificial intelligence (AI)-assisted asset identification and categorization can be automated for assets ingested into the system to extract useful information from the video and audio. The approach of the AIRcloud™ system drastically simplifies the use of automatic categorization tools that use machine learning and artificial intelligence for functions such as object detection, keyword tagging, face detection, video segment identification, as well as computer vision analysis of human performance, body mechanics, emotion perception and more. Keywords and metadata gleaned from these AI tools are a critical tool for finding, organizing, and securing media assets. Additional AI-assisted functions may employ advanced audio analysis and annotation. The inventive AIRcloud™ system may also include automatic speech to text transcriptions by integrating a third party transcription application. This capability can further streamline the process of uploading video/audio files and then downloading results. In addition, speech detection function may be used to trigger other commands within the system, for example, the mention of a particular product or person may trigger an instruction to insert into the video an image of the mentioned product or person.
With online storage, nearline backup, and offline archival long term storage connected to the system, automated backup and archiving of assets can be programmed upon project completion.
The MAM function also integrates with the approval and review process, storing all project change notes and associating those with the assets. Once an asset or project has been marked as “approved” by reviewer(s), various tasks necessary to complete, distribute, and archive the assets and/or projects for the production can be automatically triggered. Indexing of assets and projects allows for simple retrieval of complete offline archived assets by issuing a “restore project” command.
The media asset management (MAM) function within the inventive AIRcloud™ system is capable of integrating via API with third party asset management systems such as Frame.io (Adobe®), catDV™ (Quantum Corporation), and others.
Existing third party asset management systems such as Frame.io have similar systems to the AIRcloud™ system's MAM function, however, they lack integration into the entire production/post production workflow, missing the provisioning portion, cloud instance routing, storage interconnection, automated permissions and more.
Metadata, including but not limited to, pan/tilt/roll positioning from motor encoders, velocity and position data from the camera platform Inertial Measurement Unit (IMU), lens data for focus and aperture, zoom motor data on lens focal length; camera metadata including ISO, f-stop, recording format, codec, bitrate, frame rate, white balance, color profile, shutter angle, exposure value offset, RGB information, auto focus settings, audio levels; user pan/tilt gesture information, user zoom/focus/aperture information; switcher video & audio source and graphic selection information can all be captured in real time for immediate use, in conjunction with the MAM function, or leverage the metadata for other post-production processes. These and other camera features and functions are described in the aforementioned International Publication No. WO2021/195641, which is incorporated herein by reference. This information can be used as a data set to improve system performance, train autonomous systems, and automate the production process.
The statuses of all connected AIRstations™ as well as all deployed virtual compute instances can be displayed on the Master Status Dashboard (“MSD”) screens.
The present disclosure is to be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments illustrated by the figures or description below.
The present application claims the benefit of the priority of U.S. Provisional Application No. 63/316,332, which is incorporated herein by reference in its entirety.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/US2023/014542 | 3/3/2023 | WO |
Number | Date | Country | |
---|---|---|---|
63316332 | Mar 2022 | US |