Patent Application
20220109722
The examples and non-limiting embodiments relate generally to network based media processing, and more particularly, to method and apparatus for dynamic workflow task management.
It is known to provide network based media processing.
An example apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: generate a capability description document comprising requirement resource changes for migration of tasks during run-time of a workflow between cloud and device environments; and trigger follow-up actions to migrate the task during the run-time of the workflow, between the cloud and the device environments, based on the capability description document.
The apparatus may further include, wherein the capability description document includes a media processing entity (MPE) capability description document (MDD).
The apparatus may further include, wherein the apparatus includes a plurality of media processing entities (MPEs) registered for specific processing tasks.
The apparatus may further include, wherein the media processing entities exists over multiple processing environments comprising the cloud and device environments.
The apparatus may further include, wherein the capability description document includes one or more of following capabilities: a name, a description, and an identifier; a repository of built in function; total and currently available hardware resources; or issuing events in case of reducing resource through at least one of notification, reporting, or monitoring.
The apparatus may further include, wherein to generate the capability description document the apparatus is further caused to use the network based media processing (NBMP) workflow description document, wherein the NBMP description document describe at least one of notification, reporting, or monitoring of the MPE.
The apparatus may further include, wherein the NBMP descriptors includes one or more of following: scheme descriptor; general descriptor; repository descriptor; list of supported functions; requirements; or system events.
The apparatus may further include, wherein the apparatus is further caused to incorporate native functions in a processing pipeline.
The apparatus may further include, wherein the native functions join the processing pipeline via a function repository or a workflow manager.
The apparatus may further include, wherein the capability description document includes capabilities of storage definition.
The apparatus may further include, wherein the capabilities of storage definition includes one or more of persistency properties or consistency properties.
The apparatus may further include, wherein the task is implemented as a static task or a mobile task.
The apparatus may further include, wherein the task includes capability to be moved to a different MPE based on an event notification received by a workflow manager.
The apparatus may further include, wherein the mobile task further includes capability to capture an execution state and transfer the execution state to a new location.
The apparatus may further include, wherein the apparatus includes a plurality of media processing entities (MPEs) defined through different MPE capability description document (MCD).
The apparatus may further include, wherein the apparatus is further caused to manage lifecycle of one or more MPEs; and notify, to the workflow manager, a state of each MPE of the one or more MPEs.
The apparatus may further include, wherein a media processing entity includes priority policies regarding availability of resources.
The apparatus may further include, wherein the apparatus further caused to manage media processing entity administration or operation.
The apparatus may further include, wherein media processing entity administration or operation includes a dynamic device discovery, a media processing entity subscription, and a media processing entity un-subscription.
The apparatus may further include, wherein the media processing entity administration or operation is managed via one or more application programming interfaces (APIs).
The apparatus may further include, wherein the one or more application programming interfaces includes interfaces for one or more of following operations: a media processing entity discovery operation, wherein the media processing entity discovery operation includes a mechanism to discover devices with media processing capabilities using the MPE capability description document (MDD); a media processing entity subscription operation to register a device as one or more media processing entities with the capability description document; a media processing entity authentication operation to authenticate the device as one or more media processing entities with the capability description document; a media processing entity sign-off operation to un-subscribe the media processing entity of the device from a processing pipelines; or a capability change operation, used by the device, to inform a workflow manager about changes in resources of the media processing entity.
The apparatus may further include, wherein the apparatus includes one or more of a workflow manager, a user equipment, a network based media processing source, a network based media processing sink, or a server.
The apparatus may further include, wherein the apparatus includes interface and signals to support tasks running on a cloud and end-user devices by using NBMP mobile MPE clients.
Another example apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform register a media processing entity with a persistency and a consistency enabled by network based media processing device; pause a workflow; create temporary queuing tasks with the same input and output capabilities of tasks effected; change a connection map between tasks effected and temporary queuing tasks; and resume workflow and data flow.
Yet another example apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform register a media processing entity with a persistency and a consistency enabled by network based media processing device; pause a workflow; create a read/write queuing channels, wherein the read/write queuing channels includes logic endpoint universal resource locators for data consuming and producing; update input/output descriptors of tasks effected task; and resume workflow and data flow.
An example method includes generating a capability description document to includes requirement changes for migration of tasks during run-time of a workflow between a cloud and device environments; and triggering follow-up actions to the task during the run-time of the workflow, between the cloud and the device environments, based on the capability description document.
The method may further include, wherein the capability description document includes a media processing entity (MPE) capability description document (MDD).
The method may further include, wherein the capability description document includes one or more of following capabilities: a name, a description, and an identifier; a repository of built in functions; total and currently available hardware resources; or issuing events in case of reducing resource through at least one of notification, reporting, or monitoring descriptions.
The method may further include, wherein generating the capability description document includes using network based media processing (NBMP) workflow description document, wherein the NBMP description document describes at least one of notification, reporting, or monitoring of the MPE.
The method may further include, wherein the NBMP descriptors include one or more of following: scheme descriptor; general descriptor; repository descriptor; list of supported functions; requirements; or system events.
The method may further include, incorporating native functions in a processing pipeline.
The method may further include, wherein the native functions join the processing pipeline via a function repository or a workflow manager.
The method may further include, wherein the capability description document includes capabilities of storage definition.
The method may further include, wherein the capabilities of storage definition includes one or more of persistency properties or consistency properties.
The method may further include, wherein the task is implemented as a static task or a mobile task.
The method may further include, wherein the task includes capability to be moved to a different MPE based on an event notification received by a workflow manager.
The method may further include, wherein the mobile task further includes capability to capture an execution state and transfer the execution state to a new location.
The method may further include, managing lifecycle of one or more MPE; and notifying, to the workflow manager, a state of each MPE of the one or more MPEs.
The method may further include, wherein a media processing entity includes priority policies regarding availability of resources.
The method of claim may further include managing media processing entity administration or operation.
The method may further include, wherein media processing entity administration or operation includes a dynamic device discovery, a media processing entity subscription, and a media processing entity un-subscription.
The method may further include, wherein the media processing entity administration or operation is managed via one or more application programming interfaces (APIs).
The method may further include, wherein the one or more application programming interfaces includes interfaces for one or more of following operations: a media processing entity discovery operation, wherein the media processing entity discovery operation comprises a mechanism to discover devices with media processing capabilities using the MPE capability description document (MDD); a media processing entity subscription operation to register a device as one or more media processing entities with the capability description document; a media processing entity authentication operation to authenticate the device as one or more media processing entities with the capability description document; a media processing entity sign-off operation to un-subscribe the media processing entity of the device from a processing pipelines; or a capability change operation, used by the device, to inform a workflow manager about changes in resources of the media processing entity.
Another example method includes registering a media processing entity with a persistency and a consistency enabled by network based media processing device; pausing a workflow; create temporary queuing tasks with the same input and output capabilities of tasks effected; changing a connection map between tasks effected and temporary queuing tasks; and resuming workflow and data flow.
Yet another example method includes registering a media processing entity with a persistency and a consistency enabled by network based media processing device; pausing a workflow; creating a read/write queuing channels, wherein the read/write queuing channels comprise logic endpoint universal resource locators for data consuming and producing; updating input/output descriptors of tasks effected task; and resuming workflow and data flow.
An example computer program product includes a computer readable storage medium having program code portions stored thereon, the program code portions configured, upon execution, to generate a capability description document comprising requirement resource changes for migration of tasks during run-time of a workflow between cloud and device environments; and trigger follow-up actions to migrate the task during the run-time of the workflow, between the cloud and the device environments, based on the capability description document.
Another example computer program product includes a computer readable storage medium having program code portions stored thereon, the program code portions configured, upon execution, to register a media processing entity with a persistency and a consistency enabled by network based media processing device; pause a workflow; create temporary queuing tasks with the same input and output capabilities of tasks effected; change a connection map between tasks effected and temporary queuing tasks; and resume workflow and data flow.
Yet another example computer program product comprising a computer readable storage medium having program code portions stored thereon, the program code portions configured, upon execution, to register a media processing entity with a persistency and a consistency enabled by network based media processing device; pause a workflow; create a read/write queuing channels, wherein the read/write queuing channels comprise logic endpoint universal resource locators for data consuming and producing; update input/output descriptors of tasks effected task; and resume workflow and data flow.
The computer program product as described in any of the previous paragraphs, wherein the computer program product includes a non-transitory computer readable medium.
Still another example apparatus includes: at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: generate a capability description document comprising one or more of following capabilities or properties: a name, a description, or an identifier of a media processing entity; location of the media processing entity in a media processing workflow; available hardware resources; persistency properties or capabilities; or security parameters.
The example apparatus may further include, wherein the capability description document comprises capabilities of storage definition.
The example apparatus may further include, wherein the capabilities of storage definition comprise one or more of persistency properties or consistency properties.
The example apparatus may further include, wherein a media processing entity comprises priority policies regarding availability of resources.
The example apparatus may further include, wherein the apparatus further caused to manage media processing entity administration.
The example apparatus may further include, wherein media processing entity administration comprises a dynamic media processing entity device discovery, a media processing entity subscription, and a media processing entity un-subscription.
The example apparatus may further include, wherein the media processing entity administration is managed via one or more application programming interfaces (APIs).
The example apparatus may further include, wherein the one or more application programming interfaces comprises interfaces for one or more of following operations: a media processing entity discovery operation, wherein the media processing entity discovery operation comprises a mechanism to discover devices with media processing capabilities using the MPE capability description document (MDD); a media processing entity subscription operation to register a device as one or more media processing entities with the capability description document; a media processing entity authentication operation to authenticate the device as one or more media processing entities with the capability description document;
a media processing entity sign-off operation to un-subscribe the media processing entity of the device from a processing pipelines; or a capability change operation, used by the device, to inform a workflow manager about changes in resources of the media processing entity.
A still another example method includes: generating a capability description document comprising one or more of following capabilities or properties: a name, a description, or an identifier of a media processing entity; location of the media processing entity in a media processing workflow; available hardware resources; persistency properties or capabilities; or security parameters.
The example method may further include, wherein the capability description document comprises capabilities of storage definition.
The example method may further include, wherein the capabilities of storage definition comprise one or more of persistency properties or consistency properties.
The example method may further include, wherein a media processing entity comprises priority policies regarding availability of resources.
The example method may further include managing media processing entity administration.
The example method may further include, wherein media processing entity administration comprises a dynamic media processing entity device discovery, a media processing entity subscription, and a media processing entity un-subscription.
The example method may further include, wherein the media processing entity administration is managed via one or more application programming interfaces (APIs).
The example method may further include, wherein the one or more application programming interfaces comprises interfaces for one or more of following operations: a media processing entity discovery operation, wherein the media processing entity discovery operation comprises a mechanism to discover devices with media processing capabilities using the MPE capability description document (MDD); a media processing entity subscription operation to register a device as one or more media processing entities with the capability description document; a media processing entity authentication operation to authenticate the device as one or more media processing entities with the capability description document; a media processing entity sign-off operation to un-subscribe the media processing entity of the device from a processing pipelines; or a capability change operation, used by the device, to inform a workflow manager about changes in resources of the media processing entity.
A still another example computer program product includes a computer readable storage medium having program code portions stored thereon, the program code portions configured, upon execution, to: generate a capability description document comprising one or more of following capabilities or properties: a name, a description, or an identifier of a media processing entity; location of the media processing entity in a media processing workflow; available hardware resources; persistency properties or capabilities; or security parameters.
The example computer program product may further include, wherein the apparatus is further caused to perform the method as described in any of the previous paragraphs.
The example computer program product may further include, wherein the computer program product comprises a wherein the computer readable comprises a non-transitory computer readable medium.
The foregoing aspects and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:
The following acronyms and abbreviations that may be found in the specification and/or the drawing figures are defined as follows:
Some embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments of the invention are shown. Indeed, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout. As used herein, the terms “data,” “content,” “information,” and similar terms may be used interchangeably to refer to data capable of being transmitted, received and/or stored in accordance with embodiments of the present invention. Thus, use of any such terms should not be taken to limit the spirit and scope of embodiments of the present invention.
Additionally, as used herein, the term ‘circuitry’ refers to (a) hardware-only circuit implementations (e.g., implementations in analog circuitry and/or digital circuitry); (b) combinations of circuits and computer program product(s) comprising software and/or firmware instructions stored on one or more computer readable memories that work together to cause an apparatus to perform one or more functions described herein; and (c) circuits, such as, for example, a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation even if the software or firmware is not physically present. This definition of ‘circuitry’ applies to all uses of this term herein, including in any claims. As a further example, as used herein, the term ‘circuitry’ also includes an implementation comprising one or more processors and/or portion(s) thereof and accompanying software and/or firmware. As another example, the term ‘circuitry’ as used herein also includes, for example, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, other network device, and/or other computing device.
As defined herein, a “computer-readable storage medium,” which refers to a non-transitory physical storage medium (e.g., volatile or non-volatile memory device), can be differentiated from a “computer-readable transmission medium,” which refers to an electromagnetic signal.
A method, apparatus and computer program product are provided in accordance with an example embodiment in order to provide system and key interfaces with signaling for dynamic native function addressability or discovery, function registration, and deregistration in a network or distributed workflow processing environment. A method, apparatus and computer program product are provided in accordance with further example embodiment in order to provide mechanism for dynamic workflow task management in a network based media processing environment.
The following describes in detail suitable apparatus and possible mechanisms for network based media processing according to embodiments. In this regard reference is first made to
The apparatus 50 may for example be a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or a lower power device. However, it would be appreciated that embodiments of the examples described herein may be implemented within any electronic device or apparatus which may process data in communication network.
The apparatus 50 may comprise a housing 30 for incorporating and protecting the device. The apparatus 50 further may comprise a display 32 in the form of a liquid crystal display. In other embodiments of the examples described herein the display may be any suitable display technology suitable to display media or multimedia content, for example, an image or video. The apparatus 50 may further comprise a keypad 34. In other embodiments of the examples described herein any suitable data or user interface mechanism may be employed. For example the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.
The apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input. The apparatus 50 may further comprise an audio output device which in embodiments of the examples described herein may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection. The apparatus 50 may also comprise a battery (or in other embodiments of the examples described herein the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera capable of recording or capturing images and/or video. The apparatus 50 may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB/firewire wired connection.
The apparatus 50 may comprise a controller 56, processor or processor circuitry for controlling the apparatus 50. The controller 56 may be connected to memory 58 which in embodiments of the examples described herein may store both data in the form of image and audio data and/or may also store instructions for implementation on the controller 56. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and/or decoding of audio and/or video data or assisting in coding and/or decoding carried out by the controller.
The apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.
The apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and/or for receiving radio frequency signals from other apparatus(es).
The apparatus 50 may comprise a camera 42 capable of recording or detecting individual frames which are then passed to the codec 54 or the controller for processing. The apparatus may receive the video image data for processing from another device prior to transmission and/or storage. The apparatus 50 may also receive either wirelessly or by a wired connection the image for coding/decoding. The structural elements of apparatus 50 described above represent examples of means for performing a corresponding function.
With respect to
The system 10 may include both wired and wireless communication devices and/or apparatus 50 suitable for implementing embodiments of the examples described herein.
For example, the system shown in
The example communication devices shown in the system 10 may include, but are not limited to, an electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22. The apparatus 50 may be stationary or mobile when carried by an individual who is moving. The apparatus 50 may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle or any similar suitable mode of transport.
The embodiments may also be implemented in a set-top box; for example, a digital TV receiver, which may/may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and/or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and/or embedded systems offering hardware/software based coding.
Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 11 and the internet 28. The system may include additional communication devices and communication devices of various types.
The communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, 3GPP Narrowband IoT and any similar wireless communication technology. A communications device involved in implementing various embodiments of the examples described herein may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.
In telecommunications and data networks, a channel may refer either to a physical channel or to a logical channel. A physical channel may refer to a physical transmission medium such as a wire, whereas a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels. A channel may be used for conveying an information signal, for example a bitstream, from one or several senders (or transmitters) to one or several receivers.
The embodiments may also be implemented in so-called IoT devices. The Internet of Things (IoT) may be defined, for example, as an interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure. The convergence of various technologies has and may enable many fields of embedded systems, such as wireless sensor networks, control systems, home/building automation, and the like, to be included the Internet of Things (IoT). In order to utilize Internet IoT devices are provided with an IP address as a unique identifier. IoT devices may be provided with a radio transmitter, such as WLAN or Bluetooth transmitter or a RFID tag. Alternatively, IoT devices may have access to an IP-based network via a wired network, such as an Ethernet-based network or a power-line connection (PLC).
An apparatus 400 is provided in accordance with an example embodiment as shown in
The processing circuitry 402 may be in communication with the memory device 404 via a bus for passing information among components of the apparatus 400. The memory device may be non-transitory and may include, for example, one or more volatile and/or non-volatile memories. In other words, for example, the memory device may be an electronic storage device (e.g., a computer readable storage medium) comprising gates configured to store data (e.g., bits) that may be retrievable by a machine (e.g., a computing device like the processing circuitry). The memory device may be configured to store information, data, content, applications, instructions, or the like for enabling the apparatus to carry out various functions in accordance with an example embodiment of the present disclosure. For example, the memory device could be configured to buffer input data for processing by the processing circuitry. Additionally or alternatively, the memory device could be configured to store instructions for execution by the processing circuitry.
The apparatus 400 may, in some embodiments, be embodied in various computing devices as described above. However, in some embodiments, the apparatus may be embodied as a chip or chip set. In other words, the apparatus may comprise one or more physical packages (e.g., chips) including materials, components and/or wires on a structural assembly (e.g., a baseboard). The structural assembly may provide physical strength, conservation of size, and/or limitation of electrical interaction for component circuitry included thereon. The apparatus may therefore, in some cases, be configured to implement an embodiment of the present disclosure on a single chip or as a single “system on a chip.” As such, in some cases, a chip or chipset may constitute means for performing one or more operations for providing the functionalities described herein.
The processing circuitry 402 may be embodied in a number of different ways. For example, the processing circuitry may be embodied as one or more of various hardware processing means such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing element with or without an accompanying DSP, or various other circuitry including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like. As such, in some embodiments, the processing circuitry may include one or more processing cores configured to perform independently. A multi-core processing circuitry may enable multiprocessing within a single physical package. Additionally or alternatively, the processing circuitry may include one or more processors configured in tandem via the bus to enable independent execution of instructions, pipelining and/or multithreading.
In an example embodiment, the processing circuitry 402 may be configured to execute instructions stored in the memory device 404 or otherwise accessible to the processing circuitry. Alternatively or additionally, the processing circuitry may be configured to execute hard coded functionality. As such, whether configured by hardware or software methods, or by a combination thereof, the processing circuitry may represent an entity (e.g., physically embodied in circuitry) capable of performing operations according to an embodiment of the present disclosure while configured accordingly. Thus, for example, when the processing circuitry is embodied as an ASIC, FPGA or the like, the processing circuitry may be specifically configured hardware for conducting the operations described herein. Alternatively, as another example, when the processing circuitry is embodied as an executor of instructions, the instructions may specifically configure the processing circuitry to perform the algorithms and/or operations described herein when the instructions are executed. However, in some cases, the processing circuitry may be a processor of a specific device (e.g., an image or video processing system) configured to employ an embodiment of the present invention by further configuration of the processing circuitry by instructions for performing the algorithms and/or operations described herein. The processing circuitry may include, among other things, a clock, an arithmetic logic unit (ALU) and logic gates configured to support operation of the processing circuitry.
The communication interface 406 may be any means such as a device or circuitry embodied in either hardware or a combination of hardware and software that is configured to receive and/or transmit data, including video bitstreams. In this regard, the communication interface may include, for example, an antenna (or multiple antennas) and supporting hardware and/or software for enabling communications with a wireless communication network. Additionally or alternatively, the communication interface may include the circuitry for interacting with the antenna(s) to cause transmission of signals via the antenna(s) or to handle receipt of signals received via the antenna(s). In some environments, the communication interface may alternatively or also support wired communication. As such, for example, the communication interface may include a communication modem and/or other hardware/software for supporting communication via cable, digital subscriber line (DSL), universal serial bus (USB) or other mechanisms.
In some embodiments, the apparatus 400 may optionally include a user interface that may, in turn, be in communication with the processing circuitry 402 to provide output to a user, such as by outputting an encoded video bitstream and, in some embodiments, to receive an indication of a user input. As such, the user interface may include a display and, in some embodiments, may also include a keyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, a microphone, a speaker, or other input/output mechanisms. Alternatively or additionally, the processing circuitry may comprise user interface circuitry configured to control at least some functions of one or more user interface elements such as a display and, in some embodiments, a speaker, ringer, microphone and/or the like. The processing circuitry and/or user interface circuitry comprising the processing circuitry may be configured to control one or more functions of one or more user interface elements through computer program instructions (e.g., software and/or firmware) stored on a memory accessible to the processing circuitry (e.g., memory device, and/or the like).
Network-Based Media Processing (NBMP)
Network-based media processing (NBMP) is a new standard (ISO/IEC 23090-8) in MPEG MPEG-I.
As shown in
NBMP processing relies on a workflow manager, that can be virtualized, to start and control media processing. The Workflow Manager receives a Workflow Description from the NBMP Source, which instructs the Workflow Manager about the desired processing and the input and output formats to be taken and generated, respectively.
The workflow manager (the Manager) creates a workflow based on the workflow description document (WDD) that it receives from the NBMP Source. The workflow manager selects and deploys the NBMP Functions into selected media processing entities and then performs the configuration of the tasks. The WDD can include a number of logic descriptors.
The NBMP can define APIs and formats such as Function templates and workflow description document (WDD) consisting of a number of logic descriptors. NBMP uses the so-called descriptors as the basic elements for its all resource documents such as the workflow documents, task documents, and function documents. Descriptors are a group of NBMP parameters which describe a set of related characteristics of Workflow, Function or Task. Some key descriptors are General, Input, Output, Processing, Requirements, Configuration etc.
In order to hide workflow internal details from the NBMP Source, all updates to the workflow are performed through Workflow Manager. The manager is the single point of access for the creation or change of any workflows. Workflows represent the processing flows defined in WDD provided by NBMP Source (aka. the client). A workflow can be defined as a chain of tasks, specified by the “connection-map” Object in the Processing Descriptor of the WDD.
The Workflow Manager may use pre-determined implementations of media processing functions and use them together to create the media processing workflow. NBMP defines a Function Discovery API that it uses with a Function Repository to discover and load the desired Functions.
A Function, once loaded, becomes a Task, which is then configured by the Workflow Manager through the Task API and can start processing incoming media. It is noted that a cloud and/or network service providers can define their own APIs to assign computing resources to their customers.
There is an increasing deployment of distributed processing infrastructure where the processing nodes or MPEs (in NBMP context) can be located in the cloud or the edge or even on the end user device. One example of such a use case is the split-rendering process employed in cloud gaming services.
Leveraging such a distributed environment requires that an MPE, which is the task execution context, to span over different processing entities.
The exploitation of the distributed environment can range from moving a task or a workflow, partially or entirely, from one infrastructure to another. For example, transferring a last rendering task in a workflow from edge to an end user device or vice versa.
The NBMP Technology considers the following requirements for the design and development of NBMP:
Splitting Workflow
As shown in
In split-rendering, some of the tasks of a workflow may be implemented in a distributed infrastructure including the media source and/or the media sink. The embodiments herein are not restricted to only media source and/or media sink but can include any other device which may subscribe in the run-time of the workflow using the functionalities presented in this document.
Proposed Methodology in Accordance with an Embodiments
In this embodiment, to accommodate split rendering in NBMP, following parameters/criteria are considered:
MPE Capabilities Description
An MPE can be part of any mobile device/network element including the media source and/or media sink. MPE capabilities Description include, but are not limited to:
The description uses NBMP descriptors, including but not limited to, following descriptors:
The MPE Capabilities Description (MD) is defined in Table 1.
Various embodiments of the present invention, provide a mechanism to cover the communication between a mobile device and the workflow manager and other features such as security/encryption of the mobile device.
Various embodiments describe a design to allow dynamic and mobile computing environment changes. Further, various embodiments propose the following:
Persistency, for example, may refer to the availability of data from a persistent versus volatile storage. A persistent storage can be expected to be available whereas volatile storage needs to be initialized again. For example, a persistent storage is readily accessible after moving a task to a new environment during workflow execution. However, a non-persistently stored data needs to be retrieved and made available again for the task after moving to a new environment. Cloud storage is an example of persistent storage. Local storage on MPE is an example of non-persistent or volatile storage.
Consistency refers to how recent is the snapshot of the data. In case of data, which is constantly updating, there may be multiple data storage URLs with different consistency values. The value can only be greater than or equal to 0, with 0 indicating the data being consistent, e.g. no newer version is available. The value greater than 0 indicates that this version of the data is older than the most recent version of the data. In some embodiments, the value may indicate time in milliseconds.
NBMP: ISO/IEC 23090-8 Network-based Media Processing. The NBMP framework defines the interfaces including both data formats and APIs among the entities connected through the digital networks for media processing.
Workflow: A sequence of tasks connected as a graph that processes the media data.
MPE: A Media Processing Entity (MPE) runs processing tasks applied on the media data and the related metadata received from media sources or other tasks. A media processing task is a process applied to media and metadata input(s), producing media data and related metadata output(s) to be consumed by a media sink or other media processing tasks (for example, as shown in
MPE Capability Description or MPE Description (MCD/MD): a logical description of details of a NBMP media processing entity (MPE). MPE Capabilities description document (MDD) is a document containing MCD in the JSON representation format. MPE capabilities resource (MCR or MR) is a rest resource that contains MDD.
NBMP uses data plane to define media formats, the metadata, and the supplementary information formats between the Media source 902 and the task 908, as well as between the tasks 908, and Media Sink 912. Media source 902 and Media Sink 912 can be in the same UE 901, or in different UEs (901 and 904).
NBMP devices, for example, user equipments 901 and 904, edge 906, cloud 905, and workflow manager 903 communicate via the control and data planes.
Design
Design: MPE as a Portable Execution Context for NBMP Tasks
A context is an abstract layer that includes following features:
In an embodiment, tasks may be implemented as mobile or static tasks. The mobile tasks have capabilities to be moved to a different host depending on event notifications received by the workflow manager, for example task migration with same task implementation or image. Th mobile tasks have additional feature, as compared to the static or normal tasks, of capturing the execution state and transferring it to the new location, for example allowing persistency of task states. The new location can be unique identifier of an MPE, for example virtual hostname or IP address of the MPE in the operational network of the given workflow. In ISO NBMP standard, for example, the connection between two tasks is defined as a link or connection in the “connection-map” object. In the case of mobile tasks, those links may contain properties to indicate the connection states, such as “virtual” for virtual and dynamic connections; or “breakable” for breakable connections. Those property value may mandate those two connected tasks shall be run in different MPE or not.
In another embodiment, one device can have one MPE registered for specific processing task. One device can have more than one MPEs defined through different MPE Capability Descriptor (MCD) documents with different capabilities such as HW resource constraints and specific function description documents (FDDs).
In an embodiment, it is the job of the device to manage the lifecycle of one or all MPEs and notify the workflow manager the MPE states, respectively and independently. In an alternate embodiment, the workflow manager manages the lifecycle of one or all MPEs hosted in the device.
Multiple MPEs may have different priority policies regarding the underlying availability of device resources.
Communication Between Workflow Manager and MPEs
MPE and Function Registration
As illustrated in
MPE Capability Description Parameters Related to Storage
MPE capability description may include other parameters together with function descriptions. A transparent data persistency and consistency refers to the ability to move any data in one MPE to another MPE during the migration of tasks.
In an embodiment, the storage parameters like persistence-storage-url property may be used directly, or converted to the part of the input and output descriptors, so they are specific to individual input or/and output of one task. Some example parameters are described below:
Thus, data for an MPE can be described with one or more above described parameters.
Referring now to
As shown in block 1608, the apparatus includes means, such as the processor 1502, for creating by a workflow manager a temporary queuing task (e.g. FIFO queues task) with same or substantially same input and output capabilities of the task effected. As shown in block 1510, the apparatus includes means, such as the processor 1502, for changing a connection-map between the tasks and the newly created FIFO tasks. Thereafter, as shown in block 1612, the apparatus includes means, such as the processor 1502, for resuming workflow (e.g. back to run state) and data flows.
In an alternate embodiment, as shown in block 1612, the apparatus includes means, such as the processor 1502, for creating by a workflow manager temporary read/write (R/W) queuing queueing channels using cloud storage service or distributed data queueing service with unique URLS. Each channel has logic endpoint URLs for data consuming and producing. As shown in block 1612, the apparatus includes means, such as the processor 1502, for updating by the workflow manager the input/output descriptions of the tasks effected. Thereafter, as shown in block 1612, the apparatus includes means, such as the processor 1502, for resuming workflow (e.g. back to run state) and data flows.
Event Driven Approach
Design: Event Notification for Processing Capability Changes on Device MPEs
In an embodiment, the method 1900 further incorporating native functions in a processing pipeline. In another embodiment, the method 1800 includes managing lifecycle of one or more MPE and notifying the workflow manager state of each MPE of the one or more MPEs. In yet another embodiment, the method 1900 further includes managing media processing entity administration.
Turning to
The RAN node 170 in this example is a base station that provides access by wireless devices such as the UE 110 to the wireless network 100. The RAN node 170 may be, for example, a base station for 5G, also called New Radio (NR). In 5G, the RAN node 170 may be a NG-RAN node, which is defined as either a gNB or an ng-eNB. A gNB is a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface to a 5GC (such as, for example, the network element(s) 190). The ng-eNB is a node providing E-UTRA user plane and control plane protocol terminations towards the UE, and connected via the NG interface to the 5GC. The NG-RAN node may include multiple gNBs, which may also include a central unit (CU) (gNB-CU) 196 and distributed unit(s) (DUs) (gNB-DUs), of which DU 195 is shown. Note that the DU may include or be coupled to and control a radio unit (RU). The gNB-CU is a logical node hosting radio resource control (RRC), SDAP and PDCP protocols of the gNB or RRC and PDCP protocols of the en-gNB that controls the operation of one or more gNB-DUs. The gNB-CU terminates the F1 interface connected with the gNB-DU. The F1 interface is illustrated as reference 198, although reference 198 also illustrates a link between remote elements of the RAN node 170 and centralized elements of the RAN node 170, such as between the gNB-CU 196 and the gNB-DU 195. The gNB-DU is a logical node hosting RLC, MAC and PHY layers of the gNB or en-gNB, and its operation is partly controlled by gNB-CU. One gNB-CU supports one or multiple cells. One cell is supported by only one gNB-DU. The gNB-DU terminates the F1 interface 198 connected with the gNB-CU. Note that the DU 195 is considered to include the transceiver 160, for example, as part of a RU, but some examples of this may have the transceiver 160 as part of a separate RU, for example, under control of and connected to the DU 195. The RAN node 170 may also be an eNB (evolved NodeB) base station, for LTE (long term evolution), or any other suitable base station or node.
The RAN node 170 includes one or more processors 152, one or more memories 155, one or more network interfaces (N/W I/F(s)) 161, and one or more transceivers 160 interconnected through one or more buses 157. Each of the one or more transceivers 160 includes a receiver, Rx, 162 and a transmitter, Tx, 163. The one or more transceivers 160 are connected to one or more antennas 158. The one or more memories 155 include computer program code 153. The CU 196 may include the processor(s) 152, memories 155, and network interfaces 161. Note that the DU 195 may also contain its own memory/memories and processor(s), and/or other hardware, but these are not shown.
The RAN node 170 includes a module 150, comprising one of or both parts 150-1 and/or 150-2, which may be implemented in a number of ways. The module 150 may be implemented in hardware as module 150-1, such as being implemented as part of the one or more processors 152. The module 150-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the module 150 may be implemented as module 150-2, which is implemented as computer program code 153 and is executed by the one or more processors 152. For instance, the one or more memories 155 and the computer program code 153 are configured to, with the one or more processors 152, cause the RAN node 170 to perform one or more of the operations as described herein. Note that the functionality of the module 150 may be distributed, such as being distributed between the DU 195 and the CU 196, or be implemented solely in the DU 195.
The one or more network interfaces 161 communicate over a network such as via the links 176 and 131. Two or more gNBs 170 may communicate using, for example, link 176. The link 176 may be wired or wireless or both and may implement, for example, an Xn interface for 5G, an X2 interface for LTE, or other suitable interface for other standards.
The one or more buses 157 may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, wireless channels, and the like. For example, the one or more transceivers 160 may be implemented as a remote radio head (RRH) 195 for LTE or a distributed unit (DU) 195 for gNB implementation for 5G, with the other elements of the RAN node 170 possibly being physically in a different location from the RRH/DU, and the one or more buses 157 could be implemented in part as, for example, fiber optic cable or other suitable network connection to connect the other elements (for example, a central unit (CU), gNB-CU) of the RAN node 170 to the RRH/DU 195. Reference 198 also indicates those suitable network link(s).
It is noted that description herein indicates that “cells” perform functions, but it should be clear that equipment which forms the cell may perform the functions. The cell makes up part of a base station. That is, there can be multiple cells per base station. For example, there could be three cells for a single carrier frequency and associated bandwidth, each cell covering one-third of a 360 degree area so that the single base station's coverage area covers an approximate oval or circle. Furthermore, each cell can correspond to a single carrier and a base station may use multiple carriers. So if there are three 120 degree cells per carrier and two carriers, then the base station has a total of 6 cells.
The wireless network 100 may include a network element or elements 190 that may include core network functionality, and which provides connectivity via a link or links 181 with a further network, such as a telephone network and/or a data communications network (for example, the Internet). Such core network functionality for 5G may include access and mobility management function(s) (AMF(S)) and/or user plane functions (UPF(s)) and/or session management function(s) (SMF(s)). Such core network functionality for LTE may include MME (Mobility Management Entity)/SGW (Serving Gateway) functionality. These are merely example functions that may be supported by the network element(s) 190, and note that both 5G and LTE functions might be supported. The RAN node 170 is coupled via a link 131 to the network element 190. The link 131 may be implemented as, for example, an NG interface for 5G, or an S1 interface for LTE, or other suitable interface for other standards. The network element 190 includes one or more processors 175, one or more memories 171, and one or more network interfaces (N/W I/F(s)) 180, interconnected through one or more buses 185. The one or more memories 171 include computer program code 173. The one or more memories 171 and the computer program code 173 are configured to, with the one or more processors 175, cause the network element 190 to perform one or more operations.
The wireless network 100 may implement network virtualization, which is the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors 152 or 175 and memories 155 and 171, and also such virtualized entities create technical effects.
The computer readable memories 125, 155, and 171 may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The computer readable memories 125, 155, and 171 may be means for performing storage functions. The processors 120, 152, and 175 may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as non-limiting examples. The processors 120, 152, and 175 may be means for performing functions, such as controlling the UE 110, RAN node 170, network element(s) 190, and other functions as described herein.
In general, the various embodiments of the user equipment 110 can include, but are not limited to, cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, tablets with wireless communication capabilities, as well as portable units or terminals that incorporate combinations of such functions.
One or more of modules 140-1, 140-2, 150-1, and 150-2 may be configured to implement dynamic workflow task management in a network based media processing environment based on the examples described herein. Computer program code 173 may also be configured to implement dynamic workflow task management in a network based media processing environment.
As described above,
A computer program product is therefore defined in those instances in which the computer program instructions, such as computer-readable program code portions, are stored by at least one non-transitory computer-readable storage medium with the computer program instructions, such as the computer-readable program code portions, being configured, upon execution, to perform the functions described above, such as in conjunction with the flowcharts of
Accordingly, blocks of the flowcharts support combinations of means for performing the specified functions and combinations of operations for performing the specified functions for performing the specified functions. It will also be understood that one or more blocks of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by special purpose hardware-based computer systems which perform the specified functions, or combinations of special purpose hardware and computer instructions.
In some embodiments, certain ones of the operations above may be modified or further amplified. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or amplifications to the operations above may be performed in any order and in any combination.
Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe example embodiments in the context of certain example combinations of elements and/or functions, it should be appreciated that different combinations of elements and/or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and/or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.
| Number | Date | Country | |
|---|---|---|---|
| 63088610 | Oct 2020 | US |
