Patent Application
20240259126
The present disclosure relates to the technical field of multimedium, more particularly to, an audio and video synchronization method for a playback module, an audio and video synchronization device for a playback module, an electronic device and a computer readable storage medium.
Digital Televisions (DTVs) and Internet Protocol Televisions (IPTVs) are common audio-visual systems. Audio and video synchronization is an important aspect that affects the product quality and user experience of audiovisual systems.
However, in practical applications, the synchronization mechanism used by audiovisual systems such as DTV and IPTV products has some shortcomings. For example, it is difficult to support multiple-channel broadcasting synchronization, can not support cross-process audio and video synchronization. The synchronization mechanism is tightly associated with a player, is difficult to transplant, and can not be used flexibly.
The present disclosure aims to solve at least the technical problem existing in the prior art. For this purpose, a first purpose of the present disclosure is to propose an audio and video synchronization method, which can realize multi-channel play synchronization and support cross-process audio and video synchronization, and the synchronization mechanism is easier to transplant and more flexible.
A second purpose of the present disclosure is to propose an audio and video synchronization device.
A third purpose of the present disclosure is to propose an electronic device.
A fourth purpose of the present disclosure is to propose a computer readable storage medium.
In order to achieve the above purposes, an audio and video synchronization method for a playback module is provided according to an embodiment in a first aspect of the present disclosure. The playback module includes an object-oriented program language standard library and a kernel driver. The audio and video synchronization method includes: determining, in response to a playback instruction, a target player to build an audio module and a video module from the target player; creating, in the program language standard library, a synchronization instance corresponding to the target player, the synchronization instance having a unique instance identifier; and obtaining a reference clock corresponding to the synchronization instance based on the instance identifier, and controlling data outputs of the audio module and the video module corresponding to the target player based on the reference clock.
According to the embodiment of the present disclosure, the audio and video synchronization method for the playback module adopts the object-oriented design idea, carries out synchronization processing through the instance and the instance identifier, can support multiple-channel playback and cross-process synchronization scene. The audio and video synchronization module adopts the form of the program language standard library and the kernel driver, therefore any player or module can achieve audio and video synchronization by compiling the program language standard library and calling a related synchronization method of the program language standard library, thereby achieving strong portability and more flexible usage scenarios.
In order to achieve the above purposes, an audio and video synchronization device for a playback module is provided according to an embodiment in a second aspect of the present disclosure. The audio and video synchronization device for a playback a module, includes: a building block configured to determine a target player in response to a playback instruction to build an audio module and a video module from the target player; a program language standard library configured to create a synchronization instance corresponding to the target player, the synchronization instance having a unique instance identifier; and a kernel driver configured to obtain a reference clock corresponding to the synchronization instance based on the instance identifier, and to control data outputs of the audio module and the video module corresponding to the target player based on the reference clock.
According to the embodiment of the present disclosure, the audio and video synchronization device for the playback module adopts the object-oriented design idea, carries out synchronization processing through the instance and the instance identifier, can support multiple-channel playback and cross-process synchronization scene, adopts the form of the program language standard library and the kernel driver, therefore any player or module can achieve audio and video synchronization by compiling the program language standard library and calling a related synchronization method of the program language standard library, thereby achieving strong portability and more flexible usage scenarios.
In order to achieve the above purposes, an electronic device is provided according to an embodiment in a third aspect of the present disclosure. The electronic device includes: a processor; and a memory in communication connection with the processor. The memory stores a computer program that is executable by the processor. The processor, when executing the computer program, implements the audio and video synchronization method for the playback module.
According to the embodiment of the present disclosure, the electronic device, by performing the audio and video synchronization method for the playback module by its processor according to the above embodiment, can support multiple-channel playback and cross-process synchronization scene, adopts the form of the program language standard library and the kernel driver to achieve synchronous processing, therefore any player or module can achieve audio and video synchronization by compiling the program language standard library and calling a related synchronization method of the program language standard library, thereby achieving strong portability and more flexible usage scenarios.
In order to achieve the above purposes, a computer readable storage medium is provided according to an embodiment in a fourth aspect of the present disclosure. The computer readable storage medium stores a computer program thereon. The computer program, when executed by a processor, implements the audio and video synchronization method for the playback module.
According to the embodiment of the present disclosure, the computer readable storage medium stores the computer program medium thereon, and the computer program, when executed, implements the audio and video synchronization method for the playback module according to the embodiment in the first aspect of the present disclosure. The audio and video synchronization method for the playback module adopts the object-oriented design idea, carries out synchronization processing through the instance and the instance identifier, can support multiple-channel playback and cross-process synchronization scene. The synchronous processing adopts the form of the program language standard library and the kernel driver, therefore any player or module can achieve audio and video synchronization by compiling the program language standard library and calling a related synchronization method of the program language standard library, thereby achieving strong portability and more flexible usage scenarios.
Additional aspects and benefits of the present disclosure will be given partially in the description below, and partially will be obvious from the description below, or will be understood through the practice of the present disclosure.
The above and/or additional aspects and advantages of the present disclosure will become apparent and easy to understand from the description of embodiments in conjunction with the accompanying drawings, in which:
In order to make the purposes, technical solutions and advantages of the present disclosure more clear, the present disclosure will be described in detail below with reference to the attached drawings and embodiments. It should be understood that the specific embodiments described herein are intended only to interpret the present disclosure and are not intended to limit the present disclosure.
An audio and video synchronization method for a playback module according to an embodiment of the present disclosure adopts the object-oriented design idea and program language, can support multiple-channel play synchronization, and can also realize cross-process synchronization. The synchronization module is less coupling with the player, is easier to transplant, and is more flexible in use.
The object-oriented design is a software development method, i.e., a programming paradigm. In the object-oriented programming, the term “object” refers to a certain component of a computer system. An object can contain two meanings, one is data, the other is action. An object is a combination of data and action. An object can not only perform operations, but also record operation results in time. A method is an operation that an object can perform. A method also has another name, called a function. A method is a defined function in a class and its specific function is to describe an object.
In the embodiments of the present disclosure, the playback module may be a chipset or an audio and video playback product such as a television set, etc. The playback module adopts the form of an object-oriented program language standard (such as C++ library) and a kernel driver. In the program language standard such as C++ library, a synchronization instance is created, is an object in the object-oriented program, takes audio and video as objects and further includes a method in which the audio and video are taken as objects. The kernel driver saves core synchronization data, and the program language standard library interacts with the kernel driver to obtain the required core synchronization data for audio and video synchronization processing. In some embodiments, the playback module may be a plug-in, a software package or a program library.
An audio and video synchronization method for a playback module according to an embodiment of the present disclosure will be described with reference to
The audio and video synchronization method for the playback module according to the embodiment of the present disclosure includes at least the following steps S1-S4 which are described as follows.
At step S1, a target player is determined in response to a playback instruction to build an audio module and a video module from the target player.
In an embodiment, a user can input a playback instruction using a remote controller or a touchpad. The playback instruction can be a single-channel playback instruction, such as an instruction to turn on a certain video. Alternatively, the playback instruction can be a multi-channel playback instruction, such as an instruction to turn on multiple playback pictures, e.g., picture in picture during TV playback.
In an embodiment, there may be multiple idle playback channels, that is, players that are not currently in a playback state. After receiving the playback instruction, a kernel driver can use one or more idle players to play audio and video, that is, using the one or more idle players as the target player(s).
Specifically, the target player builds an audio stream running thread as an audio module, and a video stream running thread as a video module.
At step S2, a synchronization instance corresponding to the target player is created in the program language standard library.
The synchronization instance is a concept in object-oriented programming, i.e., an object, and has a unique instance identifier, that is, instance id.
In an embodiment, the kernel driver is configured with a table for managing synchronization instances, which can be called as an instance management table in the present disclosure. When a synchronization instance is created in the program language standard library such as C++ library, the kernel driver creates a node (node) in the instance management table for management, where the instance identifier is identified as a unique identifier. All operations of methods in the C++ library take the instance identifier as a parameter corresponding to the node of the instance in the kernel driver.
Specifically, an instance in which the audio module and the video module in the target player are used as objects and an instance of an operation method of the objects is created in the program language standard library such as C++ library. Here, the audio module and the video module can be present in a single process or across processes.
At step S3, a reference clock corresponding to the synchronization instance is obtained based on the instance identifier, and data outputs of the audio module and the video module corresponding to the target player are controlled based on the reference clock.
Here, the kernel driver can identify the target player based on the instance identifier, that is, identify players for playing audio and video, or that is, identify an instance that is needed to be managed. In the embodiment, the audio and video synchronization method in the embodiment of the present disclosure can support a variety of synchronization modes, such as video master mode, audio master mode, program clock reference (pcr) master mode, free run and other basic synchronization modes.
In an embodiment of the present disclosure, the reference clock may be a program reference clock for a real-time code stream or a system reference clock. The program reference clock for the real-time code stream can include a display timestamp of audio and video, an audio timestamp, etc. Some real-time code streams include a program reference clock, while some real-time code streams do not include a program reference clock. Therefore, if the real-time code stream includes a program reference clock, the program reference clock for the real-time code stream corresponding to the synchronization instance can be obtained based on the instance identifier. If the real-time code stream does not include a program reference clock, the system reference clock corresponding to the synchronization instance can be obtained.
Specifically, the kernel driver can obtain the program reference clock for the real-time code stream from demux or direct input of the third-party, including the program reference clock corresponding to the code stream of each target player, and identifies the target player based on the instance identifier and identifies the synchronization instance corresponding to the target player, so as to obtain the program reference clock or system reference clock of the real-time code stream corresponding to the synchronization instance and the audio and video timestamps. In the embodiment of the present disclosure, the playback time of the audio and video is related to its timestamp and reference clock. In addition, with reference to the playback time of the audio and video, a synchronization mode is considered, that is, the playback time of the audio and video is adjusted according to different synchronization modes. When programming is performed, the synchronization mode can be set according to the needs. That is, the data output timings of the audio module and the video module corresponding to the target player can be controlled based on the set synchronization mode, so as to cause the audio and video played by each target player to be synchronized.
In some embodiments, for each single-channel playback of multi-channel playback, the audio module and the video module of each single-channel player are controlled according to any of the above embodiments so as to perform simultaneous play, so that the audio and video synchronization effect of each single-channel player can be achieved.
In some other embodiments, the audio module and the video module across processes can also be processed synchronously by using the synchronization instance of the program language standard library such as C++ library and the instance identifier thereof in any of the above embodiments, so as to achieve the purpose of audio and video synchronization.
Therefore, according to the embodiment of the present disclosure, the audio and video synchronization method for the playback module adopts the object-oriented design idea, carries out synchronization processing through the instance and the instance identifier, can support multiple-channel playback and cross-process synchronization scene. The synchronization module adopts the form of program language standard library and kernel driver, therefore any player or module can achieve audio and video synchronization by compiling the program language standard library and calling the related synchronization methods of the program language standard library, thereby achieving strong portability and more flexible usage scenarios.
In some embodiments,
Several synchronization modes used in embodiments of the present disclosure will be explained below. During program design, the synchronization mode can be set according to specific applications and requirements.
In some embodiments, the set synchronization mode includes a clock master mode which can be used in a case in which the real-time code stream includes the program reference clock. In clock master mode, the kernel driver can identify a single-channel player to which the target player belongs based on the instance identifier, so as to obtain the synchronization data of the corresponding synchronization instance. The kernel driver obtains the audio timestamp sent by the audio module in the target player calling a method in the program language standard library such as C++ library, and obtains the program reference information of the real-time code stream of the corresponding synchronization instance of the target player, and controls the audio data output of the audio module based on the audio timestamp and the program reference clock. For example, if the audio timestamp is less than or equal to the time of the program reference clock, the audio module is controlled to output audio, On the contrary, the audio module will not output audio data.
For example, the C++ library is taken as an example of the program language standard library. The kernel driver obtains clock information PCR (program clock reference) of the real-time code stream from the hardware demux. The audio module sends the PTS (Presentation Time Stamp) corresponding to the audio pcm data of the audio module to the C++ library by calling the synchronization method in the C++ library. The C++ library obtains the PCR corresponding to the instance from the kernel driver, determines, based on the PCR, whether the audio pcm (Pulse Code Modulation) data packet is to be output. For example, if the PTS corresponding to the audio pcm data is less than or equal to the PCR of the corresponding instance, the audio pcm data packet is determined to be output. Otherwise, the audio pcm data packet is determined not to be output.
The case of the video playback of the video module is similar as the output of the audio module as above. Specifically, the kernel driver obtains a display timestamp sent by the video module in the target player calling the program language standard library, and obtains the program reference clock for the real-time code stream of the synchronization instance corresponding to the target player. The kernel driver controls video frame output of the video module based on the display timestamp and the program reference clock.
For example, the video module sends PTS corresponding to the video frame data of the video module to the C++ library by calling the synchronization method in the C++ library. The C++ library obtains the PCR of the corresponding instance from the kernel driver, and determines whether the video frame data packet is to be output based on the PCR. For example, if the PTS corresponding to the video frame data is less than or equal to the PCR of the corresponding instance, the video frame data packet is to be output; otherwise, the video frame data packet is not to be output.
In short, in the clock master mode, the audio and video synchronisation can be achieved by taking the program reference clock of the corresponding synchronization instance of the target player as a reference, and making the audio module and the video module to be synchronized with a clock of the corresponding synchronization instance.
In some embodiments, the set synchronization module includes an audio master mode, which can be used when the code stream does not contain a program reference clock.
In the audio master mode, time when data of the audio module is output is used as a reference, and is synchronized with time when data of the video module is output. Specifically, the kernel driver is configured to control the audio module to output at its own sampling rate, identify a single channel player to which the target player belongs based on the instance identifier, obtain an audio timestamp sent by the audio module in the target player calling a method in the program language standard library such as C++ library, determine the audio timestamp as synchronization anchor time of the corresponding synchronization instance of the target player, obtain the system reference clock for the corresponding synchronization instance, and control video frame output of the video module in the target player based on the synchronization anchor time and the system reference clock.
For example, the audio module performs a normal output, i.e., the audio module outputs data according to its own sampling rate. At the same time, by calling a method in the C++ library, the audio module sends PTS currently output by the corresponding audio module to the corresponding instance node of the kernel driver through the C++ library, and determines the PTS as the synchronous anchor time. The video module obtains the synchronization anchor time information of the instance corresponding to the kernel driver through the method of C++ library, decides output time and strategy of the current video frame, so as to realize the synchronous output of the video module and the audio module. For example, system reference clocks corresponding respectively to audio timestamps of two adjacent frames are determined so as to obtain a system reference clock deviation, time when a next audio frame is played (that is, time when the upcoming audio is played) is determined based on the synchronization anchor time and the system reference clock deviation, and the video frame output is controlled at the time point corresponding to the time when the upcoming audio is played, so as to achieve audio and video synchronization output.
In some embodiments, the set synchronization mode includes a video master mode, which can be used when the code stream does not include the program reference clock.
In the video master mode, time when data of the video module is output is used as a reference, and is synchronized with time when data of the audio module is output. Specifically, the kernel driver is configured to control the video module to output at its own frame rate, identify a single channel player to which the target player belongs based on the instance identifier, obtain a display timestamp sent by the video module in the target player calling a method in the program language standard library, determine the display timestamp as synchronization anchor time of the corresponding synchronization instance of the target player, and control audio data output of the audio module in the target player based on the synchronization anchor time and the system reference clock. For example, system reference clocks corresponding respectively to video timestamps of two adjacent frames are determined so as to obtain a system reference clock deviation, time when a next video frame is played (that is, the time when an upcoming video is played) is determined based on the synchronization anchor time and the system reference clock deviation, and the audio frame output is controlled at the time point corresponding to the time when the upcoming video is played to achieve audio and video synchronization output.
For example, the video module performs a normal output, i.e., the video module outputs data according to its own frame rate. At the same time, by calling a method in the C++ library, the video module sends PTS currently output by the corresponding video module to the corresponding instance node of the kernel driver through the C++ library, and determines the PTS as the synchronous anchor time. The audio module obtains the synchronization anchor time information of the instance corresponding to the kernel driver through calling the method of C++ library, so as to decide output time and strategy of pcm data of the current audio frame, so as to realize the synchronous output of the video module and the audio module.
In embodiments of the present disclosure, the audio and video synchronization method for the playback module according to the embodiments of the present disclosure can be used for the audio and video synchronization of at least one of multiplier speed playback, dead slow playback, display frame rate control, frame loss, and drop pcm. In other words, the audio and video synchronization method for the playback module in the embodiments of the present disclosure performs synchronous processing via the synchronization instance and the instance identifier based on the program language standard library such as a C++ library and the kernel driver, and thus it is suitable for any scenario of multiplier speed playback, dead slow playback, display frame rate control, frame loss, and drop pcm, and can achieve the purpose of audio and video synchronization. That is, the audio and video synchronization method of the present disclosure can fully cover the core synchronization mechanism, so that the synchronization module can be flexibly used as an independent module for various products and scenarios.
Based on the audio and video synchronization method of any of the above embodiments, an audio and video synchronization device for the playback module according to an embodiment in a second aspect of the present disclosure will be described below with reference to the attached drawings.
The building module 101 is configured to determine a target player in response to a playback instruction to build an audio module and a video module from the target player. The program language standard library 102 is configured to create a synchronization instance corresponding to the target player, the synchronization instance having a unique instance identifier. The kernel driver 103 is configured to obtain a reference clock corresponding to the synchronization instance based on the instance identifier, and to control data outputs of the audio module and the video module corresponding to the target player based on the reference clock. In some embodiments, the reference clock can be a program reference clock or a system reference clock for a real-time stream.
According to the embodiment of the present disclosure, the audio and video synchronization device 100 for the playback module adopts the object-oriented design idea, carries out synchronization processing through the instance and the instance identifier, can support multiple-channel playback and cross-process synchronization scene. The audio and video synchronization device 100 adopts the form of program language standard library 102 and kernel driver 103, therefore any player or module can achieve audio and video synchronization by compiling the program language standard library 102 and calling the related synchronization methods of the program language standard library 102, thereby achieving strong portability and more flexible usage scenarios.
In some embodiments, the audio module and the video module of the target player belong to different processes, and the program language standard library 102 can use ioctl to interact with the kernel driver 103 to achieve audio and video synchronization across processes.
In some embodiments, the set synchronization mode includes a clock master control mode. The kernel driver 103 is further configured to: identify the target player based on the instance identifier; obtain an audio timestamp sent by the audio module in the target player calling a method in the program language standard library 102 such as the C++ library, and obtain the program reference clock for the real-time code stream of the synchronization instance corresponding to the target player; and control audio data output of the audio module based on the audio timestamp and the program reference clock.
In some embodiments, the kernel driver 103 is further configured to obtain an display timestamp sent by the video module in the target player calling a method in the program language standard library 102 such as the C++ library, and obtain the program reference clock for the real-time code stream of the synchronization instance corresponding to the target player; and control video frame output of the video module based on the display timestamp and the program reference clock.
In some embodiments, the set synchronization mode includes the audio master mode. The kernel driver 103 is configured to: control the audio module to output at its own sampling rate; identify the target player based on the instance identifier; obtain an audio timestamp sent by the audio module in the target player calling a method in the program language standard library 102 such as C++ library, determine the audio timestamp as synchronization anchor time of the corresponding synchronization instance of the target player; and obtain the system reference clock for the corresponding synchronization instance; and control video frame output of the video module in the target player based on the synchronization anchor time and the system reference clock.
In some embodiments, the set synchronization mode includes a video master mode. The kernel driver 103 is configured to control the video module to output at its own frame rate; identify the target player based on the instance identifier; obtain a display timestamp sent by the video module in the target player calling a method in the program language standard library 102 such as C++ library; determine the display timestamp as synchronization anchor time of the corresponding synchronization instance of the target player; and obtain the system reference clock for the corresponding synchronization instance; and control audio data output of the audio module in the target player based on the synchronization anchor time and the system reference clock.
In some embodiments, the audio and video synchronization device 100 includes at least one of the audio and video synchronization mechanisms for multiplier speed playback, dead slow playback, display frame rate control, frame loss, and drop pcm, and can be used as an independent device.
In an embodiment, the electronic device 10 may include a digital television, an interactive network television, or other devices that can adopt the audio and video synchronization method according to any of the above embodiments of the present disclosure.
The memory 12 stores a computer program that can be executed by the processor 11. The processor 11, when executing the computer program, implements the audio and video synchronization method according to any of the above embodiments.
The electronic device 10 according to the embodiment of the present disclosure adopts the processor 11 to implement the audio and video synchronization method for the playback module according to any of the above embodiments, and thus can support the multi-channel playback and cross-process synchronization scene. The synchronous processing adopts the form of the program language standard library such as C++ library and the kernel driver, therefore any player or module can achieve audio and video synchronization by compiling the program language standard library and calling synchronization methods related to the programming language standard library, thereby achieving strong portability and more flexible usage scenarios.
In an embodiment of the fourth aspect of the present disclosure, a computer-readable storage medium on which a computer program is stored is provided. The computer program, when executed by the processor, implements the audio and video synchronization method according to any of the above embodiments. Regarding the audio and video synchronization method, reference may be made to the description of the embodiments in the first aspect above and details are omitted here.
According to the embodiment of the present disclosure, the computer readable storage medium stores the computer program medium, the computer program when executed implements the audio and video synchronization method according to the embodiment in the first aspect of the present disclosure. The method adopts the object-oriented design idea, carries out synchronization processing using the instance and the instance identifier, and thus can support multi-channel playback and cross-process synchronization scene. The synchronization processing adopts the form of program language standard libraries such as C++ libraries and the kernel driver, and any player or module can achieve audio and video synchronization by compiling the program language standard library and calling synchronization methods related to program language standard library, thereby achieving strong portability and more flexible usage scenarios.
A person of ordinary skill in the art may understand that all or part of the process in the method according to any of the above embodiments may be accomplished via instructing the relevant hardware by a computer program. The computer program may be stored in a non-transitory computer readable storage medium. The computer program, when executed, may implement a process of the method in any of the above embodiments. Any reference to a memory, a database, or other medium used in any of the embodiments provided in the disclosure can include at least one of a non-transitory memory and a transitory memory. The Non-transitory memory may include a Read-Only memory (ROM), a magnetic tape, a floppy disk, a flash memory, or an optical memory. The transitory memory can include a Random Access Memory (RAM) or an external cache memory. As an illustration rather than a limitation, RAM is available in various forms, such as a Static Random Access Memory (SRAM) and a Dynamic Random Access Memory (DRAM).
The various technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of individual technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered as falling within the scope of this description.
In the description of the present disclosure, the description with reference to the terms “an embodiment”, “some embodiments”, “illustrative embodiments”, “examples”, “specific examples”, or “some examples” indicates that the specific features, structures, materials, or features described in conjunction with the embodiment(s) or example(s) are included in at least one embodiment or example of the present disclosure. In this description, the schematic representations of the above terms do not necessarily refer to the same embodiment or example.
While embodiments of the present disclosure have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to the embodiments without departing from the principle and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202111677757.0 | Dec 2021 | CN | national |
This application is a continuation of International Application No. PCT/CN2022/115903 entitled “AUDIO AND VIDEO SYNCHRONIZATION METHOD AND DEVICE, ELECTRONIC DEVICE, AND COMPUTER READABLE STORAGE MEDIUM” filed on Aug. 30, 2022, which claims priority to Chinese Patent Application No. 202111677757.0, entitled “AUDIO AND VIDEO SYNCHRONIZATION METHOD AND DEVICE, ELECTRONIC DEVICE, AND COMPUTER READABLE STORAGE MEDIUM” filed with CNIPA on Dec. 31, 2021, all of which are incorporated herein by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2022/115903 | Aug 2022 | WO |
| Child | 18632329 | US |
