Patent Application
20250233969
The present disclosure relates to a signal processing apparatus, a control method for the signal processing apparatus, and a recording medium.
A virtual viewpoint acoustic technique that generates an acoustic signal corresponding to a designated virtual viewpoint using acoustic data obtained by collecting sound through a plurality of microphones has been attracting attention. Japanese Patent Application Laid-open No. 2021-40264 discusses a technique of associating and recording acoustic data and a time code to generate a virtual viewpoint sound.
Further, regarding synchronizing acoustic data and a time code at a time of editing work by sharing the time code between a plurality of apparatuses, Japanese Patent Application Laid-open No. 2012-114779 discusses that, when playing back a main content in which a time code superimposed acoustic signal is recorded together with an image signal, a separately recorded acoustic content is synchronized with a time code, according to a time code obtained by demodulating the time code superimposed on the acoustic signal.
Japanese Patent Application Laid-open No. 2021-40264 does not discuss a method of correcting a shift between a time code and a delay of acoustic data generated when a collected acoustic signal is delayed by a lip synchronization (lip-sync) adjustment or the like. Further, the technique discussed in Japanese Patent Application Laid-open No. 2012-114779 needs to prepare an acoustic track for the time code on a sound collection apparatus.
The present disclosure is directed to a technique for enabling a signal processing apparatus to synchronize time information and an acoustic signal regardless of whether the acoustic signal includes a time code.
According to an aspect of the present disclosure, a signal processing apparatus is provided that includes one or more memories configured to store instructions, and one or more processors configured to execute the instructions to acquire first time information, perform signal processing on an acoustic signal to generate first acoustic data including the acquired first time information, measure processing time for the signal processing based on the acquired first time information and the first acoustic data, acquire second time information indicating a time after measuring the processing time, perform signal processing on the acoustic signal to generate second acoustic data that does not include the first time information and the acquired second time information, correct the second time information based on the processing time, and record the second acoustic data synchronized in time with the corrected second time information.
Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Exemplary embodiments of the present disclosure are described herein with reference to the attached drawings. The exemplary embodiments are not intended to limit the present disclosure, the scope of which is set forth in the appended claims. In the exemplary embodiments, a plurality of features are described, regarding which non-essential features may be combined. In addition, the same or similar components are assigned the same symbols, and redundant descriptions thereof may be omitted.
A baseball game with a sound pickup target is described as an example for exemplary embodiments. For example, game sounds are collected as sound collection targets, which may include collecting a game sound that is generated at a ground level, e.g., a hitting sound and/or a catching sound; collecting a cheering sound that is generated in the stands that are elevated from the ground level; and, for television (TV) broadcasting, collecting voices for a live baseball broadcast and/or comments, with the collected acoustic signals being recorded in synchronization with a time code. However, the present disclosure is not limited to the baseball game, and may be, for example, other games, music concerts, and similar activities.
In a first exemplary embodiment, in a TV broadcast of a baseball game, an example of synchronizing and recording acoustic data based on an acoustic signal acquired from a general baseball broadcasting system, and time information based on a time code will be described.
In the present exemplary embodiment, the signal processing apparatus 100 has a measuring mode and a recording mode as operation modes. In the measuring mode, the signal processing apparatus 100 measures a delay amount (i.e., delay time) caused by the signal processing performed by the signal processing unit 130, and calculates a time code correction value based on the measured delay amount. In the recording mode, the signal processing apparatus 100 corrects the time code based on the time code correction value calculated in the measuring mode. Then, the signal processing apparatus 100 synchronizes acoustic data based on the acoustic signal with time information based on the corrected time code. The signal processing apparatus 100 records time information and the acoustic data. In the baseball game, for example, the signal processing apparatus 100 operates in the measuring mode before the baseball game starts to obtain acoustical characteristics of the baseball stadium, and operates in the recording mode during the baseball game. In this way, the signal processing apparatus 100 can synchronize and record the time information and the acoustic data.
The time code generation unit 110 generates a time code indicating a point in time position, and outputs the generated time code to the signal processing unit 130, the measurement unit 142, and the time code correction unit 143.
The time code generation unit 110 is, for example, a master clock or a device synchronizing with the master clock. In this case, the time code may be, for example, a Linear Timecode or Longitudinal Timecode (LTC) standardized by Society of Motion Picture and Television Engineers (SMPTE). The time code generation unit 110 may be, for example, a time measurement unit. The time code generated by the time code generation unit 110 is an example of first time information.
The sound collection unit 120 collects sound generated by a sound collection target, and outputs the collected sound as an acoustic signal configured of one or more channels to the signal processing unit 130. The sound collection unit 120 may be configured of, for example, a plurality of microphones and may be configured of different types of microphones. In the present exemplary embodiment, the acoustic signal output from the sound collection unit 120 is not provided with an acoustic track for a time code, and thus does not include the time code.
The signal processing unit 130 receives the time code from the time code generation unit 110 and/or the acoustic signal from the sound collection unit 120, and performs signal processing thereon. In the present exemplary embodiment, the time code from the time code generation unit 110 is input to the signal processing unit 130 in the measuring mode, and the acoustic signal is input to the signal processing unit 130 from the sound collection unit 120 in the recording mode. The signal processing unit 130 outputs the input time code or the signal-processed acoustic signal to the measurement unit 142 and the time code correction unit 143. In the present exemplary embodiment, the signal processing unit 130 outputs the input time code with its value unchanged. In the descriptions below, the time code output from the signal processing unit 130 may be referred to as a measuring target time code. The time code (measuring target time code) output from the signal processing unit 130 is an example of second time information.
In the present exemplary embodiment, the signal processing unit 130 is, for example, an audio mixer or a frame synchronizer. The signal processing unit 130 may be a device group obtained by combining a plurality of audio mixers and/or frame synchronizers. The signal processing unit 130 performs various kinds of signal processing on the input signal, as illustrated in the example provided in
In the present exemplary embodiment, in a case where the signal processing unit 130 performs the above-described signal processing on the acoustic signal, the time at which the acoustic signal is output is delayed until after output of the time code from the time code generation unit 110. For example, in a case where the signal processing unit 130 applies the delay processing on the acoustic signal, the acoustic signal output from the signal processing unit 130 delays by the delay processing amount relative to the time code output from the time code generation unit 110.
The synchronization unit 140 corrects the shift between the time code from the time code generation unit 110 and the acoustic signal from the signal processing unit 130 to synchronize the time code and the acoustic signal. In other words, the synchronization unit 140 performs processing so as to be able to record the acoustic signal together with the time code on which the processing time of the signal processing unit 130 is reflected, with the processing time corresponding to the delay amount/delay time. As described above, the synchronization unit 140 includes the control unit 141, the measurement unit 142, and the time code correction unit 143.
The control unit 141 outputs control information based on an input by a user's operation or the like, to control the operation of the signal processing apparatus 100. For example, the control unit 141 controls whether to perform processing (operation in measuring mode) of measuring a delay amount caused by the signal processing performed by the signal processing unit 130, or to perform processing (operation in recording mode) of synchronizing the acoustic signal and the time information based on the time code. For example, the control unit 141 measures the delay amount caused by the signal processing performed by the signal processing unit 130, and outputs to the measurement unit 142 a delay measuring request for causing the measurement unit 142 to calculate a time code correction value. Further, the control unit 141 sets the time code correction value acquired from the measurement unit 142 to the time code correction unit 143.
The measurement unit 142 measures the delay amount caused by the signal processing performed by the signal processing unit 130, and calculates the time code correction value, based on the delay measuring request from the control unit 141. The measurement unit 142 compares the time code acquired from the time code generation unit 110 and the time code (measuring target time code) acquired from the signal processing unit 130 to measure the delay amount caused by the signal processing unit 130. Further, the measurement unit 142 calculates the time code correction value for synchronizing and recording the acoustic signal and the time code from the measurement result, and outputs the calculated time code correction value to the control unit 141.
The time code correction unit 143 synchronizes the time code and the acoustic signal, based on the time code acquired from the time code generation unit 110, the acoustic signal acquired from the signal processing unit 130, and the time code correction value acquired from the control unit 141. The time code correction unit 143 clips the acoustic signal according to the time code. Then, the time code correction unit 143 outputs to the recording unit 150 the synchronized time code as time information, and the clipped acoustic signals as acoustic data. This synchronized time code is an example of third time information. For example, the time code correction unit 143 clips the acoustic signal in a predetermined unit (e.g., predetermined frame unit or predetermined data sample unit) corresponding to the time information. Accordingly, in a case where the acoustic signal is clipped in a predetermined unit, the time information unit is integral multiple of the predetermined unit. The predetermined unit is, for example, a time width of the frame unit or the data sample unit. The recording unit 150 records the time information and the acoustic data input from the time code correction unit 143.
In addition, in the example illustrated in
Further, in the example illustrated in
The CPU 301 controls the signal processing apparatus 100 by using programs (e.g., non-transitory computer readable media) and data stored in the ROM 302 or the RAM 303, to implement functions of the signal processing apparatus 100 illustrated in
The ROM 302 stores various programs, various pieces of data, and related information. For example, the ROM 302 stores a program for the CPU 301 to control the signal processing apparatus 100. The RAM 303 temporarily stores a program and data supplied from the auxiliary storage device 304 or the like, and data and related information supplied from an external apparatus via the communication I/F 308. For example, the auxiliary storage device 304 is configured of a hard disk drive (HDD), a solid state drive (SSD), or the like, and stores various pieces of data, such as an acoustic signal and/or setting information.
For example, the display unit 305 is configured of a liquid crystal display or light emitting diodes (LEDs), and displays a graphical user interface (GUI) for a user to operate the system. For example, the acoustic input unit 306 is configured of a microphone and an analog-to-digital (A/D) converter, and inputs the collected acoustic signal. For example, the operation unit 307 is configured as a keyboard, a mouse, a joy stick, and a touch panel to receive user's operations to input various instructions to the CPU 301. The CPU 301 functions as a display control unit for controlling the display unit 305, and an operation control unit for controlling the operation unit 307.
The communication I/F 308 is used for communications with an external apparatus. For example, in a case where the signal processing apparatus 100 is connected with an external apparatus in a wired manner, a communication cable is connected to the communication I/F 308. Further, for example, in a case where the signal processing apparatus 100 has a function of wirelessly communicating with an external apparatus, the communication I/F 308 is provided with an antenna. The bus 309 connects the units in the signal processing apparatus 100 to transmit information therebetween.
In addition, there is a possibility that a reduction in the signal level of the measuring target time code relative to the time code caused by the signal processing performed by the signal processing unit 130 may occur, and a problem may occur in analyzing the measuring target time code by the measurement unit 142. Thus, a level correction unit for correcting the signal level of the measuring target time code may be provided between the measurement unit 142 and the signal processing unit 130.
With reference to
As described above, assume that the acoustic signal delays by the signal processing (e.g., delay processing) performed by the signal processing unit 130. For example, in a case where a delay amount (DL) corresponds to one frame of the time code, as illustrated in
A value to correct the shift is the time code correction value, and as illustrated in
In addition, the time code correction value need not necessarily be the time code in a single frame unit, and the time code correction value may be, for example, time information in 1 ms unit. In this case, for the shift less than one frame, the time code and the acoustic data can be more accurately synchronized by correcting the acoustic signal in a time direction.
In addition, in the recording mode, since the time code does not need to be input to the signal processing unit 130, the time code may be prevented from being mixed with the acoustic signal to be recorded. More specifically, a switch interlocking with the control information output from the control unit 141 may be provided between the time code generation unit 110 and the signal processing unit 130, to interrupt the input of the time code from the time code generation unit 110 to the signal processing unit 130 in the recording mode. Alternatively, the signal processing unit 130 may be interlocked with the control unit 141, and in the recording mode, the input of the time code may be muted.
According to the present exemplary embodiment, the measurement unit 142 measures the delay amount caused by the signal processing performed by the signal processing unit 130 based on the time code and the measuring target time code, and calculates the time code correction value. Then, the time code correction unit 143 corrects the time code based on the time code correction value calculated by the measurement unit 142, and synchronizes the time information based on the corrected time code and the acoustic signal. In this way, it is possible to correct the shift between the time code and the delay of the acoustic signal caused by the signal processing, and synchronize the time information and the acoustic signal, even if the acoustic signal does not include the time code. For example, it is possible to synchronize and record the time information and the acoustic data without recording the time code on the acoustic track.
In a second exemplary embodiment, a method of synchronizing and recording the time information and the acoustic data of a plurality of acoustic signals different in delay amount will be described. For example, for TV broadcast of a baseball game, the method can be applied to a case where Company A is in charge of a conventional stereo broadcasting sound, and Company B is in charge of a stereophonic sound. In this case, sound collection devices and signal processing apparatuses used by Company A and those used by Company B may be configured of different devices, and may be simultaneously used. In such a case, there may be a difference in delay amount caused by the signal processing or the like between the acoustic signal by Company A and the acoustic signal by Company B, but according to the present exemplary embodiment, the time information and the acoustic data can be synchronized and recorded.
In the signal processing apparatus 600, the sound collection unit 120 includes S number of sound collection units 120-1 to 120-S. An arbitrary sound collection unit may be referred to herein as a sound collection unit 120-i. Further, the signal processing unit 130 includes S number of signal processing units 130-1 to 130-S that respectively correspond to the sound collection units 120-1 to 120-S. An arbitrary signal processing unit may be referred to herein as a signal processing unit 130-i.
The sound collection unit 120-i collects sound (sound and voice) generated from a sound collection target (object) using respective microphones, and outputs to the signal processing unit 130-i an acoustic signal acquired by collecting the sound. The signal processing unit 130-i receives data/signal from the time code generation unit 110 and/or the corresponding sound collection unit 120-i, and performs signal processing on the received acoustic signal. In the present exemplary embodiment, the time code from the time code generation unit 110 is input to the signal processing unit 130-i in the measuring mode, and the acoustic signal is input from the sound collection unit 120-i in the recording mode. Note that, in the measuring mode, the acoustic signal may be additionally input to the signal processing unit 130-i. The signal processing unit 130-i outputs the input time code (measuring target time code) or the signal-processed acoustic signal to the measurement unit 612 and the time code correction unit 613. In addition, in the present exemplary embodiment, the processing that can be performed by the signal processing unit 130-i on the time code and the acoustic signal is similar to the processing that can be performed by the signal processing unit 130 thereon in the first exemplary embodiment. The synchronization unit 610 corrects the delay between the acoustic signals output from the signal processing units 130-i, and the shift between the acoustic signals and the time code from the time code generation unit 110, to synchronize the time code and the acoustic signals. As described above, the synchronization unit 610 includes the control unit 611, the measurement unit 612, the time code correction unit 613, and the delay correction unit 614.
The control unit 611 outputs control information or the like based on an input by a user's operation, to control the operation of the signal processing apparatus 600. For example, the control unit 611 controls whether to perform processing (operation in measuring mode) of measuring a delay amount caused by the signal processing performed by the signal processing unit 130-i, or to perform processing (operation in recording mode) of synchronizing the time information based on the time code and the acoustic signal. Further, the control unit 611 measures, for example, the delay amount caused by the signal processing performed by the signal processing unit 130-i, and outputs to the measurement unit 612 a delay measuring request for causing the measurement unit 612 to calculate the delay correction value and the time code correction value. Further, the control unit 611 sets the time code correction value acquired from the measurement unit 612 to the time code correction unit 613, and outputs the delay correction value acquired from the measurement unit 612 to the delay correction unit 614. In the present exemplary embodiment, the time code correction value is a value for correcting the delay difference between the acoustic signals output from the signal processing units 130-i.
The measurement unit 612 measures a delay amount caused by the signal processing performed by the signal processing unit 130-i based on the delay measuring request from the control unit 611, and calculates a delay correction value and a time code correction value based on the measurement result. The delay correction value and the time code correction value calculated by the measurement unit 612 are output to the control unit 611.
The time code correction unit 613 synchronizes the time code and the acoustic signal based on the time code acquired from the time code generation unit 110, the acoustic signal of which the delay difference between the acoustic signals is corrected by the delay correction unit 614, and the time code correction value acquired from the control unit 611. The time code correction unit 143 clips the acoustic signal according to the time code. The acoustic signal, for example, is clipped at each frame of the time code of the acoustic data. Then, the time code correction unit 613 outputs to the recording unit 150 the synchronized time code as time information and the clipped acoustic signal as acoustic data as illustrated in
The delay correction unit 614 receives the acoustic signals from the signal processing units 130-i and the delay correction values from the control unit 611, corrects the delay amount for each acoustic signal, and outputs to the time code correction unit 613 the acoustic signal of which the delay difference between the acoustic signals is corrected.
In addition, in the example illustrated in
In the example illustrated in
In addition, the hardware configuration of the signal processing apparatus 600 in the second exemplary embodiment is similar to that of the signal processing apparatus 100 in the first exemplary embodiment illustrated in
In the measuring mode, the signal processing apparatus 600 in the present exemplary embodiment calculates the delay correction value for correcting each of the delay amounts between the plurality of the acoustic signals in addition to the time code correction value. Further, in the recording mode, the signal processing apparatus 600 corrects the delay amounts each between the acoustic signals based on the delay correction value, corrects the time code based on the time code correction value, and synchronizes and records the time information and the acoustic data.
More specifically, in the measuring mode, the signal processing apparatus 600 calculates a delay amount, which is a delay amount of the signal processing unit 130-i, and a maximum delay amount, which is the largest delay amount among the delay amounts. Next, the signal processing apparatus 600 calculates a delay correction value for correcting a delay amount of an acoustic signal acquired from the signal processing unit 130-i and the time code correction value based on the delay amount and the delay amount max.
In the recording mode, the signal processing apparatus 600 corrects the time shift between the acoustic signals by correcting the delay amount of the acoustic signal by the delay correction unit 614 based on the delay correction value. Further, the signal processing apparatus 600 corrects the shift of the time code by the time code correction unit 613 based on the time code correction value, and synchronizes and records the time information and the acoustic data.
In
After performing the processing in step S805, the measurement unit 612 starts measuring the delay amount of the signal processing unit 130-i, and the processing in steps S806 to S809 is performed for each of the signal processing units 130-i.
Thus, the measurement unit 612 repeats the processing in steps S806 to S809 by the number of the signal processing units 130, i.e., S times. In this way, the maximum delay amount and the delay amounts 1 to S respectively corresponding to the signal processing units 130-1 to 130-S are obtained. After executing the processing in steps S806 to S809 on all the signal processing units 130-i, and completing all the measurements of the delay amounts of the signal processing units 130-i, the processing in in step S810 is executed.
That is, the measurement unit 612 determines, as a delay correction value, a value obtained by subtracting the delay amount from the maximum delay amount max obtained as described above. Further, the measurement unit 612 determines the maximum delay amount obtained as described above as the time code correction value. Then, the control unit 611 receives inputs of the delay correction value calculated by the measurement unit 612 and the time code correction value, and outputs the received delay correction value to the delay correction unit 614 to set the time code correction value to the time code correction unit 613. After performing the processing in step S810, the processing returns to step S803.
According to the present exemplary embodiment, the signal processing allows for correction of the shift between the time code and the delays of the acoustic signals to synchronize the time information and the acoustic signals, even for a plurality of the acoustic signals having different delay amounts.
The present disclosure can be realized by processing of supplying a program for implementing one or more functions of the above-described exemplary embodiments to a system or an apparatus via a network or a storage medium, and one or more processors of a computer in the system or the apparatus reading and executing the program. Further, the present disclosure can also be realized by a circuit (e.g., an ASIC) that can implement one or more functions.
In addition, all the above-described exemplary embodiments are merely examples of the present disclosure and shall not be construed as limiting the technical range of the present disclosure. Thus, the present disclosure can be realized in diverse ways in accordance with the technological thought and/or features thereof.
Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., an ASIC) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., a CPU, micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a RAM, a ROM, a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray™ Disc), a flash memory device, a memory card, and similar components.
While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2024-004047, filed Jan. 15, 2024, which is hereby incorporated by reference herein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2024-004047 | Jan 2024 | JP | national |
