Patent Application
20210358506
Embodiments of this application relate to the field of audio signal processing, and in particular to an audio signal processing method and apparatus.
With rapid development of high-performance computers and signal processing technologies, people raise increasingly high requirements for voice and audio experience. Immersive audio can meet people's requirements for the voice and audio experience. For example, 4th generation mobile communications technology (the 4th generation mobile communication technology, 4G)/5th generation mobile communications technology (the 5th generation mobile communication technology, 5G) communications voice, audio service, virtual reality (virtual reality, VR), augmented reality (Augmented reality, AR), mixed reality (Mixed Reality, MR), and other video and audio technologies are gaining popularity. An immersive video and audio system requires not only a stunning visual effect but also a realistic auditory effect. Audio-visual fusion can greatly improve immersive experience.
A core of audio is a three-dimensional audio technology. Currently, three-dimensional audio is implemented by using a plurality of reproduction methods, including speaker-based speaker reproduction and headphone-based binaural reproduction. Head-related transfer function (head related transfer function, HRTF) or binaural room impulse response (binaural room impulse response, BRIR) is used to implement headphone-based binaural reproduction. When a sound wave generated by an HRTF reference sound source is propagated to an ear canal, the sound wave is reproduced under impact of scattering, reflection, and refraction of organs such as the head, the body, and an auricle. In a case of BRIR, the sound wave is reproduced based on impact of ambient reflection on the sound source. BRIR can be considered as a pulse response of a system composed of the sound source, indoor environment, and two ears (including the head, the body, and the auricle). The BRIR includes direct sound, early reflection, and late reverberation. Principles of headphone-based binaural reproduction are as follows: After it is assumed that a sound source is at a position, convolution processing is performed based on an HRTF or a BRIR corresponding to a distance from the position to a center position of a listener's head, to obtain output signals to two ears.
In a current three-dimensional audio technology, a corresponding device is selected based on an applying scenario to acquire audio signals corresponding to all wanted sound sources in a sound field, encode all the acquired audio signals, and transmit encoded audio signals to a decoder side. The decoder side decodes the audio signals, and then renders the decoded audio signals based on a playing device selected as a player. However, when there are a large quantity of sound sources in the sound field, power consumption of the decoder side is very high.
Embodiments of this application provide an audio signal processing method and apparatus, to reduce decoding power consumption while ensuring audio signal quality.
To achieve the foregoing objective, the following technical solutions are used in the embodiments of this application.
According to a first aspect of this application, an audio signal processing method is provided and is applied to an encoder side. The method may include: first acquiring at least one audio signal at a listening position, where each of the at least one acquired audio signal has a corresponding sound source, and the at least one acquired audio signal includes at least an audio signal that meets a screening condition; the screening condition may include one or more of the following conditions: volume at the listening position is greater than or equal to a volume threshold, a listening distance between the sound source corresponding to the audio signal and the listening position is less than or equal to a preset listening distance, or the sound source corresponding to the audio signal is in a range corresponding to a listening orientation; encoding the acquired audio signal to obtain an encoded audio signal; and then sending the encoded audio signal to a decoder side.
According to the audio signal processing method provided in this application, at least the audio signal that meets the screening condition at the listening position is acquired, and the acquired audio signal is encoded and sent to the decoder side, so that the decoder side decodes and plays the audio signal. In this way, the encoder side may acquire only the audio signal that meets the screening condition, and encode and send the audio signal, so that the decoder side decodes only the audio signal that meets the screening condition. Therefore, the screening condition for ensuring audio signal quality may be configured. When there are a relatively large quantity of sound sources, transmission of audio signals not only ensures audio signal quality, but also greatly reduces power consumption of the decoder side.
It should be noted that the solutions provided in this application may be applied to a sound field, or certainly may be applied to another applying scenario, for example, a game scenario, a movie scenario, or a conference scenario. The encoder side described in this application refers to a part that is in an audio and video system and that encodes an audio signal corresponding to a sound source. The encoder side acquires and encodes the audio signal corresponding to the sound source, and transmits an encoded audio signal to the decoder side. The decoder side decodes and renders the audio signal, and then plays the audio signal. The audio and video system includes but is not limited to a VR system, an AR system, an MR system, or another streaming transmission system. A type of the audio and video system to which the solutions provided in this application are applied is not specifically limited in this application.
Specifically, audio signal processing includes a real-time audio signal processing scenario and a non-real-time audio signal processing scenario. In either of the scenarios, due to mobility of a user, neither a listening position nor a listening orientation is fixed. The range corresponding to the listening orientation may include a range of all orientations or a range of a preset angle. This is not specifically limited in this application.
In the real-time audio signal processing scenario, a listening position is a position at which a listener is currently located in a sound field, and a listening orientation is an orientation toward which the listener at the listening position faces. The listening position and/or the listening orientation are both provided by the decoder side to the encoder side. The encoder side processes, based on the listening position and/or the listening orientation provided by the decoder side, an audio signal in a synchronous acquisition and transmission manner, to implement realistic restoration of a sound field and to provide immersive experience to the listener. In the non-real-time audio signal processing scenario, a listening position is any possible position in a sound field, and a listening orientation is any possible orientation.
With reference to the first aspect, in a possible implementation, in the real-time audio signal processing scenario, the audio signal processing method provided in this application may further include: receiving listening position information sent by the decoder side, where the listening position information is used to indicate the listening position.
With reference to any one of the first aspect or the possible implementation of the first aspect, in another possible implementation, when the listening orientation needs to be used in the screening condition, for example, when the screening condition includes that a corresponding sound source is in a range corresponding to the listening orientation, in the real-time audio signal processing scenario, the audio signal processing method provided in this application may further include: receiving listening orientation information sent by the decoder side, where the listening orientation information is used to indicate the listening orientation.
With reference to the first aspect or any one of the foregoing possible implementations, in another possible implementation, in the non-real-time audio signal processing scenario, a plurality of listening positions are preset or determined, and there are a plurality of listening orientations at each listening position. The encoder side needs to perform the audio signal processing method provided in this application to acquire and encode an audio signal at each listening position. The decoder side selects, based on an actual position of the listener, a matched encoded audio signal to perform decoding and rendering, to provide immersive experience to the listener.
With reference to the first aspect or any one of the foregoing possible implementations, in another possible implementation, in the non-real-time audio signal processing scenario, the listening orientation may be any listening orientation at any listening position. When the screening condition includes content related to the listening orientation, the encoder side needs to obtain an audio signal corresponding to any listening orientation. The decoder side may obtain, from the encoder side, an audio signal that meets a current screening condition, to restore a sound field. The encoder side obtains a plurality of audio signals in each listening orientation in advance, so that when the listener is in any listening orientation, a decoder may obtain, from the encoder side, an audio signal corresponding to the current listening orientation of the listener, to provide immersive experience to the listener.
With reference to the first aspect or any one of the foregoing possible implementations, in another possible implementation, only an audio signal whose volume exceeds a specific threshold can be heard. The threshold varies with a position relationship between a sound source and a listening orientation. A volume threshold may be configured as a threshold determined based on a position relationship between a sound source corresponding to an audio signal and a listening orientation. In this implementation, different thresholds are configured for different position relationships, so that a listening requirement is more accurately met, and a listening effect is more realistic.
With reference to the first aspect or any one of the foregoing possible implementations, in another possible implementation, only an audio signal whose volume exceeds a specific threshold can be heard. A volume threshold may be configured as a preset volume threshold, to acquire a high-volume sound source signal.
With reference to the first aspect or any one of the foregoing possible implementations, in another possible implementation, the audio signal processing method provided in this application may further include: The encoder side acquires audio signals corresponding to all sound sources at the listening position, encodes all the acquired audio signals to obtain encoded audio signals, sends the encoded audio signals to the decoder side, and sends indication information to the decoder side, where the indication information is used to indicate the audio signal that meets the screening condition and that is in the encoded audio signal. The decoder side decodes, according to the indication information, only the audio signal that meets the screening condition. In this way, the decoder side decodes only the audio signal that corresponds to the sound source at the listening position and that meets the screening condition.
With reference to the first aspect or any one of the foregoing possible implementations, in another possible implementation, the indication information may include characteristic information of each audio signal in the encoded audio signal, where the characteristic information includes volume information or position information of a sound source corresponding to the audio signal. The decoder side selects, based on the characteristic information, the audio signal that meets the screening condition to perform decoding.
With reference to the first aspect or any one of the foregoing possible implementations, in another possible implementation, the indication information may include an identifier of the audio signal that meets the screening condition and that is in the encoded audio signal.
According to a second aspect of this application, an audio signal processing method is provided and is applied to a decoder side. The method may include: receiving a plurality of encoded audio signals, where the plurality of audio signals are audio signals at a listening position, each audio signal has a corresponding sound source, and the plurality of audio signals include at least an audio signal that meets a screening condition; the screening condition includes one or more of the following conditions: volume at the listening position is greater than or equal to a volume threshold, a listening distance between the sound source corresponding to the audio signal and the listening position is less than or equal to a preset listening distance, or the sound source corresponding to the audio signal is in a range corresponding to a listening orientation; decoding the audio signal that meets the screening condition, to obtain a decoded signal; and rendering the decoded signal based on a sound channel transfer function in a playing device to obtain an output signal.
According to the audio signal processing method provided in this application, the decoder side decodes only the audio signal that corresponds to the sound source and that meets the screening condition. When there are a relatively large quantity of sound sources, as long as a screening condition for ensuring audio quality is configured, transmission of audio signals not only ensures audio signal quality, but also greatly reduces power consumption of the decoder side.
It should be noted that the listening position and the listening orientation have been described in detail in the first aspect, and details are not described herein again.
With reference to the second aspect, in a possible implementation, the rendering the decoded signal based on a sound channel transfer function in a playing device to obtain an output signal includes: performing convolution on each decoded signal based on a sound channel transfer function of the decoded signal in the playing device, to obtain one or more output signals.
With reference to any one of the second aspect or the possible implementation of the second aspect, in another possible implementation, in a real-time audio signal processing scenario, the audio signal processing method provided in this application may further include: The decoder side sends listening position information to the encoder side, so that the encoder side obtains, based on the listening position information, the plurality of encoded audio signals, where the listening position information is used to indicate the listening position.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, when the listening orientation needs to be used in the screening condition, for example, when the screening condition includes that a sound source corresponding to the audio signal is in a range corresponding to the listening orientation, in the real-time audio signal processing scenario, the audio signal processing method provided in this application may further include: The decoder side sends listening orientation information to the encoder side, so that the encoder side obtains the plurality of encoded audio signals based on the orientation information, where the listening orientation information is used to indicate the listening orientation.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, in a non-real-time audio signal processing scenario, the listening position is any position in an applying scenario, for example, in a sound field. Therefore, in the non-real-time audio signal processing scenario, a plurality of listening positions need to be preset or determined, and there are a plurality of listening orientations at each listening position. The encoder side needs to perform the audio signal processing method provided in this application to acquire and encode an audio signal at each listening position. The decoder side may obtain a corresponding audio signal from the encoder side based on a current listening position of the listener, perform the audio signal processing method provided in this application to restore a sound field, and obtain an audio signal at each listening position in advance. Therefore, when the listener is at any position, the decoder side may obtain, from the encoder side, an encoded audio signal at the current listening position and decode the audio signal, to provide immersive experience to the listener.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, in the non-real-time audio signal processing scenario, the listening orientation may be any listening orientation at any listening position. When the screening condition includes content related to the listening orientation, the encoder side needs to obtain an encoded audio signal that corresponds to any listening orientation. The decoder side may obtain, from the encoder side, the audio signal that meets the screening condition, to restore the sound field. The encoder side obtains an audio signal in each listening orientation in advance, so that when the listener is in any listening orientation, the decoder side may obtain, from the encoder side, an audio signal corresponding to the current listening orientation, to provide immersive experience to the listener.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, only an audio signal whose volume exceeds a specific threshold can be heard. The threshold varies with a position relationship between a sound source and a listening orientation. A volume threshold may be configured as a threshold determined based on a position relationship between a sound source and a listening orientation. In this implementation, different thresholds are configured for different position relationships, so that a listening requirement is more accurately met, and a listening effect is more realistic.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, only an audio signal whose volume exceeds a specific threshold can be heard. A volume threshold may be configured as a preset volume threshold, to acquire a high-volume sound source signal.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, the plurality of encoded audio signals include only the audio signal that meets the screening condition at the listening position. A specific implementation of decoding, by the decoder side, the audio signal that meets the screening condition, to obtain a decoded signal is: The decoder side decodes each of the plurality of audio signals to obtain the decoded signal.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, the plurality of encoded audio signals include audio signals corresponding to all sound sources at the listening position, and the audio signal processing method provided in this application may further include: The decoder side receives indication information sent by the encoder side, where the indication information is used to indicate a sound source signal that meets the screening condition and that is in the plurality of encoded audio signals; and the decoder side finds, from the plurality of encoded audio signals based on the indication information, the audio signal indicated by the indication information as the audio signal that meets the screening condition.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, the indication information may include characteristic information of each audio signal in the plurality of encoded audio signals, where the characteristic information includes volume information or position information of a sound source corresponding to the audio signal. The decoder side selects, based on the characteristic information, the audio signal that meets the screening condition to perform decoding.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, the indication information may include an identifier of the audio signal that meets the screening condition and that is in the plurality of encoded audio signals. The decoder side finds, from the plurality of encoded audio signals, an audio signal indicated by the identifier as the audio signal that meets the screening condition.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, the sound channel transfer function may include an HRTF function or a BRIR function. A type of the sound channel transfer function is not specifically limited in this application.
With reference to the second aspect or any one of the foregoing possible implementations, in another possible implementation, in a streaming transmission scenario, that the decoder side decodes the audio signal that meets the screening condition, to obtain the decoded signal specifically includes: The decoder side downloads the audio signal that meets the screening condition, and decodes the downloaded audio signal to obtain the decoded signal.
According to a third aspect of this application, an audio signal processing apparatus is provided. The apparatus may include an acquisition unit, an encoding unit, and a sending unit. The acquisition unit is configured to acquire at least one audio signal at a listening position, where each of the at least one acquired audio signal has a corresponding sound source, and the at least one acquired audio signal includes at least an audio signal that meets a screening condition; the screening condition includes one or more of the following conditions: volume at the listening position is greater than or equal to a volume threshold, a listening distance between the sound source corresponding to the audio signal and the listening position is less than or equal to a preset listening distance, or the sound source corresponding to the audio signal is in a range corresponding to a listening orientation; an encoding unit, configured to encode the acquired audio signal to obtain an encoded audio signal; and a sending unit, configured to send the encoded audio signal to a decoder side.
According to the audio signal processing apparatus provided in this application, the audio signal that meets the screening condition at the listening position is acquired, so that the decoder side decodes only the audio signal that meets the screening condition. Therefore, the screening condition for ensuring audio signal quality may be configured. When there are a relatively large quantity of sound sources, transmission of audio signals not only ensures audio signal quality, but also greatly reduces power consumption of the decoder side.
It should be noted that the audio signal processing apparatus provided in the third aspect of this application is configured to implement the audio signal processing method provided in the first aspect or any one of the possible implementations of the first aspect. Therefore, for a specific implementation of the audio signal processing apparatus provided in the third aspect of this application, refer to the first aspect or any one of the possible implementations of the first aspect. Details are not described herein again.
According to a fourth aspect of this application, another audio signal processing apparatus is provided. The apparatus may include a receiving unit, a decoding unit, and a rendering unit. The receiving unit is configured to receive a plurality of encoded audio signals, where the plurality of audio signals are audio signals at a listening position, each audio signal has a corresponding sound source, and the plurality of audio signals include at least an audio signal that meets a screening condition; the screening condition includes one or more of the following conditions: volume at the listening position is greater than or equal to a volume threshold, a listening distance between the sound source corresponding to the audio signal and the listening position is less than or equal to a preset listening distance, or the sound source corresponding to the audio signal is in a range corresponding to a listening orientation; a decoding unit, configured to decode the audio signal that meets the screening condition, to obtain a decoded signal; and a rendering unit, configured to render, based on a sound channel transfer function in a playing device, the decoded signal that is obtained by the decoding unit, to obtain an output signal.
According to the audio signal processing apparatus provided in this application, the decoder side decodes only the audio signal, in a sound field, that corresponds to the sound source and that meets the screening condition. When there are a relatively large quantity of sound sources in the sound field, as long as a screening condition for ensuring audio quality is configured, transmission of audio signals not only ensures audio signal quality, but also greatly reduces power consumption of the decoder side.
It should be noted that the audio signal processing apparatus provided in the fourth aspect of this application is configured to implement the audio signal processing method provided in the second aspect or any one of the possible implementations of the second aspect. Therefore, for a specific implementation of the audio signal processing apparatus provided in the fourth aspect of this application, refer to the second aspect or any one of the possible implementations of the second aspect. Details are not described herein again.
According to a fifth aspect of this application, this application provides an audio signal processing apparatus. The audio signal processing apparatus may implement a function of the encoder side in the foregoing method examples. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or the software includes one or more modules corresponding to the foregoing functions. The audio signal processing apparatus may exist in a form of a chip product.
With reference to the fifth aspect, in a possible implementation, a structure of the audio signal processing apparatus includes a processor and a transceiver. The processor is configured to support the audio signal processing apparatus to perform a corresponding function in the foregoing methods. The transceiver is configured to support communication between the audio signal processing apparatus and another device. The audio signal processing apparatus may further include a memory, and the memory is configured to be coupled to the processor and store program instructions and data that are necessary for the audio signal processing apparatus.
According to a sixth aspect of this application, this application provides another audio signal processing apparatus. The audio signal processing apparatus may implement a function of the decoder side in the foregoing method examples. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or the software includes one or more modules corresponding to the foregoing functions. The audio signal processing apparatus may exist in a form of a chip product.
With reference to the sixth aspect, in a possible implementation, a structure of the audio signal processing apparatus includes a processor and a transceiver. The processor is configured to support the audio signal processing apparatus to perform a corresponding function in the foregoing methods. The transceiver is configured to support communication between the audio signal processing apparatus and another device. The audio signal processing apparatus may further include a memory, and the memory is configured to be coupled to the processor and store program instructions and data that are necessary for the audio signal processing apparatus.
According to a seventh aspect of this application, an encoder is provided. The encoder includes the audio signal processing apparatus that performs the audio signal processing method provided in the first aspect or any one of the possible implementations of the first aspect. The encoder may be used by the encoder side in this embodiment of this application.
According to an eighth aspect of this application, a decoder is provided. The decoder includes the audio signal processing apparatus that performs the audio signal processing method provided in the second aspect or any one of the possible implementations of the second aspect. The decoder may be used by the decoder side in this embodiment of this application.
According to a ninth aspect of this application, an audio signal processing system is provided. The system includes the encoder provided in the seventh aspect and/or the decoder provided in the eighth aspect.
According to a tenth aspect of this application, a computer-readable storage medium including instructions is provided. When the instructions are run on a computer, the computer is enabled to perform the audio signal processing method provided in any one of the foregoing aspects or the possible implementations.
According to an eleventh aspect of this application, a computer program product including instructions is provided. When the computer program product runs on a computer, the computer is enabled to perform the audio signal processing method provided in any one of the foregoing aspects or the possible implementations.
It should be noted that various possible implementations of any one of the foregoing aspects may be combined as long as the solutions are not contradictory.
In the specification, claims, and accompanying drawings of this application, terms such as “first”, “second”, “third”, and “fourth” are intended to distinguish between different objects but do not indicate a particular order.
In the embodiments of this application, the word such as “example” or “for example” is used to give an example, an illustration, or a description. Any embodiment or design scheme described as an “example” or “for example” in the embodiments of this application should not be explained as being more preferred or having more advantages than another embodiment or design scheme. Exactly, use of the word “example”, “example”, or the like is intended to present a relative concept in a specific manner.
One or more of A, B, and C described in the embodiments of this application are used to represent the following concepts: A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.
Before the solutions provided in this application are described, a conventional audio signal processing process is first described.
The conventional audio signal processing process includes: An encoder device selects, based on an applying scenario, a corresponding device to acquire audio signals corresponding to all sound sources in a current sound field, encodes all the acquired audio signals, and transmits encoded audio signals to a decoder side. The decoder side decodes and renders all the received audio signals to obtain output signals. If a playing device is a speaker array, the decoder renders the output signals as a multi-channel speaker signal. If the playing device is a headphone device, the decoder renders the output signals as a binaural output signal.
Encoding and decoding may be performed using coding manners, such as currently common enhanced voice services (enhanced voice services, EVS), adaptive multi-rate wideband (adaptive multi-rate wideband, AMRWB), and adaptive multi-rate (Adaptive Multi-Rate, AMR). The multi-channel speaker signal is obtained by rendering by a renderer based on a specific quantity of speakers. The binaural output signal is obtained by the renderer by using the multi-channel speaker signal and HRTF/BRIR rendering. It should be noted that the foregoing coding manners used for encoding and decoding are merely an example, and another audio coding manner may also be used in an actual application.
An EVS codec is used as an example. Audio signals corresponding to the five sound sources are respectively denoted as S1(n), S2(n), S3(n), S4(n), and S5(n), which are respectively EVS-encoded into five bitstream signals S1S(n), S2S(n), S3S(n), S4S(n), and S5S(n). The five bitstream signals are connected in a frame-to-frame adjacency manner, and are sent after channel coding.
After receiving the encoded bitstream signals, a decoder first performs channel decoding to obtain the five bitstreams S1S(n), S2S(n), S3S(n), S4S(n), and S5S(n). An EVS decoder decodes the bitstreams to obtain five signals DS1(n), DS2(n), DS3(n), DS4(n), and DS5(n).
Convolution processing is separately performed on the five signals based on respective corresponding HRTF signals HRTF1(n) HRTF2(n) HRTF3(n); HRTF3(n); and HRTF4(n)) in sound channel transfer functions (HRTF functions) selected by a listening apparatus, to obtain convolution results Y1(n), Y2(n), Y3(n), Y4(n), and Y5(n). A specific convolution is denoted as:
Y1(n)=DS1(n)⊗HRTF1(n);
Y2(n)=DS2(n)⊗HRTF2(n);
Y3(n)=DS3(n)⊗HRTF3(n);
Y4(n)=DS4(n)⊗HRTF4(n);
Y5(n)=DS5(n)⊗HRTF5(n)
Finally, the convolution results are superimposed to obtain a final binaural (binaural) output signal Y(n). A calculation expression of Y(n) may be:
Y(n)=Y1(n)+Y2(n)+Y3(n)+Y4(n)+Y5(n)
In the foregoing audio signal processing process, the encoder side acquires, encodes, and transmits all the audio signals corresponding to the sound sources to the decoder side, and the decoder side decodes and renders all the received audio signals. When there are a relatively large quantity of sound source signals, power consumption of the decoder side is very high.
To resolve the foregoing problem, an embodiment of this application provides an audio signal processing method. Basic principles of the method are as follows: The decoder side selects an audio signal that meets a screening condition to perform decoding. This method replaces the conventional-technology method in which the decoder side decodes audio signals corresponding to all sound sources. When there are a large quantity of sound sources, power consumption of the decoder side is reduced. In addition, as long as the screening condition is properly configured, quality of audio signal transmission can still be well ensured while reducing the power consumption, to ensure a listening effect.
Before the solutions of this application are described, terms in this application are first explained.
A position of a listener (position of the listener) is a position at which the listener (listener) is located. A listening orientation is an orientation toward which the listener at the listening position faces. The listener may be a user who actually uses a playing device, or may be a virtual user who uses the playing device. The virtual user who uses the playing device may be at any possible listening position, and may be in any possible listening orientation.
A listening distance is a straight-line distance between a sound source and the listening position. Alternatively, the listening distance may be a distance of a preset path between the sound source and the listening position. The preset path may be a distance of bypassing an obstacle or another distance. This is not specifically limited in this application.
The following describes implementations of the embodiments of this application in detail with reference to the accompanying drawings.
The audio signal processing method provided in this application is applied to an audio and video system 10 shown in
As shown in
As shown in
It should be noted that some modules are common modules for the end-to-end audio signal processing part and the end-to-end video signal processing part included in the audio and video system 10.
The following briefly describes functions of the modules in the audio and video system 10 shown in
The acquisition module 201 is configured to acquire an audio signal corresponding to a sound source, and transmit the audio signal to the audio preprocessing module 202.
The audio preprocessing module 202 is configured to preprocess the audio signal, for example, perform filtering processing, and transmit the preprocessed audio signal to the audio encoding module 203.
The audio encoding module 203 is configured to encode the preprocessed audio signal, and transmit an encoded audio signal to the encapsulation module 204.
The acquisition module 201 is further configured to acquire a video signal. After the video combination module 210, the prediction and mapping module 211, the video encoding module 212, and the image encoding module 213 process the video signal, the encoded video signal is transmitted to the encapsulation module 204.
The encapsulation module 204 is configured to encapsulate the encoded audio signal and the encoded video signal to obtain a bitstream. The bitstream is transmitted through the delivery module 205 to the delivery module 218 on the decoder side, and then is transmitted by the delivery module 218 to the decapsulation module 206. For example, the delivery module 205 and the delivery module 218 may be wired communications modules or wireless communications modules. This is not specifically limited in this embodiment of this application.
It should be noted that, when the audio and video system 10 is a streaming transmission system, the delivery module 205 may be specifically implemented in a form of a server. To be specific, the encoder side uploads the bitstream to the server, and the decoder side downloads the bitstream from the server according to a requirement, to implement a function of the delivery module 205. Details of this process are not described again.
The decapsulation module 206 is configured to decapsulate the bitstream to obtain the encoded audio signal and the encoded video signal, transmit the encoded audio signal to the audio decoding module 207, and transmit the encoded video signal to the video decoding module 214 and the image decoding module 215.
The audio decoding module 207 is configured to decode the encoded audio signal, and transmit a decoded audio signal to the audio rendering module 208.
The audio rendering module 208 is configured to perform rendering processing on the decoded audio signal, and transmit a rendered output signal to the speaker/headphone 209.
The video decoding module 214, the image decoding module 215, and the video rendering module 216 process the encoded video signal, and transmit a processed video signal to the player 217 for playing.
It should be noted that a structure shown in
It should be further noted that the end-to-end video signal processing part and the end-to-end audio signal processing system in the audio and video system 10 shown in
The following describes the technical solutions of this application in detail with reference to the accompanying drawings.
According to one aspect, this application provides an audio signal processing apparatus.
As shown in
The following describes components of the audio signal processing apparatus 30 in detail with reference to
The memory 302 may be a volatile memory (volatile memory), for example, a RAM, or may be a non-volatile memory, for example, a read-only memory (read-only memory, ROM), a flash memory (flash memory), an HDD, an SSD, or a combination of the foregoing types of memories. The memory 302 is configured to store program code and a configuration file that can implement the method in this application.
As a control center of the audio signal processing apparatus 30, the processor 301 may be a CPU or an ASIC, or may be configured as one or more integrated circuits for implementing this embodiment of this application, for example, one or more DSPs or one or more FPGAs. The processor 301 may run or execute software programs and/or modules stored in the memory 302 and invoke data stored in the memory 302, to perform various functions of the audio signal processing apparatus 30.
The transceiver 303 is configured to implement interaction between the audio signal processing apparatus 30 and another unit. For example, the transceiver 303 may be a transceiver port of the audio signal processing apparatus 30.
Further, the audio signal processing apparatus 30 may further include a sensor 304, configured to obtain a listening position and/or a listening orientation. For example, the sensor 304 may be a gyroscope, an external camera, a motion detection apparatus, an image detection apparatus, or the like. This is not limited in this embodiment of this application.
Specifically, when the audio signal processing apparatus 30 is deployed on the encoder side, or when the audio signal processing apparatus 30 is the encoder side, the processor 301 performs the following functions by running or executing a software program and/or a module stored in the memory 302 and invoking data stored in the memory 302:
acquiring at least one audio signal at a listening position, where each of the at least one acquired audio signal has a corresponding sound source, and the at least one acquired audio signal includes at least an audio signal that meets a screening condition; the screening condition includes one or more of the following conditions: volume at the listening position is greater than or equal to a volume threshold, a listening distance between the sound source corresponding to the audio signal and the listening position is less than or equal to a preset listening distance, or the sound source corresponding to the audio signal is in a range corresponding to a listening orientation; encoding the acquired audio signal to obtain an encoded audio signal; and sending the encoded audio signal to the decoder side by using the transceiver 303.
The method, apparatus, device, or system provided in the embodiment of this application may be applied to a real-time audio signal processing scenario and a non-real-time audio signal processing scenario. When the audio signal processing apparatus 30 is applied to the real-time audio signal processing scenario, the listening position is on the decoder side, that is, a current position of a listener (listener). Specifically, the current position of the listener (listener) may be in range of a game scene, a sound field, a movie scene, a conference scenario, or another applying scenario. To better reproduce an audio of the encoder side for the listener on the decoder side, the encoder side may acquire an audio signal based on the current position of the listener (that is, the listening position) sent by the decoder side, so that the audio signal acquired by the encoder side can match the current position of the listener. When the method is applied to the non-real-time audio signal processing scenario, the listening position is a preset position. Because the listener may be at any position within the range of the applying scenario, some listening positions may be preset, and the encoder side may acquire an audio signal based on the preset listening position. In this case, when performing decoding processing on the audio signal, the decoder side may select, based on the current position of the listener, an audio signal that best matches (for example, a same position or a closest position) to perform decoding processing.
Specifically, when the audio signal processing apparatus 30 is deployed on the decoder side, or when the audio signal processing apparatus 30 is the decoder, the processor 301 performs the following functions by running or executing a software program and/or a module stored in the memory 302 and invoking data stored in the memory 302:
receiving, by using the transceiver 303, a plurality of encoded audio signals, where the plurality of audio signals are audio signals at the listening position, each audio signal has a corresponding sound source, and the plurality of audio signals include at least an audio signal that meets a screening condition, where the screening condition may include one or more of the following conditions: volume at the listening position is greater than or equal to a volume threshold, a listening distance between the sound source corresponding to the audio signal and the listening position is less than or equal to a preset listening distance, or the sound source corresponding to the audio signal is in a range corresponding to a listening orientation; decoding an audio signal that meets the screening condition, to obtain a decoded signal; and rendering the decoded signal based on a sound channel transfer function in a playing device, to obtain an output signal.
According to another aspect, an embodiment of this application provides an audio signal processing method. This method is applied to an interaction process between the encoder side and the decoder side of the end-to-end audio signal processing part in the audio and video system 10 shown in
It should be noted that, when the encoder side and the decoder side described herein interact with each other, they may directly communicate and interact with each other, or may interact with each other by uploading interaction content to a shared medium and downloading the interaction content from the shared medium. This is collectively referred to as sending/receiving in this application, and details are not described subsequently. The shared medium may be a server, a cloud server, or the like.
As shown in
S401: The encoder side obtains a listening position.
The audio signal processing includes a real-time audio signal processing scenario and a non-real-time audio signal processing scenario. In any scenario, the listening position is not fixed due to mobility of a user.
The following describes implementation of S401 in the real-time audio signal processing scenario and the non-real-time audio signal processing scenario.
1. Real-Time Audio Signal Processing Scenario
In the real-time audio signal processing scenario, a listening position is a position at which a listener is currently located in a sound field. A listening orientation is an orientation toward which the listener at the listening position faces. The listening position may be provided by the decoder side to the encoder side.
Therefore, before S401, as shown in
S401a: The decoder side obtains the listening position, and sends listening position information to the encoder side.
Specifically, the decoder side may obtain, by using a sensor or an apparatus for obtaining a position, the position at which the listener is currently located in the sound field. An implementation of obtaining the listening position by the decoder is not specifically limited in this embodiment of this application.
The listening position information is used to indicate the listening position obtained by the decoder side, and content of the listening position information may be configured based on an actual requirement. This is not limited in this embodiment of this application.
For example, the listening position information may be coordinate information or a position identifier. An identifier of each position in the sound field is preset. A specific listening position may be uniquely determined by using the identifier.
Corresponding to S401a, S401 may be specifically implemented as follows: The encoder side receives the listening position information from the decoder side, where the listening position information is used to indicate the listening position; and the encoder determines, based on the listening position information, the position at which the listener is located in the sound field.
In the real-time audio signal processing scenario, the decoder side may provide one or more listening positions for the encoder side. This is not specifically limited in this embodiment of this application.
2. Non-Real-Time Audio Signal Processing Scenario
In the non-real-time audio signal processing scenario, the listening position may be any position in the sound field. Therefore, in the non-real-time audio signal processing scenario, the encoder side needs to acquire audio signals at a plurality of possible listening positions in advance. In this case, the listening position may be any one of the plurality of preset positions.
Therefore, in the non-real-time audio signal processing scenario, S401 may be specifically implemented as follows: The encoder side determines each possible listening position in the sound field. Specifically, each possible listening position may be determined based on the plurality of preset positions, or each possible listening position may be determined by using a preset algorithm based on a current processing scenario.
Optionally, in the audio signal processing method provided in this embodiment of this application, only the listening position may be considered. In this case, S403 is performed after S401. Alternatively, according to the audio signal processing method provided in this embodiment of this application, the listening orientation may be referenced based on the listening position. In this case, S402 needs to be first performed after S401, and then S403 is performed.
S402: The encoder side obtains the listening orientation.
The listening orientation is the orientation toward which the user at the listening position faces. The listening orientation is not fixed due to mobility of the user.
The following describes implementation of S402 in the real-time audio signal processing scenario and the non-real-time audio signal processing scenario.
In the real-time audio signal processing scenario, the listening orientation is provided by the decoder side to the encoder side. Therefore, before S402, as shown in
S402a: The decoder side obtains the listening orientation, and sends listening orientation information to the encoder side.
Specifically, the decoder may obtain, by using the sensor or the apparatus for obtaining a position, an orientation toward which the listener at the current position in the sound field faces, and use the orientation as the listening orientation. An implementation of obtaining the listening orientation by the decoder side is not specifically limited in this embodiment of this application.
The listening orientation information is used to indicate the listening orientation obtained by the decoder side, and content of the listening orientation information may be configured based on an actual requirement. This is not limited in this embodiment of this application.
For example, the listening orientation information may be an orientation identifier, or the listening orientation information may be an included angle between the listening orientation and a reference orientation.
Corresponding to S402a, S402 may be specifically implemented as follows: The encoder side receives the listening orientation information from the decoder side, and the encoder side determines the listening orientation of the listener in the sound field based on the listening orientation information.
It should be noted that in the real-time audio signal processing scenario, S401 and S402 may be combined for implementation. This process is not specifically limited in this application.
In the non-real-time audio signal processing scenario, the listening orientation is each possible orientation of each listening position. Therefore, in the non-real-time audio signal processing scenario, there are a plurality of listening positions, and there are a plurality of listening orientations at each listening position.
Therefore, in the non-real-time signal processing scenario, S402 may be specifically implemented as follows: The encoder side determines each listening orientation at each possible listening position in the sound field.
S403: The encoder side acquires at least one audio signal at the listening position in the sound field, where the at least one acquired audio signal includes at least an audio signal that meets a screening condition.
In S403, each acquired audio signal has a corresponding sound source in the sound field.
Specifically, in S403, the encoder side acquires only the audio signal that meets the screening condition at the listening position in the sound field.
It should be noted that when there are a plurality of listening positions, the encoder side performs S403 at each listening position. A process of performing S403 at only one listening position is described herein, but this shall not be construed as a limitation herein.
Specifically, the encoder side performs S403 by using at least one audio signal acquisition apparatus. The audio signal acquisition apparatus may be a microphone or another apparatus. This is not specifically limited in this embodiment of this application.
The screening condition may include one or more of the following conditions: volume at the listening position is greater than or equal to a volume threshold, a listening distance between the sound source and the listening position is less than or equal to a preset listening distance, or the sound source is in a range corresponding to a listening orientation. Certainly, content of the screening condition may be alternatively configured based on an actual requirement. This is not specifically limited in this embodiment of this application.
In a possible implementation, only an audio signal whose volume exceeds a specific threshold can be heard. The threshold varies with a position relationship between a sound source and a listening orientation. A volume threshold may be configured as a threshold determined based on a position relationship between a sound source and a listening orientation.
In a possible implementation, volume thresholds corresponding to different position relationships may be preconfigured. The volume threshold is determined based on the position relationship in S403.
The position relationship may be at least one of a relative orientation relationship or a relative distance relationship between a sound source and a listening orientation. For example, the sound source is located in the front or rear of the listening orientation, and the sound source is located on the left or right side of the listening orientation. Alternatively, the position relationship may be an angle relationship between a sound source and a listening orientation. For example, a sound source is located 50 degrees to the right of a listening orientation, and a sound source is located 0 degrees to the right (straight ahead) of a listening orientation.
For example, Table 1 shows a preset correspondence between a position relationship and a volume threshold. In an actual application, a volume threshold corresponding to a position relationship is obtained through lookup. For example, when a sound source is located 30 degrees to the right of a listening orientation, and a straight-line distance between the sound source and a listening position is 0.35 meter, it is determined, by looking up Table 1, that a corresponding volume threshold is a threshold 20.
It should be noted that Table 1 describes, by way of example, and not limitation, a manner of determining the volume threshold.
In another possible implementation, a preset position relationship algorithm may be preconfigured for the volume threshold, and then the volume threshold is obtained according to the preset algorithm and based on the position relationship between the sound source and the listening position.
Certainly, for a specific implementation of obtaining volume thresholds corresponding to different position relationships, an implementation solution may be selected based on an actual requirement. This is not specifically limited in this embodiment of this application.
In another possible implementation, only a sound source whose volume exceeds a specific threshold can be heard. A volume threshold may be configured as a preset volume threshold, to acquire a high-volume audio signal. Impact of a listening orientation on a listening effect is not considered in this possible implementation.
It may be understood that the volume threshold may also be set by the listener and sent by the decoder side to the encoder side. Likewise, the listening distance may also be set by the listener and sent by the decoder side to the encoder side.
It should be noted that a specific value of the volume threshold may be configured based on an actual requirement. This is not specifically limited in this embodiment of this application.
The range corresponding to the listening orientation may include a range of all orientations or a range of a preset angle. This is not specifically limited in this application. A specific value of the preset angle may be configured based on an actual requirement. This is not specifically limited in this embodiment of this application.
S404: The encoder encodes the acquired audio signal to obtain an encoded audio signal.
It should be noted that, for an encoding process, refer to encoding in the audio signal processing process shown in
S405: The encoder sends the encoded audio signal to the decoder side.
Specifically, the encoder side sends the encoded audio signal to the decoder side through the delivery module 205 in the audio and video system 10 shown in
S406: The decoder side receives the plurality of encoded audio signals.
It should be noted that the plurality of encoded audio signals received by the decoder side in S406 are the encoded audio signals sent by the encoder side in S405.
The plurality of encoded audio signals include only the audio signal that meets the screening condition at the listening position in the sound field. The listening position herein may be the listening position provided by the decoder side in the real-time audio signal processing scenario, or may include each listening position in the sound field in the non-real-time audio signal processing scenario.
S407: The decoder side decodes the audio signal that meets the screening condition to obtain a decoded signal.
Specifically, because the plurality of encoded audio signals received in S406 include only the sound source signal that meets the screening condition, the decoder side decodes each received audio signal to obtain a decoded signal in S407. For a decoding process, refer to decoding in the audio signal processing process shown in
S408: The decoder side renders the decoded signal based on a sound channel transfer function in a playing device to obtain an output signal.
Specifically, in S408, the decoder side performs convolution on each decoded signal based on a sound channel transfer function of the decoded signal in the playing device, to obtain one or more output signals.
Specifically, the decoded signal is rendered in S408. In the audio signal processing process shown in
Optionally, the sound channel transfer function may include an HRTF function or a BRIR function. Certainly, a type of the sound channel transfer function may be configured based on an actual requirement. This is not specifically limited in this embodiment of this application.
After S408, the decoder side may play the output signals by using the playing device. Certainly, the playing device may be disposed inside or outside the decoder side. This is not specifically limited in this embodiment of this application.
According to the audio signal processing method provided in this application, the encoder side acquires only the audio signal that meets the screening condition in the sound field, and then encodes and sends the audio signal to the decoder side. The decoder side decodes the received audio signal that meets the screening condition. Therefore, the screening condition for ensuring audio signal quality is configured to ensure audio quality of the audio signals acquired by the encoder side, and the decoder side does not need to decode all the audio signals. When there are a relatively large quantity of sound sources in the sound field, compared with the conventional technology in which all sound source signals are decoded, this method greatly reduces power consumption of the decoder side.
According to still another aspect, an embodiment of this application provides another audio signal processing method. This method is applied to an interaction process between the encoder side and the decoder side of the end-to-end audio signal processing part in the audio and video system 10 shown in
As shown in
S501: The encoder side obtains a listening position.
Specifically, S501 is the same as S401. For a specific implementation, refer to S401. Details are not described herein again.
Optionally, in the audio signal processing method provided in this embodiment of this application, only the listening position may be considered. In this case, S503 is performed after S501. Alternatively, according to the audio signal processing method provided in this embodiment of this application, a listening orientation may be referenced based on the listening position. In this case, S502 needs to be first performed after S501, and then S503 is performed.
S502: The encoder side obtains the listening orientation.
Specifically, S502 is the same as S402. For a specific implementation, refer to S402. Details are not described herein again.
S503: The encoder side acquires audio signals corresponding to all sound sources at the listening position in the sound field.
It should be noted that when there are a plurality of listening positions, the encoder performs S503 at each listening position. A process of performing S503 at only one listening position is described, but this shall not be construed as a limitation herein.
Specifically, the encoder side performs S503 by using at least one audio signal acquisition apparatus. The audio signal acquisition apparatus may be a microphone or another apparatus. This is not specifically limited in this embodiment of this application.
S504: The encoder encodes the acquired audio signals to obtain encoded audio signals.
It should be noted that, for an encoding process, refer to encoding in the audio signal processing process shown in
S505: The encoder determines indication information used to indicate an audio signal that meets a screening condition and that is in the encoded audio signal.
It should be noted that the screening condition has been described in detail in S403, and details are not described herein again.
In a possible implementation, the indication information may include characteristic information of each audio signal in the encoded audio signals, and the characteristic information includes volume information or position information of a sound source corresponding to the audio signal.
Specifically, the encoder side may acquire the characteristic information of the audio signal by using a sensor or another apparatus. A specific implementation is not limited in this application.
The volume information is used to indicate volume of the sound source. The volume information may be a specific volume value, may be a volume level, or may certainly be other content. Any volume information indicating the volume of the sound source falls inside the scope of the volume information described herein.
The position information of the sound source corresponding to the audio signal is used to indicate a specific position of the sound source in the sound field. The position information may be coordinates, may be a position identifier, or may certainly be other content. Any position information indicating the specific position of the sound source in the sound field falls inside the scope of the position information described herein.
In another possible implementation, the indication information may include an identifier of the audio signal that meets the screening condition and that is in the encoded audio signal.
The identifier of the audio signal is used to uniquely indicate the audio signal. A type of the identifier of the audio signal is not specifically limited in this application. For example, the identifier of the audio signal may be a position or a sequence of the audio signal in the encoded audio signals.
S506: The encoder sends the encoded audio signals to the decoder side.
Specifically, the encoder side sends the encoded audio signals to the decoder side through the delivery module 205 in the audio and video system 10 shown in
S507: The encoder sends the indication information to the decoder side.
It should be noted that S506 and S507 may be performed together, or may be performed separately. This is not specifically limited in this application. When S506 and S507 are performed separately, they are not subject to a specific sequence.
S508: The decoder side receives the plurality of encoded audio signals.
It should be noted that the plurality of encoded audio signals received by the decoder side in S508 are the encoded audio signals sent by the encoder side in S506.
The plurality of encoded audio signals received in S508 not only include the audio signal that meets the screening condition at the listening position in the sound field, but also include another audio signal that does not meet the screening condition.
S509: The decoder side receives the indication information sent by the encoder side.
It should be noted that S508 and S509 may be performed together, or may be performed separately. This is not specifically limited in this application. When S508 and S509 are performed separately, they are not subject to a specific sequence.
S510: The decoder side decodes the audio signal that meets the screening condition to obtain the decoded signal.
In S510, the decoder side performs screening, based on the received indication information, on the plurality of encoded audio signals received in S508, to obtain the audio signal that meets the screening condition at the listening position, and performs decoding to obtain the decoded signal.
In a possible implementation, when the indication information is the characteristic information of each of the plurality of audio signals, in S510, the encoder side determines, based on the characteristic information, whether each audio signal meets the screening condition, and finds the audio signal that meets the screening condition.
In another possible implementation, when the indication information is the identifier of the audio signal that meets the screening condition, in S510, the encoder side performs screening and finds an audio signal indicated by the indication information, and uses the audio signal as the audio signal that meets the screening condition.
S511: The decoder side renders the decoded signal based on a sound channel transfer function in a playing device to obtain an output signal.
Specifically, S511 is the same as S408. For a specific implementation, refer to S408. Details are not described herein again.
According to the audio signal processing method provided in this application, the encoder side acquires the audio signals corresponding to all the sound sources, encodes the audio signals, and sends the encoded audio signals to the decoder side. At the same time, the encoder side indicates, to the decoder side by using the indication information, the audio signal that meets the screening condition. The decoder side selects, according to the indication information provided by the encoder side, the audio signal that meets the screening condition to perform decoding. When there are a relatively large quantity of sound sources, as long as a screening condition for ensuring audio signal quality is configured, transmission of audio signals not only ensures audio signal quality, but also greatly reduces power consumption of the decoder side.
It should be noted that an execution sequence of the steps included in the audio signal processing method provided in this embodiment of this application may be configured based on a requirement. The execution sequence in the accompanying drawings is merely an example, but this shall not be construed as a limitation herein.
The foregoing mainly describes the solutions provided in the embodiments of this application from a perspective of interaction between network elements. It may be understood that to implement the foregoing functions, the decoder side and the encoder side include corresponding hardware structures and/or software modules for performing the functions. A functional unit that implements the foregoing audio signal processing method on the decoder side and the encoder side is referred to as an audio signal processing apparatus. Persons skilled in the art should easily be aware that, in combination with units and algorithm steps of the examples described in the embodiments disclosed in this specification, this application may be implemented by hardware or a combination of hardware and computer software. Whether a function is performed by hardware or hardware driven by computer software depends on a particular application and a design constraint of the technical solutions. Persons skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.
In the embodiments of this application, the audio signal processing apparatus may be divided into functional modules based on the foregoing method examples. For example, functional modules may be obtained through division based on corresponding functions, or two or more functions may be integrated into one processing module. The module into which the functions are integrated may be implemented in a form of hardware, or may be implemented in a form of a software functional module. It should be noted that, in the embodiments of this application, division into modules is an example, and is merely a logical function division. In an actual implementation, another division manner may be used.
When the functional modules are obtained through division based on the corresponding functions,
Further, as shown in
When the unit into which the functions are integrated is used,
The processing module 701 may be the processor 301 in a physical structure of the audio signal processing apparatus 30 shown in
When the processing module 701 is the processor, the communications module 702 is the transceiver, and the storage module 703 is the memory, the audio signal processing apparatus 70 shown in
As described above, the audio signal processing apparatus 60 or the audio signal processing apparatus 70 provided in the embodiments of this application may be configured to implement the functions of the encoder side in the methods implemented in the foregoing embodiments of this application. For ease of description, only a part related to this embodiment of this application is shown. For specific technical details that are not disclosed, refer to the embodiments of this application.
When the functional modules are obtained through division based on the corresponding functions,
Further, as shown in
When the unit into which the functions are integrated is used,
The processing module 901 may be the processor 301 in a physical structure of the audio signal processing apparatus 30 shown in
When the processing module 901 is the processor, the communications module 902 is the transceiver, and the storage module 903 is the memory, the audio signal processing apparatus 90 shown in
As described above, the audio signal processing apparatus 80 or the audio signal processing apparatus 90 provided in the embodiments of this application may be configured to implement the functions of the decoder side in the methods implemented in the foregoing embodiments of this application. For ease of description, only a part related to this embodiment of this application is shown. For specific technical details that are not disclosed, refer to the embodiments of this application.
In another implementation of this embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed, the audio signal processing method in the foregoing method embodiments is performed.
In another implementation of this embodiment, a computer program product including instructions is provided. When the instructions are executed, the audio signal processing method in the foregoing method embodiments is performed.
It may be clearly understood by persons skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiments. Details are not described herein again.
Persons skilled in the art should be aware that in the foregoing one or more examples, functions described in this application may be implemented by hardware, software, firmware, or any combination thereof. When the functions are implemented by software, the foregoing functions may be stored in a computer-readable medium or transmitted as one or more instructions or code in a computer-readable medium. The computer-readable medium includes a computer storage medium and a communications medium, where the communications medium includes any medium that facilitates transmission of a computer program from one place to another. The storage medium may be any available medium accessible by a general purpose or dedicated computer. It may be clearly understood by persons skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiments, and details are not described herein again.
In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, division into units is merely logical function division and may be other division in an actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communications connections may be implemented through some interfaces. The indirect couplings or communications connections between the apparatuses or units may be implemented in electrical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.
In addition, function units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in a form of hardware, or may be implemented in a form of hardware plus a software functional unit.
When the foregoing integrated unit is implemented in a form of a software function unit, the integrated unit may be stored in a computer-readable storage medium. The software function unit is stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk, or an optical disc.
Finally, it should be noted that the foregoing embodiments are merely intended to describe the technical solutions of this application, but not to limit this application. Although this application is described in detail with reference to the foregoing embodiments, persons of ordinary skill in the art should understand that they may still make modifications to the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features thereof, without departing from the scope of the technical solutions of the embodiments of this application.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201910103595.6 | Jan 2019 | CN | national |
This application is a continuation of International Application No. PCT/CN2019/128908, filed on Dec. 26, 2019, which claims priority to Chinese Patent Application No. 201910103595.6, filed on Jan. 31, 2019. The disclosures of the aforementioned applications are hereby incorporated by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2019/128908 | Dec 2019 | US |
| Child | 17386800 | US |
