Patent Grant
11074921
This application is a U.S. National Phase of International Patent Application No. PCT/JP2018/010165 filed on Mar. 15, 2018, which claims priority benefit of Japanese Patent Application No. JP 2017-062305 filed in the Japan Patent Office on Mar. 28, 2017. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.
The present technology relates to an information processing device, an information processing method, and a program, and in particular relates to an information processing device, an information processing method, and a program that enable reduction of an amount of data to be transmitted in transmission of data of a plurality of audio objects.
Free-viewpoint video technologies have drawn attention as efforts of video technologies. There is a technology of combining images captured by a plurality of cameras from multiple directions to thereby retain a target object as a moving image of a point cloud, and generate a video according to a direction or distance from which the target object is viewed (NPL 1).
Once viewing of a video from a free-viewpoint is realized, people start having a demand about sounds also, demanding to listen sounds that make them feel as if they are at the place of the viewpoint. In view of this, in recent years, object-based audio technologies are drawing attention. Object-based audio data is reproduced by rendering based on metadata on waveform data of each audio object into signals of a desired number of channels depending on a system on the reproduction side.
The web site of University of Tsukuba, “HOMETSUKUBA FUTURE-#042: Customizing Sports Events with Free-Viewpoint Video,” [Retrieved: Mar. 22, 2017], <URL: http://www.tsukuba.ac.jp/notes/042/index.html>
In transmission of object-based audio data, the larger the number of audio objects to be transmitted is, the larger the data transmission amount is.
The present technology has been made in view of such a situation, and an object thereof is to enable reduction of an amount of data to be transmitted in transmission of data of a plurality of audio objects.
An information processing device according to one aspect of the present technology includes: a combining unit that combines audio objects with sounds that are undistinguishable at a predetermined supposed listening position from among a plurality of audio objects for the predetermined supposed listening position among a plurality of supposed listening positions; and a transmitting unit that transmits data of a combined audio object obtained by the combination, along with data of other audio objects with sounds that are distinguishable at the predetermined supposed listening position.
Based on audio waveform data and rendering parameters of a plurality of audio objects to be targets of the combination, the combining unit can be caused to generate audio waveform data and a rendering parameter of the combined audio object.
The transmitting unit can be caused to transmit, as the data of the combined audio object, the audio waveform data and the rendering parameter that are generated by the combining unit, and to transmit, as the data of the other audio objects, audio waveform data of each of the other audio objects and a rendering parameter for the predetermined supposed listening position.
The combining unit can be caused to combine a plurality of audio objects at positions that are away from the predetermined supposed listening position by distances which are equal to or longer than a predetermined distance.
The combining unit can be caused to combine a plurality of audio objects that is within a horizontal angle range narrower than a predetermined angle as measured from the predetermined supposed listening position as a reference position.
The combining unit can be caused to combine audio objects with sounds that are undistinguishable at the predetermined supposed listening position and belong to a same preset group.
The combining unit can be caused to perform audio object combination such that the number of audio objects to be transmitted becomes the number corresponding to a transmission bit rate.
The transmitting unit can be caused to transmit an audio bitstream including flag information representing whether an audio object included in the audio bitstream is an uncombined audio object or the combined audio object.
The transmitting unit can be caused to transmit an audio bitstream file along with a reproduction management file including flag information representing whether an audio object included in the audio bitstream is an uncombined audio object or the combined audio object.
In one aspect of the present technology, audio objects with sounds that are undistinguishable at a predetermined supposed listening position are combined from among a plurality of audio objects for the predetermined supposed listening position among a plurality of supposed listening positions; and data of a combined audio object obtained by the combination is transmitted along with data of other audio objects with sounds that are distinguishable at the predetermined supposed listening position.
The present technology enables reduction of an amount of data to be transmitted in transmission of data of a plurality of audio objects.
Note that advantages of the present technology are not necessarily limited to the advantage described here, but may be any one of advantages described in the present disclosure.
Hereinafter, embodiments for carrying out the present technology are explained. Explanations are given in the following order:
1. Configuration of Transmission System
2. Manner of Merging Objects
3. Exemplary Configuration of Each Device
4. Operations of Each Device
5. Modification Examples of Manner of Merging Objects
6. Modification Examples
<<Configuration of Transmission System>>
The transmission system illustrated in
The content generating device 1 is a device managed by a content creator, and is installed at a hall #1 where a live music performance is underway. Contents generated by the content generating device 1 are transmitted to the reproduction device 2 via the Internet 3. Content distribution may be performed via a server which is not illustrated.
On the other hand, the reproduction device 2 is a device installed in the home of a user who views and listens to contents of the live music performance generated by the content generating device 1. Although only the reproduction device 2 is illustrated as a reproduction device to which contents are distributed in the example illustrated in
Video contents generated by the content generating device 1 are a video for which one can switch the viewpoint. In addition, sound contents also are sounds for which one can switch the viewpoint (supposed listening position) such that the listening position matches the position of the video viewpoint, for example. If the viewpoint is switched, the positioning of sounds is switched.
Sound contents are prepared as object-based audio data. Audio data included in contents includes audio waveform data of each audio object, and rendering parameters as metadata for positioning the sound source of each audio object. Hereinafter, audio objects are simply called objects, as appropriate.
A user of the reproduction device 2 can select any viewpoint from a plurality of viewpoints that are prepared, and view and listen to contents through a video and sounds according to the viewpoint.
The content generating device 1 provides the reproduction device 2 with contents including video data of a video as seen from the viewpoint selected by the user, and object-based audio data of the viewpoint selected by the user. For example, such object-based audio data is transmitted in a form of data compressed in a predetermined manner such as MPEG-H 3D Audio.
Note that MPEG-H 3D Audio is disclosed at “ISO/IEC 23008-3: 2015 “Information technology—High efficiency coding and media delivery in heterogeneous environments—Part 3: 3D audio,” <https://www.iso.org/standard/63878.html>.”
Hereinafter, mainly processes related to audio data are explained. As illustrated in
For example, if a viewpoint 1 is selected from a plurality of viewpoints by the user, data of five types of objects, the bass, drums, guitar 1, guitar 2, and vocal, is transmitted as illustrated in
In addition, if the viewpoint 2 is selected by the user, the guitar 1 and the guitar 2 are merged into one object of a guitar, and data of four types of objects, the bass, drums, guitar, and vocal is transmitted as illustrated in
The viewpoint 2 is set to a position where sounds of the guitar 1 and sounds of the guitar 2 are undistinguishable by the human auditory sense since they come from the same direction, for example. In this manner, objects with sounds that are undistinguishable at a viewpoint selected by the user are merged, and transmitted as data of a single merged object.
By merging objects and transmitting them as data of a merged object as appropriate according to a selected viewpoint, it becomes possible to reduce the data transmission amount.
<<Manner of Merging Objects>>
Here, a manner of merging objects is explained.
(1) It is supposed that there is a plurality of objects. Audio waveform data of objects is defined as:
x(n, i) i=0, 1, 2, . . . , L-1
n is a time index. In addition, i represents the type of an object. Here, the number of objects is L.
(2) It is supposed that there is a plurality of viewpoints. Rendering information about objects corresponding to each viewpoint is defined as:
r(i, j) j=0, 1, 2, . . . , M-1
j represents the type of a viewpoint. The number of viewpoints is M.
(3) Audio data y(n, j) corresponding to each viewpoint is represented by Math (1):
Here, it is supposed that rendering information r is a gain (gain information). In this case, the value range of rendering information r is 0 to 1. Audio data for each viewpoint is represented by the sum of audio waveform data of all the objects, a piece of audio waveform data of each object being multiplied by a gain. A calculation like the one illustrated by Math (1) is performed at the reproduction device 2.
(4) A plurality of objects with sounds that is undistinguishable at a viewpoint are transmitted as merged data. For example, objects that are far from a viewpoint, and within a predetermined horizontal angular range from the viewpoint are selected as objects with undistinguishable sounds. On the other hand, nearby objects with distinguishable sounds at a viewpoint are not merged, but are transmitted as independent objects.
(5) Rendering information about an object corresponding to each viewpoint is defined by the type of the object, the position of the object, and the position of the viewpoint as: r(obj_type, obj_loc_x, obj_loc_y, obj_loc_z, lis_loc_x, lis_loc_y, lis_loc_z)
obj_type is information indicating the type of the object, and indicates the type of a musical instrument, for example.
obj_loc_x, obj_loc_y, and obj_loc_z are information indicating the position of the object in a three-dimensional space.
lis_loc_x, lis_loc_y, and lis_loc_z are information indicating the position of the viewpoint in the three-dimensional space.
For objects that are transmitted independently, such parameter information constituted by obj_type, obj_loc_x, obj_loc_y, obj_loc_z, lis_loc_x, lis_loc_y, and lis_loc_z is transmitted along with rendering information r. Rendering parameters are constituted by parameter information and rendering information.
Hereinafter, specific explanations are given.
(6) For example, each of the objects, the bass, drums, guitar 1, guitar 2, and vocal, is arranged as illustrated in
(7) Axes X, Y, and Z are set for the hall #1 as illustrated in
The coordinate of each object is represented as follows in meters:
Coordinate of the bass: x=−20, y=0, and z=0
Coordinate of the drums: x=0, y=−10, and z=0
Coordinate of the guitar 1: x=20, y=0, and z=0
Coordinate of the guitar 2: x=30, y=0, and z=0
Coordinate of the vocal: x=0, y=10, and z=0
(8) The coordinate of each viewpoint is represented as follows:
Viewpoint 1: x=25, y=30, and z=−1
Viewpoint 2: x=−35, y=30, and z=−1
Note that the positions of each object and each viewpoint in the figure represent merely an image of positional relations, and are not positions accurately reflecting each of the numerical values explained above.
(9) At this time, rendering information about each object for the viewpoint 1 is represented as follows:
Rendering information about the bass:
Rendering information about the drums:
Rendering information about the guitar 1:
Rendering information about the guitar 2:
Rendering information about the vocal:
obj_type of each object assumes the following values.
Bass: obj_type=0
Drums: obj_type=1
Guitar 1: obj_type=2
Guitar 2: obj_type=3
Vocal: obj_type=4
For the viewpoint 2 also, rendering parameters including parameter information and rendering information represented in the manner mentioned above is generated at the content generating device 1.
(10) Based on Math (1) illustrated above, audio data in the case where the viewpoint 1 (j=0) is selected is represented by Math (2):
[Math. 2]
y(n,0)=x(n,0)*r(0,−20,0,0,25,30,−1)+x(n,1)*r(1,0,−10,0,25,30,−1)+x(n,2)*r(2,20,0,0,25,30,−1)+x(n,3)*r(3,30,0,0,25,30,−1)+x(n,4)*r(4,0,10,0,25,30,−1) (2)
It should be noted, however, that i represents the following objects in x(n, i):
i=0: object of the bass
i=1: object of the drums
i=2: object of the guitar 1
i=3: object of the guitar 2
i=4: object of the vocal
An exemplary arrangement of respective objects as seen from the viewpoint 1 is illustrated in
(11) Similarly, audio data in the case where the viewpoint (j=1) is selected is represented by Math (3):
[Math. 3]
y(n,1)=x(n,0)*r(0,−20,0,0,−35,30,−1)+x(n,1)*r(1,0,−10,0,−35,30,−1)+x(n,2)*r(2,20,0,0,−35,30,−1)+x(n,3)*r(3,30,0,0,−35,30,−1)+x(n,4)*r(4,0,10,0,−35,30,−1) (3)
An exemplary arrangement of respective objects as seen from the viewpoint 2 is illustrated in
(12) Here, as illustrated in
(13) The angle θ1 is deemed to be an angle that allows the human auditory sense to distinguish sounds, that is, an angle that allows the human auditory sense to identify a sound of the guitar 1 and a sound of the guitar 2 as sounds that come from different directions. On the other hand, the angle θ2 is deemed to be an angle that does not allow the human auditory sense to distinguish sounds. At this time, audio data of the viewpoint 2 can be replaced using Math (4):
[Math. 4]
y(n,1)=x(n,0)*r(0,−20,0,0,−35,30,−1)+x(n,1)*r(1,0,−10,0,−35,30,−1) x(n,5)*r(5,25,0,0,−35,30,−1) x(n,4)*r(3,0,10,0,−35,30,−1) (4)
In Math (4), x (n, 5) is represented by Math (5):
[Math. 5]
x(n,5)=x(n,2)+x(n,3) (5)
That is, Math (5) represents audio waveform data of one object which is obtained by merging the guitar 1 and the guitar 2 as the sum of audio waveform data of the guitar 1 and audio waveform data of the guitar 2. obj_type of the one combined object obtained by merging the guitar 1 and the guitar 2 is obj_type=5.
In addition, for example, rendering information about the combined object is represented by Math (6) as the average of rendering information about the guitar 1 and rendering information about the guitar 2:
[Math. 6]
r(5,25,0,0,−35,30,−1)=(r(2,20,0,0,−35,30,−1)+r(3,30,0,0,−35,30,−1))/2 (6)
In this manner, the combined object represented as obj_type=5 corresponds to audio waveform data x(n, 5), and is subjected to processing using rendering information r(5, 25, 0, 0, −35, 30, −1). An exemplary arrangement of respective objects in the case where the guitar 1 and the guitar 2 are merged into one object is illustrated in
An exemplary arrangement of respective objects including the combined object as seen from the viewpoint 2 is illustrated in
(14) In this manner, objects that are auditorily undistinguishable at a selected viewpoint are merged, and transmitted as single-object data.
Thereby, the content generating device 1 can reduce the number of objects for which data is transmitted, and can reduce the data transmission amount. In addition, since the number of objects to be subjected to rendering is small, the reproduction device 2 can reduce the amount of calculation required for rendering.
Note that although there is the vocal as an object which is within the horizontal angle range of the angle θ2 as seen from the viewpoint 2 other than the guitar 1 and the guitar 2 in the example of
<<Exemplary Configuration of Each Device>>
<Configuration of Content Generating Device 1>
A CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, and a RAM (Random Access Memory) 23 are interconnected by a bus 24. The bus 24 is further connected with an input/output interface 25. The input/output interface 25 is connected with an input unit 26, an output unit 27, a storage unit 28, a communication unit 29, and a drive 30.
The input unit 26 is constituted by a keyboard, a mouse, and the like. The input unit 26 outputs signals representing contents of manipulation by a user.
The output unit 27 is constituted by a display such as an LCD (Liquid Crystal Display) or an organic EL display, and a speaker.
The storage unit 28 is constituted by a hard disk, a non-volatile memory, and the like. The storage unit 28 stores various types of data such as programs to be executed by the CPU 21, and contents.
The communication unit 29 is constituted by a network interface and the like, and performs communication with an external device via the Internet 3.
The drive 30 writes data in an attached removable media 31, and reads out data recorded in the removable media 31.
The reproduction device 2 also has a configuration which is the same as the configuration illustrated in
At least part of the configuration illustrated in
The audio encoder 51 acquires sound signals in a live music performance collected by a microphone (not illustrated), and generates audio waveform data of each object.
The metadata encoder 52 generates rendering parameters of each object for each viewpoint according to manipulation by a content creator. Rendering parameters for each of a plurality of viewpoints set in the hall #1 are generated by the metadata encoder 52.
The audio generating unit 53 associates audio waveform data generated by the audio encoder 51 with rendering parameters generated by the metadata encoder 52 to thereby generate object-based audio data for each viewpoint. The audio generating unit 53 outputs the generated audio data for each viewpoint to the content storage unit 55.
In the audio generating unit 53, a combining unit 61 is realized. The combining unit 61 performs combination of objects, as appropriate. For example, the combining unit 61 reads out audio data for each viewpoint stored in the content storage unit 55, combines objects that can be combined, and stores audio data obtained by the combination in the content storage unit 55.
The video generating unit 54 acquires data of a video captured by a camera installed at the position of each viewpoint, and encode the data in a predetermined encoding manner to thereby generate video data for each viewpoint. The video generating unit 54 outputs the generated video data for each viewpoint to the content storage unit 55.
The content storage unit 55 stores the audio data for each viewpoint generated by the audio generating unit 53 and the video data for each viewpoint generated by the video generating unit 54 in association with each other.
The transmission control unit 56 controls the communication unit 29, and performs communication with the reproduction device 2. The transmission control unit 56 receives selection viewpoint information which is information representing a viewpoint selected by a user of the reproduction device 2, and sends, to the reproduction device 2, contents consisting of video data and audio data corresponding to the selected viewpoint.
<Configuration of Reproduction Device 2>
At least part of the configuration illustrated in
If a viewpoint is selected by a user, the content acquiring unit 71 controls the communication unit 29, and sends selection viewpoint information to the content generating device 1. The content acquiring unit 71 receives and acquires contents sent from the content generating device 1 in response to the sending of the selection viewpoint information. The content generating device 1 sends contents including video data and audio data corresponding to the viewpoint selected by a user. The content acquiring unit 71 outputs the acquired contents to the separating unit 72.
The separating unit 72 separates video data and audio data included in the contents supplied from the content acquiring unit 71. The separating unit 72 outputs the video data of the contents to the video reproduction unit 74, and outputs the audio data of the contents to the audio reproduction unit 73.
Based on rendering parameters, the audio reproduction unit 73 performs rendering of audio waveform data constituting the audio data supplied from the separating unit 72, and makes sound contents output from a speaker constituting the output unit 27.
The video reproduction unit 74 decodes the video data supplied from the separating unit 72, and makes a video of contents as seen from a predetermined viewpoint displayed on a display constituting the output unit 27.
The speaker and display that are used in reproducing contents may be prepared as external equipment connected to the reproduction device 2.
<<Operations of Each Device>>
Next, operations of the content generating device 1 and reproduction device 2 having configurations like the ones mentioned above are explained.
<Operations of Content Generating Device 1>
Content Generation Process
First, processes performed by the content generating device 1 to generate contents are explained with reference to the flowchart illustrated in
The processes illustrated in
A plurality of cameras is installed in the hall #1, and videos captured by those cameras are input to the content generating device 1. In addition, microphones are installed near each object in the hall #1, and sound signals acquired by those microphones are input to the content generating device 1.
At Step S1, the video generating unit 54 acquires data of a video captured by a camera for each viewpoint, and generates a video data for each viewpoint.
At Step S2, the audio encoder 51 acquires sound signals of each object, and generates audio waveform data of each object. In the example mentioned above, audio waveform data of each of the objects, the bass, drums, guitar 1, guitar 2 and vocal, is generated.
At Step S3, the metadata encoder 52 generates rendering parameters of each object for each viewpoint according to manipulation by a content creator.
For example, if the viewpoint 1 and the viewpoint 2 are set in the hall #1 as mentioned above, a set of rendering parameters of each of the objects, the bass, drums, guitar 1, guitar 2, and vocal, for the viewpoint 1, and a set of rendering parameters of each of the objects, the bass, drums, guitar 1, guitar 2, and vocal, for the viewpoint 2 are generated.
At Step S4, the content storage unit 55 associates audio data with video data for each viewpoint to thereby generate and store contents for each viewpoint.
The processes mentioned above are performed repeatedly while the live music performance is underway. For example, when the live music performance ended, the processes of
Object Combination Processes
Next, processes performed by the content generating device 1 to combine objects are explained with reference to the flowchart illustrated in
For example, the processes illustrated in
At Step S11, the combining unit 61 pays attention to a predetermined one viewpoint among a plurality of viewpoints for which rendering parameters are generated.
At Step S12, based on parameter information included in rendering parameters, the combining unit 61 identifies the position of each object, and determines the distance to each object as measured from the viewpoint to which attention is being paid as the reference position.
At Step S13, the combining unit 61 determines whether or not there is a plurality of objects far from the viewpoint to which attention is being paid. For example, objects at positions which are at distances equal to or longer than a distance preset as a threshold are treated as distant objects. If it is determined at Step S13 that there are not a plurality of distant objects, the flow returns to Step S11, and the processes mentioned above are repeated while viewpoints to which attention is paid are switched.
On the other hand, if it is determined at Step S13 there is a plurality of distant objects, the process advances to Step S14. If the viewpoint 2 is selected as a viewpoint to which attention is being paid, for example, the drums, guitar 1, and guitar 2 are determined as distant objects.
At Step S14, the combining unit 61 determines whether or not the plurality of distant objects is within a predetermined horizontal angular range. That is, in this example, objects that are far from a viewpoint, and within a predetermined horizontal angular range from the viewpoint are processed as objects with undistinguishable sounds.
If it is determined at Step S14 that the plurality of distant objects is not within the predetermined horizontal angular range, at Step S15, the combining unit 61 sets all the objects as transmission targets for the viewpoint to which attention is being paid. In this case, if the viewpoint to which attention is being paid is selected at the time of content transmission, similar to the case where the viewpoint 1 is selected as mentioned above, audio waveform data of all the objects and rendering parameters of each object of the viewpoint are transmitted.
On the other hand, if it is determined at Step S14 that the plurality of distant objects is within the predetermined horizontal angular range, at Step S16, the combining unit 61 merges the plurality of distant objects within the predetermined horizontal angular range, and sets the combined object to a transmission target. In this case, if the viewpoint to which attention is being paid is selected at the time of content transmission, audio waveform data and rendering parameters of the combined object are transmitted along with audio waveform data and rendering parameters of uncombined, independent objects.
At Step S17, the combining unit 61 determines the sum of audio waveform data of the distant objects within the predetermined horizontal angular range to thereby generate audio waveform data of the combined object. This process is equivalent to the process of a calculation of Math (5) illustrated above.
At Step S18, the combining unit 61 determines the average of rendering parameters of the distant objects within the predetermined horizontal angular range to thereby generate rendering parameters of the combined object. This process is equivalent to the process of a calculation of Math (6) illustrated above.
The audio waveform data and rendering parameters of the combined object are stored in the content storage unit 55, and are managed as data to be transmitted when the viewpoint to which attention is being paid is selected.
After the transmission target is set at Step S15, or after the rendering parameters of the combined object are generated at Step S18, at Step S19, the combining unit 61 determines whether or not attention has been paid to all the viewpoints. If it is determined at Step S19 that there is a viewpoint to which attention has not been paid, the flow returns to Step S11, and the processes mentioned above are repeated while viewpoints to which attention is paid are switched.
On the other hand, if it is determined at Step S19 that attention has been paid to all the viewpoints, the processes illustrated in
With the processes mentioned above, objects with sounds that are undistinguishable from a viewpoint are merged into a combined object.
The processes illustrated in
Not objects that are far from the viewpoint and within a predetermined horizontal angular range as seen from the viewpoint, but simply objects that are far from the viewpoint may be processed as objects with undistinguishable sounds. In addition, objects that are within a predetermined horizontal angular range as seen from the viewpoint may be processed as objects with undistinguishable sounds.
Distances between objects may be calculated, and objects with a distance therebetween which is shorter than a threshold distance may be merged into a combined object.
If the amount of components of audio waveform data of one object that masks audio waveform data of another object is larger than a threshold, those objects may be processed as objects with undistinguishable sounds. In this manner, the manner of determination about objects with undistinguishable sounds may be arbitrary.
Content Transmission Processes
Next, processes performed by the content generating device 1 to transmit contents are explained with reference to the flowchart illustrated in
For example, the processes illustrated in
At Step S31, the transmission control unit 56 receives the selection viewpoint information sent from the reproduction device 2.
At Step S32, the transmission control unit 56 read outs, from the content storage unit 55, video data for a viewpoint selected by a user of the reproduction device 2, and audio waveform data and rendering parameters of each object for the selected viewpoint, and transmit them. For objects that are combined, audio waveform data and rendering parameters generated for audio data of a combined object are transmitted.
The processes mentioned above are performed repeatedly until content transmission is ended. When the content transmission is ended, the processes illustrated in
<Operations of Reproduction Device 2>
Next, processes performed by the reproduction device 2 to reproduce contents are explained with reference to the flowchart illustrated in
At Step S101, the content acquiring unit 71 sends information representing a viewpoint selected by a user to the content generating device 1 as selection viewpoint information.
For example, before viewing and listening of contents is started, a screen to be used for selecting from which viewpoint among a plurality of prepared viewpoints contents are to be viewed and listened to is displayed based on information sent from the content generating device 1. In response to sending of selection viewpoint information, the content generating device 1 sends contents including video data and audio data for a viewpoint selected by a user.
At Step S102, the content acquiring unit 71 receives and acquires the contents sent from the content generating device 1.
At Step S103, the separating unit 72 separates the video data and audio data included in the contents.
At Step S104, the video reproduction unit 74 decodes the video data supplied from the separating unit 72, and makes a video of contents as seen from a predetermined viewpoint displayed on a display.
At Step S105, based on rendering parameters of each object, the audio reproduction unit 73 performs rendering of audio waveform data of each object included in the audio data supplied from the separating unit 72, and makes sounds output from a speaker.
The processes mentioned above are performed repeatedly until content reproduction is ended. When the content reproduction is ended, the processes illustrated in
A series of processes mentioned above can reduce the number of objects to be transmitted, and can reduce the data transmission amount.
<<Modification Examples of Manner of Merging Objects>>
(1) Manner of Merging according to Transmission Bit Rate
The maximum number of objects may be decided according to the transmission bit rate, and objects may be merged such that the number of the objects does not exceed the maximum number.
For example, if the maximum number of objects according to a transmission bit rate is three, and the viewpoint 3 is selected, the piano, bass, vocal 1, and vocal 2 are merged into a first object based on determination according to angles like the one mentioned above. The piano, bass, vocal 1, and vocal 2 are objects within an angular range between a broken line A11 and a broken line A12 set for the left side of the stage #11 as seen from the viewpoint 3 as the reference position.
Similarly, the drums, vocal 3, and vocal 4 are merged into a second object. The drums, vocal 3, and vocal 4 are objects within an angular range between the broken line A12 and a broken line A13 set for the middle of the stage #11.
In addition, the trumpet, saxophone, guitar 1, guitar 2, vocal 5, and vocal 6 are merged into a third object. The trumpet, saxophone, guitar 1, guitar 2, vocal 5, and vocal 6 are objects within an angular range between the broken line A13 and a broken line A14 set for the right side of the stage #11.
In the manner mentioned above, audio waveform data and rendering parameters of each object (combined object) are generated, and audio data of three objects is transmitted. The number of combined objects into which objects are merged in this manner can be set to three or larger.
In the example illustrated in
The manner of merging illustrated in
By deciding the number of objects to be transmitted according to the transmission bit rate, viewing and listening with high-quality sound is allowed in the case where the transmission bit rate is high, and viewing and listening with low-quality sound is allowed in the case where the transmission bit rate is low, thus enabling content transmission at sound quality corresponding to the transmission bit rate.
For example, as audio data to be transmitted in the case where the viewpoint 3 is selected, the content storage unit 55 of the content generating device 1 stores audio data of the three objects as illustrated in
The transmission control unit 56 categorizes the communication environment of the reproduction device 2 before content transmission is started, and performs the transmission by selecting either the audio data of the three objects or the audio data of the six objects according to the transmission bit rate.
(2) Grouping of Objects
Although in the examples mentioned above, rendering information is gains, it may be reverb information. Among parameters constituting reverb information, an important parameter is reverberation amount. Reverberation amount is an amount of components of spatial reflection at walls, a floor, and the like. The reverberation amount varies depending on distances between objects (musical instruments) and a viewer/listener. Typically, the shorter the distance is, the smaller reverberation amount is, and the longer the distance is, the larger the reverberation amount is.
Other than judging whether or not sounds are distinguishable based on distances or angles to merge objects, distances between objects may be used as another index to merge objects. An example in which objects are merged also taking distances between objects into consideration is illustrated in
In the example illustrated in
Group 1: vocal 1 and vocal 2
Group 2: vocal 3 and vocal 4
Group 3: vocal 5 and vocal 6
Group 4: bass
Group 5: piano
Group 6: drums
Group 7: guitars 1 and 2
Group 8: trumpet and saxophone
In this case, as audio data to be transmitted in the case where the viewpoint 3 is selected, the content storage unit 55 of the content generating device 1 stores audio data of the eight objects.
In this manner, even objects that are within an angular range in which sounds are undistinguishable may be processed as objects to which different reverb is applied.
In this manner, it is possible to set in advance a group consisting of objects that can be merged. Only objects that satisfy conditions like the ones mentioned above based on distances and angles, and belong to the same group are to be merged into a combined object.
A group may be set according not only to distances between objects, but also to the types of objects, the positions of objects, and the like.
Note that rendering information may be not only gains or reverb information, but also equalizer information, compressor information or reverb information. That is, rendering information r can be information representing at least any one of gains, equalizer information, compressor information, and reverb information.
(3) Enhancement of Efficiency of Object Audio Encoding
In the example explained below, objects of two stringed instruments are merged into one stringed instrument object. The one stringed instrument object as a combined object is allocated a new object type (obj_type).
If it is supposed that audio waveform data of a violin 1 and audio waveform data of a violin 2 which are objects to be merged are x(n, 10) and x(n, 11), respectively, audio waveform data x(n, 14) of the stringed instrument object as a combined object is represented by Math (7) illustrated below:
[Math. 7]
x(n,14)=x(n,10)+x(n,11) (7)
Here, since the violin 1 and the violin 2 are the same stringed instruments, the two pieces of audio waveform data are highly correlated.
The difference component x(n, 15) of the audio waveform data of the violin 1 and the violin 2 indicated by Math (8) illustrated below has low information entropy, and requires only a low bit rate in case of being encoded.
[Math. 8]
x(n,15)=x(n,10)−x(n,11) (8)
By transmitting the difference component x(n, 15) indicated by Math (8) along with the audio waveform data x(n, 14) represented as the sum component, high-quality sounds can be realized at a low bit rate as explained below.
It is supposed that normally the content generating device 1 transmits the audio waveform data x(n, 14) to the reproduction device 2. Here, if conversion into high-quality sounds is performed on the reproduction device 2 side, the difference component x(n, 15) is also transmitted.
By performing calculations illustrated by Math (9) and Math (10) illustrated below, the reproduction device 2 having received the difference component x(n, 15) along with the audio waveform data x(n, 14) can reproduce the audio waveform data x(n, 10) of the violin 1 and the audio waveform data x(n, 11) of the violin 2.
[Math. 9]
(x(n,14)−x(n,15))/2=(x(n,10)+x(n,11)−x(n,10)+x(n,11))/2=x(n,11) (10)
In this case, the content storage unit 55 of the content generating device 1 stores the difference component x(n, 15) along with the audio waveform data x(n, 14) as stringed instrument object audio data to be transmitted if a predetermined viewpoint is selected.
A flag indicating that difference component data is retained is managed at the content generating device 1. The flag is sent from the content generating device 1 to the reproduction device 2 along with other information, for example, and the reproduction device 2 identifies that difference component data is retained.
In this manner, by retaining even a difference component of audio waveform data of highly correlated objects on the content generating device 1 side, it becomes possible to adjust sound quality according to the transmission bit rate at two levels. That is, if the communication environment of the reproduction device 2 is good (if the transmission bit rate is high), the audio waveform data x(n, 14) and the difference component x(n, 15) are transmitted, and if the communication environment is not good, only the audio waveform data x(n, 14) is transmitted.
Note that the amount of data of the sum of the audio waveform data x(n, 14) and the difference component x(n, 15) is smaller than the amount of data of the sum of the audio waveform data x(n, 10) and x(n, 11).
Also if the number of objects is four, the objects can be merged similarly. If four musical instruments are merged, the audio waveform data x(n, 14) of the merged object is represented by Math (11) illustrated below:
[Math. 11]
x(n,14)=x(n,10)+x(n,11)+x(n,12)+x(n,13) (11)
Here, x(n, 10), x(n, 11), x(n, 12), and x(n, 13) are audio waveform data of the violin 1, audio waveform data of the violin 2, audio waveform data of the violin 3, and audio waveform data of the violin 4, respectively.
In this case, the difference component data represented by Maths (12) to (14) illustrated below is retained by the content generating device 1.
[Math. 12]
x(n,15)=x(n,10)+x(n,11)−x(n,12)−x(n,13) (12)
[Math. 13]
x(n,16)=x(n,10)−x(n,11)+x(n,12)−x(n,13) (13)
[Math. 14]
x(n,17)=x(n,10)−x(n,11)−x(n,12)+x(n,13) (14)
It is supposed that normally the content generating device 1 transmits the audio waveform data x(n, 14) to the reproduction device 2. Here, if conversion into high-quality sounds is performed on the reproduction device 2 side, the difference components x(n, 15), x(n, 16), and x(n, 17) are also transmitted.
By performing calculations illustrated by Maths (15) to (18) below, the reproduction device 2 having received the difference components x(n, 15), x(n, 16), and x(n, 17) along with the audio waveform data x(n, 14) can reproduce the audio waveform data x(n, 10) of the violin 1, the audio waveform data x(n, 11) of the violin 2, the audio waveform data x(n, 12) of the violin 3, and the audio waveform data x(n, 13) of the violin 4.
[Math. 15]
(x(n,14)+x(n,15)+x(n,16)+x(n,17))/4=x(n,10) (15)
[Math. 16]
(x(n,14)+x(n,15)−x(n,16)−x(n,17))/4=x(n,11) (16)
[Math. 17]
(x(n,14)−x(n,15)+x(n,16)−x(n,17))/4=x(n,12) (17)
[Math. 18]
(x(n,14)−x(n,15)−x(n,16)+x(n,17))/4=x(n,13) (18)
Furthermore, it can be known from Math (19) illustrated below that if there are the audio waveform data x(n, 14) and the difference component x(n, 15), the sum (x(n, 10)+x(n, 11)) of the audio waveform data of the violin 1 and the audio waveform data of the violin 2 can be acquired. In addition, it can be known from Math (20) illustrated below that if there are the audio waveform data x(n, 14) and the difference component x(n, 15), the sum (x(n, 12)+x(n, 13)) of the audio waveform data of the violin 3 and the audio waveform data of the violin 4 can be acquired.
[Math. 19]
(x(n,14)+x(n,15))/2=x(n,10)+x(n,11) (19)
[Math. 20]
x(n,14)−x(n,15))/2=x(n,12)+x(n,13) (20)
For example, if the transmission bit rate that the reproduction device 2 can support is higher than a first threshold, and the communication environment is the best among three levels, the difference components x(n, 15), x(n, 16), and x(n, 17) are transmitted from the content generating device 1 along with the audio waveform data x(n, 14) obtained by merging the four objects.
Calculations illustrated by Maths (15) to (18) are performed at the reproduction device 2, audio waveform data of individual objects, the violin 1, violin 2, violin 3, and violin 4, is acquired, and reproduction is performed with high quality.
In addition, if the transmission bit rate that the reproduction device 2 can support is lower than the first threshold mentioned above, but is higher than a second threshold, and the communication environment is relatively good, the difference component x(n, 15) is transmitted from the content generating device 1 along with the audio waveform data x(n, 14) obtained by merging the four objects.
Calculations illustrated by Math (19) and Math (20) are performed at the reproduction device 2, audio waveform data obtained by merging the violin 1 and violin 2, and audio waveform data obtained by merging the violin 3 and violin 4 are acquired, and reproduction is performed with higher quality than that performed in the case where only the audio waveform data x(n, 14) is used.
If the transmission bit rate that the reproduction device 2 can support is lower than the second threshold mentioned above, the audio waveform data x(n, 14) obtained by merging the four objects is transmitted from the content generating device 1.
In this manner, hierarchical transmission (encoding) according to a transmission bit rate may be performed by the content generating device 1.
Such hierarchical transmission may be performed according to a fee paid by a user of the reproduction device 2. For example, if the user paid a normal fee, transmission of only the audio waveform data x(n, 14) is performed, and if the user paid a fee higher than the normal fee, transmission of the audio waveform data x(n, 14) and a difference component is performed.
(4) Cooperation with Point Cloud Moving Image Data
It is supposed that video data of contents transmitted by the content generating device 1 is point cloud moving image data. Both point cloud moving image data and object audio data have data about coordinates in a three-dimensional space, and serve as color data and audio data at those coordinates.
Note that point cloud moving image data is disclosed, for example, at “Microsoft “A Voxelized Point Cloud Dataset,” <https://jpeg.org/plenodb/pc/microsoft/>.”
The content generating device 1 retains a three-dimensional coordinate as information about the position of a vocal, for example, and in association with the coordinate, retains point cloud moving image data and audio object data. Thereby, the reproduction device 2 can easily acquire point cloud moving image data and audio object data of a desired object.
An audio bitstream transmitted by the content generating device 1 may include flag information indicating whether or not an object being transmitted by the stream is an unmerged independent object or a combined object. An audio bitstream including flag information is illustrated in
The audio bitstream illustrated in
The flag information illustrated in
Thereby, by analyzing the stream, the reproduction device 2 can identify whether data included in the stream is data of a combined object or data of an independent object.
Such flag information may be described in a reproduction management file transmitted along with a bitstream as illustrated in
Thereby, by referring to the reproduction management file, the reproduction device 2 can identify whether an object being transmitted by the stream is a combined object or an independent object.
Although it is explained that contents to be reproduced by the reproduction device 2 includes video data and object-based audio data, the contents may not include video data, but may consist of object-based audio data. If a predetermined listening position is selected from listening positions for which rendering parameters are prepared, rendering parameters for the selected listening position are used to reproduce each audio object.
Embodiments of the present technology are not limited to the embodiment mentioned above, but can be changed in various manners within a scope that does not deviate from the gist of the present technology.
For example, the present technology can have a configuration of cloud computing in which a plurality of devices shares one function via a network, and performs processes in cooperation with each other.
In addition, individual steps explained in the flowcharts mentioned above can be executed by one device, or may be executed by a plurality of devices in a shared manner.
Furthermore, if one step includes a plurality of processes, the plurality of processes included in the one step can be executed by one device, or may be executed by a plurality of devices in a shared manner.
Advantages described in the present specification are illustrated merely as examples, advantages are not limited to them, and there may be other advantages.
About Program
The series of processes mentioned above can be executed by hardware, and can also be executed by software. If the series of processes is executed by software, a program constituting the software is installed on a computer incorporated into dedicated hardware, a general-purpose personal computer, or the like.
The program to be installed is provided as a program recorded in the removable media 31 illustrated in
Note that the program to be executed by a computer may be a program to perform processes in a temporal sequence along the order explained in the present specification, or may be a program that performs processes in parallel, or at required timings when the processes are called or at different timings.
About Combinations
The present technology can also be configured in the following manners.
(1) An information processing device including:
a combining unit that combines audio objects with sounds that are undistinguishable at a predetermined supposed listening position from among a plurality of audio objects for the predetermined supposed listening position among a plurality of supposed listening positions; and
a transmitting unit that transmits data of a combined audio object obtained by the combination, along with data of other audio objects with sounds that are distinguishable at the predetermined supposed listening position.
(2) The information processing device according to (1) explained above, in which
based on audio waveform data and rendering parameters of a plurality of audio objects to be targets of the combination, the combining unit generates audio waveform data and a rendering parameter of the combined audio object.
(3) The information processing device according to (2) explained above, in which
the transmitting unit transmits, as the data of the combined audio object, the audio waveform data and the rendering parameter that are generated by the combining unit, and transmits, as the data of the other audio objects, audio waveform data of each of the other audio objects and a rendering parameter for the predetermined supposed listening position.
(4) The information processing device according to any one of (1) to (3) explained above, in which
the combining unit combines a plurality of audio objects at positions that are away from the predetermined supposed listening position by distances which are equal to or longer than a predetermined distance.
(5) The information processing device according to any one of (1) to (4) explained above, in which
the combining unit combines a plurality of audio objects that is within a horizontal angle range narrower than a predetermined angle as measured from the predetermined supposed listening position as a reference position.
(6) The information processing device according to any one of (1) to (5) explained above, in which
the combining unit combines audio objects with sounds that are undistinguishable at the predetermined supposed listening position and belong to a same preset group.
(7) The information processing device according to any one of (1) to (6) explained above, in which
the combining unit performs audio object combination such that the number of audio objects to be transmitted becomes the number corresponding to a transmission bit rate.
(8) The information processing device according to any one of (1) to (7) explained above, in which
the transmitting unit transmits an audio bitstream including flag information representing whether an audio object included in the audio bitstream is an uncombined audio object or the combined audio object.
(9) The information processing device according to any one of (1) to (7) explained above, in which
the transmitting unit transmits an audio bitstream file along with a reproduction management file including flag information representing whether an audio object included in the audio bitstream is an uncombined audio object or the combined audio object.
(10) An information processing method including steps of:
combining audio objects with sounds that are undistinguishable at a predetermined supposed listening position from among a plurality of audio objects for the predetermined supposed listening position among a plurality of supposed listening positions; and
transmitting data of a combined audio object obtained by the combination, along with data of other audio objects with sounds that are distinguishable at the predetermined supposed listening position.
(11) A program for making a computer execute processing including steps of:
combining audio objects with sounds that are undistinguishable at a predetermined supposed listening position from among a plurality of audio objects for the predetermined supposed listening position among a plurality of supposed listening positions; and
transmitting data of a combined audio object obtained by the combination, along with data of other audio objects with sounds that are distinguishable at the predetermined supposed listening position.
1: Content generating device, 2: Reproduction device, 51: Audio encoder, 52: Metadata encoder, 53: Audio generating unit, 54: Video generating unit, 55: Content storage unit, 56: Transmission control unit, 61: Combining unit, 71: Content acquiring unit, 72: Separating unit, 73: Audio reproduction unit, 74: Video reproduction unit, 73: Audio reproduction unit
| Number | Date | Country | Kind |
|---|---|---|---|
| JP2017-062305 | Mar 2017 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2018/010165 | 3/15/2018 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2018/180531 | 10/4/2018 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20050114121 | Tsingos et al. | May 2005 | A1 |
| 20050249367 | Bailey | Nov 2005 | A1 |
| 20100145487 | Oh et al. | Jun 2010 | A1 |
| 20130016842 | Schultz-Amling et al. | Jan 2013 | A1 |
| 20140023196 | Xiang et al. | Jan 2014 | A1 |
| 20140023197 | Xiang et al. | Jan 2014 | A1 |
| 20140025386 | Xiang et al. | Jan 2014 | A1 |
| 20160142846 | Herre et al. | May 2016 | A1 |
| 20160192105 | Breebaart et al. | Jun 2016 | A1 |
| 20170223476 | Breebaart et al. | Aug 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 079517 | Feb 2012 | AR |
| 2010332934 | Jul 2012 | AU |
| 2784862 | Jun 2011 | CA |
| 102859584 | Jan 2013 | CN |
| 104471640 | Mar 2015 | CN |
| 105431900 | Mar 2016 | CN |
| 105593930 | May 2016 | CN |
| 2346028 | Jul 2011 | EP |
| 2502228 | Sep 2012 | EP |
| 2830045 | Jan 2015 | EP |
| 3028273 | Jun 2016 | EP |
| 2592217 | Nov 2016 | ES |
| 2862799 | May 2005 | FR |
| 1176733 | Jul 2017 | HK |
| 5426035 | Feb 2014 | JP |
| 2016-528542 | Sep 2016 | JP |
| 2016-530803 | Sep 2016 | JP |
| 10-2012-0089369 | Aug 2012 | KR |
| 10-2015-0038156 | Apr 2015 | KR |
| 10-2016-0021892 | Feb 2016 | KR |
| 10-2016-0053910 | May 2016 | KR |
| 10-1646867 | Aug 2016 | KR |
| 10-2016-0140971 | Dec 2016 | KR |
| 2012132354 | Jan 2014 | RU |
| 2016106913 | Sep 2017 | RU |
| 201146026 | Dec 2011 | TW |
| 2011073210 | Jun 2011 | WO |
| 2014015299 | Jan 2014 | WO |
| 2015011024 | Jan 2015 | WO |
| 2015017235 | Feb 2015 | WO |
| 2018047667 | Mar 2018 | WO |
| Entry |
|---|
| Extended European Search Report of EP Application No. 18774689.6, dated Mar. 16, 2020, 06 pages. |
| International Search Report and Written Opinion of PCT Application No. PCT/JP2018/010165, dated May 15, 2018, 10 pages of ISRWO. |
| Number | Date | Country | |
|---|---|---|---|
| 20200043505 A1 | Feb 2020 | US |
