Patent Application
20200014901
The present invention relates to an information processing apparatus regarding generation of a virtual viewpoint image, a control method therefor and a computer-readable medium.
These days, attention has been paid to a technique of generating a virtual viewpoint image using a plurality of viewpoint images obtained by installing a plurality of cameras at different positions and executing synchronous shooting from multiple viewpoints. The technique of generating a virtual viewpoint image allows a user to, for example, view highlights of soccer or basketball from various angles and can give a high realistic sensation to him/her.
A virtual viewpoint image based on a plurality of viewpoint images is generated by collecting images captured by a plurality of cameras to an image processing unit such as a server and performing processes such as three-dimensional model generation and rendering by the image processing unit. The generation of a virtual viewpoint image requires setting of a virtual viewpoint. For example, a content creator generates a virtual viewpoint image by moving the position of a virtual viewpoint over time. Even for an image at a single timing, various virtual viewpoints can be necessary depending on viewer tastes and preference. In Japanese Patent Laid-Open No. 2015-187797, a plurality of viewpoint images and free viewpoint image data including metadata representing a recommended virtual viewpoint are generated. The user can easily set various virtual viewpoints using the metadata included in the free viewpoint image data.
When virtual viewpoint images are provided to a plurality of viewers of different tastes or when a viewer wants to view both a virtual viewpoint image at a given viewpoint and a virtual viewpoint image at another viewpoint, a plurality of virtual viewpoint images corresponding to a plurality of virtual viewpoints at the same timing are generated. However, if a plurality of time-series virtual viewpoints are individually set to generate a plurality of virtual viewpoint images, like the conventional technique, setting of virtual viewpoints takes a lot of time. The technique disclosed in Japanese Patent Laid-Open No. 2015-187797 reduces the labor for setting a single virtual viewpoint. However, when a plurality of virtual viewpoints are set, the setting is still troublesome.
The present invention provides a technique of enabling easy setting of a plurality of virtual viewpoints regarding generation of a virtual viewpoint image.
According to one aspect of the present invention, there is provided an information processing apparatus comprising: a setting unit configured to set a first virtual viewpoint regarding generation of a virtual viewpoint image based on multi-viewpoint images obtained from a plurality of cameras; and a generation unit configured to generate, based on the first virtual viewpoint set by the setting unit, viewpoint information representing a second virtual viewpoint that is different in at least one of a position and direction from the first virtual viewpoint set by the setting unit and corresponds to a timing common to the first virtual viewpoint.
According to another aspect of the present invention, there is provided an information processing apparatus comprising: a setting unit configured to set a first virtual viewpoint regarding generation of a virtual viewpoint image based on multi-viewpoint images obtained from a plurality of cameras; and a generation unit configured to generate, based on a position of an object included in the multi-viewpoint images, viewpoint information representing a second virtual viewpoint that is different in at least one of a position and direction from the first virtual viewpoint set by the setting unit and corresponds to a timing common to the first virtual viewpoint.
According to another aspect of the present invention, there is provided a method of controlling an information processing apparatus, comprising: setting a first virtual viewpoint regarding generation of a virtual viewpoint image based on multi-viewpoint images obtained from a plurality of cameras; and generating, based on the set first virtual viewpoint, viewpoint information representing a second virtual viewpoint that is different in at least one of a position and direction from the set first virtual viewpoint and corresponds to a timing common to the first virtual viewpoint.
According to another aspect of the present invention, there is provided a method of controlling an information processing apparatus, comprising: setting a first virtual viewpoint regarding generation of a virtual viewpoint image based on multi-viewpoint images obtained from a plurality of cameras; and generating, based on a position of an object included in the multi-viewpoint images, viewpoint information representing a second virtual viewpoint that is different in at least one of a position and direction from the set first virtual viewpoint and corresponds to a timing common to the first virtual viewpoint.
According to another aspect of the present invention, there is provided a non-transitory computer-readable medium storing a program for causing a computer to execute each step of the above-described method of controlling an information processing apparatus.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Several embodiments of the present invention will now be described with reference to the accompanying drawings. In this specification, an image is a general term of “video” “still image”, and “moving image”.
A viewpoint input unit 101 accepts a user input of a virtual viewpoint for setting a virtual camera. A virtual viewpoint designated by an input accepted by the viewpoint input unit 101 will be called an input viewpoint. A user input for designating an input viewpoint is performed via the input device 155. An another-viewpoint generation unit 102 generates a virtual viewpoint different from the input viewpoint in order to set the position of another virtual camera based on the input viewpoint designated by the user. A virtual viewpoint generated by the another-viewpoint generation unit 102 will be called another viewpoint. A material data obtaining unit 103 obtains, from the server 1102, the material data 1105 for generating a virtual viewpoint image. Based on the input viewpoint from the viewpoint input unit 101 and another viewpoint from the another-viewpoint generation unit 102, a virtual viewpoint image generation unit 104 generates virtual viewpoint images corresponding to the respective virtual viewpoints by using the material data obtained by the material data obtaining unit 103. A display control unit 105 performs control to display, on the display device 156, an image of material data (for example, one image of the multi-viewpoint images 1104) obtained by the material data obtaining unit 103 and a virtual viewpoint image generated by the virtual viewpoint image generation unit 104. A data storage unit 107 stores a virtual viewpoint image generated by the virtual viewpoint image generation unit 104, information of a viewpoint sent from the viewpoint input unit 101 or the another-viewpoint generation unit 102, and the like by using the external storage device 157. Note that the configuration of the image generation apparatus 100 is not limited to one shown in
A virtual camera 205 is a virtual camera corresponding to another viewpoint 212 set based on the input viewpoint 211 and is arranged to face the virtual camera 204. In the example of
In 2a of
In
Next, the operation of the another-viewpoint generation unit 102 will be described.
By the above-described processing, the another-viewpoint generation unit 102 generates another viewpoint in time series following a viewpoint input in time series from the viewpoint input unit 101. For example, when the input viewpoint 211 that moves so as to draw the curve 301 shown in
Next, virtual viewpoint image generation processing by the virtual viewpoint image generation unit 104 will be described.
In step S413, the virtual viewpoint image generation unit 104 determines whether it has received viewpoint information of the other viewpoint 212 from the another-viewpoint generation unit 102. If the virtual viewpoint image generation unit 104 determines in step S413 that it has received viewpoint information of the other viewpoint 212, the process advances to step S414. If the virtual viewpoint image generation unit 104 determines that it has not received viewpoint information of the other viewpoint 212, the process returns to step S413. In step S414, the virtual viewpoint image generation unit 104 arranges the virtual camera 205 based on the viewpoint information received in step S413 and generates a virtual viewpoint image to be captured by the virtual camera 205. In step S415, the virtual viewpoint image generation unit 104 determines whether reception of the viewpoint information from each of the viewpoint input unit 101 and another-viewpoint generation unit 102 has ended. If the virtual viewpoint image generation unit 104 determines that reception of the viewpoint information is completed, the process of the flowchart ends. If the virtual viewpoint image generation unit 104 determines that reception of the viewpoint information is not completed, the process returns to step S411.
Although steps S412 and S414, which are processes of generating a virtual viewpoint image, are performed in time series in the flowchart of
Next, the generation (step S403) of the other viewpoint 212 (virtual camera 205) with respect to the input viewpoint 211 (virtual camera 204) will be further explained with reference to
The content creator arranges the input viewpoint 211 behind the attacker 201 by the distance h1 and at the height h2 larger than the attacker 201. The line-of-sight direction of the input viewpoint 211 is oriented in a direction toward the defender 203 at the timing T1. In this embodiment, an intersection point of the ground and the line of sight of the input viewpoint 211 serves as the gaze point 206. In contrast, the other viewpoint 212 at the timing T1 is generated by the another-viewpoint generation unit 102 in step S403 of
Note that the gaze point 206 is set at the ground in this embodiment, but is not limited to this. For example, when the line-of-sight direction of the input viewpoint 211 represented by input line-of-sight information is parallel to the ground, the gaze point can be set at a point at the height h2 on the perpendicular 213 passing through the gaze point 206. The another-viewpoint generation unit 102 generates another viewpoint in accordance with an input viewpoint set in time series so as to maintain the relationship in distance and line-of-sight direction between the input viewpoint and the other viewpoint. Hence, the method of generating the other viewpoint 212 from the input viewpoint 211 is not limited to the above-described one. For example, the gaze point of the input viewpoint 211 and that of the other viewpoint 212 may be set individually.
In the example of
In the first embodiment, the virtual camera 205 is arranged at a position obtained by 180°—rotation about, as an axis, the perpendicular 213 passing through the gaze point 206 based on viewpoint information (for example, the viewpoint position and the line-of-sight direction) of the input viewpoint 211 created by the content creator, but is not limited to this. In
As described above, according to the first embodiment, when generating a virtual viewpoint image, an input viewpoint is set by a user input, and another viewpoint different from the input viewpoint in at least one of the position and direction is set automatically. According to the first embodiment, a plurality of virtual viewpoint images corresponding to a plurality of virtual viewpoints at a common timing can be obtained easily.
In the first embodiment, the configuration has been described, in which another viewpoint (for example, a viewpoint at which the virtual camera 205 is arranged) is set automatically based on an input viewpoint (for example, a viewpoint at which the virtual camera 204 is arranged) set by the user. In the second embodiment, another viewpoint is set automatically using the position of an object. Note that a virtual viewpoint image generation system and the hardware configuration and functional configuration of an image generation apparatus 100 in the second embodiment are the same as those in the first embodiment (
In step S803, the another-viewpoint generation unit 102 generates the viewpoint position and gaze point position (another viewpoint) of the virtual camera 502 based on the viewpoint information obtained in step S801 and the material data (coordinates of the objects) obtained in step S802. In step S804, the another-viewpoint generation unit 102 determines whether reception of the viewpoint information from the viewpoint input unit 101 has ended. If the another-viewpoint generation unit 102 determines that reception of the viewpoint information has ended, the flowchart ends. If the another-viewpoint generation unit 102 determines that the viewpoint information is being received, the process returns to step S801.
The generation of another viewpoint in step S803 will be described in detail. As shown in
Next, an initial viewpoint determination method will be explained. First, the another-viewpoint generation unit 102 obtains viewpoint information of the input viewpoint 211 including the coordinates (x1, y1, z1) of the viewpoint position and the coordinates (x2, y2, z2) of the gaze point position from the viewpoint input unit 101. Then, the another-viewpoint generation unit 102 obtains the position coordinates (information of the object position in the material data) of each player from the material data obtaining unit 103. For example, the position coordinates of the player A are (xa, ya, za). The value za in the height direction in the position coordinates of the player A can be, for example, the height of the center of the face of the player or the body height. When the body height is used, the body height of each player is registered in advance.
In this embodiment, the other viewpoint 212a (virtual camera 502) is generated behind the player B. The another-viewpoint generation unit 102 determines the gaze point of the other viewpoint 212a based on the position of the player A closest to the input viewpoint 211. In this embodiment, the position of the gaze point on the x-y plane is set as a position (xa, ya) of the player A on the x-y plane, and the position in the z direction is set as a height from the ground. In this example, the coordinates of the gaze point position are set as (x4, y4, z4)=(xa, ya, 0). The another-viewpoint generation unit 102 sets, as the viewpoint position of the other viewpoint 212a, a position spaced apart from the position of the player B by a predetermined distance on a line connecting the position coordinates of the player B and the coordinates (x4, y4, z4) of the gaze point position of the other viewpoint 212a. In
After the viewpoint position of the other viewpoint 212a is determined based on the positional relationship between the players A and B and the gaze point position is determined based on the position coordinates of the player A in this manner, the distance between the other viewpoint 212a and the player A and the line-of-sight direction are fixed. That is, after the viewpoint position and gaze point position of the other viewpoint 212a are determined in accordance with the setting of the input viewpoint 211, the distance and direction of the other viewpoint 212a with respect to the gaze point determined from the position coordinates of the player A are fixed. By this setting, even if the position coordinates of the players A and B change over time, the positional relationship between the other viewpoint 212a (virtual camera 502) and the player A is maintained. After the viewpoint information of the other viewpoint 212a is determined in accordance with the input viewpoint 211 (virtual camera 501) and the position coordinates of the players A and B, the viewpoint position and gaze point position of the other viewpoint 212a (virtual camera 502) are determined from the position coordinates of the player A.
Note that the another-viewpoint generation unit 102 needs to specify two objects of the players A and B in order to generate the other viewpoint 212a. Both the players A and B are objects included in a virtual viewpoint image from the input viewpoint 211. For example, an object closest to the input viewpoint 211 is selected as the player A, and the player B can be specified by selecting an object by the user from the virtual viewpoint image of the input viewpoint 211. Note that the user may select an object serving as the player A. Although the distance between the other viewpoint 212a and the player A and the line-of-sight direction are fixed in the above description, the present invention is not limited to this. For example, the processing of determining the other viewpoint 212a based on the positions of the players A and B (the above-described processing of determining an initial viewpoint) may be continued. Alternatively, an object (object corresponding to the player B) used to generate another viewpoint may be selected based on the attribute of the object. For example, a team to which each object belongs may be determined based on the uniform of the object, and an object belonging to the opposing team or the team of the player A may be selected as the player B from objects present in a virtual viewpoint image obtained by the virtual camera 501. A plurality of viewpoints can be set simultaneously by selecting a plurality of objects used to set another viewpoint.
The configuration has been described above, in which another viewpoint is set behind a player near the player A in response to setting the input viewpoint 211 by the content creator. However, the another-viewpoint setting method is not limited to this. As shown in
A virtual viewpoint image captured by the virtual camera 504 arranged at the other viewpoint 212c is, for example, an image as shown in
Note that a display control unit 105 displays, on a display device 156, the virtual viewpoint images of an input viewpoint and another viewpoint that are generated by a virtual viewpoint image generation unit 104. The display control unit 105 may simultaneously display a plurality of virtual viewpoint images so that the user can select a virtual viewpoint image he/she wants.
As described above, according to each of the embodiments, another viewpoint is set automatically in accordance with an operation of setting one input viewpoint by the content creator. Since a plurality of virtual viewpoints at the set timing of one virtual viewpoint are obtained in accordance with the operation of setting one virtual viewpoint, a plurality of virtual viewpoints (and virtual viewpoint images) at the same timing can be created easily. Although an input viewpoint is set by the content creator in the description of each of the embodiments, it is not limited to this and may be set by an end user or another person. Alternatively, the image generation apparatus 100 may obtain viewpoint information representing an input viewpoint from the outside and generate viewpoint information representing another viewpoint corresponding to the input viewpoint.
The image generation apparatus 100 may determine whether to set another viewpoint or the number of other viewpoints to be set, in accordance with an input user operation, the number of objects in the shooting target area, the generation timing of an event in the shooting target area, or the like. When an input viewpoint and another viewpoint are set, the image generation apparatus 100 may display both a virtual viewpoint image corresponding to the input viewpoint and a virtual viewpoint image corresponding to the other viewpoint on the display unit, or switch and display them.
Although soccer has been exemplified in the description of each of the embodiments, the present invention is not limited to this. For example, the present invention may be applied to a sport such as rugby, baseball, or skating, or a play performed on a stage. Although a virtual camera is set based on the positional relationship between players in each of the embodiments, the present invention is not limited to this and a virtual camera may be set in consideration of, for example, the position of a referee or grader.
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2018-127794, filed Jul. 4, 2018 which is hereby incorporated by reference herein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2018-127794 | Jul 2018 | JP | national |
