The present invention relates to a technique of displaying a virtual viewpoint image based on captured images.
In recent years, a technique of performing synchronous image capturing at multi-viewpoints using a plurality of cameras placed at different positions and generating not only images from camera installation positions but also an image (virtual viewpoint image) from an arbitrary viewpoint (virtual viewpoint) using a plurality of images (a plurality of viewpoint images) obtained by the image capturing has received attention.
Generation and viewing of the virtual viewpoint image based on the plurality of viewpoint images can be implemented by the following processing. That is, images captured by the plurality of cameras are collected in an image processing unit such as a server, the image processing unit executes processing such as rendering based on the virtual viewpoint to generate a virtual viewpoint image, and then the virtual viewpoint image is displayed on a user terminal.
A service using a virtual viewpoint image allows a video producer to produce a powerful viewpoint content from, for example, a video obtained by capturing a soccer game or a basketball game. In addition, the service also allows a user who is viewing the content to watch the game by freely moving his/her viewpoint, thereby providing high realism to the user. PTL 1 discloses a technique of selecting one of a plurality of virtual viewpoints by the user.
According to the invention described in PTL 1, the user can select one of a plurality of prepared virtual viewpoints to display a virtual viewpoint image at the selected virtual viewpoint. In addition, since the rough positions of the virtual viewpoints are displayed by character strings, the virtual viewpoints can be discriminated. However, for example, if the operator of the virtual viewpoints uses in live broadcasting of a sport, he/she cannot know, before switching of the virtual viewpoint, what will be shown in a virtual viewpoint image after switching. Therefore, it may be impossible to appropriately select the virtual viewpoint in accordance with a performance. The present invention provides a technique for allowing the user to select an appropriate virtual viewpoint.
PTL 1: Japanese Patent Laid-Open No. 2014-215828
According to the first aspect of the present invention, there is provided a display control apparatus comprising: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein the virtual viewpoint image corresponding to the virtual viewpoint selected as a position and direction operation target among the plurality of virtual viewpoints is generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and is transmitted to another apparatus.
According to the second aspect of the present invention, there is provided a display control apparatus comprising: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control unit is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for adding a virtual viewpoint, the display control unit newly displays a virtual viewpoint image corresponding to a position and direction of the virtual viewpoint on the operation screen.
According to the third aspect of the present invention, there is provided a display control apparatus comprising: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control unit is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a predetermined user operation, the display control unit switches from a state in which a plurality of first virtual viewpoint images corresponding to a plurality of first virtual viewpoints are displayed on the operation screen to a state in which a plurality of second virtual viewpoint images corresponding to a plurality of second virtual viewpoints are displayed on the operation screen.
According to the fourth aspect of the present invention, there is provided a display control apparatus comprising: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control unit is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for deleting a virtual viewpoint, the display control unit controls not to display, on the operation screen, a virtual viewpoint image corresponding to a position and direction of the virtual viewpoint.
According to the fifth aspect of the present invention, there is provided a display control apparatus comprising: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control unit is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for rearranging virtual viewpoint images, the display control unit changes a display order of at least some of the plurality of virtual viewpoint images displayed on the operation screen.
According to the sixth aspect of the present invention, there is provided a display control method for a display control apparatus, comprising: acquiring a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and displaying the plurality of virtual viewpoint images acquired in the acquisition on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein the virtual viewpoint image corresponding to the virtual viewpoint selected as a position and direction operation target among the plurality of virtual viewpoints is generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and is transmitted to another apparatus.
According to the seventh aspect of the present invention, there is provided a display control method for a display control apparatus, comprising: acquiring a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and displaying the plurality of virtual viewpoint images acquired in the acquisition on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for adding a virtual viewpoint, a virtual viewpoint image corresponding to a position and direction of the virtual viewpoint is newly displayed on the operation screen.
According to the eighth aspect of the present invention, there is provided a display control method for a display control apparatus, comprising: acquiring a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and displaying the plurality of virtual viewpoint images acquired in the acquisition on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a predetermined user operation, a state in which a plurality of first virtual viewpoint images corresponding to a plurality of first virtual viewpoints are displayed on the operation screen is switched to a state in which a plurality of second virtual viewpoint images corresponding to a plurality of second virtual viewpoints are displayed on the operation screen.
According to the ninth aspect of the present invention, there is provided a display control method for a display control apparatus, comprising: acquiring a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and displaying the plurality of virtual viewpoint images acquired in the acquisition on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for deleting a virtual viewpoint, it is controlled not to display, on the operation screen, a virtual viewpoint image corresponding to a position and direction of the virtual viewpoint as a deletion target.
According to the tenth aspect of the present invention, there is provided a display control method for a display control apparatus, comprising: acquiring a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and displaying the plurality of virtual viewpoint images acquired in the acquisition on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for rearranging virtual viewpoint images, a display order of at least some of the plurality of virtual viewpoint images displayed on the operation screen is changed.
According to the eleventh aspect of the present invention, there is provided a non-transitory computer-readable storage medium for storing a computer program for causing a computer to function as: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein the virtual viewpoint image corresponding to the virtual viewpoint selected as a position and direction operation target among the plurality of virtual viewpoints is generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and is transmitted to another apparatus.
According to the twelfth aspect of the present invention, there is provided a non-transitory computer-readable storage medium for storing a computer program for causing a computer to function as: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control unit is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for adding a virtual viewpoint, the display control unit newly displays a virtual viewpoint image corresponding to a position and direction of the virtual viewpoint on the operation screen.
According to the thirteenth aspect of the present invention, there is provided a non-transitory computer-readable storage medium for storing a computer program for causing a computer to function as: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control unit is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a predetermined user operation, the display control unit switches from a state in which a plurality of first virtual viewpoint images corresponding to a plurality of first virtual viewpoints are displayed on the operation screen to a state in which a plurality of second virtual viewpoint images corresponding to a plurality of second virtual viewpoints are displayed on the operation screen.
According to the fourteenth aspect of the present invention, there is provided a non-transitory computer-readable storage medium for storing a computer program for causing a computer to function as: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control unit is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for deleting a virtual viewpoint, the display control unit controls not to display, on the operation screen, a virtual viewpoint image corresponding to a position and direction of the virtual viewpoint.
According to the fifteenth aspect of the present invention, there is provided a non-transitory computer-readable storage medium for storing a computer program for causing a computer to function as: an acquisition unit configured to acquire a plurality of virtual viewpoint images corresponding to positions and directions of a plurality of virtual viewpoints; and a display control unit configured to display the plurality of virtual viewpoint images acquired by the acquisition unit on an operation screen for an operation associated with the position and direction of the virtual viewpoint, wherein at least one of the plurality of virtual viewpoint images displayed on the operation screen by the display control unit is an image generated based on a plurality of captured images obtained by capturing an image capturing target region by a plurality of cameras, and in response to a user operation for rearranging virtual viewpoint images, the display control unit changes a display order of at least some of the plurality of virtual viewpoint images displayed on the operation screen.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Embodiments will be described below with reference to the accompanying drawings. Note that the embodiments to be described below are examples of detailed implementation of the present invention or detailed examples of the arrangement described in the appended claims.
An example of the arrangement of a system according to this embodiment will be described with reference to a block diagram shown in
First, the image capturing apparatuses 109-1 to 109-n will be described. The image capturing apparatuses 109-1 to 109-n are provided to capture videos of an object from a plurality of different directions. Each of the image capturing apparatuses 109-1 to 109-n outputs an image (captured image) of each frame forming the captured video to the image generation apparatus 110. A connection mode between the image capturing apparatuses 109-1 to 109-n and the image generation apparatus 110 is not limited to a specific one. That is, the connection mode between the image capturing apparatuses 109-1 to 109-n and the image generation apparatus 110 may be a daisy chain or a star network. Furthermore, the connection mode may be a wireless network, a wired network, or a combination thereof.
The display control apparatus 100 will be described next. The display control apparatus 100 requests the image generation apparatus 110 to generate a virtual viewpoint image based on a virtual viewpoint, and performs display control of the virtual viewpoint image generated by the image generation apparatus 110 in accordance with the request. The display control apparatus 100 is formed by a computer apparatus such as a PC (Personal Computer), a smartphone, or a tablet terminal apparatus.
A CPU (Central Processing Unit) 101 executes processing using a computer program and data stored in a ROM (Read-Only Memory) 102 and a RAM (Random Access Memory) 103. Then, the CPU 101 controls the operation of the overall display control apparatus 100 while executing or controlling each process to be described later as that executed by the display control apparatus 100.
The ROM 102 stores a computer program and data which need not be rewritten, such as an activation program and setting data.
The RAM 103 has an area to store a computer program and data loaded from the ROM 102 or an HDD (Hard Disk Drive) 104. Furthermore, the RAM 103 has an area to store the data (for example, the virtual viewpoint image) received from the image generation apparatus 110 via a communication I/F (interface) 106. The RAM 103 has a work area used by the CPU 101 to execute various processes. In this way, the RAM 103 can appropriately provide various areas.
The HDD 104 saves an OS (Operating System), and computer programs and data for causing the CPU 101 to execute or control each process to be described later as that executed by the display control apparatus 100. The data saved in the HDD 104 include those described as known information in the following description. The computer programs and data saved in the HDD 104 are appropriately loaded into the RAM 103 under the control of the CPU 101 and processed by the CPU 101.
A display unit 105 is formed by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 101 using an image or characters. Note that the display unit 105 may be a projection apparatus that projects the processing result by the CPU 101 as an image or characters. The display unit 105 may be a touch panel screen, a head mounted display device such as an HMD (Head Mounted Display), a handheld display device, a smartphone, or a tablet terminal apparatus.
The communication I/F 106 functions as an interface for performing data communication with the image generation apparatus 110. A connection mode between the display control apparatus 100 and the image generation apparatus 110 is not limited to a specific one. For example, the network between the display control apparatus 100 and the image generation apparatus 110 may be a wireless network, a wired network, or a combination thereof.
An operation unit 107 is formed by a user interface such as a keyboard, a mouse, a joystick, a game pad, or a touch panel, and can be operated by the user to input various instructions to the CPU 101.
The CPU 101, the ROM 102, the RAM 103, the HDD 104, the display unit 105, the communication I/F 106, and the operation unit 107 are all connected to a bus 108. Note that the arrangement of the display control apparatus 100 shown in
The image generation apparatus 110 will be described next. In response to a request from the display control apparatus 100, the image generation apparatus 110 generates, based on images captured by one or more of the image capturing apparatuses 109-1 to 109-n, a virtual viewpoint image based on a virtual viewpoint. The image generation apparatus 110 transmits the generated virtual viewpoint image to the display control apparatus 100. The image generation apparatus 110 is formed by a computer apparatus such as a PC (Personal Computer), a smartphone, or a table terminal apparatus.
An example of the functional arrangement of the display control apparatus 100 will be described with reference to a block diagram shown in
As shown in
For example, assume that the user performs an operation of selecting (designating) the virtual viewpoint with the viewpoint ID=409 as the selected virtual viewpoint by operating the operation unit 107. In this case, the management unit 201 changes the viewpoint type corresponding to the viewpoint ID=409 to “selected virtual viewpoint”, and changes the viewpoint type corresponding to the viewpoint ID=408 to “switching virtual viewpoint”. When the user operates the operation unit 107 in this way, an arbitrary one of the six virtual viewpoints registered in the table information shown in
The management unit 201 reads out the viewpoint information of each switching virtual viewpoint registered in the table information, and outputs it to an output unit 202. The management unit 201 also reads out the viewpoint information of the selected virtual viewpoint registered in the table information. Then, the management unit 201 generates the second position by copying the position included in the readout viewpoint information, the second direction by copying the direction included in the viewpoint information, and the second angle of view by copying the angle of view included in the viewpoint information. The management unit 201 generates selected viewpoint information including the viewpoint type and the viewpoint ID included in the viewpoint information of the selected virtual viewpoint registered in the table information, the second position, the second direction, and the second angle of view, and outputs the selected viewpoint information to the output unit 202.
The output unit 202 requests the image generation apparatus 110 to generate virtual viewpoint images corresponding to the selected virtual viewpoint and the respective switching virtual viewpoints by outputting, to the image generation apparatus 110 via the communication I/F 106, the pieces of viewpoint information and the selected viewpoint information output from the management unit 201.
A reception unit 203 receives, from the image generation apparatus 110 via the communication I/F 106, the virtual viewpoint images (the virtual viewpoint image corresponding to the selected virtual viewpoint and the virtual viewpoint images corresponding to the switching virtual viewpoints) which have been generated by the image generation apparatus 110 in response to the request by the output unit 202. Note that each virtual viewpoint image output from the image generation apparatus 110 is attached with the viewpoint ID of the virtual viewpoint corresponding to the virtual viewpoint image.
A control unit 204 displays the virtual viewpoint images received by the reception unit 203 on the display screen of the display unit 105 without overlapping each other. For example, as shown in
Note that the display region 401 may correspond to the entire region of the display screen of the display unit 105 or a partial region of the display screen of the display unit 105 (for example, a region in a window displayed on the display unit 105).
An input unit 205 implements the operation of the selected virtual viewpoint by the user by performing an operation of changing at least one of the second position, the second direction, and the second angle of view included in the selected viewpoint information in response to an operation (user operation) input by operating the operation unit 107 by the user. That is, even if the user changes the position, direction, or angle of view of the selected virtual viewpoint by operating the operation unit 107, the change is reflected not on the table information but on generation of the virtual viewpoint image corresponding to the selected virtual viewpoint.
Furthermore, if the user performs an operation of selecting the selected virtual viewpoint by operating the operation unit 107, the input unit 205 instructs the management unit 201 to change the selected virtual viewpoint. The processing of changing the selected virtual viewpoint by the management unit 201 is as described above.
Note that a description will be made below by taking each functional unit (management unit 201, output unit 202, reception unit 203, control unit 204, or input unit 205) shown in
In
Processing executed by the system according to this embodiment will be described next with reference to a flowchart shown in
In step S301, the management unit 201 reads out the viewpoint information of each switching virtual viewpoint registered in the above table information, and generates selected viewpoint information from the viewpoint information of the selected virtual viewpoint registered in the above table information. The management unit 201 outputs each of the pieces of viewpoint information of the respective switching virtual viewpoints and the selected viewpoint information of the selected virtual viewpoint to the output unit 202.
In step S302, the output unit 202 determines whether all the pieces of viewpoint information of the switching virtual viewpoints output from the management unit 201 have been transmitted to the image generation apparatus 110. If, as a result of the determination processing, among the pieces of viewpoint information of the switching virtual viewpoints output from the management unit 201, the viewpoint information which has not been transmitted to the image generation apparatus 110 remains, the process advances to step S303. On the other hand, if, as a result of the determination processing, all the pieces of viewpoint information of the switching virtual viewpoints output from the management unit 201 have been transmitted to the image generation apparatus 110, the process advances to step S305.
In step S303, the output unit 202 outputs, among the pieces of viewpoint information of the switching virtual viewpoints output from the management unit 201, the untransmitted viewpoint information to the image generation apparatus 110 via the communication I/F 106. The image generation apparatus 110 generates a virtual viewpoint image based on the position, direction, and angle of view included in the viewpoint information output from the output unit 202 and images captured by one or more of the image capturing apparatuses 109-1 to 109-n. Then, the image generation apparatus 110 transmits, to the display control apparatus 100, the generated virtual viewpoint image attached with the viewpoint ID included in the viewpoint information.
In step S304, the reception unit 203 receives, via the communication I/F 106, the virtual viewpoint image transmitted from the image generation apparatus 110. Then, the control unit 204 displays the received virtual viewpoint image in the display region (one of the display regions 403 to 406) corresponding to the viewpoint ID attached to the virtual viewpoint image.
In step S305, if the user performs an operation input of changing the position/direction/angle of view of the selected virtual viewpoint by operating the operation unit 107, the input unit 205 changes the second position/second direction/second angle of view included in the selected viewpoint information in accordance with the operation input. If, for example, the user performs an operation input of changing the position of the selected virtual viewpoint by operating the operation unit 107, the input unit 205 changes the second position included in the selected viewpoint information in accordance with this operation input. If the user performs an operation input of changing the direction of the selected virtual viewpoint by operating the operation unit 107, the input unit 205 changes the second direction included in the selected viewpoint information in accordance with this operation input. If the user performs an operation input of changing the angle of view of the selected virtual viewpoint by operating the operation unit 107, the input unit 205 changes the second angle of view included in the selected viewpoint information in accordance with this operation input.
In step S306, the output unit 202 requests the image generation apparatus 110 to generate a virtual viewpoint image corresponding to the selected virtual viewpoint based on the selected viewpoint information by transmitting the selected viewpoint information to the image generation apparatus 110 via the communication I/F 106. In response to this request, the image generation apparatus 110 generates a virtual viewpoint image corresponding to the selected viewpoint information in the same manner as in step S303, and transmits the generated virtual viewpoint image attached with the viewpoint ID included in the selected viewpoint information to the display control apparatus 100.
In step S307, the reception unit 203 receives, via the communication I/F 106, the virtual viewpoint image transmitted from the image generation apparatus 110 in step S306. Then, the control unit 204 displays the received virtual viewpoint image in the display region (display region 402) corresponding to the viewpoint ID attached to the virtual viewpoint image.
In step S308, the input unit 205 determines whether the user has input a selected virtual viewpoint selection instruction by operating the operation unit 107. For example, assume that the user has performed an operation of designating the display region 405 among the display regions 403 to 406 shown in
If, as a result of the determination processing, the selected virtual viewpoint selection instruction has been input, the process advances to step S309; otherwise, the process advances to step S310.
In step S309, the management unit 201 changes, to “selected virtual viewpoint”, the viewpoint type managed by the table information for the switching virtual viewpoint corresponding to the virtual viewpoint selected in the display region 401 by operating the operation unit 107 by the user. Then, the management unit 201 changes, to “switching virtual viewpoint”, the viewpoint type managed by the table information for the virtual viewpoint image displayed in the display region 402. The management unit 201 generates the second position by copying the position managed by the table information for the virtual viewpoint whose viewpoint type has been changed to “selected virtual viewpoint”. The management unit 201 generates the second direction by copying the direction managed by the table information for the virtual viewpoint whose viewpoint type has been changed to “selected virtual viewpoint”. The management unit 201 generates the second angle of view by copying the angle of view managed by the table information for the virtual viewpoint whose viewpoint type has been changed to “selected virtual viewpoint”. That is, the second position, second direction, and second angle of view always represent the latest position, direction, and angle of view of the current selected virtual viewpoint, respectively. The management unit 201 generates selected viewpoint information including the second position, second direction, and second angle of view updated in step S309, and the viewpoint ID managed by the table information for the virtual viewpoint whose viewpoint type has been changed to “selected virtual viewpoint”. Note that the management unit 201 may set the position/direction/angle of view corresponding to the viewpoint ID set as the selected virtual viewpoint to the same values as those of the switching virtual viewpoint specified by a switching instruction.
As described above, the user (the operator of the virtual viewpoint) can operate the position, direction, and angle of view of the virtual viewpoint to obtain a desired virtual viewpoint image by operating the operation unit 107 while seeing the virtual viewpoint image displayed in the display region 402. Furthermore, the user can look for a desired viewpoint to obtain a desired virtual viewpoint image with reference to the display regions 403 to 406. If the virtual viewpoint image of the desired viewpoint is found, the virtual viewpoint image is displayed in the display region 402 by designating selection of the virtual viewpoint image, thereby making it possible to operate the virtual viewpoint. As shown in
In step S310, the input unit 205 determines whether the user has input an end instruction of the virtual viewpoint setting application by operating the operation unit 107. If, as a result of the determination processing, the end instruction of the virtual viewpoint setting application has been input, the processing according to the flowchart shown in
Note that in the table information shown in
As described above, according to this embodiment, a virtual viewpoint image corresponding to each switching virtual viewpoint is displayed as reference information to be referred to in order to select the switching virtual viewpoint as the selected virtual viewpoint next. Therefore, for example, it is possible to grasp a virtual viewpoint to be selected next more intuitively than displaying, as reference information, an image captured by each image capturing apparatus.
Note that the virtual viewpoint image (virtual viewpoint image corresponding to the selected virtual viewpoint) displayed in the display region 402 may be, for example, saved in the HDD 104 of the display control apparatus 100 or transmitted to an external apparatus via the communication I/F 106. In the latter case, for example, the virtual viewpoint image (virtual viewpoint image corresponding to the selected virtual viewpoint) displayed in the display region 402 may be broadcast for broadcasting via the communication I/F 106.
Since the operator of the virtual viewpoint need only be able to grasp the status of a target captured by the image capturing apparatus, for example, a simple virtual viewpoint image may be generated as a virtual viewpoint image. As the simple virtual viewpoint image, 3D CG (three-dimensional computer graphic) is used. First, the reception unit 203 acquires the positions of foregrounds (for example, players and a ball) in the captured image from the image generation apparatus 110. In the process of generating the virtual viewpoint image, processing of separating the foregrounds and backgrounds is normally performed. The positions of the foregrounds are generally obtained in the process. Therefore, the display control apparatus 100 can readily acquire the positions of the foregrounds from the image generation apparatus 110. Next, the display control apparatus 100 arranges the prepared 3D CG models of the players and ball at the positions acquired from the image generation apparatus 110. Assume that the 3D CG models of the field and stand of a stadium as backgrounds are additionally prepared. If these 3D models are input to a 3D CG rendering engine of the display control apparatus 100, it is possible to generate a simple virtual viewpoint image seen from the virtual viewpoint. Although it cannot be denied that this simple virtual viewpoint image is inferior to the normal virtual viewpoint image in terms of the expression capability, the operator can, for example, grasp the status of a performance and reduce the computer resource. In addition, for example, the virtual viewpoint image displayed in the display region 402 and the virtual viewpoint images corresponding to the switching virtual viewpoints displayed in the display regions 403 to 406 may be displayed at different frame rates.
As described above, according to this embodiment, the screen is displayed, in which the virtual viewpoint image corresponding to the selected virtual viewpoint selected by the user as an operation target among the plurality of virtual viewpoints and the virtual viewpoint images corresponding to the virtual viewpoints other than the selected virtual viewpoint among the plurality of virtual viewpoints are arranged without overlapping each other. This allows the operator to know, before switching to a given virtual viewpoint, what will be shown in a virtual viewpoint image after switching, thus appropriately selecting the virtual viewpoint.
In the following embodiments and modifications, differences from the first embodiment will be described, and the remaining is assumed to be the same as in the first embodiment unless it is specifically stated otherwise. In the first embodiment, one of the displayed virtual viewpoint images is designated to select, as the selected virtual viewpoint, the virtual viewpoint corresponding to the one designated virtual viewpoint image. However, a method of selecting the selected virtual viewpoint is not limited to the selection method described in the first embodiment. In this embodiment, a selection method different from that in the first embodiment will be described as the method of selecting the selected virtual viewpoint.
An example of the functional arrangement of a display control apparatus 100 according to this embodiment will be described with reference to a block diagram shown in
Then, an input unit 205 determines that the switching virtual viewpoint corresponding to the selection button pressed by the user has been selected as a selected virtual viewpoint. A management unit 201 changes the current selected virtual viewpoint in table information to a switching virtual viewpoint, and changes the switching virtual viewpoint corresponding to the selection button pressed by the user to a selected virtual viewpoint.
Processing executed by a system according to this embodiment will be described with reference to a flowchart shown in
In step S701, the processing unit 601 assigns the switching virtual viewpoint (in the case shown in
If, as described above, the display regions and the selection buttons are horizontally arranged, the display regions are assigned to the selection buttons in the arrangement order, for example, the display region at the left end is assigned to the selection button at the left end, the second display region from the left end is assigned to the second selection button from the left end, . . . . If the display regions and the selection buttons are vertically arranged, the display regions are assigned to the selection buttons in the arrangement order, for example, the display region at the upper end is assigned to the selection button at the upper end, the second display region from the upper end is assigned to the second selection button from the upper end, . . . . Note that a method of assigning a specific display region to a specific selection button is not limited to a specific one.
Note that if the number of switching virtual viewpoints is larger than that of selection buttons on the input device, for example, the switching virtual viewpoints are assigned from left as many as possible. As a method of managing assignment of a specific switching virtual viewpoint to a specific selection button, for example, in the table information, the viewpoint information of each switching virtual viewpoint may include the identification information of the selection button assigned to the switching virtual viewpoint. The management method is not limited to a specific one.
In step S308, the input unit 205 determines that the switching virtual viewpoint corresponding to the selection button pressed by the user has been selected as the selected virtual viewpoint (a selection instruction has been input).
The input device 801 is provided with a joystick 802 operated by the user to move the selected virtual viewpoint. The user can move the position and direction of the viewpoint by operating the joystick 802. Among the second position, second direction, and second angle of view included in the selected viewpoint information, the component corresponding to an operation input is changed by operating the joystick 802 by the user.
As described above, according to this embodiment, assignment of the switching virtual viewpoints to the selection buttons on the input device allows the operator of the virtual viewpoint to select the virtual viewpoint on the input device used to perform the operation of the virtual viewpoint without moving the hand to the mouse or the screen. This can select the virtual viewpoint quickly.
In this embodiment, it is possible to perform an editing operation on the table information. In this embodiment, the arrangement of the system according to the second embodiment is used. Processing executed by the system according to this embodiment will be described with reference to flowcharts shown in
In step S901, an input unit 205 determines whether the user has input an instruction (registration instruction) to register in the table information the position, direction, and angle of view of the current selected virtual viewpoint by operating an operation unit 107. A registration instruction input method is not limited to a specific one. For example, the registration instruction may be input when the user instructs a registration button displayed on a display screen of a display unit 105 by operating the operation unit 107 or when the user presses a specific button in the operation unit 107. If, as a result of the determination processing, the registration instruction has been input, the process advances to step S902; otherwise, the process advances to step S903.
In step S902, a management unit 201 issues a new viewpoint ID, generates new viewpoint information including the issued viewpoint ID, a viewpoint type “switching virtual viewpoint”, and the second position, second direction, and second angle of view included in the current selected viewpoint information, and registers the generated viewpoint information in the table information. The management unit 201 also registers, in the table information, information indicating which of display regions 403 to 406 is used to display a virtual viewpoint image corresponding to the registered virtual viewpoint or information indicating that the virtual viewpoint image is not displayed, or registers such information as that separated from the table information.
In step S903, the input unit 205 determines whether the user has input an instruction (rearrangement instruction) to change the arrangement order of the virtual viewpoint images displayed in the display regions 403 to 406 by operating the operation unit 107. If, as a result of the determination processing, the arrangement instruction has been input, the process advances to step S904; otherwise, the process advances to step S905.
In step S904, a control unit 204 changes the arrangement order of the virtual viewpoint images displayed in the display regions 403 to 406 in accordance with an operation input by the operating the operation unit 107 by the user. The operation performed by the user for this rearrangement is not limited to a specific operation. For example, if the user instructs a change button displayed on the display screen of the display unit 105 by operating the operation unit 107, the input unit 205 accepts the operation instruction for the virtual viewpoint images displayed in the display regions 403 to 406. Note that if the user presses a specific button in the operation unit 107, the input unit 205 may accept the operation instruction for the virtual viewpoint images displayed in the display regions 403 to 406.
Assume, for example, that the user drags the virtual viewpoint image in the display region 404 to move to the display region 406 by operating the operation unit 107. In this case, the control unit 204 moves the virtual viewpoint images displayed in the display regions 405 and 406 to the display regions 404 and 405, respectively. Furthermore, the control unit 204 moves, to the display region 406, the virtual viewpoint image (the virtual viewpoint image displayed in the display region 404) dragged to the display region 406. At this time, the management unit 201 changes the correspondence between each display region and each switching virtual viewpoint.
Assume, for example, that the user drags the virtual viewpoint image in the display region 404 to move to the outside of a display region 401 by operating the operation unit 107. In this case, the control unit 204 moves the virtual viewpoint images displayed in the display regions 405 and 406 to the display regions 404 and 405, respectively. The management unit 201 sets, in a non-display state, the viewpoint type of the virtual viewpoint image dragged and moved to the outside of the display region 401, and deletes a non-display state from the viewpoint type of the switching virtual viewpoint with a viewpoint ID=413 whose viewpoint type is set in the non-display state. Then, a virtual viewpoint image corresponding to the switching virtual viewpoint with the viewpoint ID=413 is generated and displayed in the display region 406, as described above. Note that there are provided various operations as an operation of rearranging the virtual viewpoint images corresponding to the switching virtual viewpoints, and the rearrangement operation is not limited to a specific one.
In step S905, the input unit 205 determines whether an instruction (deletion instruction) to delete one of the virtual viewpoint images displayed in the display regions 403 to 406 has been input by operating the operation unit 107 by the user. If, as a result of the determination processing, the deletion instruction has been input, the process advances to step S906; otherwise, the process advances to step S907.
In step S906, the management unit 201 deletes, from the table information, the viewpoint information of the virtual viewpoint as a target of the deletion instruction, and the control unit 204 erases, from the display screen of the display unit 105, the virtual viewpoint image corresponding to the virtual viewpoint as the target of the deletion instruction. An operation executed by the user for the deletion is not limited to a specific one. For example, if the user instructs a deletion button displayed on the display screen of the display unit 105 by operating the operation unit 107, the input unit 205 accepts an operation instruction for the virtual viewpoint images displayed in the display regions 403 to 406. Note that if the user presses a specific button in the operation unit 107, the input unit 205 may accept the operation instruction for the virtual viewpoint images displayed in the display regions 403 to 406.
If, for example, the user designates the display region 404 by operating the operation unit 107, the management unit 201 deletes, from the table information, the viewpoint information of the switching virtual viewpoint corresponding to the virtual viewpoint image displayed in the display region 404. Then, the control unit 204 erases the virtual viewpoint image from the display region 404.
In step S907, the input unit 205 determines whether the user has input an instruction (viewpoint set switching instruction) to switch the viewpoint information of the switching virtual viewpoint in the table information to the viewpoint information of another switching virtual viewpoint by operating the operation unit 107. If, for example, a viewpoint set switching button displayed on the display screen of the display unit 105 or a specific button in the operation unit 107 is pressed, the input unit 205 determines that the viewpoint set switching instruction has been input. If, as a result of the determination processing, the viewpoint set switching instruction has been input, the process advances to step S908: otherwise, the process advances to step S310.
In step S908, the management unit 201 creates in advance viewpoint information whose viewpoint type is “switching virtual viewpoint” in the table information, and replaces the created viewpoint information by one of virtual viewpoint sets registered in an HDD 104. The virtual viewpoint set indicates one or more pieces of viewpoint information of virtual viewpoints, as will be described alter. One or a plurality of virtual viewpoint sets may be registered in the HDD 104.
If one virtual viewpoint set is registered in the HDD 104, the management unit 201 replaces, by the one virtual viewpoint set, the viewpoint information whose viewpoint type is “switching virtual viewpoint” in the table information.
If a plurality of virtual viewpoint sets are registered in the HDD 104, the management unit 201 replaces, by one of the plurality of virtual viewpoint set, the viewpoint information whose viewpoint type is “switching virtual viewpoint” in the table information. A method of selecting one of the plurality of virtual viewpoint sets to be used for the replacement is not limited to a specific one. For example, the control unit 204 displays, on the display screen of the display unit 105, a list in which contents of the plurality of virtual viewpoint sets and file names are described. When the user selects one of the plurality of virtual viewpoint sets by operating the operation unit 107 while seeing the display screen, the management unit 201 replaces, by the selected virtual viewpoint set, the viewpoint information whose viewpoint type is “switching virtual viewpoint” in the table information.
The arrangement in which the plurality of virtual viewpoint sets are provided and one of the virtual viewpoint sets is selected and used assumes, for example, a case in which the operator of the virtual viewpoint uses the virtual viewpoint set in accordance with the status of a performance. For example, it is usable to prepare in advance a virtual viewpoint set for a corner kick, that for kick-off, that for a penalty kick, and the like. For the virtual viewpoint set for a penalty kick, it is convenient to prepare, as a set, a virtual viewpoint for seeing from the goal side to the direction of a penalty spot, a virtual viewpoint in an opposite direction, a virtual viewpoint for seeing the goal side over the penalty spot, a virtual viewpoint for seeing a bench, and the like. Note that if the table information is changed in step S908, the management unit 201 reads out the viewpoint information of each switching virtual viewpoint registered in the changed table information, and outputs the viewpoint information of each switching virtual viewpoint to the output unit 202.
In the first to third embodiments, the frame rate of the virtual viewpoint image displayed in the display region 402 is equal to that of the virtual viewpoint images displayed in the display regions 403 to 406. However, the frame rates of the virtual viewpoint images of all the virtual viewpoints need not be equal to each other. To generate a virtual viewpoint image by the image generation apparatus 110, it is necessary to read out many data and perform data processing. Consequently, an enormous computer resource is required to generate the virtual viewpoint images corresponding to all of the selected virtual viewpoint and switching virtual viewpoints at 60 fps (frames/sec), as described in the first to third embodiments, thereby increasing the cost. In this embodiment, frame rate control in consideration of the image generation capability of an image generation apparatus 110 will be described.
An example of the functional arrangement of a display control apparatus 100 according to this embodiment will be described with reference to a block diagram shown in
A processing unit 1001 sets the frame rate of a virtual viewpoint image corresponding to a selected virtual viewpoint and the frame rate of virtual viewpoint images corresponding to switching virtual viewpoints in accordance with the image generation capability of the image generation apparatus 110.
Processing executed by a system according to this embodiment will be described with reference to flowcharts shown in
In step S1101, the processing unit 1001 sets, for each virtual viewpoint registered in table information, the frame rate of a virtual viewpoint image corresponding to the virtual viewpoint. Note that in step S303 according to this embodiment, an output unit 202 outputs, to the image generation apparatus 110 via a communication I/F 106, viewpoint information corresponding to a virtual viewpoint image to be generated in the current frame among the pieces of viewpoint information of the switching virtual viewpoints output from a management unit 201. In step S306 according to this embodiment, if the current frame is a frame for generating a virtual viewpoint image corresponding to the selected virtual viewpoint, the output unit 202 transmits selected viewpoint information to the image generation apparatus 110 via the communication I/F 106.
The processing in step S1101 will be described using a practical example. Assume that the image generation capability of the image generation apparatus 110 is 120 fps (it is possible to generate images of 120 frames per sec). A case in which there exist the one selected virtual viewpoint and the four switching virtual viewpoints, as shown in
The virtual viewpoint image corresponding to the selected virtual viewpoint is an essential image for the operation of the selected virtual viewpoint, and it is necessary to display, as a virtual viewpoint image, a result of designating the selected virtual viewpoint without delay. It is understood that as the delay time from when the selected virtual viewpoint is designated until the virtual viewpoint image is displayed becomes longer, it becomes more difficult to operate the selected virtual viewpoint. Therefore, 60 fps is assigned to the selected virtual viewpoint, and the remaining 60 fps is distributed proportionally to the four switching virtual viewpoints. That is, as shown in
The virtual viewpoint image of each frame generated by the image generation apparatus 110 in steps S303 and S306 will be described with reference to
Furthermore, since the image generation capability of the image generation apparatus 110 is 120 fps and the frame rate of the virtual viewpoint image corresponding to each of the four switching virtual viewpoints is 15 fps, the virtual viewpoint image corresponding to each switching virtual viewpoint is generated and displayed once per eight frames. For example, in
Therefore, the output unit 202 outputs, to the image generation apparatus 110, viewpoint information of a virtual viewpoint corresponding to a virtual viewpoint image to be generated and displayed for the current frame, and the image generation apparatus 110 generates a virtual viewpoint image corresponding to the viewpoint information received from the output unit 202. By equally distributing the timings of generating the virtual viewpoint images, it is possible to display the images more smoothly.
Note that the above numerical values are merely examples used to describe the practical example. The above operation is not limited to the numerical values. That is, the numerical values are set appropriately in accordance with the capability of the system and the requirements of a broadcasting system that broadcasts a virtual viewpoint image.
As described above, according to this embodiment, the frame rate for generating a virtual viewpoint is distributed appropriately to the selected virtual viewpoint and the switching virtual viewpoints, thereby making it possible to implement the operation of the virtual viewpoint and selection of the virtual viewpoint according to the performance or the like without increasing the computer resource.
In the first to fourth embodiments, the virtual viewpoint image corresponding to the selected virtual viewpoint and the virtual viewpoint images corresponding to the switching virtual viewpoints are displayed on one display screen. The present invention, however, is not limited to this. For example, the virtual viewpoint image corresponding to the selected virtual viewpoint and the virtual viewpoint images corresponding to the switching virtual viewpoints may be displayed on separate display devices.
In the third and fourth embodiments, the virtual viewpoint registration processing, rearrangement processing, deletion processing, and viewpoint set switching processing are performed in the processing loop of generation and display of the virtual viewpoint image. The present invention, however, is not limited to this. For example, the virtual viewpoint registration processing, rearrangement processing, deletion processing, and viewpoint set switching processing may be performed in threads different from the processing loop of generation and display of the virtual viewpoint image.
Note that some or all of the above-described embodiments and modifications may be used in combination appropriately. Alternatively, some or all of the above-described embodiments and modifications may be selectively used.
An arrangement according to the present invention can allow the user to select an appropriate virtual viewpoint.
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
Number | Date | Country | Kind |
---|---|---|---|
2017-245392 | Dec 2017 | JP | national |
This application is a Continuation of International Patent Application No. PCT/JP2018/042948, filed Nov. 21, 2018, which claims the benefit of Japanese Patent Application No. 2017-245392, filed Dec. 21, 2017, both of which are hereby incorporated by reference herein in their entirety.
Number | Name | Date | Kind |
---|---|---|---|
20050018045 | Thomas et al. | Jan 2005 | A1 |
20110293241 | Umeyama | Dec 2011 | A1 |
20170293413 | Matsushita et al. | Oct 2017 | A1 |
Number | Date | Country |
---|---|---|
1578435 | Feb 2005 | CN |
101014123 | Aug 2007 | CN |
2005242606 | Sep 2005 | JP |
2011135138 | Jul 2011 | JP |
2012-015990 | Jan 2012 | JP |
2014-215828 | Nov 2014 | JP |
2017208702 | Nov 2017 | JP |
2017057006 | Apr 2017 | WO |
2017204174 | Nov 2017 | WO |
Entry |
---|
Qiang Yao, et al., Automatic Camera Self-Calibration for Immersive Navigation of Free Viewpoint Sports Video, 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), 2016, pp. 1-6, Montreal, QC, XP33041409A. |
Zhebin Zhang, et al., A Virtual View Genetation Method for Free-Viewpoint Video System, 2007 International Symposium on Intelligent Signal Processing and Communication Systems, Nov. 28-Dec. 1, 2007, pp. 156-159, Xiamen, China, X31211491A. |
Number | Date | Country | |
---|---|---|---|
20200105174 A1 | Apr 2020 | US |
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2018/042948 | Nov 2018 | US |
Child | 16701910 | US |