Patent Application
20240404178
The present technology relates to an information processing device, an information processing method, and a program.
Technologies for enhancing comfort of a user in control of display of a virtual space are known. For example, the following Patent Document 1 discloses a technology for reducing discomfort such as motion sickness (so-called virtual reality sickness (VR sickness)) of a user, which is likely to occur during viewpoint change accompanied by continuous movement of the user in a virtual space.
Meanwhile, as a method of changing the viewpoint in the virtual space, there is a method of performing movement (discrete movement) that is not accompanied by continuous movement of a user. In this case, the motion sickness of the user can be reduced as compared with the viewpoint change accompanied by the continuous movement, but it is difficult to grasp a self-state (for example, a position, an orientation, and the like) in the virtual space. Furthermore, it is difficult for the user to confirm in advance an appearance of the virtual space at a viewpoint change destination. In this manner, there is room for improving the comfort for the user in the conventional technology.
An object of the present technology is to propose an information processing device, an information processing method, and a program capable of displaying an image comfortable for a user.
The present technology relates to
The present technology relates to
The present technology relates to a
An embodiment to be described hereinafter is a preferred specific example of the present technology, and various technically preferable limitations are given. However, the scope of the present technology is not limited to the following embodiment unless otherwise specified in the following description. Note that components having substantially the same functional configuration in the present specification and the drawings will be denoted by the same reference sign, and the redundant description thereof will be omitted as appropriate. The present technology will be described in the following order.
First, an outline of an information processing system according to the present embodiment will be described with reference to
The control device 10 generates a display image of a viewpoint (first-person viewpoint) of the user U using an image of the virtual space. The image of the virtual space is an image of virtual reality (VR), augmented reality (AR), mixed reality (MR) including these, or the like. The image includes a still image and a moving image (video). Note that the image of the virtual space may be an image obtained by capturing an image of an actual scene. That is, the image of the virtual space may be an image representing a captured subject, specifically, an actually captured image, a computer graphic generated on the basis of the actually captured image, or the like. The display image is an image obtained in a case where the virtual space is imaged by virtual cameras installed at predetermined positions in the virtual space. For example, the display image means an image seen in a field of view of the user U in the virtual space. Specifically, the control device 10 sets a virtual user viewpoint that is the viewpoint of the user U in the virtual space, and generates the display image seen from the virtual user viewpoint. Here, the virtual user viewpoint is specified on the basis of attitude detection information transmitted from the display device 20. The attitude detection information is generated by an attitude detection unit incorporated in the display device 20. Then, the control device 10 causes the display device 20 to display the generated display image, thereby causing the display device 20 to display the virtual space.
The display device 20 is a so-called head-mounted display (HMD), and is worn around the eyes of the user U. The above-described display image is displayed on the display device 20. Then, when the virtual user viewpoint is changed by a viewpoint change operation (for example, moving the face, performing an operation of instructing “so-called warp movement (discrete movement)”, or the like) by the user U, the display image is changed accordingly. Therefore, the user U can enjoy a virtual experience as if the user U exists in the virtual space.
The operation device 30 is a device (for example, a game controller or the like) that is operated for the control in the control device 10 according to an operation intention of the user U. Specifically, the operation device 30 is operated by being held in the hand of the user U as illustrated in
The communication line 40 connects the control device 10 and the display device 20 and enables information communication therebetween. The communication line 50 connects the control device 10 and the operation device 30 and enables information communication therebetween. The communication lines 40 and 50 may adopt any connection scheme such as a wired or wireless scheme. Note that any combination of connections may be adopted as long as each of the display device 20 and the operation device 30 is connected to the control device 10. For example, the combination of connections is not limited to one illustrated in
Here, the information processing system 1 may be configured such that at least two or more of the control device 10, the display device 20, and the operation device 30 are integrated. For example, the display device 20 may have the function of the control device 10. Furthermore, the display device 20 is not limited to a head-mounted display, and may be another type of display. For example, the display device 20 may be a stationary display or a display of a mobile terminal (for example, a mobile phone, a smartphone, a smart tablet, a portable game machine, or the like). In this case, the attitude detection unit of the display device 20 described above can be achieved, for example, by providing an imaging device on these display and mobile terminals.
In a conventional information processing system, a situation that is not comfortable for a user sometimes occurs in a case where a virtual space accompanied by viewpoint change is displayed. For example, viewpoint change accompanied by continuous movement of the user in the virtual space causes motion sickness of the user. Therefore, Patent Document 1 described above discloses a technology for reducing such discomfort.
Specifically, first, a map image is arranged in the virtual space, and a target point (viewpoint change position) image and a character image are arranged on the map image. Then, the character image is moved to the target point image that is being gazed by the user. That is, the character image is moved following a virtual user viewpoint. Next, an actual scene image (scene image) is displayed in the virtual space around the character image, and the map image and the character image are enlarged with the character image as the center. Then, a display image seen from the virtual user viewpoint is generated and displayed. Therefore, the actual scene image is displayed in front of the eyes. Then, a target point image closest to a gaze point image that is being gazed by the user of the actual scene image is specified to generate a moving viewpoint image (an actual scene image seen from the virtual user viewpoint moving from a gaze point to a target point), the generated image is displayed on an image display frame arranged in the virtual space, thereby performing the viewpoint change accompanied by continuous movement of the user. At this time, the image is displayed in a state where the actual scene image around the image display frame is fixed to mitigate an uncomfortable feeling, such as the motion sickness of the user, and to prevent a feeling of unexpected movement during the viewpoint change.
In this technology, it is necessary to display the moving viewpoint image, and it is difficult to say that viewpoint change with discrete movement is performed. Furthermore, it is difficult to set the viewpoint by searching for a position of an appearance desired by the user at an any place in the virtual space. Therefore, in the embodiment of the present technology, when the display of the virtual space accompanied by viewpoint change is controlled, the control is performed to display a model of the virtual space in a manner changeable to any appearance desired by the user and to display the virtual space by moving the virtual user viewpoint to a position on the virtual space at which the same appearance as an appearance of the model is obtained. Hereinafter, the information processing system 1 according to the present embodiment will be described in detail.
The communication unit 11 includes, for example, a communication device or the like, and communicates with the display device 20 via the communication line 40 (see
The communication unit 11 outputs the information (for example, attitude detection information given from the display device 20, and operation information and attitude detection information given from the operation device 30), obtained by each communication described above, to the control unit 13. Furthermore, the communication unit 11 transmits information given from the control unit 13, for example, a display image to the display device 20.
The storage unit 12 includes, for example, a ROM, a RAM, a hard disk, and the like, and stores information necessary for the processing by the control device 10. For example, the storage unit 12 stores a program, various images, and the like. Here, images to be stored in the storage unit 12 include an image of a virtual space and the like.
The control unit 13 includes, for example, a CPU and the like, and controls each component of the control device 10. Furthermore, the control unit 13 generates a display image to be displayed on the display device 20, and outputs the display image to the communication unit 11. Specifically, the control unit 13 sets a virtual user viewpoint that is a viewpoint of a user in the virtual space, and generates the display image seen from the virtual user viewpoint. At that time, the control unit 13 generates a model of the virtual space, and causes the generated model to be included in the display image and displayed. For example, the model is generated on the basis of the image of the virtual space, the attitude detection information given from the operation device 30, and the like.
Here, the model of the virtual space is a target that is included in the virtual space and used as a reference for viewpoint change. For example, in a case where the virtual space is a live venue, examples of the target include the entire live venue, a stage portion, a specific artist, and the like. Specifically, this model is enlarged or reduced with respect to a display size (real size) of the virtual space. Therefore, the model can be displayed in an appropriate size regardless of the real size. Note that the equal magnification may be applied. Furthermore, this model includes the entire virtual space displayed by the display images, or is selected by the user from the virtual space displayed by the display images. This model is preferably a copy of the virtual space. Therefore, it is possible to eliminate an uncomfortable feeling in a case where the display image is switched in accordance with the viewpoint change as described later.
The communication unit 21 includes, for example, a communication device or the like, and communicates with the control device 10 via the communication line 40. The communication unit 21 outputs information obtained by each communication described above, for example, a display image, to the display control unit 24. Furthermore, the communication unit 21 transmits information given from the display control unit 24, for example, attitude detection information as described later, to the control device 10.
The attitude detection unit 22 includes, for example, an attitude detection sensor, and detects an attitude of display device 20. The attitude detection sensor may be, for example, a gyro sensor. The attitude detection unit 22 generates attitude detection information regarding the attitude of the display device 20, and outputs the attitude detection information to the display control unit 24. Here, the attitude of the display device 20 changes depending on an orientation of the head of the user U. Then, a viewpoint of the user U can be roughly specified on the basis of the orientation of the head of the user U. Therefore, the attitude detection information is used to specify a virtual user viewpoint in the present embodiment. In this manner, the virtual user viewpoint is specified on the basis of the attitude detection information by the attitude detection unit 22. Therefore, the user U can move the virtual user viewpoint by changing the orientation of the head.
Here, the attitude detection sensor may be a line-of-sight detection sensor (that is, an imaging device that captures an image of the vicinity of user's eyes) that detects a line of sight of the user U. In this case, the attitude detection sensor can directly detect the line-of-sight of the user U. Furthermore, the user U can move the virtual user viewpoint by changing the line-of-sight.
The display unit 23 includes, for example, a head-mounted display panel, and displays the display image or the like under the control of the display control unit 24. The display control unit 24 includes, for example, a CPU and the like, and controls each component of the display device 20. Furthermore, the display control unit 24 causes the display unit 23 to display the display image provided from the communication unit 21. Furthermore, the display control unit 24 outputs the attitude detection information provided from the attitude detection unit 22 to the communication unit 21.
The communication unit 31 includes, for example, a communication device or the like, and communicates with the control device 10 via the communication line 50. The communication unit 31 transmits information given from the operation control unit 34, for example, attitude detection information and operation information, to the control device 10.
The attitude detection unit 32 includes, for example, an attitude detection sensor, and detects an attitude of the operation device 30. The attitude detection sensor may be, for example, a gyro sensor. The attitude detection unit 32 generates attitude detection information regarding the attitude of the operation device 30 and outputs the attitude detection information to the operation control unit 34. Here, the attitude of the operation device 30 changes in accordance with a motion of the hand of the user U holding the operation device 30. Therefore, this attitude detection information is used to move the above-described model in the present embodiment. Note that the information used at the time of moving the model may be operation information generated as the user U operates the operation unit 33.
The operation unit 33 includes, for example, an operation switch such as an operation button or an operation lever, generates operation information according to a user operation, and provides the generated operation information to the operation control unit 34. The operation control unit 34 includes, for example, a CPU and the like, and controls each component of the operation device 30. Furthermore, the operation control unit 34 outputs information given from the attitude detection unit 32, for example, the attitude detection information and information given from the operation unit 33, for example, the operation information, to the communication unit 31.
Note that the information processing system 1 may include a sensor (for example, an imaging device) that detects an attitude of the entire body of the user U. Then, the control unit 13 of the control device 10 may specify a virtual user viewpoint on the basis of detection information given from the sensor. That is, in this case, the control unit 13 uses orientations of the user's face, body, and line-of-sight in a comprehensive manner. Similarly, the control unit 13 may specify an attitude of the operation device 30 on the basis of the detection information given from the sensor.
Next, state transition of a display image accompanied by viewpoint change will be described. Note that a case where a virtual reality space (VR space) is used as the above-described virtual space will be described as an example in the following description. Furthermore, a case where a miniature model in which a model including all display objects in the VR space is reduced with respect to the real size is displayed will be described as an example.
As illustrated in
Then, when an instruction to turn on miniature display is made from the initial state A, a miniature model M of the VR space S is displayed together with the VR space S described above (a miniature display state B) as illustrated in
Note that the miniature model M is preferentially displayed so as to be seen at a position closer to the user U than the VR space S. For example, the miniature model M is displayed at a position where the center position thereof is closer to the user U than the center position of the VR space S. Therefore, the miniature model M can be made to be easily seen by the user U. As illustrated in the drawing, the miniature model M is displayed specifically in the vicinity of the user U, more specifically, close to the hand of the user U. Note that the display position of the miniature model M is not limited to these positions.
Then, a positional relationship between a virtual user viewpoint and the miniature model M is moved (a viewpoint-moved state C) by the user U from the miniature display state B as illustrated in
In the example illustrated in
In the example illustrated in
Specifically, the position of the virtual user viewpoint relative to the miniature model M is represented by a coordinate position (miniature coordinate system viewpoint position) with a predetermined position in the virtual space as a reference point. Then, a position of the virtual user viewpoint in the VR space corresponding to the viewpoint position (VR space coordinate system viewpoint position) is determined in response to the determination of the miniature coordinate system viewpoint position. Details of this correspondence will be described later. Specifically, the position of the virtual user viewpoint in the VR space is represented by a coordinate position (VR space coordinate system viewpoint position) with a predetermined position in the virtual space as a reference point.
Next, the VR space S is deleted (a VR space-deleted state E) as illustrated in
Furthermore, a movement effect on the display of the display image is started as illustrated in
After the VR space S after the viewpoint movement is displayed, the display of the miniature model M is deleted as illustrated in
In a case where the viewpoint change is performed again at a viewpoint change destination, the miniature model M is displayed again (a miniature display state B1) to control the display image as illustrated in
The miniature model M is not limited to one including all the display objects in the predetermined VR space S as described with reference to
The control of the display image accompanied by the viewpoint change in this case is started from a state (the initial state A) in which the VR space S is displayed by the display image as illustrated in
Then, when an instruction to turn on miniature display is made from the initial state A, the miniature model M selected from among the display objects of the VR space S is displayed together with the VR space S described above (the miniature display state B) as illustrated in
Then, a positional relationship between a virtual user viewpoint and the miniature model M is moved (the viewpoint-moved state C) by the user U from the miniature display state B, and any positional relationship desired by the user U is determined as illustrated in
Then, a miniature coordinate system viewpoint position is determined in response to the determination of the viewpoint position, and a VR space coordinate system viewpoint position corresponding to the miniature coordinate system viewpoint position is determined.
Next, the VR space S is deleted (specifically, displayed in dark) (the VR space-deleted state E) as illustrated in
After the VR space S after the viewpoint movement is displayed, the display of the miniature model M is deleted as illustrated in
Next, display control processing by the control device 10 that performs the control of the display image will be described in detail.
In this display control processing, first, control is performed to display the miniature model M in response to an instruction for miniature display (step S1). For example, the instruction for the miniature display is instructed by the user U using the operation device 30.
Next, it is determined whether or not the user U is viewing the miniature model M on the basis of attitude detection information or the like given from the display device 20, and a background process of blurring the background of the miniature model M so as not to be conspicuous is performed in a case where it is determined that the user is viewing (step S2). When the background is blurred, the miniature model M can be focused on by the user U. Note that this background process is not limited to the blurring, and may be performed by display with white fill or the like, and may be appropriately performed as necessary.
Next, it is determined whether or not the user is gazing at a target to be a reference for the viewpoint change, that is, the miniature model M (step S3). In a case where it is determined in step S3 that the target is not being gazed (NO), the process in step S3 is repeated until it is determined that the target is being gazed. For example, in a case where it is determined that the target is not being gazed even after a lapse of a predetermined period of time, the miniature model M may be deleted to end the processing.
In a case where it is determined in step S3 that the target is being gazed (YES), a position of the user viewpoint in a world coordinate system (VR space coordinate system viewpoint position) is calculated on the basis of a position of the user viewpoint in a local coordinate system (miniature coordinate system viewpoint position) determined as described above (step S4). Specifically, the position of the user viewpoint in the world coordinate system (VR space coordinate system viewpoint position) is calculated on the basis of a position vector of the user viewpoint in the local coordinate system. This VR space coordinate system viewpoint position can be obtained by a known calculation method.
As illustrated in
Next, the VR space S is deleted (specifically, darkened) by bringing out a movement effect in the periphery of the field of view with the gaze point as the center (step S6). That is, the VR space S around the miniature model M gazed by the user U is deleted.
In a case where the 3D display is performed as described above, a stereoscopic effect is strongly felt when the distance to a display object is short, but the stereoscopic effect is weakened when the distance to the display object is long. That is, a far display object appears to be more planar than a near display object. Therefore, when the miniature model M is to be deleted as described above, the uncomfortable feeling is generated due to a sudden change in the stereoscopic effect if the deletion is suddenly performed to switch to the real size of the VR space S. Therefore, in the present embodiment, the stereoscopic effect of the miniature model M is gradually made to be the same as the stereoscopic effect of the VR space S (target V) after deletion of the miniature model M before the miniature model M is deleted to display the VR space from which the miniature model M is deleted instead of suddenly performing the deletion to switch the display at the time of deleting the miniature model M to display the VR space from which the miniature model M is deleted. Specifically, the stereoscopic effect is adjusted by gradually changing the binocular parallax of the display image described above. That is, the binocular parallax on the display panel is gradually changed to gradually make the stereoscopic effect be the same as the stereoscopic effect at the time of viewing the VR space S. Specifically, the binocular parallax on the display panel can be changed by changing the interocular distance. For example, the interocular distance can be controlled by adjusting the distance between the virtual cameras when the display image is generated. That is, the interocular distance can be changed by adjusting the distance between a left-eye virtual camera and a right-eye virtual camera by software.
Therefore, as illustrated in
As described above, the uncomfortable feeling is generated if the sudden switching from the state on the left side to the state on the right side in
In this case, the interocular distance IPD is changed until the binocular parallax P1 is gradually changed to P2 before the miniature model M is deleted. For example, in the case of the illustrated example, the stereoscopic effect is gradually weakened. At this time, it is preferable to set a mode and time of a change (for example, at a degree that the user U does not notice) that do not cause the uncomfortable feeling. Therefore, it is possible to reduce the uncomfortable feeling caused by a sudden change in the sense of distance to the target V in the VR space S when the miniature model M has been deleted. Therefore, the fact that the stereoscopic effect gradually changes (is weakened in the illustrated example) is the only difference in appearance (change in image) so that natural viewpoint change becomes possible according to the present embodiment.
As described above, under the control of the control unit 13, the user U moves seamlessly (while viewing the miniature model M) to a place at which the same appearance as that of the viewpoint seen from the miniature model M is obtained. Since the viewpoint is seamlessly changed by confirming the appearance with the miniature model M, the viewpoint is changed while grasping a self-position in the VR space (without requiring continuous movement). Therefore, discomfort such as motion sickness of the user can be dramatically mitigated. That is, when the miniature model M is deleted, what is the same as the miniature model M that has been viewed is displayed as being viewed, and thus, VR motion sickness is not caused and confusion is not caused.
As described above, during control of the display of the VR space S accompanied by the viewpoint change, the control unit 13 performs control to display the miniature model M of the VR space S in a manner changeable to any appearance desired by the user U and to display the VR space S by moving the virtual user viewpoint to a position on the VR space S at which the same appearance as an appearance of the miniature model M is obtained.
Therefore, the user U can view the miniature model M from any position and freely select the position of the virtual user viewpoint. Since the appearance is the same as the appearance of the miniature model M, it is possible to continuously confirm the appearance even if the position of the virtual user viewpoint is discontinuous (the viewpoint is discretely changed). Furthermore, it is possible to move to a desired viewpoint position without being accompanied by continuous movement and without recognizing discontinuity. That is, it is possible to move the viewpoint without being accompanied by continuous movement while grasping a self-position in the real size space.
Although the embodiment of the present technology has been specifically described above, the present technology is not limited to the above-described embodiment, and various modifications based on the technical idea of the present technology are possible. For example, various modifications as described below are possible. Furthermore, one or a plurality of arbitrarily selected modes of the modifications to be described below can be also appropriately combined. Furthermore, the configurations, methods, steps, shapes, materials, numerical values, and the like of the above-described embodiment can be combined with each other without departing from the gist of the present technology.
The display image has been described as the stereoscopic view image (3D image) in the above-described embodiment, but can be a 2D image.
Although the miniature model obtained by reducing the target V has been exemplified as the model of the VR space in the above-described embodiment, the model can be obtained by equal magnification or enlargement of the target V.
Although the copy of the target V has been exemplified as the model of the VR space in the above-described embodiment, the model of the VR space may be one that is clearer than the target V, may be one that is obtained by simplifying or deforming the target V, or the like.
Note that the present technology can also have the following configurations.
(1)
An information processing device including
The information processing device according to (1), in which
The information processing device according to (2), in which
The information processing device according to (3), in which
The information processing device according to (4), in which
The information processing device according to any one of (1) to (5), in which
The information processing device according to any one of (1) to (6), in which
The information processing device according to any one of (1) to (7), in which
The information processing device according to any one of (1) to (8), in which
The information processing device according to any one of (1) to (9), in which
The information processing device according to any one of (1) to (10), in which
The information processing device according to (11), in which
The information processing device according to any one of (1) to (12), in which
The information processing device according to any one of (1) to (13), in which
The information processing device according to any one of (1) to (14), in which
An information processing method including
A program causing a computer to
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-060165 | Mar 2020 | JP | national |
The present application is a continuation application of U.S. patent application Ser. No. 17/906,525, filed Sep. 16, 2022, which is a National Stage Entry of Patent Application No. PCT/JP2021/012319 filed Mar. 24, 2021, which claims priority from prior Japanese Patent Application JP 2020-060165 filed in the Japan Patent Office on Mar. 30, 2020, the entire contents of which are hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17906525 | Sep 2022 | US |
| Child | 18800708 | US |
