The invention relates to a non-transitory computer-readable storage medium, a display apparatus, a head-mounted display apparatus, and a marker.
An HMD (head mounted display) is one type of apparatus for experiencing a virtual space. A user can wear the HMD on their head to observe a video image of a virtual space that is presented before their eyes. For example, PTL1 discloses a technique for executing specific processing corresponding to an orientation of an operation unit, along with displaying an image corresponding to the orientation of the head of an operator on a head-mounted display worn on the head.
PTL1: Japanese Patent No. 5944600
In the technique disclosed by PTL1, a motion sensor for detecting motion of the head of an operator is used in order to display an image corresponding to an orientation of the head of the operator. The present invention provides, with simple processing using a captured image, a technique for causing an image in accordance with a position or an orientation of the head of an operator to be displayed, even if such a motion sensor is not used.
An aspect of the present invention provides a non-transitory computer-readable storage medium storing a computer program for causing a computer of a display apparatus to function as: a detection unit configured to detect, from a moving image obtained by capturing a marker, a change of a shape of an image region of the marker; and a display control unit configured to control a shape of a display image in accordance with the change detected by the detection unit and cause a display screen of the display apparatus to display the display image after the controlling.
Another aspect of the present invention provides a display apparatus, comprising: a detection unit configured to detect, from a moving image obtained by capturing a marker, a change of a shape of an image region of the marker; and a display control unit configured to control a shape of a display image in accordance with the change detected by the detection unit and cause a display screen of the display apparatus to display the display image after the controlling.
Still another aspect of the present invention provides a head-mounted display apparatus having a display apparatus, the display apparatus comprising: a detection unit configured to detect, from a moving image obtained by capturing a marker, a change of a shape of an image region of the marker; and a display control unit configured to control a shape of a display image in accordance with the change detected by the detection unit and cause a display screen of the display apparatus to display the display image after the controlling.
Yet another aspect of the present invention provides a marker in which a first rectangle and a second rectangle, which results from shrinking and rotating the first rectangle in order for the second rectangle to be inscribed in the first rectangle, are arranged, wherein a defined mark is arranged in the second rectangle.
By virtue of a configuration of the invention, with simple processing using a captured image, it is possible to cause an image in accordance with a position or an orientation of a head of an operator to be displayed, even if a motion sensor is not used.
Further features and advantages of the present invention will become apparent from the following description with reference to the attached drawings. Note, in the accompanying drawings, the same reference numerals are added for same or similar configuration elements.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
Below, explanation will be given for embodiments of the present invention with reference to the accompanying drawings. Note that the embodiments to be described below are examples of detailed implementation of the present invention or detailed examples of the arrangement described in the claims.
In the present embodiment, a display image (may be a still image, may be a moving image, and may be both of these) is displayed on a display screen of an HMD, but a shape of the display image is controlled in accordance with a change of a shape of a marker that is provided on a wall surface and is in a captured image captured by a camera provided in the HMD. In the following description, it is assumed that “shape” includes the meanings of both “outer shape” and “size”.
Firstly,
The display apparatus 101 is a device that has a display screen which is capable of displaying text or an image based on data that it holds or is received from an external unit, and for the display apparatus 101, it is possible to apply a smart phone, a tablet terminal device, or the like, for example. Reference numeral 114 denotes a camera, and it captures a moving image of a physical space. The display screen of the display apparatus 101 is provided on a surface opposite to a surface to which a camera 114 is attached.
The HMD main body portion 102 is a device to which it is possible to attach the display apparatus 101 so that the display screen of the display apparatus 101 is positioned before the eyes of a user who wears the HMD 100. Reference numerals 190a and 190b in the HMD main body portion 102 denote speakers for outputting audio to a right ear and a left ear, respectively, of a user who wears the HMD 100.
An example of wearing the HMD 100 is illustrated in
Note that
Next, description is given using the block diagram of
Firstly, description is given regarding an example of the hardware configuration of the display apparatus 101.
A CPU 111 uses a computer program and data stored in a RAM 112 to perform operation control of the entirety of the display apparatus 101 and to execute or control various processing that is described later as something that the display apparatus 101 performs.
The RAM 112 has an area for storing a computer program and data which are loaded from a non-volatile memory 113, and a work area used when the CPU 111 executes various processing. Furthermore, it also has an area for storing a computer program or data received from an external device if the display apparatus 101 is a device capable of data communication with an external device. In this way, the RAM 112 can appropriately provide various areas.
An OS (operating system), and computer programs and data for causing the CPU 111 to execute or control various processing described later as something the display apparatus 101 performs are saved in the non-volatile memory 113. A computer program for detecting a region of a marker in a captured image, a display image reproducing program, an audio reproducing program, and the like are included in the computer programs saved in the non-volatile memory 113. In addition, data of a display image, audio data, information handled as known information in the following description, and the like is included in the data saved in the non-volatile memory 113. Note that, if the display apparatus 101 is a device that is capable of data communication with an external device, the above-described various programs and data or the like may be received from the external device and saved in the non-volatile memory 113.
The computer programs and data saved in the non-volatile memory 113 are appropriately loaded to the RAM 112 in accordance with control by the CPU 111, and become a target of processing by the CPU 111.
The camera 114 is something attached to a surface that is opposite to that of a display screen 115 in the display apparatus 101, and captures a moving image of a physical space. An image (a captured image) of each frame is outputted from the camera 114, and the outputted captured image is saved to the non-volatile memory 113.
The display screen 115 is configured by a liquid crystal screen, a touch panel screen, or the like, and is capable of displaying a result of processing by the CPU 111 with an image, text, or the like.
An OF (interface) 116 is an interface for output of audio to the HMD main body portion 102, and, as described later, is configured so that data communication with an OF 121 on a side of the HMD main body portion 102 is possible. Data communication between the OF 116 and the OF 121 may be wireless communication and may be wired communication.
The CPU 111, the RAM 112, the non-volatile memory 113, the camera 114, the display screen 115, and the OF 116 are each connected to a bus 117.
Next, description is given regarding an example of the hardware configuration of the HMD main body portion 102.
The OF 121 is an interface for receiving an audio signal outputted from the display apparatus 101 via the OF 116 and sending the received audio signal to an audio output unit 122, and is configured so that data communication with the OF 116 described above is possible.
The audio output unit 122 is something for outputting an audio signal received from the OF 121 as audio. The audio output unit 122 includes the above-described speaker 190a for a right ear and speaker 190b for a left ear, and outputs audio based on audio signals with respect to respective speakers.
Next, description using
Next, a flow chart of
In step S601, the CPU 111 obtains a captured image (a captured image of an Nth frame) outputted from the camera 114, and saves the obtained captured image in the non-volatile memory 113.
In step S602, the CPU 111 detects, as a marker region, an image region in which the marker 301 is appearing in the captured image of the Nth frame (the captured image of the current frame) obtained in step S601. There are various techniques for a technique for detecting a marker region from a captured image, and, in this step, any technique may be used to detect the marker region from the captured image. For example, a recognition model that has learned the marker in advance may be used to detect, as a marker region, an image region of the marker from the captured image, and, by a pattern matching technique, an image region from the captured image that most matches an image of the marker may be detected as a marker region.
In step S603, the CPU 111 determines whether or not the captured image obtained in step S601 is a captured image for which a marker region was first detected after activation of the display apparatus 101. As a result of this determination, if the captured image obtained in step S601 is a captured image for which a marker region was first detected after activation of the display apparatus 101, the processing proceeds to step S604, and if the captured image obtained in step S601 is a second or subsequent captured image for which a marker region was detected after activation of the display apparatus 101, the processing proceeds to step S605.
In step S604, the CPU 111 stores in the RAM 112 the marker region detected in step S602, in other words shape information defining a shape of the marker region first detected after activation of the display apparatus 101. The shape information is image coordinates of four corners of the marker region in the captured image, for example. Furthermore, the CPU 111 displays a display image having a base shape (may be referred to as a base display image) on the display screen 115. For the description below, description is given for a display image 502 illustrated in
Meanwhile, in step S605, the CPU 111 detects a change from the shape indicated by the shape information stored in the RAM 112 in step S604 to a shape of the marker region detected in step S602 for the captured image of the Nth frame.
In step S606, the CPU 111 controls the shape of the base display image in accordance with the change of the shape obtained in step S605 to generate a new display image, and causes the display screen 115 to display the generated display image.
Here, the processing in step S605 and step S606 is described by taking
For example, assume that, by capturing the marker 301 from in front at a position separated by a distance d1 in the normal direction of the marker 301 from the position of the marker 301, a captured image 401a of
Here, when the marker 301 is captured from in front at a position separated by a distance d2 (d2>d1) in the normal direction of the marker 301 from the position of the marker 301, as illustrated by
In addition, when a position separated from the position of the marker 301 by a distance d1 in the normal direction of the marker 301 is set as a position P, when the marker 301 is captured from a position separated from the position P by an appropriate distance to the left when facing the marker 301, as illustrated in
In addition, when the marker 301 is captured from a position separated from the position P by an appropriate distance to the right when facing the marker 301, as illustrated by
In this way, in accordance with a positional relationship between the marker 301 and the camera 114, the appearance of the marker 301 in a captured image changes. In the present embodiment, when a marker region is first detected from a captured image after activation of the display apparatus 101, a base display image (here the display image 502) is displayed on the display screen 115, irrespective of the shape of the detected marker region. In other words, when
For example, assume that a captured image in which a marker region was first detected after activation of the display apparatus 101 is the captured image 402a of
Subsequently, assume that the captured image 401a is obtained as the captured image of the Nth frame. In this case, a change from the shape of the marker 402b to the shape of the marker 401b is obtained. There are various methods for a method of obtaining the shape change, but an example thereof is given here. Firstly, image coordinates of the four corners of the marker 401b in the captured image 401a are obtained. A coordinate conversion parameter (a region transformation parameter for causing the shape of the marker region of the marker 402b to transform to the shape of the marker region of the marker 401b) for conversion from the image coordinates of the four corners of the marker 402b which is indicated by the shape information to the image coordinates of the four corners of the marker 401b is obtained. By using the coordinate conversion parameter to convert the image coordinates of the four corners of the display image 502, a display image 501 into which the display image 502 is transformed is generated, and the display screen 115 is caused to display the generated display image 501. As a result, as illustrated by
In addition, in a case where the captured image for which a marker region is first detected after the activation of the display apparatus 101 is the captured image 401a of
In addition, in a case where the captured image for which a marker region is first detected after the activation of the display apparatus 101 is the captured image 401a of
In addition, in a case where the captured image for which a marker region is first detected after the activation of the display apparatus 101 is the captured image 401a of
By virtue of such a configuration, in a case of performing an image display in accordance with a position and orientation of an HMD (a head), it is possible to perform an image display in accordance with the position and orientation of the HMD (head), in accordance with only change of the shape of a marker region in a captured image, without needing to perform complicated calculation processing such as generating an image from a viewpoint in a virtual three-dimensional space by considering a three-dimensional position and orientation of a marker or information collected using a sensor or the like, where the position and orientation of the HMD is measured using the sensor, as in the past.
In the first embodiment, shape information is set as information indicating a shape of a marker region first detected from a captured image after activation of the display apparatus 101, but may be information indicating the shape of a marker region in another frame. For example, information indicating the shape of the marker region detected recently may be set as shape information.
In addition, in the first embodiment, a hand-held display apparatus, a smart phone, and a tablet terminal device may be used instead of an HMD, but regardless of which apparatus, there is a necessity to have a camera that captures a moving image of a physical space.
In addition, in a case of displaying a display image on the display screen 115, an image captured by the camera 114 may be displayed. For example, a display image may be displayed overlapping a marker region on a captured image. Of course, the display position of the display image is not limited to a position that overlaps a marker region.
The present invention is not limited to the embodiments described above, and it is possible to make various modifications or changes without straying from the spirit and scope of the present invention. Therefore, to apprise the public of the scope of the present invention, the following claims are appended.
This application is a continuation of International Patent Application No. PCT/JP2016/089106 filed on Dec. 28, 2016, the entire disclosures of which is incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2016/089106 | Dec 2016 | US |
Child | 16454195 | US |