Patent Application
20210264678
The present disclosure relates to a video display system that displays a video for supporting a user's work.
A technique relating to mixed reality (hereinafter referred to as “MR”) is known of displaying virtual information using computer graphics or the like with respect to a real object arranged in a real space (see, for example, Patent Document 1). Patent Document 1 discloses a technique of synthesizing display of a virtual video and a real video obtained by capturing an image of an architectural model that is a real object.
As a method of using an MR technique, associating a virtual video for supporting a user's work through a head mount display (HMD) (hereinafter referred to as an “MR device”) or the like with a real object and providing it to the user can be considered. For example, when welding steel members, work such as reading and understanding of drawings, ruling on steel members, and confirmation of ruling is required. Even skilled engineers take time to understand complicated drawings, and reworking due to simple mistakes is needed when there are many portions to be welded. For this reason, conventional welding of steel members has required skilled techniques. Consequently, superimposition display of a virtual object indicating a mounting member on a real steel member using the MR device can be considered.
However, the difficulty of mounting work may not be improved by only superimposition display of a virtual object indicating a mounting member on a structure such as a steel member. For example, in a state in which a mounting member having relatively large dimensions or weight is inclined with respect to a structure, mounting work such as welding may be performed between the mounting member and the structure. In such a case, it is difficult to perform the mounting work while maintaining a state in which the mounting member is arranged in accordance with display of the virtual object.
The present disclosure was invented in view of the above problems, and an object thereof is to provide a video display system that makes it possible to easily mount a member on a structure.
In order to achieve the above object, according to the present disclosure, there is provided a video display system used in mounting a member on a structure, including: a storage unit configured to store information on arrangement of a jig for positioning the member on the structure; a structure arrangement detection unit configured to detect an arrangement of the structure arranged in a real space; and a control unit configured to perform a display process indicating the arrangement of the jig on the structure arranged in the real space on the basis of the information on the arrangement of the jig stored in the storage unit and a detection result obtained by the structure arrangement detection unit.
In the video display system according to the present disclosure, information on the arrangement of a jig for positioning a member on a structure is stored, and a display process indicating the arrangement of the jig is performed on the structure arranged in a real space. A user can easily arrange the jig in accordance with the display process indicating the arrangement of the jig. As a result, the member is positioned by the jig on the structure, and thus it is possible to easily mount the member on the structure.
According to the present disclosure, it is possible to provide a video display system that makes it possible to easily mount a member on a structure.
Hereinafter, an embodiment of a video display system according to the present disclosure will be described in detail with reference to the accompanying drawings. Meanwhile, in the description of the drawings, the same components are denoted by the same reference numerals and signs, and thus description thereof will not be repeated.
[Outline of Video Display System]
First, a video display system according to the present embodiment will be described.
The video display system 100 performs image capturing in the same image capturing direction as a user's eye line, or the like. The video display system 100 superimposes a virtual object on a captured image to display the result. By viewing this display, a user can view a virtual object that does not exist in reality according to the arrangement of a real object arranged in a real space. The “arrangement” referred to here means the state of six degrees of freedom defined by coordinates on predetermined three-dimensional coordinate axes and rotation around each of the three-dimensional coordinate axes (orientation in three-dimensional coordinates).
The video display system 100 displays a virtual object using a virtual space (a screen coordinate system) which is three-dimensional coordinates. The video display system 100 arranges a virtual object at preset position in a virtual space, and calculates a correspondence relation between the virtual space and the real space. The arrangement of a virtual object in a virtual space is referred to as “display arrangement.” In the following description, a coordinate axis indicating the position of a virtual object in a virtual space is referred to as a “virtual coordinate axis α.” In the video display system 100, an image viewed from a position and direction in a virtual space corresponding to an image capturing position and image capturing direction in a real space is defined as display of a virtual object. That is, an image of a virtual object in a virtual space viewed from a virtual user's visual line is displayed. The display of a virtual object can be carried out using an MR technique of the related art. The video display system 100 detects a user's movement, and changes a display process in accordance with the movement so as to match a change in the user's visual line according to the user's movement.
The video display system 100 is used in mounting a member on a structure 110 arranged in a real space (hereinafter referred to as a “mounting member”).
The jig 120 is a member for positioning a mounting member on the structure 110. For example, in a state in which a mounting member is positioned on the structure 110 by the jig 120, a user performs mounting work such as welding between the mounting member and the structure. In the present embodiment, the above-described virtual object contains information indicating the structure 110 in a real space, information indicating a mounting member scheduled to be mounted on the structure 110, and information indicating a distance from a reference position to a position at which the jig 120 is arranged. The reference position is an end 110a of the structure 110.
In the present embodiment, as shown in
In the present embodiment, the video display system 100 displays a shortest distance from the end 110a of the structure 110 in its extending direction to a mounting member and a longest distance from the end 110a of the structure 110 in its extending direction to the mounting member as distances from the reference position to the position at which the jig 120 is arranged. As shown in
The video display system 100 detects the reference position on the basis of a reference marker 125 arranged on the structure 110 in a real space. The reference marker 125 is arranged in a predetermined arrangement relation with the structure 110 before work is performed. In the present embodiment, the reference marker 125 is provided along the end 110a of the structure 110 on a face plate 112 of the structure 110. The reference marker 125 has a feature point that makes it possible for the video display system 100 to recognize the three-dimensional arrangement of the reference marker 125 in a real space through image recognition. For example, the reference marker 125 may contain a diagram having such a feature point (for example, a two-dimensional code) or the like. Insofar as the video display system 100 can recognize the three-dimensional arrangement of the reference marker 125 in a real space, the reference marker 125 is not limited to this form. For example, the feature point of the reference marker 125 may be shown by the color or blinking of the reference marker 125. The three-dimensional position of the reference marker 125 may be identified through a method of identifying a three-dimensional position using a magnetic signal, a sound wave signal, or the like.
The face plate 111 has main surfaces 111a and 111b that face each other. The face plate 112 has main surfaces 112a and 112b that face each other. The face plate 113 has main surfaces 113a and 113b that face each other. The main surfaces 111a, 111b, 112a, 112b, 113a, and 113b of the face plates 111, 112, and 113 extend in a longitudinal direction and are in contact with the mounting member. The face plate 111 is connected to the face plates 112 and 113 so as to be perpendicular thereto. The central portion of the main surface 112a of the face plate 112 and the central portion of the main surface 113a of the face plate 113 are connected to a pair of long sides of the face plate 111.
As shown in
As shown in
On the basis of the display of the video display system 100, the flat surface 121a of the first jig 121 is arranged at the position of the shortest distance from the end 110a in the extending direction of the structure 110 to the mounting member 131. On the basis of the display of the video display system 100, the flat surface 122a of the second jig 122 is arranged at the position of the longest distance from the end 110a in the extending direction of the structure 110 to the mounting member.
The video display system 100 also displays the virtual mounting member 201 in addition to display of the information 206 and 207 indicating distances from the reference position to the position at which the jig 120 is arranged. The first jig 121 is arranged to be in contact with a portion of the virtual mounting member 201 which is located at the shortest distance from the end 110a in the extending direction of the structure 110. The second jig 122 is arranged to be in contact with a portion of the virtual mounting member 201 which is located at the longest distance from the end 110a in the extending direction of the structure 110. In the present embodiment, the first jig 121 is arranged so that the flat surface 121a is in contact with the corner portion 131a of the virtual mounting member 201, and the second jig 122 is arranged so that the flat surface 122a is in contact with the corner portion 131d of the virtual mounting member 201. In this case, the flat surface 121a of the first jig 121 and the flat surface 122a of the second jig 122 face each other. Thereby, the jig 120 is arranged to restrict the position of the mounting member 131 in the extending direction of the structure 110; the angle of inclination of the mounting member 131 with respect to the main surface 111a of the structure 110; and the angle of rotation of the mounting member 131 with respect to an axis orthogonal to the main surface 111a. For example, the mounting member 131 is arranged to abut the main surface 111a of the structure 110; the flat surface 121a of the first jig 121; and the flat surface 122a of the second jig 122, so that the mounting member 131 is positioned with a desired arrangement.
[Configuration of Video Display System]
As shown in
At least one of the image capturing unit 101 and the display unit 103 in the video display system 100 is portable. The video display system 100 may not include the image capturing unit 101. When the display unit 103 is carried without carrying the image capturing unit 101, the display unit 103 and the inertial measurement unit 102 are housed in one housing. For example, the video display system 100 can be used not only when an HMD is used, but also when a user carries a tablet or the like in which the image capturing unit 101 does not operate and when the user remotely operates the movement of a terminal having the image capturing unit 101 while making confirmation with the display unit 103 at hand. In the present embodiment, a case in which the image capturing unit 101, the inertial measurement unit 102, and the display unit 103 are housed in one housing and an HMD having the housing fixed to a user's head is used will be mainly described.
The image capturing unit 101 captures an image of a real space, and sequentially transmits the captured image to the environmental information detection unit 104. In the present embodiment, the image capturing unit 101 is fixed to a user's head, and the position and posture of the image capturing unit 101 fluctuate in accordance with the movement of the user's head. The position of the image capturing unit 101 is a coordinate position of the image capturing unit 101. The posture of the image capturing unit 101 is rotation around each axis of three-dimensional coordinates (orientation in three-dimensional coordinates). The image capturing unit 101 is constituted by a plurality of cameras such as a depth camera and an RGB camera. The image capturing unit 101 includes not only a front camera that captures an image of a user's visual line direction, but also an environment camera that captures an image of the user's side or the like.
The inertial measurement unit 102 measures external force acting on the image capturing unit 101 or the display unit 103, and sequentially transmits measurement results to the environmental information detection unit 104. In the present embodiment, the inertial measurement unit 102 is constituted by an acceleration sensor, a gyro sensor, an orientation sensor, and the like which are fixed to a user's head.
The display unit 103 includes a display that performs display. The display unit 103 displays a virtual video including a virtual object in accordance with the position and posture of the image capturing unit 101 or the display unit 103. In the present embodiment, the display unit 103 is fixed to a user's head in a state in which the user can visually recognize a virtual object appropriately. The display unit 103 is configured such that the position and posture of the display unit 103 fluctuate in accordance with the movement of the user's head. The position of the display unit 103 is a coordinate position of the display unit 103. The posture of the display unit 103 is rotation around each axis of three-dimensional coordinates (orientation in three-dimensional coordinates). In the present embodiment, the display unit 103 performs display indicating the structure 110 in a real space, display indicating a mounting member scheduled to be mounted on the structure 110, and display of information indicating a distance from the reference position to the position at which the jig 120 is arranged, in accordance with instructions from the control unit 107 to be described later.
The environmental information detection unit 104 detects various types of information on the basis of at least one of an image captured by the image capturing unit 101 and a measurement result of the inertial measurement unit 102. The video display system 100 may not include the inertial measurement unit 102. In this case, the video display system 100 performs various processes in the environmental information detection unit 104 on the basis of the image captured by the image capturing unit 101. In a case where the video display system 100 does not include the image capturing unit 101, the video display system 100 performs various processes in the environmental information detection unit 104 on the basis of the inertial measurement unit 102. The environmental information detection unit 104 includes a reference position detection unit 151, a structure arrangement detection unit 152, a user position detection unit 153, a visual line direction detection unit 154, a display arrangement detection unit 155, and a jig arrangement detection unit 156.
The reference position detection unit 151 detects a reference position for arranging the origin N of a virtual coordinate axis α. In the present embodiment, the reference position detection unit 151 detects a feature point that makes it possible to recognize the three-dimensional arrangement of the reference marker 125 in a real space from the image captured by the image capturing unit 101. The reference position detection unit 151 detects a reference position in a real space from the detected feature point. For example, in a state in which the reference marker 125 is arranged along the end 110a of the structure 110, the reference position detection unit 151 detects the center of the end 110a in the face plate 112 from the feature point of the reference marker 125 as the reference position. The result of detection of the reference position performed by the reference position detection unit 151 is, for example, a relative position of the reference position from the image capturing unit 101 or the display.
The reference position detection unit 151 may detect the reference position not from the reference marker 125 but from the arrangement of the edge of the structure 110 in the image captured by the image capturing unit 101, the surrounding environment of the structure 110 in the image captured by the image capturing unit 101, or the like. The reference position detection unit 151 may separately acquire a signal indicating the reference position in a real space in a wireless or wired manner. The result of detection of the reference position performed by the reference position detection unit 151 may be a position on the image captured by the image capturing unit 101.
The structure arrangement detection unit 152 detects the arrangement of the structure 110 arranged in a real space. In the present embodiment, the structure arrangement detection unit 152 detects the arrangement of the structure 110 on the basis of the reference position detected by the reference position detection unit 151 and another feature point for specifying the direction of the structure 110. The other feature point for specifying the direction of the structure 110 may be the direction of the reference marker 125, the arrangement of a marker provided in the structure 110 separately from the reference marker 125, the arrangement of the edge of the structure 110, or the surrounding environment of the structure 110. For example, the structure arrangement detection unit 152 detects the arrangement of the edge of the structure 110 in a real space from the image captured by the image capturing unit 101.
The result of detection of the arrangement of the structure 110 performed by the structure arrangement detection unit 152 is, for example, a relative arrangement of the structure 110 with respect to the image capturing unit 101 or the display. The structure arrangement detection unit 152 may separately acquire a signal indicating the arrangement of the structure 110 in a real space in a wireless or wired manner. The result of detection of the arrangement of the structure 110 performed by the structure arrangement detection unit 152 may be an arrangement on the image captured by the image capturing unit 101.
The user position detection unit 153 detects the position of a user who performs mounting work. In the present embodiment, the user position detection unit 153 detects a relative position of the image capturing unit 101 with respect to the reference position as the user's position. For example, the user position detection unit 153 detects a feature point of the surrounding environment from the image captured by the image capturing unit 101. The user position detection unit 153 detects the position of the image capturing unit 101 in a real space from the detected feature point. The user position detection unit 153 may detect the position of the image capturing unit 101 on the basis of the image captured by the image capturing unit 101, the result of detection of the reference position, and the measurement result of the inertial measurement unit 102.
The user position detection unit 153 may detect a relative position of the display of the display unit 103 with respect to the reference position as the user's position. The user position detection unit 153 may detect the relative position of the image capturing unit 101 with respect to the reference position as the position of the display. The user position detection unit 153 may detect the position of the display on the basis of the image captured by the image capturing unit 101, the result of detection of the reference position, and the measurement result of the inertial measurement unit 102. The user position detection unit 153 may detect the position of the display without using the image captured by the image capturing unit 101. The user position detection unit 153 may separately acquire a signal indicating the position of the image capturing unit 101 or the display in a real space from, for example, an external image capturing unit or the like.
The user position detection unit 153 may detect the position of the user with respect to any position of the structure 110 rather than the position of the user with respect to the reference position. The user position detection unit 153 may detect the position of the user with respect to the jig 120.
The visual line direction detection unit 154 detects the visual line direction of the user who performs mounting work. Meanwhile, the “visual line direction” may be a direction in which a front camera is directed, may be the center of a captured image, may be a direction in which the display is directed, or may be a direction calculated from the direction of the user's eyeball. In the present embodiment, the visual line direction detection unit 154 detects the posture of the image capturing unit 101 with respect to the reference position as the visual line direction. For example, the visual line direction detection unit 154 detects a feature point of the surrounding environment from the image captured by the image capturing unit 101. The visual line direction detection unit 154 detects the posture of the image capturing unit 101 in a real space from the detected feature point. The visual line direction detection unit 154 may detect the posture of the image capturing unit 101 on the basis of the image captured by the image capturing unit 101, the result of detection of the reference position, and the measurement result of the inertial measurement unit 102.
The visual line direction detection unit 154 may detect the posture of the display of the display unit 103 with respect to the reference position as the visual line direction. The visual line direction detection unit 154 may detect the posture of the image capturing unit 101 with respect to the reference position as the posture of the display. The visual line direction detection unit 154 may detect the posture of the display on the basis of the image captured by the image capturing unit 101, the result of detection of the reference position, and the measurement result of the inertial measurement unit 102. The visual line direction detection unit 154 may detect the posture of the display without using the image captured by the image capturing unit 101. The visual line direction detection unit 154 may separately acquire a signal indicating the posture of the image capturing unit 101 or the display in a real space from, for example, an external image capturing unit or the like.
The display arrangement detection unit 155 detects the arrangement of the display. In the present embodiment, the display arrangement detection unit 155 detects the arrangement of the display with respect to the reference position from the detection result obtained by the user position detection unit 153 and the detection result obtained by the visual line direction detection unit 154.
The jig arrangement detection unit 156 detects the arrangement of the jig 120 arranged in a real space. In the present embodiment, the jig arrangement detection unit 156 detects a feature point of each of the first jig 121 and the second jig 122 from the image captured by the image capturing unit 101. As shown in
As shown in
The storage unit 105 previously stores various types of information (CG data) required for display of a virtual object. The storage unit 105 stores information on the arrangement of the jig 120 on the structure 110. In the present embodiment, the information on the arrangement of the jig 120 includes information on the arrangement of the first jig 121 and the second jig 122 of which the flat surface 121a and the flat surface 122a are in contact with the mounting member. The information on the arrangement of the first jig 121 and the second jig 122 includes information on the arrangement of the jig that restricts the position of the mounting member in the extending direction of the structure 110; the angle of inclination of the mounting member with respect to the main surface 111a of the structure 110; and the angle of rotation of the mounting member with respect to the axis orthogonal to the main surface 111a.
For example, the storage unit 105 previously stores the display arrangement of the virtual mounting members 201, 202, 203, and 204 associated in advance with the virtual coordinate axis α in a virtual space and the information 206 and 207 indicating the distance from the reference position to the position at which the jig 120 is arranged. The storage unit 105 stores the display arrangement of the virtual mounting members 201, 202, 203, and 204 and the coordinates on the virtual coordinate axis a of the position at which the jig 120 is arranged. The display arrangement of the virtual mounting members 201, 202, 203, and 204 corresponding to the position and posture of the image capturing unit 101 in a real space and the position at which the jig 120 is arranged are determined by associating the virtual coordinate axis α with the real space.
The storage unit 105 also previously stores a predetermined arrangement relation between the reference marker 125 arranged in a real space and the virtual coordinate axis a. In the present embodiment, the position of a reference point M in a real space of the reference marker 125 and the reference position at which the origin N of the virtual coordinate axis α is arranged are associated with each other. That is, the arrangement of the reference marker 125 in a real space and the origin N of the virtual coordinate axis a are associated with each other.
The operation detection unit 106 detects a user's operation. For example, in the touch panel display of the video display system 100, a position touched by a user is detected. The operation detection unit 106 sequentially transmits detected information to the control unit 107. For example, the operation detection unit 106 transmits signals indicating selected virtual mounting members 201, 202, 203, and 204 to the control unit 107 in accordance with a user's operation. The user's operation which is detected by the operation detection unit 106 may be a hand gesture, a blink, or a button input.
The control unit 107 performs various types of control on the basis of the detection result obtained by the environmental information detection unit 104, the detection result obtained by the operation detection unit 106, and information stored in the storage unit 105. The control unit 107 includes a reference arrangement setting unit 157; a display control unit 158; and a member determination unit 159.
The reference arrangement setting unit 157 sets a relationship between the virtual coordinate axis α for specifying the display arrangement of a virtual object and the position and posture of the image capturing unit 101 or the display detected by the user position detection unit 153 and the visual line direction detection unit 154. In the present embodiment, the reference arrangement setting unit 157 associates the virtual coordinate axis a with a real space in accordance with the arrangement of the structure 110 detected by the structure arrangement detection unit 152. In other words, the reference arrangement setting unit 157 arranges the origin N of the virtual coordinate axis α at the reference position on the basis of the detection result detected by the structure arrangement detection unit 152, and sets the direction of the virtual coordinate axis a in a real space.
The display control unit 158 performs a display process indicating the arrangement of the jig 120 on the structure 110 arranged in a real space on the basis of the information on the arrangement of the jig 120 stored in the storage unit 105 and the detection result obtained by the structure arrangement detection unit 152. For example, the display control unit 158 instructs the display unit 103 to display information indicating the arrangement of a plurality of members, and causes the display unit 103 to display the virtual mounting members 201, 202, 203, and 204 on the structure 110 arranged in a real space as shown in
The display control unit 158 may change the display arrangement of information indicating the arrangement of the jig 120 in accordance with the arrangement of the display detected by the display arrangement detection unit 155. For example, when a user has changed the arrangement of the display, the display control unit 158 may change the display arrangement of the structure 110, the display arrangement of the virtual mounting members 201, 202, 203, and 204, and the display arrangement of the information 206 and 207 indicating the distance to the position at which the jig 120 is arranged, on the display, in accordance with the change.
The display control unit 158 displays information indicating a position shift of the jig 120 on the basis of the arrangement of the jig 120 detected by the jig arrangement detection unit 156 and the information on the arrangement of the jig 120 stored in the storage unit 105. For example, when the arrangement of the jig 120 detected by the jig arrangement detection unit 156 shifts from the information on the arrangement of the jig 120 stored in the storage unit 105 by an allowable error or more, the display control unit 158 performs a display process indicating the occurrence of the shift on the display unit 103. For example, when it is determined that the shift has occurred, the display control unit 158 causes the display unit 103 to display information 160 indicating that the shift has occurred, as shown in
The member determination unit 159 determines a mounting member on which mounting work is performed by a user from a plurality of mounting members corresponding to a plurality of virtual mounting members stored in the storage unit 105. For example, the member determination unit 159 determines at least one mounting member out of the plurality of mounting members as the mounting member on which mounting work is performed by the user on the basis of, for example, at least one of the detection result obtained by the operation detection unit 106, the position of the user with respect to the structure 110, and the user's visual line direction. The display control unit 158 determines information indicating the arrangement of the jig 120 corresponding to the mounting member determined by the member determination unit 159 as information used in the arrangement of the jig 120 by the user, and performs a display process on information indicating the arrangement of the jig 120 corresponding to the other mounting members. For example, when the member determination unit 159 has determined the mounting member 131 corresponding to the virtual mounting member 201 as the mounting member on which mounting work is performed by the user, the display control unit 158 instructs the display unit 103 not to display information indicating the arrangement of the jig 120 corresponding to the remaining mounting members. A display process for the remaining jig 120 is not limited to non-display, but includes changing display such as semi-transparent display, blinking display, contour display, color change, and display density change. For the remaining jig 120, only information indicating the distance of the jig 120 may be changed in display.
The display control unit 158 may change display of information indicating the arrangement of the jig 120 corresponding to the mounting member determined by the member determination unit 159.
For example, the member determination unit 159 determines a mounting member corresponding to the user's operation among a plurality of mounting members on the basis of the detection result obtained by the operation detection unit 106 and the information on the arrangement of the jig 120 stored in the storage unit 105. The display control unit 158 performs a display process of information indicating the arrangement of the jig 120 corresponding to the determined mounting member. For example, when the user touches the virtual mounting member 201, the member determination unit 159 determines the mounting member 131 corresponding to the virtual mounting member 201 operated by the user as the mounting member on which mounting work is performed by the user. The display control unit 158 may leave display of the virtual mounting member 201 operated by the user and display of a distance corresponding thereto, as shown in
For example, the member determination unit 159 determines a mounting member corresponding to the detection result obtained by the visual line direction detection unit 154 among a plurality of mounting members on the basis of the detection result obtained by the visual line direction detection unit 154 and the information on the arrangement of the jig 120 stored in the storage unit 105. The display control unit 158 performs a display process of information indicating the arrangement of the jig 120 relating to the determined mounting member. For example, as shown in
For example, the member determination unit 159 determines a mounting member corresponding to the detection result obtained by the user position detection unit 153 among a plurality of mounting members on the basis of the detection result obtained by the user position detection unit 153 and the information on the arrangement of the jig 120 stored in the storage unit 105. The display control unit 158 performs a display process of information indicating the arrangement of the jig 120 relating to the determined mounting member. For example, the member determination unit 159 determines where on the left, right, top and bottom the user is located relative to the structure 110 on the basis of the detection result obtained by the user position detection unit 153 and the information on the arrangement of the jig 120 stored in the storage unit 105. For example, as shown in
The member determination unit 159 may calculate a distance between the user and the structure 110 from the detection result obtained by the user position detection unit 153, and determine a mounting member in accordance with the distance. For example, the member determination unit 159 may output a virtual mounting member located within two meters of the user as a determination result. In this case, the display control unit 158 may, for example, display a virtual mounting member located within two meters of the user in blue, and may not display a virtual mounting member located two meters or more away from the user or display it semi-transparently. Even in the case of a virtual mounting member located within two meters of the user, the display control unit 158 may not display the virtual mounting member located on the opposite side of the user with the structure 110 interposed therebetween. For example, in
[Process in Video Display System]
Next, an example of processing which is performed in the video display system 100 will be described with reference to
In step S101, the reference position detection unit 151 detects a reference position for arranging the origin N of the virtual coordinate axis α. Subsequently, the process proceeds to step S102.
In step S102, the structure arrangement detection unit 152 detects the arrangement of the structure 110 arranged in a real space. Subsequently, the process proceeds to step S103.
In step S103, the display arrangement detection unit 155 detects the arrangement of the display of the display unit 103. Subsequently, the process proceeds to step S104.
In step S104, the jig arrangement detection unit 156 detects the arrangement of the jig 120 arranged in a real space. Subsequently, the process proceeds to step S105.
In step S105, the control unit 107 performs a display process on the basis of information detected in steps S102 to S104. The display unit 103 displays various types of information in accordance with instructions from the control unit 107.
Next, a display process which is performed by the control unit 107 will be described with reference to
In step S111, the control unit 107 determines whether there is information on the selection of a member. The information on the selection of a member is, for example, a determination result determined by the member determination unit 159 on the basis of at least one of the detection result obtained by the operation detection unit 106, the position of the user with respect to the structure 110, and the user's visual line direction. In a case where it is determined that there is no information on the selection of a member (NO in step S111), the control unit 107 advances the process to step S112. In a case where it is determined that there is the information on the selection of a member (YES in step S111), the control unit 107 advances the process to step S113.
In step S112, the control unit 107 causes the display control unit 158 to display all the virtual mounting members stored in the storage unit 105 on the display unit 103, and advances the process to step S117.
In step S113, the control unit 107 determines whether the jig arrangement has been detected in the jig arrangement detection unit 156. In a case where it is determined that the jig arrangement is not detected (NO in step S113), the control unit 107 advances the process to step S114. In a case where it is determined that the jig arrangement is detected (YES in step S113), the control unit 107 advances the process to step S115.
In step S114, the control unit 107 causes the display control unit 158 to display a virtual mounting member corresponding to a mounting member determined by the member determination unit 159 on the display unit 103, and advances the process to step S117.
In step S115, the control unit 107 determines whether the jig arrangement is appropriate on the basis of the detection result obtained by the jig arrangement detection unit 156. In a case where it is determined that the jig arrangement is appropriate (YES in step S115), the control unit 107 advances the process to step S114. In a case where it is determined that the jig arrangement is not appropriate (NO in step S115), the control unit 107 advances the process to step S116.
In step S116, the control unit 107 causes the display control unit 158 to display a virtual mounting member corresponding to a mounting member determined by the member determination unit 159 and an error on the display unit 103, and the control unit 107 advances the process to step S117. In this case, the display control unit 158 displays, for example, the information 160 indicating that a shift has occurred on the display unit 103.
In step S117, the control unit 107 determines whether to end the display process. In a case where it is determined that the display process is not ended (NO in step S117), the control unit 107 advances the process to step S111. In a case where it is determined that the display process is ended (YES in step S117), the control unit 107 ends the display process.
[Operations and Effects of Video Display System]
Next, main operations and effects of the video display system 100 according to the present embodiment will be described.
In the video display system 100, the information on the arrangement of the jig 120 for positioning a mounting member on the structure 110 is stored, and the display process indicating the arrangement of the jig 120 is performed on the structure 110 arranged in a real space. A user can easily arrange the jig 120 in accordance with the display process indicating the arrangement of the jig 120. As a result, the mounting member is positioned by the jig 120 on the structure 110, and thus it is possible to easily mount the mounting member on the structure 110.
For example, the mounting member 131 is mounted on the structure 110 in a state in which it is also inclined with respect to the main surface 112a of the face plate 112 and the main surface 113a of the face plate 113 while being inclined with respect to the main surface 111a of the face plate 111. It is very difficult to perform mounting work in such a state. The jig 120 includes the first jig 121 having the flat surface 121a and the second jig 122 having the flat surface 122a facing the flat surface 121a. The information on the arrangement of the jig 120 includes the information on the arrangement of the jig 120 of which the flat surface 121a and the flat surface 122a abut the mounting member. Therefore, according to the video display system 100, it is possible to easily and appropriately arrange the jig 120 and to easily position the mounting member 131 on the structure 110.
The structure 110 has the main surface 111a which extends in a predetermined direction and with which the mounting member is in contact. The information on the arrangement of the jig 120 includes the information on the arrangement of the jig 120 for restricting the position of the mounting member in a predetermined direction, the angle of inclination of the mounting member with respect to the main surface 111a, and the angle of rotation of the member with respect to the axis orthogonal to the main surface 111a. Therefore, according to the video display system 100, it is possible to easily and appropriately arrange the jig 120 and to easily position the mounting member 131 on the structure 110.
The video display system 100 includes the display arrangement detection unit 155 that detects the arrangement of the display that performs display based on the display process. The control unit 107 changes display of the structure and the display arrangement of information indicating the arrangement of the jig 120 on the display in accordance with the arrangement of the display detected by the display arrangement detection unit 155. Therefore, the user can confirm the arrangement of the jig 120 on the structure at any position by moving the display.
The video display system 100 includes the jig arrangement detection unit 156 that detects the arrangement of the jig 120 arranged in a real space. The control unit 107 displays the information 160 indicating a position shift of the jig 120 on the basis of the arrangement of the jig 120 detected by the jig arrangement detection unit 156 and the information on the arrangement of the jig 120 stored in the storage unit 105. Therefore, the user can easily confirm whether the arrangement of the jig 120 is appropriate.
The video display system 100 includes the operation detection unit 106 that detects a user's operation. The control unit 107 performs a display process of information indicating the arrangement of the jig relating to a mounting member corresponding to the user's operation among a plurality of mounting members on the basis of the detection result obtained by the operation detection unit 106 and the information on the arrangement of the jig 120 stored in the storage unit 105. Therefore, the user can select information indicating the arrangement of the jig to be displayed.
The storage unit 105 stores information on the arrangement of a plurality of mounting members on the structure 110. The control unit 107 gives an instruction for display of information indicating the arrangement of a plurality of mounting members on the basis of the information on the arrangement of the plurality of members stored in the storage unit 105. The control unit 107 determines a mounting member corresponding to the detection result obtained by the operation detection unit 106 among the plurality of mounting members, and performs a display process indicating the arrangement of the jig 120 relating to the determined mounting member. Therefore, the user can select information indicating the arrangement of the jig by operating the displayed mounting member.
The video display system 100 includes the user position detection unit 153 that detects a user's position. The control unit 107 performs a display process of information indicating the arrangement of the jig relating to a mounting member corresponding to the user's position among a plurality of mounting members on the basis of the detection result obtained by the user position detection unit 153 and the information on the arrangement of the jig 120 stored in the storage unit 105. Therefore, information indicating appropriate arrangement of the jig can be displayed in accordance with the user's position.
The video display system 100 includes the visual line direction detection unit 154 that detects a user's visual line direction. The control unit 107 performs a display process of information indicating the arrangement of the jig relating to a mounting member corresponding to the user's visual line direction among a plurality of mounting members on the basis of the detection result obtained by the visual line direction detection unit 154 and the information on the arrangement of the jig 120 stored in the storage unit 105. Therefore, information indicating appropriate arrangement of the jig can be displayed in accordance with the user's visual line direction.
Meanwhile, the block diagram used in the description of the above embodiment represents blocks in units of functions. These functional blocks (constituent elements) are realized by any combination of at least one of hardware and software. In addition, a method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device which is physically or logically coupled, or may be realized using two or more devices which are physically or logically separated from each other by connecting the plurality of devices directly and/or indirectly (for example, using a wired or wireless manner or the like). The functional block may be realized by combining software with the one device or the plurality of devices.
Examples of the functions include determining, deciding, judging, calculating, computing, process, deriving, investigating, search, ascertaining, receiving, transmitting, output, access, resolving, selecting, choosing, establishing, comparing, assuming, expecting, considering, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (or mapping), assigning, and the like, but there is no limitation thereto. For example, a functional block (constituent element) for causing transmitting to function is referred to as a transmitting unit or a transmitter. As described above, any realization methods are not particularly limited.
For example, the video display system in an embodiment of the present disclosure may function as a computer that performs processing in the present embodiment.
Meanwhile, in the following description, the word “device” can be replaced with a circuit, a unit, or the like. The hardware configuration of the video display system 100 may be configured to include one or a plurality of devices shown in the drawing, or may be configured without including some devices.
The processor 1001 performs an arithmetic operation by reading predetermined software (a program) onto hardware such as the processor 1001 or the memory 1002, and thus each function of the video display system 100 is realized by controlling communication in the communication device 1004 or controlling at least one of reading-out and writing of data in the memory 1002 and the storage 1003.
The processor 1001 controls the whole computer, for example, by operating an operating system. The processor 1001 may be constituted by a central processing unit (CPU) including an interface with a peripheral device, a control device, an arithmetic operation device, a register, and the like. For example, the environmental information detection unit 104, the control unit 107, and the like may be realized by the processor 1001.
In addition, the processor 1001 reads out a program (a program code), a software module, data, or the like from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various types of processes in accordance therewith. An example of the program which is used includes a program causing a computer to execute at least some of the operations described in the foregoing embodiment. For example, the display unit 103, the environmental information detection unit 104, the operation detection unit 106, and the control unit 107 are stored in the memory 1002, and may be realized by a control program which is operated in the processor 1001. Similarly, other functional blocks may be realized. The execution of various types of processes described above by one processor 1001 has been described, but these processes may be simultaneously or sequentially executed by two or more processors 1001. One or more chips may be mounted in the processor 1001. Meanwhile, the program may be transmitted from a network through an electrical communication line.
The memory 1002 is a computer readable recording medium, and may be constituted by at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), and the like. The memory 1002 may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store a program (a program code), a software module, or the like that can be executed in order to carry out a wireless communication method according to an embodiment of the present disclosure.
The storage 1003 is a computer readable recording medium, and may be constituted by at least one of, for example, an optical disc such as a compact disc ROM (CD-ROM), a hard disk drive, a flexible disk, a magnetooptic disc (for example, a compact disc, a digital versatile disc, or a Blu-ray (registered trademark) disc), a smart card, a flash memory (for example, a card, a stick, or a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. The storage 1003 may be referred to as an auxiliary storage device. The foregoing storage medium may be, for example, a database including at least one of the memory 1002 and the storage 1003, a server, or another suitable medium.
The communication device 1004 is hardware (a transmitting and receiving device) for performing communication between computers through at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like.
The input device 1005 is an input device (such as, for example, a keyboard, a mouse, a microphone, a switch, a button, or a sensor) that receives an input from the outside. The output device 1006 is an output device (such as, for example, a display, a speaker, or an LED lamp) that executes an output to the outside. Meanwhile, the input device 1005 and the output device 1006 may be an integrated component (for example, a touch panel). In addition, respective devices such as the processor 1001 and the memory 1002 are connected to each other through the bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using a different bus between devices.
In addition, the video display system 100 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), or some or all of the respective functional blocks may be realized by the hardware. For example, the processor 1001 may be mounted using at least one of the hardware.
The aspects/embodiments described in the present disclosure may be used alone, may be used in combination, or may be switched during implementation thereof. In addition, notification of predetermined information (for example, notification of “X”) is not limited to explicit transmission, and may be performed by implicit transmission (for example, the notification of the predetermined information is not performed).
Hereinbefore, the present disclosure has been described in detail, but it is apparent to those skilled in the art that the present disclosure should not be limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and changed aspects without departing from the spirit and scope of the present disclosure, which are determined by the description of the scope of claims. Therefore, the description of the present disclosure is intended for illustrative explanation only, and does not impose any limited interpretation on the present disclosure.
The aspects/embodiments described in the present disclosure may be applied to at least one of a system employing long term evolution (LTE), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, a 4th generation mobile communication system (4G), a 5th generation mobile communication system (5G), Future Radio Access (FRA), new Radio (NR), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (registered trademark), or other appropriate systems and a next-generation system extended on the basis thereof. In addition, a plurality of systems may be combined (for example, 5G and at least one of LTE and LTE-A are combined or the like) and be applied.
The order of the processing sequences, the sequences, the flow charts, and the like of the aspects/embodiments described above in the present disclosure may be changed as long as they are compatible with each other. For example, in the methods described in the present disclosure, various steps as elements are presented using an exemplary order but the methods are not limited to the presented order.
The input or output information or the like may be stored in a specific place (for example, a memory) or may be managed using a management table. The input or output information or the like may be overwritten, updated, or added. The output information or the like may be deleted. The input information or the like may be transmitted to another device.
Determination may be performed using a value (0 or 1) which is expressed by one bit, may be performed using a Boolean value (true or false), or may be performed by comparison of numerical values (for example, comparison thereof with a predetermined value).
Information, a signal or the like described in the present disclosure may be expressed using any of various different techniques. For example, data, an instruction, a command, information, a signal, a bit, a symbol, and a chip which can be mentioned in the overall description may be expressed by a voltage, a current, an electromagnetic wave, a magnetic field or magnetic particles, an optical field or photons, or any combination thereof.
Information, a signal or the like described in the present disclosure may be expressed using any of various different techniques. For example, data, an instruction, a command, information, a signal, a bit, a symbol, and a chip which can be mentioned in the overall description may be expressed by a voltage, a current, an electromagnetic wave, a magnetic field or magnetic particles, an optical field or photons, or any combination thereof.
The term “determining” which is used in the present disclosure may include various types of operations. The term “determining” may include regarding operations such as, for example, judging, calculating, computing, processing, deriving, investigating, looking up/search/inquiry (for example, looking up in a table, a database or a separate data structure), or ascertaining as an operation such as “determining.” In addition, the term “determining” may include regarding operations such as receiving (for example, receiving information), transmitting (for example, transmitting information), input, output, or accessing (for example, accessing data in a memory) as an operation such as “determining.” In addition, the term “determining” may include regarding operations such as resolving, selecting, choosing, establishing, or comparing as an operation such as “determining.” That is, the term “determining” may include regarding some kind of operation as an operation such as “determining” In addition, the term “determining” may be replaced with the term “assuming,” “expecting,” “considering,” or the like.
An expression “on the basis of ˜” which is used in the present disclosure does not refer to only “on the basis of only ˜,” unless otherwise described. In other words, the expression “on the basis of ˜” refers to both “on the basis of only ˜” and “on the basis of at least ˜.”
Any reference to elements having names such as “first” and “second” which are used in the present disclosure does not generally limit amounts or an order of the elements. The terms can be conveniently used to distinguish two or more elements in the present disclosure. Accordingly, reference to first and second elements does not mean that only two elements are employed or that the first element has to precede the second element in any form.
In the present disclosure, when the terms “include” and “including” and modifications thereof are used, these terms are intended to have a comprehensive meaning similarly to the term “comprising.” Further, the term “or” which is used in the present disclosure is intended not to mean an exclusive logical sum.
In the present disclosure, when articles are added by translation like, for example, “a,” “an” and “the” in English, the present disclosure may include that nouns that follow these articles are plural forms.
In the present disclosure, an expression “A and B are different” may mean that “A and B are different from each other.” Meanwhile, the expression may mean that “A and B are different from C.” The terms “separated,” “coupled,” and the like may also be construed similarly to “different.”
100: Video display system; 105: Storage unit; 106: Operation detection unit; 107: Control unit; 110: Structure; 111a: Main surface; 120: Jig; 121: First jig; 121a, 122a: Flat surface; 122: Second jig; 131, 132: Mounting member; 152: Structure arrangement detection unit; 153: User position detection unit; 154: Visual line direction detection unit; 155: Display arrangement detection unit; 156: Jig arrangement detection unit; 201, 202, 203, 204: Virtual mounting member.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2019-084245 | Apr 2019 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2019/050957 | 12/25/2019 | WO | 00 |
