The present invention relates to a validation system for VR HMDs, i.e. for virtual reality head mounted displays.
CN111583310A provides a virtual-real registration method and device for virtual reality and augmented reality equipment, and the method comprises the steps: calculating a projection matrix through an image photographed by a camera, and avoiding a subjective error caused by human eye setting. The sizes and distances of the test pattern and the virtual object can be adjusted according to actual conditions and the device can adapt to different models. Only two cameras and two computers are needed and are fixed, a moving device is not present during operation, and virtual-real mismatching caused by large equipment assembly difference can be effectively reduced in view of tilt and rotation.
CN111610638A provides an adjustment testing device and method for an optical module in binocular head-mounted equipment. The testing device comprises a first camera and a second camera for simulating two eyes, a camera positioning mechanism for calibrating the positions of the first camera and the second camera, a first adjusting mechanism and a second adjusting mechanism for respectively adjusting a first optical module and a second optical module, and a bracket fixing device for fixing a binocular bracket. The adjusting mechanism adjust the angle of the optical module according to the obtained rotation angle and inclination amount so as to compensate adverse effects of structural tolerance and assembling tolerance of the optical module on imaging quality.
US2021215940A1 is related to a calibration system that includes a calibration target having one or more calibration objects, a head mounted display (HMD), and a rendering engine configured to render, based on an estimated pose of the HMD, respective virtual calibration objects in a viewing area of the HMD at respective locations corresponding to the one or more calibration objects visible in the viewing area, wherein the HMD is configured to output the virtual calibration targets. The calibration system further includes an image capture system configured to capture image data comprising images of the viewing area of the HMD, and a calibration engine configured to compute, from the image data, difference data comprising respective distances between the one or more calibration objects and the respective, corresponding virtual calibration objects.
Segura Alvaro et al. have disclosed in the article “Improved virtual reality perception with calibrated stereo and variable focus for industrial use” in INTERNATIONAL JOURNAL ON INTERACTIVE DESIGN AND MANUFACTURING (IJIDEM); Springer (Paris), Paris, Vol. 12, no. 1, 8 Feb. 2017, pages 95-103, XP036423659, ISSN: 1955-2513, DOI: 10.1007/S12008-017-0377-0, a system for VR HMDs having a left eye display and a right eye display and provides a method to find the projection matrices for the two displays of an HMD for a fixed value IPD.
US 2021/215940 A1 relates to a calibration system for augmented or mixed reality HMDs with the goal of overlapping physical and virtual target object, which is a monocular problem since both sides need to match the physical target independently.
U.S. Pat. No. 10,277,893 B1 discloses a profiling system to characterize optical distortion of an HMD and measures effects of inter-pupillary distances (IPD) on the device under test, whereas projection is not part of the measurements and different camera IPDs are used to assess the effect of mismatched IPD's on the distortion.
In VR aviation training applications using HMDs (with VR for virtual reality and HMD for head mounted displays) the correct distance estimation and depth perception is essential, especially when the application is related to helicopter simulations where depth perception is essential to land the aircraft safely. There is an unacceptable risk of negative training on the simulator if depth perception and geometry are not correct.
Some HMDs have electronically sliding lenses and displays which can have play. Additionally, the lens may have a hysteresis, which means that the true lens position depends on whether the lens is shifted from the inside to the outside or from the outside to the inside.
The user's depth perception is manipulated and/or distorted if the displays of the HMD are not in the correct position. In addition, it can cause nausea and excessive eye strain for the user if the interpupillary distance of the lenses is not correct.
Based on this prior art it is an object of the present invention to provide a device allowing to check the binocular depth representation in a VR environment. This requirement stems from the difference between surreal applications like computer games and e.g. the present VR aviation application that in computer games a relative depth perception is sufficient and the real distance perception is unimportant.
Measurement of the interpupillary distance (in short IPD) in real time and adjusting the IPD accordingly is not precise enough. Therefore, the approach used in the framework of the present application, the interpupillary distance of a user is measured in advance or already known and then the HMD is manually adjusted to this value. This value must then also be precisely correctly reflected for the reasons mentioned above, so the device according to the invention provides the measurement and verification system of this value.
The verification system is based on a measuring device connected with a control unit configured to control the measuring device and presenting an HMD to be verified with the corresponding simulated images.
The measuring device for validating the IPD of an HMD has two cameras instead of the eyes of a user, wherein the cameras are intrinsically and extrinsically calibrated. These cameras can be moved in at least one dimension, i.e. inwards/outwards (IPD). Additional dimensions can be added to measure and validate, when the users eyes are not completely aligned with the height of the horizontal centerline of the HMD, i.e. the cameras can be moved upwards/downwards. The movement forwards and backwards check the influence of the position of the eyes in the eye sockets on the validation process. The degrees of freedom are used to perform the validation with deviation to the Design Eye Point (DEP). The HMD to be validated can have an electronically shiftable displays/lenses or manually shiftable lenses with an electronic displacement meter which shows the chosen IPD in the display. Other HMDs have a number of specific, fixed IPDs as e.g. 59, 64, 69 millimeter, or the distance between the displays and lenses is chosen by a graduated slider. It is noted that HMDs may show hysteresis, i.e. the shown or communicated IPD in the case of an electronic displacement meter for a given position of the slider or of one of the fixed positions of the HMD IPD is different between the two scenarios, if the position is reached starting from a lower IPD or from a higher IPD. Therefore, preferably the measurements are taken twice, once from lowest possible HMD IPD to highest possible HMD IPD and then vice versa. The measurement can then be decided to be passed, if a supplementary condition is met, i.e. that the distance difference between the determined distance for the same HMD IPD reached from both sides (high to low and low to high) is within a given threshold or ratio.
None of the cited documents mention the use of different IPD values for the HMD. Furthermore, they do not validate the accuracy of the IPD mechanism (motor and/or sensor) to ensure that the desired IPD matches the actual IPD of the HMD. This can be explained with an example: The projection can be calibrated with the teaching of the Segura article for a fixed HMD IPD of e.g. 64 mm and it passes the test as presented within Segura (or the US '893 B2 document). When the IPD of the HMD is set to a different desired value, e.g. 71 mm, the actual IPD of the HMD could be 69 mm. This can lead to the problems mentioned above.
The measuring device is connected to a control unit, configured to generate specific images for the two displays of the HMD creating a virtual target in a predetermined target distance. These images comprise one or more virtual target objects. The correct projection of the simulation is not the focus of the validation, but is also indirectly checked, since, if the projection is not correct, no successful measurement can be made. The correct projection is a necessary but not a sufficient condition.
The target for the IPD validation corresponds to a black area with a white dot in the center. Other targets are also possible, e.g. chessboard, black area with several white dots, etc., The two displays of the HMD, which show the virtual target, are captured with the two cameras.
A validation system for VR HMDs have a left eye display and a right eye display with an adjustable HMD IPD between them. A measurement device is configured in a way that an HMD to be validated is attachable to it in a way as such an HMD would be removably attached to the head of a user. The measurement device comprises a left and a right camera, wherein each camera is intrinsically and extrinsically calibrated. Each camera is mounted within the validation system for a movement of the cameras to adjust the camera IPD between the two cameras. Finally, a control unit is connected to the two cameras as well as configured to deliver control signals to the HMD, when the HMD is removably attached to the measurement device and when the HMD IPD of this VR HMD is manually or automatically adjusted to the prechosen camera IPD, wherein the control unit is configured to effect a validation measurement comprising the steps of:
Step a.) relates to the creation of the coordinates of the virtual target object vis-à-vis the user, i.e. the view of the user, wherein the user is here the unit comprising the left camera and the right camera. The coordinates have to be such that the virtual target object can be seen in the field of view of the measurement system.
These coordinate values have to be transformed by the control unit through calculation of control signals for controlling the display of the HMD to represent the image of this virtual object on this display. This comprises a transmitting part from the control unit towards the display as such.
Steps a.) and b.) are related to the images of the virtual target object. They are represented, usually as one or more dots on the displays of the HMD. Then, taking images of the displays with the left camera and the right camera is a feature related to the image of the displays.
The validation measurement with steps a.) to f.) can be conducted for a plurality of positions of a virtual target object having different predetermined virtual distances from the validation system and optionally different positions in the field of view of the HMD. Then, the IPD validation is only passed, if the distance validation of all or of a predetermined percentage of distance validations are passed. The intervals of validation can be different for different virtual distances.
The control unit can be configured to conduct steps a.) to f.) for at least two of the plurality of virtual target objects in one pass. Then, two or more virtual points are shown at the same time on the displays and the detection and calculation by the control unit are performed while such an image is displayed.
The validation measurement can comprise the distance validation of distances for a plurality of different HMD IPDs. This allows the check of an adjustable HMD which can then be validated for a number of different or range of IPDs. The distance validation is performed with a reduction of the target HMD IPD, starting from the highest possible IPD of the HMD to the lowest possible IPD of the HMD providing a IPD range validation for each IPD value of the HMD, or vice versa, wherein the device validation is only passed, if the IPD range validation of all or of a predetermined percentage of IPD interval validations are passed.
Preferably, the validation measurement further comprises the validation of distances for the same plurality of different HMD IPDs a second time, wherein the validation is performed with an increase of the target HMD IPD, starting from the lowest possible IPD of the HMD to the highest possible IPD of the HMD, or vice versa. Vice versa is here applied, if the first above mentioned device validation is performed from the lowest to highest. The hysteresis validation is only passed, if the difference or relative error between the IPD range validation value of the upward adjusted HMD IPD and the IPD range validation value of the downward adjusted HMD IPD is inside a hysteresis validation interval.
Each camera can be mounted within the validation system for a movement of the cameras to adjust the height position of the two cameras vis-à-vis the height of the horizontal centreline of the HMD when removably attached, creating height deviating from the DEP (design eye point) validation results. DEP can also be read as design eye position.
Each camera can also be mounted within the validation system for a movement of the cameras to adjust the relief distance of the two cameras from the HMD when removably attached, creating camera-eye HMD display distance deviating validation results.
Further embodiments of the invention are laid down in the dependent claims.
Preferred embodiments of the invention are described in the following with reference to the drawings, which are for the purpose of illustrating the present preferred embodiments of the invention and not for the purpose of limiting the same. In the drawings,
The control unit 100 is connected to the measurement device 10 with a communication line 102 to move the measurement device 10 as explained in connection with
The measurement device 10 has instead of two eyes two cameras, i.e. a left camera 11 and a right camera 12, respectively. Both cameras 11 and 12 are intrinsically and extrinsically calibrated. Such a calibration can e.g. be effected by using the approach of a Tsai calibration as published in R. Y. Tsai, “Metrology Using Off-the-Shelf TV Cameras and Lenses” in IEEE Journal of Robotics and Automation, Vol. 3, No. 4, pp. 323-344, August 1987. Other calibration models can be used as well.
The left and right cameras 11 and 12 can be moved in three dimensions: an inward/outward (IPD) movement 131 to set the camera IPD 130, a forward/backward or relief movement 111 representing the position and thus distance 110 of the eyes from the HMD 20 and an upwards/downwards or vertical movement 121 to represent the height position of the eyes vis-à-vis the height position of the optical system with the optics 23, 24 of the HMD 20. The three directions are orthogonally one to each other and span a Cartesian coordinate system. The vertical movement 121 for the height position 120 as well as the relief movement 111 for the eye socket position 110 are performed for both cameras 11 and 12 whereas the IPD movement for IPD 130 is per definition a movement of the two cameras 11 and 12 independently one from the other but preferably symmetrically to the mirror axis between the two cameras 11 and 12. It is noted that according to the invention and the main claim only one movement is needed: an inward/outward (IPD) movement 131 to set the camera IPD 130.
The total IPD movement 131 is divided in two identical portions, i.e. 50% of the movement to the right for the left camera 11 and 50% of the movement to the left right for the right camera 12. It is an advantage to have a correlated left and right movement of the camera 11 and 12, since it is then possible to give the measurement device the form of a human head with a nose, centering the HMD on the nose, when attached with a band 25 around the head, and having half of the IPD to the left and half of the IPD to the right of the vertical symmetry plane being the sagittal plane.
Reference numerals 130′ for the camera IPD, 120′ for the height position of the cameras as well as 110′ for the eye socket position are the positions after the movement and adjustment of the measurement device. The height of the horizontal centreline of the HMD compared to the horizontal centreline of the eyes of the user is mainly predetermined by the way the HMD 20 is positioned on the head. Therefore, using the IPD validation method for different height positions of the cameras 11 and 12, e.g. a low, a centred and a high height position allows to determine the influence of this departure from the DEP (Design Eye Point).
The movement of the cameras 11 and 12 from their initial position towards the shown positions 11′ and 12′ allow the positioning aligned before the left eye display 21 and right eye display 22 of the HMD. In front of the left and right eye displays 21 and 22, respectively are positioned a left side optics 23 and a right side optics 24. The optics 23 and 24 are mechanically connected to the respective displays 21 and 22 for a connected sideways movement according to arrow 231 being the adjustment of the HMD IPD 230. In other words, arrow 231 represents the possibility for the user to adapt the HMDs IPD 230 to his personal IPD. The aim of the present measurement device 10 is to validate the HMD 20 for one chosen HMD IPD 230 which can then be adjusted by the user for his own experience, preferably based on a previously measured human IPD, i.e. adjusting the HMD IPD value to this known value. The validation method can be reduced to be only conducted for one single IPD. However, in view of a validation procedure for a predetermined HMD device 20, it is contemplated that such an HMD is validated for specific chosen plurality of HMD IPDs, preferably for a range of IPD values covering the entire adjustable range of the HMD, e.g. at least a minimal IPD, a middle IPD and a maximal IPD, optimally for every settable value of the HMD 20.
The degrees of freedom are used to perform the validation with deviation to the Design Eye Point (DEP) or Design Eye Reference Point (DERP), or to meet the different DEPs of different HMD models. This is reflected by the measurement IPD distance. It is also referred to as Design Eye Position.
The HMD 20 can have electronically shiftable displays 21, 22 and lenses (optics) 23, 24. The lenses (optics) 23, 24 can also be manually shiftable lenses with an electronic displacement meter. This is mainly related to current technology of VR-HMDs 20.
A flight simulation software can be used to generate the images 400L, 400R or 410L, 410R for the two displays 21, 22 of the HMD 20 as shown in
The (virtual object) target 401 for the IPD validation corresponds to a black area 400 with a white dot 401 in the center. Other targets are also possible, e.g. chessboard, black area with several white dots, etc. and the target 411 is e.g. a corner of four areas. The content of the two displays 21 and 22 of the HMD 20, which show the virtual target 401, are captured with the two cameras 21 and 22. The only important common feature of these representations is the clear distinction of providing a point as target having a specific distance (i.e. Z-position) as well as an X position and a Y position.
It is also possible to provide different target configurations and especially providing several targets in one image checking either several distances at the same time or the same distance at different X-Y positions, i.e. different positions in the field of view of the HMD display in order to speed up the sequence of tested images in real time.
Otherwise, usually a plurality of distances and positions are tested and a decision step 340 refers the method back via a new parameter changing step 350 to the new placement step. The parameters of the different distances to be tested are preferably initially predetermined and stored in a database. When all distance tests for the virtual target have been passed within the given threshold parameters, then the test has been successfully passed with the test result 370.
In this context it is mentioned that different threshold values can be given, i.e. predetermined and stored, for the comparison test step 330, i.e. a different absolute value for a first near distance smaller than e.g. 10 to 20 meter and a different greater threshold for a middle distance greater than 10 to 20 meter, respectively, and a further greater threshold for a greater distance greater than 50 to 100 meter. It is also possible to provide a threshold curve as e.g. a monotone increasing threshold for any specific distance to be tested. The threshold can be a percentage of the intended distance.
However, said
| Number | Date | Country | Kind |
|---|---|---|---|
| 21214021.4 | Dec 2021 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2022/085573 | 12/13/2022 | WO |