Patent Grant
12204098
Not Applicable.
Not Applicable.
Not Applicable.
The invention generally relates to a head-mounted display system. More particularly, the invention relates to a head-mounted display system configured to be used for balance testing and training.
Measurement and testing systems are utilized in various fields to detect and analyze many different measurable quantities. For example, in biomedical applications, measurement and testing systems are used for gait analysis, assessing balance and mobility, evaluating sports performance, and assessing ergonomics. However, conventional measurement and testing systems have numerous limitations and drawbacks.
For example, conventional measurement and testing systems with large measurement surface areas and large visual displays are complex, difficult to install, and are not easily adaptable to different space configurations in a building. Also, these conventional measurement and testing systems are typically cost prohibitive for small clinical applications.
What is needed, therefore, is a head-mounted display system for assessing balance and mobility. Moreover, a head-mounted display system is needed for enhancing the visual motor performance of individuals. Furthermore, a need exists for a head-mounted display system as part of the rehabilitation regime for an orthopedic and/or neurological injury.
Accordingly, the present invention is directed to a head-mounted display system that substantially obviates one or more problems resulting from the limitations and deficiencies of the related art.
In accordance with one or more other embodiments of the present invention, there is provided a head-mounted display system that includes an input device, the input device configured to output an input signal based upon an input response by the user; a head-mounted visual display device having an output screen, the head-mounted visual display device configured to display one or more screen images on the output screen so that the one or more screen images are viewable by the user; and at least one data processing device, the at least one data processing device operatively coupled to the input device and the head-mounted visual display device. The at least one data processing device being programmed to: (i) generate and display at least one visual target on the output screen of the head-mounted visual display device; (ii) receive an input signal from the input device based upon an input response by the user; (iii) determine an orientation angle of a body portion of the user based upon the input signal received from the input device; and (iv) determine how closely the orientation angle of the body portion of the user corresponds to a tilt of the at least one visual target on the output screen of the head-mounted visual display device.
In a further embodiment of the present invention, the input device comprises a head position sensing device and the input signal comprises one or more measurement signals outputted by the head position sensing device, the one or more measurement signals being generated based upon a head movement of the user, and the orientation angle of the body portion of the user comprises a head angle of the user based upon the one or more measurement signals outputted by the head position sensing device.
In yet a further embodiment, the head-mounted visual display device comprises the head position sensing device.
In accordance with one or more other embodiments of the present invention, there is provided a head-mounted display system that includes an input device, the input device configured to output an input signal based upon an input response by a user; a head-mounted visual display device having an output screen, the head-mounted visual display device configured to display one or more images on the output screen so that the one or more images are viewable by the user; at least one camera for enabling the user to see one or more images of a real-world environment outside of the head-mounted visual display device, the at least one camera configured to capture the one or more images of the real-world environment outside of the head-mounted visual display device; and at least one data processing device, the at least one data processing device operatively coupled to the input device and the head-mounted visual display device. The at least one data processing device being programmed to: (i) generate augmented digital content; (ii) receive an input signal from the input device based upon an input response by the user; and (iii) in response to the input signal from the input device, display at least a portion of the one or more images of the real-world environment captured by the at least one camera on the output screen of the head-mounted visual display device, and overlay the augmented digital content over the one or more images of the real-world environment so that the user is able to quickly check his or her surroundings without removing the head-mounted visual display device from his or her head.
In a further embodiment of the present invention, the at least one data processing device is further programmed to: (iv) receive a defined area within the real-world environment from the user that establishes one or more boundaries of the real-world environment; and (v) display the one or more boundaries of the real-world environment within a virtual reality environment or an augmented reality environment so as to prevent the user from colliding with one or more objects in the real-world environment.
In yet a further embodiment, the at least one data processing device is further programmed to: (iv) manipulate the augmented digital content such that the augmented digital content interacts with one or more actual objects in the real-world environment.
In still a further embodiment, the at least one data processing device is further programmed to: (iv) overlay instructions or data over the one or more images of the real-world environment so that the user is able to receive guidance as to how to interact with the real-world environment.
In yet a further embodiment, the at least one data processing device is further programmed to: (iv) capture one or more hand movements of the user in the real-world environment using the at least one camera; and (v) enable the user to interact with one or more virtual elements using the one or more hand movements of the user captured from the real-world environment.
In still a further embodiment, the at least one data processing device is further programmed to: (iv) perform object recognition on the one or more images of the real-world environment captured by the at least one camera so as to identify one or more real-world objects; and (v) generate the augmented digital content based upon the one or more real-world objects identified in the real-world environment so as to provide a context-aware augmented reality experience.
In accordance with yet one or more other embodiments of the present invention, there is provided a system for body pose estimation in a virtual reality or augmented reality environment that includes one or more egocentric cameras coupled to a user, the one or more egocentric cameras configured to capture video data from a perspective of the user; an instrumented flooring system, the instrumented flooring system including one or more sensors configured to detect foot placement and/or movement of the user, and generate sensor output data based on the detected foot placement and/or movement of the user; and at least one data processing device, the at least one data processing device operatively coupled to the one or more egocentric cameras and the one or more sensors of the instrumented flooring system, the at least one data processing device being configured to receive and integrate the video data from the one or more egocentric cameras and the sensor output data from the instrumented flooring system using sensor fusion algorithms, thereby generating a comprehensive body pose estimation for the user.
In a further embodiment of the present invention, the one or more egocentric cameras coupled to the user comprise a plurality of egocentric cameras that are attached to different parts of the body of the user to capture a range of perspectives and movements of the user, the different parts of the body of the user being selected from a group consisting of: (i) a head of the user, (ii) a chest of the user, (iii) one or more arms of the user, (iv) one or more legs of the user, and (v) combinations thereof.
In yet a further embodiment, the one or more sensors of the instrumented flooring system comprise a plurality of sensors configured to detect a weight distribution, foot placement, and/or gait patterns of the user, and the plurality of sensors are selected from a group consisting of: (i) pressure sensors, (ii) capacitive sensors, (iii) load cells, and (iv) combinations thereof.
In still a further embodiment, the at least one data processing device is further configured to execute synchronization protocols for aligning the video data from the one or more egocentric cameras and the sensor output data from the instrumented flooring system in a common coordinate system, thereby ensuring coherent and accurate pose estimation.
In yet a further embodiment, the at least one data processing device is further configured to perform real-time processing of the video data from the one or more egocentric cameras and the sensor output data from the instrumented flooring system so as to provide quasi-instantaneous updates to the virtual reality or augmented reality environment based on body movements of the user.
In still a further embodiment, the sensor fusion algorithms executed by the at least one data processing device include error correction mechanisms to compensate for potential drift in the video data from the one or more egocentric cameras using stable reference points from the instrumented flooring system.
In yet a further embodiment, the at least one data processing device is further configured to dynamically update the virtual reality or augmented reality environment based on the body pose estimation for the user, thereby enhancing user interaction and immersion in the virtual reality or augmented reality environment.
In still a further embodiment, the sensor fusion algorithms executed by the at least one data processing device include feedback loops to continuously refine the body pose estimation for the user by compensating for drift in the video data from the one or more egocentric cameras using reference points from the instrumented flooring system.
In yet a further embodiment, the at least one data processing device is further configured to provide real-time feedback and interaction capabilities in the virtual reality or augmented reality environment based on body movements of the user.
In still a further embodiment, the at least one data processing device is further configured to dynamically adjust training scenarios for the user in response to the body movements of the user, thereby enhancing the effectiveness of a training session.
In accordance with still one or more other embodiments of the present invention, there is provided a head-mounted display system that includes a head-mounted visual display device having an output screen, the head-mounted visual display device configured to display one or more screen images on the output screen so that the one or more screen images are viewable by the user; a body pose estimation subsystem, the body pose estimation subsystem configured to capture body pose information for the user of the head-mounted visual display device, and generate corresponding pose output data; and at least one data processing device, the at least one data processing device operatively coupled to the head-mounted visual display device and the body pose estimation subsystem, the at least one data processing device being programmed to modify and/or augment the one or more screen images displayed on the output screen of the head-mounted visual display device by using the pose output data received from the body pose estimation subsystem.
In a further embodiment of the present invention, the body pose estimation subsystem captures the body pose information for the user using one or more of the following means: (i) marker-based motion capture using cameras, (ii) markerless motion capture using cameras, (iii) inertial measurement units (IMUs) attached to the user, (iv) egocentric cameras attached to the user, (v) WiFi network based pose estimation, (vi) wall-mounted LIDAR based pose estimation, (vii) instrumented floor-based pose sensors, and (viii) combinations thereof.
In accordance with yet one or more other embodiments of the present invention, there is provided a method for estimating full-body pose in a virtual reality or augmented reality environment. The method comprises the steps of: (i) capturing video data from a perspective of a user by using a plurality of egocentric cameras coupled to a body of the user; (ii) detecting foot placement and/or movement data for the user using an instrumented flooring system comprising one or more sensors; (iii) integrating, using at least one data processing device, the video data from the plurality of egocentric cameras and the foot placement and/or movement data from the one or more sensors of the instrumented flooring system using sensor fusion algorithms to generate a comprehensive full-body pose estimation for the user; and (iv) dynamically updating, using the at least one data processing device, the virtual reality or augmented reality environment based on the full-body pose estimation for the user.
In a further embodiment of the present invention, the method further comprises the step of calibrating the egocentric cameras and the instrumented flooring system to operate in a common coordinate system, ensuring accurate data integration.
In yet a further embodiment, the sensor fusion algorithms include feedback loops to continuously refine pose estimation by compensating for drift in the video data from the plurality of egocentric cameras by using reference points from the instrumented flooring system.
In still a further embodiment, the method further comprises the step of providing real-time feedback and interaction capabilities in the virtual reality or augmented reality environment based on full-body movements of the user.
In accordance with still one or more other embodiments of the present invention, there is provided a non-transitory computer-readable medium containing instructions that, when executed by a processor of the at least one data processing device, cause the processor to perform at least the steps of: (i) integrating the video data from the plurality of egocentric cameras and the foot placement and/or movement data from the one or more sensors of the instrumented flooring system using sensor fusion algorithms to generate a comprehensive full-body pose estimation for the user; and (ii) dynamically updating the virtual reality or augmented reality environment based on the full-body pose estimation for the user.
In accordance with yet one or more other embodiments of the present invention, there is provided a virtual reality or augmented reality training system that includes one or more egocentric cameras coupled to a trainee, the one or more egocentric cameras configured to capture video data from a perspective of the trainee; an instrumented flooring system, the instrumented flooring system including one or more sensors configured to detect foot placement and/or movement of the trainee, and generate sensor output data based on the detected foot placement and/or movement of the trainee; at least one data processing device, the at least one data processing device operatively coupled to the one or more egocentric cameras and the one or more sensors of the instrumented flooring system, the at least one data processing device being configured to receive and integrate the video data from the one or more egocentric cameras and the sensor output data from the instrumented flooring system using sensor fusion algorithms, thereby generating a comprehensive body pose estimation for the trainee; and a feedback module configured to provide real-time feedback to the trainee based on the comprehensive body pose estimation for the trainee.
In a further embodiment of the present invention, the feedback module adjusts training scenarios dynamically in response to movements of the trainee, thereby enhancing the effectiveness of a training session.
It is to be understood that the foregoing summary and the following detailed description of the present invention are merely exemplary and explanatory in nature. As such, the foregoing summary and the following detailed description of the invention should not be construed to limit the scope of the appended claims in any sense.
The invention will now be described, by way of example, with reference to the accompanying drawings, in which:
Throughout the figures, the same parts are always denoted using the same reference characters so that, as a general rule, they will only be described once.
The present invention is described herein, in an exemplary manner, with reference to computer system architecture and exemplary processes carried out by the computer system. In one or more embodiments, the functionality described herein can be implemented by computer system instructions. These computer program instructions may be loaded directly onto an internal data storage device of a computing device (e.g., an internal data storage device of a laptop computing device and/or a data processing device within a head-mounted display). Alternatively, these computer program instructions could be stored on a portable computer-readable medium (e.g., a flash drive, etc.), and then subsequently loaded onto a computing device such that the instructions can be executed thereby. In other embodiments, these computer program instructions could be embodied in the hardware of the computing device, rather than in the software thereof. It is also possible for the computer program instructions to be embodied in a combination of both the hardware and the software.
This description describes in general form the computer program(s) required to carry out the functionality of the head-mounted display system described herein. Any competent programmer in the field of information technology could develop a system using the description set forth herein.
For the sake of brevity, conventional computer system components, conventional data networking, and conventional software coding will not be described in detail herein. Also, it is to be understood that the connecting lines shown in the block diagram(s) included herein are intended to represent functional relationships and/or operational couplings between the various components. In addition to that which is explicitly depicted, it is to be understood that many alternative or additional functional relationships and/or physical connections may be incorporated in a practical application of the system.
1. Illustrative Head-Mounted Display System
An illustrative embodiment of a head-mounted display (HMD) system is seen generally at 100 in
In the illustrative embodiment, the head-mounted display 30 may have the following exemplary specifications: (i) a single liquid crystal display (LCD) binocular display, (ii) a resolution of at least 1830×1920 per eye, (iii) a refresh rate of at least 90 Hz, (iv) a horizontal visible field of view (FOV) of at least 98 degrees, (v) a vertical visible field of view (FOV) of at least 90 degrees, (vi) a built-in eye tracking device, (vii) a Qualcomm Snapdragon 865 chip set, (viii) at least 6 GB memory, and (ix) at least 128 GB storage.
Referring again to the illustrative embodiment of
In addition, as shown in the illustrative embodiment of
In the illustrative embodiment, while performing the training routines and tests described hereinafter, the user may use the user input devices 34, 36 in order to enter and transmit his or her responses to the at least one data processing device (e.g., to the data processing device in the head-mounted display 30 and/or the remote laptop 20). For example, the user may use the user input devices 34, 36 to select a particular visual object on the output screen of the visual display device 32 of the head-mounted display 30.
In the illustrative embodiment, the head-mounted display system 100 may further include headset auto-discovery and connection functionality. In particular, the software running on the headset 30 may respond to a status ping over standard WiFi networking UDP broadcasts, initiating a connection back to the base station (e.g., the laptop computing device 20) that sent the status ping. This in turn creates a handshaking opportunity for the two systems to negotiate a secure crypto client-server protocol.
In the illustrative embodiment, the head-mounted display system 100 may further include a headset hands-off configuration. In particular, if during the auto discovery and connection, the base station (e.g., the laptop computing device 20) determines that the headset 30 is running outdated software or an updated configuration is needed, the user is prompted to plug the headset 30 into the base station via a standard USB cable. The laptop software will then configure the headset 30 and restart it, performing all needed updates and configuration steps without the user's involvement or actions (typically, this would need the user to wear the headset and use the controllers to navigate through several sub-menus and programs, and copy files from the computer or USB thumb drive into the headset). In the illustrative embodiment, the total automated configuration time is normally under two (2) minutes.
Now, turning again to the illustrative embodiment of
Referring again to
In the illustrative embodiment, the force measurement assembly 10 is operatively coupled to the data processing device 20 by virtue of an electrical cable. In one embodiment, the electrical cable is used for data transmission, as well as for providing power to the force measurement assembly 10. Various types of data transmission cables can be used for the cable of the force measurement assembly 10. For example, the cable can be a Universal Serial Bus (USB) cable or an Ethernet cable. Preferably, the electrical cable contains a plurality of electrical wires bundled together, with at least one wire being used for power and at least another wire being used for transmitting data. The bundling of the power and data transmission wires into a single electrical cable advantageously creates a simpler and more efficient design. In addition, it enhances the safety of the training environment for the user. However, it is to be understood that the force measurement assembly 10 can be operatively coupled to the data processing device 20 using other signal transmission means, such as a wireless data transmission system. If a wireless data transmission system is employed, it is preferable to provide the force measurement assembly 10 with a separate power supply in the form of an internal power supply or a dedicated external power supply.
Now, the acquisition and processing of the load data carried out by the illustrative embodiment of the head-mounted display system 100 will be described. Initially, a load is applied to the force measurement assembly 10 by the user disposed thereon. The load is transmitted from the first and second plate components 12, 14 of the dual force plate 10 to its force transducer beams. In the illustrative embodiment, each plate component 12, 14 of the dual force plate 10 is supported on a pair of force transducer beams disposed thereunder. In the illustrative invention, each of the force transducer beams includes a plurality of strain gages wired in one or more Wheatstone bridge configurations, wherein the electrical resistance of each strain gage is altered when the associated portion of the associated beam-type force transducer undergoes deformation (i.e., a measured quantity) resulting from the load (i.e., forces and/or moments) acting on the first and second plate components 12, 14. For each plurality of strain gages disposed on the force transducer beams, the change in the electrical resistance of the strain gages brings about a consequential change in the output voltage of the Wheatstone bridge (i.e., a quantity representative of the load being applied to the measurement surface). Thus, in the illustrative embodiment, the pair of force transducer beams disposed under the plate components 12, 14 output a total of six (6) analog output voltages (signals). In the illustrative embodiment, the six (6) analog output voltages from dual force plate are then transmitted to a preamplifier board (not shown) for preconditioning. The preamplifier board is used to increase the magnitudes of the transducer analog voltages, and preferably, to convert the analog voltage signals into digital voltage signals as well. After which, the force measurement assembly 10 transmits the force plate output signals SFPO1-SFPO6 to a main signal amplifier/converter. Depending on whether the preamplifier board also includes an analog-to-digital (A/D) converter, the force plate output signals SFPO1-SFPO6 could be either in the form of analog signals or digital signals. The main signal amplifier/converter further magnifies the force plate output signals SFPO1-SFPO6, and if the signals SFPO1-SFPO6 are of the analog-type (for a case where the preamplifier board did not include an analog-to-digital (A/D) converter), it may also convert the analog signals to digital signals. In the illustrative embodiment, the force plate output signals SFPO1-SFPO6 may also be transformed into output forces and/or moments (e.g., FLz, MLx, MLy, FRz, MRx, MRy) by the firmware of the dual force plate by multiplying the voltage signals SFPO1-SFPO6 by a calibration matrix prior to the force plate output data being transmitted to the data processing device 20. Alternatively, the data acquisition/data processing device 20 may receive the voltage signals SFPO1-SFPO6, and then transform the signals into output forces and/or moments (e.g., FLz, MLx, MLy, FRz, MRx, MRy) by multiplying the voltage signals SFPO1-SFPO6 by a calibration matrix.
After the voltage signals SFPO1-SFPO6 are transformed into output forces and/or moments (e.g., FLz, MLx, MLy, FRz, MRx, MRy), the center of pressure for each foot of the user (i.e., the x and y coordinates of the point of application of the force applied to the measurement surface by each foot) may be determined by the data acquisition/data processing device 20. If the force transducer technology described in U.S. Pat. No. 8,544,347 is employed, it is to be understood that the center of pressure coordinates (xP
In one or more alternative embodiments, the data processing device 20 determines the vertical forces FLz, FRz exerted on the surface of the first and second force plates by the feet of the user and the center of pressure for each foot of the user, while in another embodiment where a six component force plate is used, the output forces of the data processing device 20 includes all three (3) orthogonal components of the resultant forces acting on the two plate components 12, 14 (i.e., FLx, FLy, FLz, FRx, FRy, FRz) and all three (3) orthogonal components of the moments acting on the two plate components 12, 14 (i.e., MLx, MLy, MLz, MRx, MRy, MRz). In yet other embodiments of the invention, the output forces and moments of the data processing device 20 can be in the form of other forces and moments as well.
In the illustrative embodiment, where a single set of overall center of pressure coordinates (xP, yP) are determined for the force measurement assembly 10, the center of pressure of the force vector F applied by the user to the measurement surface of the force plate 22 is computed as follows:
where:
In addition, in a further embodiment, the head-mounted display system 100 further comprises a data interface configured to operatively couple the data processing device 20 to a remote computing device (e.g., remote laptop or desktop computing device) so that data from the data processing device 20 is capable of being transmitted to the remote computing device. In one or more embodiments, the data interface may comprise a wireless data interface or a wired data interface operatively coupling the data processing device 20 to the remote computing device.
2. Testing and Training Functionality of the Head-Mounted Display System
Now, with reference to the screen images of
An exemplary screen image of an operator/clinician home screen 50 of the head-mounted display system 100 is shown in
Turning to
In the first set of training routines carried out by the illustrative head-mounted display system 100, the user input device comprises one or more hand controllers 34, 36 of the head-mounted visual display device 30, and the input signal to the at least one data processing device (i.e, first data processing device and/or second data processing device) comprises one or more hand control signals outputted by the one or more hand controllers 34, 36. The one or more hand control signals are generated based upon a hand movement of the user. The at least one data processing device is configured to receive the one or more hand control signals that are generated based upon the hand movement of the user, and to control the movement of at least one displaceable visual object on the output screen 32 of the head-mounted visual display device 30 towards at least one visual target.
In particular, in one subset of the first set of illustrative training routines, the visual scene on the visual display device 32 of the head-mounted display 30 is designed to assess the user's ability to align a rod with respect to the gravitational vertical (0°) or horizontal by using the hand controller 34, 36 or middle finger trigger of the controller. The objective is for the user to use the controller 34, 36 to align the rod with respect to the gravitational vertical (SVV and R&F) or horizontal (SVH) in a fully immersive virtual reality (VR) scene. In these training routines, the at least one data processing device is configured to determine how closely the user is able to align the at least one displaceable visual object (e.g., the rod) relative to the at least one visual target (gravitational vertical or horizontal). During the training routines, the user (e.g., the patient) is asked to use the right and/or left controllers 34, 36 to position the rod at the gravitational vertical or horizontal. He or she will submit their response by using the index trigger button 38. Once the response is submitted, the correct rod angle will display. Both rods will then disappear, and a new rod will show. Various levels of difficulty can be implemented with optokinetic and visual flow options.
In the illustrative embodiment, the operator can choose from different optokinetic scenes that will be used in the background of the rod (e.g., stripped or starfield). Also, the operator can define the color of the rod (e.g., black, red, or green). The operator additionally can define the direction of the optokinetic movement (e.g., up, down, left, right), the speed of the optokinetic movement (e.g., speed ranges from 0-25 in increments of 5), the density of the optokinetic scene (e.g., low, medium, or high).
In the illustrative embodiment, the operator can choose the type of visual flow scene that will be used in the background of the rod (e.g., park, boardwalk, or driving). Also, the operator can define the speed of the scene's movement (e.g., slow, medium, or fast).
In another one of the first set of illustrative training routines, the user is positioned either sitting or standing, with the headset 30 on and the controllers 34, 36 in the correct hands. The user is shown a sequence of items 144, 146 sitting on a grocery store shelf 142 (see screen image 140 in
In yet another one of the first set of illustrative training routines, the user is positioned either sitting or standing, with the headset 30 on and the controllers 34, 36 in the correct hands. The user is shown several items 154, 156 on a grocery shelf 152 at the same time (see screen image 150 in
In still another one of the first set of illustrative training routines, the user is either sitting or standing with the headset 30 on and the controllers 34, 36 in the correct hands. He or she is shown a grocery store shelf 162 with an item 164 on it that will be randomly highlighted in one of two colors (see screen image 160 in
In yet another one of the first set of illustrative training routines, the user is either sitting or standing with the headset 30 on and the controllers 34, 36 in the correct hands. A grocery store shelf scene will be displayed on the headset 30 (see screen image 170 in
With reference to the screen image 180 in
In the second set of training routines carried out by the illustrative head-mounted display system 100, the user input device comprises a force measurement assembly 10 (see e.g., user 40 disposed on dual force plate 10) and the input signal comprises one or more measurement signals outputted by one or more force measurement devices of the force measurement assembly. The one or more measurement signals are generated based upon the user's contact with a surface of the force measurement assembly 10. The at least one data processing device is configured to receive the one or more measurement signals that are generated based upon the user's contact with the surface of the force measurement assembly 10 and to compute one or more numerical values using the one or more measurement signals. The data processing device is configured to control the movement of the at least one displaceable visual object on the output screen 32 of the head-mounted visual display device 30 towards the at least one visual target by using the one or more computed numerical values.
In the second set of training routines carried out by the illustrative head-mounted display system 100, the center of pressure (center-of-gravity) of the user, which is determined by the at least one data processing device from the force and moment output data of the force measurement assembly 10 described above, is used to control the displacement of a visual object (e.g., a cursor) on the output screen 32 of the head-mounted visual display device 30.
In particular, in one subset of the second set of illustrative training routines, the user is asked to maintain balance throughout a moving scene while having the option of dodging objects by shifting his or her center-of-gravity (COG), answering simple questions, various distractions, and noise distractions. In this training routine, the at least one data processing device is configured to generate and display a displaceable scene (see e.g.,
In another subset of the second set of illustrative training routines, the user is positioned on the force/balance plate 10 in the correct position and the headset 30 is placed on the user's head. The operator/clinician then chooses one of the quick training protocols and chooses one of the seven scene options. Each protocol has a different target area, and the user is to stay on the balance plate 10 and shift his or her weight towards the plurality of targets being displayed on the headset 30 (see e.g.,
An exemplary operator/clinician screen 108 for the fountain quick training scene is depicted in
In the third set of training routines carried out by the illustrative head-mounted display system 100, the user input device comprises a head position sensing device (e.g., an inertial measurement unit for measuring a user's head position, rotation, velocity, and acceleration), and the input signal comprises one or more measurement signals outputted by the head position sensing device. The one or more measurement signals are generated based upon a head movement of the user, and the at least one data processing device is configured to determine a head angle of the user based upon the one or more measurement signals outputted by the head position sensing device. In the illustrative embodiment, the head-mounted visual display device 30 may comprise the head position sensing device (i.e., the head-mounted visual display device 30 may comprise the inertial measurement unit disposed therein).
In particular, in one subset of the third set of illustrative training routines, using a rod on the screen, a user must align his or her head with the rod and use the index trigger on the hand controller 34, 36 to submit his or her response when he or she is aligned. In the illustrative embodiment, the rod will disappear on submission and a new rod will appear at a new angle, at which point the user will align with the new rod. This routine continues for the number of trials selected. The objective in this training routine is for the user to tilt his or her head (+/−45 degrees max) to align with the rod or scene inside the headset 30. The visual scene is designed for the user to align his or her head with the rod tilt angle or scene that appears on the screen 32 of the headset 30 while the user is sitting or standing. In the illustrative embodiment, there are three (3) separate Head Tilt Response (HTR) training protocols: (i) head tilt response (HTR), (ii) head tilt response visual flow (HTR-VF), and (iii) head tilt response optokinetics (HTR-OPK). An example of head tilt response optokinetics is shown in
Next, referring to
In a further illustrative embodiment, the head-mounted display system 100 may include headset data collection and synthesis functionality. In particular, during the specific training/test protocols, the headset 30 records and transmits position and rotation values, along with velocity (e.g., now at 5 degrees left and moving at 2.3 degrees per second). The headset 30 also reports eye tracking movement and gaze lingering, along with hand controller position/rotation/velocity. This data from the headset 30 is sampled at a high rate and combined with the force plate data to present a complete ‘picture’ of the user at any given moment in time (the data rate is sampled at 1000 Hz to match the force plate 10). Combining the data together into existing protocols—e.g., Quick Training where the user must follow the dots by shifting body weight—it can be determined if the user is swaying his or her upper body, turning his or her head to look at the target pattern, and/or if his or her eyes are tracing the displaceable cursor 128, 138 as it moves into the target (or if he or she looks at the target first then move). The visual flow scenes, such as the boardwalk and forest scenes, provide additional opportunities for data synthesis with the eye tracking the various distractors, such as the birds flying through the forest in the forest scene
In the third set of illustrative training routines, the inertial measurement unit (IMU) forming the head position sensing device in the headset 30 may comprise a triaxial (three-axis) accelerometer sensing linear acceleration {right arrow over (a)}′, a triaxial (three-axis) rate gyroscope sensing angular velocity {right arrow over (ω)}′, a triaxial (three-axis) magnetometer sensing the magnetic north vector {right arrow over (n)}′, and a central control unit or microprocessor operatively coupled to each of accelerometer, gyroscope, and the magnetometer.
Next, an illustrative manner in which the at least one data processing device of the head-mounted display system 100 performs the inertial measurement unit (IMU) calculations will be explained in detail. In particular, this calculation procedure will describe the manner in which the orientation and position of the head of the user could be determined using the signals from the inertial measurement unit (IMU) of the system 100. As explained above, in the illustrative embodiment, the inertial measurement unit may include the following three triaxial sensor devices: (i) a three-axis accelerometer sensing linear acceleration {right arrow over (a)}′, (ii) a three-axis rate gyroscope sensing angular velocity {right arrow over (ω)}′, and (iii) a three-axis magnetometer sensing the magnetic north vector {right arrow over (n)}′. The inertial measurement unit senses in the local (primed) frame of reference attached to the IMU itself. Because each of the sensor devices in the IMU is triaxial, the vectors {right arrow over (a)}′, {right arrow over (ω)}′, {right arrow over (n)}′ are each 3-component vectors. A prime symbol is used in conjunction with each of these vectors to symbolize that the measurements are taken in accordance with the local reference frame. The unprimed vectors that will be described hereinafter are in the global reference frame.
The objective of these calculations is to find the orientation {right arrow over (θ)}(t) and position {right arrow over (R)}(t) in the global, unprimed, inertial frame of reference. Initially, the calculation procedure begins with a known initial orientation {right arrow over (θ)}0 and position {right arrow over (R)}0 in the global frame of reference.
For the purposes of the calculation procedure, a right-handed coordinate system is assumed for both global and local frames of reference. The global frame of reference is attached to the Earth. The acceleration due to gravity is assumed to be a constant vector 4. Also, for the purposes of the calculations presented herein, it is presumed the sensor devices of the inertial measurement unit (IMU) provide calibrated data. In addition, all of the signals from the IMUs are treated as continuous functions of time. Although, it is to be understood the general form of the equations described herein may be readily discretized to account for IMU sensor devices that take discrete time samples from a bandwidth-limited continuous signal.
The orientation {right arrow over (θ)}(t) is obtained by single integration of the angular velocity as follows:
where {right arrow over (θ)}(t) is the matrix of the rotation transformation that rotates the instantaneous local frame of reference into the global frame of reference.
The position is obtained by double integration of the linear acceleration in the global reference frame. The triaxial accelerometer of the IMU senses the acceleration {right arrow over (a)}′ in the local reference frame. The acceleration {right arrow over (a)}′ has the following contributors: (i) the acceleration due to translational motion, (ii) the acceleration of gravity, and (iii) the centrifugal, Coriolis and Euler acceleration due to rotational motion. All but the first contributor has to be removed as a part of the change of reference frames. The centrifugal and Euler accelerations are zero when the acceleration measurements are taken at the origin of the local reference frame. The first integration gives the linear velocity as follows:
where 2{right arrow over (ω)}×{right arrow over (v)}′(t) is the Coriolis term, and where the local linear velocity is given by the following equation:
{right arrow over (v)}′(t)={right arrow over (Θ)}−1(t){right arrow over (v)}(t) (7)
The initial velocity {right arrow over (v)}0 can be taken to be zero if the motion is being measured for short periods of time in relation to the duration of Earth's rotation. The second integration gives the position as follows:
At the initial position, the IMU's local-to-global rotation's matrix has an initial value {right arrow over (Θ)}(0)≡{right arrow over (θ)}0. This value can be derived by knowing the local and global values of both the magnetic north vector and the acceleration of gravity. Those two vectors are usually non-parallel. This is the requirement for the {right arrow over (θ)}0({right arrow over (g)}′, {right arrow over (n)}′, {right arrow over (g)}, {right arrow over (n)}) to be unique. The knowledge of either of those vectors in isolation gives a family of non-unique solutions {right arrow over (θ)}0({right arrow over (g)}′, {right arrow over (g)}) or {right arrow over (θ)}0(({right arrow over (n)}′, {right arrow over (n)}) that are unconstrained in one component of rotation. The {right arrow over (θ)}0({right arrow over (g)}′, {right arrow over (n)}′, {right arrow over (g)}, {right arrow over (n)}) has many implementations, with the common one being the Kabsch algorithm. As such, using the calculation procedure described above, the at least one data processing device of the system 100 may determine the orientation {right arrow over (θ)}(t) and position {right arrow over (R)}(t) of one or more body portions of the user. For example, the orientation of the head of the user may be determined by computing the orientation {right arrow over (θ)}(t) and position {right arrow over (R)}(t) of two points on the head of the user (i.e., at the respective locations of two inertial measurement units (IMUs) disposed on the head of the user).
In one or more alternative embodiments, rather than using an inertial measurement unit (IMU) that includes an accelerometer, a gyroscope, and a magnetometer, a single accelerometer may be used to simply measure the displacement of the head of the user (e.g., by using equation (8) described above). As explained above, the acceleration output from the accelerometer may be integrated twice in order to obtain the positional displacement of the head of the user.
In yet a further illustrative embodiment, with reference to
With reference again to
In this further illustrative embodiment, the at least one data processing device 20 is further programmed to determine the one or more output forces and/or output moments for the user from the plurality of output signals from the plurality of force measurement assemblies; and determine performance metrics for the user based upon the one or more output forces and/or output moments. In this further illustrative embodiment, the one or more performance metrics determined by the at least one data processing device 20 for the user are selected from a group consisting of: (i) a postural stability parameter, (ii) a center of pressure for the user, (iii) a center of gravity for the user, and (iv) combinations thereof. In the illustrative embodiment, the force data from the instrumented floor 15 can be used to characterize the kinetic (ground reaction forces) of the subject standing on the force tile(s) 17. One also can derive various metrics about the user/subject based on the force data. For example, one can assess the stability of the user, given certain visual cues (e.g. the user standing at the edge of a virtual cliff, on a virtual tight rope, etc.). As another example, the center of pressure (CoP) and thus the center of gravity (CoG) can be derived and used for the detection of modalities of user reactions to certain “virtual” events, such as falling, virtual object avoidance, etc.
In this further illustrative embodiment, the head position sensing device of the head-mounted visual display device 30 (e.g., the inertial measurement unit (IMU) disposed in the head-mounted visual display device 30) may be configured to detect at least one of head position, head movement, head orientation, head velocity, and head acceleration of the user, and the at least one data processing device may be further programmed to: (i) determine the one or more output forces and/or output moments for the user from the one or more output signals from the one or more force measurement assemblies of the instrumented floor 15; (ii) determine the at least one of the head position, the head movement, the head orientation, the head velocity, and the head acceleration of the user from the head position and/or movement sensing device; and (iii) assess balance and/or body stability for the user based upon both the one or more output forces and/or output moments from the one or more force measurement assemblies and the at least one of the head position, the head movement, the head orientation, the head velocity, and the head acceleration of the user from the head position and/or movement sensing device. In this further illustrative embodiment, the one or more output forces for the user may comprise the ground reaction force (GRF) for the user, which is the force the ground exerts on any object in contact with it, such as the human body during movements like walking, running, or jumping. The important aspects of the ground reaction force (GRF) include: (i) magnitude and direction, and (ii) GRF components. Considering magnitude and direction, the GRF varies in intensity depending on the activity, being higher in running than walking, for instance. The GRF direction is typically opposite to the applied force. Considering the GRF components, the GRF may be analyzed in three dimensions: (i) vertical (supporting body weight), (ii) anterior-posterior (forward and backward motion), and (iii) medial-lateral (side-to-side stability). In gait analysis, ground reaction force (GRF) plays a pivotal role. The GRF is used to study walking and running patterns, helping in rehabilitation, enhancing sports performance, and diagnosing movement disorders. Integrating GRF measurements from the instrumented floor 15 with data from the head position sensing device of the head-mounted visual display device 30 (e.g., the inertial measurement unit (IMU) disposed in the head-mounted visual display device 30) profoundly enriches gait analysis because this combination yields a comprehensive view of an individual's gait dynamics. The significance of this integration is twofold. First, the integration results in comprehensive gait dynamics where the force plates of the instrumented floor 15 provide precise GRF data including magnitude, direction, and application point during gait, essential for understanding body-ground interaction. The HMD-based head position sensing device complements the force plate data of the instrumented floor 15 by capturing head movements, orientation, and acceleration, crucial for balance and stability analysis. Secondly, the integration results in a comprehensive balance and stability analysis where the HMD-based head position sensing device monitors head movements, pivotal for balance, while the force plates of the instrumented floor 15 provide precise GRF data. Advantageously, analyzing head stability alongside GRF data allows for a thorough assessment of an individual's balance and body stabilization during motion, vital for pinpointing gait irregularities.
In this further illustrative embodiment, a plurality of different users and/or objects may be disposed on the instrumented floor 15, and the at least one data processing device 20 is further programmed to determine positions and/or movements for the plurality of different users and/or objects disposed on the instrumented floor 15 using the one or more output signals. In the illustrative embodiment, the instrumented floor 15 can also measure and keep track of multiple users/subjects and objects resting or moving on the instrumented floor 15. Due to the high sensitivity of the instrumented floor 15, minute perturbations can be measured for any object or user/subject on the instrument floor 15. Also, in the illustrative embodiment, the force data from the instrumented floor 15 can enhance the virtual video projected to the user. For example, the tapping of one's foot versus the stomping of one's foot on the instrumented floor 15 can be used to project virtual indentations on the virtual project ground/surface.
In this further illustrative embodiment, the at least one data processing device 20 is further programmed to generate at least one of visual feedback, auditory feedback, and/or tactile feedback for the user based upon the one or more output signals from the instrumented floor 15. The at least one data processing device 20 may be further programmed to generate the visual feedback for the user based upon the one or more output signals from the instrumented floor 15 by displaying one or more virtual objects to the user using the head-mounted visual display device 30. In this further illustrative embodiment, the instrumented floor 15 comprises one or more vibratory devices for providing the tactile feedback for the user, and the at least one data processing device 20 may be further programmed to generate the tactile feedback for the user based upon the one or more output signals from the instrumented floor 15 by generating one or more tactile sensations for the user using the one or more vibratory devices. In the illustrative embodiment, the force data for the instrumented floor 15 also can be used to generate acoustic feedback for the user.
The head-mounted display system 100′ illustrated in
For sports training, the head-mounted display system 100′ can be used for performance analysis (e.g., athletes can receive feedback on their footwork, balance, and force distribution as they train). Also, for sports training, the head-mounted display system 100′ can be used for skill development (e.g., in-game situations can be simulated and the head-mounted display system 100′ can provide athletes with real-time feedback on their biomechanics).
For virtual reality gaming, the head-mounted display system 100′ can be used for creating immersive experiences. As a result of the instrumented floor 15 of the head-mounted display system 100′, games can be designed where players must use their whole body, and the feedback from the floor 15 can be used to inform game mechanics, like jumping, running, or dodging. Also, for virtual reality gaming, the head-mounted display system 100′ can be used for creating real-world effects (e.g., if a player is “hit” in a game, the floor could simulate the force, adding a tactile dimension to the gameplay).
For balance and posture training, the head-mounted display system 100′ can be used for creating interactive scenarios. For example, users could be placed in virtual environments that challenge their balance, like a tightrope or shifting platform. The instrumented floor 15 provides data on how they adjust and maintain balance. Also, for balance and posture training, the head-mounted display system 100′ can be used for biofeedback. For example, real-time data on weight distribution can be visualized in the head-mounted visual display device 30, helping users understand and correct their posture.
For research, the head-mounted display system 100′ can be used for studying human biomechanics. For example, the head-mounted display system 100′ can be used to study the intricacies of human movement in various scenarios, collecting data on how different visual stimuli affect movement and balance. Also, for research, the head-mounted display system 100′ can be used for ergonomics assessments. For example, the head-mounted display system 100′ can be used to test how people move and distribute their weight in various work settings or when interacting with different objects/tools.
For safety and evacuation training, the head-mounted display system 100′ can be used for simulating scenarios. For example, the head-mounted display system 100′ can be used to create virtual emergencies, like a building fire or ship sinking, and train individuals on how to evacuate safely. The instrumented floor 15 can provide data on the efficiency of movements, possible trip hazards, etc.
For entertainment and interactive exhibits, the head-mounted display system 100′ can be used for creating immersive experiences. For example, the head-mounted display system 100′ can be used by museums or theme parks to create interactive exhibits where visitors navigate virtual scenarios, with the floor capturing their reactions and movements.
For architecture and design, the head-mounted display system 100′ can be used for simulating environments. For example, before building, architects and designers can use the head-mounted display system 100′ to simulate spaces, observing how individuals navigate them. The data from the instrumented floor 15 can inform design adjustments to optimize flow and safety.
In this further illustrative embodiment, the at least one data processing device 20 is further programmed to determine locations for one or more objects or persons based upon the one or more output signals from the instrumented floor 15; and depict, using the head-mounted visual display device 30, the one or more objects or persons in a virtual reality environment or augmented reality environment based upon the locations determined from the one or more output signals from the instrumented floor 15. In the illustrative embodiment, if there is more than one subject or object on the instrumented floor 15, more accurate locations can be estimated and better projected in the simulated virtual reality (VR) environment. Similarly, better object localization can be done in augmented reality projects based on the CoP data from the instrumented floor 15.
In this further illustrative embodiment, the at least one data processing device 20 is further programmed to generate one or more virtual terrains, virtual obstacles, and/or virtual people on the output screen of the head-mounted visual display device 30 so that the user is able to interact with the one or more virtual terrains, virtual obstacles, and/or virtual people in a virtual reality environment; and determine a performance of the user in the virtual reality environment with the one or more virtual terrains, virtual obstacles, and/or virtual people based upon the one or more output signals from the instrumented floor 15.
In still a further illustrative embodiment, with reference again to
The pass-through mode of the head-mounted visual display device 30 has various benefits, such as improving the spatial awareness of the user and enhancing interactive gaming. For spatial awareness, the pass-through mode of the head-mounted visual display device 30 allows users to quickly check their surroundings without taking off the headset. The pass-through mode of the head-mounted visual display device 30 is especially useful in environments where users are walking around because the mode provides a safety “pass-through” to real-world visuals. For example, in the illustrative embodiment, if a patient is feeling dizzy or nauseous and needs to escape the virtual reality scene, they can do so by pressing “X” or “A” on the input device 34, 36 (e.g., hand controllers 34, 36). In the illustrative embodiment, the pass-through mode is a black-and-white or color live view of the surrounding environment. Also, in the illustrative embodiment, when the pass-through toggle is activated on one or more graphical user interfaces of the head-mounted visual display device 30, the head-mounted visual display device 30 enables the pass-through function which disables the virtual world and shows a user his or her surroundings so that the user can orient himself or herself. When the pass-through toggle clicked again, the head-mounted visual display device 30 will turn off the pass-through function and shows the virtual world.
In this further embodiment, the at least one data processing device 20 is further programmed to receive a defined area within the real-world environment from the user that establishes one or more boundaries of the real-world environment; and display the one or more boundaries of the real-world environment within a virtual reality environment or an augmented reality environment so as to prevent the user from colliding with one or more objects in the real-world environment. In the illustrative embodiment, when setting up a virtual reality (VR) system, users can define a play area (i.e., a boundary setting). The pass-through mode allows the user to see the real world set to these boundaries, ensuring he or she does not collide into real-world objects.
In this further embodiment, the at least one data processing device 20 is further programmed to manipulate the augmented digital content such that the augmented digital content interacts with one or more actual objects in the real-world environment. In an interactive gaming application, game elements may be overlaid onto the real-world. For example, the user is playing a game where his or her living room becomes the game board, and virtual characters interact with the actual furniture in the living room.
In this further embodiment, the at least one data processing device 20 is further programmed to overlay instructions or data over the one or more images of the real-world environment so that the user is able to receive guidance as to how to interact with the real-world environment. In an education and training application, augmented reality (AR) may be used to overlay instructions or data onto real-world objects. For example, a mechanic wearing the head-mounted visual display device 30 could see annotated instructions on how to repair an engine.
In this further embodiment, the at least one data processing device 20 is further programmed to capture one or more hand movements of the user in the real-world environment using the at least one camera 31; and enable the user to interact with one or more virtual elements using the one or more hand movements of the user captured from the real-world environment. That is, the cameras 31 of the head-mounted visual display device 30 can be used for hand tracking, allowing users to interact with virtual elements using natural hand movements.
In this further embodiment, the at least one data processing device 20 is further programmed to perform object recognition on the one or more images of the real-world environment captured by the at least one camera 31 so as to identify one or more real-world objects; and generate the augmented digital content based upon the one or more real-world objects identified in the real-world environment so as to provide a context-aware augmented reality experience. Object recognition can enable context-aware augmented reality (AR) experiences. For example, pointing the cameras 31 of the head-mounted visual display device 30 at a book might bring up reviews or a summary.
In yet a further illustrative embodiment, a body pose of a user may be estimated while the user is wearing the head-mounted display system. In this illustrative embodiment, the pose of the user may modify and/or augment the scene and/or environment displayed inside the head-mounted display (HMD). In this illustrative embodiment, the body pose of the user may be estimated by using one or more of the following means: (i) marker-based motion capture using cameras, (ii) markerless motion capture using cameras, (iii) inertial measurement units (IMUs) attached to the user, (iv) egocentric cameras attached to the user, (v) WiFi network based pose estimation, (vi) wall-mounted LIDAR based pose estimation, (vii) instrumented floor-based pose sensors, and (viii) combinations thereof.
In this further illustrative embodiment, the full-body pose of the user wearing the HMD may be estimated using one or more of the pose estimation techniques mentioned above. First, marker-based motion capture using cameras will be described. Marker-based motion capture involves attaching reflective or light emitting diode (LED) markers to key points on the user's body, such as joints and limbs. These markers may be tracked by multiple cameras positioned around the environment. The cameras capture the light reflected or emitted by the markers, and software algorithms triangulate the positions of these markers in three-dimensional space. This data is then used to construct a detailed and accurate model of the user's movements and pose. The setup for this type of pose estimation technique involves a controlled environment with multiple cameras and a calibration process to ensure precision. The markers need to be securely attached to the user and consistently visible to the cameras.
Second, markerless motion capture using cameras will be described. Markerless motion capture uses advanced computer vision techniques to track the subject's body without the need for physical markers. This method relies on depth-sensing cameras or standard RGB cameras combined with machine learning algorithms to identify and follow the natural features of the body, such as the contours of limbs and the shape of the torso. The machine learning algorithms analyze the captured frames to detect and estimate the pose of the user in real-time. Markerless motion capture is particularly useful in settings where attaching markers is impractical, such as live performances, public demonstrations, and at-home VR experiences. Markerless motion capture also is useful for real-time applications like gaming and interactive installations. The setup for this type of pose estimation technique typically involves a fewer number of cameras, as compared to marker-based systems, and can often be used in more varied and uncontrolled environments. The system needs to be trained to recognize a wide range of body types and movements. For example, in the illustrative embodiment, the markerless motion capture system may utilize the 3D pose estimation system described in U.S. Pat. No. 10,853,970, the entire disclosure of which is incorporated herein by reference. Also, in the illustrative embodiment, the markerless motion capture system may utilize a mesh estimation module that takes 2D keypoints as inputs and predicts a human body mesh through parametric mesh model/body model parameters, such as Skinned Multi-Person Linear (SMPL) Model parameters. The human body mesh is useful in determining the actual volume/space occupied by the user in the real world, and thus its impact in the virtual world.
Third, inertial measurement units (IMUs) attached to the user will be described. Inertial measurement units (IMUs) are comprised of accelerometers, gyroscopes, and sometimes magnetometers as well. Accelerometers measure linear acceleration along multiple axes, gyroscopes measure rotational velocity, and magnetometers detect the orientation relative to the Earth's magnetic field. By integrating the data from these sensors, an IMU can provide continuous, real-time information about the position and orientation of the body parts to which the IMUs are attached. Typically, IMUs can be integrated into wearable devices like suits, gloves, or footwear, allowing for a mobile and flexible setup. Periodic calibration is typically involved to maintain accuracy and reduce sensor drift.
Fourth, egocentric camera(s) attached to the user will be described. Egocentric cameras are typically attached to the user, capturing video from his or her point of view. Recent research has advanced this technique to enable full body pose estimation by using multiple egocentric cameras placed strategically on the body, such as on the head, chest, and limbs. These cameras capture the surroundings and the subject's body parts from various angles. Advanced algorithms then process the video streams to reconstruct the full body pose by analyzing the relative positions and movements of the body parts visible in the camera feeds. Egocentric cameras are particularly useful for applications where external cameras are impractical, such as outdoor activities, immersive gaming, and virtual training scenarios. This method enhances the user's immersion by providing a highly personal perspective. The setup of egocentric cameras involves careful placement and synchronization of multiple cameras. The processing algorithms need to be robust to handle occlusions and varying lighting conditions. This method can be combined with other sensors to improve accuracy.
Fifth, WiFi network-based pose estimation will be described. WiFi-based pose estimation leverages the properties of WiFi signals to determine the position and movement of a user. By analyzing the distortions and reflections of WiFi signals caused by the human body, machine learning algorithms can infer the user's pose. Multiple WiFi access points and receivers are used to triangulate the user's position and movements within the coverage area. WiFi-based pose estimation is suitable for indoor environments where the WiFi infrastructure is already in place, such as smart homes, offices, and public spaces. This technique can be used for non-intrusive monitoring and interaction in AR/VR applications. The setup for WiFi-based pose estimation involves a network of WiFi devices positioned strategically around the environment. The system needs to be trained to distinguish between different types of movements and poses.
Sixth, wall-mounted LIDAR-based pose estimation will be described. LIDAR (Light Detection and Ranging) technology uses laser pulses to measure distances to objects in the environment. Wall-mounted LIDAR sensors can create a detailed 3-D map of the surroundings, including a person's body. By continuously scanning the area, LIDAR can track the position and movement of the user with high accuracy. This data is processed to determine the pose of the user in real-time. LIDAR is ideal for environments where precise spatial mapping is required, such as in industrial applications, virtual reality arenas, and motion capture studios. LIDAR is also used in autonomous vehicles for navigation. The setup for LIDAR-based pose estimation involves the installation of LIDAR sensors around the environment. The sensors need to be calibrated and synchronized to provide a coherent 3D map. LIDAR works well in various lighting conditions, but can be expensive. In the illustrative embodiment, similar to the markerless motion capture system described above, the WiFi network-based pose estimation system and/or wall-mounted LIDAR-based pose estimation system may utilize a mesh estimation module that takes 2D keypoints as inputs and predicts a human body mesh through parametric mesh model/body model parameters, such as Skinned Multi-Person Linear (SMPL) Model parameters. The human body mesh is useful in determining the actual volume/space occupied by the user in the real world, and thus its impact in the virtual world.
Seventh, instrumented floor-based pose sensors will be described. Instrumented floors are embedded with various types of sensors, such as pressure sensors, capacitive sensors, or load cells, to detect foot placement and body movements. These sensors capture data about the distribution of weight, movement patterns, and pressure points, which can be used to infer the pose of the entire body. The data is analyzed to track the position and motion of the person as they move across the floor. An instrumented floor setup may comprise a floor equipped with a dense array of sensors and connected to a processing unit that can handle the data in real-time. This setup is typically limited to specific areas and requires significant installation effort.
Each of the full body pose estimation techniques described above offers unique advantages and challenges, making them suitable for different applications and environments in VR/AR. By leveraging the strengths of multiple methods, one can create more accurate, immersive, and interactive virtual experiences for users.
In the illustrative embodiment, knowing the full-body pose of the user of the head-mounted display (HMD) 30 provides several significant advantages that enhance the control and realism of the scenes inside the HMD 30. Illustrative examples of how full-body pose estimation impacts and improves the VR/AR experience will be explained hereinafter.
First, knowing the full-body pose of the user of the head-mounted display (HMD) 30 enhances interaction and immersion with respect to: (i) natural movement replication, and (ii) improved control mechanisms. With regard to natural movement replication, the full body pose tracking allows the virtual avatar to replicate the user's movements accurately. This creates a more natural and immersive experience, as the user sees their virtual self moving as he or she does in real life. For example, when a user ducks, jumps, or leans, these actions are mirrored in the virtual world, providing a more engaging and realistic interaction. With regard to improved control mechanisms, accurate body tracking enables more intuitive and diverse control methods. Users can perform complex gestures and movements that are recognized and interpreted by the system, enhancing the range of possible interactions. For example, in a VR game, users can use hand gestures to cast spells, perform martial arts moves, or manipulate virtual objects, leading to more dynamic and interactive gameplay.
Second, knowing the full-body pose of the user of the head-mounted display (HMD) 30 enhances contextual environmental responses with respect to: (i) dynamic scene adjustment, and (ii) obstacle avoidance and safety. With regard to dynamic scene adjustment, the virtual environment can dynamically adjust based on the user's body position and movements. This ensures that the environment feels responsive and alive. For example, as the user moves closer to virtual objects, those objects can react appropriately, such as opening doors, triggering events, or changing perspectives to provide a better view. With regard to obstacle avoidance and safety, full body tracking helps the system to recognize potential physical obstacles in the real world and provide warnings or adjustments to avoid collisions. For example, if the user is about to walk into a wall, the system can alert them or create a virtual barrier to prevent injury.
Third, knowing the full-body pose of the user of the head-mounted display (HMD) 30 results in enhanced gameplay and application scenarios with respect to: (i) realistic interactions with virtual objects, and (ii) multi-user and collaborative experiences. With regard to realistic interactions with virtual objects, full body tracking allows users to interact with virtual objects in a more lifelike manner. This is crucial for applications that require precise and realistic manipulation of objects. For example, in a VR surgery simulation, accurate hand and finger tracking enable precise control of surgical instruments, providing a realistic training environment. With regard to multi-user and collaborative experiences, full body pose estimation is needed for multi-user VR/AR experiences, where users can see and interact with each other's avatars in the virtual space. For example, in a collaborative VR workspace, users can point to objects, gesture during discussions, and perform collaborative tasks with natural body movements, enhancing communication and teamwork.
Fourth, knowing the full-body pose of the user of the head-mounted display (HMD) 30 results in improved physical training and rehabilitation with respect to: (i) fitness and sports training, and (ii) rehabilitation and therapy. With regard to fitness and sports training, VR/AR systems can use full body tracking to provide real-time feedback on physical activities, helping users improve their form and technique. For example, a virtual personal trainer can guide users through exercises, providing corrections and encouragement based on their movements. With regard to rehabilitation and therapy, full body tracking in VR can be used for physical therapy, where patients perform exercises and movements in a controlled virtual environment. Therapists can monitor progress and adjust therapy sessions remotely. For example, patients recovering from injuries can perform rehabilitation exercises in VR, with the system tracking their movements and providing feedback to ensure they are performing the exercises correctly.
Fifth, knowing the full-body pose of the user of the head-mounted display (HMD) 30 results in enhanced realism in storytelling and simulation for: (i) immersive storytelling, and (ii) training simulations. With regard to immersive storytelling, full body tracking allows users to become active participants in virtual narratives, where their movements influence the storyline and outcomes. For example, in an interactive VR movie, the user's actions can determine the plot's direction, making the experience personalized and immersive. With regard to training simulations, accurate body pose tracking is critical for training simulations, such as military, aviation, or emergency response, where realistic body movements and interactions are necessary for effective training. For example, pilots in training can use VR to simulate emergency procedures, with full body tracking ensuring that their actions are accurately represented and evaluated.
Overall, knowing the full body pose of the wearer of an HMD significantly enhances the realism, interactivity, and safety of VR/AR experiences. Also, knowing the full body pose of the wearer allows for more natural and intuitive user interactions, dynamic and responsive virtual environments, improved physical training and rehabilitation applications, and more engaging storytelling and simulation scenarios.
In still a further illustrative embodiment, with reference to
In the illustrative embodiment, with reference to
In the illustrative embodiment, various integration strategies may be employed to combine the egocentric camera-based pose estimation with the instrumented flooring system. First, data fusion may be employed to combine the camera output data of the egocentric camera-based pose estimation with the sensor output data of the instrumented flooring. The data fusion may include sensor fusion algorithms to combine the data from the egocentric cameras and floor sensors. These algorithms integrate visual data from cameras with pressure and movement data from the floor to create a coherent and accurate full-body pose estimation. The data fusion may include the implementation of real-time processing to ensure that the system responds promptly to user movements, maintaining immersion and interactivity.
Secondly, additional integration strategies for the egocentric camera-based pose estimation with the instrumented flooring system may include calibration and synchronization. In the illustrative embodiment, an initial calibration may be performed to align the data from the cameras and floor sensors, ensuring they operate in the same coordinate system. Also, in the illustrative embodiment, continuous synchronization may be performed that uses synchronization protocols to continuously align and update the data from both sources, maintaining accuracy over time. For example, in the illustrative embodiment, the at least one data processing device 20 is configured to execute synchronization protocols for aligning the video data from the plurality of egocentric cameras and the sensor output data from the instrumented flooring system in a common coordinate system, thereby ensuring coherent and accurate pose estimation.
Thirdly, additional integration strategies for the egocentric camera-based pose estimation with the instrumented flooring system may include error correction and drift compensation. For example, in the illustrative embodiment, the sensor fusion algorithms executed by the at least one data processing device 20 include error correction mechanisms to compensate for potential drift in the video data from the one or more egocentric cameras 21 using stable reference points from the instrumented flooring system 15. Also, in the illustrative embodiment, the sensor fusion algorithms executed by the at least one data processing device 20 include feedback loops to continuously refine the body pose estimation for the user by compensating for drift in the video data from the one or more egocentric cameras using reference points from the instrumented flooring system. Using this integration strategy, the complementary strengths of each system may be used to compensate for the weaknesses of the other system. For example, the instrumented floor sensors can provide stable reference points to correct any drift in the camera-based tracking. Also, feedback loops may be implemented where the data from the floor sensors is used to adjust and refine the pose estimates from the egocentric cameras. Combining egocentric camera-based pose estimation with instrumented flooring technologies presents an innovative approach to full-body tracking in VR/AR. By leveraging the complementary strengths of both systems, this hybrid solution can provide highly accurate, immersive, and interactive experiences across a wide range of applications.
In the illustrative embodiment, the plurality of egocentric cameras may be attached to different parts of the body of the user to capture a range of perspectives and movements of the user. In the illustrative embodiment, the different parts of the body of the user may be selected from a group consisting of: (i) a head of the user, (ii) a chest of the user, (iii) one or more arms of the user, (iv) one or more legs of the user, and (v) combinations thereof. For example, with reference again to
In the illustrative embodiment, the at least one data processing device 20 is further configured to perform real-time processing of the video data from the plurality of egocentric cameras 21 and the sensor output data from the instrumented flooring system 15 so as to provide quasi-instantaneous updates to the virtual reality or augmented reality environment of the HMD 30 based on body movements of the user. Also, in the illustrative embodiment, the at least one data processing device 20 is further configured to dynamically update the virtual reality or augmented reality environment of the HMD 30 based on the body pose estimation for the user, thereby enhancing user interaction and immersion in the virtual reality or augmented reality environment.
In the illustrative embodiment, the at least one data processing device 20 is further configured to provide real-time feedback and interaction capabilities in the virtual reality or augmented reality environment of the HMD 30 based on body movements of the user. Also, in the illustrative embodiment, the at least one data processing device 20 is further configured to dynamically adjust training scenarios for the user in response to the body movements of the user, thereby enhancing the effectiveness of a training session.
It is readily apparent that the illustrative head-mounted display system 100, 100′, 100″ described above offers numerous advantages and benefits. For example, the head-mounted display system 100, 100′, 100″ is very beneficial for assessing balance and mobility. As another example, the head-mounted display system 100, 100′, 100″ is useful for enhancing the visual motor performance of individuals. As yet another example, the head-mounted display system 100, 100′, 100″ may be used as part of the rehabilitation regime for an orthopedic and/or neurological injury.
While reference is made throughout this disclosure to, for example, “an illustrative embodiment”, “one embodiment”, or a “further embodiment”, it is to be understood that some or all aspects of these various embodiments may be combined with one another as part of an overall embodiment of the invention. That is, any of the features or attributes of the aforedescribed embodiments may be used in combination with any of the other features and attributes of the aforedescribed embodiments as desired.
Although the invention has been shown and described with respect to a certain embodiment or embodiments, it is apparent that this invention can be embodied in many different forms and that many other modifications and variations are possible without departing from the spirit and scope of this invention.
Moreover, while exemplary embodiments have been described herein, one of ordinary skill in the art will readily appreciate that the exemplary embodiments set forth above are merely illustrative in nature and should not be construed as to limit the claims in any manner. Rather, the scope of the invention is defined only by the appended claims and their equivalents, and not, by the preceding description.
This is a continuation-in-part of U.S. patent application Ser. No. 18/389,191, entitled “Head-Mounted Display System”, filed on Nov. 13, 2023; which is a continuation-in-part of U.S. patent application Ser. No. 17/965,745, entitled “Head-Mounted Display System”, filed on Oct. 13, 2022, now U.S. Pat. No. 11,816,258, the disclosure of each of which is hereby incorporated by reference as if set forth its entirety herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6038488 | Barnes et al. | Mar 2000 | A |
| 6113237 | Ober et al. | Sep 2000 | A |
| 6152564 | Ober et al. | Nov 2000 | A |
| 6295878 | Berme | Oct 2001 | B1 |
| 6354155 | Berme | Mar 2002 | B1 |
| 6389883 | Berme et al. | May 2002 | B1 |
| 6936016 | Berme et al. | Aug 2005 | B2 |
| 8181541 | Berme | May 2012 | B2 |
| 8315822 | Berme et al. | Nov 2012 | B2 |
| 8315823 | Berme et al. | Nov 2012 | B2 |
| D689388 | Berme | Sep 2013 | S |
| D689389 | Berme | Sep 2013 | S |
| 8543540 | Wilson et al. | Sep 2013 | B1 |
| 8544347 | Berme | Oct 2013 | B1 |
| 8643669 | Wilson et al. | Feb 2014 | B1 |
| 8700569 | Wilson et al. | Apr 2014 | B1 |
| 8704855 | Berme et al. | Apr 2014 | B1 |
| 8764532 | Berme | Jul 2014 | B1 |
| 8847989 | Berme et al. | Sep 2014 | B1 |
| D715669 | Berme | Oct 2014 | S |
| 8902249 | Wilson et al. | Dec 2014 | B1 |
| 8915149 | Berme | Dec 2014 | B1 |
| 9032817 | Berme et al. | May 2015 | B2 |
| 9043278 | Wilson et al. | May 2015 | B1 |
| 9066667 | Berme et al. | Jun 2015 | B1 |
| 9081436 | Berme et al. | Jul 2015 | B1 |
| 9168420 | Berme et al. | Oct 2015 | B1 |
| 9173596 | Berme et al. | Nov 2015 | B1 |
| 9200897 | Wilson et al. | Dec 2015 | B1 |
| 9277857 | Berme et al. | Mar 2016 | B1 |
| D755067 | Berme et al. | May 2016 | S |
| 9404823 | Berme et al. | Aug 2016 | B1 |
| 9414784 | Berme et al. | Aug 2016 | B1 |
| 9468370 | Shearer | Oct 2016 | B1 |
| 9517008 | Berme et al. | Dec 2016 | B1 |
| 9526443 | Berme et al. | Dec 2016 | B1 |
| 9526451 | Berme | Dec 2016 | B1 |
| 9558399 | Jeka et al. | Jan 2017 | B1 |
| 9568382 | Berme et al. | Feb 2017 | B1 |
| 9622686 | Berme et al. | Apr 2017 | B1 |
| 9763604 | Berme et al. | Sep 2017 | B1 |
| 9770203 | Berme et al. | Sep 2017 | B1 |
| 9778119 | Berme et al. | Oct 2017 | B2 |
| 9814430 | Berme et al. | Nov 2017 | B1 |
| 9829311 | Wilson et al. | Nov 2017 | B1 |
| 9854997 | Berme et al. | Jan 2018 | B1 |
| 9916011 | Berme et al. | Mar 2018 | B1 |
| 9927312 | Berme et al. | Mar 2018 | B1 |
| 10010248 | Shearer | Jul 2018 | B1 |
| 10010286 | Berme et al. | Jul 2018 | B1 |
| 10085676 | Berme et al. | Oct 2018 | B1 |
| 10117602 | Berme et al. | Nov 2018 | B1 |
| 10126186 | Berme et al. | Nov 2018 | B2 |
| 10216262 | Berme et al. | Feb 2019 | B1 |
| 10231662 | Berme et al. | Mar 2019 | B1 |
| 10264964 | Berme et al. | Apr 2019 | B1 |
| 10331324 | Wilson et al. | Jun 2019 | B1 |
| 10342473 | Berme et al. | Jul 2019 | B1 |
| 10390736 | Berme et al. | Aug 2019 | B1 |
| 10413230 | Berme et al. | Sep 2019 | B1 |
| 10463250 | Berme et al. | Nov 2019 | B1 |
| 10527508 | Berme et al. | Jan 2020 | B2 |
| 10555688 | Berme et al. | Feb 2020 | B1 |
| 10646153 | Berme et al. | May 2020 | B1 |
| 10722114 | Berme et al. | Jul 2020 | B1 |
| 10736545 | Berme et al. | Aug 2020 | B1 |
| 10765936 | Berme et al. | Sep 2020 | B2 |
| 10803990 | Wilson et al. | Oct 2020 | B1 |
| 10853970 | Akbas et al. | Dec 2020 | B1 |
| 10856796 | Berme et al. | Dec 2020 | B1 |
| 10860843 | Berme et al. | Dec 2020 | B1 |
| 10945599 | Berme et al. | Mar 2021 | B1 |
| 10966606 | Berme | Apr 2021 | B1 |
| 11033453 | Berme et al. | Jun 2021 | B1 |
| 11052288 | Berme et al. | Jul 2021 | B1 |
| 11054325 | Berme et al. | Jul 2021 | B2 |
| 11074711 | Akbas et al. | Jul 2021 | B1 |
| 11097154 | Berme et al. | Aug 2021 | B1 |
| 11158422 | Wilson et al. | Oct 2021 | B1 |
| 11182924 | Akbas et al. | Nov 2021 | B1 |
| 11262231 | Berme et al. | Mar 2022 | B1 |
| 11262258 | Berme et al. | Mar 2022 | B2 |
| 11301045 | Berme et al. | Apr 2022 | B1 |
| 11311209 | Berme et al. | Apr 2022 | B1 |
| 11321868 | Akbas et al. | May 2022 | B1 |
| 11337606 | Berme et al. | May 2022 | B1 |
| 11348279 | Akbas et al. | May 2022 | B1 |
| 11458362 | Berme et al. | Oct 2022 | B1 |
| 11521373 | Akbas et al. | Dec 2022 | B1 |
| 11540744 | Berme | Jan 2023 | B1 |
| 11604106 | Berme et al. | Mar 2023 | B2 |
| 11631193 | Akbas et al. | Apr 2023 | B1 |
| 11688139 | Karagoz et al. | Jun 2023 | B1 |
| 11705244 | Berme | Jul 2023 | B1 |
| 11712162 | Berme et al. | Aug 2023 | B1 |
| 11790536 | Berme et al. | Oct 2023 | B1 |
| 11798182 | Karagoz et al. | Oct 2023 | B1 |
| 11816258 | Berme et al. | Nov 2023 | B1 |
| 11826601 | Berme | Nov 2023 | B1 |
| 11850078 | Berme | Dec 2023 | B1 |
| 11857331 | Berme et al. | Jan 2024 | B1 |
| 11865407 | Berme et al. | Jan 2024 | B1 |
| 11911147 | Berme et al. | Feb 2024 | B1 |
| 11992746 | Berme | May 2024 | B1 |
| 12013542 | Berme et al. | Jun 2024 | B1 |
| 20030216656 | Berme et al. | Nov 2003 | A1 |
| 20080228110 | Berme | Sep 2008 | A1 |
| 20110277562 | Berme | Nov 2011 | A1 |
| 20120266648 | Berme et al. | Oct 2012 | A1 |
| 20120271565 | Berme et al. | Oct 2012 | A1 |
| 20150096387 | Berme et al. | Apr 2015 | A1 |
| 20160245711 | Berme et al. | Aug 2016 | A1 |
| 20160334288 | Berme et al. | Nov 2016 | A1 |
| 20180024015 | Berme et al. | Jan 2018 | A1 |
| 20190078951 | Berme et al. | Mar 2019 | A1 |
| 20200139229 | Berme et al. | May 2020 | A1 |
| 20200408625 | Berme et al. | Dec 2020 | A1 |
| 20210333163 | Berme et al. | Oct 2021 | A1 |
| 20220178775 | Berme et al. | Jun 2022 | A1 |
| Entry |
|---|
| Notice of Allowance in U.S. Appl. No. 17/965,745, mailed on Jul. 10, 2023. |
| Notice of Allowance in U.S. Appl. No. 18/389,191, mailed on Feb. 14, 2024. |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18389191 | Nov 2023 | US |
| Child | 18745864 | US | |
| Parent | 17965745 | Oct 2022 | US |
| Child | 18389191 | US |
