Patent Application
20190265806
The present invention relates generally to Virtual reality and in particular to a reactive animation enhanced Virtual Reality
This section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present invention that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
In recent years, Virtual Reality (VR) has become the subject of increased attention. This is because VR can be used practically in every field to perform various functions including testing, entertaining and teaching. For example, engineers and architects can use VR in modeling and testing of new designs. Doctors can use VR to practice and perfect difficult operations ahead of time and military experts can develop strategies by simulating battlefield operations. VR is also used extensively in the gaming and entertainment industries to provide interactive experiences and enhance audience enjoyment. VR enables the creation of a simulated environment that feels real and can accurately duplicate real life or fictional situations. Furthermore, VR covers remote communication environments which provide virtual presence of users with the concepts of tele-presence and tele-existence or virtual artifact (VA). VR also provides a better sense of design and engineering because in many instances it allows conversion of two dimensional images into visually accessible three dimensional virtual structures. Most prior art VR devices, are cumbersome and expensive. The changes presented by mobile device technology has offered virtual reality applications with more accessible alternatives. Consequently, in recent years, however, it has become desirous to provide VR functions through incorporating everyday personal mobile devices such as cellular phones so as to improve accessibility and availability.
Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. For a better understanding of the invention with advantages and features, refer to the description and to the drawings.
An apparatus and method is provided for determining user input. In one embodiment, the method comprises receiving continuous total magnetic field readings via a processor, wherein some of the magnetic field measurements are due to a magnet connected to a user interface and input to the user interface causes positional changes to said magnet. The processor removes any ambient magnetic field components from the magnetic field readings and analyzes changes in the magnetic field readings to determine when the changes are due to positional changes of a magnet in proximity to the user interface. The processor also initiates at least one command based on tracking positional changes of the magnet.
The present disclosure will be better understood and illustrated by means of the following embodiment and execution examples, in no way limitative, with reference to the appended figures on which:
Wherever possible, the same reference numerals will be used throughout the figures to refer to the same or like parts.
It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the present invention, while eliminating, for purposes of clarity, many other elements found in typical digital multimedia content delivery methods and systems. However, because such elements are well known in the art, a detailed discussion of such elements is not provided herein. The disclosure herein is directed to all such variations and modifications known to those skilled in the art.
Ĥ, in three dimensions [Ĥx,Ĥy,Ĥz].
Magnetometers sense magnetic fields and have become cheap and ubiquitous with the rise of smartphones. These sensors are capable of measuring the strength of the magnetic field vector on its three orthogonal axis as shown in
Fundamentally, a magnetic field H (μT) can be decomposed into two orthogonal component vectors, tangential Hr and radial Hθ:
where K is a constant (in units of field μT in cm3) related to the magnetic moment and depends on the specific permanent magnet used. r (cm) is the distance from the magnet to the sensor and θ is the angle from the north pole of the magnet to the sensor as was shown in
The above formulas allow calculation of a 2D coordinate (x,y) for use with a user interface from the magnetic field reading [Hx, Hy ]. Note that this derivation assumes the magnet and sensor are aligned with the pole parallel to the x-axis and Hz=0. If the configuration is different, the known rotation T needs to be applied to the raw sensor readings.
Applying this concept to an embodiment such as shown in
In one embodiment, as shown in
The interaction surface where the magnet moves is the right hand vertical surface (the YZ plane). In a VR enclosure, the phone is mounted in the back of the device (away from the user) in the XY plane. In one embodiment, a test apparatus can be used that has a bar for allowing the mounting of a magnetometer in a similar configuration.
Furthermore, the magnetometer can be disposed in one of a plurality of locations. In one example, three different locations can be used for simplicity. In this example a distance from the interaction surface (along the x axis) of 1.5 cm, 7.0 cm, or 12.5 cm is used. The interaction surface in this example has an area of 8 cm×8 cm and the magnet can be moved vertically (y between −3.5 cm and +4.5 cm) and horizontally (z between 8 cm and 0 cm). In one embodiment a stand-alone magnetometer and an accelerometer/gyroscope can be connected directly to sensor. Given this configuration, Equations 3 and 4 can be used for the final calculation. When the user moves the magnet in the YZ plane which is the know distance x from the magnetometer. Furthermore, the magnet is axis aligned with the sensor. Since the magnetic field is rotationally symmetric about the pole, only need to consider is that of the plane coincident with the x-axis that passes through the pole of the permanent magnet. The magnetic field can now be measured by rotating the reading along the x-axis:
α=a tan 2(Ĥz,Ĥy).
Now using Equations 3 and 4:
H
x
=Ĥ
z and Hy=√{square root over (Ĥy2+Ĥz2)}
Since x is known then there is only variable (y) to solve. To solve for y, it can also be transformed back into the Cartesian coordinate frame of the magnetometer:
ŷ=y cos α,{circumflex over (z)}=y sin α
The above formula may not always take into account the situations that include more than a single magnet. In addition to the magnet used to create the field for input, there is the magnetic field of the Earth and other ambient sources to consider. Therefore, the impact of this field must be accounted for otherwise the distance calculations will be incorrect. Furthermore, using magnetic field sensing in this way has other parameters that must be considered. In particular, some parameters to consider are the strength of the magnet, distance from the magnet to the sensors, and sensor gain that will all play a role in an overall system design.
In addition,
Taking this into consideration, Table 1 provide results for a static test (original value, after calibration) as provided:
The first experiment demonstrates applying these equations to sensor data in a VR enclosure configuration. The static test can be performed, for example, on a table with our apparatus so that a constant ambient magnetic field, G can be achieved. Given the above analysis, the only report shown is for data in the middle sensor position (x=7:0 cm) and a magnet where K was measured to be approximately 8200. The sensitivity can also be increased to achieve better results. For all of the magnetic field readings in the figures, first a calibration can be made for hard and soft iron effects. This means to remove the permanent magnet from the area and rotate the device around all axes. If the magnetometer can be perfectly calibrated, all of these points would lie on a sphere centered at (0,0,0) with the radius being the magnitude of the ambient magnetic field. In practice, however, there can be hard iron effects (the center is offset) and soft iron effects (the sphere is deformed into an ellipsoid). To compensate for these effects, an ellipsoid to the data is fit after removing outliers. The parameters of the ellipsoid (centroid and major/minor axes) provide the needed information to transform the raw magnetometer readings into ones where these effects are corrected. This calibration procedure is performed once and applied to every sensor reading taken from the magnetometer.
In
H=Ĥ−G (5)
In this example. To achieve representative data, a printed template was used attached to a 8 cm×8 cm interaction with 9 positions and manually placed the magnet in each location as was previously shown in
For comparison, the magnet itself is 1.27 cm in diameter. Overall, this data provides the conclusion that if somehow the ambient magnetic field can be removed, one can obtain good 2D position measurements for user input on a VR enclosure. Unfortunately, as discussed the ambient field can have a very large impact on the position measurements.
In one embodiment, a device rotation scheme can be used to resolve these issues. In one embodiment, a VR enclosure is provided that will not be stationary and therefore the strategy from the previous experiment of estimating the ambient field first and assuming it remains constant can provide to be unrealistic. To demonstrate the impact of not accounting for the direction of the ambient field, a second experiment similar to the previous one can be conducted. Here, the device is oriented so that the ambient field is aligned with the Z axis (0 degrees) and measure the ambient magnetic field at the beginning of the experiment as before:
(G=[2.35,−26.39,20.38],|G|=33.42).
In one embodiment, the magnet is placed in a single position (in the middle at z=4 cm, y=0.5 cm) and rotate the whole apparatus about the Y axis at 22.5 degree increments through a full circle. At each position, we again collect 500 readings and subtract the initially determined magnetic field vector G from each (Eq. 5). The mean position of the magnet can then be calculated.
To compensate for the ambient field, fundamentally, the challenge is with using magnetic field sensing for VR enclosure. That is that the field can change for one of two reasons. It varies as the user moves the magnet to provide input as intended. It also changes as the user moves the VR enclosure around to look at different elements in the virtual world. The above data and analyses show the error introduced by movement can be enough to make the distance calculations useless for input. While carefully selecting the strength of the magnet or position of the magnetometer might mitigate the impact, it does not address this issue. It is impossible to differentiate between these two with just the one magnetometer in a phone. In one embodiment the inertial sensors in the phone is used.
In one embodiment, phone movement may be tracked by allowing the user only move the magnet or only the VR enclosure, As long as both of these two components are not moved at the same time, there will be no issues. When just the magnet is moving, the above solution can be applied where first the value is estimated and then geomagnetic field (Eq. 5) is subtracted. G in this case will be treated as a constant. H is determined by applying Equations 3 and 4. When the VR enclosure moves and the magnet is stationary, the opposite can be performed as shown below:
G=Ĥ−H (6)
The field generated by the magnet H is treated as constant since the user is not moving it and we re-determine G. Having a mode for tracking phone movement versus allowing input could be permissible depending on the interactions exposed to the user. However, asking the user to stay perfectly stationary while providing input is not feasible. Consequently, a way to track the changing orientation of G is needed while also allowing the user to move the magnet. Fortunately, the VR enclosures must already estimate the rotation of the phone so it can create the right viewport into the virtual environment. The same approach can be used to track the orientation of the ambient magnetic field as the user moves. In other situations, orientation can be tracked using the accelerometer and the magnetometer. These sensors form a basis (measuring gravity pointing down and the Earth's magnetic field pointing north) for determining absolute orientation. However, since some devices use the magnet for input, they forego the magnetometer and instead must rely on fusing the accelerometer and gyroscope. For example, a mobile device may use web browsers which uses a complimentary filter to fuse accelerometer and gyroscope sensor readings. By integrating the measurements of rotational velocity over time and fusing them with the gravity vector, the complimentary filter provides an estimate of the rotation matrix R that transforms the starting reference frame to the current one.
In one embodiment, an algorithm or a conventional graphics software can be used for this rotation to create the right view port into the virtual world. This can be used to track G. As such, a gyroscope and accelerometer was added to the test apparatus (
G=R·G
i (7)
This approach allows the user move both their head (and the VR enclosure) as well as the magnet simultaneously if desired. The inertial sensors can track G as the user moves and Equation 5 (and in turn Equations 3 and 4) can be used to obtain the 2D position of the magnet when accurate devices are used. However, where precision instruments are not used, the use of the gyroscopes can cause have both drift and noise. To make account of this situation (and given how sensitive it is to have an error or any incorrect magnetic field readings—see
In another embodiment and using another approach to minimize the impact of drift, one can have magnet return to a known position after each input session. In some earlier prior art devices, a second fixed magnet was used on the inside of the device so that input magnet would re-center. Unfortunately, later devices have removed this option and the option so this option no longer available to determine the location of the two separate magnets (and associated magnetic field) with just the one magnetometer. Therefore, an embodiment can be provided that is designed so the user positions the magnet in a known location. In this case, there is the potential for some drift during input, but it does not accumulate from session to session. For instance, a detent could be created on the surface of the device so the user can return the magnet to a known position. In one example, the user can make gestures along the edge of the device to enter letters of the alphabet. However by design, the user also always stops a given gesture in a known corner. That position would let the system reset to a known magnetic field reading H and determine the current ambient field reading G.
As can be appreciated by those skilled in the art, different implementations can be provided. In one example, as was shown and discussed in conjunction with
In a second example, a version of the system is implemented for use in a VR enclosure. In one embodiment, a mobile phone or smart phone can be used and sensor data can be obtain (for example Objective C code can be used to obtain “raw” sensor data). For example, startMagnetometerUpdatesToQueue:withHandler can be used to obtain CMMagnetomete sensor events. In such an example, the result that is obtained will show that the readings are directly acquired from the sensor. In other embodiments, further processing by the processor is needed (by obtaining information from both firmware or at operating system level) when strong magnetic fields are present in close proximity and affecting the calibration of the sensor data. In another embodiment, (where raw events are used) hard and soft iron corrections for the smart phone can also be used. The latter step can provide solutions to minimize significant hard iron effects, likely due to the close proximity of the magnetometer such as to the ear of a user/speaker.
Unlike the previous example of a test apparatus, the exact position of the phone magnetometer may not be known. However, by examining magnetometer data, one can estimate that it is approximately ion a certain range. For example, in one mobile phone example, the measure range was about 1.1 cm from the top of the phone and 1.9 cm from the left.
In instances where the magnetometer is provided close to the edge, as was shown in
To aid understanding, several different interaction examples can now be used in conjunction with the different embodiments. Each example is a plane positioned in three dimensional (3D) environment. In the examples, in one embodiment, to enter the input mode, the user places a “gaze” cursor on a component (i.e. a widget) by rotating the VR device. After some time, as short as two seconds, the system switches modes which is visually indicated by changing the color of the surface. At this point, the system starts tracking device rotation to estimate G and also calculates the 2d position of the magnet which is used by the user interface. When the gaze cursor leaves the component or the widget, the system returns to tracking only mode. To demonstrate this in one example, the depiction of
In a second example is provided in
In a third example can also be understood in conjunction with
In using magnetic field sensing for 2D input, the ambient magnetic field can have a significant impact. However, a detailed tracking device orientation using the inertial sensors, allows the embodiments discussed to successfully use the magnet for user input. In addition, the APIs available do not provide the same level of access to the sensors as a direct hardware API. Also, mobile phone platforms offer several soft sensors that utilize a variety of filtering and sensor fusion algorithms to overcome some of the limitations of the sensors or compensate for known but proprietary factors impacting the sensors. In one embodiment, an API can be created that allows a developer to leverage the processing the phones are doing on the sensor data but that also takes into account the presence of relatively large moving magnetic field that can be used for input in some embodiments.
In addition, in one embodiment, a holistic system that tracks all of the unknown variables is provided. Given the above API issues, in one embodiment a standalone tracking system based on a complementary filter can be used but in alternate embodiments.
In a different embodiment, where a strong pair of magnets is needed to create a sufficiently large signal, it can be difficult to slide the magnet over the surface due to the force exerted between the magnets. In such cases, a coating can be applied to the magnets with a lower coefficient of friction to help with handling. In a different embodiment, a handle can be added for the magnet to provide for easier input. The handle could both reduce friction between the magnet and the device cardboard surface and even provide a better point for grasping the magnet with the fingers allowing for easier actuation.
Finally, while simple interaction examples were discussed to demonstrate the capability of our system to provide 2D input, it should be understood that interactions where the position of the magnet at the start or end of a movement is known can also be considered. Doing so would minimize any cumulative tracking errors. In alternate embodiment, a switch between tracking and input modes can be explored and in other embodiments techniques to create or adapt existing VR input to make use of our continuous 2D pointing capability can be implemented.
Virtual Reality enclosures provide a cheap and simple way to experience VR content. However, given that cost and simplicity is such a driving factor in their design, there are limited opportunities for input. In different embodiments, as discussed, different ways to use a magnet for input can be explored from the simple to the more complex including replacing magnets only for use as a binary to using the magnetic field sensing to provide continuous 2D input. By tracking the orientation of the ambient magnetic field using the inertial sensors in the phone, one can successfully calculate the 2D position of the magnet using the embodiments discussed.
While some embodiments has been described, it will be understood that those skilled in the art, both now and in the future, may make various improvements and enhancements which fall within the scope of the claims which follow. These claims should be construed to maintain the proper protection for the invention first described.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US17/49627 | 8/31/2017 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62383873 | Sep 2016 | US |
