Patent Application
20160267720
The present invention relates generally to the viewing of a reality, and in particular to a comfortable, pleasant, informative and realistic viewing of such a reality by a user. The reality can be a virtual reality, an augmented reality or a mixed reality.
When an item moves without constraints in a three-dimensional environment with respect to stationary objects, knowledge of the item's distance from and inclination to these objects can be used to derive a variety of the item's parameters of motion as well as its pose. Particularly useful stationary objects for pose recovery purposes include a ground plane, fixed points, lines, reference surfaces and other known features.
Over time, many useful coordinate systems and methods have been developed to parameterize stable reference frames defined by stationary objects. The pose of the item, as recovered and expressed in such stable frames with parameters obtained from the corresponding coordinate description of the frame, is frequently referred to as the item's absolute pose. Based on the most up-to-date science, we know that no absolute or stationary frame is available for defining truly absolute parameters. Stable frame is thus not to be construed to imply a stationary frame. More precisely stated, the stable frame in which the absolute pose is parameterized is typically not a stationary or even an inertial frame (for example, a reference frame defined on the Earth's surface is certainly stable, but not stationary and non-inertial due to gravity and Earth's rotation). Nevertheless, we shall refer to poses defined in stable frames as “absolute” in adherence to convention.
Many conventions have also been devised to track temporal changes in absolute pose of the items as it undergoes motion in the three-dimensional environment. Certain types of motion in three dimensions can be fully described by corresponding equations of motion (e.g., orbital motion, simple harmonic motion, parabolic motion, curvilinear motion, etc.). These equations of motion are typically expressed in the stable frame defined by the stationary objects.
The parameterization of stable frames is usually dictated by the symmetry of the situation and overall type of motion. For example, motion exhibiting spherical symmetry is usually described in spherical coordinates, motion exhibiting cylindrical symmetry in cylindrical coordinates and generally linear motion in Cartesian coordinates. More advanced situations may even be expressed in coordinates using other types of parameterizations, e.g., sets of linearly independent axes.
Unconstrained motion of items in many three-dimensional environments, however, may not lend itself to a simple description in terms of equations of motion. Instead, the best approach is to recover a time sequence of the item's absolute poses and reconstruct the motion from them. For a theoretical background, the reader is referred to textbooks on classical mechanics and, more specifically, to chapters addressing various types of rigid body motion. An excellent overall review is found in H. Goldstein et al., Classical Mechanics, 3rd Edition, Addison Wesley Publishing, 2002.
Items associated with human users, e.g., items that are manipulated or worn by such users, generally do not move in ways that can be described by simple equations of motion. That is because human users exercise their own will in moving such items in whatever real three-dimensional environment they find themselves. It is, however, precisely the three-dimensional motion of such items that is very useful to capture and describe. That is because such motion may communicate the desires and intentions of the human user. These desires and intentions, as expressed by corresponding movements of the item (e.g., gestures performed with the item), can form the basis for user input and interactions with the digital domain (e.g., data input or control input).
In one specific field, it is important to know the absolute pose of an item associated with a human user to derive the position of its tip while it contacts a plane surface. Such position represents a subset of the absolute pose information. Various types of items, such as elongate objects, can benefit from knowledge of their pose, which includes the position of their tip. More precisely, such items would benefit from knowing the absolute position (in world coordinates parameterizing the stable frame) of their tip while it is in contact with a plane surface embedded in the three-dimensional environment. These items include walking canes when in touch with the ground, pointers when in touch with a display or projection surface, writing devices when in touch with a writing surface, and styluses when in touch with an input screen.
The need to determine the absolute position of the tip or nib is deeply felt in the field of input devices such as pens and styluses. Here, the absolute position of the tip has to be known in order to analyze the information written or traced by the user on the writing surface. Numerous teachings of pens and related input devices providing relative tip position and absolute tip position are discussed in the prior art. Some of these teachings rely on inertial navigation devices including gyroscopes and accelerometers as described in U.S. Pat. Nos. 6,492,981; 6,212,296; 6,181,329; 5,981,884; 5,902,968. Others combine inertial navigation with force sensing as described in U.S. Pat. Nos. 6,081,261; 5,434,371. Still other techniques rely on triangulation using signal receivers and auxiliary devices on or adjacent to the writing surface as found in U.S. Pat. Nos. 6,177,927; 6,124,847; 6,104,387; 6,100,877; 5,977,958 and 5,484,966. Furthermore, various forms of radiation including short radio-frequency (RF) pulses, infra-red (IR) pulses, and even sound waves in the form of ultrasound pulses have been taught for triangulation and related techniques. A few examples of yet another set of solutions employing digitizers or tablets are discussed in U.S. Pat. Nos. 6,050,490; 5,750,939; 4,471,162.
The prior art also addresses the use of optical systems to provide relative, and in some cases, absolute position of the tip of a pen or stylus on a surface. For example, U.S. Pat. No. 6,153,836 teaches emitting two light beams from the stylus to two receivers that determine angles with respect to a two-dimensional coordinate system defined within the surface. The tip position of the stylus is found with the aid of these angles and knowledge of the location of the receivers. U.S. Pat. No. 6,044,165 teaches integration of force sensing at the tip of the pen with an optical imaging system having a camera positioned in the world coordinates and looking at the pen and paper. Still other teachings use optical systems observing the tip of the pen and its vicinity. These teachings include, among others, U.S. Pat. Nos. 6,031,936; 5,960,124; 5,850,058. According to another approach, the disclosure in U.S. Pat. No. 5,103,486 proposes using an optical ballpoint in the pen. More recently, optical systems using a light source directing light at paper have been taught, e.g., as described in U.S. Pat. Nos. 6,650,320; 6,592,039 as well as WO 00217222 and U.S. Patent Appl. Nos. 2003-0106985; 2002-0048404.
In some prior art approaches the writing surface is provided with special markings that the optical system can recognize. Some early examples of pens using special markings on the writing surface include U.S. Pat. Nos. 5,661,506; 5,652,412. More recently, such approach has been taught in U.S. Patent Appl. 2003-0107558 and related literature. For still further references, the reader is referred to U.S. patent application Ser. Nos. 10/640,942 and 10/745,371 and the references cited therein.
The rich stream of information expressing an item's absolute pose combines its three linear or translational degrees of freedom with its three rotational degrees of freedom. Typically, translations are measured along linearly independent axes such as the X, Y, and Z-axes. The translation or displacement along these axes is usually measured by the position (x,y,z) of a reference point on the item (e.g., the center of mass of the item). The three-dimensional orientation of the item is typically expressed by rotations taken around three linearly independent axes. The latter are typically expressed with three rotation angles, such as the Euler angles (φ,θ,ψ).
However the prior art comes short on several fronts when it comes to providing a rich, and comfortable virtual reality, augmented reality or mixed reality experience to the user. In particular, the prior art does not teach a system or methods for utilizing a viewing mechanism to view a reality/environment comprising real and/or virtual objects, where the virtual objects may be altered or modified based on one or more properties of an inside-out camera. Examples of such one or more properties include the pose of the camera or a reduced homography. The reality viewed by the user may be a virtual reality, an augmented reality or a mixed reality. The above alteration or modification of the virtual objects may be necessary to avoid motion sickness for the user. Such a sickness occurs because of the conflict between the vestibular and ocular responses of the brain, as a result of the motion of the user and the system's inability to render appropriate and timely changes to the images/reality viewed by the user.
The prior art is also silent about the many different choices available for the properties of the inside-out camera according to which the virtual object(s) in the reality may be modified as described above. Such modification(s) may be necessary to enhance the experience of the user in viewing the reality. The prior art is also silent about the fact that the reality may be viewed from either a user viewpoint or the viewpoint of a device which is detached from the user.
In view of the shortcomings of the prior art, it is an object of the present invention to teach systems and methods for providing a rich, pleasant and comfortable virtual reality, augmented reality or mixed reality experience to the user.
It is also an object of the invention to provide techniques for modifying the appearance of one or more virtual objects in a reality or environment viewed by the user. The reality is viewed by the user via a viewing mechanism, and the modification is based on one or more properties of an inside-out camera. The appropriate properties of the inside-out camera are suitably chosen according to the application at hand.
It is further an object of the invention to allow an array of choices for the properties based on which the above modification is performed. These choices include the pose of the camera, a reduced homography and any other property recoverable from the output of the camera.
It is further an object of the invention to allow for the user to view the reality from his/her own viewpoint or from the viewpoint of another device.
The numerous objects and advantages of the systems and methods of the invention will become apparent upon reading the ensuing description in conjunction with the appended drawing figures.
The objects and advantages of the present invention are secured by a system having a viewing mechanism, in which a user views one or more virtual objects in an environment or scene. The environment is viewed by the user from his or her viewpoint or vantage point. The system further employs a projection mechanism that projects or displays the environment that is viewed by the user via the viewing mechanism. There is an inside-out camera or more than one inside-out cameras with various properties that can be measured or recovered from its/their output. Then based on one or more such properties, the projection mechanism alters or modifies the one or more virtual objects in the environment viewed by the user from his/her viewpoint.
In the preferred embodiment, the environment or scene viewed by the user is preferably a virtual reality. In a variation, the environment is an augmented reality. In another related variation, the environment is a mixed reality. Preferably, the property of the inside-out camera based on which the one or more virtual objects in the scene/environment are modified for the user, is the pose (position and orientation) of the user. More specifically, it is the pose of the inside-out camera employed by the user, and the inside-out camera is used in the recovery of the pose. In a related embodiment, the property is a homography.
Preferably, the inside-out camera is mounted on or worn by the user. In another variation, the alteration or modification to the one or more virtual objects in the environment viewed by the user are done so as to reinforce or improve a perception of the presence of the user in a virtual reality. In a similar embodiment, the alteration or modification to the one or more virtual objects in the environment viewed by the user are done so as to reinforce/improve a perception of the presence of the one or more virtual objects in an augmented or mixed reality. In another variation, the alteration is done to correct the positioning of the one or more virtual objects in the environment viewed by the user from his/her viewpoint. In yet another variation, the alteration is done to enhance the information communicated to the user through his or her visual senses.
In still another variation, the alteration done by the projection mechanism to the viewed environment is consonant to a movement of the user. In a related variation, the alteration is done so as to minimize the motion discomfort or sickness of the user as a result of his/her movement and the corresponding changes needed to be made in the viewed environment. In another related variation, the movement of the user is constrained. The constraint on the movement of the user results in a reduced homography which is used as the property of the inside-out camera based on which the changes to the virtual object(s) in the environment are made.
In an advantageous set of embodiments, the alteration of the VR/AR/MR scene is done by the projection mechanism by changing the associated programmable and configurable parameters. Preferably these are the programmable and configurable parameters of the corresponding graphics rendering pipeline. More preferably still, these parameters are the vertex operations of the graphics rendering pipeline. It is yet more preferable, that these parameters be the fragment operations. In another variation, these parameters are the pixel-based operations of the graphics rendering pipeline. In still related embodiments, the parameters are model transformations, or view transformations or camera transformation of the graphics rendering pipeline. In still other related embodiments, these parameters are shading, diffusion and light-scattering effects applied to image fragments of the one or more virtual items being altered/modified.
In other advantageous embodiments, the inside-out camera is employed by a second user. In these embodiments the first user or set of users view the environment from the viewpoint of the second user. The inside-out camera(s) may preferably be mounted on or worn by the second user. The inside-out camera may preferably be affixed to the viewing mechanism, or it may be integrated with it, or connected to it or attached to it. Furthermore, the viewing mechanism may preferably utilize a display unit. The display unit may preferably be affixed to the viewing mechanism, or it may be integrated with it, or connected to it or attached to it. More preferably still, the display unit may be a heads-up display (HUD) or a head-mounted display (HMD). In another variation, the viewing mechanism may employ optics, and thusly be called viewing optics.
Preferably the viewing mechanism is replicated so that the environment is viewed in stereo by the user. More preferably still, the inside-out camera(s) and/or the projection mechanism are replicated for stereoscopically performing the alteration(s) and/or modification(s) of one or more virtual objects in the environment seen by the user from his/her viewpoint. As stated, the alteration/modification is done based on one or more properties of the inside-out camera(s).
In a highly preferred set of embodiments, the viewing mechanism is a virtual reality eyewear, an augmented reality eyewear or a mixed reality eyewear. In related embodiments, this eyewear is a set of goggles, eyeglasses, or even a telescope or binoculars.
In another set of preferred embodiments, the user employs a control device for performing the above mentioned alteration/modification to the VR/AR/MR scene. In one of these embodiments, the control device is worn by or mounted on the user. In another one of these embodiments, the control device is used to control the power or another signal delivered to the viewing mechanism and/or the inside-out camera. In still another embodiment, the control device is used to control the appearance of an image in the viewing mechanism. Preferably, the control device is a joystick, a game controller, a touch sensor, a gesture sensor (e.g. the ones used in games and smartphones), a digital pen (e.g. a stylus), a proximity sensor (e.g. a capacitive, photoelectric or inductive sensor), a vicinity sensor (e.g. using radio frequency identification (RFID) technology), an electromagnetic sensor, an inertial sensor (e.g. an accelerometer or a vibration sensor) or one of the many types of motion sensors.
In similar embodiments, instead of a control device, the system uses an auxiliary sensor for controlling the appearance/modification of the one or more virtual objects. Preferably, the auxiliary sensor is an optical sensor, an inertial sensor (e.g. an accelerometer or a gyroscopic sensor), a magnetometer, an optical flow sensor, a displacement sensor, an acoustic sensor, a Radio Frequency (RF) sensor.
In a highly preferred set of variations, the system employs a device which has the inside-out camera(s). Now, the viewing mechanism is used by the user to view the environment containing one or more virtual objects/items from the device viewpoint (instead of from the user viewpoint as in prior embodiments). Then based one or more properties of the inside-out camera(s) employed by the device, appropriate modification(s) and/or alteration(s) to the one or more virtual objects in the scene/environment are performed by the projection mechanism. All other teachings of the prior embodiments still apply to these variations, except that the environment now is viewed from the device viewpoint.
The device in the above embodiments may be controlled by the user, or it may be an autonomous or semi-autonomous device. The device may be a drone, a robot, a remotely controlled tool or implement, a remotely controlled automotive equipment, etc. The viewing mechanism and projection mechanisms may preferably be integrated with each other, and in fact be the same. The projection mechanism may preferably employ a display unit.
The display unit may be integrated in the device from whose viewpoint the environment is being seen by the user, or it may be affixed to it, attached to it, or operably connected to it. Similar to prior embodiments, the above mentioned alteration/modification to the scene (specifically the virtual objects in it), may be done consonant to a motion of the device. The motion of the device may preferably be constrained, resulting in a reduced homography. Generally, the device in these embodiments may be a manipulated device/item, i.e., it is moved or operated directly by the user (e.g., by hand), or the device is a wearable device, which is carried or worn by the user.
In still another set of embodiments, the system employs optical sensor(s) for imaging preferably non-collinear points in an environment that is viewed by a viewer. Another mechanism then layers one or more virtual objects onto that environment that is viewed by the viewer. These one or more virtual objects are then modified/altered by the above mechanism based on the tracking of the movements of the viewer by the system. The viewer is preferably a human whose discomfort is minimized. In an alternative variation, the one or more virtual items may be modified/altered based on one or more properties of the optical sensor(s). Preferably, the optical sensor(s) is/are embodied in an HUD/HMD. Preferably, the optical sensor(s) are affixed to a gear that is mounted on, manipulated by or worn by a user.
The methods of the invention further provide the steps required to accrue its benefits. Specifically, the methods provide for layering of one or more virtual objects by a projection mechanism onto an environment viewed by a viewer from its viewpoint. Then appropriate modifications to the one or more virtual objects may be performed based on one or more properties of an inside-out camera. The inside-out camera may be mounted on or worn by the viewer.
According to the methods, in a preferred embodiment, the environment seen by the viewer is a virtual reality scene, an augmented reality scene or a mixed reality scene. The property (or properties) based on which the above modification/alteration of the one or more virtual objects is done is preferably a pose, a homography, or other properties recoverable/measurable from the output of the inside-out camera.
According to the methods of the invention, the viewer is preferably a sentient being such as a human or an animal. The viewer may also preferably be a machine or a device such as a drone, a robot, a manipulated tool, a remotely controlled tool, a remotely controlled implement, a remotely controlled automotive equipment, or an artificially intelligent agent. As in prior embodiments, the alteration/modification of the virtual object(s) in the scene is preferably consonant with the motion of the viewer.
The above motion of the viewer is preferably constrained, resulting in a reduced homography, rather than a full or regular homography.
This is preferably because of the presence of structural uncertainties in the optics of the viewing mechanism, or because of structural redundancies caused by the conditioned motion of the viewer.
The specifics of the invention and enabling details are described below with reference to the appended drawing figures.
The figures and the following description relate to preferred embodiments of the present invention by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of the claimed invention.
Reference will now be made in detail to several embodiments of the present invention(s), examples of which are illustrated in the accompanying figures. It is noted that wherever practicable, similar or like reference numbers may be used in the figures and may indicate similar or like functionality. The figures depict embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.
The various aspects of the invention will be best understood by initially referring to system 100 containing an environment 130 as shown in
Environment 130 also shows a mobile phone 110 with a viewing mechanism or screen or display 112 and an inside-out camera 114. The schematic diagram of camera 114 is shown as 114′. Schematic 114′ illustrates how camera lens 116 projects the images of real world objects 102, 104, 106 and 108, or any other objects that may lie within the camera's angular field-of-view 115 onto photo-sensor 111. These objects are projected through a single viewpoint 113 of camera 114. Objects 102 and 104 are configured to contain features convenient for calculating the pose of camera 114 with respect to the stable environment 130.
The pose thus recovered may then be used to alter the appearance of virtual object 120 to correspond with any changes occurring in the pose of camera 114 as mobile phone 110 is moved by the user to different poses within stable environment 130. It should be noted that while the present embodiment depicts a single viewpoint 113, the present invention is capable of working with similar embodiments in less than ideal conditions of a strictly single viewpoint. In other words, the present teachings, and the pose recovery techniques of the related references that the present invention utilizes, can work in the presence of some image aberration if viewpoint 113 was a little “smeared out”.
Although inside-out-camera 114 in the example shown in
As shown in
Environment 130 can also have any number of virtual objects. In the example of
As shown in
As will be further explained below, the mixed reality scene may be viewed from two viewpoints of reality 140 illustrated in
The above two viewpoints for the left and right eyes may be created synthetically by knowing the positions of the user's eyes with respect to the inside-out camera's viewpoint 113. Then optical projection system 155 (shown disassembled from head-mounted display (HMD) 150) is used to project the image information from two viewpoints emanating from different regions of display 112 onto the user's respective retinas. In some cases, optical system 155 may simply comprise two lenses 160A and 160B as shown in
It should be noted that in the above example, even though smartphone 110 has only one camera 114, we are still rendering both left and right images in the display via the two synthetically created viewpoints. Virtual objects, such as virtual object 120, can be rendered correctly in both the left and right display, but the real scene is captured only through viewpoint of camera 114. This may not be sufficient for creating a ‘true’ stereo image. In order to render the real world objects for a true stereo vision, one must have two cameras corresponding to left and right eyes, or camera 114 must capture depth information, image disparity or a similar property allowing stereoscopic rendering.
Thus camera 114 can be a stereo camera or depth camera. Alternatively, camera 114 can be a conventional camera capturing a sequence of images of the same scene from slightly different viewpoints. Indeed, in an interesting variation of the embodiment shown in
A second inside-out camera may also be employed to increase the collective field-of-view for gathering the images of real world objects. Additionally, the second inside-out camera may be used for providing auxiliary pose information that may be required in cases when the particular real world objects that contain features convenient for calculating the pose (i.e., objects 102 and 104) lie outside the field-of-view of the first inside-out camera.
As understood in the art, a mixed reality (MR) refers to a system that combines real and virtual objects and information that may be fused or layered together to give the viewer an enhanced viewing experience, compared to just the virtual reality or purely real or real reality environments. As shown in
As used in this disclosure the terms scene, environment, image, rendering and reality may be used interchangeably to represent a scene or a sequence of scenes being observed by a user of the system, and any distinction will be drawn as and if needed. Also, the distinction between VR, AR, MR may be drawn only as needed knowing that the principles of the invention apply to any scene or environment observable through a viewing mechanism in concert with inside-out camera(s) and associated elements of the system as taught in this disclosure.
Recovering three dimensional (3D) pose (position and orientation) of manipulated, worn or remotely controlled objects is a hard problem. There are two approaches that choose fundamentally different camera placements to achieve this purpose. The outside-in camera method places camera(s) in the environment to track the user's/viewer's VR/AR/MR gear. The inside-out camera method places one or more cameras on the user's/viewer's VR/AR/MR gear to track the pose of the user/viewer based on the same rules of perspective geometry as humans apply naturally.
The pose of the inside-out camera can be recovered from certain features of objects in the environment that lie within the field-of-view of the inside-out camera. The pose of the inside-out camera can then be transformed into a new pose corresponding to a different camera orientation and/or a different camera position. That is often useful for projecting virtual images onto a user's retina corresponding to the different viewpoints of each the user's eyes while viewing a scene.
For systems having stereoscopic displays (i.e. using a different region of the display for each eye), it is useful to know the position and orientation of the user's eyes (i.e. the pose of each eye) with respect to the pose of the inside-out camera. That is in order to facilitate the calculations of the absolute pose of each eye (with respect to the environment), and for projecting virtual objects that are displayed properly to each eye for stereoscopic vision of the virtual objects.
The above can be accomplished by coordinate transformations that take advantage of the knowledge of the spatial relationships between the inside-out camera and the user's eyes. In some cases, multiple inside-out cameras may be used to provide a larger combined field-of-view (i.e., one inside-out camera facing forward and a second inside-out camera facing backward) to insure that real world objects having certain features convenient for recovering pose are always within the field-of-view of at least one of the inside-out cameras. With the inside-out camera techniques as employed by the instant invention, instrumentation of the environment is not a requirement.
For a detailed treatment of pose recovery techniques using an inside-out camera, the reader is referred to U.S. Pat. No. 7,826,641, U.S. Pat. No. 7,961,909, U.S. Pat. No. 8,553,935, U.S. Pat. No. 8,897,494, U.S. Pat. No. 9,235,934, U.S. patent application Ser. No. 14/992,748, U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856 and U.S. patent application Ser. No. 14/926,435.
According to the main aspects, virtual item 120 in
Since in
Note further that throughout this disclosure when referring to such alteration or modification or correction or adjustment or compensation, we may use the noun in the singular with the understanding that more than one such alterations or modifications or corrections or adjustments or compensations to the images/scenes may be made based on one or more properties of the inside-out camera(s). These properties may encompass extrinsic parameters of the inside-out camera(s) i.e. orientation and pose, its intrinsic parameters i.e. focal length fx, fy, optical center (x0,y0) and axis skew, as well as many other properties that will be taught later in this specification.
Now recall the embodiment of
The result is either a VR experience, or in conjunction with the phone's camera an AR or MR experience for the user. A non-exhaustive list of such headsets includes Google Cardboard, Freefly, VR One, ColorCross, etc. As in the case of the embodiment of
In addition, a variety of other higher-end wearables are also available in the market that comprise the entire optical and electronic circuitry to provide a complete VR/AR/MR experience, without requiring a smart phone or handset. A non-exhaustive list of such eyewear products includes the Oculus Rift, Sony PlayStation VR, HTC Vive Pre, ODG R-7 Smart Glasses, Microsoft HoloLens, FOVE VR, Avegant Glyph, etc. An example of such an integrated HUD/HMD 152 is shown in
Note, that since environment 132 is actually also seen through the optics of eyeglasses 152 onto which virtual object(s) are projected, as such the eyeglasses 152 would typically be characterized as AR/MR glasses, rather than VR glasses. The latter typically do not require and have a “see through” capability. Such a capability is also sometimes referred to as “Optical see-through” capability in the art. A related capability called “Camera see-through” in the art is sometimes used to refer to a VR setup where two cameras are used to provide binocular vision for the user of the environment, and onto which virtual objects may be layered.
However, to avoid undue detraction from the teachings of this disclosure, we will consider eyeglasses 152 to be VR, AR and/or MR (or more simply VR/AR/MR), while knowing the above subtle difference and distinguishing as and if necessary. Moreover, one could conceive of eyeglasses 152 to be VR type as well, if the see-through capability is blocked and the entire scene projected onto the user's retinas is virtual.
Device 152 in
Images 142 and 142′ are projected onto the retinas of eyes 151A and 151B respectively by image projectors (i.e., pico-projectors) 159A and 159B. The complete projection system of device 152 also includes the eyeglass lenses that provide see-through capability. As indicated by the two respective angled arrows, they also act as reflectors for deflecting the image information emanating from projectors 159A and 159B into eyes 151A and 151B respectively.
By employing the techniques taught in the above mentioned references, once the pose of inside-out camera 153A with respect to stable environment 132 is established, then the pose of the user's left and right eyes 151A and 151B respectively can also be calculated. This can be accomplished by familiar coordinate transformations derived using known spatial relationships of inside-out cameras 153A-B and user's eyes 151A-B. These relationships may be established by the specific design specifications of the head-mounted display (HMD). The accuracy and fidelity of the projected environment may be further improved by using eye-tracking hardware that utilize techniques well known in the art of HMD design.
A representation of the user's eye 151 is shown in
Each user eye 151A, 151B also has a natural angular field-of-view 156, which allows viewing of real world objects in environment 132 that lie within this angular field-of-view range. Note for clarity in
Projector 159B projects a virtual image of virtual bird/object 120 onto the retina of eye 151B, thereby producing a composite viewing of both real and virtual images on the user's right retina. This is an example of augmented reality (AR), whereby virtual objects are projected on the retina having the appearance of a real world object. The same process is replicated for left eye 151A through the use of projector 159A. The result is an Optical see-through capability introduced earlier.
In an alternate variation of this AR/MR system, the eyeglass lenses may be opaque, thus not allowing eye 151B to directly view the real world objects in environment 132. Instead, the images of the real world objects are gathered by camera 153B and projected by projector 151B onto the retina of eye 151B in the same location, and with same or altered appearance as compared to when the eye is directly viewing through a transparent eyeglass. The same process is repeated for left eye 151A by employing camera 153A and projector 159A.
The result is a Camera see-through capability introduced earlier. Moreover, virtual object 120 can be simultaneously projected onto the retina to produce a mixed reality experience for the user. This is an example that represents a case of augmented virtuality (AV), whereby real world objects are projected on the retina having the appearance of virtual objects. Both of these cases described above represent different versions of mixed reality (MR) systems.
The rendered scenes 142 and 142′, projected by projectors 159A and 159B onto the left and right panes of eyeglasses 152 respectively, are slightly different, each corresponding to the vision of one eye of the user. Rendered scenes 142 and 142′ may include unaltered portions of the captured images, or they may be further corrected/processed using knowledge of the actual pose of each eye 151A, 151B using known dimensional properties of headwear 152.
Having multiple cameras further facilitates in the quick and accurate recovery of user/camera pose using the techniques taught in the above references. However, the invention admits of having just a single inside-out camera for pose recovery (typically situated at the center of the eyeglasses at or slightly above the center of eyes of the user). Therefore, in subsequent drawings we may not explicitly label cameras 153A and 153B with the admission that one or multiple such cameras may be present on the wearable used by or mounted on the user.
In these and other related embodiments, the invention employs a projection mechanism, which is actually responsible for displaying/projecting the VR/AR/MR scenes/environment onto the retinas of the user's eyes as the user views the VR/AR/MR scenes/environment. The projection mechanism may employ appropriate hardware and software technologies to generate the requisite scenes and images that are seen by the user via the viewing mechanism.
Often times, the viewing mechanism and the projection/display mechanisms are integrated and/or are one and the same. In other words, the display unit/screen of the projection mechanism is the same as the viewing mechanism used by the user to view the VR/AR/MR scenes. That is the case with HUD/HMD 150 of
In the case of eyeglasses 152 of
Having such a setup of integrated viewing and projection mechanisms, is typical of smartphones, tablets, eyewear and many other devices employed in these teachings where the screen/window onto which the environment is displayed is also the same screen/window viewed by the user. A viewing mechanism is any facility or capability of viewing the VR/AR/MR environment, while the projection mechanism is responsible for performing graphics rendering and the associated computations, and adjustments/alterations/corrections to the virtual objects/scene of the VR/AR/MR environment being viewed. These adjustments are based on one or more properties of the inside-out camera(s) as explained throughout this disclosure.
As such, the projection mechanism may not be explicitly illustrated in the drawing figures nor explicitly referenced in associated teachings, but is presumed to exist and be integrated with, affixed to, attached to or operably connected with the viewing mechanism/display. As stated, it is responsible for generating the images and any adjustments to the images according to the invention that are seen by the user via the viewing mechanism, and as also stated the two mechanisms are often integrated in practice.
It should be noted that having an imaging and rendering system that employs a lens or optics is not an absolute requirement for pose recovery. In alternative setups photosensor(s) may be used “bare” to register photons without an intervening lens or optics. With appropriate light modulation utilizing techniques known in the art, pose recovery can be accomplished using the techniques taught in the above mentioned references. In other words, although having an imaging system is useful and often required to project the scene/reality to a viewer, that is not a requirement for pose recovery. As such, the principles of this invention apply well to any VR/AR/MR apparatus, whether or not it employs any lensing or optics.
Returning to
For clarity, in subsequent descriptions of the invention, we will mostly use this simplified configuration to illustrate different uses of VR/AR/MR glasses. This configuration will have a centrally located inside-out camera used for pose recovery, and for gathering images of real world objects to be displayed or projected onto the user's retinas for viewing an VR/AR/MR environment. In
The principles of the invention apply well to any device whether it be just a wearable headset that can accommodate a handset device such as an iPhone or an Android phone, or whether it be a fully functional wearable device without requiring the insertion of a handset. In
In related systems of the prior art, without appropriate adjustments or compensation to the image/scene viewed by the user, a discomfort motion sickness is normally induced in the user due to his/her or camera's motion. In the embodiment shown in
Motion sickness or discomfort is a common problem in such VR/AR/MR systems of the prior art because they fail to compensate for the movement of the camera and consequently its changing pose to make appropriate corrections to the projected images/scenes. Specifically, they fail to make appropriate alterations/corrections/adjustments to virtual object 120 (or other virtual objects if present) in
In the examples of
Those skilled in the art will understand that in the context of the present applications, a feeling of unease or discomfort, customarily called motion sickness, is attributed to the conflicts that occur between the human vestibular system and the human visual or ocular system. This motion sickness is especially a problem in AR/MR environment because of the need for a realistic integration of real and virtual objects, particularly in the near-field i.e. less than 10 feet away. As the distance increases and human eyes are more and more focused at infinity, the objects appear to move less with increasing distance, and the problem is minimized. Also, sometimes in a pure VR environment such as games, blurring and fog effects on near-field objects are used to trick the eyes into focusing on just the ‘active’ objects in the field. But this is an unnatural effect, because in actual reality, real fog blurs far away objects more so than close ones.
However, for AR/MR applications, especially in the near-field, the vestibular-visual delay can cause a serious conflict between the vestibular and visual/ocular inputs of the brain if the image is not corrected for head/camera motion. There is a partial conflict when the image correction is inaccurate, but it is also the time delay (latency) between the motion of camera 114 on phone 110 (see
In order to remove the above motion sickness from VR/AR/MR scenes for objects in the near-field, the system must meet the following minimum requirements:
The present invention is able to achieve the above objectives because it is able to accurately and rapidly determine the pose of camera 114 and phone 110 (and that of the user's head/face for the embodiments of
These adjustments can be accurately made only if the pose of the viewer/user can be determined accurately and quickly using the available computing resources. Once the pose is determined on a near real-time basis by employing the techniques taught in above provided references, the projection mechanism can provide corrections to virtual object 120 or virtual objects if present, quickly enough so as to make the brain perceive a smooth and immersive VR/AR/MR experience.
Sometimes, a real object in an environment may be “cloaked” or made to disappear in an AR/MR scene by superimposing or cloaking it with a virtual object. The examples of this disclosure easily extend to such scenarios also, as will be obvious to a reader of average skill.
Again, such cloaking can only be effectively possible if the pose of the user/camera can be determined quickly and accurately while providing the VR/AR/MR experience to the user.
Techniques for recovering the pose of the camera, devices and viewers having six degrees of freedom (6 DOF) in a variety of settings have been extensively taught in related patent references. For a detailed treatment of pose recovery of a variety of viewers and/or optical apparatuses/objects, the reader is referred to U.S. Pat. No. 7,826,641, U.S. Pat. No. 7,961,909, U.S. Pat. No. 8,553,935, U.S. Pat. No. 8,897,494, U.S. Pat. No. 9,235,934, U.S. patent application Ser. No. 14/992,748, U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856 and U.S. patent application Ser. No. 14/926,435.
Let us now analyze what happens to the virtual object(s) projected by the projection mechanism of VR/AR/MR eyeglasses 152 in
The viewpoint location 154 and orientation 164 of a centrally located inside-out camera (not shown) represents the pose of the inside-out camera and thus also represents the pose of eyeglasses 152.
Specifically, as shown in
As shown in
Let us first take the three rotational DOF of user 202 of
That is because of the ‘expectation’ of human brain to see a corresponding correction or change to user's FOV. Such a corrected or expected version of the image of
As stated, the above provided references extensively teach the techniques of the recovery of the new pose of user 202. Using those techniques, the present invention is able to immediately correct for the new pose user 202 leading to the corrected/altered projected image of virtual object 120 as shown in
Similarly,
The accurate and timely recovery of pose afforded by the teachings provided in the above mentioned references, allows for requisite image/scene compensation to take place as represented by the above examples. Even if the image/scene compensation is done but not on a timely basis, motion sickness can still occur for the user. Recall the speed/latency requirements between the vestibular and ocular systems as taught above. Appropriate image/scene compensation needs to occur within a certain maximum allowable latency, to avoid conflict between the visual/ocular and vestibular systems of human body that causes motion sickness. Thus if an application is able to produce corrected images of
The techniques for providing the above corrections by the projection mechanism to images/scenes in the above examples, once the new pose of user 202 is known, are well understood in the art. For techniques regarding fast and accurate pose recovery of the user and related optical apparatuses in a variety of settings, the reader is again referred to U.S. Pat. No. 7,826,641, U.S. Pat. No. 7,961,909, U.S. Pat. No. 8,553,935, U.S. Pat. No. 8,897,494, U.S. Pat. No. 9,235,934, U.S. patent application Ser. No. 14/992,748, U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856 and U.S. patent application Ser. No. 14/926,435.
After having reviewed the effects of the three rotational DOF of the camera/user, let us now consider the three translational DOF of the camera/user and the resulting effects on the environments/scenes viewed. For this, consider
Environment 300 is similar to environment 200 of earlier examples including real objects 210 consisting of hills, trees and the sun. However, reference numerals 210 from environment 200 have been omitted in environment 300 for clarity. This allows us to better concentrate on real bird 212 of interest and virtual bird 220 in virtual cage 214, all of which are marked as such in environment 300 of
In the canonical position shown, real bird 212 and virtual bird 220 appear to user 202 side by side of each other at about the same height, as shown in
Without the above correction, real bird 212 and virtual bird 220 (in virtual cage 214) would still be side by side at the same height as in their original positions shown in
Similarly,
To summarize,
In the absence of the above corrections, virtual objects 220 and 214 would stay static as in
Once again, the techniques for providing the above corrections to images/scenes in the above examples, once the new pose of user 202 is known, are well understood in the fields of VR, AR and MR and graphics rendering. As for the techniques relating to fast and accurate pose recovery of user and optical apparatuses in a variety of settings, the reader is again referred to U.S. Pat. No. 7,826,641, U.S. Pat. No. 7,961,909, U.S. Pat. No. 8,553,935, U.S. Pat. No. 8,897,494, U.S. Pat. No. 9,235,934, U.S. patent application Ser. No. 14/992,748, U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856 and U.S. patent application Ser. No. 14/926,435.
As amply taught in the above mentioned references, the movement of a user having an inside-out camera mounted on a wearable such as glasses, along the available 6 DOF can be represented by a collineation or homography. This collineation or homography (often denoted by A or H) is expressed as
where
Note from
In other words, there is a linear distance between vantage/viewpoint 154 of user 202 as in
Specifically, the above offset is accounted for in the translation vector
U.S. Pat. No. 8,553,935, U.S. Pat. No. 8,897,494, U.S. Pat. No. 9,235,934, U.S. patent application Ser. No. 14/992,748, U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856 and U.S. patent application Ser. No. 14/926,435.
For best stereo vision that can be generated for the user, the inside-out cameras responsible for capturing the scene/environment onto which virtual object(s) are augmented, should be as close to each eye of the user as possible. Referring to
Technologies such as virtual retinal display (VRD) or retinal scan display (RSD) or retinal projector (RP) may be useful for the above purpose. However, these technologies still do not address the fact that the cameras are not mounted on the same location as the eye. As will be apparent that for VR, this is not a problem to mount the camera over the eyes because the user does not watch the real scene. But AR is more challenging because one would require mounting the cameras on the same optical axes as the two eyes, and still allow the user to see through unobstructed. Of course, for augmented virtuality (AV) one can design a standard VR headset with the cameras in front of the eyes. Then one would render a combination of the captured video/image/scene and the virtual objects. The user therefore would not directly see the world but a high-fidelity facsimile of it, i.e. the real world translated into the digital space.
At this juncture, let us study the typical body movements associated with a human user wearing eyeglasses of the previous examples and enjoying a VR/AR/MR experience. The canonical position of such a user is shown in
One section of the body is the head that pivots around the upper neck as indicated by pivot point 350 in
Now, as shown in
Similarly, as shown in
Regardless of the various movements of the portions of the body of the user as shown in the above examples of
The above alteration/adjustment/modification of the VR/AR/MR scene/image(s) preferably compensates for motion sickness of the user by ensuring that the image seen through the viewing mechanism or viewing optics of his/her wearable such as glasses, conforms the vestibular responses of human brain to the visual system of the body, thereby avoiding motion sickness. As taught above, this typically involves adjusting the image dynamically on a near real-time basis to account for a consistent visual perception by the eyes as observed naturally in the real world. However, there may be other reasons for the above alteration/adjustment of the VR/AR/MR scene/images and changing the viewing experience of the user. These reasons may include specific medical, psychological, mechanical or other needs of the application at hand.
As already stated, pose recovery techniques taught in the above mentioned references allow for a fast and accurate recovery of the pose of the user/camera. A very fast and efficient pose recovery algorithm allows plenty of time for other computing tasks. Many of these (e.g., rendering) are contingent on knowing the pose first. Thus, latency and drift in the image projection system can be reduced as the location and orientation of the virtual object(s) is calculated with time to spare. Note that the image for each eye will be different due to the separation between them. Optometrists call this offset between images perceived by the eyes, the parallax effect due to binocular vision.
It is also important to know where the user is focusing their sight in an AR/MR environment (i.e. close or far away). In general, this is not actively measured but rather inferred or assumed to be in the points/spots of interest. When the user focuses on real and/or virtual objects very far away the parallax effect or problem is simplified. The AR/MR system can then project an orthographic view of the virtual object(s). Such projection works well when the user's eyesight can be thought of “focused at infinity”. Parallax plays no role in such situations and other ocular accommodation issues are minimized. For virtual objects at intermediate distances, human optometry is an issue. It becomes an acute problem when projecting virtual objects in the near field.
Therefore, certain applications use a “window” for looking at AR/MR to side-step the parallax problem. These applications typically use a screen on which AR/MR is projected/displayed. Ideally, the screen is rather large. Still, a smart phone screen may be sufficient under many situations. A big advantage of the window method is that it does not require two separate images of the virtual object normally generated for the left and the right eye separately. Instead, the user looks at the screen and makes the necessary optometric adjustments with their own eyes.
The perception of depth in an AR/MR scene when using the window method does not result from stereoscopic images displayed to the left and right eyes. Instead, depth is inferred/perceived through other means such as textures, shades, shape familiarity, and is built up as the display window is moved around the virtual object(s). This allows the user to see each scene containing the real and virtual objects as they would appear from the various view angles and distances at which the window is placed. In other words, the AR/MR experience is achieved by combining monocular views from the different positions and orientations of the window.
One such application using the above described window method is shown in
Because the above application uses the window method explained above, only one image/scene 400 needs to be displayed on window 402 without having to be concerned with the stereoscopic vision for left and right eyes typical of HUD/HMD devices as in
Again the above applications are possible if a quick and accurate estimation of the changing pose of device 414 and its inside-out camera(s) (not shown) can be made while the user moves the device around object 410. Such quick and accurate estimations of pose in a variety of settings are thoroughly taught in U.S. Pat. No. 7,826,641, U.S. Pat. No. 7,961,909, U.S. Pat. No. 8,553,935, U.S. Pat. No. 8,897,494, U.S. Pat. No. 9,235,934, U.S. patent application Ser. No. 14/992,748, U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856 and U.S. patent application Ser. No. 14/926,435.
As mentioned earlier, the window method adopted by above and other related embodiments simplifies the projection requirements of the AR/MR scene. It is now enough to project a single scene rather than two different stereo projections, one suited for each eye. This stereoscopic projection required for human binocular vision is especially important for head-worn devices such as HUD/HMD.
Table 502 has two reference points 504, 506 and a reference edge 508 that help recovery of pose of user 202 wearing glasses 152 as taught in the above mentioned references. As in previous embodiments, glasses 152 have two inside-out cameras 153A and 153B, although having two (or more) cameras is not a requirement in order to accrue the benefits of the instant invention. In other words, eyeglasses can have only one inside-out camera as well. As shown in
Now let us see what happens in order to resolve the parallax or optometry problem of human binocular vision explained above. For this, let us first see
Those skilled in the art will understand that vergence is the simultaneous movement in opposite directions of both eyes to obtain or maintain single binocular vision. When the two views corresponding to the two eyes are slightly mismatched as in
Finally, if the optometry/parallax problem is addressed correctly, then the user would see a realistic and natural image for a distinct virtual wine glass 510 as shown in
Furthermore, environment 600 viewed by user 202 through and on windscreen 604 also has one or more virtual objects. One such virtual object 610 is shown, which is a road-sign. In addition to virtual object 610 there may be a virtual tachometer or other gauge or readings projected on windshield 604 of car 602 by the projection mechanism involved (not shown) of the application. Such projection mechanism may include appropriate hardware and software technologies available in the art for rendering images on windscreen 604. Alternatively, or in addition, the whole or part of the dashboard of car 602 may be virtual.
In the above embodiment, it is very useful if virtual road sign 610 or other virtual objects as desired e.g. various dashboard gauges, moved on windscreen 604 according to the movement of user's head. That way, the important safety and driving information is always available to the driver in his/her view, wherever on windscreen 604 he/she may be focusing. This is even more important for pilots who would like their flight data to be displayed onto the windscreen/windshield in concert with the movement of their head.
Of course, the above is easily accomplished by a timely and accurate recovery of the pose of user's eyes/head, utilizing the inside-out cameras of eyewear 152. Based on the changing pose, the position of the virtual object(s) can be changed on the viewing mechanism such as windscreen 604 of
In a close variation of the above embodiment, the projection mechanism does not physically display/project the AR/MR scenes onto windscreen 604 but rather the AR/MR images are projected only onto the lenses/optics of eyeglasses 152 worn by user 202. In other words, the user views windscreen 604 through the viewing optics of his/her eyeglasses onto which the projection mechanism projects virtual road sign 610. Obviously, the user further sees real objects 606, 608 (and others if present) through the same viewing optics onto which virtual object 610 (and any other virtual objects if present) are projected. It is also conceivable to have a combination of the above two variations.
In any case, as in prior embodiments, in order to provide a pleasant/comfortable and useful projection of environment 600 whether onto eyeglasses 152 or onto windscreen 604, the pose of user's head or glasses 152 with cameras 153A-B must be known. Based on that pose, as elaborated in the above examples, the appropriate alterations/corrections to virtual object 610 and other virtual objects if present, can be made.
One such image/scene adjustment or correction can be the resolution of the optometry or the parallax problem. For the automotive embodiments of
Specifically, in
According to the present invention, the appearance of one or more virtual objects displayed/projected on a viewing optics or mechanism is altered or modified according to a property (or properties) of an inside-out camera utilized by the system. Preferably, the property is the pose (position and orientation, also sometimes referred to as the extrinsic parameters) of the camera. The inside-out camera is utilized to capture the reference objects (points, edges, etc.) in the environment, based on which the pose of the camera is estimated, as per the above references. Preferably still, the property is a homography induced by some surface in the real scene. The homography implicitly conveys the pose of the camera.
Recall that in an AR/MR system, the scene/images captured by the inside-out camera are overlayed by one or more virtual objects. In a pure VR environment, the entire scene created for the user and displayed on the viewing optics/mechanism is virtual. In this case, the user is transposed to a reality that is artificial and synthesized. This is achieved by blocking the user's view of the surroundings with closed goggles/glasses/visors that also serve as displays for images rendered by a computer. The user feels immersed or ‘present’ in the synthesized reality if the rendered virtual scene reacts to the user's actions in the same manner the eye view of a natural scene would do.
Therefore, in a highly advantageous embodiment, the above alteration/modification of one or more virtual objects is done so as to reinforce this sense of “presence” of the user in the VR environment. This reinforcement of the presence of the user manipulates the scene(s)/object(s) viewed by the user and their reactions to user actions and movements, so as to make them appear and feel as desired to enhance the sense of presence. One requirement for such manipulation could be to make the VR scene/object(s) appear and react to user's motion as naturally as possible compared to how they would appear and react if they were real.
In an AR/MR environment, user's view of the natural environment is not blocked but rather captured by a camera and then ‘augmented’ by the layering one or more virtual objects on/in it. This is typically achieved by a see-through display of some type, be it goggles, contact-lenses, car windshields, or display windows/screens. A computer renders the image of one or more virtual objects in such a way that the user believes that the object(s) (rather than the user himself/herself as in the case of VR) are part of the actual surroundings. The virtual item(s) feel “present” if the rendered images for the user react to the user's actions in the same manner as natural item(s) would.
Therefore, in another advantageous set of embodiments, the above alteration/modification of one or more virtual objects is done so as to reinforce this sense of presence of the object(s) in the AR/MR environment. This reinforcement of the presence of the item(s) or virtual object(s) involves manipulating them to react to user actions and movements, such that they appear and feel as natural objects to the user. The manipulation of item(s)/object(s) may also be to satisfy any other requirement specific to the application at hand for reinforcing the sense of presence of the item(s).
Of course a critical aspect of the reinforcement of the sense of presence, whether it be that of the user in a VR, or of one or more objects in AR/MR, is the knowledge of the pose of the user and/or the camera. Based on the changing pose of the user/camera, the objects in the VR/AR/MR can be manipulated to appear as they would naturally to the eyes.
In still other related embodiments, the above alteration/modification of the one or more virtual objects is done so as to apply a certain texture or color to the virtual object(s). This could be used in AR/MR applications where a red color or prickly texture can be overlaid on top of an object in response to the user getting too close to the object(s) (or the object(s) suddenly becoming dangerous). Similarly, a virtual obstruction can be used to cue the user to avoid an area or path in an AR/MR application.
Now let us understand the technical mechanisms typically involved in the manipulation of the virtual objects rendered for the user in the above embodiments. As already stated, the rendering is done by a projection mechanism that typically renders the scene/object(s) on some viewing mechanism/optics. Such viewing mechanism/optics may involve viewing lenses of eyewear, or display windows/screen of an electronic device (e.g. a tablet). Those skilled in the art will understand that in 3D computer graphics, the rendering pipeline or graphics pipeline refers to the sequence of steps/stages that are required to create a 2D raster representation of a 3D scene/image.
Once a 3D model of an object has been created, the graphics pipeline is the process of rendering that 3D model onto a display. The following few paragraphs in relation to
Output of one stage/step is fed as input to the next stage. A vertex has attributes such as position in (x, y, z) coordinates, color (RGB or RGBA), vertex-normal (nx, ny, nz) and texture. A primitive is made up of one or more vertices. The rasterizer raster-scans each primitive to produce a set of grid-aligned fragments, by interpolating the vertices. Vertex processing shown in
Model, view and camera transformations are done at the vertex processing stage. Model transformation refers to the arrangement of objects within the synthesized scene or world. The view transformation refers to the position and orientation of the view that is to be presented to the user. Camera transformation refers to the (virtual) lens parameters through which the scene is to be visualized. All these transformations are programmable as shown in
Continuing further in
After rasterization, fragment processing also performs texturing of each fragment, lighting and fog effects, fragment culling tests (such as scissor test, alpha test, stencil test and depth buffer test), and finally pixel-based operations (such as blending, dithering, logical operations and bit-masking). All these operations are either programmable or configurable.
After the above primer on 3D graphics rendering pipeline, let us now turn our attention to the relevant embodiments of the invention. A person skilled in the art can readily see that the appearance of one or more virtual objects in the above embodiments, can be changed in numerous ways given the versatility of the graphics rendering pipeline. Therefore, in another set of highly advantageous embodiments, the alteration/modification of the one or more virtual items in the above embodiments, entails changing one or more configurable or programmable parameters of a graphics rendering pipeline.
The configurable and programmable parameters of a graphics rendering pipeline have already been introduced above and are associated with configurable and programmable functions/boxes shown in
As already taught above, the alteration of one or more virtual objects is done based on one or more properties of the inside-out camera of the system. One such property can be a pose of the camera. This use-case makes it highly applicable to VR/AR/MR environments. However, many other properties utilizing the inside-out camera can be used. These properties can be recovered, reconstructed or measured from the output of the inside-out camera. A non-exhaustive list includes: parallax, image sharpness, lens distortion, image blur or defocus, vignetting, lens flare, brightness, image texture, image disparity, z-depth, optical flow, image noise or grain, lighting and shading, edge and corners, SiFT features, foreground silhouettes, occlusions, vanishing points, foci of expansion, motion blur and spatiotemporal intensity fluctuations.
We have already explained parallax above and provided several examples of how the projected image/scene may be altered to resolve the parallax or optometry problem (see.
Preferably, the property based on which the one or more virtual objects are altered, is image sharpness. In other words, based on the desired or required level of image sharpness of the output of the camera, the one or more virtual objects are altered accordingly. Preferably, the property is lens distortion. This embodiment may involve a defect in the lens of the inside-out camera that results in an image distortion of the AR/MR scene being viewed. This distortion is then compensated for virtually using graphics rendering techniques provided above so a corrected image/scene can be presented to the user.
In the present embodiment, lens distortion can also be a desirable effect to enhance the sense of space or vastness. In such case, the appearance of a virtual object can be appropriately distorted as a result of changing the camera viewpoint. In a variation of this embodiment, a drop in image sharpness may be the result of fog or haze in the real scene. The sharpness of the virtual object can then be similarly reduced to match the perceived physical reality.
Preferably, the above mentioned property of the inside-out camera is an image blur or defocus that needs to be corrected/compensated for. Alternatively, this property may be used to have the appearance of the virtual object appropriately match the blur or defocus conditions. Several examples of this were provided above in relation to resolving the optometry or parallax problem. In addition, the image blur or defocusing may happen due to any number of other reasons, including the optical properties of the lens(es). In any case, these may be corrected for the user by employing the techniques provided herein.
Preferably, the above mentioned property of the inside-out camera based on which one or more virtual objects are altered, is vignetting. In photography and optics, vignetting is a technique for drawing attention to the center by reducing the image's brightness or saturation at the periphery compared to the image center. Therefore, based on the vignetting required for our VR/AR/MR scene, the virtual object(s) may be altered as desired. For example, if a virtual object is at the periphery of the scene it may be intentionally dimmed to draw more attention to the center.
Preferably, the above mentioned property of the inside-out camera is lens flare. Lens flare is the light scattered in the system through usually (unwanted) image formation mechanisms. These can be internal reflection and scattering from inhomogeneities in the lens material. However, a lens flare may be used deliberately to invoke a sense of drama. It may also be added to an artificial or augmented image to give it a sense of realism—implying that the image is an un-edited original image of a real-life scene. Therefore, in this embodiment, lens flare may be used as the basis for the alteration/modification of the one or more virtual object(s) in the VR/AR/MR scene. Reasons for doing this alteration may be to reduce the effects of unwanted lens flare, or to deliberately enhance the effect of lens flare for dramatic or real-life effects.
Preferably, the above mentioned property of the inside-out camera is brightness. Depending on the brightness level of the scene captured by the inside-out camera, a variety of appropriate alterations to the virtual object(s) may be warranted. For example, if the image is bright, brighten the virtual object(s) also, and vice versa.
Preferably, the above mentioned property of the inside-out camera is image texture. As understood in the art, an image texture is a set of metrics in image processing for quantifying the perceived texture of an image. Image texture provides information about the spatial arrangement of color or intensities in an image or selected region of the image. Thus based on the texture of the image/scene of the inside-out camera, appropriate alterations to the virtual object(s) may be warranted. One example would include retexturing the virtual object according to the texture of the rest of the image. One could texture the virtual object to match or contrast with the rest of the scene/image.
Preferably, the above mentioned property of the inside-out camera is binocular disparity, which refers to the difference in image location of an object seen by the left and right eyes, resulting from the eyes' horizontal separation (parallax). It is used by the brain to extract depth information from the two-dimensional retinal images in stereopsis. Therefore, objects or images may be virtually manipulated to produce the desired level of binocular disparity. As would be obvious from above, one reason for doing that may be to provide stereo vision for the user.
Preferably, the above mentioned property of the inside-out camera is z-depth. A common use of this property is for depth keying, which consists of grouping pixels based on their relative distance to the background. Thus, this property can be employed in a number of ways to alter the virtual images of our VR/AR/MR scene. The virtual object (or parts of the virtual object) can be rendered translucent over pixels with a low key (i.e., further into the background), or opaque over pixels with a high key (i.e., closer to the foreground).
Preferably, the above mentioned property of the inside-out camera is optical flow which is the pattern of apparent motion of objects, surfaces and edges in a visual scene, caused by the relative motion between an observer and the scene. The observer can be a person or a camera. As an example, a VR/AR/MR scene in the present embodiment may be manipulated to provide an illusion of movement.
Preferably, the above mentioned property of the inside-out camera is image noise. Image noise can take many forms, and is random variation of brightness or color in images. Usually it is an aspect of electronic noise or the ‘graininess’ of the film. Image noise is an undesirable by-product of image capture that adds spurious and extraneous information. Hence an example use-case would be to virtually manipulate the VR/AR/MR scene to counter image noise, or in an alternative scenario to enhance the noise for any reasons.
Preferably, the above mentioned property of the inside-out camera is shading (or conversely lighting). Shading means depicting depth perception in 3D models or illustrations by varying the levels of darkness/shading. Thus in the present embodiment, the VR/AR/MR scene may be altered based on the shading/lighting requirements of the scene. An example could use applying color/media to the image more densely or darkly for areas that should be perceived to be dark, and applying the colors/media lightly to areas that should be perceived to be lighter.
Preferably, the above mentioned property of the inside-out camera is one or more edges in the VR/AR/MR scene. Based on the edges any type and number of manipulations/alterations to the scene/objects may be desired. In a similar embodiment, the above mentioned property of the inside-out camera is one or more corners in the VR/AR/MR scene. Based on the corners any type and number of manipulations/alterations to the scene/objects may be desired. Examples of these include varying the details of the scene/objects in order to conform, align or to contrast with the edges and corners of existing objects in the scene.
Preferably, the above mentioned property of the inside-out camera is Scale-invariant Feature Transform (SiFT or SIFT) features. SIFT is an algorithm of computer vision to detect and describe local features in images. Thus in the present embodiment, a VR/AR/MR scene may be manipulate/altered according to the features extracted by SIFT. Examples of such manipulation/alteration include varying the contrast of the scene/objects according to high-contrast edges and corners detected by SIFT. Since SIFT is useful in image recognition, there are numerous possibilities of using computer vision to study the VR/AR/MR scene thusly manipulated in the present embodiment based on SIFT features.
Preferably, the above mentioned property of the inside-out camera is foreground silhouettes. Thus in this embodiment, VR/AR/MR scene may be manipulated to accentuate, highlight or annotate the silhouettes of one or more objects in the scene. Examples of this manipulation include increasing the lighting of objects surrounding the silhouetted object or darkening the silhouetted object compared to the surroundings.
Preferably, the above mentioned property of the inside-out camera is vanishing points. Recall from perspective geometry that a vanishing point is a point in the picture plane that is the intersection of the projections of a set of parallel lines in space on to the picture plane. The classic example is the point where railway tracks appear to intersect in the distance in a picture viewed from the front. Thus in the present embodiment, depending on the vanishing points present in the VR/AR/MR scene, certain manipulation/alterations of the scene/objects may be warranted. Examples of such manipulations include scaling the object according to how far/close those vanishing points need to be.
Preferably, the above mentioned property of the inside-out camera is foci of expansion. When the camera/observer is moving forward, the corresponding optical flow contains a focus of expansion. It is a point from where the objects in the image appear to be expanding. A classic example is when the camera is moving inwards towards a point in the scene, the objects around it expand or become bigger and appear closer and then eventually disappear out of bounds at the periphery. The center or the point around which this expansion occurs is the focus of expansion. It is the point towards which the camera is moving inwards in the above example.
Thus in this embodiment, depending on one or more of such foci of expansions in the scene based on the camera movement, virtual objects/images in the VR/AR/MR scene may be altered or manipulated. Examples of such manipulations/alterations include enlarging or expanding the objects around the focus of expansion so that the movement appears real-world and realistic.
Preferably, the above mentioned property of the inside-out camera is motion blur, which is the apparent streaking of rapidly moving objects in a still image or in a sequence of images. It occurs when the image being projected or recorded changes during a single exposure, either due to rapid movement of objects and/or extended length of the exposure. Thus in this embodiment the virtual/augment/mixed scenes or objects are manipulated according to the presence/absence of the motion blur. For example, the manipulation may involve blurring the virtual object(s) in the scene so they appear to match the amount of motion blur of other real objects in the scene. Still in another example, the virtual object(s) may be blurred so as to create a perception of the movement of the objects.
Preferably, the above mentioned property of the inside-out camera is spatiotemporal intensity fluctuations. In this embodiment, the manipulation or alteration of virtual object(s) may be warranted due to the intensity changes in space and time. An example scenario includes changing the light intensity of the virtual object(s) to account for changes in light intensity on other objects in space and time.
As will be evident, the above examples offer a vast number of possibilities for the property (or properties) of inside-out camera based on which image alteration of a VR/AR/MR scene is performed according to the invention. Furthermore, the alteration or manipulation of the scene could be simple or complex. Preferably, the alteration is merely a change in the position of one or more virtual objects. Among the many possible choices for the reasons for such alteration include improving stereoscopic vision, reinforcing the sense of presence for the user and/or the object(s), or providing a more comfortable/pleasant and natural experience to the user.
Preferably, the alteration is consonant to a movement of the user. Among the many possible choices for the reasons for such alteration include reducing motion sickness for the user by keeping the projected image/scene consonant with the voluntary or involuntary movements of the user. Under such circumstances it is advantageous for the movements of the user to be constrained. Such constraint on the motion of the user/camera results in a reduced homography associated with the changing pose of the user/camera.
A reduced homography is preferably possible because of the presence of structural uncertainties in the optics of the viewing mechanism, or because of structural redundancies caused by the conditioned motion of the viewer. The reduced homography employs a reduced representation that is much more efficient to compute for estimating the user/camera pose, than regular homography. For a detailed treatment on pose recovery using homography, the reader is referred to U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856 and U.S. patent application Ser. No. 14/926,435.
In an interesting embodiment of the invention there are two users of the system. First user views the projected/displayed VR/AR/MR scene as before, while the second user is associated with the inside-out camera. Preferably the second user carries or wears the inside-out camera. This way the scene that is being projected and altered for the first user is actually from the perspective of the second user. An example use-case for such an application is a video game where the user plays a game (or otherwise uses the system) from the perspective of the second user. Note here that although the above explanation uses a first user in the singular, there could be any number of such users present that view the projected/displayed VR/AR/MR scene from the perspective of the second user/avatar.
In alternative embodiments, there is only one real user (or set of real users), while the virtual object is an avatar of a second user thereby giving the illusion of the presence of a second user. The virtual user or avatar is then manipulated according to the one or more properties of the inside-out camera per above teachings. The inside-out camera is preferably mounted on the first user. In a variation, the virtual object could also be a tool or an implement which is altered or manipulated based on the pose of the inside-out camera.
The viewing optics or mechanism may preferably be integrated with the inside-out camera. This would be typical of a set of VR/AR/MR eyeglasses or goggles or visors where the left and right optics through which the user views or sees the environment are the same respective left and right optics which are used by the inside-out camera/cameras as its/their lenses. Alternatively, the viewing optics or mechanism may just be affixed to the inside-out camera. Examples of such a setup include eyeglasses 152 of
As already described, the viewing optics or viewing mechanism may employ a display unit such as a screen/window. Devices such as a smartphone, a tablet as well as HUD/HMD/eyeglasses 150 of earlier embodiments would be examples of that. Preferably the projection mechanism used to project/display the VR/AR/MR scenes/objects is integrated with the display unit. This would be typical of a smartphone or tablet, although many other possibilities exist. For example, retinal projections onto the user's eyes would utilize the natural lenses inside user's eyes as the viewing optics through which he/she views the environment and onto which the alteration to the virtual object(s) occurs. Alternatively, viewing optics may be just attached, connected or affixed to the display unit.
As extensively taught in the above embodiments, the viewing optics may be replicated/duplicated for producing a stereo vision for the user. Recall the discussion around the stereoscopy, optometry or the parallax problem. In fact, the viewing optics may take many other forms, e.g. a telescope, binoculars, etc. In other words, the user may view a VR/AR/MR environment through the optics of a telescope that has a projection mechanism for projecting/displaying virtual object(s) on its optics. Same principle applies to binoculars.
In another set of highly advantageous embodiments, the system utilizes a control device or a controller for controlling the one or more virtual objects in the VR/AR/MR scene. Preferably, the controller is a wearable device. Preferably the controller is used to control the alterations/modifications to the virtual object(s) in the scene. Examples of such a control device or controller include a joystick, a game controller (e.g. Nintendo Wii), a touch sensor (e.g. Apple Magic Trackpad, Lenovo K5923 Multi-gesture Touchpad), a gesture sensor (e.g. the ones used in games and smartphones), a digital pen (e.g. a stylus), a proximity sensor (e.g. a capacitive, photoelectric or inductive sensor), a vicinity sensor (e.g. a sensor using radio frequency identification (RFID) technology), an electromagnetic sensor, an inertial sensor (e.g. an accelerometer or a vibration sensor) or one of the many types of motion sensors.
In similar embodiments, instead of a control device, the system simply uses an auxiliary sensor for controlling the appearance/modification of the one or more virtual objects. Preferably, the auxiliary sensor is an optical sensor, an inertial sensor (e.g. a gyroscopic sensor, or an accelerometer), a magnetometer, an optical flow sensor, a displacement sensor, an acoustic sensor, a Radio Frequency (RF) sensor. All the above sensors and sensor technologies are well understood in the art and they will not be delved further into this specification.
There are many interesting applications conceivable for using a control device or an auxiliary sensor in the above embodiments for the alteration/modification of the one or more virtual objects/items. Examples include game controllers, e.g. joystick or other types of input devices for effecting alterations to the appearance of the virtual object(s). Other examples include instrumenting the gamer with the auxiliary sensor so that his/her actions, such as movements or gestures or sounds, may be tracked/measured by the sensor and adjustments made to the one or more virtual objects accordingly.
In an interesting variation of the above embodiments, the user of a device views the VR/AR/MR scene from a device viewpoint (instead of a user viewpoint as in earlier embodiments). As before, a projection mechanism is employed to alter the appearance of one or more virtual images/objects in the VR/AR/MR scene as seen by the user from the device viewpoint. There are many interesting use cases for such a scenario.
One such use case is illustrated in
In the embodiment shown in
Alternatively, inside-out camera(s) can be separately mounted on, affixed to or in some other way operably connected to device 652. Furthermore, as before, there can be one or more inside-out camera(s) on device 652 providing either monocular, binocular/stereoscopic or other types of vision of environment 650 to user 202. Environment 650 contains a virtual object 656 and potentially other virtual and real objects which are not explicitly shown in
Obviously the important difference in the present embodiments is that instead of viewing the environment from the viewpoint of the user himself or herself, the environment as projected/displayed by the projection mechanism is viewed from the viewpoint of a device. This detachment of the viewpoint and associated viewing optics/mechanism from the user himself/herself provides a lot of interesting applications. One such application is shown in
However, many other applications of the present embodiments are possible. For example, the device can be a robot controlled by a control mechanism such as a computer software/hardware or by the user himself/herself either manually or through an electronic control mechanism. The device can be an instrument or an implement or a tool controlled by the user, typically from some distance. Still possibly, the device may be any remotely controlled automotive equipment, such as a car, train, truck, etc. Alternatively, device 652 may be autonomous, or semi-autonomous with little or no control over it exercised by user 202.
In alternative variations, a light-field camera (such as a Lytro camera or a Pelican camera) can be used to collect composite optical information and permit rendering from many vantage points within a certain volume. A light-field camera is advantageous for pose recovery using the techniques of the above provided references, because it captures intensity as well as the direction of the light emanating from the reference points in the environment. A light-field camera is typically more resource intensive than a conventional camera in terms of power and computation requirements. Hence, appropriate resources need to be made available to the camera, whether it is placed on a drone/device or otherwise made available to the viewer, such as in an HUD/HMD.
As used in the present variations, a device in general may fall into two broad categories. It is either an implement/tool operated by the user either directly (e.g., by hand) or autonomously, or the device is a wearable device, which is carried or worn by the user. In the former category of manipulated devices/items, the device may be attached to a mechanical linkage having up to six degrees of freedom that allow total freedom of motion or a constrained freedom of motion. The device may further be wireless or attached by a flexible tether (with or without stress relief of torque relief).
The category of implements/tools generally includes wands, flying drones, remotely controlled cameras, portable phones, portable electronic devices, medical implements, digitizers, hand-held tools, gaming controls, gaming items, digital inking devices, pointers, remote touch devices, TV remotes and magic wands. In terms of use-cases, the manipulated device/item is a portable phone that is used to control a user device which is a game console, a television, a stereo, an electronic picture frame, a computer, a tablet, an RF transmitter unit, a set-top box, a base station, a portable user device having a display, a non-portable user device having a display, an appliance or the like.
The category of wearable devices/items generally includes items affixed on headgear, on glasses, on gloves, on rings, on watches, on articles of clothing, on accessories, on jewelry, on accoutrements and the like. Any of such wearable devices can be used to control a user device that is a game console, a television, a stereo, an electronic picture frame, a computer, a tablet, an RF transmitter unit, a set-top box, a base station, a portable user device having a display, a non-portable user device having a display, an appliance or the like.
All other teachings of the earlier embodiments including where the VR/AR/MR environment was seen from a user viewpoint, also apply to the present embodiments where the VR/AR/MR environment is viewed from a device viewpoint. For example, one or more virtual objects/images layered on the environment by the projection mechanism, as seen by the user from the device viewpoint can be altered or manipulated based on one or more properties of the inside-out camera(s) as taught earlier. The myriad of choices of such properties have already been taught above. Similarly, the many choices for the type of viewing optics/mechanisms, wearables, etc. have also been taught above. Furthermore, as before, projection mechanism may include a display screen unit or screen/window and the associated teachings of earlier embodiments apply. Still further, the various types of alterations/modifications to the scene and the motivations behind them as taught above, also apply to the present embodiments.
Preferably, the alteration/modification of the one or more virtual objects/images/scene is consonant to a movement of the device—notice the contrast to the earlier embodiments where the alteration was consonant to a movement of the user. However, again among the many possible choices for the reasons for such alteration include reducing motion sickness for the user by keeping the projected image/scene consonant with the voluntary or involuntary movements of the device. Under such circumstances it is advantageous for the movements of the device to be constrained. Such constraint on the motion of the camera results in reduced homography associated with the changing pose of the device/camera.
The reduced homography employs a reduced representation that is much more efficient to compute for estimating the device/camera pose, than regular homography. For a detailed treatment on pose recovery using homography, the reader is referred to U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856 and U.S. patent application Ser. No. 14/926,435.
In another set of embodiments of the instant invention an optical sensor is used for imaging space points of a reality that is viewed by a viewer. The space points would preferably be non-collinear. The optical sensor may be an inside-out camera as taught earlier. Again, there may be multiple optical sensors employed for producing stereoscopic vision for the viewer, or for other reasons pertinent to the application. Then a mechanism is used to generate one or more virtual objects/items that are layered on the reality viewed by the viewer. Such a mechanism can employ graphics rendering pipeline capabilities and embody the projection mechanism of earlier examples. Then, utilizing the optical sensor(s), the system tracks the movement of the viewer and the above mechanism modifies the one or more virtual objects/items on the reality according to that tracking.
As taught earlier, one reason for tracking the movement of the viewer may be to improve his/her viewing experience, such as, by reducing the motion sickness associated with his/her voluntary/involuntary movements. Still other reasons may include changing his viewing experience according to specific medical, psychological, mechanical or other needs of the application. The optical sensor(s) of this embodiment may be worn in a HUD/HMD or other gear of the earlier embodiments.
The modification of the one or more virtual items may also be according to one or more properties of the optical sensor. Accordingly, the types of properties of the inside-out camera taught earlier, and the teachings of the prior embodiments apply to the present variation(s) employing optical sensor(s) also. In another embodiment, the inside-out camera is attached to an autonomous, or semi-autonomous device, and the scene is displayed to a sentient being (e.g. human). In this embodiment, the scene is displayed and updated with virtual objects for the human but from the viewpoint of the autonomous or semi-autonomous device (e.g. robot, drone, etc.).
In another set of highly interesting embodiments, an immersive sports experience may be provided for the user. In such embodiments, the user with HUD/HMD wearables or some other appropriate viewing and projection mechanisms, that may be the same as or similar to earlier embodiments or entirely different designs, may be virtually transported into a sports event. The sports event or game may be a Soccer, NFL, MLB, NBA, NHL, etc. game, and the user may be able to interact with the virtual objects present in the VR/AR/MR scene/game.
The experience in above embodiments could be an AR/MR or VR experience as taught earlier. The virtual objects may be completely fictitious or a rendition of actual objects and/or players of an actual game/team. It is easy to extend these embodiments to fantasy team, and further to completely fictitious objects related to sports or otherwise, that one may be able to interact/play with in the VR/AR/MR embodiments afforded by the instant invention. Indeed, many other applications are conceivable according to the teachings and within the scope of the invention.
The methods of the invention further provide the steps required to layer one or more virtual objects/items onto a VR/AR/MR scene. The scene is viewed by a viewer from a viewer viewpoint using a viewing optics/mechanism. The viewer can be a machine such as a robot, a manipulated/controlled tool or implement, or an artificial agent. Alternatively, the viewer can be a sentient being such as a human, or an animal. The layering is performed by a projection mechanism that is capable of displaying the scene to the viewer using any number of mechanisms available in the art.
The methods further provide that the appearance of the one or more virtual objects/images in the scene viewed by the viewer may be altered based on one or more properties of an inside-out camera. Teachings of the earlier embodiments, including the types of properties of the inside-out camera, the choices of how the inside-out camera may be connected to capture the real environment, the types and choices of projection mechanism, viewing optics, types and choices of alterations/modifications to the scene, etc. still apply.
The methods provide that the alteration of the one or more virtual objects/images/scene is preferably consonant to the motion of the viewer. As taught earlier, a constraint on such a motion results in homography that only requires a reduced representation and is more efficient to compute than a full homography requiring a regular/full representation. Such a reduced and computationally efficient representation for the homography is possible due to structural uncertainty present in the viewing optics/mechanism.
In addition, the reduced representation of the homography may also be possible due to structural redundancy caused by the conditioned motion of the viewer. For a full treatment of reduced homography and pose recovery in the presence of structural uncertainty and structural redundancy, and the associated topics, the reader is referred to U.S. Pat. No. 8,970,709, U.S. Pat. No. 9,189,856, U.S. patent application Ser. No. 14/926,435.
It will be evident to a person skilled in the art that the present invention admits of various other embodiments. Therefore, its scope should be judged by the claims and their legal equivalents.
This application is a Continuation-in-part of U.S. patent application Ser. No. 14/965,544 filed on Dec. 10, which is a Continuation-in-part of U.S. patent application Ser. No. 13/199,239 filed on Aug. 22, 2011 now U.S. Pat. No. 9,229,540, which is a Continuation-in-part of U.S. patent application Ser. No. 10/769,484 filed on Jan. 30, 2004 now U.S. Pat. No. 8,542,219. The above referenced U.S. patent application Ser. No. 14/965,544 filed on Dec. 10, 2015 is also a Continuation-in-part of U.S. patent application Ser. No. 14/551,367 filed on Nov. 24, 2014 now U.S. patent Ser. No. 9,235,934, which is a Continuation of U.S. patent application Ser. No. 13/889,748 filed on May 8, 2013 now U.S. Pat. No. 8,897,494, which is a Division of U.S. patent application Ser. No. 13/134,006 filed on May 25, 2011 now U.S. Pat. No. 8,553,935, which is a Division of U.S. patent application Ser. No. 12/586,226 filed on Sep. 18, 2009 now U.S. Pat. No. 7,961,909, which is a Continuation-in-part of U.S. patent application Ser. No. 12/584,402 filed on Sep. 3, 2009 now U.S. Pat. No. 7,826,641, which is a Continuation-in-part of U.S. patent application Ser. No. 11/591,403 filed on Oct. 31, 2006 now U.S. Pat. No. 7,729,515 which claims priority from U.S. Provisional Patent Application No. 60/780,937 filed on Mar. 8, 2006. All the above numbered applications are incorporated by reference herein in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| 60780937 | Mar 2006 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13134006 | May 2011 | US |
| Child | 13889748 | US | |
| Parent | 12586226 | Sep 2009 | US |
| Child | 13134006 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13889748 | May 2013 | US |
| Child | 14551367 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14965544 | Dec 2015 | US |
| Child | 15165955 | US | |
| Parent | 13199239 | Aug 2011 | US |
| Child | 14965544 | US | |
| Parent | 10769484 | Jan 2004 | US |
| Child | 13199239 | US | |
| Parent | 14551367 | Nov 2014 | US |
| Child | 14965544 | US | |
| Parent | 12584402 | Sep 2009 | US |
| Child | 12586226 | US | |
| Parent | 11591403 | Oct 2006 | US |
| Child | 12584402 | US |
