Patent Grant
9164581
Sensing technologies are available which make it possible to manipulate computer generated content via simple gestures in front of a display screen. Also, display technologies are available that allow a user to view the computer generated content in 3 dimensions (3D). Current display systems with gesture based control systems are often problematic in that the user's hands tend to block the user's view of the content being manipulated on the display screen.
The figures depict implementations/embodiments of the invention and not the invention itself. Some embodiments are described, by way of example, with respect to the following Figures.
The drawings referred to in this Brief Description should not be understood as being drawn to scale unless specifically noted.
For simplicity and illustrative purposes, the principles of the embodiments are described by referring mainly to examples thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one of ordinary skill in the art, that the embodiments may be practiced without limitation to these specific details. Also, different embodiments may be used together. In some instances, well known methods and structures have not been described in detail so as not to unnecessarily obscure the description of the embodiments.
In one embodiment of the present invention, the display screen is comprised of a transparent material so that a user positioned in front of the display screen can easily see through it. Viewpoint assessment sensors are positioned to capture information about a user and his position in front of the display screen. In addition the display system includes object tracking sensors positioned to track a physical object positioned behind the screen and manipulated by the user. The object behind the screen being manipulated by the user can be the user's own hand or a physical object being held or otherwise manipulated by the user. Responsive to data output from the viewpoint assessment sensors and the object tracking sensors, an image is generated on the display screen that overlays or augments objects placed behind the screen. These overlaid images are registered with the objects positioned behind the screen based on the user's viewpoint (eye position). The sensed position of the user's hands relative to the objects can be used to enable manipulation of the overlaid images on the display screen.
Referring to
In addition, the display system 100 includes one or more object tracking sensors 148a, 148b covering the space behind the display screen to sense objects (including the user's hands) positioned behind the display screen.
Referring to
The display system includes a display screen 112 for displaying content. In one embodiment, the display screen is a transparent screen which allows a user interacting with the display to see objects positioned behind the display. In an alternative embodiment, the display screen is opaque but transparency is simulated by displaying a view of objects behind the display screen captured by a video camera that captures video behind the display screen. In another alternative embodiment, the display screen is capable of operating either in a first mode (a transparent screen) and a second mode (an opaque screen), and switching between the two modes based on system or user controlled instructions.
Different screen materials may be used to provide the transparent display screen 112. In one embodiment, the transparent display screen is a transparent OLED (organic light-emitting diode) screen. In an alternative embodiment, the display screen is comprised of transparent LCDs (liquid crystal display). However, the transparent LCD screen implementation assumes a well lit working environment. In a third alternative embodiment, the display screen is comprised of a partially diffusing material and the content is projected onto the partially diffusing material. Although many alternative implementations are possible, the transparent display screen operates so that objects positioned behind the display screen can be easily seen or viewed by a user positioned in front of the display screen. The transparent display allows the user to have a clear view of the objects behind the screen that are being manipulated in real time and to instantaneously see the effect of their manipulation on the object.
As previously described, in one embodiment the display screen is capable of switching between display modes. For example, the display screen would be transparent in one mode and in a second, opaque, mode the display screen would operate as a conventional display screen. A display screen capable of switching between two modes could be implemented in one embodiment as a transparent OLED screen with a liquid crystal shutter positioned behind the OLED screen. In a first mode the transparent LED screen would be transparent (liquid crystal shutter open) and in a second mode the screen would appear to be opaque (liquid crystal shutter behind the display screen closed) for use as a conventional display.
In one embodiment, the display screen 112 includes a frame 154 or bezel, typically located around the periphery of the display screen. Referring to
Because in the embodiment described with respect to
Referring to
Although multiple viewpoint assessment and object tracking sensors are shown around the boundary of the display screen in the embodiment shown in
Although the term “user” is used throughout, when used with respect to the viewpoint assessment sensors, the term user refers to the features of the user (eyes, head and shoulders, face, etc.) necessary for the viewpoint assessment component, to determine the user viewpoint. With reference to the object tracking device, the object being tracked is a physical object capable of being manipulated by a user. The tracked object being manipulated by the user behind the display screen, can be the user's own hand or alternatively, another physical object (e.g. block, pliers, etc.) that the user is directly or indirectly manipulating. For example, a user could be directly manipulating a physical object (the block shown in
The display screen can be the size of a large desktop monitor, as shown in the
Referring to
Although different definitions of user viewpoint can be used, with respect to
In one embodiment, determining the user's eye position is found using IR (infrared) illumination. In this implementation, an IR illumination source floods IR light in the vicinity of the user. In this case, the viewpoint assessment sensors 140a, 140b are IR sensitive devices, typically an IR sensitive camera. IR light is reflected off the user's retina and the IR sensitive camera (the viewpoint assessment sensor) captures the user's eye location relative to the display screen and the object positioned behind the display screen. Based on the location of the user's eyes, the viewpoint assessment component determines the user's viewpoint with respect to the screen and the object.
In an alternative embodiment, information about the user's eye position (used to determine the user's viewpoint) can be achieved using face detection software 180. In this case, the viewpoint assessment sensors 140a, 140b can be simple cameras. Based on the feature detection software 180 in the viewpoint assessment component 116, the eye position of the user is located. Based on the eye position of the user, the user viewpoint can be determined.
In an alternative embodiment, some form of active tracking of the user's position is used. For example, the user could wear a device that is capable of being tracked in 3D space. In one implementation, retro-reflective markers could be applied to an object worn by the user (i.e. a headset, a pair of glasses, etc.). Based on the position of the tracked object, the viewpoint assessment component software 116 provides an accurate approximation of the user's eye location.
In another alternative embodiment, the eye location of the user can be approximated using depth cameras as the viewpoint assessment sensor 140a, 140b. In this implementation, a depth camera is used to locate the user (user's head and shoulder at a depth or depth range nearer to the depth camera) and the background (at a depth or depth range further from the camera). The viewpoint location can then be approximated as being at the center of the head shaped region in the depth image. Any of the above techniques listed in the preceding paragraphs can be combined to give a more accurate determination of the viewpoint location.
In one embodiment, the display is an auto-stereo display. In these displays, after determining the location of each eye, different views are provided for each eye. The display system thus needs to know the location of each eye, so that content can be generated in the right location for each eye. Some of the embodiments described above give the position for each eye directly, in those that don't the typical human inter-occular distance can be used to calculate eye positions from a single viewpoint.
Referring to
Although the object being manipulated by the user can be a physical or virtual object (an augmented image) being displayed on the display screen, the object being tracked behind the display screen by the object tracking sensors is a physical object. Further, a single or multiple objects may be tracked. For example, in the case of a user manipulating a virtual image as shown in
Although many types of object tracking sensors 148a, 148b can be used, in one embodiment the object tracking sensors are image capture devices. In one implementation, the image capture device (object tracking sensor) is a depth camera. For applications that require sensing gestures and objects in the volume of space behind the screen, a depth camera would be appropriate as it provides information regarding the shape and location of the object in 3D. For example, for the case where the tracked object is the user's hand, the depth camera would provide information regarding where the user's fingers are and the position of the user's fingers in relationship to the second tracked object (the rectangular cube.)
Although depth cameras may be used to provide information about the shape and location of the object in 3D space, other types of sensors and techniques for determining position and shape of the tracked object could be used. For example, sensors and techniques that are used for sensing gestures and object in front of the display screen, might also be applied for tracking objects behind the display screen.
In one embodiment, the objects to be tracked have a code embedded on the object or registration marks on the object and the object tracking sensor is a camera. For example, a barcode label could be attached to the tracked object 120. In this case, the camera capturing an image of the code or registration marks and based on the location of those marks, could also determine where the object is physically located. Further, software can interpret the code or markings to provide information about the object (i.e. size, shape, use instructions, etc.).
The display controller component 130 includes: a viewpoint assessment component 116, an object tracking component 124 and a display generation component 126. The display generation component 126 creates an overlaid image that is displayed on the transparent screen that provides a natural way to augment user interactions with real world objects (located behind the display screen). Because the display screen 112 is transparent, the user can easily view and interact with both the physical object 120 located behind the screen and the overlaid image 510 displayed on the transparent display.
The display system creates an “overlaid” image 510 on the display screen 112—where the overlaid image 510 is an image generated on the display screen that is between the user's viewpoint and the object 120 behind the screen that it is “overlaid” on. The overlaid image is dependent upon the user's viewpoint. Thus, the position of the overlaid image with respect to the object behind the display screen stays consistent even as the user moves their head and/or the object behind the display screen.
For the embodiments shown in
Step 430 shown in
Referring again to
In the embodiment shown in
Referring to
Referring to
The embodiment shown in
In one embodiment, the detailed 3D information about the object behind the screen (the user's hand) is provided by depth cameras. In other words, in this example the object tracking sensors 148a, 148b would be depth cameras. Since the sensors described above can provide three-dimensional information on the position and shape of the user's hand, the display system 100 can determine what parts of the computer generated content the user's hand should occlude and not display those parts on the overlaid image of the cube 610a. Thus, simulating occlusion from the point of view of the user.
Also provided to the display system 100 is a description of the overlaid image 184 that is shown on the display screen. In one embodiment, the description of the overlaid image 184 is a 2D description of the object available for user interaction or manipulation. In one embodiment, the description of the overlaid image 184 is a 3D description of the object available for user interaction or manipulation. In one example, the description of the overlaid image is provided by the display system and another example, the description of the overlaid image is provided by the user.
In one example, graphic rendering software 190 capable of rendering a 3D overlaid object on a 2D screen is included in the display generation component. Thus a 2D description of the overlaid image can be converted into a 3D overlaid object on a 2D display screen. Assume an example where the display system has a 3D description of the physical object behind the screen and a 3D description of the 2D overlaid image displayed on the display screen. Using the 3D information provided for the overlaid image, the display system can simulate/determine how the overlaid virtual image interacts with a physical object positioned behind the screen. From this simulation, determinations can be made about which areas of the virtual image and which areas of the physical objects are shown and which areas are occluded, as well as how the user's movements result in interacting with the virtual content (e.g. has the user grasped a virtual object, allowing them to move it).
As previously stated with respect to the image shown in
In this opaque screen implementation, the display screen attempts to simulate a transparent screen by capturing the view of the physical object 710a located behind the screen and displaying this content on the display screen 112. This captured image provides an approximation of the actual view that would be shown by a transparent screen, but with an offset viewpoint. Instead of being the captured image reflecting the user's view through the transparent screen, the viewpoint seen on the display screen is from the perspective of a camera 148c positioned on the back of the display screen.
In the embodiment shown in
For the opaque screen embodiment, at least one object tracking sensor is a video camera. In one embodiment (not shown), a single object tracking sensor is used. In this embodiment, the object tracking sensor is a depth video camera or other image capture device capable of (1) providing information about the object location (O) and (2) providing a video image of the area or scene behind the display screen. In another embodiment (shown in
In one embodiment, the image output to the display (the image transmitted to simulate the transparent screen) is modified. For example, if the sensor information about the user position in front of the display screen is available, the system could try to modify the image to more closely approximate the viewpoint of the user—instead of the viewpoint of the object tracking sensor 148c.
Steps 710 and 720 are similar to the steps 410 and 420 in the method 400. However, in the implementation described with respect to
Some or all of the operations set forth in the method 400 may be contained as utilities, programs or subprograms, in any desired computer accessible medium. In addition, the method 400 may be embodied by computer programs, which may exist in a variety of forms both active and inactive. For example, they may exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats. Any of the above may be embodied on a computer readable medium, which include storage devices and signals, in compressed or uncompressed form.
The computing apparatus 800 includes one or more processor(s) 802 that may implement or execute some or all of the steps described in the methods 400. Commands and data from the processor 802 are communicated over a communication bus 804. The computing apparatus 800 also includes a main memory 806, such as a random access memory (RAM), where the program code for the processor 802, may be executed during runtime, and a secondary memory 808. The secondary memory 808 includes, for example, one or more hard drives 810 and/or a removable storage drive 812, representing a removable flash memory card, etc., where a copy of the program code for the method 800 may be stored. The removable storage drive 812 reads from and/or writes to a removable storage unit 814 in a well-known manner.
Exemplary computer readable storage devices that may be used to implement the present invention include but are not limited to conventional computer system RAM, ROM, EPROM, EEPROM and magnetic or optical disks or tapes. Concrete examples of the foregoing include distribution of the programs on a CD ROM or via Internet download. In a sense, the Internet itself is a computer readable medium. The same is true of computer networks in general. It is therefore to be understood that any electronic device and/or system capable of executing the functions of the above-described embodiments are encompassed by the present invention.
Although shown stored on main memory 806, any of the memory components described 806, 808, 814 may also store an operating system 830, such as Mac OS, MS Windows, Unix, or Linux; network applications 832; and a display controller component 130. The operating system 830 may be multiparticipant, multiprocessing, multitasking, multithreading, real-time and the like. The operating system 830 may also perform basic tasks such as recognizing input from input devices, such as a keyboard or a keypad; sending output to the display 820; controlling peripheral devices, such as disk drives, printers, image capture device; and managing traffic on the one or more buses 804. The network applications 832 includes various components for establishing and maintaining network connections, such as software for implementing communication protocols including TCP/IP, HTTP, Ethernet, USB, and FireWire.
The computing apparatus 800 may also include an input devices 816, such as a keyboard, a keypad, functional keys, etc., a pointing device, such as a tracking ball, cursors, etc., and a display(s) 820, such as the screen display 110 shown for Example in
The processor(s) 802 may communicate over a network, for instance, a cellular network, the Internet, LAN, etc., through one or more network interfaces 824 such as a Local Area Network LAN, a wireless 802.11x LAN, a 3G mobile WAN or a WiMax WAN. In addition, an interface 826 may be used to receive an image or sequence of images from imaging components 828, such as the image capture device.
The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the invention. The foregoing descriptions of specific embodiments of the present invention are presented for purposes of illustration and description. They are not intended to be exhaustive of or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations are possible in view of the above teachings. The embodiments are shown and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents:
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US2010/053860 | 10/22/2010 | WO | 00 | 4/22/2013 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2012/054063 | 4/26/2012 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4997263 | Cohen et al. | Mar 1991 | A |
| 5252950 | Saunders et al. | Oct 1993 | A |
| 5266930 | Ichikawa et al. | Nov 1993 | A |
| 6481851 | McNelley et al. | Nov 2002 | B1 |
| 6710797 | McNelley et al. | Mar 2004 | B1 |
| 6767099 | Perkins et al. | Jul 2004 | B2 |
| 6771294 | Pulli et al. | Aug 2004 | B1 |
| 7177144 | Ha et al. | Feb 2007 | B2 |
| 7385600 | Marion | Jun 2008 | B2 |
| 7567436 | Jeong | Jul 2009 | B2 |
| 7685524 | Rekimoto | Mar 2010 | B2 |
| 7724511 | Jacobs | May 2010 | B2 |
| 7983920 | Sinclair, II | Jul 2011 | B2 |
| 8046719 | Skourup et al. | Oct 2011 | B2 |
| 20020044152 | Abbott et al. | Apr 2002 | A1 |
| 20030107643 | Yoon | Jun 2003 | A1 |
| 20030142068 | DeLuca et al. | Jul 2003 | A1 |
| 20030151726 | Perkins et al. | Aug 2003 | A1 |
| 20050041009 | Kuroda | Feb 2005 | A1 |
| 20050046953 | Repetto et al. | Mar 2005 | A1 |
| 20050083713 | Boks | Apr 2005 | A1 |
| 20050166163 | Chang et al. | Jul 2005 | A1 |
| 20060017654 | Romo | Jan 2006 | A1 |
| 20060108483 | Wolff et al. | May 2006 | A1 |
| 20070164988 | Ryu et al. | Jul 2007 | A1 |
| 20070200547 | Chen | Aug 2007 | A1 |
| 20070236485 | Trepte | Oct 2007 | A1 |
| 20070253600 | Furukawa et al. | Nov 2007 | A1 |
| 20070291008 | Wigdor et al. | Dec 2007 | A1 |
| 20080018555 | Kuo et al. | Jan 2008 | A1 |
| 20080024523 | Tomite et al. | Jan 2008 | A1 |
| 20080084395 | Dawson et al. | Apr 2008 | A1 |
| 20080088602 | Hotelling | Apr 2008 | A1 |
| 20080211813 | Jamwal et al. | Sep 2008 | A1 |
| 20080211832 | Kumon | Sep 2008 | A1 |
| 20080218434 | Kelly et al. | Sep 2008 | A1 |
| 20080284729 | Kurtenbach et al. | Nov 2008 | A1 |
| 20090135135 | Tsurumi | May 2009 | A1 |
| 20090288889 | Carlvik et al. | Nov 2009 | A1 |
| 20090298548 | Kim et al. | Dec 2009 | A1 |
| 20100007613 | Costa | Jan 2010 | A1 |
| 20100035658 | Lee | Feb 2010 | A1 |
| 20100045569 | Estevez et al. | Feb 2010 | A1 |
| 20100053151 | Marti et al. | Mar 2010 | A1 |
| 20100073404 | Brown et al. | Mar 2010 | A1 |
| 20100087229 | Maatta | Apr 2010 | A1 |
| 20100107099 | Frazier et al. | Apr 2010 | A1 |
| 20100138766 | Nakajima | Jun 2010 | A1 |
| 20100153313 | Baldwin et al. | Jun 2010 | A1 |
| 20100164959 | Brown et al. | Jul 2010 | A1 |
| 20100177035 | Schowengerdt et al. | Jul 2010 | A1 |
| 20100207888 | Camiel | Aug 2010 | A1 |
| 20100253593 | Seder et al. | Oct 2010 | A1 |
| 20100277439 | Charlier et al. | Nov 2010 | A1 |
| 20100287500 | Whitlow et al. | Nov 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 2602454 | Feb 2004 | CN |
| 7234743 | Sep 1995 | JP |
| 20030058204 | Jul 2003 | KR |
| WO-2010007426 | Jan 2010 | WO |
| Entry |
|---|
| Billinghurst et al ˜ “Towards Ambient Augmented Reality With Tangible Interfaces”˜ Human Computer Interaction ˜ Part III HCII 2009 ˜ pp. 387396. |
| D. Wigdor et al., “LucidTouch: A see-Through mobile device,” Proceedings of UST'07, Oct. 7-10, 2007; Newport, RI. |
| http://www.youtube.com/watch?v=V4Yj1Y3Pg20&feature=related > Publication Date: Jul. 12, 2008 ˜ video ˜ ARTToolKirt ARDeskTop ˜ No Copy to Be Sent. |
| International Search Report and Written Opinion in parent PCT patent application PCT/US2010/053860, dated Jul. 29, 2011. |
| N R. Hedley et al., “Explorations in the use of augmented reality for geographic visualization,” Presence: Teleoperators and Virtual Enviroments-Mixed Reality journal, vol. 11, issue 2, Apr. 2002, pages 119-133. |
| Nishina et al.; “Photometric Registration by Adaptive High Dynamic Range image Generation for Augmented Reality”, IEEE Int'l Symp Mixed & Augmented Reality, Sep. 2008, 4 pgs. |
| D. Wigdor et al., “LucidTouch: A see-through mobile device,” Proceedings of UST'07, Oct. 7-10, 2007, Newport, RI. |
| N.R. Hedley et al., “Explorations in the use of augmented reality for geographic visualization,” Presence: Teleoperators and Virtual nviroments—Mixed Reality journal, vol. 11, issue 2, Apr. 2002, pp. 119-133. |
| International search report and written opinion in parent PCT patent application PCT/US2010053860, dated Jul. 29, 2011. |
| Number | Date | Country | |
|---|---|---|---|
| 20130207896 A1 | Aug 2013 | US |
