A computer uses a desktop environment to enable a user to visually interact with an operating system of the computer. Desktop environments allow the user to move items within a two-dimensional surface, which is generally defined by a display on a monitor of the computer. In some cases, the desktop environment may be designed to give an illusion of depth to provide a three-dimensional experience for the user. However, despite this illusion of depth, the interaction typically remains a two-dimensional interaction. By providing a true three-dimensional desktop environment, it is possible for a user to interact with the computer in an entirely new way.
For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.
For illustrative purposes only, the three-dimensional space 106 is shown as having a cube-like volume corresponding to the device 102. A coordinate system 125 may be defined over the space 106. In general, the coordinate system 125 may be defined with respect to the display 104 which generates the three-dimensional desktop environment 100. One direction such as a z-direction of the coordinate system 125 extends perpendicularly out of display 104, with z=0 defined at a surface of the display 104. The x-direction is generally oriented along a horizontal axis of the display 104 and the y-direction is generally oriented along a vertical axis of the display 104.
Icons displayed within the desktop environment 100 may be either isolated icons or grouped icons. In
Since the three-dimensional desktop environment 100 is projected into a three-dimensional space 106 existing in real space, a user may move a user object such as a hand or a stylus, pointer, wand, etc., within the three-dimensional space 106. In various embodiments, at least one ultrasonic transducer may be used to determine a location of the user object within the three-dimensional space 106. Thus, the user may be enabled to employ the user object to interact with the three-dimensional desktop environment 100 as well as with an image or icon within the three-dimensional desktop environment 100, as discussed below.
Additionally, receivers 203R, 205R and 207R of the other transducers 203, 205 and 207 can receive ultrasonic pulses 307b, 307c and 307d, respectively, that are reflected from the user object 302 corresponding to the generated ultrasonic pulse 305 from transmitter 201T. Time delays between generating the ultrasonic pulse 305 and receiving the reflected pulses 307b, 307c and 307d may also be recorded. Due to known spatial distances between transmitter 201T and receivers 203R, 205R and 207R, time delays determined for each of these transmitter-receiver pairings may be used to determine a horizontal and vertical distance (i.e., x-coordinate and y-coordinate) of the user object 302 with respect to the display 200.
In addition to measuring reflected pulses that are based on the ultrasonic pulse generated at transmitter 201T, this method of determining a spatial relation of the user object to the display may be implemented using the other transmitters 203T, 205T and 207T. Thus, the transmitters and receivers of each ultrasonic transducer 201, 203, 205 and 207 may be used in any suitable combination to determine the location of the user object 302. Additionally, transmitters 201T, 203T, 205T and 207T may be activated in a selected pattern. In one embodiment, the relative location of the user object 302 may be determined by triangulation of the distances determined using any or all of the ultrasonic transducers 201, 203, 205 and 207.
In addition to determining a location of the user object 302, the signals transmitted and received by ultrasonic transducers 201, 203, 205 and 207 may be used to determine a user action by the user object 302. A user action may include, for example, a position of the user object 302, a motion of the user object 302 with respect to the device 110, and/or a gesture or movement of the user object 302. The processor 312 may interpret the user action and perform a corresponding action based upon the user action.
There generally exists a one-to-one correspondence between the virtual space 512 and the real space 502 so that it is possible to map a position in real space 502 to a corresponding position in virtual space 512 and vice-versa. For example, a virtual object 516 that has position (xv, yv, zv)virtual object in virtual space 512 may be mapped to a position (xr, yr, zr)virtual object in real space 502 at which a corresponding image 506 is projected. Also, a position of the user object (xr, yr, zr)user object in real space 502 may be mapped to a corresponding position (xv, yv, zv)user object of a representation 514 of the user object 504 in virtual space 512. As the user moves the user object 504 in the real space 502, transmitted and received signals from the ultrasonic transducers are used by the processor to determine the position (xr, yr, zr)user object of the user object 504 in real space 502 and map this real position to the corresponding virtual position (xv, yv, zv)user object of its virtual representation 514.
In addition to mapping a virtual object 516 to real space 502 and mapping a user object 504 to virtual space 512, the processor 312 may determine a relative distance d between the location (xr, yr, zr)user object of the user object 504 in real space and the location (xr, yr, zr)virtual object of the projected image 506 in real space 502. Processor 312 determines distance d using the virtual position (xv, yv, zv)user object of the representation 514 of user object 504 and the virtual position (xv, yv, zv)virtual object of the virtual object 516. When the determined distance d is less than a selected threshold value ε, the processor 312 may determine that the virtual representation 514 substantially coincides with or “touches” the selected virtual object 516. The touching of the virtual representation 514 to the virtual object 516 in virtual space 512 corresponds with the experience of the user seeing the user object 504 as “touching” the projected image 506 of the virtual object in real space 502. When the distance d between the virtual representation 514 and the virtual object 516 is greater than this selected threshold value ε, the processor 312 may determine that the virtual representation 514 is not touching the virtual object 516 and therefore that the user object 504 is not “touching” the projected image 506.
Once the user object 504 is determined to be touching the projected image 506, the user may perform a user action to select the projected image 506 and to thereby select the corresponding virtual object 516. For example, the user may depress a key or a group of keys or a sequence of keys on a keyboard to select the projected image 506. Alternatively, the user may perform an action using a mouse, a touch pad or a touch-sensitive surface of the device, such as a surface of the display 501. Alternatively, if the user object 504 is a wand or stylus or other suitable user object, the user may select a button at the user object 504 that generates a signal to the device to indicate the user's intent to select the projected image 506. Alternatively, the user may perform a gesture or a motion within the real space 502 of the desktop environment that may be interpreted at the processor 312 as the user having selected the projected image 506. In an embodiment in which the user object 504 is part of a user's hand and the user may perform a pinching motion or other suitable motion at the position of the projected image 506 to perform a “grabbing” of the projected image 506. Alternatively, the user action may include maintaining the user object 504 at the substantial position of the projected image 506 for an extended period of time, such as for longer than one second, to select the projected image 506. Various user actions may similarly be taken to deselect a projected image.
Once the user has selected the projected image 506, the user may perform a user action on the projected image 506. In various embodiments, the user may move the projected image 506 from one position in the desktop environment to another position in the desktop environment, rotate the projected image 506, increase a size of the projected image 506, decrease a size of the projected image 506, etc. In addition, the user may activate or open an application or a program by selected the projected image 506.
Field 808 indicates a user rights or user access with respect to the pixel volume. User access rights may include, for example and without limitation, “Normal” 812 “Locked” 814 “System Restricted” 816 and “Others” 818. A Normal 812 access right may mean that the user is free to interact with the pixel volume. A Locked 814 access right may exclude the user from interacting with the pixel volume. For example, a power level may be locked for a given user or locked for a given application, etc. A System Restricted 816 access right may mean that the pixel volume is only for displaying system objects that are not to be tampered with by the user. An “Others” 818 access right may be a user-defined access right or application-specific access right, for example.
Field 810 indicates to which sub-groups the pixel volume may belong. A sub-group 820 may include pixel volumes within a same power level. Alternatively, a sub-group 822 may be a system-defined or user-defined group or category. Images or icons within the three-dimensional desktop environment may be grouped into the sub-group according to an affiliation with the category. For example, a sub-group may include Microsoft Office applications and their icons for Word, Excel, Powerpoint, etc. Examples of user-defined sub-groups may include icons grouped according to user's hobbies, clubs, career advancement documents, financial documents, pictures, etc. Icons in a same group may be flagged with a selected marker, such as a virtual tag, etc.
Therefore, in accordance with one aspect of the disclosure, a method of performing an action with a processor includes: generating, using the processor, a three-dimensional desktop environment and projecting the three-dimensional desktop environment into a real space; propagating an ultrasonic pulse into the real space; receiving a reflection of the ultrasonic pulse from a user object located within the real space; determining a user action of the user object using the ultrasonic pulse and the reflected ultrasonic pulse; and performing the action with the processor based on the determined user action.
In accordance with another aspect of the disclosure, an electronic device includes: a display configured to project a three-dimensional desktop environment into a real space; at least one ultrasonic transducer configured to propagate an ultrasonic pulse into the real space and receive a reflection of the ultrasonic pulse from a user object in the real space; and a processor configured to: determine a user action of a user object from the reflection of the ultrasonic pulse, and perform an action based on the determined user action of the user object.
In accordance with yet another aspect of the disclosure, an interface for providing a user interaction with an electronic device includes: a display configured to project a three-dimensional desktop environment into a real space; at least one ultrasonic transducer configured to propagate an ultrasonic pulse into the real space and receive a reflection of the ultrasonic pulse from a user object in the real space; and a processor configured to: determine a user action of a user object from the reflection of the ultrasonic pulse, and perform an action based on the determined user action of the user object.
It should be understood at the outset that although illustrative implementations of one or more embodiments of the present disclosure are provided below, the disclosed systems and/or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.
While several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.
Also, techniques, systems, subsystems and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the spirit and scope disclosed herein.
Number | Name | Date | Kind |
---|---|---|---|
6023277 | Osaka | Feb 2000 | A |
6064354 | DeLuca | May 2000 | A |
6559813 | DeLuca et al. | May 2003 | B1 |
6822635 | Shahoian et al. | Nov 2004 | B2 |
7148875 | Rosenberg et al. | Dec 2006 | B2 |
8063893 | Rosenberg et al. | Nov 2011 | B2 |
8279168 | Glomski et al. | Oct 2012 | B2 |
8416187 | Margolis et al. | Apr 2013 | B2 |
8432390 | Givon | Apr 2013 | B2 |
8493354 | Birnbaum et al. | Jul 2013 | B1 |
8494497 | Seo et al. | Jul 2013 | B2 |
20030058265 | Robinson et al. | Mar 2003 | A1 |
20030184574 | Phillips et al. | Oct 2003 | A1 |
20050277448 | Castaneda et al. | Dec 2005 | A1 |
20070103449 | Laitinen et al. | May 2007 | A1 |
20070139391 | Bischoff | Jun 2007 | A1 |
20080068348 | Rosenberg et al. | Mar 2008 | A1 |
20080153554 | Yoon et al. | Jun 2008 | A1 |
20080252595 | Boillot | Oct 2008 | A1 |
20090018445 | Schers et al. | Jan 2009 | A1 |
20090058829 | Kim et al. | Mar 2009 | A1 |
20090077504 | Bell et al. | Mar 2009 | A1 |
20090102805 | Meijer et al. | Apr 2009 | A1 |
20090160813 | Takashima et al. | Jun 2009 | A1 |
20090267892 | Faubert | Oct 2009 | A1 |
20090267920 | Faubert et al. | Oct 2009 | A1 |
20090294186 | Fontanella et al. | Dec 2009 | A1 |
20090322496 | da Costa | Dec 2009 | A1 |
20100020036 | Hui et al. | Jan 2010 | A1 |
20100033479 | Hirayama et al. | Feb 2010 | A1 |
20100066662 | Tomisawa et al. | Mar 2010 | A1 |
20100095206 | Kim | Apr 2010 | A1 |
20100110018 | Faubert et al. | May 2010 | A1 |
20100156814 | Weber et al. | Jun 2010 | A1 |
20100156818 | Burrough et al. | Jun 2010 | A1 |
20100225734 | Weller et al. | Sep 2010 | A1 |
20110001707 | Faubert et al. | Jan 2011 | A1 |
20110119639 | Tartz | May 2011 | A1 |
20110138277 | Grant et al. | Jun 2011 | A1 |
20110141046 | Sato et al. | Jun 2011 | A1 |
20110195782 | Mao et al. | Aug 2011 | A1 |
20110205151 | Newton et al. | Aug 2011 | A1 |
20120001875 | Li et al. | Jan 2012 | A1 |
20120068957 | Puskarich et al. | Mar 2012 | A1 |
20120162384 | Vesely et al. | Jun 2012 | A1 |
20120229400 | Birnbaum et al. | Sep 2012 | A1 |
20120293513 | Krishnaswamy | Nov 2012 | A1 |
20140160087 | Mercea et al. | Jun 2014 | A1 |
Number | Date | Country |
---|---|---|
2708673 | Dec 2010 | CA |
2270627 | Jan 2011 | EP |
2518590 | Oct 2012 | EP |
2005258666 | Sep 2005 | JP |
2009515351 | Apr 2009 | JP |
2004025517 | Mar 2004 | WO |
2009099280 | Aug 2009 | WO |
Entry |
---|
http://wn.wikipedia.org/wiki/Autosteroscopy—published at least as early as Nov. 2010. |
Canadian office action for corresponding Canadian Patent Application No. 2,708,673, dated Feb. 19, 2013, 2 pages. |
http://techon.nikkeibp.co.jp/english/NEWS—EN/20090310/166952/?P=11—published Mar. 11, 2009. |
http://www.immersence.info/—published at least as early as Nov. 2010. |
http://www.japantrends.com/tobidasu-3d-table-menu-from-tec/—published Mar. 12, 2008. |
Korean first office action for Korean Patent Application No. 10-2010-0060536, dated Dec. 19, 2011, plus English transaction, 13 pages. |
http:/wn.wikipedia.org/wiki/Lenticular-lens—published at least as early as Nov. 2010. |
http://silkehilsing.de/impress/blog/—published May 24, 2009. |
http://singularityhub.com/2009/11/13/haptic-ring-lets-you-feel-objects-in-augmented-reality-video/—published Nov. 13, 2009. |
https:/wiki.cc.gatech.edu/ccg/—media/classes/7570/7470-f06/gwpaper.pdf?id=classes:7570:7470-f06:projects—ii&cache=cache—published at least as early as Nov. 2010. |
Amemiya, Tomohiro, et al.; “Phantom-Drawn: Direction Guidance Using Rapid and Asymmetric Acceleration Weighted by Nonlinearity of Perception”; ICAT; 2005; 8 pages. |
Hoggan, Eve, et al.; “Mobile Multi-Actuator Tactile Displays”; HAID; 2007; 12 pages. |
Nakamura, Norio, et al.; “Development of a Force and Torque Hybrid Display “GyroGubeSick””; IEEE; 2005; 2 pages. |
European Extended Search Report; Application No. 14184405.0; Jan. 21, 2015; 13 pages. |
European Search Report; Application No./Patent No. 13178166.8-1972; Date of Mailing: Jun. 13, 2014; pp. 1-12. |
Mankowski, Peter; U.S. Appl. No. 14/024,006, filed Sep. 11, 2013; Title: Three Dimensional Haptics Hybrid Modeling. |
Office Action dated May 22, 2015; U.S. Appl. No. 14/024,006, filed Sep. 11, 2013; 39 pages. |
Final Office Action dated Dec. 3, 2015; U.S. Appl. No. 14/024,006, filed Sep. 11, 2013; 49 pages. |
Number | Date | Country | |
---|---|---|---|
20150033191 A1 | Jan 2015 | US |