Patent Grant
10409443
Providing an intuitive feedback system has an important role in achieving a perceptibly responsive computing system. Without feedback, users may have difficulty understanding the effect of their input on the computer system, and indeed whether their input is even being registered by the computing system. Conventional computing systems inherently provide aural and haptic feedback to the user through depressible buttons or clickers. For example, as a user types, the springs of a conventional keyboard provide physical resistance felt by the user's fingers and emit sounds, providing users with immediate feedback that their input is being registered by the keyboard. However, applying these feedback systems becomes challenging in a holographic environment that may utilize the user's gaze direction and hand gestures as input, rather than a physical input device that provides haptic and aural feedback when touched.
To address these issues, a head mounted display device and a method for providing contextual cursors in a virtual or augmented reality environment is provided. According to one aspect, the head mounted display device comprises a camera configured to capture a series of images, a display, and a processor configured to: display, to a user of the head mounted display device, a virtual cursor on the display so as to appear at a location in a three dimensional environment, determine whether motion of a hand of the user in the images is trackable, and in response to at least determining that motion of the hand is trackable, modify a visual appearance of the virtual cursor to indicate that motion of the hand is trackable, and in response to at least determining that motion of the hand is not trackable, modify the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
The present description relates to a head mounted display device and method for providing contextual based cursors in a virtual or augmented reality environment. More particularly, the present description relates to modifying a visual appearance of a virtual cursor that is displayed to a user of the head mounted display device. Utilizing contextual information, the displayed visual appearance of the virtual cursor is modified in order to provide feedback to the user. The visual appearance of the virtual cursor may afford the user understanding of whether the user's hand is currently trackable for the head mounted display, as well as what programmatic function the head mounted display has selected based on the contextual information. Accordingly, the user is provided with feedback that allows the user to understand, prior to entering user input, not only if the user input will have an effect, but also what effect the user input will have on the virtual or augmented reality environment being viewed by the user at that time.
For example, the head mounted display device 10 may include an image production system 14 that is configured to display virtual objects to the user with the stereoscopic display 12. In the augmented reality configuration with an at least partially see-through display, the virtual objects are visually superimposed onto the physical environment that is visible through the display so as to be perceived at various depths and locations. In the virtual reality configuration, the image production system 14 may be configured to display virtual objects to the user with the non-see-through stereoscopic display, such that the virtual objects are perceived to be at various depths and locations relative to one another. In one embodiment, the head mounted display device 10 may use stereoscopy to visually place a virtual object at a desired depth by displaying separate images of the virtual object to both of the user's eyes. Using this stereoscopy technique, the head mounted display device 10 may control the displayed images of the virtual objects, such that the user will perceive that the virtual objects exist at a desired depth and location in the viewed physical environment. In one example, the virtual object may be a virtual cursor that is displayed to the user, such that the virtual cursor appears to the user to be located at a desired location in the virtual three dimensional environment. In the augmented reality configuration, the virtual object may be a holographic cursor that is displayed to the user, such that the holographic cursor appears to the user to be located at a desired location in the real world physical environment.
The head mounted display device 10 includes an optical sensor system 16 that may include one or more optical sensors. In one example, the optical sensor system 16 includes an outward facing optical sensor 18 that may be configured to detect the real-world background from a similar vantage point (e.g., line of sight) as observed by the user through the at least partially see-through stereoscopic display 12. The optical sensor system 16 may additionally include an inward facing optical sensor 20 that may be configured to detect a gaze direction of the user's eye. It will be appreciated that the outward facing optical sensor 18 may include one or more component sensors, including an RGB camera and a depth camera. The RGB camera may be a high definition camera or have another resolution. The depth camera may be configured to project non-visible light and capture reflections of the projected light, and based thereon, generate an image comprised of measured depth data for each pixel in the image. This depth data may be combined with color information from the image captured by the RGB camera, into a single image representation including both color data and depth data, if desired. In a virtual reality configuration, the color and depth data captured by the optical sensor system 16 may be used to perform surface reconstruction and generate a virtual model of the real-world background that may be displayed to the user via the display 12. Alternatively, the image data captured by the optical sensor system 16 may be directly presented as image data to the user on the display 12.
The head mounted display device 10 may further include a position sensor system 22 that may include one or more position sensors such as accelerometer(s), gyroscope(s), magnetometer(s), global positioning system(s), multilateration tracker(s), and/or other sensors that output position sensor information useable as a position, orientation, and/or movement of the relevant sensor.
Optical sensor information received from the optical sensor system 16 and/or position sensor information received from position sensor system 22 may be used to assess a position and orientation of the vantage point of head mounted display device 10 relative to other environmental objects. In some embodiments, the position and orientation of the vantage point may be characterized with six degrees of freedom (e.g., world-space X, Y, Z, pitch, roll, yaw). The vantage point may be characterized globally or independent of the real-world background. The position and/or orientation may be determined with an on-board computing system (e.g., on-board computing system 24) and/or an off-board computing system.
Furthermore, the optical sensor information and the position sensor information may be used by a computing system to perform analysis of the real-world background, such as depth analysis, surface reconstruction, environmental color and lighting analysis, or other suitable operations. In particular, the optical and positional sensor information may be used to create a virtual model of the real-world background. In some embodiments, the position and orientation of the vantage point may be characterized relative to this virtual space. Moreover, the virtual model may be used to determine positions of virtual objects in the virtual space and add additional virtual objects to be displayed to the user at a desired depth and location within the virtual world.
Additionally, the optical sensor information received from the optical sensor system 16 may be used to identify and track objects in the field of view of optical sensor system 16. For example, depth data captured by optical sensor system 16 may be used to identify and track motion of a user's hand. The tracked motion may include movement of the user's hand in three-dimensional space, and may be characterized with six degrees of freedom (e.g., world-space X, Y, Z, pitch, roll, yaw). The tracked motion may also be used to identify and track a hand gesture made by the user's hand. For example, one identifiable hand gesture may be moving a forefinger upwards or downwards. It will be appreciated that other methods may be used to identify and track motion of the user's hand. For example, optical tags may be placed at known locations on the user's hand or a glove worn by the user, and the optical tags may be tracked through the image data captured by optical sensor system 16.
It will be appreciated that the following examples and methods may be applied to both a virtual reality and an augmented reality configuration of the head mounted display device 10. In a virtual reality configuration, the display 12 of the head mounted display device 10 is a non-see-through display, and the three dimensional environment is a virtual environment displayed to the user. The virtual environment may be a virtual model generated based on image data captured of the real-world background by optical sensor system 16 of the head mounted display device 10. Additionally, a virtual cursor having a modifiable visual appearance is also displayed to the user on the display 12 as having a virtual location within the three dimensional environment. In an augmented reality configuration, the virtual cursor is a holographic cursor that is displayed on an at least partially see-through display, such that the virtual cursor appears to be superimposed onto the physical environment being viewed by the user.
As another non-limiting example, the location 30 may be calculated based on a detected gaze direction 34 and a recognized object that intersects with the detected gaze direction. In this example, the recognized object may be a real object in the three dimensional environment. This example is illustrated in
Additionally, in the example illustrated in
In the above embodiment, the head mounted display device 10 communicates to the user whether motion of the user's hand is trackable. In this embodiment, in response to at least determining that motion of the hand is trackable, the head mounted display device 10 modifies the visual appearance of the holographic cursor to indicate that motion of the hand is trackable. In the example illustrated in
Further in this embodiment, in response to at least determining that motion of the hand is not trackable, the head mounted display device 10 modifies the visual appearance of the holographic cursor to indicate that motion of the hand is not trackable. As illustrated in
Now turning to
Based on this contextual information, the head mounted display device 10 may then select an appropriate programmatic function. In the example shown in
After the programmatic function has been selected, the head mounted display device 10 may be modify the visual appearance of the holographic cursor to indicate the selected programmatic function. In the example illustrated in
However, if motion of the user's hand is currently trackable, the head mounted display device 10 may be configured to track motion of the hand in the images to identify a user hand gesture. In response to at least identifying the user hand gesture, the head mounted display device may execute a programmatic function that may be selected as described above. In response to at least identifying the hand gesture, the head mounted display device 10 may be further configured to determine whether the hand gesture is being maintained based on the tracked motion of the hand. That is, the head mounted display may determine whether the user is currently holding the same hand gesture over a period of time. If the user maintains the hand gesture, the head mounted display device may continue to execute the selected programmatic function.
Additionally, the head mounted display device may further modify the visual appearance of the holographic cursor is indicate that the selected programmatic function is currently being executed. In the example illustrated in
As illustrated in
Now turning to
It will be appreciated that the virtual object 42 does not necessarily take the form illustrated in
Additionally, the visual appearance 28E of the holographic cursor indicates not only that the resizing function has been selected, but also indicates a possible user input associated with the selected programmatic function based on the determined context. In this specific example, the two chevrons point in two different directions. These chevron icons indicate to the user that based on the form of the virtual object 42, that two user inputs associated with the resizing function are available. In this example, the user may move the hand 38 in the same directions indicated by the two chevrons of the visual appearance 28E of the holographic cursor in order to change the size of the virtual object 42.
In the second example illustrated in
Proceeding from step 602 to step 604, the method 600 may include determining a context based on a captured series of images. Step 604 may include substeps 606, 608, 610, and 612, as follows. At substep 606, method 600 may include detecting a recognized object in the three dimensional environment. Proceeding to substep 608, the method 600 may include determining a gaze direction of the user of the head mounted display device. It will be appreciated the recognized object may be either a virtual object or a real object having positions in the three dimensional environment. In one embodiment, the recognized object is detected by the optical sensor system 16 of the head mounted display device 10. In another embodiment, the recognized object is determined by performing visual analysis on the series of images captured by the optical sensor system 16. Additionally, the gaze direction of the user may be determined based on visual data received by the inward facing optical sensor 20 of the optical sensor system 16, although it will be appreciated that other methods are possible.
After detecting the recognized object and determining the gaze direction of the user, method 600 advances from substep 608 to 610, and may include determining an intersection between the recognized object and the gaze direction of the user. Proceeding from sub step 610 to substep 612, the method 600 may include determining the context based on the intersection and the recognized object.
After determining the context, method 600 advances from step 604 to step 614 and may include selecting the programmatic function based on the determined context. It will be appreciated that the selection of the programmatic function may be determined based on a variety of suitable contextual information. In one example, the programmatic function is selected based on the type of object the recognized object is and the location that the gaze direction intersected with the recognized object. As a few other non-limiting examples, the suitable contextual information may be a current application running on the head mounted display device, a position of the user in the three dimensional environment, or past user input of the user.
Proceeding from step 614 to step 616, the method 600 may include modifying the visual appearance of the holographic cursor to indicate the selected programmatic function. It will be appreciated that the visual appearance of the holographic cursor may be modified in a variety of ways. As a few non-limiting examples, the visual appearance may be modified by changing a color, changing a shape, adding or removing an icon, or changing a size of the holographic cursor.
Advancing from step 616 to step 618, the method 600 may include determining whether motion of the hand of the user in the series of images captured by a camera of the head mounted display device is trackable.
In response to at least determining that motion of the hand is not trackable, the method 600 proceeds from step 618 to step 620 and may include modifying the visual appearance of the holographic cursor to indicate that motion of the hand is not trackable. It will be appreciated that this modification to the visual appearance of the holographic cursor may be made in substitution of or in addition to any previous modifications to the visual appearance of the holographic cursor. Next, the method 600 may return to step 618 after a period of time to determine whether the motion of the hand has become trackable.
In response to at least determining that motion of the hand is trackable, the method 600 advances from step 618 to step 622 and may include modifying the visual appearance of the holographic cursor to indicate that motion of the hand is trackable. It will be appreciated that the visual appearance may be modified in a variety of ways as discussed above, and these modifications may be made in substitution of and in addition to any previous modifications to the visual appearance of the holographic cursor.
Proceeding from step 622 to step 624, the method 600 may include tracking motion of the hand in the images to identify a user hand gesture. In response to at least identifying the hand gesture, the method 600 advances to step 626 and may include executing a programmatic function. In one embodiment, the executed programmatic function is the programmatic function selected in step 614. It will be appreciated that any suitable hand gesture may be used. In one non-limiting example, the hand gesture may be the raising or lowing or a forefinger of the user's hand while the other fingers remain clasped.
Now turning to
In response to at least determining that the hand gesture is being maintained, the method 600 advances from step 628 to step 632 and may include continuing to execute the selected programmatic function. It will be appreciated that during execution of the selected programmatic function, the head mounted display device may continue to track motion of the hand of the user. The tracked motion of the hand may be used as input for the execution of the selected programmatic function.
Proceeding from step 632 to step 634, the method 600 may include modifying the visual appearance of the holographic cursor to indicate that the selected programmatic function is currently being executed. It will be appreciated that the visual appearance may be modified in a variety of way as discussed above, and these modifications may be made in substitution of and in addition to any previous modifications to the visual appearance of the holographic cursor.
In response to at least determining that the intersection is located at the planar surface of the recognized object, the method 600 proceeds from step 636 to substep 638A and may include selecting a drawing function as the programmatic function, the drawing function being a programmatic function to draw an image onto the planar surface of the recognized object.
Alternatively, in response to at least determining that the intersection is located at the resizing area of the recognized object, the method 600 proceeds from step 636 to substep 638B and may include selecting a resizing function as the programmatic function, the resizing function being a programmatic function to change a size of the recognized object.
Alternatively, in response to at least determining that the intersection is located at the moving area of the recognized object, the method 600 proceeds from step 636 to substep 638C and may include selecting a moving function as the programmatic function, the moving function being a programmatic function to move the object location of the recognized object to a different location in the three dimensional environment.
In response to at least selecting a programmatic function, the method 600 may advance from the expanded step 614 to step 616. It will be appreciated that while a drawing function, a resizing function, and a moving function are described in the expanded step 614, a variety of programmatic functions are possible.
In some embodiments, the methods and processes described herein may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
Computing system 90 includes a logic processor 92 volatile memory 93, and a non-volatile storage device 94. Computing system 90 may optionally include a display subsystem 96, input subsystem 98, communication subsystem 100, and/or other components not shown in
Logic processor 92 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
The logic processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. Processors of the logic processor 92 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic processor optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic processor may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration. In such a case, these virtualized aspects are run on different physical logic processors of various different machines, it will be understood. Logic processor 92 may include the image production subsystem 14 described above, which may be included in hardware-implemented logic (such as an ASIC or system-on-chip) or as stored software or firmware executed by logic processor 92.
Non-volatile storage device 94 includes one or more physical devices configured to hold instructions executable by the logic processors to implement the methods and processes described herein. When such methods and processes are implemented, the state of non-volatile storage device 94 may be transformed—e.g., to hold different data.
Non-volatile storage device 94 may include physical devices that are removable and/or built-in. Non-volatile storage device 94 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), or other mass storage device technology. Non-volatile storage device 94 may include nonvolatile, dynamic, static, read/write, read-only, sequential-access, location-addressable, file-addressable, and/or content-addressable devices. It will be appreciated that non-volatile storage device 94 is configured to hold instructions even when power is cut to the non-volatile storage device 94.
Volatile memory 93 may include physical devices that include random access memory. Volatile memory 93 is typically utilized by logic processor 92 to temporarily store information during processing of software instructions. It will be appreciated that volatile memory 93 typically does not continue to store instructions when power is cut to the volatile memory 93.
Aspects of logic processor 92, volatile memory 93, and non-volatile storage device 94 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example. When computing system 90 is formed as the head mounted display device 10 of
The terms “module,” “program,” and “engine” may be used to describe an aspect of computing system 90 implemented to perform a particular function. In some cases, a module, program, or engine may be instantiated via logic processor 92 executing instructions held by non-volatile storage device 94, using portions of volatile memory 93. It will be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
When included, display subsystem 96 may be used to present a visual representation of data held by non-volatile storage device 94. When computing system 90 is formed as the head mounted display device 10 of
When included, input subsystem 98 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller. When computing system 90 is formed as the head mounted display device 10 of
When included, communication subsystem 100 may be configured to communicatively couple computing system 90 with one or more other computing devices. Communication subsystem 100 may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem may allow computing system 90 to send and/or receive messages to and/or from other devices via a network such as the Internet. Thus, the head mounted display system 10 of
The following paragraphs provide additional support for the claims of the subject application. One aspect provides a head mounted display device for contextual cursors in a virtual or augmented reality environment comprising: a camera configured to capture a series of images, a display, and a processor configured to: display, to a user of the head mounted display device, a virtual cursor on the display so as to appear at a location in a three dimensional environment, determine whether motion of a hand of the user in the images is trackable, in response to at least determining that motion of the hand is trackable, modify a visual appearance of the virtual cursor to indicate that motion of the hand is trackable, and in response to at least determining that motion of the hand is not trackable, modify the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable. The head mounted display device may additionally or alternatively include, wherein the three dimensional environment is a physical environment, the display is an at least partially see-through display configured to display holograms superimposed onto the three dimensional environment, and the virtual cursor is a holographic cursor. The head mounted display device may additionally or alternatively include, wherein to modify the visual appearance of the holographic cursor, the processor is further configured to: change a color, change a shape, add or remove an icon, or change a size of the holographic cursor. The head mounted display device may additionally or alternatively include, wherein in response to at least determining that motion of the hand is trackable, the processor is further configured to: track motion of the hand in the images to identify a user hand gesture, and in response to at least identifying the hand gesture, execute a programmatic function. The head mounted display device may additionally or alternatively include, wherein the processor is further configured to: determine a context based on the captured series of images, and select the programmatic function based on the determined context. The head mounted display device may additionally or alternatively include, wherein the processor is further configured to: modify the visual appearance of the holographic cursor to indicate the selected programmatic function. The head mounted display device may additionally or alternatively include, wherein to determine the context based on the captured series of images, the processor is further configured to: detect a recognized object in the three dimensional environment, determine a gaze direction of the user of the head mounted display device, determine an intersection between the recognized object and the gaze direction of the user, and determine the context based on the intersection and the recognized object. The head mounted display device may additionally or alternatively include, wherein in response to at least identifying the hand gesture, the processor is further configured to: determine whether the hand gesture is being maintained based on the tracked motion of the hand, in response to at least determining that the hand gesture is being maintained, continue executing the selected programmatic function and modify the visual appearance of the holographic cursor to indicate that the selected programmatic function is currently being executed, and in response to at least determining that the hand gesture is not maintained, modify the visual appearance of the holographic cursor to indicate that the selected programmatic function is not currently being executed. The head mounted display device may additionally or alternatively include, the processor is further configured to: in response to at least selecting the programmatic function based on the determined context, modify the visual appearance of the holographic cursor to indicate to the user, a possible user input associated with the selected programmatic function based on the determined context. The head mounted display device may additionally or alternatively include, wherein the recognized object is a virtual object having an object location in the three dimensional environment, and the processor is further configured to: determine whether the intersection between the recognized object and the gaze direction of the user is located at a planar surface of the recognized object, a resizing area of the recognized object, or a moving area of the recognized object, in response to at least determining that the intersection is located at the planar surface of the recognized object, select a drawing function as the selected programmatic function, the drawing function being a programmatic function to draw an image onto the planar surface of the recognized object, in response to at least determining that the intersection is located at the resizing area of the recognized object, select a resizing function as the selected programmatic function, the resizing function being a programmatic function to change a size of the recognized object, and in response to at least determining that the intersection is located at the moving area of the recognized object, select a moving function as the selected programmatic function, the moving function being a programmatic function to move the object location of the recognized object to a different location in the three dimensional environment.
Another aspect provides a method for contextual cursors in a virtual or augmented reality environment comprising: displaying a virtual cursor to a user on a display of a head mounted display device so as to appear at a location in a three dimensional environment, determining whether motion of a hand of the user in a series of images captured by a camera of the head mounted display device is trackable, in response to at least determining that motion of the hand is trackable, modifying a visual appearance of the virtual cursor to indicate that motion of the hand is trackable, and in response to at least determining that motion of the hand is not trackable, modifying the visual appearance of the virtual cursor to indicate that motion of the hand is not trackable. The method may additionally or optionally include, wherein the three dimensional environment is a physical environment, the display is an at least partially see-through display configured to display holograms superimposed onto the three dimensional environment, and the virtual cursor is a holographic cursor. The method may additionally or optionally include, wherein modifying the visual appearance of the holographic cursor further comprises: changing a color, changing a shape, adding or removing an icon, or changing a size of the holographic cursor. The method may additionally or optionally include, wherein in response to at least determining that motion of the hand is trackable, the method further comprises: tracking motion of the hand in the images to identify a user hand gesture, and in response to at least identifying the hand gesture, executing a programmatic function. The method may additionally or optionally include, determining a context based on the captured series of images, selecting the programmatic function based on the determined context, and modifying the visual appearance of the holographic cursor to indicate the selected programmatic function. The method may additionally or optionally include, wherein determining the context based on the captured series of images, further comprises: detecting a recognized object in the three dimensional environment, determining a gaze direction of the user of the head mounted display device, determining an intersection between the recognized object and the gaze direction of the user, and determining the context based on the intersection and the recognized object. The method may additionally or optionally include, wherein in response to at least identifying the hand gesture, the method further comprises: determining whether the hand gesture is being maintained based on the tracked motion of the hand, in response to at least determining that the hand gesture is being maintained, continuing to execute the selected programmatic function and modifying the visual appearance of the holographic cursor to indicate that the selected programmatic function is currently being executed, and in response to at least determining that the hand gesture is not maintained, modifying the visual appearance of the holographic cursor to indicate that the selected programmatic function is not currently being executed. The method may additionally or optionally include, in response to at least selecting the programmatic function based on the determined context, modifying the visual appearance of the holographic cursor to indicate to the user, a possible user input associated with the selected programmatic function based on the determined context. The method may additionally or optionally include, wherein the recognized object is a virtual object having an object location in the three dimensional environment, and further comprising: determining whether the intersection between the recognized object and the gaze direction of the user is located at a planar surface of the recognized object, a resizing area of the recognized object, or a moving area of the recognized object, in response to at least determining that the intersection is located at the planar surface of the recognized object, selecting a drawing function as the selected programmatic function, the drawing function being a programmatic function to draw an image onto the planar surface of the recognized object, in response to at least determining that the intersection is located at the resizing area of the recognized object, selecting a resizing function as the selected programmatic function, the resizing function being a programmatic function to change a size of the recognized object, and in response to at least determining that the intersection is located at the moving area of the recognized object, selecting a moving function as the selected programmatic function, the moving function being a programmatic function to move the object location of the recognized object to a different location in the three dimensional environment.
Another aspect provides a head mounted display device for contextual cursors in a holographic environment comprising: a camera configured to capture a series of images, an at least partially see-through display configured to display holograms, and a processor configured to: display, to a user of the head mounted display device, a holographic cursor on the at least partially see-though display so as to appear at a location in a three dimensional environment, determine a context based on the captured series of images, select a programmatic function based on the determined context, modify a visual appearance of the holographic cursor to indicate the selected programmatic function, and track motion of the hand in the images to identify a user hand gesture, and in response to at least identifying the hand gesture, execute the selected programmatic function.
It will be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6021403 | Horvitz et al. | Feb 2000 | A |
| 6842877 | Robarts et al. | Jan 2005 | B2 |
| 7493153 | Ahmed | Feb 2009 | B2 |
| 7561143 | Milekic | Jul 2009 | B1 |
| 7684592 | Paul | Mar 2010 | B2 |
| 8479110 | Johnson et al. | Jul 2013 | B2 |
| 8566717 | Markiewicz et al. | Oct 2013 | B2 |
| 8671365 | Berus | Mar 2014 | B2 |
| 8773355 | Perez | Jul 2014 | B2 |
| 8881051 | Frey | Nov 2014 | B2 |
| 8964298 | Haddick | Feb 2015 | B2 |
| 9170674 | Forutanpour | Oct 2015 | B2 |
| 9703373 | Kohlhoff | Jul 2017 | B2 |
| 9710130 | Mulcahy | Jul 2017 | B2 |
| 20020006222 | Inagaki et al. | Jan 2002 | A1 |
| 20020044152 | Abbott, III | Apr 2002 | A1 |
| 20040131232 | Meisner | Jul 2004 | A1 |
| 20040201857 | Foxlin | Oct 2004 | A1 |
| 20050206583 | Lemelson et al. | Sep 2005 | A1 |
| 20090027330 | Aida | Jan 2009 | A1 |
| 20100103075 | Kalaboukis | Apr 2010 | A1 |
| 20110083112 | Matsubara | Apr 2011 | A1 |
| 20110148798 | Dahl | Jun 2011 | A1 |
| 20110175932 | Yu et al. | Jul 2011 | A1 |
| 20120113223 | Hilliges | May 2012 | A1 |
| 20120206322 | Osterhout et al. | Aug 2012 | A1 |
| 20120249587 | Anderson | Oct 2012 | A1 |
| 20120319949 | Lee | Dec 2012 | A1 |
| 20130007668 | Liu | Jan 2013 | A1 |
| 20130055150 | Galor | Feb 2013 | A1 |
| 20130135288 | King et al. | May 2013 | A1 |
| 20130154913 | Genc | Jun 2013 | A1 |
| 20130194164 | Sugden | Aug 2013 | A1 |
| 20130271363 | Poznansky | Oct 2013 | A1 |
| 20130301926 | Eilat | Nov 2013 | A1 |
| 20130307764 | Denker et al. | Nov 2013 | A1 |
| 20130307771 | Parker et al. | Nov 2013 | A1 |
| 20130307875 | Anderson | Nov 2013 | A1 |
| 20140063050 | Fateh | Mar 2014 | A1 |
| 20140125577 | Hoang | May 2014 | A1 |
| 20140152558 | Salter et al. | Jun 2014 | A1 |
| 20140168267 | Kim | Jun 2014 | A1 |
| 20140180972 | Boldyrev et al. | Jun 2014 | A1 |
| 20140184550 | Hennessey | Jul 2014 | A1 |
| 20140237366 | Poulos | Aug 2014 | A1 |
| 20140258942 | Kutliroff | Sep 2014 | A1 |
| 20140282162 | Fein | Sep 2014 | A1 |
| 20140282282 | Holz | Sep 2014 | A1 |
| 20140285520 | Park | Sep 2014 | A1 |
| 20140361988 | Katz | Dec 2014 | A1 |
| 20140368535 | Salter | Dec 2014 | A1 |
| 20150062000 | Saito | Mar 2015 | A1 |
| 20150103003 | Kerr et al. | Apr 2015 | A1 |
| 20150130740 | Cederlund | May 2015 | A1 |
| 20160004306 | Maltz | Jan 2016 | A1 |
| 20160132139 | Du | May 2016 | A1 |
| 20160140763 | Seichter | May 2016 | A1 |
| 20160162082 | Schwesinger | Jun 2016 | A1 |
| 20160224123 | Antoniac | Aug 2016 | A1 |
| 20160232715 | Lee | Aug 2016 | A1 |
| 20160253044 | Katz | Sep 2016 | A1 |
| 20160284134 | Kamhi | Sep 2016 | A1 |
| 20160321841 | Christen | Nov 2016 | A1 |
| 20160379414 | Brown | Dec 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 2013033842 | Mar 2013 | WO |
| Entry |
|---|
| Definition of cursor, accessed Apr. 11, 2018, 1 page. |
| IPEA European Patent Office, Second Written Opinion Issued in PCT Application No. PCT/US2016/036112, dated Jun. 1, 2017, WIPO, 10 pages. |
| Ajanki et al., “An Augmented Reality Interface to Contextual Information”, In Virtual Reality—Special Issue on Augmented Reality, vol. 15, Issue 2-3, Jun. 1, 2011, 15 Pages. |
| Meers et al., “Haptic Gaze-Tracking Based Perception of Graphical User Interfaces”, In Proceedings of 11th International Conference on Information Visualisation, Jul. 4, 2007, 15 pages. |
| Doswell et al., “Augmenting Human Cognition with Adaptive Augmented Reality”, In Proceedings of Foundations of Augmented Cognition Advancing Human Performance and Decision-Making through Adaptive Systems Lecture Notes in Computer Science vol. 8534, Retrieved on : May 25, 2015, 4 Pages. |
| Porta et al., “ceCursor, a Contextual Eye Cursor for General Pointing in Windows Environments”, In Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications, Mar. 22, 2010, 8 pages. |
| Chapuis et al., “UIMarks: Quick Graphical Interaction with Specific Targets”, In Proceedings of the 23th annual ACM Symposium on User Interface Software and Technology, Oct. 3, 2010, 10 pages. |
| ISA European Patent Office, International Search Report and Written Opinion Issued in PCT Application No. PCT/US2016/036112, dated Sep. 1, 2016, WIPO, 13 pages. |
| “International Preliminary Report on Patentability Issued in PCT Application No. PCT/US2016/036112”, dated Sep. 11, 2017, 11 Pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20160378294 A1 | Dec 2016 | US |
