Patent Application
20220011924
Virtual experience computing systems (e.g., virtual reality or augmented reality devices) receive input from input devices to select objects in virtual environments and augmented reality environments (i.e., collectively virtual experiences). For example, a virtual reality controller can be used to make a selection in a virtual experience based on tracking motion of the virtual reality controller, where the motion is communicated to a virtual reality computing system of the virtual experience to indicate the selection. Selecting virtual reality objects or augmented reality objects (“virtual objects”) can further be complimented with annotation inputs to associate additional information with the selections. As such, an interactivity system for selection and annotation is an integral part of interacting with virtual experience computing systems.
Embodiments of the present invention are directed to a multi-device mixed interactivity system for selection and annotation. The multi-device mixed interactivity system (“interactivity system”) includes paired mixed-input devices for interacting with and controlling virtual objects using displays, real inputs and virtual inputs, sensors, and passive and active haptic feedback associated with the paired mixed-input devices. Real device space tracker data and virtual device space tracker data is accessed via the paired mixed-input devices, respectively, to determine real input and virtual input that are integrated and processed to determine selection inputs and annotation inputs. The real device space tracker data and virtual device space tracker data are also used in different interaction contexts for determining selection inputs and annotation inputs.
The interactivity system includes an auxiliary mixed-input device (e.g., a selection device or annotation device) and a virtual mixed-input device (e.g., a mixed-reality device). The auxiliary mixed-input device is configured to determine real input based on real device space tracker data associated with the auxiliary mixed-input device. The auxiliary mixed-input device is a mobile device having a first display that displays a virtual object. The virtual mixed-input device is configured to determine virtual input based on virtual device space tracker data associated with the virtual mixed-input device. The virtual mixed-input device is a head-mounted display having a second display that displays the virtual object. The virtual object is an augmented reality object via the second display.
An integrated processing component is configured to determine a selection input or annotation input based on the real input and the virtual input. The selection input is used to determine a selected portion of the virtual object. The annotation input is used to determine an annotation for the virtual object. The integrated processing component includes a selection manager configured to process selection inputs and interfaces and an annotation manager configured to process annotation inputs and interfaces. The integrated processing component is further configured to cause display on the first display associated with the auxiliary mixed-input device the selected portion or the annotation of the virtual object. The integrated processing component is further configured to cause display on the second display associated with virtual mixed-input device the selected portion or the annotation of the virtual object.
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in isolation as an aid in determining the scope of the claimed subject matter.
The present invention is described in detail below with reference to the attached drawing figures, wherein:
Virtual experience computing systems (e.g., virtual reality or augmented reality devices) receive input from input devices to select objects in virtual environments and augmented reality environments (i.e., collectively virtual experiences). An input device can refer to a peripheral used to provide data and control signals for processing via an input processing system of a virtual experience device (e.g., head mounted display device) or mobile device (e.g., smartphone). Virtual experiences (e.g., augmented reality, immersive games or other multidimensional experiences) use different types of input devices. For example, a virtual reality controller can be used to make a selection in a virtual experience based on tracking motion of the virtual reality controller, where the motion is communicated to a virtual reality computing system of the virtual experience to indicate the selection. Selecting virtual reality objects or augmented reality objects (“virtual objects”) can further be complimented with annotation inputs to associate additional information with the selections. In particular, selection and annotation of augmented reality objects can be performed using an augmented reality device that supports the presentation of augmented reality images. Augmented reality images include augmented reality objects, which are virtualized objects or entities (e.g., holographic content or mixed-reality content), that are rendered in a real world environment of a user associated with the augmented reality device. The augmented reality objects can be rendered based on the real world environment, where the real world environment can be captured in a digital representation by the augmented reality device. Understanding the real world environment can be based on several different techniques that provide the augmented reality device or mixed-reality device with information about the environment. For example, an environment can be scanned in real-time and a mesh representation of the environment can be dynamically generated to provide the augmented reality device with information about the environment. As such, augmented reality devices can support augmented reality or mixed-reality experiences. Such experiences include selecting in the real world environment augmented reality objects or annotating augmented reality objects in combination with the real world environment. Augmented reality objects can include a hologram catalogue of 3D objects or virtual images that can be placed and scaled around a user. Augmented reality objects can be used in a variety of different implementations (e.g., video games, entertainment, productivity tools, etc.) as entities that can be controlled through a graphical user interface from actuators from the user via the mixed-input pointing device described herein.
With conventional augmented reality virtual computing systems, selecting and annotating virtual objects can be challenging because of the inherent properties of the virtual reality interactivity. At a basic level, selecting a portion of a virtual object can be cumbersome because a selection interface can impede actions associated with making the selection. For example, a user may have to hold their hand in front the virtual object during selection; however the user risks occluding, with their hand, the virtual object being selected. In addition, current selection and annotation methods in virtual reality can also be inconsistent with traditional intuitive computing selection methods that users have become accustomed to in non-virtual computing interfaces. Virtual experiences offer 3D interactivity; however users would like to use familiar 2D data input paradigms. Users in virtual experiences lack traditional tools in virtual experiences, making identifying selections and providing notations in a virtual experience context difficult. For example, the free space display of augmented reality virtual objects provides a singular context for making selections and annotations. As such, a comprehensive interactivity system can be implemented to improve selection and annotation of virtual objects for users in virtual experiences.
Embodiments of the present inventions provide simple and efficient methods and systems for providing a multi-device mixed interactivity system for selection and annotation. At a high level, the multi-device mixed interactivity system (“interactivity system”) includes paired mixed-input devices for interacting and controlling virtual objects using displays, real inputs and virtual inputs, sensors, and passive and active haptic feedback associated with the paired mixed-input devices. The interactivity system includes several components including: an auxiliary mixed-input device (i.e., a selection device or annotation device) having a first display, a virtual mixed-input device (i.e., a mixed-reality device) having a second display, an integrated processing component, on the auxiliary mixed input device and/or the virtual mixed-input device, having a selection manager and an annotation manager. Real device space tracker data and virtual device space tracker data is accessed via the paired mixed-input devices, respectively, to determine real input and virtual input that are integrated and processed to determine selection inputs and annotation inputs. The real device space tracker data and virtual device space tracker data are also used in different interaction contexts for determining selection inputs and annotation inputs. An interaction context determines the operations, interfaces and controls for interacting with a virtual object.
In operation, for a selection operation, a selection input associated with a virtual object is accessed. The selection input is based on real input associated with a selection device and virtual input associated with a mixed-reality device. The selection device has a first display that displays the virtual object and the mixed-reality device has a second display that displays the virtual object. A selection profile is generated based on the selection input. The selection profile comprises one or more selection profile attributes for isolating a portion of the virtual object. A selected portion of the virtual object is determined based on the selection profile. The selected portion of the virtual reality object is caused to be displayed on the first display of the selection device.
In operation, for an annotation operation, a selection profile associated with a virtual object is accessed. The selection profile is generated based on a selection input determined using real input associated with a selection device and virtual input associated with a mixed-reality device. The selection device has a first display and the mixed-reality device has a second display that both display the virtual object. An annotation input for the virtual object based on a selected portion corresponding to the selection profile is received. An annotation profile based on the annotation input is generated. The annotation profile includes annotation profile attributes for annotating a portion of the virtual object. An annotation of the selected portion of the virtual reality object is caused to be displayed.
Embodiments of the interactivity system can be described based on a specific exemplary implementation and corresponding components. The features of the embodiment described below are meant to be exemplary and not intended to limit the scope of the invention, unless otherwise stated. At a high level, multi-device mixed interactivity system for selection and annotation uses a mobile device (e.g., tablet, smartphone), in particular a display and user interface (e.g., touchscreen interface) of the mobile device, a mixed-reality device (e.g., head-mounted display) and the display of the mixed-reality device to provide selection operations and annotation operations in augmented reality experiences. It is contemplated that touchscreen interface can refer to a touch surface interface that does not include a display.
Users operating data and visual interfaces often identify portions of the visual interfaces for interaction and selection of specific elements. Embodiments of the present disclosure can be described based on interaction contexts that demonstrate the functionality of the interactivity system. As discussed, selection inputs and annotation inputs are based on real input and virtual input corresponding to real device space tracker data and virtual device space tracker data. In particular, real device space tracker data (e.g., finger motion, display or touchscreen input, smartphone orientation, etc.) can generally refer to data that captured via the mobile device (i.e., auxiliary mixed-input device) and virtual device space tracker data (e.g., hand motion, head motion, eye motion, auxiliary mixed-input device orientation etc.) can generally refer to data that is captured via the HMD (i.e., virtual mixed-input device). The mobile device and HMD can also support casting for further processing selection inputs. For example, casting (e.g., ray-casting or shape-casting) can be identify a portion of the virtual object to be selected or define a volume selection for selection inputs.
Integrating and processing the real input and virtual input supports determining the selection inputs and annotation inputs. An integrated processing component can be implemented via the mobile device and/or the HMD. The mobile device has several sensors that support tracking and the HMD also has several sensors that support tracking to define real input and virtual input for determining selection inputs and annotation inputs. The real device space tracker data and virtual device space tracker data are also used to generate interaction contexts. An interaction context determines the operations, interfaces and controls for interacting with a virtual object. In particular, context-based interfaces can be used to make selection configurations and annotation configurations with corresponding interface elements for the context-based interface.
The interactivity system can support initiating different interaction contexts in order to begin a selection and/or annotation. An interactivity context can be initiated in several ways. For example, a user can execute controls (e.g., gesture, voice, selection input) on a user interface of the mobile device or HMD to initialize an interaction context. User interface as used herein can refer to a mobile device and/or the HMD user interface unless stated otherwise. Based on the interaction context, the mobile device display can operate to display the virtual object using different display configuration. The mobile device display can mirror or substantially mirror (i.e., at least portions of the HMD display of the virtual object are mirrored) the HMD display of the virtual object. In this regard, the mobile device display and the HMD display can be in sync for viewing the virtual object. The mobile device display can be operate independently of the HMD display and further operate to receive selection inputs and annotation inputs and other interaction controls for the virtual object. It is contemplated that the different context can be manually triggered based on user input or automatically triggered based on positioning of the mobile device.
In a freehand device motion selection context, a mobile device can be used as a freehand selection tool (e.g., a lasso) in space. The mobile device is used to make a selection by moving the mobile device to trace a portion of the virtual object for a selection. The user can make a selection by moving the mobile device relative to a virtual object to trace a selection of the virtual object. Relative to a virtual object can refer to moving mobile device across the virtual object projected in the real world or moving the mobile device at a position away from the virtual object with reference to the virtual object. In a device interface selection context, a display of the mobile device can be used for receiving a selection input (i.e., input data). For example, display can be a touchscreen interface that can receive user freehand selection input with reference to a virtual object or a selection input based on predefined selection inputs or other selection inputs for isolating one or more parts of the virtual object.
It is contemplated that the selection can occur directly (i.e., direct interaction) where the mobile device overlaps portions of the display of virtual object for making a selection or indirectly (i.e., indirect interaction) where the mobile device relative to where the virtual object is displayed via the HMD. In addition, a selection input on the virtual object can be performed on a virtual object at different display configuration (i.e., scale or orientation display of the virtual object). For example, the size or orientation of the virtual object can be increased or reduced to improve ease of selection depending on the specific user intent for the selection.
Selection inputs are processed to determine selected portions of the virtual object. In particular, a selection input is used to define a selection profile of the selection input. The selection profile supports isolating a portion of the virtual object. The selection profile can be based on a freehand device motion selection input, freehand selection input via a touchscreen interface, or a selection input of a predefined selection profile. Selection input can be dynamically defined based on different types of real inputs, virtual inputs or additional inputs. A predefined selection profile can be selected from any number of pre-defined shapes (e.g., circle, square, octagon, etc.) that are selectable via the user interface. In this regard, the selection profile can refer to a defined shape that is used for identifying a selected portion of a virtual object.
The selection profile as defined as a shaped can be displayed via the user interface. The selection profile can be displayed on the mobile device display and/or HMD display with reference to the virtual object. The selection profile can include selection attributes which are specific properties of the selection profile. The selection profile attributes can include size, length, display configuration, visualization, haptic feedback, and additional metadata associated with the selection operation. The selection profile includes attributes that instruct on how to isolate, present, and interact with portions of a virtual object. For example, a display configuration attribute of the selection profile can indicate display the selected profile and a visualization attribute indicates how to distinguish a selected portion. The selection profile can be stored as a data structure defined as a selection profile object. The selection profile object stores the attributes of the selection profile. The user interface can support modifying attributes of the selection profile. For example, a size or length of the selection profile can be modified via the touchscreen interface of the mobile device.
The selection input can also be used to define the selection profile that is further augmented as a volume selection (i.e., volume selection attribute) based on the initial selection profile. For example, a selection loop or closed curve selection input can define the selection profile. A volume selection can be generated based on an extrusion of the selection profile. In an exemplary embodiment, the selection profile can be extruded automatically extruded or manually. Further, the selection profile can be extruded in an orthogonal direction from the mobile device, in particular, according to the current mobile device location and orientation. In another exemplary embodiment, the extrusion can simply be longitudinal extrusion of the selection profile.
A selection profile can be based on two selection inputs. A compound selection profile can be generated based on combining of determining an intersection of two selection inputs or two existing selection profiles to generate a selected portion of the virtual object based on both. A compound selection profile can be generated based selection input define by a on a user walking around the virtual object making multiple selections or based on a collaborative selection between two mobile devices. In this regard, the selection profile can be based on selection inputs from multiple perspectives.
The selection profile is used to isolate a portion of the virtual object. A selected portion of the virtual object can be determined based on identifying portions of the virtual object surrounded or enclosed by the selection profile. The selected portion is displayed based on a specific interaction context. For example, the selected portion can be displayed on the mobile device and/or the HMD based on continuing operations with the selected portion. In particular, the selection profile attributes can instruct on how the selected portion is displayed and where to display the selected portion. For example, a visualization attribute can instruct on interface elements (e.g., outlines, color, and text) can be used to display the selected portion.
The interactivity system is operable support several interactivity features to enhance the selection operation, including: selecting and presenting context-based interfaces where specific interface menus are presented based on the interaction context, interface elements that provide visual feedback of the selection operation, haptic feedback for the selection operation, collaborative selection between two mobile devices, additional input modes (e.g., gesture, voice) and mapping of the virtual object display between the mobile device and the HMD. To further describe aspects of the above-identified features by way of examples, a user may scroll through or be presented with different selection menus for providing selection inputs and identify and modifying selection profiles. In order to present a selection menu, the interactivity system determines the active interaction context and provides a selection menu specifically based on the context. The selection menu within a particular interaction context determines the operations and controls for selection inputs and selection profiles interactivity. It is contemplated that the context-based interfaces functionality can be supported via the mobile device or in combination with the HDM device.
Further, with reference to visual feedback, the display of the virtual object can be highlighted with different types of interface elements to show the selection profile, volume selection, the selected portion of the virtual object and further distinguish them from unselected portions of the virtual object. In a collaborative selection, for example, a selection from a first mobile device can be highlighted different from a selection of a second mobile device. It is contemplated that a combination or an intersection of a collaborative selection from two mobile devices can define the identified compound selection profile. The selection operation can also be enhanced with haptic feedback as the user makes selection inputs and identifies and adjust different selection configurations of the selection profile. Haptic feedback can be triggered via the mobile device and/or the HMD. For example, haptic feedback can be generated when performing certain operations or when selecting different subsection of a virtual object.
Several other user interface controls (e.g., gaze, gestures, audio and touch input) can be implemented via the interface to further define, refine, and adjust attributes of the selection profile. For example, the user eye position and the virtual object on the mobile device can define a frustum for making a selection input and corresponding selection profile of the virtual object, in combination with, making an input selection for a position attribute of the selection profile. Speech recognition inputs can also control aspects of the selection operations. For example, a speech command can be used to add or subtract sub-volumes of the selection profile. It is contemplated that a vocabulary for speech commands for additional controls can be defined explicitly for the selection operation to increase rate of success of the specific controls associated with the selection process.
In one embodiment, the mobile device display surface can mapped to the virtual object display in different configurations for receiving selection input. The surface of the mobile device can be mapped as 3D or two-dimensional (2D) interfaces relative to the virtual object. For example, the mapping can be spherical coordinate mapping the volume selection profiles. In this regard, a location of the user's finger is mapped to the space where a selection profile is located to make adjustments to the selection profile. The mapping of the mobile device to the virtual object can be defined such that a display of the virtual object via the HMD is based on input from the touchscreen of the mobile device. For example, when a finger of the user is approaching the touchscreen, the corresponding location is displayed on the display of the virtual object via the HMD. The location can be computed in several different ways. For example, the mobile device display can represent a point of view of the user of the virtual object in the HMD display. The center of the mobile device display can correspond to the center of the user view. In this regard, the user can interface via the mobile device display to a direct mapping to the virtual object in the HMD display (e.g., homographic transformation).
As mentioned, the interactivity system can also support interaction contexts for annotation. In particular, an annotation operation and features are interrelated with the selection operation and features described above. In this regard, the annotation operation is an extension of the selection operation with additional annotation functionality as described below. By way of example, with an annotation operation, a user can provide analysis or communicate information via annotation input that is captured based on annotation profile (i.e., an annotation profile object) associated with a virtual object.
The interactivity system can support initiating an interaction context for annotation, as discussed above. A user can trigger an annotation mode or an annotation mode can be automatically triggered. In a selection profile annotation context, a user can simply select an existing selection profile associated with the virtual object to make an annotation on the selected portion of the virtual object associated with the virtual object using the annotation device. In a freehand slice annotation context, a mobile device can be used to make selection by moving the mobile device to make a selection input based on a slice. A slice refers to a cross-section based representation of the virtual object. A selection profile based on a slice is associated with a selected portion of the virtual object that is represented as a cross section of the virtual object. During selection in the freehand slice annotation context, a display configuration of the virtual object on the mobile substantially mirrors a display configuration of the virtual object on the second display. The selection can be based on a direct interaction with the virtual object on the second display. In a freehand annotation context, a selection can be made similar to freehand device motion selection context, as discussed above. In a stamp annotation context, the annotation is received prior to making a selection. An annotation is received via the touchscreen of the mobile device and stamped (i.e., aligned with the geometry of the virtual object) by placing mobile device (directly or indirectly via an offset) of the virtual object.
Generally, annotation input is received with reference to a selection profile in order to generate an annotation profile. An integrated processing component can be implemented via the mobile device and/or the HMD to support receiving annotation inputs. Different types of annotations that define annotation inputs are contemplated, as described below in more detail. In operation, mobile device has several sensors that support tracking and the HMD also has several sensors that support tracking to define real input and virtual input for determining selection inputs and annotation inputs. The real device space tracker data and virtual device space tracker data are also used to generate the interaction contexts and determine annotation input. The interaction context determines the operations, interfaces and controls for interacting with a virtual object. In particular, context-based interfaces can be used to make annotation configurations with corresponding interface elements for the context-based interface.
The annotation input is used to generate an annotation profile having annotation attributes. The annotation device can support receiving the annotation input using the mobile device. An annotation device operates similarly to a selection device for receiving selection inputs associated with annotations. Annotation inputs are processed to determine annotations for the virtual object. In particular, an annotation input is used to define annotation profile of the selection input or a selection input to be determined (e.g., stamped annotation). The annotation profile functionally operates similar to a selection profile as described above, however the annotation profile further includes attributes associated with the annotation operation and annotation metadata. The annotation profile is similar in that, a selection input prior to incorporating an annotation onto a selection profile of the selection input is based on the selection operation. For example, the selection profile for an annotation profile can be generated based on user-defined selection or a predefined selection generated via the mobile device. Also, predefined selection profile for an annotation profile can be a slice that identifies a singular cross section of the virtual object as a selected portion of the virtual object. It is contemplated that any other predefined selection profiles, as described hereinabove can be used with reference to the annotation profile. In this regard, the selection profile identified to receive an annotation input, and the annotation operation proceeds with generating the annotation profile with annotation.
The user interface can support modifying attributes of the annotation profile. The annotation profile attributes can include annotation notes, annotation type, display configuration, annotation instructions, and additional metadata associated with the annotation operation. An annotation profile can define and display annotations (e.g., an annotation icon, note, input) at specific location with based on the annotation profile. An annotation can be added via mobile device to a selected portion of the virtual object. Annotations can be textual, visual information, video and other types of media. The annotation can be integrated to selected virtual objects. Annotation can be managed, processing, accessed, interacted with and displayed like other types of virtual objects.
It is contemplated that different annotations profiles or annotation can be viewed from different distances. The annotation profile supports managing display and presentation elements of the annotations. For example, annotation instructions determine how the annotations can be retrieved and viewed. For example, an annotation instruction can indicate, for an annotation, the presentation and positioning information for a user to properly view the annotation. A display configuration attribute of the annotation profile can determine how to distinguish an annotation profile from the virtual object. For example, for a virtual object associated with multiple annotations, interface elements can be used to distinguish the different annotation profiles and further highlight an actively selected annotation profile on either displays of the virtual object as a user traverses the different annotation profiles.
The interactivity system can support several interactivity features to enhance the annotation operations, including: selecting and presenting context-based interfaces where specific interface menus are presented based on the context, interface elements that provide visual feedback of the annotation operation, haptic feedback for the annotation operation, collaborative annotation between two mobile devices, additional input modes (e.g., gesture, voice) and mapping of the virtual object display between the mobile device and the HMD. These interactivity features can be consistent with the description above for the selection operation and as such are not further discussed.
The multi-device mixed interactivity system for selection and annotation described above advantageously overcomes some of the shortfalls of conventional virtual reality selection and annotation tools. The solution implements mixed-input devices in combination to allow for improved selection inputs and annotation inputs as described above. The mobile device display also provides a traditional 2D input paradigm, high definition display, and passive and haptic feedback that users are accustomed to for selection and annotation. In addition, the interactivity system provide for flexible context-based interfaces and interactions, such as, the mobile device selectively switching between an augmented reality context and an auxiliary context to support interactions with and controls of a virtual object.
With reference to
The virtual mixed-input device 120 can be a head mounted display (“HMD”) augmented reality (“AR”) device and may be any type of HMD augmented reality device having one or more components of the HMD described below with reference to
Having identified various components of interactivity system 100, it is noted that any number of components may be employed to achieve the desired functionality within the scope of the present disclosure. The various components of
Other arrangements and elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by one or more entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory.
Turning to the components of
At a high level, the virtual mixed-input device 120 can operate to provide a virtual experience. The virtual experience can be an augmented reality experience that generates or display virtual object in a real world environment. The virtual mixed-input device 120 can include additional sensors for tracking motion to make selections and annotations with reference to the virtual experience. The virtual mixed-input device 120 can operate a virtual-mixed input component 130 for providing additional novel functionality associated with embodiments of the present disclosure.
The auxiliary mixed-input device 110 supports generating real-input for the interactivity system 100. The auxiliary mixed-input device 110 operates with the real device sensor 112, real device space tracker 114 and the haptic feedback component 118 to provide the functionality described herein. The real device sensor 112 can be a type of sensor transmitting a signal associated with particular the type of sensor. The real device sensor 112 can also, in addition or in the alternative, operate as a component that accesses sensor data. Different types of sensor data are contemplated with reference the auxiliary mixed-input device. Real device sensor 112 can specifically be associated with wide angle sensor and a hover sensor for capturing different types of movement. The real device space tracker 114 can operate with sensor data to determine real input. The real device space tracker 114 is responsible for identifying and classifying different types of real device space information (e.g., finger motion, display or touchscreen input, smartphone orientation etc.) which can be communicated as real input. Real input can refer to information gathered from the perspective of the auxiliary mixed-input device. The haptic feedback component 116 can provide active haptic feedback during different types of interactions via the auxiliary mixed-input device for real input, virtual input and determined selection inputs and annotation inputs. The auxiliary mixed-input device 110 supports a display 121A for displaying virtual objects and supporting functionality in accordance with embodiments described herein.
The virtual mixed-input device 120 supports generating virtual input for the interactivity system 100. The virtual mixed-input device 120 operates with the virtual device sensor 132 and the virtual device space tracker 134 to provide the functionality described herein. The virtual device sensor 132 can be a type of sensor transmitting a signal associated with the particular type of sensors. The virtual device sensor 132 can also, in addition or in the alternative, operate as a component that accesses sensor data. Different types of sensor data are contemplated with reference the virtual mixed-input device. The virtual device space tracker 134 can operate with sensor data to determine virtual input. The virtual device space tracker 134 is responsible for identifying and classifying different types of virtual device space information (e.g., hand motion head motion, eye motion, smartphone orientation etc.) which can be communicated as virtual input. Virtual input can refer to information gathered from the perspective of the virtual mixed-input device. The virtual mixed-input device 120 supports a display 120A for displaying virtual objects and supporting functionality in accordance with embodiments described herein.
The auxiliary integrated processing component 116 and the integrated processing component 140 (collectively “mixed-input integrated processing components”) support generating selection inputs and annotation inputs for the interactivity system 100. The integrated mixed-input processing components can receive the real input and virtual input to determine selection input and annotation input for a virtual object.
The mixed-input integrated processing components can further use the real input and virtual input via their respective selection managers and annotation managers to support different types of interaction contexts as discussed in more detail below. For example, the auxiliary integrated processing component 116 can operate with the selection manager 116A and the annotation manger 116B to provide interaction contexts associated with the auxiliary mixed-input device 110 while the integrated processing component 140 can operate with the selection manager 142 and annotation manager 144 to provide interaction contexts associated with the virtual mixed-input device 120. The mixed-input integrated processing components can access, receive or generate the real input and the virtual input to determine the virtual space input. The selection managers and annotation managers can operate to selectively or automatically generate different types of interaction contexts based on real device space tracker data and virtual device space tracker data. The auxiliary integrated processing component 116 can support can selectively switching the display between an augmented reality context and an auxiliary context to support interactions with and controls associated with virtual object.
Embodiments described herein can further be described based on exemplary selection and annotation operations and interfaces provided based on components of the interactivity system 100. For example, the interactivity system 100 can support selection contexts and interfaces and annotation contexts and interfaces via the auxiliary mixed-input device 110 (e.g., selection device or annotation device) and the virtual mixed-input device 120. The display 110A of the auxiliary mixed-input device 110 can automatically or manually switch between an augmented reality context and an auxiliary context to support selection inputs and annotation inputs for a virtual object. Similarly, the display 120A of the virtual mixed-input device 120 can automatically or manually switch between an augmented reality context and an auxiliary context to support selection inputs and annotation input for a virtual object. Real input based on real device space tracker data and virtual input based on virtual device space tracker data can be generated via auxiliary mixed-input device 110 and the virtual mixed-input device respectively. Real device space tracker data and virtual device space tracker data can be associated with different types of interaction contexts that further define the real input and the virtual input that are integrated and processed to generate a selection input or an annotation input, where tracker data is determined based at least in part on sensor data. The selection input or an annotation input is managed, based on a determined context, using the auxiliary integrated process component 116 and specifically the selection manager 116A or the annotation manager 116B and similarly, the selection input or an annotation input is managed, using the integrated processing component 140 and specifically the selection manager 142 or the annotation manager 144. As used herein, managing can refer to accessing, receiving, processing, referencing, communicating, comparing, translating, triggering, computing instructions, messages, interactivity features and causing display of virtual objects in accordance with embodiments of the present disclosure.
With reference to
An integrated processing components (e.g., auxiliary integrated processing component 116 and integrated processing component 140) are configured to determine a selection input or annotation input based on the real input and the virtual input. The selection input is used to determine one or more selected portions of the virtual object based on a selection profile. The annotation input is used to determine one or more annotations for the virtual object based on an annotation profile.
With reference to
With reference to
With reference to
With reference to
Turning to
With reference to
With reference to
With reference to
With reference to
With reference to
With reference to
Turning to
Turning to
Turning to
Turning to
With reference to the multi-device mixed interactivity system, supports paired mixed-input devices for interacting with and controlling virtual objects using displays, real inputs and virtual inputs, sensors, and passive and active haptic feedback associated with the paired mixed-input devices. The multi-device mixed interactivity system components refer to integrated components for multi-device mixed interactivity. The integrated components refer to the hardware architecture and software framework that support distributed traffic management functionality using distributed the traffic management system 100. The hardware architecture refers to physical components and interrelationships thereof and the software framework refers to software providing functionality that can be implemented with hardware embodied on a device. The end-to-end software-based multi-device mixed interactivity system can operate within the multi-device mixed interactivity system components to operate computer hardware to provide multi-device mixed interactivity system functionality. As such, the multi-device mixed interactivity system components can manage resources and provide services for the multi-device mixed interactivity system functionality. Any other variations and combinations thereof are contemplated with embodiments of the present invention.
By way of example, the multi-device mixed interactivity system can include an API library that includes specifications for routines, data structures, object classes, and variables may support the interaction between the hardware architecture of the components and the software framework of the multi-device mixed interactivity system. These APIs include configuration specifications for the multi-device mixed interactivity system such that the different components therein can communicate with each other in the multi-device mixed interactivity system, as described herein.
With reference to
Turning to
Light ray representing the virtual image 1102 is reflected by the display component 1128 toward a user's eye, as exemplified by a light ray 1110, so that the user sees an image 1112. In the augmented-reality image 1112, a portion of the real-world scene 1104, such as, a cooking oven is visible along with the entire virtual image 1102 such as a recipe book icon. The user can therefore see a mixed-reality or augmented-reality image 1112 in which the recipe book icon is hanging in front of the cooking oven in this example.
Other arrangements and elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by one or more entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory.
Having described embodiments of the present invention, an exemplary operating environment in which embodiments of the present invention may be implemented is described below in order to provide a general context for various aspects of the present invention. Referring initially to
The invention may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules including routines, programs, objects, components, data structures, etc. refer to code that perform particular tasks or implement particular abstract data types. The invention may be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general-purpose computers, more specialty computing devices, etc. The invention may also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network.
With reference to
Computing device 1200 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 1200 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media.
Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 1200. Computer storage media excludes signals per se.
Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.
Memory 1212 includes computer storage media in the form of volatile and/or nonvolatile memory. The memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical-disc drives, etc. Computing device 1200 includes one or more processors that read data from various entities such as memory 1212 or I/O components 1220. Presentation component(s) 1216 present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc.
I/O ports 1218 allow computing device 1200 to be logically coupled to other devices including I/O components 1220, some of which may be built in. Illustrative components include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc.
Embodiments described in the paragraphs above may be combined with one or more of the specifically described alternatives. In particular, an embodiment that is claimed may contain a reference, in the alternative, to more than one other embodiment. The embodiment that is claimed may specify a further limitation of the subject matter claimed.
The subject matter of embodiments of the invention is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms “step” and/or “block” may be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.
For purposes of this disclosure, the word “including” has the same broad meaning as the word “comprising,” and the word “accessing” comprises “receiving,” “referencing,” or “retrieving.” Further the word “communicating” has the same broad meaning as the word “receiving,” or “transmitting” facilitated by software or hardware-based buses, receivers, or transmitters” using communication media described herein. Also, the word “initiating” has the same broad meaning as the word “executing or “instructing” where the corresponding action can be performed to completion or interrupted based on an occurrence of another action. In addition, words such as “a” and “an,” unless otherwise indicated to the contrary, include the plural as well as the singular. Thus, for example, the constraint of “a feature” is satisfied where one or more features are present. Also, the term “or” includes the conjunctive, the disjunctive, and both (a or b thus includes either a or b, as well as a and b).
For purposes of a detailed discussion above, embodiments of the present invention are described with reference to a distributed computing environment; however the distributed computing environment depicted herein is merely exemplary. Components can be configured for performing novel aspects of embodiments, where the term “configured for” can refer to “programmed to” perform particular tasks or implement particular abstract data types using code. Further, while embodiments of the present invention may generally refer to the interactivity system and the schematics described herein, it is understood that the techniques described may be extended to other implementation contexts.
Embodiments of the present invention have been described in relation to particular embodiments which are intended in all respects to be illustrative rather than restrictive. Alternative embodiments will become apparent to those of ordinary skill in the art to which the present invention pertains without departing from its scope.
From the foregoing, it will be seen that this invention is one well adapted to attain all the ends and objects hereinabove set forth together with other advantages which are obvious and which are inherent to the structure.
It will be understood that certain features and sub-combinations are of utility and may be employed without reference to other features or sub-combinations. This is contemplated by and is within the scope of the claims.
This application is a Continuation of U.S. patent application Ser. No. 15/640,363, titled “ANNOTATION USING A MULTI-DEVICE MIXED INTERACTIVITY SYSTEM,” filed Jun. 30, 2017, which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15640363 | Jun 2017 | US |
| Child | 17335795 | US |
