A simulated reality system can be used to present simulated reality content on a display device. In some examples, simulated reality content includes virtual reality content that includes virtual objects that a user can interact with using an input device. In further examples, simulated reality content includes augmented reality content, which includes images of real objects (as captured by an image capture device such as a camera) and supplemental content that is associated with the images of the real objects. In additional examples, simulated reality content includes mixed reality content (also referred to as hybrid reality content), which includes images that merge real objects and virtual objects that can interact.
Some implementations of the present disclosure are described with respect to the following figures.
Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements. The figures are not necessarily to scale, and the size of some parts may be exaggerated to more clearly illustrate the example shown. Moreover, the drawings provide examples and/or implementations consistent with the description; however, the description is not limited to the examples and/or implementations provided in the drawings.
In the present disclosure, use of the term “a,” “an”, or “the” is intended to include the plural forms as well, unless the context clearly indicates otherwise. Also, the term “includes,” “including,” “comprises,” “comprising,” “have,” or “having” when used in this disclosure specifies the presence of the stated elements, but do not preclude the presence or addition of other elements.
Simulated reality content can be displayed on display devices of any of multiple different types of electronic devices. In some examples, simulated reality content can be displayed on a display device of a head-mounted device. A head-mounted device refers to any electronic device (that includes a display device) that can be worn on a head of a user, and which covers an eye or the eyes of the user. In some examples, a head-mounted device can include a strap that goes around the user's head so that the display device can be provided in front of the user's eye. In further examples, a head-mounted device can be in the form of electronic eyeglasses that can be worn in the similar fashion as normal eyeglasses, except that the electronic eyeglasses include a display screen (or multiple display screens) in front of the user's eye(s).
When viewing simulated reality content displayed by a head-mounted device or another type of electronic device, a user may not be able to see a real physical input device, which may be held in the users hand(s) or may be laying on a surface (e.g., desk surface, tabletop, or any other surface). For example, the head-mounted device may obstruct the user's view of the real world. As another example, when looking at the display device of an electronic device, the user's head may be turned away from the physical input device.
As a result, the user may not be able to determine where the user's fingers are relative to control elements of the physical input device. For example, if the physical input device is a keyboard, the user would not be able to determine which keys the user's fingers are over when the user is viewing simulated reality content on the head-mounted device. As another example, if the physical input device is a trackpad that has a trackpad surface and control buttons, the user would not be able to determine where the user's fingers are relative to the trackpad surface and the control buttons. As a further example, if the physical input device is a touch sensitive display screen, the user looking at simulated reality content while wearing a head-mounted device or looking at the display device of another type of electronic device may not be able to see control icons or other control elements displayed by the touch sensitive display screen.
In accordance with some implementations of the present disclosure, a virtual input device corresponding to a real input device can be displayed along with simulated reality content presented by a simulated reality system. The virtual input device can be part of the simulated reality content, or the virtual input device can be separately generated but can be overlaid over the simulated reality content.
As used here, a “real input device” refers to a physical input device that exists in the real world and that is actually used by a user to enter inputs to the simulated reality system. Inputs can be entered by user touch of a portion of the physical input device with the user's finger (or fingers), or using a different input object, such as a stylus, a digital pen, and so forth.
The pressure sensitive surface 104 is responsive to touch input on the pressure sensitive surface 104 using an input object (such as a user's finger, a stylus, a digital pen, etc.). In addition, the pressure sensitive surface 104 is responsive to different touch pressures applied on a particular location of the pressure sensitive surface, The different touch pressures are caused by different forces applied on the pressure sensitive surface at the particular location by the input object, As explained further below, different touch pressures applied on the particular location of the pressure sensitive surface 104 can cause different actions to be performed.
Although reference is made to use of a head-mounted device in some examples, it is noted that solutions according to some implementations of the present disclosure can be applied to other types of electronic devices, such as a handheld electronic device, a notebook computer, a desktop computer, and so forth.
The simulated reality system 106 can be implemented using a computer, such as a smartphone, a tablet computer, a desktop computer, a notebook computer, and so forth. In examples where the simulated reality system 106 is sufficiently small, such as a smartphone or a game appliance, the simulated reality system 106 can be part of or can be mounted to the head-mounted device 110.
In further examples, the simulated reality system 106 is separate from the head-mounted device 110, but is able to communicate with the head-mounted device 110 over a communication link 114, where the communication link 114 can include a wired link (in the form of a cable, for example) or a wireless link. A wireless link can include a Bluetooth link, a Wi-Fi network connection, a cellular network connection, and so forth.
Although the simulated reality system 106 and the real input device 102 are shown as separate components, it is noted that in other examples, the pressure sensitive surface 104 can be part of the simulated reality system 106.
The virtual input device management engine 108 can cause display of a virtual input device 204 on the display device 202 of
Note that although an arrangement of keys 206 of a keyboard is depicted as part of the virtual input device 204 of
As discussed further below, in other examples, a keyboard overlay can be provided over the pressure sensitive surface 104, where the keyboard overlay can include structures that correspond to keys of a keyboard to give the user 112 tactile feedback regarding where the user's fingers are on the real input device 102.
In other examples, the virtual input device 204 can be a different type of input device, such as a trackpad with control buttons, a touch sensitive display surface, and so forth. In each case, the real input device 102 may not actually depict such other types of input devices—rather, the pressure sensitive surface 104 of the real input device 102 may be a blank surface that is capable of detecting touch input at different locations.
The simulated reality system 106 further includes a simulated reality content generation engine 109 to generate simulated reality content 208 for display on the display device 202. As used here, “simulated reality content” can refer to any of the following: virtual reality content, augmented reality content, or mixed reality content. Virtual reality content refers to content that is completely simulated, with no depiction of any real object. Augmented reality content refers to a depiction of a real object (such as captured by a camera) along with supplemental content, which can include text, graphical images, or other information. Mixed reality content refers to content that merges captured images of real objects and virtual objects that can interact.
The simulated reality system 106 can cause the simulated reality content 208 to be displayed in conjunction with the virtual input device 204. The displayed virtual input device 204 can be part of the simulated reality content 208, or the displayed virtual input device 204 can be separately generated but can be overlaid over the simulated reality content 208.
In further examples, the simulated reality system 106 is able to track a position of the pressure sensitive surface 104. In some examples, the real input device 102 includes markings 120 that can be detected by the simulated reality system 106, such as by a camera or other sensor of the simulated reality system 106. As the real input device 102 is moved around, the camera or other sensor of the simulated reality system 106 is able to detect this movement based on tracking the markings 120. The markings 120 can include visible elements, invisible elements (e.g., infrared elements), optical sources, and so forth, that are detectable by a camera or another sensor.
In response to the tracking of the pressure sensitive surface 104 (and more specifically tracking the markings 120 provided on the real input device 102), the simulated reality system 106 is able to adjust a position of the virtual input device 204 as displayed by the display device 202.
The real input device 102 is able to detect a location of a touch input on the pressure sensitive surface 104, and an amount of pressure applied on the pressure sensitive surface 104 at the location. The real input device 102 is able to transmit over a communication link 111 information to the simulated reality system 106, where the transmitted information can indicate the location of a touch input and an amount of pressure applied at the location.
The virtual input device management engine 108 can receive indications of different touch pressures from the real input device 102 that includes the pressure sensitive surface 104.
In response to the user's finger 300 applying a first touch pressure on the location 302 on the pressure sensitive surface 104, the virtual input device management engine 108 causes the “S” key of the virtual keyboard 204 to be highlighted, such as by changing a color of the “S” key relative to the color of the other keys, or by darkening or lightening the “S” key relative to the other keys. Alternatively, touching the location 302 on the pressure sensitive surface 104 with the first touch pressure causes the “S” key to be enlarged from a normal size to an enlarged size, while the remaining keys of the virtual keyboard 204 remain at the normal size. The normal size of a key refers to a size of a key that is displayed when no touch input is made on the pressure sensitive surface 104.
As used here, a “first touch pressure” can refer to a pressure that is within a first predefined pressure range. Any touch pressure that is within the first predefined pressure range is considered to be a “first touch pressure.” A touch pressure refers to an amount of force imparted on a surface by an input object. Similarly, a “second touch pressure” can refer to a pressure that is within a second predefined pressure range. Any touch pressure that is within the second predefined pressure range is considered to be a “second touch pressure.”
The touching of the location 302 with the first touch pressure does not cause the virtual input device management engine 108 to activate the “S” key of the virtual keyboard 204. Rather, the highlighting of the “S” key of the virtual keyboard 204 provides feedback (in the form of the highlighting of the “S” key of the virtual keyboard 204 displayed on the display device 202 of
In other examples, instead of highlighting the “S” key in response to the first touch pressure applied by the user's finger 300 on the location 302 of the pressure sensitive surface 104, the first touch pressure applied on the location 302 can cause the virtual input device management engine 108 to display an input object over the “S” key in the image displayed by the display device 202. The input object can be a representation of the user's finger or some other input object, such as a stylus or digital pen.
More generally, in response to the first touch pressure applied on a given location of the pressure sensitive surface 104, a feedback indication relative to a control element of the virtual input device is provided to the user on the display device 202, where the feedback indication can include highlighting a control element of the virtual input device 204, or the feedback indication can include a representation of an input object provided over the control element.
If the user desires to activate a specific key, such as the “S” key, then the user can apply a greater second touch pressure (greater than the first touch pressure) onto a corresponding location of the pressure sensitive surface 104. For example, the user's finger 300 can apply the second touch pressure on the location 302 on the pressure sensitive surface 104, to cause the “S” key to be activated. The applied second touch pressure on the location 302 of the pressure sensitive surface 104 can be communicated to the virtual input device management engine 108, which interprets the detected second touch pressure as an indication that the “S” key is to be activated, This can cause a respective task to be performed. For example, if the user is entering text, then the letter “S” would be entered into a respective region displayed on the display device 202. In other examples, the “S” key can correspond to a different action, such as an action of an electronic game that is being played by the user.
With respect to the location 302 of the pressure sensitive surface 104, different tasks are performed by the simulated reality system 106 in response to different touch pressures applied on the location 302 of the pressure sensitive surface 104. A first task of the different tasks includes highlighting the “S” key of the virtual keyboard 204, while a second task of the different tasks corresponding to a second touch pressure causes activation of the “S” key.
More generally, a first touch pressure on a given location of the pressure sensitive surface 104 causes feedback to be provided to the user regarding the touching of a control element of a virtual input device displayed by the display device 202, while a second touch pressure applied on the given location of the pressure sensitive surface 104 causes activation of the control element of the virtual input device 204.
In further examples, the user 112 is able to toggle between different types of input devices for use with the simulated reality system 106. The toggling to different modes can be performed in response to activation of a physical control element 121 on the real input device 102, where the control element 121 can include a control button, a control switch, or any other type of control element. In other examples, the mode toggling control element can be a virtual mode toggling control element 220 displayed on the display device 202. The user can activate the virtual mode toggling control element 220, such as by touching a respective location on the pressure sensitive surface 104, or touching some other input control element.
When the user activates the mode toggling control element (121 or 220), the virtual input device management engine 108 receives an indication of actuation of a mode toggle control element. In response to the actuation of the mode toggle control element, the virtual input device management engine 108 causes a switch from the display of a first virtual input device (e.g., a virtual keyboard) to a display of a different second virtual input device (e.g., a trackpad, a touchscreen, an arrangement of function buttons, etc.). Once the switch has occurred and the second virtual input device is displayed, the virtual input device management engine 108 can receive an indication of touch input (at respective touch pressures) related to the second virtual input device on the pressure sensitive surface 104.
The simulated reality input device 800 includes a pressure sensitive surface 804 and a controller 806. The controller 806 can include a hardware processing circuit or a combination of a hardware processing circuit and machine-readable instructions executable on the hardware processing circuit. Examples of a hardware processing circuit can include any or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable gate array, a programmable integrated circuit device, and so forth.
The controller 806 can perform various tasks, which include a first touch pressure indicating task 808 that sends a first indication to a simulated reality system (e.g., 106 in
The controller 806 can further perform a second touch pressure indicating task 810, which sends a second indication to the simulated reality system in response to a different second touch pressure applied at the given location on the pressure sensitive surface 804.
The simulated reality input device 802 further includes a vibrator 904, which can cause vibration of a portion of the pressure sensitive surface 804, in response to any of the following: a user touching the pressure sensitive surface 804 with a first touch pressure, a user touching the pressure sensitive surface 804 with a second touch pressure, a control element being activated, or in response to the user touch on the pressure sensitive surface 804 moving between different locations of the pressure sensitive surface 804.
The simulated reality input device 802 also includes a mode toggle control element 906, which can be actuated by a user. In response to actuation of the mode toggle control element 906, the controller 806 can send a mode toggle indication 908 to cause the simulated reality system to switch from displaying a first virtual input device to displaying a different second virtual input device.
In the example of
In other examples, the physical overlay 1002 can be a different type of physical overlay, such as a physical overlay for a touchpad.
A processor performing a task can refer to a single processor performing the task, or to multiple processors performing the task. A processor can include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit.
The machine readable instructions further include second touch pressure responding instructions 1206 to activate the control element of a virtual input device in response to a second indication of a different second touch pressure applied on the given location of the pressure sensitive surface.
The storage medium 1200 can include any or some combination of the following: a semiconductor memory device such as a dynamic or static random access memory (a DRAM or SRAM), an erasable and programmable read-only memory (EPROM), an electrically erasable and programmable read-only memory (EEPROM) and flash memory; a magnetic disk such as a fixed, floppy and removable disk; another magnetic medium including tape; an optical medium such as a compact disk (CD) or a digital video disk (DVD); or another type of storage device. Note that the instructions discussed above can be provided on one computer-readable or machine-readable storage medium, or alternatively, can be provided on multiple computer-readable or machine-readable storage media distributed in a large system having possibly plural nodes. Such computer-readable or machine-readable storage medium or media is (are) considered to be part of an article (or article of manufacture). An article or article of manufacture can refer to any manufactured single component or multiple components. The storage medium or media can be located either in the machine running the machine-readable instructions, or located at a remote site from which machine-readable instructions can be downloaded over a network for execution.
In the foregoing description, numerous details are set forth to provide an understanding of the subject disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations from the details discussed above. It is intended that the appended claims cover such modifications and variations.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/US2017/028158 | 4/18/2017 | WO | 00 |