Patent Application
20150109191
Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
Computing devices such as personal computers, laptop computers, tablet computers, cellular phones, and countless types of Internet-capable devices are increasingly prevalent in numerous aspects of modern life. Over time, the manner in which these devices are providing information to users is becoming more intelligent, more efficient, more intuitive, and/or less obtrusive.
The trend toward miniaturization of computing hardware, peripherals, as well as of sensors, detectors, and image and audio processors, among other technologies, has helped open up a field sometimes referred to as “wearable computing.” In the area of image and visual processing and production, in particular, it has become possible to consider wearable displays that place a very small image display element close enough to a wearer's (or user's) eye(s) such that the displayed image fills or nearly fills the field of view, and appears as a normal sized image, such as might be displayed on a traditional image display device. The relevant technology may be referred to as “near-eye displays.” In some configurations, wearable computers can receive inputs from input devices, such as keyboards, computer mice, touch pads, and buttons. In other configurations, wearable computers can accept speech inputs as well or instead via voice interfaces.
Emerging and anticipated uses of wearable displays include applications in which users interact in real time with an augmented or virtual reality. Such applications can be mission-critical or safety-critical, such as in a public safety or aviation setting. The applications can also be recreational, such as interactive gaming.
In one aspect, an example method can include: (a) defining a range of voice-activation gaze directions using a computing device, (b) determining a gaze direction using the computing device, (c) determining whether the gaze direction is within the range of voice-activation gaze directions using the computing device, and (d) in response to determining that the gaze direction is within the range of voice-activation gaze directions, activating a voice interface of the computing device.
In another aspect, an example computing device can include a processor, a voice interface, a non-transitory computer-readable medium and program instructions stored on the non-transitory computer-readable medium. The program instructions are executable by the processor to cause the wearable computing device to perform functions. The functions can include: (a) defining a range of gaze directions, wherein each gaze direction in the range of gaze directions is capable of triggering activation of the voice interface (b) determining a gaze direction, (c) determining whether the gaze direction is within the range of gaze directions, and (d) in response to determining that the gaze direction is within the range of gaze directions, activating the voice interface.
In yet another aspect, an article of manufacture can include a non-transitory computer-readable medium having instructions stored thereon that, when executed by a computing device, cause the computing device to perform functions. The functions can include: (a) defining a range of voice-activation gaze directions, (b) determining a gaze direction, (c) determining whether the gaze direction is within the range of voice-activation gaze directions, and (d) in response to determining that the gaze direction is within the range of voice-activation gaze directions, activating a voice interface.
These as well as other aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings.
One problem in using a voice interface is the “voice segmentation problem”, which is a problem of determining when to activate and deactivate the voice interface. The voice segmentation problem involves segmenting speech (or other audio information) into a portion of speech which is directed to a speech recognition system of the voice interface and a portion of speech that is directed to other people. A desired solution to the voice segmentation problem would enable both easy switching between speaking to the speech recognition system and speaking to human conversation partners and clearly indicating to whom each speech action is directed.
One approach to address the voice segmentation problem is to use gaze detection to activate or deactivate a voice interface. In particular, a gaze can be detected in a range of “voice-activation gaze directions”, where a voice-activation gaze direction is a gaze direction capable of triggering activation, deactivation, or toggling of an activation state of a voice interface. In the context of wearable computers with heads-up displays, when a wearer is gazing straight ahead, such as when looking directly at a conversational partner, the system can recognize that that the wearer's gaze is not directed in a voice-activation gaze direction and so speech is not to be directed to the voice interface.
However, when the wearer's gaze is directed to one or more predefined portions of the heads-up display, such as a portion positioned slightly upward from a typical gaze, then the system can recognize that the wearer's gaze is directed in a voice-activation gaze direction. This (slight) upward gaze can provide a social cue from the wearer to the conversational partner that the wearer is not currently involved in the conversation. By picking up the social cue, the conversational partner can recognize that any speech is not directed toward him/her, but rather directed elsewhere. Then, upon recognizing the gaze is in a voice-activation gaze direction, speech can be directed to the voice interface.
In other embodiments, gazing at an electromagnetic emissions sensor (EES) or a camera can toggle activation of the voice interface. For example, suppose a deactivated speech recognition system is equipped with a camera for detecting gazes. Then, in response to a first gaze at the camera, the speech recognition system can detect the first gaze as being in a voice-activation gaze direction and activate the speech recognition system. Later, in response to a second gaze at the camera, the speech recognition system can detect the second gaze as being in a voice-activation gaze direction and deactivate the speech recognition system. Subsequent gazes detected in voice-activation gaze directions can continue toggling an activation state (e.g., activated or deactivated) of the speech recognition system.
The concept of using gaze detection to activate speech recognition systems of voice interfaces can be expanded to other devices as well. For example, in the home, a television set could have a camera mounted out of the way of normal viewing angles. Then, when the television detected a television watcher was looking at the camera, the television could activate a speech recognition system, perhaps after muting any sound output of the television. The voice interface could be used to instruct the television set using voice commands, such as to change the channel or show a viewing guide, and then the television watcher could look away from the camera to stop using the speech recognition system. Other devices, such as, but not limited to, mobile phones, vehicles, information kiosks, personal computers, cameras, and other devices, could use gaze detection to activate/deactivate speech recognition system and voice interfaces as well.
Method 100 begins at block 110. At block 110, a computing device can define a range of voice-activation gaze directions. In some embodiments, defining a range of voice-activation gaze directions can include defining a range of social-cue gaze directions. The range of social-cue gaze directions can overlap the range of voice-activation gaze directions. In other embodiments, defining a range of voice-activation gaze directions can include defining a range of deactivation gaze directions. The range of deactivation gaze directions can be selected not to overlap the range of voice-activation gaze directions.
At block 120, the computing device can determine a gaze direction.
At block 130, the computing device can determine whether the gaze direction is within the range of voice-activation gaze directions.
At block 140, the computing device can, in response to determining that the gaze direction is within the range of voice-activation gaze directions, activate the voice interface of the computing device.
In some embodiments, in response to determining the gaze direction, the computing device can determine whether the gaze direction is within the range of social-cue gaze directions. Then, in response to determining that the gaze direction is within the range of social-cue directions, the computing device can activate the voice interface. Alternatively, in response to determining that the gaze direction is not within the range of social-cue directions, the computing device can deactivate the voice interface.
In other embodiments, after activating the voice interface, the computing device can receive speech input via the activated voice interface. A textual interpretation of at least part of the speech input can be generated. A command can be provided to an application, such as but not limited to a software application executing on the computing device, based on the textual interpretation.
In even other embodiments, the computing device can determine whether the gaze direction remains within the range of voice-activation gaze directions. In response to determining that the gaze direction does not remain within the range of voice-activation gaze directions, the computing device can deactivate the voice interface.
In yet other embodiments, the computing device can display a voice activation indicator on a display of the computing device. Then, the range of voice-activation gaze directions can comprises a range of gaze-directions from an eye toward the voice activation indicator. In particular of these embodiments, the voice activation indicator can be configured to indicate whether or not the voice interface is activated.
In still other embodiments, the computing device is configured to maintain an activation status of the voice interface that corresponds to the activation of the voice interface. That if, the voice interface is activated, the activation status is activated, and if the voice interface is not activated, the activation status is not activated. Then, the computing device can determine whether the gaze direction remains within the range of voice-activation gaze directions. In response to determining that the gaze direction does not remain within the range of voice-activation gaze directions, the computing device can maintain the activation status of the voice interface. Then, after maintaining the activation of the voice interface, the computing device can determine whether a later gaze direction is within the range of voice-activation gaze directions. In response to determining that the later gaze direction is within the range of voice-activation gaze directions, the computing device can toggle the activation status of the voice interface.
Example Scenario Using Gaze Detection for Voice
Scenario 200 continues, as shown in
Upon determining that gaze 204b is directed at voice activation indicator 210b, wearable computing device 202 can activate its voice interface. In some embodiments, such as shown in
Generally, wearable computing device 202 can designate one or more voice-indicator (VI) portions 212a, 212b of display 208 to be associated with activating/deactivating the voice interface, as shown in the bottom-left portion of
At the bottom-right portion of
When a gaze of wearer 230 is determined to be within a social-cue gaze-direction range 214a or 214b, wearable computing device 202 can activate the voice interface. For example, social-cue gaze-direction range 214a contains voice activation indicator 210b; that is, when the gaze of wearer 230 is at voice activation indicator 210b the gaze of wearer 230 can also be determined to be within social-cue gaze-direction range 214a, and thus activate the voice interface. In some embodiments not shown in the Figures, a voice activation indicator can be displayed within social-cue gaze-direction range 214b instead of or as well as voice activation indicator 210b displayed within social-cue gaze-direction range 214a.
In embodiments not shown in the Figures, the voice interface can be activated in response to both the gaze of wearer 230 being within a social-cue gaze-direction range 214a, 214b and at least one secondary signal. The secondary signal can be generated by wearer 230, such as a blink, an additional gaze, one or more spoken words (e.g., “activate speech interface”), pressing buttons, keys, etc. on a touch-based user interface, and/or by other techniques.
Also or instead, a secondary signal can be generated by wearable computing device 202. For example, if wearable computing device 202 determines that the gaze of wearer 230 is within a social-cue gaze-direction range 214a, 214b longer than a threshold period of time, such as 1-3 seconds, then wearable computing device 202 can generate the secondary signal. In some embodiments, the secondary signal can be generated before the gaze of wearer 230 is detected within a social-cue gaze detection range 214a, 21b. In other embodiments, use of the secondary signal in partially activating and/or confirming activation of the voice interface can be enabled and/or disabled, perhaps by wearer 230 interacting with an appropriate user interface of wearable computing device 202. In still other embodiments, multiple secondary signals can be required to confirm activation of the voice interface. Many other possibilities for generating and using secondary signals to partially activate and/or confirm activation of voice interfaces are possible as well.
If an eye of wearer 230 gazes through either social-cue gaze-direction range 214a or range 214b of display 208, then the conversational partner of wearer 230 can infer wearer 230 is not talking to the conversational partner. The conversational partner can make this inference via a social cue, since wearer 230 is not looking directly at the conversational partner. For example, when the eyes of wearer 230 gaze with gaze 204 in a gaze direction within social-cue gaze-direction range 214a, such as shown in
Turning to
In scenario 200, the first word “calculate” of speech 206c can be interpretation as a request for a calculator application. Based on this request, the speech recognition system can direct subsequently generated textual interpretations to the calculator application. For example, upon generating the textual interpretation of the remainder of speech 206c of “24 times 365,” the speech recognition system can then provide text of “24 times 365” to the calculator application.
The calculator application can generate calculator application text 220 indicating that the calculator application was activated with the output “Calculator”, output text and/or symbols confirming the reception of the words “24”, “times”, and “365” to text, and consequently output a calculated result of 8,760. Upon generating part or all of the output text and/or the calculated result, the calculator application can provide the output text and/or the calculated result to wearable computing device 202 for display. In other examples, other applications, output modes (video, audio, etc.), and/or calculations can be performed and/or used.
Wearable computing device 202 can use a timer and a threshold amount of time to determine that speech 206c is or is not being received. For example, the timer can expire when no audio input is received at the voice interface for at least the threshold amount of time. In some embodiments, the threshold amount is selected to be long enough to permit short, natural pauses in speech. In other embodiments, a threshold volume can be used as well. e.g., speech input 206c is considered to be received unless audio input falls below the threshold volume for at least the threshold amount of time.
Scenario 200 continues as shown in
Wearer 230 can then inform the conversational partner that a year has 8,760 hours via uttering speech 206d of
Example Head Mountable Devices for Determining Pupil Positions
Each of electromagnetic emitter/sensors 320a-320d can be configured to emit and/or sense electromagnetic radiation in one or more frequency ranges. In one example, each of electromagnetic emitter/sensors 320a-320d can be configured to emit and sense infrared light. The emitted electromagnetic radiation can be emitted at one or more specific frequencies or frequency ranges, such as an infrared frequency, to both aid detection and to distinguish the emitted radiation from background radiation, such as ambient light.
In some embodiments, the emitted electromagnetic radiation can be emitted using a specific pattern of frequencies or frequency ranges to better distinguish emitted radiation from background radiation and to increase the likelihood of detection of the emitted radiation after reflection from the eye. For example, one or more infrared or ultraviolet frequencies can be used as a specific pattern of frequencies or as part or all of a frequency range. In other embodiments, one or more of electromagnetic emitter/sensors 320a-320d can be implemented using a camera facing toward an eye of a wearer of head mountable device 300.
Electromagnetic emitter/sensors 320a-320d can be configured to emit electromagnetic radiation toward a right eye of a wearer of head mountable device 300 and subsequently detect electromagnetic radiation reflected from the right eye to determine a position of a portion of the right eye. For example, electromagnetic emitter/sensor 320a can be configured to emit and receive electromagnetic radiation at or near the upper-right-hand portion of the right eye and electromagnetic emitter/sensor 320c can be configured to emit and receive electromagnetic radiation at or near the lower-left-hand portion of the right eye of the wearer.
For example, suppose at a time TA, the iris and pupil of the right eye of the wearer were located at position A shown in
A position of the glint can be determined relative to other glints received by electromagnetic emitter/sensors 320a-320d to determine a relative direction of an iris and/or pupil of an eye. The iris and pupil of a human eye are covered by the cornea, which is a transparent, dome-shaped structure that bulges outward from the rest of the eyeball. The rest of the eyeball is also covered by a white, opaque layer called the sclera. As such, when emitted electromagnetic radiation strikes the eyeball, electromagnetic radiation reflected from the cornea and/or sclera can received at an electromagnetic emitter/sensor.
When electromagnetic radiation reflects from a leading surface of the cornea rather than the sclera (or a trailing surface of the cornea), the electromagnetic radiation can have less distance to travel before being reflected. As such, when the cornea is close to a specific sensor, a corresponding glint appears closer to the sensor as well. Also, when the cornea is farther from a specific sensor, a corresponding glint appears farther from the sensor.
As the sensors in head mountable device 300 are mounted on lens frame 308 near the edges of lens 306, when the cornea is near a closer edge of lens 306, corresponding glints appear closer to the closer edge. Thus, a pair of glints reflecting electromagnetic radiation emitted from sensors mounted on the closer edge appear farther apart than a pair of glints reflecting electromagnetic radiation emitted from sensors mounted on an edge opposite to the closer edge.
Based on the data about the received reflected electromagnetic radiation, a computing device, perhaps associated with head mountable device 300, can determine an estimated position PA of the iris and pupil of the right eye at TA is approximately centered within lens 308. That is, at time TA, electromagnetic emitter/sensors 320a-320d can emit electromagnetic radiation toward the right eye and the emitted light can be reflected from the surface of the right eye as a glint pattern. The resulting glint pattern can have a square or rectangular shape, with distances between each pair of horizontally aligned glints being roughly equal, and distances between each pair of vertically aligned glints being roughly equal.
As another example, suppose at a time TB, the cornea, including iris and pupil of the right eye of the wearer, is located at position B shown in
At time TB, electromagnetic emitter/sensors 320a-320d can emit electromagnetic radiation toward the right eye and the emitted light can be reflected from the surface of the right eye as a glint pattern. The resulting glint pattern can have a trapezoidal shape, with distances between each pair of horizontally aligned glints being roughly equal, and a distance between a pair of vertically aligned glints closest to position B being shorter than a distance between a pair of vertically aligned glints farthest from position B. The computing device can utilize this data to determine that the cornea, and corresponding pupil and iris, is closer to sensors 320a and 320b, and thus closer to a right edge of lens 308.
In some embodiments, each of electromagnetic emitter/sensors 320a-320d can be configured to emit and/or sense electromagnetic radiation in one or more frequency ranges. In one example, each of electromagnetic emitter/sensors 320a-320d can be configured to emit and sense infrared light. The emitted electromagnetic radiation can be emitted at one or more specific frequencies or frequency ranges, such as an infrared frequency, to both aid detection and to distinguish the emitted radiation from background radiation, such as ambient light. In other embodiments, the emitted electromagnetic radiation can be emitted using a specific pattern of frequencies or frequency ranges to better distinguish emitted radiation from background radiation and to increase the likelihood of detection of the emitted radiation after reflection from the eye.
Electromagnetic emitter/sensors 320a-320d can be configured to emit electromagnetic radiation toward an eye of a wearer of head mountable device 300 and subsequently detect reflected electromagnetic radiation to determine a position of a portion of the eye of the wearer.
For example, suppose at a time TD the iris and pupil of the right eye of the wearer were located at position D shown in
In this example, suppose the amounts of received light at each of electromagnetic emitter/sensors 320a-320d as shown in Table 2 below:
In other embodiments, one or more of electromagnetic emitter/sensors 320a-320d can provide more or less information about received light to head mountable device 300. As one example, the amount of received light can be expressed using either a single bit, with 0 being dark and 1 being light. As another example, a finer scale than a 1-10 scale can be used; e.g., a 0 (dark) to 255 (bright) scale. Additionally or instead, information about frequencies, direction, and/or other features of received light can be provided by one or more of electromagnetic emitter/sensors 320a-320d. Upon receiving the light, each of sensors 320a-320d can determine the amount of received light, generate an indication of the amount of received light using one or more of the numerical scales, and provide the indication to head mountable device 300.
Based on the information about received reflected electromagnetic radiation, a computing device, perhaps associated with head mountable device 300, can determine an estimated position of the iris and pupil PD of the right eye at TD. As the amount of reflected light received at sensor 320c is considerably smaller than the amounts of light received at sensors 320a, 320b and 320d, head mountable device 300 can determine that there is a relatively-large probability that PD is very near to sensor 320c. Then, considering that the amount of reflected light at sensor 320b is equal to the reflected light at sensor 320d, and the head mountable device 300 can determine that there is a relatively-larger probability that PD is equidistant between sensors 320b and 320d, and, based on the relatively large difference in light values between sensors 320a and 320c, that PD is on or near a line connected sensors 320a and 320c (equidistant from sensors 320b and 320d) and considerably closer to sensor 320c, and thus PD is near the lower-left edge of lens 308 very near to sensor 320c.
Other portions of the eye can be detected as well. For example, suppose each of electromagnetic emitter/sensors 320a-320d receive approximately equal amounts of received electromagnetic radiation, and each amount is relatively high. More light is reflected from a closed eye than from an open eye. Then a computing device, perhaps part of head mountable device 300, can infer the electromagnetic radiation is not being reflected from the eye, but from an eyelid.
In this case, by inferring the electromagnetic radiation is reflected from an eyelid, the computing device can infer that the eye is closed and that the wearer is either blinking (closed the eye for a short time) or has shut their eyes (closed the eye for a longer time).
To determine if the wearer is blinking or has shut their eyes, the computing device can wait a predetermined amount of time, and then request a second set of indications of reflected electromagnetic radiation from the electromagnetic emitter/sensors.
The predetermined amount of time can be based on a blink duration and/or a blink interval. In adult humans, a blink duration, or how long the eye is closed during a blink is approximately 300-400 milliseconds for intentional blinks and approximately 150 milliseconds for a reflex blink (e.g., a blink for spreading tears across the surface of the eye), and a blink rate, or how often the eye is blinked under typical conditions, is between two and ten blinks per minute; i.e., one blink per every six to thirty seconds. Using additional sets of indications of received electromagnetic radiation taken from another eye of the wearer can determine if the wearer has both eyes open, both eyes closed, or has one eye open; e.g., is winking. Also, indications of received electromagnetic radiation taken from another eye can be used to confirm values of received electromagnetic radiation when both eyes can be inferred to be closed.
Other techniques can be used to determine a position of an eye beyond those described herein.
Microphone 362 can capture the received audio and transmit captured audio 366 to speech recognition system 364, which in turn can process captured audio 366 to generate, as an output, a textual interpretation of captured audio 366. The textual interpretation of captured audio 366 is shown in
In some embodiments, display 370 is part of voice interface 360 and is perhaps part of a touch screen interface. In other embodiments, more or fewer sources of audio; e.g., speakers 350a and 352a, can be used with voice interface 360. In still other embodiments, more or fewer copies of text 368 can be generated by voice interface 360.
Additionally, at 400A of scenario 400, mobile device 410 includes an indicator 414a with “Stare for Voice” and an electromagnetic radiation emitter/sensor 416. If the user of mobile device gazes at indicator 414a for longer than a threshold period of time (a.k.a. stares at indicator 414a), then mobile device 410 can detect the gaze using electromagnetic-radiation emitter/sensor 416 and switch from utilizing the touch interface shown at 400A to utilizing a voice interface.
In some embodiments, electromagnetic radiation emitter/sensor 416 can include a camera to capture an image of an eye of the user. In other embodiments, electromagnetic radiation emitter/sensor 416 can emit a known frequency of electromagnetic radiation, wait a period of time for the emitted electromagnetic radiation to reflect from the eye to electromagnetic radiation emitter/sensor 416, and determine an eye position based on the reflected radiation received by electromagnetic radiation emitter/sensor 416 at mobile device 410. In still other embodiments, mobile device 410 can compare an eye position received at a first time to an eye position received at a later, second time to determine whether the eye positions at the first and second times indicate a user is staring at indicator 414a. Upon determining that the user is staring at indicator 414a, then mobile device can switch between touch and voice interfaces.
Scenario 400 continues at 400B with mobile device 410 utilizing a voice interface. The voice interface includes microphone 418 to capture sounds, including voice commands such as voice command 432, dialed digit display 420b to show previously-entered digits, phone/text indicator 428 to indicate if a communication is a voice call or text message and voice dialing display 430 to inform a user about voice commands that can be used with the voice interface. Mobile device 410 also includes a speaker 412 to output sounds to a user of mobile device 410.
As shown in
For example,
In some embodiments, voice dialing display 430 is not provided by mobile device 410 while utilizing the voice interface. In particular, a user of mobile device 410 may be able to select providing or not providing voice dialing display 430. In other embodiments, one or more languages other than English can be used by the voice interface.
Additionally, at 400B of scenario 400, mobile device 410 includes an indicator 414a with “Stare for Touch” and an electromagnetic radiation emitter/sensor 416. A gaze or stare at electromagnetic radiation emitter/sensor 416 can be detected using the techniques discussed above at least in the context of 400A of
By use of electromagnetic radiation emitter/sensors 516a-516d, multiple selections of interfaces can be utilized throughout the vehicle. For example, indicators 516a and 516c indicate that a user is to “stare for voice” to convert an interface from a touch interface to a voice interface, while indicators 516b and 516d indicate that a user is to “stare for touch” to convert an interface from a voice interface to a touch interface.
Beyond the movie player interface shown in
Example Systems and Devices
Systems and devices in which example embodiments may be implemented will now be described in greater detail. In general, an example system may be implemented in or may take the form of a wearable computer. However, an example system may also be implemented in or take the form of other devices, such as a mobile phone, among others. Further, an example system may take the form of non-transitory computer readable medium, which has program instructions stored thereon that are executable by at a processor to provide the functionality described herein. An example, system may also take the form of a device such as a wearable computer or mobile phone, or a subsystem of such a device, which includes such a non-transitory computer readable medium having such program instructions stored thereon.
As illustrated in
Each of the frame elements 604, 606, and 608 and the extending side-arms 614 and 616 may be formed of a solid structure of plastic or metal, or may be formed of a hollow structure of similar material so as to allow wiring and component interconnects to be internally routed through the head-mountable device 602. Other materials may possibly be used as well.
One or both of lens elements 610 and 612 may be formed of any material that can suitably display a projected image or graphic. One or both of lens elements 610 and 612 may also be sufficiently transparent to allow a wearer to see through the lens element. Combining these two features of lens elements 610, 612 can facilitate an augmented reality or heads-up display where the projected image or graphic is superimposed over a real-world view as perceived by the wearer through the lens elements.
The extending side-arms 614 and 616 each may be projections that extend away from the frame elements 604 and 606, respectively, and are positioned behind a wearer's ears to secure the head-mountable device 602. The extending side-arms 614 and 616 may further secure the head-mountable device 602 to the wearer by extending around a rear portion of the wearer's head. Additionally or alternatively, for example, head-mountable device 602 may connect to or be affixed within a head-mounted helmet structure. Other possibilities exist as well.
Head-mountable device 602 may also include an on-board computing system 618, video camera 620, sensor 622, and finger-operable touchpads 624, 626. The on-board computing system 618 is shown on the extending side-arm 614 of the head-mountable device 602; however, the on-board computing system 618 may be positioned on other parts of the head-mountable device 602 or may be remote from head-mountable device 602; e.g., the on-board computing system 618 could be wired to or wirelessly-connected to the head-mounted device 602.
The on-board computing system 618 may include a processor and memory, for example. The on-board computing system 618 may be configured to receive and analyze data from video camera 620, sensor 622, and the finger-operable touchpads 624, 626 (and possibly from other sensory devices, user interfaces, or both) and generate images for output from the lens elements 610 and 612 and/or other devices.
The sensor 622 is shown mounted on the extending side-arm 616 of the head-mountable device 602; however, the sensor 622 may be provided on other parts of the head-mountable device 602. The sensor 622 may include one or more of a gyroscope or an accelerometer, for example. Other sensing devices may be included within the sensor 622 or other sensing functions may be performed by the sensor 622.
In an example embodiment, sensors such as sensor 622 may be configured to detect head movement by a wearer of head-mountable device 602. For instance, a gyroscope and/or accelerometer may be arranged to detect head movements, and may be configured to output head-movement data. This head-movement data may then be used to carry out functions of an example method, such as method 100, for instance.
The finger-operable touchpads 624, 626 are shown mounted on the extending side-arms 614, 616 of the head-mountable device 602. Each of finger-operable touchpads 624, 626 may be used by a wearer to input commands. The finger-operable touchpads 624, 626 may sense at least one of a position and a movement of a finger via capacitive sensing, resistance sensing, or a surface acoustic wave process, among other possibilities. The finger-operable touchpads 624, 626 may be capable of sensing finger movement in a direction parallel or planar to the pad surface, in a direction normal to the pad surface, or both, and may also be capable of sensing a level of pressure applied. The finger-operable touchpads 624, 626 may be formed of one or more transparent or transparent insulating layers and one or more transparent or transparent conducting layers. Edges of the finger-operable touchpads 624, 626 may be formed to have a raised, indented, or roughened surface, so as to provide tactile feedback to a wearer when the wearer's finger reaches the edge of the finger-operable touchpads 624, 626. Each of the finger-operable touchpads 624, 626 may be operated independently, and may provide a different function.
The lens elements 610 and 612 may act as a combiner in a light projection system and may include a coating that reflects the light projected onto them from the projectors 628 and 632. In some embodiments, a special coating may not be used (e.g., when the projectors 628 and 632 are scanning laser devices).
In alternative embodiments, other types of display elements may also be used. For example, the lens elements 610, 612 themselves may include: a transparent or semi-transparent matrix display, such as an electroluminescent display or a liquid crystal display, one or more waveguides for delivering an image to the wearer, or other optical elements capable of delivering an in focus near-to-eye image to the wearer. A corresponding display driver may be disposed within the frame elements 604 and 606 for driving such a matrix display. Alternatively or additionally, a laser or light-emitting diode (LED) source and scanning system could be used to draw a raster display directly onto the retina of one or more of the wearer's eyes. Other possibilities exist as well.
While
The outward-facing video camera 620 is shown to be positioned on the extending side-arm 614 of the head-mountable device 602; however, the outward-facing video camera 620 may be provided on other parts of the head-mountable device 602. The outward-facing video camera 620 may be configured to capture images at various resolutions or at different frame rates. Many video cameras with a small form-factor, such as those used in cell phones or webcams, for example, may be incorporated into an example of wearable computing device 600.
Although
In some embodiments not shown in
As shown in
Display 708 is shown to be provided in a center of a lens of head-mountable device 702; however, the display 708 may be provided in other positions. The display 708 can be controlled using on-board computing system 704 coupled to display 708 via an optical waveguide 710.
Head-mountable device 802 may include a single lens element 830 configured to be coupled to one of the side-arms 823 and/or center frame support 824. The lens element 830 may include a display such as the display described with reference to
Thus, computing device 902 may include display system 930, processor 940, and display 950. Display 950 may be, for example, an optical see-through display, an optical see-around display, or a video see-through display. Processor 940 may receive data from remote device 920 and configure the data for display on display 950. Processor 940 may be any type of processor, such as a micro-processor or a digital signal processor, for example.
Computing device 902 may further include on-board data storage, such as memory 960 coupled to the processor 940. Memory 960 may store software that can be accessed and executed by the processor 940. For example, memory 960 may store software that, if executed by processor 940 is configured to perform some or all of the functionality described herein, for example.
Remote device 920 may be any type of computing device or transmitter including a laptop computer, a mobile telephone, or tablet computing device, etc., that is configured to transmit and/or receive data to/from computing device 902. Remote device 920 and computing device 902 may contain hardware to establish, maintain, and tear down communication link 910, such as processors, transmitters, receivers, antennas, etc.
In
Example methods and systems are described herein. The example embodiments described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein.
The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.
With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and/or communication may represent a processing of information and/or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and/or messages may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer blocks and/or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.
A block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and/or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.
The computer readable medium may also include non-transitory computer readable media such as computer-readable media that stores data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and/or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.
Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and/or hardware modules in the same physical device. However, other information transmissions may be between software modules and/or hardware modules in different physical devices.
While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.
