External user interface for head worn computing

Information

  • Patent Grant
  • 12174378
  • Patent Number
    12,174,378
  • Date Filed
    Tuesday, September 5, 2023
    a year ago
  • Date Issued
    Tuesday, December 24, 2024
    a day ago
Abstract
Aspects of the present invention relate to external user interfaces used in connection with head-worn computers (HWC).
Description
BACKGROUND
Field of the Invention

This invention relates to head worn computing. More particularly, this invention relates to external user interfaces related to head worn computing.


Description of Related Art

Wearable computing systems have been developed and are beginning to be commercialized. Many problems persist in the wearable computing field that need to be resolved to make them meet the demands of the market.


SUMMARY

This Summary introduces certain concepts of head worn computing, and the concepts are further described below in the Detailed Description and/or shown in the Figures. This Summary should not be considered to describe essential features of the claimed subject matter, nor used to determine or limit the scope of the claimed subject matter.


Aspects of the present invention relate to external user interfaces used in connection with head-worn computers (“HWC”). These and other systems, methods, objects, features, and advantages of the present invention will be apparent to those skilled in the art from the following detailed description of the preferred embodiment and the drawings. All documents mentioned herein are hereby incorporated in their entirety by reference.





BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are described with reference to the following Figures. The same numbers may be used throughout to reference like features and components that are shown in the Figures:



FIG. 1 illustrates a head worn computing system in accordance with the principles of the present invention.



FIG. 2 illustrates an external user interface in accordance with the principles of the present invention.



FIG. 3A to 3C illustrate distance control systems in accordance with the principles of the present invention.



FIG. 4A to 4C illustrate force interpretation systems in accordance with the principles of the present invention.



FIG. 5A to 5C illustrate user interface mode selection systems in accordance with the principles of the present invention.



FIG. 6 illustrates interaction systems in accordance with the principles of the present invention.



FIG. 7 illustrates external user interfaces in accordance with the principles of the present invention.



FIG. 8 illustrates a pattern recognition system and process in accordance with the principles of the present invention.



FIG. 9 illustrates external interface systems in accordance with the principles of the present invention.





While the invention has been described in connection with certain preferred embodiments, other embodiments would be understood by one of ordinary skill in the art and are encompassed herein.


DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Aspects of the present invention relate to head-worn computing (“HWC”) systems. HWC involves, in some instances, a system that mimics the appearance of head-worn glasses or sunglasses. The glasses may be a fully developed computing platform, such as including computer displays presented in each of the lenses of the glasses to the eyes of the user. In embodiments, the lenses and displays may be configured to allow a person wearing the glasses to see the environment through the lenses while also seeing, simultaneously, digital imagery, which forms an overlaid image that is perceived by the person as a digitally augmented image of the environment, or augmented reality (“AR”).


HWC involves more than just placing a computing system on a person's head. The system may need to be designed as a lightweight, compact and fully functional computer display, such as wherein the computer display includes a high resolution digital display that provides a high level of emersion comprised of the displayed digital content and the see-through view of the environmental surroundings. User interfaces and control systems suited to the HWC device may be required that are unlike those used for a more conventional computer such as a laptop. For the HWC and associated systems to be most effective, the glasses may be equipped with sensors to determine environmental conditions, geographic location, relative positioning to other points of interest, objects identified by imaging and movement by the user or other users in a connected group, and the like. The HWC may then change the mode of operation to match the conditions, location, positioning, movements, and the like, in a method generally referred to as a contextually aware HWC. The glasses also may need to be connected, wirelessly or otherwise, to other systems either locally or through a network. Controlling the glasses may be achieved through the use of an external device, automatically through contextually gathered information, through user gestures captured by the glasses sensors, and the like. Each technique may be further refined depending on the software application being used in the glasses. The glasses may further be used to control or coordinate with external devices that are associated with the glasses.


Referring to FIG. 1, an overview of the HWC system 100 is presented. As shown, the HWC system 100 comprises a HWC 102, which in this instance is configured as glasses to be worn on the head with sensors such that the HWC 102 is aware of the objects and conditions in the environment 114. In this instance, the HWC 102 also receives and interprets control inputs such as gestures and movements 116. The HWC 102 may communicate with external user interfaces 104. The external user interfaces 104 may provide a physical user interface to take control instructions from a user of the HWC 102 and the external user interfaces 104 and the HWC 102 may communicate bi-directionally to affect the user's command and provide feedback to the external device 108. The HWC 102 may also communicate bi-directionally with externally controlled or coordinated local devices 108. For example, an external user interface 104 may be used in connection with the HWC 102 to control an externally controlled or coordinated local device 108. The externally controlled or coordinated local device 108 may provide feedback to the HWC 102 and a customized GUI may be presented in the HWC 102 based on the type of device or specifically identified device 108. The HWC 102 may also interact with remote devices and information sources 112 through a network connection 110. Again, the external user interface 104 may be used in connection with the HWC 102 to control or otherwise interact with any of the remote devices 108 and information sources 112 in a similar way as when the external user interfaces 104 are used to control or otherwise interact with the externally controlled or coordinated local devices 108. Similarly, HWC 102 may interpret gestures 116 (e.g., captured from forward, downward, upward, rearward facing sensors such as camera(s), range finders, IR sensors, etc.) or environmental conditions sensed in the environment 114 to control either local or remote devices 108 or 112.


We will now describe each of the main elements depicted on FIG. 1 in more detail; however, these descriptions are intended to provide general guidance and should not be construed as limiting. Additional description of each element may also be further described herein.


The HWC 102 is a computing platform intended to be worn on a person's head. The HWC 102 may take many different forms to fit many different functional requirements. In some situations, the HWC 102 will be designed in the form of conventional glasses. The glasses may or may not have active computer graphics displays. In situations where the HWC 102 has integrated computer displays the displays may be configured as see-through displays such that the digital imagery can be overlaid with respect to the user's view of the environment 114. There are a number of see-through optical designs that may be used, including ones that have a reflective display (e.g., LCoS, DLP), emissive displays (e.g., OLED, LED), hologram, TIR waveguides, and the like. In addition, the optical configuration may be monocular or binocular. It may also include vision corrective optical components. In embodiments, the optics may be packaged as contact lenses. In other embodiments, the HWC 102 may be in the form of a helmet with a see-through shield, sunglasses, safety glasses, goggles, a mask, fire helmet with see-through shield, police helmet with see through shield, military helmet with see-through shield, utility form customized to a certain work task (e.g., inventory control, logistics, repair, maintenance, etc.), and the like.


The HWC 102 may also have a number of integrated computing facilities, such as an integrated processor, integrated power management, communication structures (e.g., cell net, WiFi, Bluetooth, local area connections, mesh connections, remote connections (e.g., client server, etc.)), and the like. The HWC 102 may also have a number of positional awareness sensors, such as GPS, electronic compass, altimeter, tilt sensor, IMU, and the like. It may also have other sensors such as a camera, rangefinder, hyper-spectral camera, Geiger counter, microphone, spectral illumination detector, temperature sensor, chemical sensor, biologic sensor, moisture sensor, ultrasonic sensor, and the like.


The HWC 102 may also have integrated control technologies. The integrated control technologies may be contextual based control, passive control, active control, user control, and the like. For example, the HWC 102 may have an integrated sensor (e.g., camera) that captures user hand or body gestures 116 such that the integrated processing system can interpret the gestures and generate control commands for the HWC 102. In another example, the HWC 102 may have sensors that detect movement (e.g., a nod, head shake, and the like) including accelerometers, gyms and other inertial measurements, where the integrated processor may interpret the movement and generate a control command in response. The HWC 102 may also automatically control itself based on measured or perceived environmental conditions. For example, if it is bright in the environment the HWC 102 may increase the brightness or contrast of the displayed image. In embodiments, the integrated control technologies may be mounted on the HWC 102 such that a user can interact with it directly. For example, the HWC 102 may have a button(s), touch capacitive interface, and the like.


As described herein, the HWC 102 may be in communication with external user interfaces 104. The external user interfaces may come in many different forms. For example, a cell phone screen may be adapted to take user input for control of an aspect of the HWC 102. The external user interface may be a dedicated UI, such as a keyboard, touch surface, button(s), joy stick, and the like. In embodiments, the external controller may be integrated into another device such as a ring, watch, bike, car, and the like. In each case, the external user interface 104 may include sensors (e.g., IMU, accelerometers, compass, altimeter, and the like) to provide additional input for controlling the HWD 104.


As described herein, the HWC 102 may control or coordinate with other local devices 108. The external devices 108 may be an audio device, visual device, vehicle, cell phone, computer, and the like. For instance, the local external device 108 may be another HWC 102, where information may then be exchanged between the separate HWCs 108.


Similar to the way the HWC 102 may control or coordinate with local devices 106, the HWC 102 may control or coordinate with remote devices 112, such as the HWC 102 communicating with the remote devices 112 through a network 110. Again, the form of the remote device 112 may have many forms. Included in these forms is another HWC 102. For example, each HWC 102 may communicate its GPS position such that all the HWCs 102 know where all of HWC 102 are located.


Referring to FIG. 2, we now turn to describe a particular external user interface 104, referred to generally as a pen 200. The pen 200 is a specially designed external user interface 104 and can operate as a user interface, such as to many different styles of HWC 102. The pen 200 generally follows the form of a conventional pen, which is a familiar user handled device and creates an intuitive physical interface for many of the operations to be carried out in the HWC system 100. The pen 200 may be one of several user interfaces 104 used in connection with controlling operations within the HWC system 100. For example, the HWC 102 may watch for and interpret hand gestures 116 as control signals, where the pen 200 may also be used as a user interface with the same HWC 102. Similarly, a remote keyboard may be used as an external user interface 104 in concert with the pen 200. The combination of user interfaces or the use of just one control system generally depends on the operation(s) being executed in the HWC's system 100.


While the pen 200 may follow the general form of a conventional pen, it contains numerous technologies that enable it to function as an external user interface 104. FIG. 2 illustrates technologies comprised in the pen 200. As can be seen, the pen 200 may include a camera 208, which is arranged to view through lens 202. The camera may then be focused, such as through lens 202, to image a surface upon which a user is writing or making other movements to interact with the HWC 102. There are situations where the pen 200 will also have an ink, graphite, or other system such that what is being written can be seen on the writing surface. There are other situations where the pen 200 does not have such a physical writing system so there is no deposit on the writing surface, where the pen would only be communicating data or commands to the HWC 102. The lens configuration is described in greater detail herein. The function of the camera is to capture information from an unstructured writing surface such that pen strokes can be interpreted as intended by the user. To assist in the predication of the intended stroke path, the pen 200 may include a sensor, such as an IMU 212. Of course, the IMU could be included in the pen 200 in its separate parts (e.g., gyro, accelerometer, etc.) or an IMU could be included as a single unit. In this instance, the IMU 212 is used to measure and predict the motion of the pen 200. In turn, the integrated microprocessor 210 would take the IMU information and camera information as inputs and process the information to form a prediction of the pen tip movement.


The pen 200 may also include a pressure monitoring system 204, such as to measure the pressure exerted on the lens 202. As will be described in greater herein, the pressure measurement can be used to predict the user's intention for changing the weight of a line, type of a line, type of brush, click, double click, and the like. In embodiments, the pressure sensor may be constructed using any force or pressure measurement sensor located behind the lens 202, including for example, a resistive sensor, a current sensor, a capacitive sensor, a voltage sensor such as a piezoelectric sensor, and the like.


The pen 200 may also include a communications module 218, such as for bi-directional communication with the HWC 102. In embodiments, the communications module 218 may be a short distance communication module (e.g., Bluetooth). The communications module 218 may be security matched to the HWC 102. The communications module 218 may be arranged to communicate data and commands to and from the microprocessor 210 of the pen 200. The microprocessor 210 may be programmed to interpret data generated from the camera 208, IMU 212, and pressure sensor 204, and the like, and then pass a command onto the HWC 102 through the communications module 218, for example. In another embodiment, the data collected from any of the input sources (e.g., camera 108, IMU 212, pressure sensor 104) by the microprocessor may be communicated by the communication module 218 to the HWC 102, and the HWC 102 may perform data processing and prediction of the user's intention when using the pen 200. In yet another embodiment, the data may be further passed on through a network 110 to a remote device 112, such as a server, for the data processing and prediction. The commands may then be communicated back to the HWC 102 for execution (e.g., display writing in the glasses display, make a selection within the UI of the glasses display, control a remote external device 112, control a local external device 108), and the like. The pen may also include memory 214 for long or short term uses.


The pen 200 may also include a number of physical user interfaces, such as quick launch buttons 222, a touch sensor 220, and the like. The quick launch buttons 222 may be adapted to provide the user with a fast way of jumping to a software application in the HWC system 100. For example, the user may be a frequent user of communication software packages (e.g., email, text, Twitter, Instagram, Facebook, Google+, and the like), and the user may program a quick launch button 222 to command the HWC 102 to launch an application. The pen 200 may be provided with several quick launch buttons 222, which may be user programmable or factory programmable. The quick launch button 222 may be programmed to perform an operation. For example, one of the buttons may be programmed to clear the digital display of the HWC 102. This would create a fast way for the user to clear the screens on the HWC 102 for any reason, such as for example to better view the environment. The quick launch button functionality will be discussed in further detail below. The touch sensor 220 may be used to take gesture style input from the user. For example, the user may be able to take a single finger and run it across the touch sensor 220 to affect a page scroll.


The pen 200 may also include a laser pointer 224. The laser pointer 224 may be coordinated with the IMU 212 to coordinate gestures and laser pointing. For example, a user may use the laser 224 in a presentation to help with guiding the audience with the interpretation of graphics and the IMU 212 may, either simultaneously or when the laser 224 is off, interpret the user's gestures as commands or data input.



FIGS. 3A-C illustrate several embodiments of lens and camera arrangements 300 for the pen 200. One aspect relates to maintaining a constant distance between the camera and the writing surface to enable the writing surface to be kept in focus for better tracking of movements of the pen 200 over the writing surface. Another aspect relates to maintaining an angled surface following the circumference of the writing tip of the pen 200 such that the pen 200 can be rolled or partially rolled in the user's hand to create the feel and freedom of a conventional writing instrument.



FIG. 3A illustrates an embodiment of the writing lens end of the pen 200. The configuration includes a ball lens 304, a camera or image capture surface 302, and a domed cover lens 308. In this arrangement, the camera views the writing surface through the ball lens 304 and dome cover lens 308. The ball lens 304 causes the camera to focus such that the camera views the writing surface when the pen 200 is held in the hand in a natural writing position, such as with the pen 200 in contact with a writing surface. In embodiments, the ball lens 304 should be separated from the writing surface to obtain the highest resolution of the writing surface at the camera 302. In embodiments, the ball lens 304 is separated by approximately 1 to 3 mm. In this configuration, the domed cover lens 308 provides a surface that can keep the ball lens 304 separated from the writing surface at a constant distance, such as substantially independent of the angle used to write on the writing surface. For instance, in embodiments the field of view of the camera in this arrangement would be approximately 60 degrees.


The domed cover lens, or other lens 308 used to physically interact with the writing surface, will be transparent or transmissive within the active bandwidth of the camera 302. In embodiments, the domed cover lens 308 may be spherical or other shape and comprised of glass, plastic, sapphire, diamond, and the like. In other embodiments where low resolution imaging of the surface is acceptable. The pen 200 can omit the domed cover lens 308 and the ball lens 304 can be in direct contact with the surface.



FIG. 3B illustrates another structure where the construction is somewhat similar to that described in connection with FIG. 3A; however, this embodiment does not use a dome cover lens 308, but instead uses a spacer 310 to maintain a predictable distance between the ball lens 304 and the writing surface, wherein the spacer may be spherical, cylindrical, tubular or other shape that provides spacing while allowing for an image to be obtained by the camera 302 through the lens 304. In a preferred embodiment, the spacer 310 is transparent. In addition, while the spacer 310 is shown as spherical, other shapes such an oval, doughnut shape, half sphere, cone, cylinder or other form may be used.



FIG. 3C illustrates yet another embodiment, where the structure includes a post 314, such as running through the center of the lensed end of the pen 200. The post 314 may be an ink deposition system (e.g., ink cartridge), graphite deposition system (e.g., graphite holder), or a dummy post whose purpose is mainly only that of alignment. The selection of the post type is dependent on the pen's use. For instance, in the event the user wants to use the pen 200 as a conventional ink depositing pen as well as a fully functional external user interface 104, the ink system post would be the best selection. If there is no need for the ‘writing’ to be visible on the writing surface, the selection would be the dummy post. The embodiment of FIG. 3C includes camera(s) 302 and an associated lens 312, where the camera 302 and lens 312 are positioned to capture the writing surface without substantial interference from the post 314. In embodiments, the pen 200 may include multiple cameras 302 and lenses 312 such that more or all of the circumference of the tip 314 can be used as an input system. In an embodiment, the pen 200 includes a contoured grip that keeps the pen aligned in the user's hand so that the camera 302 and lens 312 remains pointed at the surface.


Another aspect of the pen 200 relates to sensing the force applied by the user to the writing surface with the pen 200. The force measurement may be used in a number of ways. For example, the force measurement may be used as a discrete value, or discontinuous event tracking, and compared against a threshold in a process to determine a user's intent. The user may want the force interpreted as a ‘click’ in the selection of an object, for instance. The user may intend multiple force exertions interpreted as multiple clicks. There may be times when the user holds the pen 200 in a certain position or holds a certain portion of the pen 200 (e.g., a button or touch pad) while clicking to affect a certain operation (e.g., a ‘right click’). In embodiments, the force measurement may be used to track force and force trends. The force trends may be tracked and compared to threshold limits, for example. There may be one such threshold limit, multiple limits, groups of related limits, and the like. For example, when the force measurement indicates a fairly constant force that generally falls within a range of related threshold values, the microprocessor 210 may interpret the force trend as an indication that the user desires to maintain the current writing style, writing tip type, line weight, brush type, and the like. In the event that the force trend appears to have gone outside of a set of threshold values intentionally, the microprocessor may interpret the action as an indication that the user wants to change the current writing style, writing tip type, line weight, brush type, and the like. Once the microprocessor has made a determination of the user's intent, a change in the current writing style, writing tip type, line weight, brush type, and the like may be executed. In embodiments, the change may be noted to the user (e.g., in a display of the HWC 102), and the user may be presented with an opportunity to accept the change.



FIG. 4A illustrates an embodiment of a force sensing surface tip 400 of a pen 200. The force sensing surface tip 400 comprises a surface connection tip 402 (e.g., a lens as described herein elsewhere) in connection with a force or pressure monitoring system 204. As a user uses the pen 200 to write on a surface or simulate writing on a surface the force monitoring system 204 measures the force or pressure the user applies to the writing surface and the force monitoring system communicates data to the microprocessor 210 for processing. In this configuration, the microprocessor 210 receives force data from the force monitoring system 204 and processes the data to make predictions of the user's intent in applying the particular force that is currently being applied. In embodiments, the processing may be provided at a location other than on the pen (e.g., at a server in the HWC system 100, on the HWC 102). For clarity, when reference is made herein to processing information on the microprocessor 210, the processing of information contemplates processing the information at a location other than on the pen. The microprocessor 210 may be programmed with force threshold(s), force signature(s), force signature library and/or other characteristics intended to guide an inference program in determining the user's intentions based on the measured force or pressure. The microprocessor 210 may be further programmed to make inferences from the force measurements as to whether the user has attempted to initiate a discrete action (e.g., a user interface selection ‘click’) or is performing a constant action (e.g., writing within a particular writing style). The inferencing process is important as it causes the pen 200 to act as an intuitive external user interface 104.



FIG. 4B illustrates a force 408 versus time 410 trend chart with a single threshold 418. The threshold 418 may be set at a level that indicates a discrete force exertion indicative of a user's desire to cause an action (e.g., select an object in a GUI). Event 412, for example, may be interpreted as a click or selection command because the force quickly increased from below the threshold 418 to above the threshold 418. The event 414 may be interpreted as a double click because the force quickly increased above the threshold 418, decreased below the threshold 418 and then essentially repeated quickly. The user may also cause the force to go above the threshold 418 and hold for a period indicating that the user is intending to select an object in the GUI (e.g., a GUI presented in the display of the HWC 102) and ‘hold’ for a further operation (e.g. moving the object).


While a threshold value may be used to assist in the interpretation of the user's intention, a signature force event trend may also be used. The threshold and signature may be used in combination or either method may be used alone. For example, a single-click signature may be represented by a certain force trend signature or set of signatures. The single-click signature(s) may require that the trend meet a criteria of a rise time between x any y values, a hold time of between a and b values and a fall time of between c and d values, for example. Signatures may be stored for a variety of functions such as click, double click, right click, hold, move, etc. The microprocessor 210 may compare the real-time force or pressure tracking against the signatures from a signature library to make a decision and issue a command to the software application executing in the GUI.



FIG. 4C illustrates a force 408 versus time 410 trend chart with multiple thresholds 418. By way of example, the force trend is plotted on the chart with several pen force or pressure events. As noted, there are both presumably intentional events 420 and presumably non-intentional events 422. The two thresholds 418 of FIG. 4C create three zones of force: a lower, middle and higher range. The beginning of the trend indicates that the user is placing a lower zone amount of force. This may mean that the user is writing with a given line weight and does not intend to change the weight, the user is writing. Then the trend shows a significant increase 420 in force into the middle force range. This force change appears, from the trend to have been sudden and thereafter it is sustained. The microprocessor 210 may interpret this as an intentional change and as a result change the operation in accordance with preset rules (e.g., change line width, increase line weight, etc.). The trend then continues with a second apparently intentional event 420 into the higher-force range. During the performance in the higher-force range, the force dips below the upper threshold 418. This may indicate an unintentional force change and the microprocessor may detect the change in range however not effect a change in the operations being coordinated by the pen 200. As indicated above, the trend analysis may be done with thresholds and/or signatures.


Generally, in the present disclosure, instrument stroke parameter changes may be referred to as a change in line type, line weight, tip type, brush type, brush width, brush pressure, color, and other forms of writing, coloring, painting, and the like.


Another aspect of the pen 200 relates to selecting an operating mode for the pen 200 dependent on contextual information and/or selection interface(s). The pen 200 may have several operating modes. For instance, the pen 200 may have a writing mode where the user interface(s) of the pen 200 (e.g., the writing surface end, quick launch buttons 222, touch sensor 220, motion based gesture, and the like) is optimized or selected for tasks associated with writing. As another example, the pen 200 may have a wand mode where the user interface(s) of the pen is optimized or selected for tasks associated with software or device control (e.g., the HWC 102, external local device, remote device 112, and the like). The pen 200, by way of another example, may have a presentation mode where the user interface(s) is optimized or selected to assist a user with giving a presentation (e.g., pointing with the laser pointer 224 while using the button(s) 222 and/or gestures to control the presentation or applications relating to the presentation). The pen may, for example, have a mode that is optimized or selected for a particular device that a user is attempting to control. The pen 200 may have a number of other modes and an aspect of the present invention relates to selecting such modes.



FIG. 5A illustrates an automatic user interface(s) mode selection based on contextual information. The microprocessor 210 may be programmed with IMU thresholds 514 and 512. The thresholds 514 and 512 may be used as indications of upper and lower bounds of an angle 504 and 502 of the pen 200 for certain expected positions during certain predicted modes. When the microprocessor 210 determines that the pen 200 is being held or otherwise positioned within angles 502 corresponding to writing thresholds 514, for example, the microprocessor 210 may then institute a writing mode for the pen's user interfaces. Similarly, if the microprocessor 210 determines (e.g., through the IMU 212) that the pen is being held at an angle 504 that falls between the predetermined wand thresholds 512, the microprocessor may institute a wand mode for the pen's user interface. Both of these examples may be referred to as context based user interface mode selection as the mode selection is based on contextual information (e.g., position) collected automatically and then used through an automatic evaluation process to automatically select the pen's user interface(s) mode.


As with other examples presented herein, the microprocessor 210 may monitor the contextual trend (e.g., the angle of the pen over time) in an effort to decide whether to stay in a mode or change modes. For example, through signatures, thresholds, trend analysis, and the like, the microprocessor may determine that a change is an unintentional change and therefore no user interface mode change is desired.



FIG. 5B illustrates an automatic user interface(s) mode selection based on contextual information. In this example, the pen 200 is monitoring (e.g., through its microprocessor) whether or not the camera at the writing surface end 208 is imaging a writing surface in close proximity to the writing surface end of the pen 200. If the pen 200 determines that a writing surface is within a predetermined relatively short distance, the pen 200 may decide that a writing surface is present 502 and the pen may go into a writing mode user interface(s) mode. In the event that the pen 200 does not detect a relatively close writing surface 504, the pen may predict that the pen is not currently being used to as a writing instrument and the pen may go into a non-writing user interface(s) mode.



FIG. 5C illustrates a manual user interface(s) mode selection. The user interface(s) mode may be selected based on a twist of a section 508 of the pen 200 housing, clicking an end button 510, pressing a quick launch button 222, interacting with touch sensor 220, detecting a predetermined action at the pressure monitoring system (e.g., a click), detecting a gesture (e.g., detected by the IMU), etc. The manual mode selection may involve selecting an item in a GUI associated with the pen 200 (e.g., an image presented in the display of HWC 102).


In embodiments, a confirmation selection may be presented to the user in the event a mode is going to change. The presentation may be physical (e.g., a vibration in the pen 200), through a GUI, through a light indicator, etc.



FIG. 6 illustrates a couple pen use-scenarios 600 and 601. There are many use scenarios and we have presented a couple in connection with FIG. 6 as a way of illustrating use scenarios to further the understanding of the reader. As such, the use-scenarios should be considered illustrative and non-limiting.


Use scenario 600 is a writing scenario where the pen 200 is used as a writing instrument. In this example, quick launch button 122A is pressed to launch a note application 610 in the GUI 608 of the HWC 102 display 604. Once the quick launch button 122A is pressed, the HWC 102 launches the note program 610 and puts the pen into a writing mode. The user uses the pen 200 to scribe symbols 602 on a writing surface, the pen records the scribing and transmits the scribing to the HWC 102 where symbols representing the scribing are displayed 612 within the note application 610.


Use scenario 601 is a gesture scenario where the pen 200 is used as a gesture capture and command device. In this example, the quick launch button 122B is activated and the pen 200 activates a wand mode such that an application launched on the HWC 102 can be controlled. Here, the user sees an application chooser 618 in the display(s) of the HWC 102 where different software applications can be chosen by the user. The user gestures (e.g., swipes, spins, turns, etc.) with the pen to cause the application chooser 618 to move from application to application. Once the correct application is identified (e.g., highlighted) in the chooser 618, the user may gesture or click or otherwise interact with the pen 200 such that the identified application is selected and launched. Once an application is launched, the wand mode may be used to scroll, rotate, change applications, select items, initiate processes, and the like, for example.


In an embodiment, the quick launch button 122A may be activated and the HWC 102 may launch an application chooser presenting to the user a set of applications. For example, the quick launch button may launch a chooser to show all communication programs (e.g., SMS, Twitter, Instagram, Facebook, email, etc.) available for selection such that the user can select the program the user wants and then go into a writing mode. By way of further example, the launcher may bring up selections for various other groups that are related or categorized as generally being selected at a given time (e.g., Microsoft Office products, communication products, productivity products, note products, organizational products, and the like)



FIG. 7 illustrates yet another embodiment of the present invention. FIG. 700 illustrates a watchband clip on controller 700. The watchband clip on controller may be a controller used to control the HWC 102 or devices in the HWC system 100. The watchband clip on controller 700 has a fastener 718 (e.g., rotatable clip) that is mechanically adapted to attach to a watchband, as illustrated at 704.


The watchband controller 700 may have quick launch interfaces 708 (e.g., to launch applications and choosers as described herein), a touch pad 714 (e.g., to be used as a touch style mouse for GUI control in a HWC 102 display) and a display 712. The clip 718 may be adapted to fit a wide range of watchbands so it can be used in connection with a watch that is independently selected for its function. The clip, in embodiments, is rotatable such that a user can position it in a desirable manner. In embodiments the clip may be a flexible strap. In embodiments, the flexible strap may be adapted to be stretched to attach to a hand, wrist, finger, device, weapon, and the like.


In embodiments, the watchband controller may be configured as a removable and replaceable watchband. For example, the controller may be incorporated into a band with a certain width, segment spacing's, etc. such that the watchband, with its incorporated controller, can be attached to a watch body. The attachment, in embodiments, may be mechanically adapted to attach with a pin upon which the watchband rotates. In embodiments, the watchband controller may be electrically connected to the watch and/or watch body such that the watch, watch body and/or the watchband controller can communicate data between them.


The watchband controller may have 3-axis motion monitoring (e.g., through an IMU, accelerometers, magnetometers, gyroscopes, etc.) to capture user motion. The user motion may then be interpreted for gesture control.


In embodiments, the watchband controller may comprise fitness sensors and a fitness computer. The sensors may track heart rate, calories burned, strides, distance covered, and the like. The data may then be compared against performance goals and/or standards for user feedback.


Another aspect of the present invention relates to tracking pen movements with the assistance of a camera and displayed content in a HWC 102. In embodiments, content is presented in a see-through display of a head-worn computer to provide a virtual guide for the wearer who wants to make motions with a pen, finger, or other interface and have the motions interpreted for pattern recognition. As described in connection with pen embodiments disclosed herein elsewhere, an IMU or pen-tip camera may be used to monitor the motion of a pen in order to predict what patterns are being drawn. The IMU and/or pen tip camera may suffer from electronic or optical drift and the drift may cause inaccuracies in the pattern prediction. In embodiments, to augment the IMU and/or pen tip camera motion predictions the virtual guide is provided to compensate for the drift. The pen motions may be captured by a camera on-board the HWC 102 while the wearer is writing with the guidance of the virtual line. Knowing that the wearer is using the virtual line as a guide, the relative position between the pen tip and virtual line can be used to reduce or eliminate drift issues.


In embodiments, digital content is presented to a wearer of the HWC 102 and the wearer moves the pen 200 along a writing surface guided by the digital content for pattern recordation, recognition and presentation assistance. In embodiments, a camera in the HWC 102 images and tracks the positions of the pen 200 for pattern recordation and recognition assistance. In embodiments, both the digital content and the camera capturing the pen positions are used for pattern recordation and recognition assistance. In embodiments, the digital content, camera capture, in-pen camera capture, in-pen IMU, etc. may be used in combination for pattern recordation and recognition assistance. In embodiments, the relative positions of the pen strokes to the virtual line may be presented in the HWC 102 displays in relation to the virtual line. For example, the wearer of the HWC 102 may be scribing without ink in relation to the virtual line that he perceives and as presented in the HWC 102 display, the on-board HWC 102 camera may capture the scribing, a processor may interpret the imaged scribing in relation to the line such that the scribing can be converted into digital content to be displayed in the HWC 102 display in relation to the virtual line.



FIG. 8 illustrates a system where a camera in the HWC 102 is used to track pen 200 motions and digital content is presented to the wearer of the HWC 102 to assist the wearer with writing within a structure. In this embodiment, digital content in the form of a line 804 is presented in an FOV 802 of the HWC 102. The wearer can see through the FOV 802 so the line 804 appears to augment the surrounding environment's view for the wearer. The line may be ‘fixed’ to a spot in the environment such that when the wearer turns his head and hence changes the position of the HWC 102, the line appears to stay in position with respect to the environment. In embodiments, the camera in the HWC 102 may image the environment and track the relative movement of the HWC 102 with respect to the environment such that the line 804 can be positioned and moved within the FOV in accordance with the imaged movements to maintain visual alignment of the line with a point, object, marker, etc. in the environment. This configuration presents a virtual line in the environment that does not appear to move as the wearer's head moves. The virtual line can provide the wearer with guidance on where to make pen strokes. The line can be thought of as a line on a piece of paper so the wearer can write, or make strokes in a writing pattern, along the virtual line to make prediction of the lines pattern more accurate and overcome drift errors that may otherwise be apparent when attempting to record the movements and predict the patterns.


With the virtual line presented and virtually connected to a position in the environment, the wearer can use the line for guidance when making writing patterns. The HWC 102 camera can also be used to track the movements of the pen 200 relative to the position of the virtual line. This may be used to better predict the patterns indicated by the wearer's pen strokes. As described herein elsewhere, the pen 200 may track its motions through a pen tip camera and IMU. In embodiments, the pen tip camera and IMU may track the pen's motion and the camera may be used to track the motion of the pen relative to the virtual line. Each of these inputs may be used to track, record and predict what it being written.


In embodiments, the camera in the HWC 102 captures images of the wearer's pen's motion while the wearer is using the pen to make patterns with the virtual line as a guide. The virtual line may then be overlaid on the captured images of the motion to assist with the pattern analysis. In embodiments, once the overlay is made, one can see or analyze how the pen pattern moved with respect to the position of the virtual line as the wearer may be viewed the virtual line. The pattern analysis may involve interpreting the IMU motion detection, in-pen motion detection, and/or the pen's motion as captured through the HWC 102 camera relative to the virtual line. For example, if the IMU indicates that the pen shifted away from the wearer but the position of the pen relative to the virtual line indicates the pen was not moving, the portion of IMU data that indicated the shift may be discounted in the prediction analysis. The virtual line pattern analysis may be done in real-time, after the fact, etc. The pattern recognition may be done on a processor on-board the HWC 102, remote from the HWC 102, or partially on-board and remotely.


In embodiments, the virtual line may take any number of forms. For example, the virtual line may be a line, part of a virtual note, part of a virtual message template, etc. The line may also change positions and shapes depending on the wearer's needs. For example, the wearer may want to trace a pattern that is being displayed as digital content and the digital content may be presented as a consolidated image, part of an image, image in a line-by-line presentation format, etc. In embodiments, this system may be used for lessons on writing, painting, drawing, etc.


Another aspect of the present invention relates to a vehicle-specific external user interface 104. In embodiments, the vehicle-specific external (“VSE”) user interface 104 includes a mechanical mounting system adapted to mount the VSE interface 104 on the steering wheel of the vehicle. The mounting system may secure the VSE interface in a position that tends to be near the driver's hands, such as on the wheel portion around the 1:00 to 3:00 position or 9:00 to 11:00 position. The VSE interface may be secured with a Velcro style system, clip, strap, etc. In embodiments, the VSE interface is adapted to provide the driver with a system for the interaction with the HWC 102 when driving where the interactions are intended to enhance the driver's driving experience. For example, a driver may preset applications, screens, content sets, etc. for access while he is driving and the VSE interface may provide a physical interface for the launching of an application, toggling, switching, or changing applications or screens or content sets, etc. The presentation of display content controlled by the VSE interface may involve navigation, vehicle systems, point-of-interest information, advertisements, etc. and the driver may be able to switch between the applications very quickly through the interaction of a button or more than one button. In embodiments, the preset screens, content sets, or applications may be launched through dedicated quick launch buttons. For example, the navigation application button may be in the upper right of the VSE interface.


In embodiments, a pre-programmed button or set of buttons may be set to clear the display of the HWC 102 to be free of content or reduce the amount of content that is otherwise displayed to increase the driver's see-through view of the surroundings. The button(s) may be set to switch content display modes between two pre-determined content types relating to the vehicle (e.g., switching between pre-set driving applications). The button(s) may be set to change the amount of content-free area in the field-of-view of the HWC 102. The button(s) may be set to move content within the field-of-view. The button(s) may be set to change the HWC 102 display brightness and contrast or control other aspects of the HWC 102, such as to change audio volume, sensor settings, etc. While many embodiments refer to the use of “button(s)” it should be understood that this is for simplicity in illustration only and that other forms of user controllable interfaces are envisioned by the present invention, such as, switches, toggles, touch screens, touch pads, etc.



FIG. 9 illustrates several VSE interfaces according to the principles of the present invention. The VSE interface 904 is illustrated as being mounted on the steering wheel 902 and illustrated in various control configurations. The VSE interface may have hot or launch buttons on the side 908, front face 910 or otherwise such that the driver can touch and interact with them while driving. The VSE interface may also have a fixed hot button 912 to perform a dedicated function such as clearing the display of the HWC 102 of content or limiting the type or amount of content that is permitted to be displayed in the display. The VSE interface may also have one or more touch pads or screens. A touch pad or screen may, for example, be used as a button style interface as well as a cursor control style interface. The VSE interface may also be virtually modified with a virtual active layer 914. The virtual active layer 914 may be presented as digital content in the display of the HWC 102 and be locked in position with respect to the physical VSE interface such that the driver perceives the virtual content as augmenting the physical VSE interface. For example, virtual button labels may be provided as digital content and overlaid or set next to the VSE interface such that the driver perceives the labels as being associated with the buttons. The virtual content may be used in coordination with a new command set. For example, a new command set relating to navigation may be set on the HWC 102 and a label or image may be set to appear in a position locked to the VSE interface. In embodiments, there may not be a physical button and the interaction that causes a control command may be initiated when the user virtually interacts with the content by touching a portion of the VSE controller that intersects, from the driver's perspective through the display, with the virtual content.


In embodiments the VSE interface may be mechanically adapted to connect to other systems within the natural grasp of the driver. For example, a VSE interface may be mechanically adapted to attach to a gear shift, handle, post, etc.


Although embodiments of HWC have been described in language specific to features, systems, computer processes and/or methods, the appended claims are not necessarily limited to the specific features, systems, computer processes and/or methods described. Rather, the specific features, systems, computer processes and/or methods are disclosed as non-limited example implementations of HWC.


All documents referenced herein are hereby incorporated by reference.

Claims
  • 1. A system comprising: a wearable device comprising a see-through display;a vehicle comprising a physical control, the physical control configured to control an operation associated with the vehicle and further configured to move with respect to an interior of the vehicle, the vehicle configured to communicate with the wearable device; andan external user interface, the external user interface mechanically coupled to the physical control and configured to communicate with the wearable head device, wherein: the wearable device is configured to present, via the see-through display, concurrently with presenting a view of the physical control, virtual content to a user;the presenting the virtual content comprises: at a first time, presenting via the see-through display a first user interface element aligned with the external user interface and at a first position with respect to the interior of the vehicle,at a second time, in response to a movement of the physical control with respect to the interior of the vehicle, presenting via the see-through display the first user interface element aligned with the external user interface and at a second position with respect to the interior of the vehicle, a displacement between the first position with respect to the interior of the vehicle and the second position with respect to the interior of the vehicle corresponding to the movement of the external user interface, andpresenting via the see-through display a second user interface element at a third position with respect to the interior of the vehicle, the third position different from the first position with respect to the interior of the vehicle and the second position with respect to the interior of the vehicle; anda control command associated with the operation of the vehicle is generated in response to an interaction of the user with the first user interface element.
  • 2. The system of claim 1, wherein the physical control comprises one or more of a steering wheel, a shift lever, and a handle.
  • 3. The system of claim 1, wherein: the physical control is configured to provide input to the wearable device; andthe wearable device is further configured to: in response to receiving the input from the physical control, update the presentation of the virtual content via the see-through display.
  • 4. The system of claim 3, wherein the updating the presentation of the virtual content via the see-through display comprises increasing or decreasing an amount of virtual content presented via the see-through display.
  • 5. The system of claim 3, wherein the updating the presentation of the virtual content via the see-through display comprises clearing a region of the see-through display of virtual content.
  • 6. The system of claim 1, further comprising a third user interface element comprising one or more of a button, a touch pad, a touch screen, and a switch.
  • 7. The system of claim 1, wherein the virtual content comprises text corresponding to a command set selected via the wearable device.
  • 8. The system of claim 7, wherein the command set comprises one or more navigation commands.
  • 9. The system of claim 1, wherein the virtual content comprises navigation information.
  • 10. The system of claim 1, wherein the virtual content comprises system information for the vehicle.
  • 11. The system of claim 1, wherein the virtual content comprises point-of-interest information.
  • 12. The system of claim 1, wherein the virtual content comprises an advertisement.
  • 13. The system of claim 1, wherein one or more of the first user interface element and second user interface element are associated with an application associated with the vehicle or the wearable device.
  • 14. A method, comprising: controlling, via a physical control of a vehicle, an operation associated with the vehicle, wherein the physical control is configured to move with respect to an interior of the vehicle;presenting, via a see-through display of a wearable device, concurrently with presenting a view of the physical control, virtual content to a user, wherein: the vehicle is configured to communicate with the wearable device,an external user interface is mechanically coupled to the physical control and is configured to communicate with the wearable head device,the presenting the virtual content comprises: at a first time, presenting via the see-through display a first user interface element aligned with the external user interface and at a first position with respect to the interior of the vehicle,at a second time, in response to a movement of the physical control with respect to the interior of the vehicle, presenting via the see-through display the first user interface element aligned with the external user interface and at a second position with respect to the interior of the vehicle, a displacement between the first position with respect to the interior of the vehicle and the second position with respect to the interior of the vehicle corresponding to the movement of the external user interface, andpresenting via the see-through display a second user interface element at a third position with respect to the interior of the vehicle, the third position different from the first position with respect to the interior of the vehicle and the second position with respect to the interior of the vehicle; andin response to an interaction of the user with the first user interface element, generating a control command associated with the operation of the vehicle.
  • 15. The method of claim 14, wherein the physical control comprises one or more of a steering wheel, a shift lever, and a handle.
  • 16. The method of claim 14, further comprising: providing, via the physical control, an input to the wearable device; andin response to receiving the input from the physical control, updating the presentation of the virtual content via the see-through display.
  • 17. The method of claim 14, wherein the the vehicle further comprises a third user interface element comprising one or more of a button, a touch pad, a touch screen, and a switch.
  • 18. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: controlling, via a physical control of a vehicle, an operation associated with the vehicle, wherein the physical control is configured to move with respect to an interior of the vehicle;presenting, via a see-through display of a wearable device, concurrently with presenting a view of the physical control, virtual content to a user, wherein: the vehicle is configured to communicate with the wearable device,an external user interface is mechanically coupled to the physical control and is configured to communicate with the wearable head device,the presenting the virtual content comprises: at a first time, presenting via the see-through display a first user interface element aligned with the external user interface and at a first position with respect to the interior of the vehicle,at a second time, in response to a movement of the physical control with respect to the interior of the vehicle, presenting via the see-through display the first user interface element aligned with the external user interface and at a second position with respect to the interior of the vehicle, a displacement between the first position with respect to the interior of the vehicle and the second position with respect to the interior of the vehicle corresponding to the movement of the external user interface, andpresenting via the see-through display a second user interface element at a third position with respect to the interior of the vehicle, the third position different from the first position with respect to the interior of the vehicle and the second position with respect to the interior of the vehicle; andin response to an interaction of the user with the first user interface element, generating a control command associated with the operation of the vehicle.
CLAIM TO PRIORITY

This application is a continuation of U.S. patent application Ser. No. 17/670,373, filed Feb. 11, 2022, which is a continuation of U.S. patent application Ser. No. 17/338,484, filed Jun. 3, 2021, now U.S. Pat. No. 11,294,180, issued on Apr. 5, 2022, which is a continuation of U.S. patent application Ser. No. 16/915,919, filed Jun. 29, 2020, now U.S. Pat. No. 11,054,645, issued on Jul. 6, 2021, which is a continuation of U.S. patent application Ser. No. 15/785,505, filed Oct. 17, 2017, now U.S. Pat. No. 10,698,212, issued Jun. 30, 2020, which is a continuation of U.S. patent application Ser. No. 14/307,114, filed Jun. 17, 2014, now U.S. Pat. No. 9,810,906, issued on Nov. 7, 2017, each of which is incorporated by reference in its entirety.

US Referenced Citations (902)
Number Name Date Kind
1897833 Benway Feb 1933 A
3305294 Alvarez Feb 1967 A
4034401 Mann Jul 1977 A
4668155 Kaufmann May 1987 A
4811739 Silver Mar 1989 A
4852988 Velez Aug 1989 A
4928301 Smoot May 1990 A
D327674 Kuo Jul 1992 S
5151722 Massof Sep 1992 A
5257094 Larussa Oct 1993 A
D352930 Tsuji Nov 1994 S
5483307 Anderson Jan 1996 A
D375748 Hartman Nov 1996 S
D376790 Taylor Dec 1996 S
5596451 Handschy Jan 1997 A
5621424 Shimada Apr 1997 A
5699057 Ikeda Dec 1997 A
5699194 Takahashi Dec 1997 A
5717422 Fergason Feb 1998 A
D392959 Edwards Mar 1998 S
5729242 Margerum Mar 1998 A
5767841 Hartman Jun 1998 A
5788195 Rice Aug 1998 A
5808800 Handschy Sep 1998 A
5808802 Hur Sep 1998 A
D410638 Loughnane Jun 1999 S
5949583 Rallison Sep 1999 A
5991084 Hildebrand Nov 1999 A
6028608 Jenkins Feb 2000 A
6034653 Robertson Mar 2000 A
6046712 Beller Apr 2000 A
6147805 Fergason Nov 2000 A
6160552 Wilsher Dec 2000 A
6160666 Rallison Dec 2000 A
6195136 Handschy Feb 2001 B1
6204974 Spitzer Mar 2001 B1
6297749 Smith Oct 2001 B1
D451892 Carrere Dec 2001 S
6347764 Brandon Feb 2002 B1
6359723 Handschy Mar 2002 B1
6369952 Rallison Apr 2002 B1
6379009 Fergason Apr 2002 B1
6384982 Spitzer May 2002 B1
6392656 Someya May 2002 B1
D460071 Sheehan Jul 2002 S
6433760 Vaissie Aug 2002 B1
6456438 Lee Sep 2002 B1
6461000 Magarill Oct 2002 B1
6480174 Kaufmann Nov 2002 B1
6491389 Yaguchi Dec 2002 B2
6491391 Blum et al. Dec 2002 B1
D470144 Li Feb 2003 S
6535182 Stanton Mar 2003 B2
D473871 Santos Apr 2003 S
6563626 Iwasaki May 2003 B1
D478052 Thomas, Jr. Aug 2003 S
6642945 Sharpe Nov 2003 B1
6717075 Stavely Apr 2004 B1
6747611 Budd Jun 2004 B1
6771294 Pulli Aug 2004 B1
6795041 Ogawa Sep 2004 B2
6847336 Lemelson Jan 2005 B1
6906836 Parker Jun 2005 B2
6943754 Aughey Sep 2005 B2
6956558 Rosenberg Oct 2005 B1
D512027 Sarasjoki Nov 2005 S
D513233 Stauffer Dec 2005 S
6977776 Volkenandt et al. Dec 2005 B2
6987787 Mick Jan 2006 B1
D514525 Stauffer Feb 2006 S
7003308 Fuoss Feb 2006 B1
7016116 Dolgoff Mar 2006 B2
7030925 Tsunekawa Apr 2006 B1
D521493 Wai May 2006 S
7088234 Naito Aug 2006 B2
D529467 Rose Oct 2006 S
D541226 Wakisaka Apr 2007 S
7199934 Yamasaki Apr 2007 B2
7206134 Weissman Apr 2007 B2
D559793 Fan Jan 2008 S
7347551 Fergason et al. Mar 2008 B2
D571816 Corcoran Jun 2008 S
7380936 Howell Jun 2008 B2
7401918 Howell Jul 2008 B2
7414791 Urakawa Aug 2008 B2
7457040 Amitai Nov 2008 B2
7481531 Howell Jan 2009 B2
7488294 Torch Feb 2009 B2
7500747 Howell Mar 2009 B2
7522344 Curatu Apr 2009 B1
7542210 Chirieleison, Sr. Jun 2009 B2
7543943 Hubby, Jr. Jun 2009 B1
7646540 Dolgoff Jan 2010 B2
7677723 Howell Mar 2010 B2
7690799 Nestorovic Apr 2010 B2
7728799 Kerr Jun 2010 B2
7771046 Howell Aug 2010 B2
7777690 Winsor Aug 2010 B2
7777723 Namiki Aug 2010 B2
7777960 Freeman Aug 2010 B2
7792552 Thomas et al. Sep 2010 B2
7806525 Howell Oct 2010 B2
7812842 Gordon Oct 2010 B2
7830370 Yamazaki Nov 2010 B2
7850301 Dichiara Dec 2010 B2
7855743 Sako Dec 2010 B2
D630213 Fidali Jan 2011 S
D631881 Quinn Feb 2011 S
D631882 Odgers Feb 2011 S
7928926 Yamamoto Apr 2011 B2
8004765 Amitai Aug 2011 B2
8018579 Krah Sep 2011 B1
8079713 Ashkenazi Dec 2011 B2
8092007 Dichiara Jan 2012 B2
8166421 Magal Apr 2012 B2
8212859 Tang Jul 2012 B2
8223024 Petrou Jul 2012 B1
8228315 Starner Jul 2012 B1
8235529 Raffle Aug 2012 B1
8246170 Yamamoto Aug 2012 B2
D669066 Olsson Oct 2012 S
8337013 Howell Dec 2012 B2
8376548 Schultz Feb 2013 B2
8378924 Jacobsen Feb 2013 B2
8384999 Crosby Feb 2013 B1
D680112 Monahan Apr 2013 S
D680152 Olsson Apr 2013 S
8427396 Kim Apr 2013 B1
8430507 Howell Apr 2013 B2
8434863 Howell May 2013 B2
D685019 Li Jun 2013 S
8464863 Pierret Jun 2013 B2
8473241 Foxlin Jun 2013 B2
8487838 Lewis Jul 2013 B2
8489326 Na Jul 2013 B1
8494215 Kimchi Jul 2013 B2
8505430 Miralles Aug 2013 B2
D689862 Liu Sep 2013 S
8531394 Maltz Sep 2013 B2
D690684 Lee Oct 2013 S
8553910 Dong Oct 2013 B1
8564883 Totani Oct 2013 B2
8570273 Smith Oct 2013 B1
8570656 Weissman Oct 2013 B1
8576276 Bar-Zeev Nov 2013 B2
8576491 Takagi Nov 2013 B2
8587869 Totani Nov 2013 B2
8593795 Chi Nov 2013 B1
8594467 Lu Nov 2013 B2
D696668 Chen Dec 2013 S
8611015 Wheeler Dec 2013 B2
D697914 Bates Jan 2014 S
8638498 Bohn et al. Jan 2014 B2
8662686 Takagi Mar 2014 B2
8670183 Clavin Mar 2014 B2
8678581 Blum Mar 2014 B2
8696113 Lewis Apr 2014 B2
8698157 Hanamura Apr 2014 B2
8711487 Takeda Apr 2014 B2
8730129 Solomon May 2014 B2
8733927 Lewis May 2014 B1
8733928 Lewis May 2014 B1
8743052 Keller Jun 2014 B1
8745058 Garcia-barrio Jun 2014 B1
8750541 Dong Jun 2014 B1
8752963 Mcculloch Jun 2014 B2
8760765 Gupta Jun 2014 B2
8767306 Miao Jul 2014 B1
8770742 Howell Jul 2014 B2
8786675 Deering Jul 2014 B2
8786686 Amirparviz Jul 2014 B1
8787006 Golko Jul 2014 B2
8803867 Oikawa Aug 2014 B2
8814691 Haddick Aug 2014 B2
8823071 Oyamada Sep 2014 B2
8824779 Smyth Sep 2014 B1
8832557 Fadell Sep 2014 B2
8836768 Rafii Sep 2014 B1
8837880 Takeda Sep 2014 B2
8854433 Rafii Oct 2014 B1
8866702 Wong Oct 2014 B1
8866849 Cho Oct 2014 B1
8867139 Gupta Oct 2014 B2
D716808 Yeom Nov 2014 S
D716813 Deng Nov 2014 S
D718309 Roberts Nov 2014 S
8878749 Wu Nov 2014 B1
D719568 Heinrich Dec 2014 S
D719569 Heinrich Dec 2014 S
D719570 Heinrich Dec 2014 S
8922530 Pance Dec 2014 B2
8929589 Publicover et al. Jan 2015 B2
8947323 Raffle Feb 2015 B1
8955973 Raffle Feb 2015 B2
8964298 Haddick Feb 2015 B2
D724083 Olsson Mar 2015 S
8970495 Biffle Mar 2015 B1
8971023 Olsson Mar 2015 B2
8982014 Evans Mar 2015 B2
8982471 Starner Mar 2015 B1
D727317 Olsson Apr 2015 S
9010929 Lewis Apr 2015 B2
9020832 Fisher Apr 2015 B2
D728573 Deng May 2015 S
9024842 Prada Gomez May 2015 B1
9031273 Dong May 2015 B2
9033502 Nistico May 2015 B2
D732025 Heinrich Jun 2015 S
9046999 Teller Jun 2015 B1
9063563 Gray Jun 2015 B1
9064436 Patel Jun 2015 B1
D733709 Kawai Jul 2015 S
9076368 Evans Jul 2015 B2
9096920 Gomez Aug 2015 B1
9107622 Nistico Aug 2015 B2
9116337 Miao Aug 2015 B1
D738373 Davies Sep 2015 S
9122054 Osterhout Sep 2015 B2
9128281 Osterhout Sep 2015 B2
9129157 Chao Sep 2015 B2
9129295 Border Sep 2015 B2
9141194 Keyes Sep 2015 B1
9143693 Zhou Sep 2015 B1
9158115 Worley Oct 2015 B1
9158116 Osterhout Oct 2015 B1
D743963 Osterhout Nov 2015 S
9176582 Johnson Nov 2015 B1
D745007 Cazalet Dec 2015 S
9202233 Siegel Dec 2015 B1
9223136 Braun Dec 2015 B1
9225934 Cho Dec 2015 B2
9229233 Osterhout Jan 2016 B2
9229234 Osterhout Jan 2016 B2
9235051 Salter Jan 2016 B2
D751551 Ho Mar 2016 S
D751552 Osterhout Mar 2016 S
9274338 Robbins et al. Mar 2016 B2
9286728 Osterhout Mar 2016 B2
9292082 Patel Mar 2016 B1
9292973 Bar-zeev et al. Mar 2016 B2
9298001 Border Mar 2016 B2
9298002 Border Mar 2016 B2
9298007 Border Mar 2016 B2
9299194 Border Mar 2016 B2
D753114 Osterhout Apr 2016 S
9310610 Border Apr 2016 B2
9316833 Border Apr 2016 B2
9323325 Perez et al. Apr 2016 B2
9324189 Vaughn Apr 2016 B2
D756363 Mathis May 2016 S
D757006 Cazalet May 2016 S
9329387 Border May 2016 B2
9354445 Weaver May 2016 B1
9366867 Border Jun 2016 B2
9366868 Border Jun 2016 B2
9374655 Lee Jun 2016 B1
9377625 Border Jun 2016 B2
9400390 Osterhout Jul 2016 B2
9401540 Osterhout Jul 2016 B2
9423612 Border Aug 2016 B2
9423842 Osterhout Aug 2016 B2
9436006 Border Sep 2016 B2
9448409 Border Sep 2016 B2
9523856 Osterhout Dec 2016 B2
9529195 Osterhout Dec 2016 B2
9529199 Osterhout Dec 2016 B2
9529385 Connor Dec 2016 B2
9575321 Osterhout Feb 2017 B2
9658473 Lewis May 2017 B2
9682622 Kim Jun 2017 B2
9690763 Lee Jun 2017 B1
D794637 Osterhout Aug 2017 S
9720505 Gribetz et al. Aug 2017 B2
9810906 Osterhout Nov 2017 B2
9939934 Osterhout Apr 2018 B2
10013053 Cederlund et al. Jul 2018 B2
10025379 Drake et al. Jul 2018 B2
10185147 Lewis Jan 2019 B2
10698212 Osterhout Jun 2020 B2
10732721 Clements Aug 2020 B1
11054645 Osterhout Jul 2021 B2
11294180 Osterhout et al. Apr 2022 B2
20010017614 Hara Aug 2001 A1
20010019240 Takahashi Sep 2001 A1
20010050817 Travers Dec 2001 A1
20020005108 Ludwig Jan 2002 A1
20020021498 Ohtaka Feb 2002 A1
20020024675 Foxlin Feb 2002 A1
20020109903 Kaeriyama Aug 2002 A1
20020148655 Cho Oct 2002 A1
20020149545 Hanayama Oct 2002 A1
20020183101 Oh Dec 2002 A1
20020191297 Gleckman Dec 2002 A1
20030030597 Geist Feb 2003 A1
20030030912 Gleckman Feb 2003 A1
20030142065 Pahlavan Jul 2003 A1
20030151834 Penn Aug 2003 A1
20030209953 Park Nov 2003 A1
20030234823 Sato Dec 2003 A1
20040003453 Urakawa Jan 2004 A1
20040024287 Patton Feb 2004 A1
20040027312 Owada Feb 2004 A1
20040032392 Chi Feb 2004 A1
20040066363 Yamano Apr 2004 A1
20040066547 Parker Apr 2004 A1
20040080541 Saiga Apr 2004 A1
20040130522 Lin Jul 2004 A1
20040150631 Fleck Aug 2004 A1
20040194880 Jiang Oct 2004 A1
20040227994 Bruzzone Nov 2004 A1
20050010563 Gross Jan 2005 A1
20050041289 Berman Feb 2005 A1
20050122319 Sakurai Jun 2005 A1
20050156915 Fisher Jul 2005 A1
20050212980 Miyazaki Sep 2005 A1
20060017654 Romo Jan 2006 A1
20060023158 Howell et al. Feb 2006 A1
20060047339 Brown Mar 2006 A1
20060047386 Kanevsky Mar 2006 A1
20060050146 Richardson Mar 2006 A1
20060061542 Stokic Mar 2006 A1
20060092131 Kuroki May 2006 A1
20060098293 Garoutte May 2006 A1
20060119794 Hillis Jun 2006 A1
20060132457 Rimas-ribikauskas Jun 2006 A1
20060132924 Mimran Jun 2006 A1
20060152686 Yeralan Jul 2006 A1
20060170652 Bannai Aug 2006 A1
20060173351 Marcotte Aug 2006 A1
20060178827 Aoyama Aug 2006 A1
20060215111 Mihashi Sep 2006 A1
20060224238 Azar Oct 2006 A1
20060238550 Page Oct 2006 A1
20060239629 Qi Oct 2006 A1
20060250322 Hall Nov 2006 A1
20060279549 Zhang Dec 2006 A1
20060285315 Tufenkjian Dec 2006 A1
20060288233 Kozlay Dec 2006 A1
20070003168 Oliver Jan 2007 A1
20070004451 C. Anderson Jan 2007 A1
20070024750 Wing Feb 2007 A1
20070024763 Chung Feb 2007 A1
20070024764 Chung Feb 2007 A1
20070024820 Chung Feb 2007 A1
20070024823 Chung Feb 2007 A1
20070025273 Chung Feb 2007 A1
20070030243 Ishii Feb 2007 A1
20070030456 Duncan Feb 2007 A1
20070035563 Biocca Feb 2007 A1
20070038960 Rekimoto Feb 2007 A1
20070058868 Seino Mar 2007 A1
20070070859 Hirayama Mar 2007 A1
20070075989 Cohen Apr 2007 A1
20070100637 Mccune May 2007 A1
20070120806 Schmidt May 2007 A1
20070120836 Yamaguchi May 2007 A1
20070132662 Morita Jun 2007 A1
20070153639 Lafever Jul 2007 A1
20070156364 Rothkopf Jul 2007 A1
20070178950 Lewis Aug 2007 A1
20070194902 Blanco et al. Aug 2007 A1
20070229458 Moon Oct 2007 A1
20070233376 Gershony Oct 2007 A1
20070236450 Colgate Oct 2007 A1
20070263174 Cheng Nov 2007 A1
20070273611 Torch Nov 2007 A1
20070282682 Dietz Dec 2007 A1
20070296684 Thomas Dec 2007 A1
20080005702 Skourup Jan 2008 A1
20080066973 Furuki Mar 2008 A1
20080121441 Sheets May 2008 A1
20080143954 Abreu Jun 2008 A1
20080186255 Cohen Aug 2008 A1
20080191965 Pandozy Aug 2008 A1
20080219025 Spitzer Sep 2008 A1
20080266645 Dharmatilleke Oct 2008 A1
20080291277 Jacobsen Nov 2008 A1
20090015735 Simmonds Jan 2009 A1
20090040296 Moscato Feb 2009 A1
20090108837 Johansson Apr 2009 A1
20090110241 Takemoto Apr 2009 A1
20090147331 Ashkenazi Jun 2009 A1
20090183929 Zhang Jul 2009 A1
20090251441 Edgecomb Oct 2009 A1
20090279180 Amitai Nov 2009 A1
20100001572 Masunaga Jan 2010 A1
20100007852 Bietry Jan 2010 A1
20100046075 Powell Feb 2010 A1
20100056274 Uusitalo Mar 2010 A1
20100060713 Snyder Mar 2010 A1
20100079508 Hodge Apr 2010 A1
20100079733 Lu Apr 2010 A1
20100082368 Gecelter Apr 2010 A1
20100085325 King-smith Apr 2010 A1
20100094161 Kiderman Apr 2010 A1
20100097580 Yamamoto Apr 2010 A1
20100103075 Kalaboukis Apr 2010 A1
20100130140 Waku May 2010 A1
20100134848 Lynggaard Jun 2010 A1
20100149073 Chaum Jun 2010 A1
20100178101 Day Jul 2010 A1
20100182561 Ikeda Jul 2010 A1
20100194682 Orr Aug 2010 A1
20100240988 Varga Sep 2010 A1
20100241450 Gierhart Sep 2010 A1
20100254017 Martins Oct 2010 A1
20100283774 Bovet Nov 2010 A1
20100290127 Kessler Nov 2010 A1
20100309097 Raviv Dec 2010 A1
20100329301 Pang Dec 2010 A1
20110006982 Rhee Jan 2011 A1
20110007081 Gordon Jan 2011 A1
20110012874 Kurozuka Jan 2011 A1
20110089325 Ottney Apr 2011 A1
20110096100 Sprague Apr 2011 A1
20110102234 Adams May 2011 A1
20110109587 Ferencz May 2011 A1
20110130958 Stahl Jun 2011 A1
20110131495 Bull Jun 2011 A1
20110157236 Inoue Jun 2011 A1
20110164047 Pance Jul 2011 A1
20110164163 Bilbrey Jul 2011 A1
20110164221 Tilleman Jul 2011 A1
20110176106 Lewkowski Jul 2011 A1
20110196610 Waldman Aug 2011 A1
20110199171 Prest Aug 2011 A1
20110199305 Suh Aug 2011 A1
20110201213 Dabov Aug 2011 A1
20110202823 Berger Aug 2011 A1
20110205209 Kurokawa Aug 2011 A1
20110211056 Publicover et al. Sep 2011 A1
20110213664 Osterhout Sep 2011 A1
20110221658 Haddick Sep 2011 A1
20110221672 Osterhout Sep 2011 A1
20110221896 Haddick Sep 2011 A1
20110227820 Haddick Sep 2011 A1
20110234631 Kim Sep 2011 A1
20110248963 Lawrence Oct 2011 A1
20110285638 Harris Nov 2011 A1
20110285764 Kimura Nov 2011 A1
20120021806 Maltz Jan 2012 A1
20120026088 Goran Feb 2012 A1
20120035934 Cunningham Feb 2012 A1
20120049759 Pezzutti Mar 2012 A1
20120050140 Border Mar 2012 A1
20120050493 Ernst Mar 2012 A1
20120056093 Poteet Mar 2012 A1
20120062444 Cok Mar 2012 A1
20120062445 Haddick Mar 2012 A1
20120062594 Campbell Mar 2012 A1
20120062998 Schultz Mar 2012 A1
20120068913 Bar-zeev Mar 2012 A1
20120069413 Schultz Mar 2012 A1
20120075168 Osterhout Mar 2012 A1
20120078628 Ghulman Mar 2012 A1
20120081800 Cheng Apr 2012 A1
20120092328 Flaks Apr 2012 A1
20120092329 Koo Apr 2012 A1
20120096095 Bhargava Apr 2012 A1
20120113514 Rodman May 2012 A1
20120119978 Border May 2012 A1
20120120103 Border May 2012 A1
20120120498 Harrison May 2012 A1
20120127062 Bar-zeev May 2012 A1
20120127284 Bar-zeev May 2012 A1
20120133885 Howell May 2012 A1
20120139915 Muikaichi Jun 2012 A1
20120154920 Harrison Jun 2012 A1
20120162270 Fleck Jun 2012 A1
20120163013 Buelow, II Jun 2012 A1
20120169608 Forutanpour Jul 2012 A1
20120176682 Dejong Jul 2012 A1
20120188245 Hyatt Jul 2012 A1
20120194550 Osterhout Aug 2012 A1
20120194553 Osterhout Aug 2012 A1
20120194784 Shih Aug 2012 A1
20120200935 Miyao Aug 2012 A1
20120212398 Border Aug 2012 A1
20120212484 Haddick Aug 2012 A1
20120212499 Haddick Aug 2012 A1
20120212593 Na Aug 2012 A1
20120218301 Miller Aug 2012 A1
20120223885 Perez Sep 2012 A1
20120229248 Parshionikar Sep 2012 A1
20120229367 Magyari Sep 2012 A1
20120233000 Fisher Sep 2012 A1
20120235885 Miller Sep 2012 A1
20120237085 Meier Sep 2012 A1
20120242251 Kwisthout Sep 2012 A1
20120242697 Border Sep 2012 A1
20120249741 Maciocci Oct 2012 A1
20120249797 Haddick Oct 2012 A1
20120250152 Larson Oct 2012 A1
20120256944 Crumly Oct 2012 A1
20120264510 Wigdor Oct 2012 A1
20120287398 Baker Nov 2012 A1
20120293548 Perez Nov 2012 A1
20120294478 Publicover Nov 2012 A1
20120302347 Nicholson Nov 2012 A1
20120306850 Balan Dec 2012 A1
20120326948 Crocco Dec 2012 A1
20120327040 Simon et al. Dec 2012 A1
20120327116 Liu Dec 2012 A1
20130002545 Heinrich Jan 2013 A1
20130002724 Heinrich Jan 2013 A1
20130009366 Hannegan Jan 2013 A1
20130009907 Rosenberg Jan 2013 A1
20130022274 Lawrence Jan 2013 A1
20130027341 Mastandrea Jan 2013 A1
20130038729 Chang Feb 2013 A1
20130069985 Wong Mar 2013 A1
20130070344 Takeda Mar 2013 A1
20130077049 Bohn Mar 2013 A1
20130077147 Efimov Mar 2013 A1
20130083009 Geisner Apr 2013 A1
20130083055 Piemonte Apr 2013 A1
20130088413 Raffle Apr 2013 A1
20130100259 Ramaswamy Apr 2013 A1
20130106674 Wheeler May 2013 A1
20130120224 Cajigas May 2013 A1
20130120841 Shpunt May 2013 A1
20130127906 Sugita May 2013 A1
20130127980 Haddick May 2013 A1
20130135198 Hodge May 2013 A1
20130135244 Lynch May 2013 A1
20130141434 Sugden Jun 2013 A1
20130154913 Genc Jun 2013 A1
20130162632 Varga Jun 2013 A1
20130169560 Cederlund Jul 2013 A1
20130176533 Raffle Jul 2013 A1
20130185052 Boyd Jul 2013 A1
20130194389 Vaught Aug 2013 A1
20130196757 Latta Aug 2013 A1
20130201080 Evans Aug 2013 A1
20130201081 Evans Aug 2013 A1
20130207887 Raffle Aug 2013 A1
20130207970 Shpunt Aug 2013 A1
20130215149 Hayashi Aug 2013 A1
20130222270 Winkler Aug 2013 A1
20130222919 Komatsu Aug 2013 A1
20130230215 Gurman Sep 2013 A1
20130234914 Fujimaki Sep 2013 A1
20130235331 Heinrich Sep 2013 A1
20130241805 Gomez Sep 2013 A1
20130241948 Kimura Sep 2013 A1
20130242405 Gupta Sep 2013 A1
20130248691 Mirov Sep 2013 A1
20130249778 Morimoto Sep 2013 A1
20130249787 Morimoto Sep 2013 A1
20130249849 Wong Sep 2013 A1
20130249946 Kimura Sep 2013 A1
20130250207 Bohn Sep 2013 A1
20130250430 Robbins Sep 2013 A1
20130250503 Olsson Sep 2013 A1
20130257622 Davalos Oct 2013 A1
20130257709 Raffle Oct 2013 A1
20130257899 Baron Oct 2013 A1
20130258111 Frank Oct 2013 A1
20130265212 Kato Oct 2013 A1
20130265227 Julian Oct 2013 A1
20130278631 Border Oct 2013 A1
20130288761 Santos Paiva Ferraz Conceicao Oct 2013 A1
20130293530 Perez Nov 2013 A1
20130293580 Spivack Nov 2013 A1
20130300637 Smits Nov 2013 A1
20130300652 Raffle Nov 2013 A1
20130321265 Bychkov Dec 2013 A1
20130321271 Bychkov Dec 2013 A1
20130321932 Hsu Dec 2013 A1
20130335301 Wong Dec 2013 A1
20130335461 Rekimoto Dec 2013 A1
20130336528 Itani Dec 2013 A1
20130336629 Mulholland Dec 2013 A1
20130342564 Kinnebrew Dec 2013 A1
20130342571 Kinnebrew Dec 2013 A1
20130342981 Cox Dec 2013 A1
20130346245 Desore Dec 2013 A1
20140002496 Lamb Jan 2014 A1
20140028704 Wu Jan 2014 A1
20140043682 Hussey Feb 2014 A1
20140062854 Cho Mar 2014 A1
20140063054 Osterhout Mar 2014 A1
20140063055 Osterhout Mar 2014 A1
20140063473 Pasolini Mar 2014 A1
20140078043 Kim Mar 2014 A1
20140078282 Aoki Mar 2014 A1
20140091984 Ashbrook Apr 2014 A1
20140101608 Ryskamp Apr 2014 A1
20140104142 Bickerstaff Apr 2014 A1
20140104692 Bickerstaff Apr 2014 A1
20140111838 Han Apr 2014 A1
20140115520 Itani Apr 2014 A1
20140125785 Na May 2014 A1
20140129328 Mathew May 2014 A1
20140139655 Mimar May 2014 A1
20140146394 Tout May 2014 A1
20140147829 Jerauld May 2014 A1
20140152530 Venkatesha Jun 2014 A1
20140152558 Salter Jun 2014 A1
20140152676 Rohn Jun 2014 A1
20140153173 Pombo Jun 2014 A1
20140159995 Adams Jun 2014 A1
20140160055 Margolis Jun 2014 A1
20140160137 Martin Jun 2014 A1
20140160157 Poulos Jun 2014 A1
20140160170 Lyons Jun 2014 A1
20140165000 Fleizach Jun 2014 A1
20140168056 Swaminathan Jun 2014 A1
20140168266 Kimura Jun 2014 A1
20140168716 King Jun 2014 A1
20140168735 Yuan Jun 2014 A1
20140176591 Klein Jun 2014 A1
20140176603 Kumar Jun 2014 A1
20140177023 Gao Jun 2014 A1
20140183269 Glaser Jul 2014 A1
20140191942 Kobayashi Jul 2014 A1
20140195918 Friedlander Jul 2014 A1
20140204062 Goto Jul 2014 A1
20140204759 Guo et al. Jul 2014 A1
20140213280 Sandel Jul 2014 A1
20140222929 Grossman Aug 2014 A1
20140225814 English Aug 2014 A1
20140232651 Kress Aug 2014 A1
20140240103 Lake et al. Aug 2014 A1
20140240223 Lake Aug 2014 A1
20140240313 Varga Aug 2014 A1
20140247286 Chi Sep 2014 A1
20140253588 Mandala Sep 2014 A1
20140253605 Border Sep 2014 A1
20140266988 Fisher Sep 2014 A1
20140267010 Pasquero Sep 2014 A1
20140279528 Slaby Sep 2014 A1
20140285631 Janky Sep 2014 A1
20140306866 Miller Oct 2014 A1
20140310075 Ricci Oct 2014 A1
20140320383 Goto Oct 2014 A1
20140320389 Scavezze Oct 2014 A1
20140320971 Gupta Oct 2014 A1
20140341441 Slaby Nov 2014 A1
20140361957 Hua Dec 2014 A1
20140363797 Hu Dec 2014 A1
20140372957 Keane Dec 2014 A1
20140375542 Robbins Dec 2014 A1
20140375545 Ackerman Dec 2014 A1
20140375680 Ackerman Dec 2014 A1
20140375683 Salter Dec 2014 A1
20150002371 Burgess Jan 2015 A1
20150012581 Kim Jan 2015 A1
20150016777 Abovitz Jan 2015 A1
20150029088 Kim Jan 2015 A1
20150040040 Balan Feb 2015 A1
20150042544 Sugihara Feb 2015 A1
20150085333 Theytaz Mar 2015 A1
20150097719 Balachandreswaran Apr 2015 A1
20150123887 Shadle May 2015 A1
20150128251 Yoon May 2015 A1
20150143297 Wheeler May 2015 A1
20150145839 Hack May 2015 A1
20150146004 Kritt May 2015 A1
20150156716 Raffle Jun 2015 A1
20150161913 Dominguez Jun 2015 A1
20150168731 Robbins Jun 2015 A1
20150169953 Border Jun 2015 A1
20150178932 Wyatt Jun 2015 A1
20150181383 Schulz Jun 2015 A1
20150185476 Lee Jul 2015 A1
20150186636 Tharappel Jul 2015 A1
20150198807 Hirai Jul 2015 A1
20150201834 Border Jul 2015 A1
20150201835 Border Jul 2015 A1
20150201836 Border Jul 2015 A1
20150202962 Habashima Jul 2015 A1
20150205035 Border Jul 2015 A1
20150205100 Border Jul 2015 A1
20150205101 Border Jul 2015 A1
20150205102 Border Jul 2015 A1
20150205103 Border Jul 2015 A1
20150205104 Border Jul 2015 A1
20150205105 Border Jul 2015 A1
20150205107 Border Jul 2015 A1
20150205108 Border Jul 2015 A1
20150205111 Border Jul 2015 A1
20150205112 Border Jul 2015 A1
20150205113 Border Jul 2015 A1
20150205114 Border Jul 2015 A1
20150205115 Border Jul 2015 A1
20150205116 Border Jul 2015 A1
20150205117 Border Jul 2015 A1
20150205118 Border Jul 2015 A1
20150205119 Osterhout Jul 2015 A1
20150205120 Border Jul 2015 A1
20150205121 Border Jul 2015 A1
20150205122 Border Jul 2015 A1
20150205123 Border Jul 2015 A1
20150205124 Border Jul 2015 A1
20150205125 Border Jul 2015 A1
20150205126 Schowengerdt Jul 2015 A1
20150205127 Border Jul 2015 A1
20150205128 Border Jul 2015 A1
20150205129 Border Jul 2015 A1
20150205130 Border Jul 2015 A1
20150205131 Border Jul 2015 A1
20150205132 Osterhout Jul 2015 A1
20150205135 Border Jul 2015 A1
20150205346 Border Jul 2015 A1
20150205347 Border Jul 2015 A1
20150205348 Nortrup Jul 2015 A1
20150205349 Nortrup Jul 2015 A1
20150205351 Osterhout Jul 2015 A1
20150205373 Osterhout Jul 2015 A1
20150205378 Osterhout Jul 2015 A1
20150205384 Osterhout Jul 2015 A1
20150205385 Osterhout Jul 2015 A1
20150205387 Osterhout Jul 2015 A1
20150205388 Osterhout Jul 2015 A1
20150205401 Osterhout Jul 2015 A1
20150205402 Osterhout Jul 2015 A1
20150205494 Scott Jul 2015 A1
20150205566 Osterhout Jul 2015 A1
20150206008 Border Jul 2015 A1
20150206173 Nortrup Jul 2015 A1
20150212324 Osterhout Jul 2015 A1
20150212327 Osterhout Jul 2015 A1
20150212647 Kim Jul 2015 A1
20150213584 Ishikawa Jul 2015 A1
20150213650 Barzuza Jul 2015 A1
20150226966 Osterhout Aug 2015 A1
20150226967 Osterhout Aug 2015 A1
20150228099 Osterhout Aug 2015 A1
20150228119 Osterhout Aug 2015 A1
20150228120 Osterhout Aug 2015 A1
20150229019 Osterhout Aug 2015 A1
20150234508 Cho Aug 2015 A1
20150235422 Lohse Aug 2015 A1
20150235429 Miller Aug 2015 A1
20150235622 Border Aug 2015 A1
20150241963 Nortrup Aug 2015 A1
20150241964 Nortrup Aug 2015 A1
20150241965 Nortrup Aug 2015 A1
20150241966 Nortrup Aug 2015 A1
20150243039 Holz Aug 2015 A1
20150245131 Facteau Aug 2015 A1
20150260986 Nortrup Sep 2015 A1
20150261015 Han Sep 2015 A1
20150277113 Border Oct 2015 A1
20150277116 Richards Oct 2015 A1
20150277118 Border Oct 2015 A1
20150277120 Border Oct 2015 A1
20150277122 Border Oct 2015 A1
20150277549 Border Oct 2015 A1
20150277559 Vescovi Oct 2015 A1
20150279104 Border Oct 2015 A1
20150279107 Border Oct 2015 A1
20150279108 Border Oct 2015 A1
20150287048 Nortrup Oct 2015 A1
20150293587 Wilairat Oct 2015 A1
20150293738 Lee Oct 2015 A1
20150294156 Border Oct 2015 A1
20150294627 Yoo Oct 2015 A1
20150301593 Border Oct 2015 A1
20150302646 Osterhout Oct 2015 A1
20150302647 Osterhout Oct 2015 A1
20150304368 Vaccari Oct 2015 A1
20150309313 Border Oct 2015 A1
20150309314 Border Oct 2015 A1
20150309317 Osterhout Oct 2015 A1
20150309534 Osterhout Oct 2015 A1
20150309562 Shams Oct 2015 A1
20150309995 Osterhout Oct 2015 A1
20150316766 Weaver Nov 2015 A1
20150316769 Border Nov 2015 A1
20150316770 Border Nov 2015 A1
20150316771 Border Nov 2015 A1
20150316772 Border Nov 2015 A1
20150331241 Haddick Nov 2015 A1
20150338661 Osterhout Nov 2015 A1
20150346496 Haddick Dec 2015 A1
20150346511 Osterhout Dec 2015 A1
20150347823 Monnerat Dec 2015 A1
20150355466 Border Dec 2015 A1
20150355468 Osterhout Dec 2015 A1
20150356772 Osterhout Dec 2015 A1
20150356775 Osterhout Dec 2015 A1
20150356776 Osterhout Dec 2015 A1
20150356777 Osterhout Dec 2015 A1
20150356778 Osterhout Dec 2015 A1
20150356779 Osterhout Dec 2015 A1
20150382305 Drincic Dec 2015 A1
20160005003 Norris Jan 2016 A1
20160011417 Border Jan 2016 A1
20160015470 Border Jan 2016 A1
20160018640 Haddick Jan 2016 A1
20160018641 Haddick Jan 2016 A1
20160018642 Haddick Jan 2016 A1
20160018644 Border Jan 2016 A1
20160018645 Haddick Jan 2016 A1
20160018646 Osterhout Jan 2016 A1
20160018647 Osterhout Jan 2016 A1
20160018648 Osterhout Jan 2016 A1
20160018649 Osterhout Jan 2016 A1
20160018650 Haddick Jan 2016 A1
20160018651 Haddick Jan 2016 A1
20160018652 Haddick Jan 2016 A1
20160018653 Haddick Jan 2016 A1
20160018654 Haddick Jan 2016 A1
20160019715 Haddick Jan 2016 A1
20160019719 Osterhout Jan 2016 A1
20160021304 Osterhout Jan 2016 A1
20160025974 Osterhout Jan 2016 A1
20160025977 Osterhout Jan 2016 A1
20160025979 Border Jan 2016 A1
20160025980 Osterhout Jan 2016 A1
20160026239 Border Jan 2016 A1
20160027211 Osterhout Jan 2016 A1
20160027414 Osterhout Jan 2016 A1
20160048019 Haddick Feb 2016 A1
20160048021 Border Feb 2016 A1
20160048023 Haddick Feb 2016 A1
20160048160 Haddick Feb 2016 A1
20160049008 Haddick Feb 2016 A1
20160054566 Osterhout Feb 2016 A1
20160056647 Choi Feb 2016 A1
20160062118 Osterhout Mar 2016 A1
20160062121 Border Mar 2016 A1
20160062122 Border Mar 2016 A1
20160077342 Osterhout Mar 2016 A1
20160078278 Moore Mar 2016 A1
20160080897 Moore Mar 2016 A1
20160085071 Border Mar 2016 A1
20160085072 Haddick Mar 2016 A1
20160085278 Osterhout Mar 2016 A1
20160091718 Border Mar 2016 A1
20160091719 Border Mar 2016 A1
20160098086 Li Apr 2016 A1
20160109709 Osterhout Apr 2016 A1
20160109711 Border Apr 2016 A1
20160109713 Osterhout Apr 2016 A1
20160116738 Osterhout Apr 2016 A1
20160116745 Osterhout Apr 2016 A1
20160116979 Border Apr 2016 A1
20160131904 Border May 2016 A1
20160131911 Border May 2016 A1
20160131912 Border May 2016 A1
20160132082 Border May 2016 A1
20160133201 Border May 2016 A1
20160137312 Osterhout May 2016 A1
20160140766 Balachandreswaran May 2016 A1
20160147063 Border May 2016 A1
20160147064 Border May 2016 A1
20160147065 Border May 2016 A1
20160147070 Border May 2016 A1
20160154242 Border Jun 2016 A1
20160154244 Border Jun 2016 A1
20160161743 Osterhout Jun 2016 A1
20160161747 Osterhout Jun 2016 A1
20160170207 Haddick Jun 2016 A1
20160170208 Border Jun 2016 A1
20160170209 Border Jun 2016 A1
20160170699 Border Jun 2016 A1
20160171769 Haddick Jun 2016 A1
20160187651 Border Jun 2016 A1
20160187658 Osterhout Jun 2016 A1
20160202946 Osterhout Jul 2016 A1
20160207457 Border Jul 2016 A1
20160216516 Border Jul 2016 A1
20160216517 Border Jul 2016 A1
20160231571 Border Aug 2016 A1
20160239985 Haddick Aug 2016 A1
20160240008 Haddick Aug 2016 A1
20160246055 Border Aug 2016 A1
20160252731 Border Sep 2016 A1
20160259166 Border Sep 2016 A1
20160274361 Border Sep 2016 A1
20160282626 Border Sep 2016 A1
20160286177 Border Sep 2016 A1
20160286203 Border Sep 2016 A1
20160286210 Border Sep 2016 A1
20160299569 Fisher Oct 2016 A1
20160329634 Osterhout Nov 2016 A1
20160375306 Gu Dec 2016 A1
20170017323 Yu Jan 2017 A1
20170024007 Pelis Jan 2017 A1
20170024035 Pelis Jan 2017 A1
20170031395 Osterhout Feb 2017 A1
20170075155 Oikawa Mar 2017 A1
20170100664 Osterhout Apr 2017 A1
20170123492 Marggraff May 2017 A1
20170153672 Shin Jun 2017 A1
20170160812 Park Jun 2017 A1
20170185214 Kojima Jun 2017 A1
20170187855 Hoellwarth Jun 2017 A1
20170293351 Li Oct 2017 A1
20170322416 Osterhout Nov 2017 A1
20170322627 Osterhout Nov 2017 A1
20170322641 Osterhout Nov 2017 A1
20170324860 Novet Nov 2017 A1
20170336872 Osterhout Nov 2017 A1
20180011571 Colgate Jan 2018 A1
20180113504 Osterhout Apr 2018 A1
20180113524 Osterhout Apr 2018 A1
20180113590 Osterhout Apr 2018 A1
20220188563 Baughman Jun 2022 A1
Foreign Referenced Citations (39)
Number Date Country
2316473 Jan 2001 CA
2362895 Dec 2002 CA
2388766 Dec 2003 CA
111142673 May 2020 CN
368898 May 1990 EP
777867 Jun 1997 EP
1326121 Jan 2007 EP
2207164 Jul 2010 EP
2486450 Aug 2012 EP
2502410 Sep 2012 EP
2491984 Dec 2012 GB
H07110735 Apr 1995 JP
200102036 Apr 2000 JP
2005138755 Jun 2005 JP
2009171505 Jul 2009 JP
5017989 Sep 2012 JP
2012212990 Nov 2012 JP
1020110101944 Sep 2011 KR
2011143655 Nov 2011 WO
2012040030 Mar 2012 WO
2012058175 May 2012 WO
2012064546 May 2012 WO
2012082807 Jun 2012 WO
2012118573 Sep 2012 WO
2012118575 Sep 2012 WO
2013043288 Mar 2013 WO
2013049248 Apr 2013 WO
2013050650 Apr 2013 WO
2013103825 Jul 2013 WO
2013110846 Aug 2013 WO
2013176079 Nov 2013 WO
2015109145 Jul 2015 WO
2015164276 Oct 2015 WO
2015179877 Nov 2015 WO
2015195444 Dec 2015 WO
2016044035 Mar 2016 WO
2016073734 May 2016 WO
2016133886 Aug 2016 WO
WO-2017095790 Jun 2017 WO
Non-Patent Literature Citations (49)
Entry
US 8,743,465 B2, 06/2014, Totani (withdrawn)
US 8,792,178 B2, 07/2014, Totani (withdrawn)
US 9,195,056 B2, 11/2015, Border (withdrawn)
PCT/US2015/059264, , “International Application Serial No. PCT/US2015/059264, International Search Report and Written Opinion mailed Feb. 19, 2016”, Osterhout Group, Inc., 11 Pages.
Allison, R S. et al. “Tolerance of Temporal Delay In Virtual Environments,” VR '01 Proceedings of the Virtual Reality 2001 Conference (VR'01), Centre for Vision Research and Departments of Computer Science and Psychology, Mar. 2001, 2-8.
Bezryadin, et al. “Brightness Calculation in Digital Image Processing,” Technologies for Digital Fulfillment 2007, Las Vegas, NV, 2007, pp. 1-6.
Fathi, A. et al. (2012). “Social interactions: A First-Person Perspective,” Computer Vision and Pattern Recognition (CVPR), IEEE Conference on. IEEE, 2012, eight Pages.
Final Office Action mailed Oct. 11, 2019, for U.S. Appl. No. 15/785,505, filed Oct. 17, 2017, eleven pages.
Final Rejection mailed Jan. 5, 2017,, for U.S. Appl. No. 14/307,114, filed Jun. 17, 2014, ten pages.
Huang, Jin-Bin , “Image Completion Using Planar Structure Guidelines”, ACM Transactions on Graphics, vol. 33, No. 4, Article 129, Jul. 2014, pp. 1-10.
Janin, Adam L. et al., “Calibration of Head-Mounted Displays for Augmented Reality Applications”, Research and Technology Boeing Computer Services MS ?L-48 P.O. Box 24346 Seattle, WA 98124-0346 Virtual Reality Annual International Symposium, 1993., 1993 IEEE,, 1993, 10 Pages.
Lang, Manuel et al. “Nonlinear Disparity Mapping For Stereoscopic 3D”, Jul. 2010, pp. 1-10.
Logbar Inc., “Ring: Shortcut Everything”, https://www.kickstarter.com/projects/1761670738/ring-shrotcut-everything, Dec. 2014, 22 pages.
Mastandrea. “Mycestro, The Next Generation 3D Mouse,” https://www.kickstarter.com/projects/mycestro/mycestrotm-the-next-generation-3d-mouse. Dec. 2014, 22 pages.
Non-Final Rejection mailed Apr. 8, 2016,, for U.S. Appl. No. 14/307,114, filed Jun. 17, 2014, ten pages.
Non-Final Rejection mailed Jul. 1, 2019, for U.S. Appl. No. 15/785,505, filed Oct. 17, 2017, seven pages.
Notice of Allowance mailed Feb. 27, 2020, for U.S. Appl. No. 15/785,505, filed Oct. 17, 2017, eight pages.
Notice of Allowance mailed Jul. 28, 2017,, for U.S. Appl. No. 14/307,114, filed Jun. 17, 2014, eight pages.
Optical head-mounted display. (Jun. 9, 2014). Retrieved Oct. 7, 2019, from https://en.wikipedia.org/w/index.php?title=Optical_head-mounted_display&oldid=612157920, thirteen pages.
Pamplona, Vitor R. et al., “Photorealistic Models for Pupil Light Reflex and Iridal Pattern Deformation”, ACM Transactions on Graphics, vol. 28, No. 4, Article 106, Publication date: Aug. 2009, pp. 1-12.
PCT/US2015/011697, “International Application Serial No. PCT/US2015/011697, International Search Report and Written Opinion mailed Apr. 13, 2015”, Osterhout Group, Inc., 14 pages.
PCT/US2015/026704, “International Search Report and Written Opinion” mailed Aug. 21, 2015, 15 pages.
PCT/US2015/035192, , “International Application Serial No. PCT/US2015/035192, International Search Report and Written Opinion mailed Sep. 3, 2015”, Osterhout Group, Inc., 11 pages.
PCTU.S. Pat. No. 2015033379, , “International Application Serial No. PCT/US2015/033379, International Search Report and Written Opinion mailed Nov. 30, 2015”, Osterhout Group, Inc., 12 Pages.
Plainis, et al., “The Physiologic Mechanism of Accommodation”, Cataract & Refractive Surgery Today Europe, Apr. 2014, pp. 23-29.
Schedwill, “Bidirectional OLED Microdisplay”, Fraunhofer Research Institution for Organics, Materials and Electronic Device COMEDD, Apr. 11, 2014, 2 pages.
Vogel, et al., “Data glasses controlled by eye movements”, Information and communication, Fraunhofer-Gesellschaft J9/22/2013, 2 pages.
Walton, Z. “Wear This Smartphone Controller on Your Finger,” http://www.webpronews.com/wear-this-smartphone-controller-on-your-finger-2012-06, 5 pages.
Ye, Hui et al., “High Quality Voice Morphing”, Cambridge University Engineering Department Trumpington Street, Cambridge, England, CB2 1PZ, 2004, 1-9-1-12.
“Audio Spotlight,” by Holosonics, http://www.holosonics.com, accessed Jul. 3, 2014, three pages.
“Genius Ring Mice,” http://www.geniusnet.com/Genius/wSite/productCompare/compare.jsp, Dec. 23, 2014, one page.
“Help Requested! Comments and input needed for new coaxial UAS-DIY Drones,” http://diydrones.com/profiles/blogs/ help-requested-comments-and-input-needed-for-new-coaxial-uas, Mar. 5, 2015, 1-3.
“How Ascent AeroSystems is looking to add to your outdoor adventure,” http://droneblog.com/2015/03/23/how-ascent-aerosystems-is-looking-to-add-to-your-outdoor-adventure/#!prettyPhoto, Mar. 23, 2015, 1-10.
“Lightberry,” https://web.archive.org/web/20131201194408/http:1/lightberry.eu/, Dec. 1, 2013, 11 pages.
“Meet Nod, the Bluetooth Ring That Wants to Replace your Mouse,” http://www.pcmag.com/article2/0.2817.2457238.00.asp, Apr. 29, 2014, 6 pages.
“Sound from Ultrasound,” Wikipedia entry, http://en.m.wikipedia.org/wiki/Sound_from_ultrasound, accessed Jul. 3, 2014, 13 pages.
Azuma, Ronald T. (Aug. 1997). “A Survey of Augmented Reality,” In Presence: Teleoperators and Virtual Environments 6, 4, Hughes Research Laboratories, Malibu, CA, located at: https://web.archive.org/web/20010604100006/http://www.cs.unc.edu/˜azuma/ARpresence.pdf , retrieved on Oct. 26, 2020.
Bimber, Oliver et al. (2005). “Spatial Augmented Reality: Merging Real and Virtual Worlds,” A. K. Peters, Ltd., Wellesley, MA.
Wikipedia. Head-mounted display. (May 11, 2014). Retrieved Oct. 21, 2020, from https://en.wikipedia.org/w/index.php?title=Head-mounted displayoldid=608091 327 (Year: 2014).
Non-Final Rejection mailed Oct. 23, 2020, for U.S. Appl. No. 16/915,919, filed Jun. 29, 2020, 11 pages.
Notice of Allowance mailed Mar. 4, 2021, for U.S. Appl. No. 16/915,919, filed Jun. 29, 2020, 8 pages.
Notice of Allowance mailed Oct. 21, 2021, for U.S. Appl. No. 17/338,484, filed Jun. 3, 2021, 8 pages.
Notice of Allowance mailed Jan. 11, 2022 for U.S. Appl. No. 17/338,484, filed Jun. 3, 2021, 9 pages.
Non-Final Office Action mailed Apr. 27, 2023, for U.S. Appl. No. 17/670,373, filed Feb. 11, 2022, six pages.
Notice of Allowance mailed Aug. 9, 2023, for U.S. Appl. No. 17/670,373, filed Feb. 11, 2022, five pages.
Jacob, R. “Eye Tracking in Advanced Interface Design”, Virtual Environments and Advanced Interface Design, Oxford University Press, Inc. (Jun. 1995).
Rolland, J. et al., “High- resolution inset head- mounted display”, Optical Society of America, vol. 37, No. 19, Applied Optics, (Jul. 1, 1998).
Tanriverdi, V. et al. (Apr. 2000). “Interacting With Eye Movements In Virtual Environments,” Department of Electrical Engineering and Computer Science, Tufts University, Medford, MA 02155, USA, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, eight pages.
Yoshida, A. et al., “Design and Applications of a High Resolution Insert Head Mounted Display”, (Jun. 1994).
Related Publications (1)
Number Date Country
20230418064 A1 Dec 2023 US
Continuations (5)
Number Date Country
Parent 17670373 Feb 2022 US
Child 18461436 US
Parent 17338484 Jun 2021 US
Child 17670373 US
Parent 16915919 Jun 2020 US
Child 17338484 US
Parent 15785505 Oct 2017 US
Child 16915919 US
Parent 14307114 Jun 2014 US
Child 15785505 US