Patent Grant
11681369
There are many human interface input devices for data, control, and command entry into computing devices and other systems that are in common use. These input devices are typified by a keyboard, mouse, touch pad, joystick, graphics tablet, electronic pen, and various motion sensitive or motion activated controllers. These input devices are routinely encountered in home, office, or industrial settings or in the rapidly expanding areas of console, computer, and on-line gaming There are also many interface devices that have been custom designed for automation and robotic control or to provide alternative control and command capture methods for a wide range of specialized devices and applications.
One type of human interface input device that frees the wearer from many of the drawbacks of traditional input devices uses touch sensors directly embedded or otherwise integrated into garments, such as gloves, that are worn by an individual. Signals generated by touching these sensors are routed through various circuits sewn, woven, or otherwise integrated into or attached to the garment and conveyed to signal processing circuits mounted at strategic locations therein or that may be transmitted to signal processing circuits external to the garment. The signal processing circuits analyze these signals and construct appropriate messages corresponding to the signals that can be sent to a computer or other similar equipment to simulate or mimic traditional wearer input devices.
One common application that uses garments, such as gloves, as a human interface input device is virtual reality (VR). In VR applications, the garment wearer's movements need to be translated into virtual movements within a virtual world. There are a number of companies recreating physical instrument panels in training situations using VR with a corresponding real-world control panel training setup that the user can touch and feel. In addition to the use of VR for video gaming systems, VR has many other applications. For example, drug companies use VR to train employees when handling, mixing, or even just picking up special items. As another example, aerospace training simulators use VR to mimic a real-world cockpit or control panels therein. Similarly, VR may be used to train operators of tractors, heavy equipment, military vehicles, security stations, cranes, etc.
Various embodiments include methods, systems and gloves for detecting control-point activation conditions initiated by a wearer of a glove for virtual reality applications. The glove may include a glove body that may be configured to receive a hand of the wearer therein. The glove body may include at least one finger cavity for receiving a finger of the hand, wherein the at least one finger cavity extends away from a palm region toward a fingertip region of the glove body. The glove may also include a conductive path attached to the glove body and extending from a proximal end of the at least one finger cavity, closest to the palm region, toward a distal end of the at least one finger cavity, closest to the fingertip region. In addition, the glove may include an expanded conductive area conductively coupled to the conductive path and attached to the glove body. The expanded conductive area may be wider than individual portions of the conductive path. Further, the glove may include a processor coupled to the conductive path, wherein the processor is configured to process received inputs from the expanded conductive area via the conductive path representing a control-point activation condition. The control-point activation condition may correspond to a portion of the glove body coming in contact with or being within a threshold distance from at least one of another portion of the glove body or a control surface separate from the glove body.
In various embodiments, the conductive path comprises at least one of a coiled wire and/or a conductive printed ink. In addition, a conductive thread may attach the conductive path to the glove body. The conductive area may include at least one of a capacitive sensor or an inductive sensor. A conductive thread may attach the conductive area to the conductive path. The glove body may be formed from a unitary thin elastic polymer material.
In various embodiments, the glove may include a radio frequency identification (RFID) reader may be coupled to the conductive path, wherein the RFID reader is configured to detect an RFID tag on a control surface separate from the glove body. The glove may also include a transceiver coupled to the processor and configured to transmit a message corresponding to the received input. In addition, the glove may include at least one camera mounted on the glove body and configured to obtain images of at least one finger portion of the glove body, wherein the processor is configured to process the obtained images received from the at least one camera. One or more portions of the glove body may include an imprinted code configured to be detected by the at least one camera. The glove may include at least one camera mounted on the glove body and configured to obtain images of control surface remote from the glove. A processor of the glove may be configured to process the obtained images received from the at least one camera.
Various embodiments may include methods performed by a processor (e.g., a processor within a glove or in a computing device coupled to the glove) for control-point activation detection for generating control signals. The method may include receiving a control-point input suitable for tracking location, angle, and/or relative position of one or more portions of a glove. The method may determine whether a control-point activation condition is detected and generate control signals associated with the detected control-point activation condition in response to determining the control-point activation condition is detected.
In some embodiments, the control-point input may be received from an expanded conductive area via a conductive path coupled thereto, which is attached to a glove body, wherein the expanded conductive area is wider than individual portions of the conductive path. The control-point input may be received from a camera mounted on a glove body. The control-point input may include an image of a marking visible on at least one of the glove body or a control panel surface remote from the glove. The control-point input may be received from contact between a portion of a glove and a control panel surface remote from the glove. The control-point activation condition may be received in response to a portion of a glove hovering close to at least one of another portion of the glove or a control panel surface remote from the glove.
In some embodiments, the method may include determining whether at least one of a location, an angle, or a relative position of the one or more portions of the glove has changed. In addition, a virtual image of the glove may be generated based on the changed at least one of the location, the angle, or the relative position of the one or more portions of the glove in response to determining at least one of the location, the angle, or the relative position of the one or more portions of the glove has changed. The method may further include adjusting the received control-point input with calibration parameters. The calibration parameters may be determined using at least two control-point inputs previously received for simultaneously tracking and comparing the same location, angle, or relative position of the same one or more portions of the glove.
In some embodiments, the method may include receiving an image of a glove with an object or markings of predetermined size positioned in a glove size designated location on the glove. A glove size may be determined by comparing a first imaged size of the object or markings in the image to a second imaged size of the glove in the image, using the predetermined size and designated location of the object or markings. Also, the glove size may be output based on the comparison of first and second imaged sizes.
Various embodiments may include methods performed by a processor (e.g., a processor within a glove or in a computing device coupled to the glove) for sizing a virtual reality hand. The method may include receiving an image of a glove with an object or markings of predetermined size positioned in a designated location on the glove. In addition, a glove size may be determined by comparing a first imaged size of the object or markings in the image to a second imaged size of the glove in the image, using the predetermined size and designated location of the object or markings. Also, the glove size may be output based on the comparison of first and second imaged sizes.
Various embodiments may include methods performed by a processor (e.g., a processor within a glove or in a computing device coupled to the glove) for sizing a virtual reality hand that includes receiving an image of a hand or glove and determining a virtual reality hand size or orientation by comparing the received image of the hand or glove to a previously saved image of the hand or glove. In addition, the virtual reality hand size or orientation may be output based on the comparison of the received image and the previously saved image. The received image may include indications of a visible object or markings of predetermined size positioned in a designated location on the glove. Determining the virtual reality hand size or orientation may include comparing a first imaged size and orientation of the object or markings in the received image to the previously saved image of the glove.
Various embodiments may include methods performed by a processor (e.g., a processor within a glove or in a computing device coupled to the glove) for control-point activation detection for generating control signals that includes receiving control-point input information indicating a location or region on the glove in which a first control-point activation condition occurred. Historical control-point information may be updated with the received control-point input. An order of priority may be determined for cycling through a check of locations or regions on the glove to determine whether a second control-point activation condition occurred based on the updated historical control-point information. The locations or regions on the glove may be checked, in the determined order of priority, to determine whether the second control-point activation condition occurred.
Further aspects include a glove including a processor configured with processor-executable instructions to perform operations of any of the methods summarized above. Further aspects include a non-transitory processor-readable storage medium having stored thereon processor-executable software instructions configured to cause a processor of a glove to perform operations of any of the methods summarized above. Further aspects include a processing device for use in a glove and configured to perform operations of any of the methods summarized above. Further embodiments include a computing device configured to be coupled to a glove of various embodiments and having a processor configured with processor-executable instructions to perform operations of any of the methods summarized above.
The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate exemplary embodiments of the invention, and together with the general description given above and the detailed description given below, serve to explain the features of the invention.
The various embodiments will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made to particular examples and implementations are for illustrative purposes, and are not intended to limit the scope of the invention or the claims.
Various embodiments include a glove that may be used as a human interface or input device, enabling the identification of various hand movements, gestures, finger position inputs, and/or contact with or proximity to switches or control surfaces by a wearer of a glove. Various embodiments include a glove configured to be worn by a wearer and detect a position, orientation, and/or proximity of one or more portions of the glove relative either to other portions of the glove or to particular surfaces apart from the glove. Unlike conventional input devices that map signals to an individual symbol or command, key or combination of keys, various embodiments include elements and/or techniques for identifying portions of the glove, referred to as control-points, that have come in contact with or in close proximity to either another control-point of the glove or a control surface separate from the glove.
Various embodiments include a conductive path extending across one or more portions of the glove. The conductive path may be configured to convey received inputs from control-point activation conditions, which correspond to when one or more portions of the glove come in contact with or in close proximity to either another portion of the glove or a control surface separate from the glove. A processor may analyze the received inputs conveyed along the conductive path, and/or transmit those inputs, as a message corresponding to the received inputs, to a remote processor wirelessly.
In various embodiments, the glove may include expanded conductive areas at predetermined locations along the conductive path. The expanded conductive areas may be configured to expand the detection of control-point activation conditions beyond the narrow width of the conductive path and responsive to certain finger touches. For example, expanded conductive areas may be located within a palm of the glove, at finger tips, and/or at a thumb tip. The expanded conductive areas may be configured to complete a circuit that indicates that control-point activation conditions have been met. As a more specific example, when an expanded conductive area of the glove comes in contact with or comes within close proximity of a control surface, resistance through that path is reduced and current through that path is increased indicating whether a switch associated with that control surface has been engaged or approached. Similarly, when one expanded conductive area, for example on a side or tip of a finger, comes in contact with or close proximity to another expanded conductive area, for example on a thumb, the change in resistance or capacitance through the resulting conductive path indicates a particular gesture or command has been made by a wearer of the glove. Thus, by providing the expanded conductive areas of the glove, the wearer is able to input commands, symbols or letters that correspond to certain finger touches together or to the palm.
In some embodiments, the detected change in resistance or capacitance may be conveyed to a processor located within the glove. The processor may process the signal(s) to generate an indication of whether an associated control-point activation condition has been met (e.g., a first gesture or switch activation has occurred). The processor may alternatively or additionally pass along the generated indication, via further portions of the conductive path, to a transceiver located within the glove. The transceiver may be configured to transmit the generated indication of the associated control-point activation condition, as a message corresponding to the associated control-point activation condition.
Traditional control panel training applications that use VR have difficulty presenting accurate hand presence to the trainee. The precise position of the trainee's hand or hands in the VR space must match a real-world position of the hand or hands relative to the physical control panel on which they are training. Various embodiments are particularly suited to improve VR-based control panel training applications. For example, in accordance with various embodiment the control panel itself may be equipped with switches (i.e., control surfaces) that include metallic elements or surfaces. Thus, a glove in accordance with various embodiments, touching one of those control surfaces may act like a multimeter and measure resistance. Accordingly, a measurement of resistance by the glove may provide an indication of proximity to or contact with (also referred to herein as a “control-point activation condition”) a control surface. Different icons, buttons or portions of the control surface may be configured with different resistance values, and thus the processor may determine the icon, button or panel location that was touched by the resistance through the circuit through the glove to the touched surface.
A further aspect of various embodiments includes configuring or measuring the resistance at different locations on control surfaces (e.g., buttons, switches, icons, etc.) so that each has a known resistance that can be used by the processor to recognize when a particular item is touched. The glove and processor system may be configured to detect differences in resistance of as little as 5-10 ohms. In addition, adding a resistor to a switch may change its resistance, so each control surface may be modified (i.e., by adding one or more resistors) to exhibit a unique or at least different resistance. In this way, proximity and/or activation of a particular control surface (e.g., buttons, switches, icons, etc.) may be recognized based on comparing measured resistance to the known resistance of different items on the control surface. For example, a 50-ohm reading may correspond to the control surface (i.e., switch) designated for putting an aircraft's flaps up, while a 100-ohm reading may correspond to the control surface designated for deploying landing gear.
A further aspect of various embodiments may use a grid of RFID tags embedded in or mounted on the surface of a control panel with control surfaces. Alternatively, one RFID tag may be located on each control surface, providing a unique identifier for each control surface. The RFID tags may then be read by an RFID reader included in/on or associated with the glove. The RFID reader antenna may be located in or on a suitable portion of the glove, such as embedding an antenna loop in the pointer finger or thumb portion of the glove. Alternatively, or additionally, the expanded conductive areas and conductive paths may act like an antenna between the RFID reader and the RFID tags. In addition, RFID signals may trigger on proximity and do not require contact between surfaces.
A further aspect of various embodiments relates to detecting bending of one or more fingers the glove to provide inputs useful for VR systems. In addition, the tracking and/or detection of finger movements may be used to detect certain gestures being made by a user wearing the glove. Thus, various embodiments include a glove with one or more cameras mounted on the glove and configured to obtain images of at least one finger portion of the glove body, in order to detect finger movements and bending. In addition, portions of the glove, such as the finger tips, may include a marking, such as imprinted codes, bar codes, quick response (QR) codes, or other unique symbols, which may be readily identifiable by the one or more cameras. In addition, one or more control surfaces remote from the glove may include imprinted codes, such as bar codes, quick response (QR) codes, or other unique symbols, which may be used to identify or confirm the identity of the control surface (i.e., to identify or confirm proximity or engagement with a particular switch, button, or other control).
A further aspect of various embodiments relates to scaling a wearer's hands when generating a representation of those hands in a VR environment (i.e., virtual hands). Current systems tend to take a one size fits all approach. Various embodiments perform dynamic virtual hand scaling in order to present wearers of the gloves with a more realistic representation of their actual hand size.
A further aspect of various embodiments relates to orienting a wearer's hands when generating virtual hands in a VR environment. Current systems tend to make orientation determinations based on an offset from other known system elements, like a paddle controller that may be held by the wearer. Unfortunately, this may lead to an inaccurate representation of virtual hand orientations. Various embodiments perform dynamic virtual hand orienting in order to present wearers of the gloves with a more accurate representation of the orientation of their hands.
A further aspect of various embodiments relates to improving the processing of signals received from the expanded conductive areas through the conductive paths of the glove. In order to determine whether there has been a change in resistance or capacitance at any particular location or through any particular conductive path on the glove, processors may cycle through every location on the glove, checking for a change in resistance, capacitance or other electric characteristic. Various embodiments perform smart location cycling, which takes into account the most frequent touch locations and/or most recently active locations on the glove, in order to speed up the identification of where control-point activation conditions may be occurring.
A further aspect of various embodiments may include sensors connected to one or more of the expanded conductive areas and/or one or more of the conductive paths. In this way, sensors placed in special locations on a glove may be used to detect or confirm certain hand gestures or hand movements have been made.
As used herein, the expressions a “control-point input” refers to one or more signals received from an input element of a glove. The signals may be received when certain conditions occur at a location on particular portion of a glove. For example, a conductive element at a first location on the glove may make contact with a conductive element at a second location on the glove, which forms a closed-circuit loop and generates an identifiable input associated with the two locations and is considered a control-point input. Similarly, a sensor at a particular location on the glove may trigger under certain circumstances and the triggering of that sensor, which is an identifiable input associated with a particular location on the glove is considered a control-point input. Also, camera images of a particular location on the glove in combination with image analysis may be configured to identify certain conditions associated with that location and may be considered a control-point input.
As used herein, the expression “control-point activation condition” refers to a predefined condition that occurs when one or more recognized control-point inputs are received that correspond to predetermined gestures, motions, or control surface interaction. The control surface interactions may include both direct contact or close-contact (i.e., hovering) between the glove and another portion of the glove or a control panel surface remote from the glove.
As used herein, the terms “component,” “system,” and the like include a computer-related entity, such as, but not limited to, hardware, firmware, a combination of hardware and software, software, or software in execution, which are configured to perform particular operations or functions. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a communication device and the communication device may be referred to as a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one processor or core and/or distributed between two or more processors or cores. In addition, these components may execute from various non-transitory computer readable media having various instructions and/or data structures stored thereon. Components may communicate by way of local and/or remote processes, function or procedure calls, electronic signals, data packets, memory read/writes, and other known computer, processor, and/or process related communication methodologies.
As used herein, the term “proximal” refers to elements situated nearer to the center of a body, such as the body of a wearer, or nearer to the point of attachment of a portion of the body. In contrast, as used herein, the term “distal” refer to elements situated away from the center of the body or from the point of attachment; meaning the opposite of proximal.
Various embodiments provide solutions for tracking hand movement in VR applications. In particular, various embodiments may determine when a user touches a switch or control surface and which switch or control surface they are touching, approaching, or hovering over. Various aspects of the glove and related methods disclosed herein may eliminate the need to use mock instrument panels that are wired and functional to train with a VR application.
The glove body 110 may be a covering for all or part of a hand of the wearer and thus configured to receive the hand or parts of the hand therein. The glove body 110 is formed from the main sections of material configured to cover parts of the hand Thus, the glove body 110 may include at least one finger cavity for receiving a finger of the hand. In various embodiments, the glove body 110 may include more than one finger cavity, such as a cavity for one or more other fingers or a separate cavity for each finger. As with a conventional glove, the at least one finger cavity may extend away from a palm region toward a fingertip region of the glove body 110. The glove body may be formed from any of a variety of materials and may be a combination of materials. For example, the glove body 110 may comprise a fabric blend of polyester and spandex.
The conductive path 120 may be attached to the glove body 110 and extend across the wrist, palm, and/or finger regions of the glove body 110. The conductive path 120 may extend between the processor 150 and a portion of the glove body 110 configured to receive inputs. For example, the conductive path 120 may extend from the processor 150 located at a proximal end of the glove body 110, such as the wrist region, to a fingertip region located at a distal end of at least one of the finger cavities. The conductive path 120 may form a closed-circuit loop for each extent between the proximal and distal ends. In various embodiments, the conductive path 120 may be formed from a flexible conductive material configured to bend and/or move, along with corresponding portions of the glove body 110 following hand movements of the wearer. For example, the flexible conductive material may be formed from wire, a conductive trace, conductive ink, or other conductive element. For example, the conductive path 120 may be formed as a linear resistor, such as from a forty ohm (40Ω) coiled wire, which may be soldered or otherwise bonded to a circuit board. The spiral pattern of a coiled wire, which may be 1 millimeter in width, may provide more flexibility and a wider coverage than a straight piece of wire.
In various embodiments, the glove body 110 may additionally include one or more expanded conductive areas 130a, 130b, 130c, 130d, 130e (i.e., 130a-130e), 132a, 132b, 132c, 132d, 132e, 132f, 132g (i.e., 132a-132g), 134a, 134b, 134c, 134d (i.e., 134a-134d), strategically located on the glove body in locations such as the finger tips, sides of fingers, palm, and the like. The expanded conductive areas 130a-130e, 132a-132g, and 134a-134d may provide localized surface regions in which distinct resistive changes may be detected when one or more portions of the glove 200 are touching switches, special surfaces, or other conductive areas. In this way, when thumb tip and fingertip are touching, a resistance may lower by approximately 30 ohms, indicating a switch is being touched. In addition, if the wearer slides a glove-finger along one of the expanded conductive areas 130a-130e, 132a-132g, and 134a-134d, maintaining engagement therewith, the continuous signal with changing resistance may represent control-point activation conditions associated with a slider or potentiometer operations, such as for scrolling, zooming in/out, or other functions.
In contrast to the conductive path 120, individual segments of which may be relatively narrow, the conductive areas may be formed to cover broader areas, significantly wider than the conductive path 120. Expanding the sensing capability of the conductive path, the conductive areas may connect directly or indirectly to the conductive path 120. In various embodiments, each of the expanded conductive area may be formed by a pad, patch or plate made of conductive material. For example, a first set of conductive pads may be the expanded conductive areas 130a-130e, located at or near a tip of each finger portion of the glove body 110, including the thumb. A second set of conductive pads may be the finger-side conductive pads 132a-132g, located at more central and base regions of the finger portions of the glove body 110. A third set of conductive pads may be the palm conductive pads 134a-134d, located at or near the palm portions of the glove body 110. The number and precise position of the individual conductive pads or sets of conductive pads (e.g., 130a-130e, 132a-132g, and 134a-134d) is illustrated for ease of explanation. However, more or fewer of these conductive pads may be located at any portion of the glove body 110. For example, additional conductive pads may be included on the back-side of the glove, in the orientation shown in
Additionally, the conductive pads may be enhanced for providing proximity detection, in addition to contact detection. All or a portion of the conductive pads may be replaced with or supplemented by RFID tags embedded or attached in the fabric of the glove 100. RFID reader antennas may be located in various areas of the glove 100, such as the thumb or palm thereof. For example, the RFID reader(s) may be incorporated into the conductive pads (e.g., 130a-130e, 132a-132g, and 134a-134d). In addition, or alternatively, the RFID reader antenna may be incorporated into the conductive pads and/or the conductive path(s) (e.g., 120). By adjusting the signal strength of the RFID reader antenna, the proximity at which the RFID tags are activated may be adjusted. Thus, by using an RFID tag and reader system various proximity detections can be made. For example, the proximity may be finely tuned so that the RFID tag needs to be touched by the antenna to activate, or the proximity may be tuned so that the RFID tag only needs to come within an inch (1″) of the RFID reader antenna. Alternatively, other distances or proximity methods could be used, such as measuring time between ultrasonic pulses originating from a location on the glove with receivers on the finger tips.
In some embodiments, the glove 100 may include a processor 150 which is configured to receive a generated signal and process the generated signal to identify movements, gestures, or even proximity to switches or control surfaces by a wearer with the glove 100. Although the processor 150 is shown in
The processor 150 may be configured to execute machine learning processes to train the system to recognize hand position and switch proximity in control panel training applications. The processor 150 may include a microcontroller, configurable analog and digital circuits, firmware, software, conversion formulae, lookup tables, and/or other control and measurement elements. The processor 150 may be configured to deliver an indication that a control-point activation condition has occurred. In addition, the processor 150 may transmit the identified gesture to a transceiver 130. In these embodiments, the transceiver 130 may be configured to transmit the control-point input(s) of the control-point activation condition (e.g., an identified gesture), as a message associated with control-point activation condition.
The processor 150 may include one or more cores, and each processor/core may perform operations independent of the other processors/cores. One or more of the processors may be configured with processor-executable instructions to perform operations of methods of various embodiments (e.g., methods 1400, 1500, 1600, 1700, and 1800 described herein with reference to
In some embodiments, the glove 100 may include one or more components for enabling one-way or two-way wireless communications. For example, the glove 100 may have a transceiver 160 (e.g., Bluetooth®, Zigbee®, Wi-Fi, HF, VHF, RF radio, cellular, etc.), including an antenna, for sending and receiving wireless transmissions 165, from and/or to the processor 150. The transceiver 160 may be used with the above-mentioned circuitry to implement various wireless transmission protocol stacks and interfaces.
In some embodiments, the expanded conductive areas 230a-230e, 232a-2321 may be formed from higher resistance material than the conductive path 120. This combination of higher resistance and lower resistance may encourage the electrons to flow more directly from the expanded conductive areas 230a-230e, 232a-2321 to the conductive path 120, following the quickest path to least resistance. In this way, higher electrical resistance areas feeding electrons into lower resistance “highways,” may provide a means for location detection of a contact point on the glove.
More or fewer expanded conductive areas may be provided on the glove body 110 than illustrated in the figures. Also, each of the expanded conductive areas may be larger or smaller than illustrated in the figures. In addition, as an alternative, one or more expanded conductive areas may replace one or all of the palm conductive pads 134a-134d.
In some embodiments, substantially all of the glove body 110 may be made of conductive fabric that has a higher resistance than that of the conductive path 120. For example, the conductive path may have a resistance of 40 ohms, while the fabric of the glove body 110 may have a resistance 60-80 ohms. In this way, touching almost anywhere on the glove body 110 may be differentiated by a few ohms difference that may be detected in response to that touch. The current will follow the path of least resistance down the conductive path 120, and thus may not travel all the way down the conductive fabric of the glove body 110, with its higher resistance. In this way, the point along one particular conductive path 120, nearest to a point of contact on the conductive fabric of the glove body 110, may be considered an estimate of the control-point providing an activation condition. By using a measurement of the resistance, the processor 150 may not be able to determine precisely where the point of contact was made on the conductive fabric, but may use the estimated control-point as an indication of the general region of the glove body 110 from which the signal originated. One advantage to having providing substantially all of the glove body 110 may be made out of conductive fabric that has a higher resistance than that of the conductive path 120 is that it may eliminate dead zones on the glove body 110 in which contact does not register as an activation condition. In addition, if used in combination with imaging systems configured to take and analyze images of the glove or portions thereof (e.g., cameras on the glove or remote from the glove), the estimated control-point may be used as a confirmation or calibration between two different tracking systems. Further, the timing of the detected activation condition from the estimated control-point, the estimated control-point location, and any other activation condition detection input (e.g., from another tracking system, such as an imaging system) may be used for machine learning to train machine learning models to more accurately correlate control-point activation conditions with a precise contact timing and/or glove location, orientation, and/or motions.
In addition, if the wearer slides a glove-finger along a portion of the higher resistance conductive fabric, but maintaining engagement therewith, the continuous signal with changing resistance may represent control-point activation conditions associated with a slider or potentiometer operations, such as for scrolling, zooming in/out, or other functions.
In some embodiments in which substantially all of the glove body 110 is made out of conductive fabric, providing dual resistive/capacitive conductive pads (e.g., in the expanded conductive areas 130a-130e, 132a-132g, and 134a-134d), may enable control-point inputs to be generated from almost anywhere on the entire glove 100. For example, with reference to
With reference to
The conductive path 120 may include portions of conductive ink and other portions of other conductive material, such as coiled wire. For example, coiled wire may secure directly to the processor, providing fly leads that may be secured to the conductive ink. The coiled wire may be sewn, bonded (e.g., by soldering or with conductive paste), or otherwise secured to the conductive ink of the conductive path 620 at one or more appropriate attachment locations 625. Alternatively, the conductive ink may extend from the processor (e.g., 150), to a distal-most expanded conductive area 330a, and loop back to the processor, surrounding any expanded conductive areas 332a, 332b, 332c, 332d along the way.
The conductive path 620 may widen in locations in which the expanded conductive areas 330a, 332a, 332b, 332c, 332d are disposed. In this way, each of the expanded conductive areas 330a, 332a, 332b, 332c, 332d may be surrounded by conductive material, namely the conductive ink.
Various embodiments may include a conductive wire, buried under the conductive ink as a redundancy or “bridge” in the event of cracks or breaks in the printed ink. In this way, the wire and/or the conductive ink may serve as a primary conductor and the other serves as a secondary conductor.
Various embodiments may include an encapsulant over the conductive ink. The encapsulant may be a non-conductive layer of material, such as polyurethane, that covers the conductive ink to reinforce and/or provide extra strength thereto. In addition, the encapsulant may cover areas of the ink not intended to be exposed. The encapsulant may protect the integrity of the conductive ink and be provided as a separate graphics layer.
The control strap 750 may be a separate removable element configured to be mounted onto a wrist portion of the glove 700. In this way, the glove 700 may be disposable and replaced with a fresh glove 700, while the control strap 750 may be transferred to the fresh glove 700 and thus reused. For example, the control strap 750 may include conductive contact strips 752 configured to align with and engage individual branches of the conductive path 720 at the wrist region. When strapped firmly around the wearer's wrist, the control strap 750 may force the conductive contact strips 752 into engagement with the wrist portions of the conductive path 720. The control strap 750 may additionally include a transceiver and circuitry for wireless communications.
Similar to the glove 700, the conductive sticker set 800 may be configured to work in conjunction with the control strap 750 that may house a processor (e.g., 150) and optionally a transceiver (e.g., 160). The control strap 750 may include conductive contact strips 752 configured to align with and engage individual branches of the conductive path 820 at the wrist region. When strapped firmly around the wearer's wrist, the control strap 750 may force the conductive contact strips 752 into engagement with the wrist portions of the conductive path 820.
In various embodiments, portions of the glove, such as the finger tips/sides or back/palm, may optionally include imprinted markers A, B, C, D, E, such as bar codes, QR codes, or other unique symbols, which may be readily identifiable from images taken by the one or more wrist-mounted cameras 970. A greater or fewer number of imprinted codes may be included on the glove 900.
The glove 900 may otherwise have aspects of one or more of the gloves 100, 200, 300, 305, 600, 700 described above with respect to other embodiments. In this way, the wrist-mounted cameras 970 may not only provide a visual indication of glove finger position and orientation, but also have a calibration input for the image tracking from the conductive surfaces/paths used to detect control-point activation conditions, described above with regard to the conductive paths and expanded conductive areas. Alternatively, the conductive surfaces/paths may provide the main control-point activation detection, with the visual indications from the wrist-mounted cameras 970 providing the calibration input. This calibration between two different tracking systems may provide an especially powerful and accurate technique for tracking finger movements, particularly of the thumb. Additionally, thumb touch locations on fingers can be tuned and tracked using kinematic models.
As an alternative the gloves 900, 1100 may be made as fingerless gloves, since the cameras (e.g., 970, 1170) may be used to track finger-tip movements and thus detect control-point activation conditions.
When a conductive area of the glove, such as a conductive pad, is brought into contact with or gets within close proximity to one another, a signal may be generated indicative of a gesture being formed by the user wearing the glove. For example, when the glove is formed into a first gesture by the user's hand, certain conductive pads located in finger tips, along the thumb, and within the palm of the glove may be brought into contact or within close proximity Circuit(s) established based on this contact or close proximity may generate a signal indicative of the first gesture. In some embodiments, such generated signal may be carried via circuitry within the glove.
In various embodiments, the glove (e.g., 100, 200, 300, 305, 600, 700, 900, 1100) and other input devices (e.g., control panel surfaces) may be used to detect control-point activation conditions associated with glove touches that generate control-point inputs. For example, when two conductive portions of the glove touch or when a conductive portion of the glove touches a conductive control panel surface separate from the glove control-point inputs may be generated, received by a processor (e.g., 150) and analyzed to determine whether a control-point activation condition exists. A processor in the glove may control its own control-point activation condition detection system. In addition, the control panel, which is remote from the glove, may optionally include its own control-point activation condition detection systems. In yet further embodiments, the glove and/or the control panel may transmit received control-point inputs, as a message associated with the control-point activation condition, to a remote computing device for processing and running control-point activation condition detection systems.
In block 1410, the processor may receive a control-point input suitable for tracking location, angle, and/or relative position of one or more portions of a glove. The glove position information may come from a control-point input, such as from conductive pads or other conductive materials on the glove or a control panel, sensors, imaging data, or other inputs associated with tracking location, angle, and/or relative position of one or more portions of the glove.
The control-point input may be received in response to a wearer of the glove causing one or more control-point activation conditions to occur, such as by forming a gesture or interacting with a control surface remote from the glove. For example when forming a gesture, a wearer may bring two or more portions of the glove, such as the expanded conductive areas (e.g., 130a-130e, 132a-132g, and 134a-134d), conductive path (e.g., 120) or other conductive areas of the glove into contact or close proximity with one another, which may generate a control-point input signal. In this way, forming gestures such as a fist, an “OK” sign, a “V” for victory sign, pointing using an index finger, a thumbs up/down gesture, or any of various other static or dynamic gestures may cause control-point inputs to be generated. Similarly, just moving the glove and orienting the glove in a particular way may be detected by inertial or other sensors, which may generate a control-point input signal.
Alternatively, the control-point input information may be received in response to a wearer of the glove moving and/or positioning the glove, but not actually causing a control-point activation condition. This may occur when the wearer moves the glove between gestures, does not intend to form a recognized gesture, or otherwise moves or arranges the glove without causing a control-point activation condition.
Circuitry connected to the expanded conductive areas may convey an electron flow (i.e., a signal) from points of contact in the expanded conductive areas to the processor. In some embodiments, the signal may be indicative of the gesture being formed. In some embodiments, the signal may be an indication of which two or more expanded conductive areas were brought into contact or close proximity. In some embodiments, the signal may provide an indication of a location of the contact or close proximity. In some embodiments, the signal may include multiple signals. For example, when a formed gesture brings multiple pairs or combinations of expanded conductive areas into contact or close proximity, each pair or combination of areas may convey a separate signal and these signals may be combined or otherwise collected into the signal received by the processor.
In some embodiments, one or more sensors may be used to generate signals suitable for tracking location, angle, and/or relative position of one or more portions of a glove. For example, using camera imaging or gyroscopic sensor data, a thumbs up or thumbs down gesture may be detected, which both include a similar gesture but with different hand orientations. The generated signal may not only include an indication of hand orientation, but also include hand motion and/or acceleration for identifying dynamic gestures. For example, an open palm being brought down into a first may represent an instruction to stop. In this way, in addition to static hand gestures, a multitude of dynamic gestures may be used/detected. The signals received from sensors may also be received as glove position information suitable for tracking location, angle, and/or relative position thereof.
In determination block 1415, the processor may determine whether a control-point activation condition is detected. In particular, the processor may determine whether the received control-point input matches an input associated with a control-point activation condition. There may be many control-point activation conditions, each associated with functions or commands used in a VR application and each one may be associated with a particular one or more control-point inputs. If the received control-point input matches those of one of the control-point activation conditions, then the corresponding control-point activation condition has been detected. Otherwise, if the received control-point input does not match any inputs associated with a control-point activation condition, then no control-point activation condition has been detected. Thus, in response to a control-point activation condition being detected (i.e., determination block 1415=“Yes”), the processor may generate control signals associated with the detected control-point activation condition in block 1420.
In determination block 1425, following the generation of control signals in block 1420 or in response to the determination that control-point activation condition is detected in determination block 1415, the processor may determine whether a location, angle, and/or relative position of the one or more portions of the glove have changed.
In response to determining that the location, angle, and/or relative position of the one or more portions of the glove have changed (i.e., determination block 1425=“Yes”), the processor may generate a virtual image based on the changed location, angle, and/or relative position. The processor may treat circumstances in which no previous virtual image has been generated because the virtual imaging process has just begun as the location, angle, and relative position having changed. In response to determining that none of the location, angle, or relative position of the one or more portions of the glove have changed (i.e., determination block 1425=“No”) or following the generation of the virtual image in block 1430, the processor may await receipt of a further control-point input so the process of the method 1400 may start again in block 1410.
In various embodiments, the glove (e.g., 100, 200, 300, 305, 600, 700, 900, 1000 and related control-point activation detection systems may be used to train a processor (e.g., 150) to not only detect touch-type control-point activation conditions but also non-contact control-point activation conditions, such as hovering or other non-contact gestures. Using machine learning the processor may be “trained” to more accurately determine glove position, orientation, and movements or better recognize control-point activation conditions. The processor may be trained to detect a precise moment of contact with a control panel remote from the glove, such as in a mock-up cockpit. Not only may the processor learn how to deal with unique instrument panels, but also different users (i.e., glove wearers) may orient their hands differently depending on the proximity of the control panel to that user. For example, a user may orient their hand one way when interacting with a control panel switch that is located far away from the user and orient that hand differently when interacting with a different control panel switch or one that is much closer.
In some embodiments following the operations of block 1410 of the method 1400 (
The detection of a touch-type contact or proximity sensor alert, which generate control-point input signals, may be used to calibrate image sensing of control-point activation conditions. Thus, when a conductive finger tip of a glove touches another conductive surface of the glove, the processor may be provided with a precise indication of where and when such contact was made. Accordingly, an image recognition system may then use images of the same event to determine calibration parameters that may be used to fine-tune/calibrate the image recognition system. The image recognition system may use machine learning/modeling to get a better idea of where one or more glove fingers actually are and how images look when those locations make contact with other predetermined surfaces on the glove or elsewhere. In addition, such calibration parameters may be used to subsequently identify the same finger or other portion of the glove merely hovering (i.e., not making contact), which may be associated with a different set of functions, operations, or a different virtual image of the relative position of the user's hand in a VR environment.
In determination block 1515, the processor may determine whether another control-point input has been received for simultaneously tracking location, angle, and/or relative position of the same one or more portions of the glove. Like the two control-point inputs used to determine the calibration parameters, another control-point input received following or coincident with the control-point input received in block 1410 may be used for updating or generating new calibration parameters. In response to the processor determining that another control-point input of this type has been received (i.e., determination block 1515=“Yes”), the processor may compare the two control-point inputs to determine whether calibration parameters need to be saved or updated in block 1520.
In block 1530, the processor may save the calibration parameters or updated calibration parameters in response to determining the calibration parameters need to be saved or updated.
In response to the processor determining that no other control-point input of this type has been received (i.e., determination block 1515=“No”) or following the operations of block 1530, the processor may perform the operations of determination block 1415 of the method 1400 (
A further aspect of various embodiments relates to virtual hand scaling. Various embodiments perform dynamic virtual hand scaling in order to present wearers of the gloves with a more realistic representation of their actual hand size. Glove size may not be an accurate representation of the size of the wearer's hands since the gloves may be made of elastic material that may shrink or stretch as much as 40%. In one aspect of dynamic virtual hand scaling, a processor may analyze images of the glove worn by the user, such as images received from wrist-mounted cameras (e.g., 970) or even images received from headset-mounted cameras, as shown in
In block 1610, the processor may receive an image of a glove with an object or markings of predetermined size positioned in a designated location on the glove. For example, an image of a logo or other graphic of known size and that may be found in the same location on numerous different gloves may be the object or markings used to perform dynamic virtual hand scaling. Similarly, a power pack, control module, or other object mounted in a designated location on the glove may also be used in this way.
In block 1620, the processor may determine a glove size by comparing a first imaged size of the object or markings in the image to a second imaged size of the glove in the image, using the predetermined size and designated location of the object or markings.
In block 1630, the processor may output the glove size based on the comparison of first and second imaged sizes.
A further aspect of various embodiments relates to virtual hand orienting. Various embodiments perform dynamic virtual hand orienting in order to present wearers of the gloves with a more accurate representation of virtual hand orientations. In one aspect of dynamic virtual hand orienting, a processor may analyze images of the glove worn by the user, such as images received from wrist-mounted cameras (e.g., 970) or even images received from headset-mounted cameras, as shown in
In block 1710, the processor may receive an image of a hand or glove with an object or markings of predetermined size positioned in a designated location on the glove. For example, an image of a logo or other graphic of known size and that may be found in the same location on numerous different gloves may be the object or markings used to perform dynamic virtual hand scaling. Similarly, a power pack, control module, or other object mounted in a designated location on the glove may also be used in this way.
In some embodiments, the received image of the hand or glove may include indications of a visible object or markings of predetermined size positioned in a designated location on the glove. Using the indications of the visible object or markings, the virtual reality hand size or orientation may be determined by comparing a first imaged size and orientation of the object or markings in the received image to the previously saved image of the glove.
In block 1720, the processor may determine a virtual reality hand size or orientation by comparing the received image of the hand or glove to a previously saved image of the hand or glove.
In block 1730, the processor may output the virtual reality hand size or orientation based on the comparison of the received image and the previously saved image.
A further aspect of various embodiments relates to control-point activation conditions detection. Various embodiments perform smart location cycling, which takes into account the most frequent locations and/or most recently active locations on the glove, in order to speed up the identification of where control-point activation conditions may be occurring.
Various embodiments may include a method of prioritizing a search for locations or regions in which control-point activation conditions may exist. The prioritized search may determine an order and/or frequency with which to scan locations or regions using a search algorithm, known as a Baier algorithm or method. Using the Baier methods, the prioritized search may scan more frequently and/or more recently used locations or regions of a glove first and/or more often for control-point activation conditions, which may increase the speed in which control-point activation conditions may be detected. Some embodiments may modify the order of the prioritized search to account for common circumstances, customized settings/environments, or other situations if needed. In fact, the prioritized search may scan certain areas multiple times before scanning other areas.
For example, with reference to
In block 1810, the processor may receive control-point input information indicating a location or region on the glove in which a first control-point activation condition occurred. The location or region associated with the received control-point input information may be correlated to one of the expanded conductive areas (e.g., 130a-130e, 132a-132g, and 134a-134d) or a point along a conductive path (e.g., 120) on the glove.
In block 1820, the processor may update historical control-point information with the received control-point input information.
In block 1830, the processor may determine an order of priority for cycling through a check of locations or regions on the glove to determine whether a second control-point activation condition occurred based on the updated historical control-point information. In this way, the most frequently detected locations or regions (i.e., locations or regions on the glove in which control-point activation conditions are detected) may be given priority over less frequently detected or never detected locations or regions.
In block 1840, the processor may check the locations or regions on the glove, in the determined order of priority, to determine whether the second control-point activation condition occurred.
Alternatively, rather than making scanning priority adjustments based on the last control-point activation detection, the processor may use a standard scanning sequence, such as one determined to be the most efficient for the most common circumstances.
Various embodiment gloves may be connected to and various embodiment methods may be implemented in a variety of computing devices, such as a laptop computer 1900 as illustrated in
The processors may be any programmable microprocessor, microcomputer or multiple processor chip or chips that may be configured by software instructions (applications) to perform a variety of functions, including the functions of the various embodiments described in this application. In some mobile devices, multiple processors may be provided, such as one processor dedicated to wireless communication functions and one processor dedicated to running other applications. Typically, software applications may be stored in the internal memory before they are accessed and loaded into the processor. The processor may include internal memory sufficient to store the application software instructions.
Various embodiments illustrated and described are provided merely as examples to illustrate various features of the claims. However, features shown and described with respect to any given embodiment are not necessarily limited to the associated embodiment and may be used or combined with other embodiments that are shown and described. Further, the claims are not intended to be limited by any one example embodiment. For example, one or more of the operations of the methods may be substituted for or combined with one or more operations of the methods.
The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the claims.
The hardware used to implement the various illustrative logics, logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuitry that is specific to a given function.
In one or more embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable medium or non-transitory processor-readable medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a non-transitory computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable storage media may be any storage media that may be accessed by a computer or a processor. By way of example but not limitation, such non-transitory computer-readable or processor-readable media may include RAM, ROM, EEPROM, FLASH memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable medium and/or computer-readable medium, which may be incorporated into a computer program product.
The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the claims. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.
This application claims the benefit of priority to U.S. Provisional Patent Application No. 62/900,808 entitled “Control-Point Activation Condition Detection For Generating Corresponding Control Signals” filed Sep. 16, 2019, the entire contents of which are hereby incorporated by reference for all purposes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4414537 | Grimes | Nov 1983 | A |
| 5906004 | Lebby et al. | May 1999 | A |
| 6080690 | Lebby et al. | Jun 2000 | A |
| 6128004 | McDowall | Oct 2000 | A |
| 6141643 | Harmon | Oct 2000 | A |
| 6210771 | Post et al. | Apr 2001 | B1 |
| 6670894 | Mehrin et al. | Dec 2003 | B2 |
| 6727197 | Wilson et al. | Apr 2004 | B1 |
| 6729025 | Farrell et al. | May 2004 | B2 |
| 6942496 | Sweetland et al. | Sep 2005 | B2 |
| 7498956 | Baier | Mar 2009 | B2 |
| 8704758 | Figley | Apr 2014 | B1 |
| 9606630 | Underkoffler | Mar 2017 | B2 |
| 10561367 | Salada | Feb 2020 | B1 |
| 10593101 | Han | Mar 2020 | B1 |
| 10642364 | Minnen | May 2020 | B2 |
| 10802657 | Ahne | Oct 2020 | B1 |
| 11292236 | Wang | Apr 2022 | B1 |
| 20040036678 | Zngf | Feb 2004 | A1 |
| 20040051694 | Backman et al. | Mar 2004 | A1 |
| 20040210166 | Soh et al. | Oct 2004 | A1 |
| 20040263358 | Madsen et al. | Dec 2004 | A1 |
| 20050052291 | Backman et al. | Mar 2005 | A1 |
| 20050052412 | McRae et al. | Mar 2005 | A1 |
| 20060248478 | Liau | Nov 2006 | A1 |
| 20070291016 | Philipp | Dec 2007 | A1 |
| 20100090966 | Gregorio | Apr 2010 | A1 |
| 20130104285 | Nolan | May 2013 | A1 |
| 20150013044 | Baacke | Jan 2015 | A1 |
| 20150091858 | Rosenberg | Apr 2015 | A1 |
| 20150091859 | Rosenberg | Apr 2015 | A1 |
| 20150258431 | Stafford | Sep 2015 | A1 |
| 20160171907 | Moore | Jun 2016 | A1 |
| 20160180594 | Todeschini | Jun 2016 | A1 |
| 20160259408 | Messingher | Sep 2016 | A1 |
| 20160306431 | Stafford | Oct 2016 | A1 |
| 20170086712 | Mauro | Mar 2017 | A1 |
| 20170196513 | Longinotti-Buitoni | Jul 2017 | A1 |
| 20170319950 | Buchanan, IV | Nov 2017 | A1 |
| 20190101981 | Elias | Apr 2019 | A1 |
| 20190346938 | Wang | Nov 2019 | A1 |
| 20200022433 | Lu | Jan 2020 | A1 |
| 20200257384 | Ahne | Aug 2020 | A1 |
| 20220018689 | Whitehead | Jan 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 9737340 | Oct 1997 | WO |
| 2015048584 | Apr 2015 | WO |
| Entry |
|---|
| The Internatonal Bureau of WIPO, International Preliminary Report on Patentability issued in International Application No. PCT/IB2020/000744, dated Mar. 31, 2022 (10 pages). |
| Post, et al., “Smart Fabric, or Washable Computing”, http://web.media.mit.edu/˜rehmi/fabric, 4 pages, Nov. 5, 2009. |
| International Search Report and the Written Opinion of the International Searching Authority received from the Canadian Intellectual Property Office in related International Application No. PCT/IB2020/000744 dated Jan. 18, 2021. |
| Number | Date | Country | |
|---|---|---|---|
| 20210081042 A1 | Mar 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62900808 | Sep 2019 | US |
