Patent Application
20220058942
The present invention relates to systems and methods for operating a controlled device.
The desire for hands-free operation of controlled devices arises in many contexts. For example, U.S. Pat. No. 7,184,903 describes a hands-free, mouth-activated switch disposed within a cup-shaped, rigid portion of a pilot's oxygen mask. Among the elements controllable by such a switch is a night vision compatible light. U.S. Patent Application Publication 2012/0229248 describes a hands-free controller that monitors facial expressions of a wearer and other body motions and generates commands for a controlled device based on the combination of the facial expressions and other monitored motions.
Embodiments of the invention include systems and methods for operating a controlled device via an activation accessory of a wearable device that includes a movable actuator having a range of travel between a fully extended position and fully compressed position, a sensor, and a communication element. The sensor is coupled to a controller, which has an output coupled to a control signal interface. The controller is programmed to receive and evaluate input signals from the sensor that are responsive to movements of the movable actuator to determine whether or not they represent a command for the controlled device by assessing the input signals for a signal pattern indicative of a plurality of volitional actions (e.g., jaw clenches) of a wearer of the wearable device. If/when the processor determines that the input signals represent the command, then it decodes the command and transmits an associated control signal to the controlled device via the control signal interface.
In one example, the activation accessory of the wearable device includes a Hall effect sensor, and a magnet is positioned on the movable actuator so that it causes the Hall effect sensor to output signals to the controller due to movements of the movable actuator. The controller includes a processor and a memory coupled thereto which stores processor-executable instructions that, when executed by the processor, cause the processor to receive and evaluate input signals from the Hall effect sensor. In particular, the controller evaluates the input signals to determine whether or not they represent a command for the controlled device by assessing the input signals for a signal pattern indicative of any of a plurality of such commands. If/when the processor determines that the input signals represent one of the plurality of commands, then it decodes the respective command and transmits an associated control signal to the controlled device via the control signal interface. The controller may also provide feedback to the wearer by providing an activation signal to a vibration motor. On the other hand, if the processor determines that the input signals from the sensor do not represent a command, no control signal or activation signal is transmitted and the processor proceeds to evaluate further/new input signals from the Hall effect sensor in a like manner as the original input signals.
A communication element, which may be a part of the activation accessory or otherwise included/integrated in the wearable device, is coupled to the control signal interface and is adapted to transmit the control signal from the processor to the controlled device. For example, the communication element may be a cable having a plug configured to mate with a jack at the controlled device, or a transmitter adapted for radio frequency communication with a receiver at the controlled device.
In various embodiments, the movable actuator may be supported in or by a mount on the wearable device, such as a temple piece or the frame of eyewear (e.g., glasses, goggles, AR/VR headset, etc.), a headset, or another arrangement. For example, the movable actuator may be movable with respect to a temple piece or frame of the eyewear, or a frame of a headset, so as to permit operation of the activation accessory at different positions on the wearer. In one example, the movable actuator of the actuation accessory may be positioned on the movable device so that when the movable device is being worn the movable actuator touches the skin of the wearer overlying an area of the wearer's temporalis muscle, or the tendon which inserts onto the coronoid process of the mandible, or masseter muscle. The temporalis muscle and masseter muscle can generally be felt contracting while the jaw is clenching and unclenching, and it is such clench actions which, by virtue of the resulting movement of the movable actuator, can cause the sensor to output signals to the controller.
In some cases, the movable actuator of the activation accessory may be supported in a helmet or mask (e.g., a helmet or mask used by a firefighter, a diver, an aircrew member, or another wearer), where the mask is configured to position the movable actuator so as to be overlying an area of the wearer's temporalis or masseter muscle. Alternatively, the entire activation accessory may be included in a module having an adhesive applied to a surface thereof to enable a module encasing the activation accessory to be worn directly on the face or head of the wearer. Such an adhesive may, in one case, be in the form of a removable film adhered to the surface of the module that encloses the activation accessory.
The activation accessory may include more than one Hall effect sensor, and/or sensors of different types, with the multiple sensors arranged with respect to one another so as to permit individual and/or group activation thereof by associated volitional jaw clench (or other muscle activity) actions of the wearer. Further, in addition to a vibrational motor, a visual activation indicator may be present. Such a visual activation indicator (e.g., an LED) may be coupled to receive a visual activation indication signal from the controller and the processor-executable instructions, when executed by the processor, may further cause the processor to perform transmit the visual activation indication signal to the visual activation indicator if/when the processor determines that input signals from one or more of the sensors represent a command for the controlled device.
When assessing the input signals from a Hall effect sensor or other sensor for the signal pattern indicative of a command for the controlled device, the processor may evaluate the input signals against a stored library of command signal representations, where each command signal representation characterizes an associated command for the controlled device. Alternatively, or in addition, the input signals may be assessed according to respective power spectral densities thereof within specified time periods. Or the input signals may be assessed according to count values of the Hall effect sensor(s) received within a specified time period. Still further, the input signals may be evaluated against a trained model of command signal representations, where each command signal representation characterizes an associated command for the controlled device.
These and still more embodiments of the invention are described in detail below with reference to the accompanying drawings.
The present invention is illustrated by way of example, and not limitation, in the figures of the accompanying drawings, in which:
Described herein are systems and methods for switched operation, in many cases hands-free operation, of controlled devices, for example illumination systems, push-to-talk systems, computer user interface cursors and other user interface elements, and other devices. These systems and methods are characterized, in part, by a wearable device, such as eyewear (e.g., glasses, AR/VR headsets, goggles, etc.), earphones, headphones, audio headsets, masks, helmets, headbands, garments, hats, caps, modules, etc., that are configured with an activation accessory for a controlled device and a communication element coupled to the activation accessory. The communication element may be a component of the activation accessory or a separate component. The activation accessory generally includes a movable actuator having a range of travel between a fully extended position and fully compressed position and configured to be positioned in and to maintain the fully extended position until acted upon by a force, e.g., by a spring bias, a hinge, or other means. The activation accessory is positioned on or in the wearable device so that when the wearable device is worn the movable actuator of the activation accessory presses against the skin of the wearer, preferably overlying the wearer's temporalis muscle, the tendon which inserts onto the coronoid process of the mandible, or masseter muscle. In other instances, the activation accessory may be positioned on or in the wearable device so that when the wearable device is worn the movable actuator of the activation accessory presses against the skin of the wearer overlying a different muscle or tendon. For convenience, the reminder of the discussion below refers to the movable actuator of the activation accessory being positioned so as to overlie the wearer's temporalis muscle so as to be responsive to jaw clenches of the wearer, but readers should recognize this is only for convenience in explaining aspects of the invention and other positionings of the movable actuator of the activation accessory, through wearing of wearable devices on parts of the body other than the head or face so that the movable actuator is responsive to volitional movements of other muscles (e.g., those lying under areas of the body where the movable actuator would be adjacent to), are contemplated.
When the wearable device is worn so that, for example, the movable actuator of the activation accessory is positioned so as to overlie the wearer's temporalis muscle so as to be responsive to jaw clenches or other jaw movements of the wearer, hands free activation, deactivation, and/or operation of one or more controlled devices is possible. For example, to activate, deactivate, and/or operate a controlled device that is communicably coupled to the activation accessory, e.g., via the communication element, the wearer of the wearable device can perform one or more jaw clench actions. By clenching and unclenching his/her jaw, the wearer's temporalis muscle will be engaged and will expand and contract in the region of the wearer's temple. Because the movable actuator of the activation accessory on/in the wearable device is positioned so as to overlie the wearer's temporalis muscle in the region of the wearer's temple, when the wearer's temporalis muscle expands and contracts in accordance with the wearer's jaw clench actions, the movable actuator, which presses on the skin of the wearer in the temple region, is moved. In one example, a jaw clench or movement causes the movable actuator to move laterally with respect to the wearer and a jaw unclenching or other movement causes the movable actuator to move medially with respect to the wearer. Other motions of the movable actuator, such as rotations, may also be invoked through muscle movement. As described below, these movements of the movable actuator are registered by a sensor associated with the movable actuator and recognized as commands for the controlled device. Once so recognized, the commands are issued to the controlled device via the communication element. For the remainder of the discussion, jaw clenching and unclenching will be described, however, other movements of the jaw, for example lateral-medial movements, are contemplated for actuation of a movable actuator and have proved to be useful when positioning such actuators over a wearer's masseter muscle. Lateral-medial movements, clenching and unclenching, and other jaw movements that result in flexing and relaxing of a wearer's masseter and/or temporalis muscle are contemplated and are generally referred to herein as volitional movements. Similarly, for movable actuators positioned overlying other muscles of a wearer, a variety of volitional movements may be used to manipulate the movable actuator of an activation accessory configured in accordance with the present invention.
In addition to this hands-free operation of the controlled device (in the above example, only a jaw clench/unclench action was used to provide a command to the controlled device), the same activation accessory can be used to activate, deactivate, and/or control the controlled device via touch actions of the wearer. For example, considering the same activation accessory on/in the wearable device with the movable actuator of the activation accessory positioned so as to overlie the wearer's temporalis muscle in the region of the wearer's temple as in the above example, the wearer may cause the movable device to move with respect to its associated sensor by touching/pressing the wearable device instead of by clenching/unclenching his/her jaw. If say the wearable device were eyewear and the movable actuator were positioned along one of the temple pieces of the eyewear so as to contact the wearer's skin in the region of the wearer's temple, then when the wearer pressed the temple piece of the eyewear towards his/her head (i.e., moved the temple piece medially towards his/her head), the movable actuator would move with respect to its sensor and cause the sensor to produce a signal just as if the movable actuator had moved responsive to a jaw clench. And, when the wearer released the temple piece of the eyewear and the temple piece of the eyewear moved laterally away from the wearer's head, the movable actuator would return to its original position with respect to the sensor, still touching the wearer's skin in the region of the wearer's temple, but now extended from the position it was in when the wearer was pressing on the temple piece. This touch/press responsiveness of the activation accessory in addition to its responsiveness to hand-free actions of the wearer provides a very versatile set of operating characteristics for the activation accessory of the wearable device and a wide range of potential operating commands for the controlled device could be made up of successive hands-free/touch-press actions of the wearer.
As noted, the sensor or sensors of the activation accessory is/are responsive to movements of the movable actuator. One such sensor is a Hall effect sensor that is responsive to movements of a magnet in the movable actuator. Other sensors could be used and several examples are discussed below. The sensor is communicably coupled to a controller of the activation accessory (or another controller that is included in the wearable device), and the controller has an output coupled to a control signal interface. Generally, the controller may include a processor and a memory coupled to the processor, which memory stores processor-executable instructions that, when executed by the processor, cause the processor to perform various operations. For example, the stored processor-executable instructions, when executed by the processor, may cause the processor to receive, from the one or more sensors, input signals that are produced as outputs of the sensor(s) responsive to movements of the movable actuator. The instructions may cause the processor further to evaluate the input signals to determine whether or not the input signals represent a command for said controlled device. Since the activation accessory is part of or attached to a wearable device, it is conceivable that some motion of the movable actuator, and, hence, some signals output by the sensor(s) to the processor of the controller, may be associated with movements of the wearer that are not intended as movements representing commands for the controlled device. An example might be the wearer talking or eating. Such actions can be expected to cause the wearer's temporalis muscle to expand and contract, thereby causing a movable actuator positioned so as to be overlying the wearer's temporalis muscle in the region of the wearer's temple to move. This movement of the movable actuator would, in turn, cause the associated sensor(s) to produce output signals to the processor of the controller, but those signals should not cause the processor to issue commands to the controlled device because the wearer's movements were not intended to be interpreted as such commands. To address this situation and mitigate the effect of such movements of the wearer vis-á-vis commands issued to the controlled device, a filtering and/or analysis process may be used by the controller to distinguish volitional actions of the wearer that are intended as commands from those which are not.
Examples of the filtering and analysis process may include such things as band-pass filtering of the signals output by the sensor(s) so as to prevent high and/or low frequency signals, associated with high and/or low speed movements of the movable actuator, from being interpreted as signals associated with commands. Signals of a relatively high frequency may be regarded as being associated with rapid movements of the movable actuator, which may be indicative of movements of the wearer's jaw or other muscle when engaged in activities not associated with issuing commands for a controlled device (e.g., eating, talking, etc.). Similarly, relatively low frequency signals may be regarded as being associated with relatively slow movements of the movable actuator, which may be indicative of movements of the wearer's jaw or other muscle when engaged in activities not associated with issuing commands for a controlled device (e.g., stretching). By filtering out such relatively high and/or low frequency signals before they are provided to the processor of the controller for analysis (or by filtering of such relatively high and/or low frequency signals by the processor as a first step in any analysis), the present invention can avoid the issue of unintended commands to the controlled device.
Other actions in place of or in addition to this kind of filtering can be employed. For example, a microphone could be used in conjunction with the activation accessory (or as part thereof) and signals produced by the microphone when the wearer of the activation accessory is speaking provided to the processor. The stored processor-executable instructions, when executed by the processor, may be such that the processor, upon recognizing that the wearer is speaking, may ignore signals from the sensor(s) associated with the movable actuator as any such signals are likely to be the result of movement of the wearer's temporalis muscle (and, hence, the movable actuator) due to such speaking and not the result of the wearer issuing a command for the controlled device. Of course, the processor could be programmed so as to search for special signal patterns that indicate command sequences even when speaking is detected so that the activation accessory can be sued to activate, deactivate, and/or control a controlled device even when the wearer is engaged in a conversation.
Further, and as discussed in greater detail below, the stored processor-executable instructions, when executed by the processor, may cause the processor to assess the input signals from the sensor(s) for one or more signal patterns indicative of a command for a controlled device, for example, by comparing time domain or frequency domain representations of such signals to a stored library of command signal representations. By digitizing and then transforming received input signals from the sensor(s) using a Fast Fourier Transform algorithm, the processor may compare patterns of received input signals to stored replicas of known command clench and/or touch/press operations of the activation accessory and issue commands to the controlled device accordingly.
If the processor determines that the input signals from the sensor(s) represent a command for the controlled device, then the stored processor-executable instructions, when executed by the processor, may cause the processor to decode the command and, subsequently, transmit an associated control signal to the control signal interface. Otherwise, if the processor determines that the input signals from the sensor(s) do not represent a command for the controlled device, then the stored processor-executable instructions, when executed by the processor, will cause the processor to not transmit such a control signal and instead to proceed to evaluate further or new input signals from the sensor.
The communication element, which may be part of the activation accessory or another component of the wearable device, is coupled to the control signal interface and is adapted to transmit control signals from the processor to the controlled device. For example, the communication element may be a simple a cable having a plug configured to mate with a jack at the controlled device. Or the communication element may be a transmitter adapted for radio frequency communication with a receiver at the controlled device. Any of several kinds of radio frequency communications may be used, for example, Bluetooth, Bluetooth Low Energy (BLE), Zigbee, infrared, WiFi HaLow (IEEE 802.22h), Z-wave, Thread, SigFox, Dash7, or other form of radio frequency communication.
As noted, the activation accessory may be integrated into or attached to a wearable device such that when the wearable device is worn on a person the movable actuator of the activation accessory is touching the person at an area overlying the person's temporalis or masseter muscle. Such a wearable device may be a headset, in which case the movable actuator is preferably movable with respect to a portion of the headset, eyewear, in which case the movable actuator may be supported in a temple piece or a frame of the eyewear, or other device, garment, module, or accessory, as described herein.
In further embodiments, the present invention provides a wearable sensor module configured to detect muscle movement of a wearer and to control an electronic device (e.g., an illumination element, etc.). In some cases, the electronic device may be a wearable device that incorporates or includes the wearable sensor module or to which the wearable sensor module is attached. In other cases, it may be a device remote from the wearable sensor module.
The wearable sensor module has a movable control portion and a detection portion. The movable control portion has a defined range of travel in relation to the detection portion between a fully extended position and a fully seated position. In some instances, the movable control portion may be biased (e.g., by a spring, hinge, or other arrangement) so as to maintain its fully extended position until compressed towards its fully seated position by an outside force. When worn, the wearable sensor module contacts the wearer so that the movable control portion partially compresses. This partial compression results in the detection portion producing an initial signal; for example, upon the wearer donning the wearable sensor module, the detection portion may produce the initial input signal as a result of movement (compression) of the movable control portion when coming into contact with the wearer's body in a region overlying the wearer's temporalis muscle (e.g., at or near the wearer's temple). The initial signal may cause the wearable sensor module to wake from a sleep or inactive state so that subsequent movements of the movable control portion caused by flexing and relaxing of the wearer's muscle(s) over which the sensor module is positioned cause the sensor to produce further signals that, when recognized by a controller of the wearable sensor module or the wearable device in which it is instantiated or to which it is attached, result in commands for controlling the electronic device to be generated.
The detection portion of the wearable sensor module is preferably configured to detect varying degrees of movement of the movable control portion, which varying degrees of movement result in commands for controlling the electronic device to be generated. That is, it is recognized that the wearable device in which the wearable sensor module is instantiated or to which it is attached may be worn by different individuals, some or all of which may have heads of different shapes and sizes. So, the movable control portion of the wearable sensor module may be actuated to different degrees by the different wearers. The detection portion is arranged and situated with respect to the movable control portion so as to be responsive to these different degrees of actuation of the movable control portion, e.g., different lengths of travel or movement thereof due to jaw clenching or other muscle movement.
As mentioned, when the movable control portion is not experiencing any external forces acting upon it, it is biased open from the detection portion, e.g., by a hinge or layer of over-molded elastic polymer that provides spring-like bias. Then, when the movable control portion contacts the wearer, e.g., being donned by the wearer, it is partially compressed along its length of travel with respect to the detection portion. This may happen, for example, when the movable control portion contacts an area of the wearer's head or face overlaying the temporalis muscle or overlaying the masseter muscle.
The detection portion of the wearable sensor module may be removably attached or slidably attached to the wearable electronic device. Such configurations may allow for replacement of broken or damaged detection portions. Alternatively, the detection portion is integrated as part of the wearable electronic device or of the wearable device to which the wearable electronic device is attached. And, as described above, input actuations of the movable control portion may be generated both in a hand-free manner and/or manually by tapping or pressing the wearable electronic device to cause the movable control portion to compress against or extend away from an area of the body over which the wearable sensor module is positioned. The wearable sensor module is thus configured to detect the movement of the movable control portion as having been affected by tapping or pressing on the medial or lateral side of the wearable electronic device when the wearable electronic device is being worn.
In still further embodiments, the present invention provides a wearable sensor module configured to detect muscle movement and to control an electronic wearable device while attached thereto. The wearable sensor module has a movable control portion, movement of which may be effected by a wearer of the electronic wearable device flexing and/or relaxing his/her temporalis and/or masseter muscle, and a detection portion, which may be attached to the wearable electronic device by an adjustable mounting interface. The movable control portion has a defined range of travel in relation to the detection portion, between a fully extended position (in which the movable control portion is biased when not acted upon by any external forces) and a fully seated position. Accordingly, the movable control portion maintains its fully extended position with respect to the detection portion unless or until compressed towards its fully seated position by an outside force. When worn by a wearer, the wearable sensor module contacts the wearer so that the movable control portion partially compresses, establishing an initial signal by the detection portion, for example upon the wearer donning the wearable sensor module. The initial signal and subsequent outputs of the detection portion being effected by movement of the movable control portion caused by coming into contact with the person of the wearer and there after flexing and relaxing of muscles associated with volitional jaw movements of the wearer. The initial signal established as a result of partial compressing of the movable control portion when the wearer dons the wearable sensor module is generated at a point at which the movable control portion is positioned at location between its fully extended position and fully seated position so as to provide for subsequent adequate movement of the movable control portion with respect to the detection portion in order to generate commands for a controlled device upon the position of the movable control portion being affected by volitional jaw movements of the wearer.
Activation elements of wearable devices configured in accordance with embodiments of the present invention may be employed in combination with eyewear (e.g., eyeglasses, goggles, AR/VR headsets, etc.), headsets, masks, garments, accessories, or other head/face-worn articles used in a variety of contexts, including military, law enforcement, health care, and others (e.g., consumer). The head/face-worn article positions the movable actuator of the activation element so that it overlies an area of the wearer's temporalis muscle so that clenching/flexing of the wearer's jaw moves the movable actuator with respect to the sensor, thereby allowing for hand-free operation of the controlled device. Other embodiments of the invention make use of the movable actuator as part of other head-worn articles, including but not limited to illumination, imaging, and/or communication systems. In some instances, the movable actuator may be positioned in locations other than over the wearer's temporalis muscle, allowing activation/deactivation/operation of controlled devices by means of muscles associated with a wearer's eyebrow, jaw, or other body part.
As used herein, when referencing an area of a wearer's head or face overlying the temporalis muscle, we mean that movable actuator is positioned to contact the right or left side of the wearer's head or face within an area generally behind the eye and forward of the ear, near an area where the frontal, parietal, temporal, and sphenoid bones of the skull fuse. In other cases, for example where the movable actuator is positioned by a headset or similar arrangement, it may be positioned above the wearer's ear. The movable actuator is responsive to a relaxed condition and a flexed condition of the wearer's jaw, that is, it is movable with respect to the sensor responsive to the user clenching and unclenching his/her jaw, thereby allowing the wearer to generate input signals for operating/activating/deactivating controlled devices, such as electronic system components, via such clench actions. Note that while much of the discussion herein refers to actions of a wearer's jaw, such as clenching/unclenching, activation elements configured in accordance with embodiments of the present invention may be employed in connection with other volitional acts of a user moving his/her muscles. Also, while jaw clenches/unclenches are a preferred form of manipulation of a movable actuator, hand/finger presses can also be used. For example, in the case of an activation element mounted on eyewear or a headset, a hand/finger press on the outside of the eyewear temple piece or earphone cup may cause the movable actuator to move with respect to the sensor of the activation element, resulting in signals being provided from the sensor to the controller of the activation element for operation/activation/deactivation of the controlled device.
The support for the activation element may be adjustable in terms of the positioning of movable actuator so that it overlies a portion of the wearer's temporal muscle. Further, the movable actuator may be arranged so as to be at its fully extended position when the activation element is not being worn. For example, the movable actuator may include a spring or hinge that is biased so as to be extended or open when the activation element is not being worn. Then, when a user dons the activation element, e.g., by putting on eyewear or a headset that includes the activation element, the movable actuator may be partially compressed or moved, e.g., by contacting the wearer's head or face, to a semi-closed position between its fully extended position and fully compressed position. This movement of the activation element with respect to the sensor may cause the sensor to issue an output signal which the controller may interpret as a wake-from-sleep or similar command to begin sampling the sensor output for possible controlled device command signals.
The use of “clench interactions” has been recognized as a viable control technique. For example, the present applicant's U.S. PGPUB 2020/0097084, Xu et al., “Clench Interaction: Novel Biting Input Techniques,” Proc. 2019 CHI Conference on Human Factors in Computing Systems (CHI 2019), May 4-9, 2019, Glasgow, Scotland UK, and Koshnam, E. K. et al., “Hands-Free EEG-Based Control of a Computer Interface based on Online Detection of Clenching of Jaw,” in: Rojas I., Ortuño F. (eds) Bioinformatics and Biomedical Engineering, IWBBIO 2017, pp. 497-507 (Apr. 26-28, 2017) all provide examples of such techniques. In Xu et al., the use of bite force interfaces may afford some advantages in some applications, however, the present invention adopts a different approach inasmuch as it relies on sensors placed outside a user's oral cavity. Such sensors are more suitable for applications where the presence of sensors inside one's mouth may be uncomfortable or impractical. In Koshnam et al., the EEG sensors were external to the oral cavity, having been placed at temporal sites T7 and T8 on the wearer's head, but there was no provision for alerting the wearer when a command signal was recognized as having been initiated through a jaw clench action. Accordingly, the system was perceived as having excessive lag time in recognizing and implementing a clench action, which adversely impacted its use as a control element for a remote device.
Referring to
Examples of movable actuators and sensors are further illustrated in
Other sensors that can be used include fiber optic compression sensors in which the illuminance of photonically energized fiber optic cable as detected by a photosensor is varied according to the compression of a sleeve or other attenuator surrounding or enclosing the fiber optic cable (e.g., by the action of a movable actuator responsive to jaw clench or other muscle movements of a wearer). Photosensor output is analyzed and processed as an input command for controlling electronic devices in the manner described herein. Such a sensor/controller arrangement is very lightweight and unobtrusive, requires no electronic components (and so is highly rugged/waterproof), features low-compute signal processing, provides variable input and is very low cost. The sensor's actuator can be placed away from the light source and photosensor expanding design flexibility. The sensor/controller could be positioned on eyewear and actuated by the temporalis muscle via jaw clenching to control electronic devices or positioned over other areas of the body to provide hands-free input by detecting movement. For example, one or more sensors could be attached to a glove and positioned over one or more knuckle bones in order to detect grasping/clenching. As the glove tightens over the knuckles while grasping/clenching, photonic output from the fiber optic cable would be reduced and detected by the photosensor, resulting in input commands being generated
The processor 22 of controller 18 is also coupled to a memory 26, which stores processor-executable instructions that, when executed by processor 22, cause processor 22 to receive and evaluate input signals from the Hall effect sensor 16. Controller 18 (i.e., processor 22) evaluates the input signals to determine whether or not they represent a command for the controlled device by assessing the input signals for a signal pattern indicative of such a command. As more fully discussed below, if/when the processor 22 determines that the input signals from Hall effect sensor 16 represent the command for the controlled device, then processor 22 decodes the command and transmits an associated control signal to the controlled device (not shown in this view) via the control signal interface 24, and optionally transmits an activation signal to the vibration motor 12. On the other hand, if the processor 22 determines that the input signals from Hall effect sensor 16 do not represent the command for the controlled device, no control signal or activation signal is transmitted and processor 22 proceeds to evaluate further/new input signals from the Hall effect sensor 16 in a like manner. In one embodiment, the activation signal for the vibration motor is a pulse width modulated signal. The haptic feedback provided by vibration motor 12 may also be activated by another user (e.g., through a communication to the wearer of activation accessory 10) to provide a means for silent communication.
Referring now to
In addition to the above-described examples, the processor 22 may also communicate with and control other peripherals, such as a heads-up display, audio input/output unit, off-headset unit, etc. Processor 22 is a hardware-implemented module and may be a general-purpose processor, or dedicated circuitry or logic, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)), or other form of processing unit. Memory 26 may be a readable/writeable memory, such as an electrically erasable programmable read-only memory, or other storage device.
Referring now to
In some cases, as shown in
In still other embodiments, the activation accessory 10 may be supported on a temple piece of eyewear 64, as shown in
Further, additional sensors such as for wearer vital signs monitoring may also be integrated into the temple 66 to provide remote biomonitoring of the wearer, as the temple area has been proven to be an effective location for sensing certain vital signs. Such sensors may be integrated into the eyewear temples 66, permanently attached as an accessory, or attached to the inside of the temple using adhesive tape, glue, magnets, hook and loop fasteners, screws, or a tongue and groove or dovetail profile connection mechanism. The sensor signal may be routed through a powered cable/tether or via a wireless connection such as Bluetooth or Near Field Magnetic Induction. In other embodiments, one or more biomonitoring sensors 65 may be integrated onto/into a movable actuator of the activation accessory 10, for example at a point of contact between the movable actuator and the wearer's skin.
The activation accessory 10 may include more than one Hall effect sensor 16, with the multiple sensors arranged with respect to one another so as to permit individual and/or group activation thereof by associated volitional jaw clench actions of the wearer. For example,
Further, as shown in
In the various embodiments, activation accessory 10 is positioned so that the movable actuator contacts the wearer's head or face, over the temporalis muscle so that clenching/flexing of the jaw activates the Hall effect sensor 16. Power supply and control electronics for the activation accessory 10 may be incorporated within the activation accessory 10 itself, and/or in a frame, helmet, or mask that supports the activation accessory 10 or elsewhere. In the arrangement shown in
In the various embodiments, the movable actuator 8 may be hingibly attached to or within activation accessory 10, for example by a spring-loaded hinge that keeps the movable actuator 8 against the wearer's head or face even when the wearer moves his/her head, unless moved away from the wearer's head/face by an amount sufficient to engage a detent that prevents return to a position adjacent a wearer's face unless manually adjusted by the wearer. Such a hingible arrangement may incorporate a spring-loaded hinge of any type, for example a spring-loaded piano hinge, butt hinge, barrel hinge, butterfly hinge, pivot hinge, or other arrangement. Other embodiments include the use of a living hinge or an elastic/memory effect produced by wholly or partially encapsulating the movable actuator in an over-molded elastic polymer which produces a stretchable membrane effect, and which also provides water resistance.
As should be apparent from the above discussion, use of the activation accessory does not require donning a headset or mask. Instead, the activation accessory can be worn by itself, e.g., through use of an adhesive. Incorporating the activation accessory in headsets would typically be the norm for any member of an aircraft flight or operations crew, but headsets such as the one illustrated in the above-referenced figures are not restricted to use by flight/aircraft crews and may be employed by ground forces, naval/coast guard personnel, and civilians. For example, headsets such as the ones described herein may be employed by workers in and around constructions sites, sports arenas, film and television production locations, amusement parks, and many other locations. By employing headgear equipped with activation accessories such as those described herein, wearers thereof have ready access to activation/deactivation/operation of illumination, imaging, gaming, and/or communications system(s)) in a hands-free fashion. Note that although
When assessing the input signals from the Hall effect sensor(s) 16 for a signal pattern indicative of a command, the processor 22 may evaluate the input signals against a stored library of command signal representations, where each command signal representation characterizes an associated command for the controlled device. Alternatively, or in addition, the input signals may be assessed according to respective power spectral densities thereof within specified time periods. Or the input signals may be assessed according to count values of the Hall effect sensor(s) received within a specified time period. Still further, the input signals may be evaluated against a trained model of command signal representations, where each command signal representation characterizes an associated command for the controlled device.
An example of an input signal received by processor 22 from Hall effect sensor 16 is illustrated in
As illustrated in
In general, Hall effect sensor 16 is a device that requires little or no mechanical displacement of a control element associated with movable actuator 8 in order to signal or effect a change (or desired change) in state of a controlled system. Other examples of such a device which may be used in place of the Hall effect sensor 16 include an EMG sensor or a piezo switch, such as the Piezo Proximity Sensor produced by Communicate AT Pty Ltd. of Dee Why, Australia. Piezo switches generally have an on/off output state responsive to electrical pulses generated by a piezoelectric element. The electrical pulse is produced when the piezoelectric element is placed under stress, for example as a result of compressive forces resulting from movement of a movable actuator 8 responsive to a wearer clenching his/her jaw so that pressure is exerted against the piezoelectric element. Although the pulse is produced only when the compressive force is present (e.g., when the wearer's jaw is clenched), additional circuitry may be provided so that the output state of the switch is maintained in either an “on” or an “off” state until a second actuation of the switch occurs. For example, a flip-flop may be used to maintain a switch output logic high or logic low, with state changes occurring as a result of sequential input pulses from the piezoelectric element. One advantage of such a piezo switch is that there are no moving parts (other than a front plate that must deform by a few micrometers each time a wearer's jaw is clenched) and the entire switch can be sealed against the environment, making it especially useful for marine and/or outdoor applications.
Another example is a micro tactile switch. Although tactile switches employ mechanical elements subject to wear, for some applications they may be more appropriate than Hall effect sensors or piezo switches because they provide mechanical feedback to the user (although the haptic feedback provided by vibration motor 12 also provides an acceptable level of feedback for a user and so may be sufficient in the majority of instances). This feedback can provide assurance that the switch has been activated or deactivated. Momentary contact tactile switches may also be used, but because they require continual force (e.g., as provided by clenching one's jaw against the switch), they are best suited to applications where only a momentary or short engagement of the active element under the control of switch is desired, for example, signal light flashes, burst transmissions, or other short duration applications, or where a flip flop is used to maintain an output state until a subsequent input is received, as discussed above. Other forms of switches include a ribbon switch (e.g., as made by Tapeswitch Corporation of Farmingdale, N.Y.) and conductive printed circuit board surface elements activated via carbon pucks on an overlaid keypad.
Further, in various embodiments, the controlled device may consist of one or more LEDs, which emit light in one or more wavelengths. Further, the controlled device may include one or more cameras for digital still and/or video imaging. In some instances, a lighting element may be worn on one side of the headset while an imaging system is worn on the opposite side, each being controlled by separate activation accessories mounted on respective opposite sides of the headset, or by activation accessory if the lighting and illumination systems are responsive to different command signals, similar to the way in which computer cursor control devices (e.g., touch pads, mice, etc.) may be separately responsive to single, double, triple, or other multiple clicks. Indeed, the activation accessory may itself be used to control a cursor as part of a user-computer interface. For example, any or all of cursor type, cursor movement, and cursor selection may be controlled using an activation accessory 10 positioned so that the movable actuator is flush against the wearer's face (or nearly so), over the area of the temporalis muscle so that clenching/flexing of the jaw activates the Hall effect sensor 16 or other sensor. Applications for such uses include computer gaming interfaces, which today commonly include head-worn communication equipment. One or more activation accessories 10 configured in accordance with embodiments of the invention may be fitted to such headgear (either when manufactured or as an after-market addition) to provide cursor control capabilities. Conventional wired or wireless communication means may be employed to provide a connection to a console, personal computer, tablet, mobile phone, or other device that serves as the gaming or other host. The use of such human-machine interfaces may find particular application for users that have no or limited use of their hands and afford them a convenient means of interacting with a personal computer, tablet, mobile phone, or similar device.
Further, the controlled device(s) may include one or more microphones. Such microphones may be mounted or integral to a headset and make use of bone conduction transducers for transmission of audio signals. Alternatively, or in addition, activation accessory 10 may be used to adjust the presence, absence, and/or volume of audio played through one or more earphones or other earpieces. Also, activation accessory 10 may be used to control off-headset equipment, for example, via a wireless transmitter.
One or more of the above-described embodiments may permit signal generation via a control surface that can be activated by direct or indirect force, hinged paddle, touch-sensitive surface, or other tactile actuation device. Devices configured in accordance with these embodiments may employ movable structures (e.g., paddles) that house Hall effect sensors to detect a change in an electromagnetic field when a corresponding magnet is moved in proximity to the sensor. Such devices may be in the form of an accessory to a remote (e.g., hand-held) device or fully integrated into a wearable form factor such as eyewear and headsets. Other sensors, as discussed herein, may also be used.
By providing both a left and right activation means (or any number of them) which may be configured to allow for input of various command sequences (e.g., different numbers of activations similar to single-, double- or other mouse clicks), a user may provide different commands for an associated device. For example, different command activation sequences may be used for zooming a camera, panning a direction in a virtual/visual environment, or a host of other commands to control cameras, audio transmissions (volume up or down), etc. In addition to the foregoing, the use of gyros and/or accelerometers while clenching and holding can allow for selecting and moving objects in the virtual field. This is similar to a click-and-hold followed by movement of a cursor with a mouse or joystick in that it allows a user to move objects (e.g., icons) around on a virtual desktop, to open menus, and to select commands, etc. by clenching and moving one's head. The gyros and/or accelerometers may be incorporated in activation accessory 10 or elsewhere (e.g., in a frame supporting the activation accessory 10).
Thus, systems and methods for operating a controlled device in a hands-free or other manner through volitional jaw clench actions of a wearer, and in particular using an activation accessory for a controlled device that includes a movable actuator, sensor (e.g., a Hall effect sensor), and a communication element have been described. In various embodiments, the present invention improves the functionality of controllable electronic devices by providing improved hands-free and tactile input and control methods that cater to both fully abled and disabled users, Moreover, because the movable actuator of the activation accessory has a range of travel between its fully extended position and fully compressed position, when worn on temple pieces of eyewear and the like the activation accessory accommodates a wide variety of wearers, e.g., those with wide or thin faces, those with or without facial hair, etc. The position of the movable actuator when the activation accessory, or an instrumentality in which it is positioned/included, is donned may contribute to an initial output signal of the Hall effect sensor, but this initial signal can be taken as a baseline value and accommodated when analyzing the output signal of the sensor for commands. Beyond jaw clenching, an input signal can be produced manually by tapping or pressing the activation accessory or an instrumentality in which it is positioned/included at a location that causes the movable actuator to first be compressed then extended, or conversely, for it to first be extended then compressed. The sensor can detect if the tapping or pressing is generated from the right or left side of a head-worn device depending on whether it first detects compression or extension of the movable actuator surface when tapping/pressing force is applied. The movable actuator allows for varying levels of input by detecting movement (e.g., travel, speed, duration, etc.) of the movable actuator caused by clenching, tapping, or pressing, from very light to very hard. While described with reference to eyewear and similar articles, the activation accessory may be worn on/with other wearables that feature a flexible or rigid band (e.g., AR/VR headsets), the inside surface of which defines a plane alongside the wearer's face/head or other wearable bands that define other planes when worn on other parts of the body (e.g., wristbands, etc.). Any such wearable will permit the movable actuator to be moved with respect to the sensor, allowing the sensor to output signals responsive to the wearer moving/flexing associated muscles.
This is a NONPROVISIONAL of, claims priority to, and incorporates by reference U.S. Provisional Application Nos. 63/110,463, filed 6 Nov. 2020, and 63/260,499, filed 23 Aug. 2021, and is a CONTINUATION-IN-PART of and incorporates by reference U.S. application Ser. No. 17/247,976, filed 4 Jan. 2021, which claims priority to U.S. Provisional Application Nos. 62/705,524, filed 2 Jul. 2020, and 63/110,463, filed 6 Nov. 2020.
| Number | Date | Country | |
|---|---|---|---|
| 63110463 | Nov 2020 | US | |
| 63260499 | Aug 2021 | US | |
| 62705524 | Jul 2020 | US | |
| 63110463 | Nov 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17247976 | Jan 2021 | US |
| Child | 17453717 | US |
