Patent Grant
11165938
At least one embodiment of the present disclosure pertains to head-mountable display and viewing devices, and more particularly, to a head-mountable display device designed to be worn by an animal and having a first person viewing system.
Head mountable display (HMD) devices are available on the market today, but they are designed only for use by humans. Thus, their current form factors, optical arrays and software are not suitable for use with animals. Additionally, certain camera products are available that can be put on an animal and used to acquire images approximately from the animal's perspective, such as GoPro cameras, which can be put on a dog's back. However, these products are also generally intended for human use and are not optimal for use with animals, due to problems with image stability and other factors.
One or more embodiments of the present disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements.
In this description, references to “an embodiment”, “one embodiment” or the like, mean that the particular feature, function, structure or characteristic being described is included in at least one embodiment of the technique introduced here. Occurrences of such phrases in this specification do not necessarily all refer to the same embodiment. On the other hand, the embodiments referred to also are not necessarily mutually exclusive.
Introduced herein are a device and system for communication between an animal and a human, that enables direct, specific communication of instructions to the animal, such as regarding desired direction and/or desire of the human. Note that the term “animal” as used herein is intended to exclude humans. The system can include an Animal Wearable Head Mountable Display (AW-HMD) device that provides visual cues directly to an animal at the direction of the handler, who may be located remotely from the animal (i.e., not within the animal's normal range of hearing or vision). The system also or alternatively include an Animal Wearable First Person View (AW-FPV) system that enables direct viewing by a human of the animal's visual perspective. Further, the system may provide nonvisual signals to the animal, such as tactile (e.g., haptic), auditory or olfactory signals, or any combination thereof, which may be based on remote user inputs from the human.
I. Animal Wearable Head-Mounted Display System
At least some of the embodiments of the AW-HMD device introduced here comprise a headset portion that includes a head fitting in the form of a ruggedized goggle designed to accommodate the distinct form factor of an animal's head (generally though not necessarily designed to optimally fit a particular animal species), a light source, a lens array, camera optics, and controls and feedback mechanisms for auditory, olfactory and/or tactile data. The AW-HMD enables viewing by a human of an animal's direct line of sight and visual communication of images back into the line of sight of the animal. The device can be used to provide signals to the animal so that there is no ambiguity in direction, target, or any other visual reference point. The AW-HMD device can include non-visual sensors, multiple cameras and integrated optical equipment and associative computing and processing power. These components may include optics equipment (e.g., display, one or more lenses, beamsplitter) to display visual content to the animal, an outward facing camera to capture the animal's line of sight, and an inward facing camera to address alignment between the animal's eye and optics and/or to perform eye tracking. Additional sensory and spectral detection sensors can also be included, along with custom-designed software to address issues related to mammal-specific visual acuity and/or other species-specific visual acuity.
The AW-HMD device disclosed herein is designed for animals, with at least some contemplated embodiments being particularly designed for mammals, and more particularly, for canines. However, the principles introduced here can additionally or alternatively be applied and/or adapted for use with other species, including non-mammalian species. At least some embodiments of the system introduced here have the following capabilities: (1) permit delivery of information into the line of sight and/or perception of the animal without discernment of such information by third parties; (2) provide direct line of sight (“first-person” viewing) of what the animal sees (here the term “person” in “first-person” does not refer to a human); (3) integrate olfactory responses into the design; (4) adjust for varied lighting scenarios; (5) automatically calibrate to the eye position of the animal; (6) integrate haptic controls for additional communications; and (7) integrate audio controls for bi-directional audio communication.
At least some embodiments of the AW-HMD system may also have one or more of the following: a near field-light source, an optical array, a flexible near-to-eye display (NED), an infrared camera and imaging to detect heat sources, optical arrays for use in veterinary/clinical practices to test animal visual acuity and perception, spectral sensors to detect external anomalies, audio components, pupil tracking, and/or haptics.
The AW-HMD device 1 may have any of various different form factors to fit any of various different functional and anthropomorphic requirements. In some embodiments, the AW-HMD device 1 may be designed with conventional goggles 2, as noted above. The goggles 2 may at least partially enclose active computer graphics displays, which may be configured as see-through (at least partially transparent) displays, such that digital imagery and objects can be overlaid on the animal's view of its environment. Any of various see-through display technologies and optical designs may be used, such as emissive displays (e.g., light emitting diode (LED), organic LED (OLED), or active-matrix OLED (AMOLED)), holographic displays, or the like. The optical configuration may include a single reflexive display, monocular display, or binocular display.
The AW-HMD device 1 may also have a number of integrated computing elements, which may include one or more integrated microprocessors and/or digital signal processors (DSPs), power management, as well as wired and wireless communications transceivers (e.g., for universal serial bus (USB), cellular, WiFi, Bluetooth, mesh connections, etc.). The AW-HMD device 1 may also have one or more positional sensors, such as global positioning system (GPS) circuitry, accelerometers, inertial measurement unit (IMU), or the like. It may also have other sensors, such as a camera(s), rangefinders, microphones, speakers, hyper-spectral camera, spectral illuminators, temperature sensors, olfactory detection, Geiger counter, and the like.
Some embodiments of the AW-HMD device 1 may include eye-tracking sensors to determine and assist in the alignment of the optical display(s) to the animal's eye(s). Data gathered by eye-tracking sensors may also be made available through a networked connection 4 to remote user interfaces and devices 3, such that the external interfaces and devices 3 can be alerted to changes in the eye-box relative to the animal's active field of view (FOV). Additionally, embodiments of the AW-HMD device 1 may also optimize the delivery of imagery to the animal through the display such that colors are optimized for the animal or animal species (e.g., dichromacy for canines) as well as for optimal contrast. Additionally, the AW-HMD device 1 may generate images and objects for display to the animal by using shapes optimized for the visual perception capabilities of a particular animal or animal species, which shapes may include linear, oblong, elliptical, orbicular, lanceolate, ovate, oblanceolate, obvate, triangular and deltate shape types.
The AW-HMD device 1 may also have integrated command and control features, which may include contextually based control, active control, user control, remote control, passive control, or the like. For example, the AW-HMD device 1 may have an integrated sensor, such as a camera and associated detection circuitry that can perform object recognition (e.g., facial, landmark, or the like) on captured images, such that the integrated processing system can interpret an object and relay an indication of the identified object to the remote user interfaces and device 3 through a networked connection 4. The AW-HMD device 1 may also provide various types of signals to the animal directly (automatically and/or based on user inputs) through the optical display.
The AW-HMD device 1 may also adjust itself based on measured or perceived environmental conditions, such as ambient light. For example, in dark environments the AW-HMD device 1 may decrease the level of contrast of a digitally displayed object or image. In some embodiments, the control technology may be mounted on the AW-HMD device 1 such that a user can make manual adjustments directly to the AW-HMD device 1. Alternatively or additionally, some embodiments may include the ability to make adjustments to the contrast of displayed objects through an remote user interface and device 3 through a networked connection 4.
Additionally, some embodiments of the AW-HMD device 1 may include active sensory feedback. For example, embodiments of the AW-HMD device 1 may have integrated olfactory delivery capability, such that a user connected through a networked remote user interface 3 may trigger delivery of olfactory cues through an olfaction sensor and delivery component 7 on the AW-HMD device 1, to provide signals to the animal. Some embodiments of AW-HMD device 1 may also include haptic feedback elements 8 to provide signals to the animal. For example, a user, through a network connection 4 and an remote user interface or device 3, may input into the remote user interface 3 to activate the haptic feedback elements 8 on AW-HMD device 1.
Some embodiments of the AW-HMD device 1 may have sensors that detect movement of the animal (e.g., pace change, position, and the like) including accelerometers, gyros, and other internal measurements, where the integrated processors may interpret the tracked movement and provide quantitative and qualitative measurements of the animal's position and/or motion to the remote user interfaces and device 3.
As noted above, the AW-HMD device 1 may be in communication with remote user interfaces 3. The remote user interfaces 3 may be generated by devices having any of various different forms. For example, a cell phone screen may be adapted to receive input from AW-HMD device 1 and may be set up to actively control functional aspects of the AW-HMD device 1. The remote user interface 3 may additionally or alternatively include other forms, such as one or more tablets, laptop computers, desktop computers, or the like. In each case, the remote user interface may also include sensors (e.g., IMU, accelerometers, compass, temperature, and the like) to provide additional input in controlling the AW-HMD device 1 and instructing the animal.
As mentioned above, the AW-HMD device 1 may include one or more optical modules to display images to the animal.
A portion of the light that is emitted by the display 202 is reflected by the reflective beamsplitter/combiner 203 (perpendicularly to the plane of the drawing) into the eye of the animal (not shown). Another portion of that emitted light is passed through the beamsplitter/combiner 203 and captured by the camera 205, such that any images produced by the display 202 can be monitored and tracked in three-dimensional space by remote user interfaces and devices 3. Additionally, a portion of the light that impinges on the beamsplitter/combiner 203 from the animal's environment is reflected by the beamsplitter/combiner 203 to the camera 205, which provides images of the animal's optical perspective for transmission to remote user interfaces and devices 3.
The optical module 201 can be lightweight and compact such that it fits into a small portion of an AW-HMD device 1. This allows integration of the optical module 201 into off-the-shelf canine goggles, such as canine goggles made by Rex Specs. An example of the manner of integration is illustrated in
The manner of operation of the optical module of the AW-HMD 1 is further explained now with reference to
As mentioned above, in at least some embodiments displays digital image content to an eye 239 of an animal wearing the AW-HMD 1 and allows the animal to see through the display, such that the digital content is perceived by the animal as augmenting the animal's view of the surrounding environment, i.e., as augmented reality (AR) and/or mixed reality (MR) images. The optical configuration may have a variably transmissive optical element that is in-line with the animal's view of its surroundings, such that the degree of transmission of the see-through view can be increased and decreased. The variably transmissive optical element may be or include, for example, the beamsplitter/combiner 203 and/or one or more of the lenses 207. This feature may be helpful in situations where the animal would be better served with a high degree of transmission of see-through view and when, in the same AW-HMD device 1, the animal would be better served with a lower degree of transmission of see-through transmission. The lower degree of see-through transmission may be used, for example, in bright conditions and/or in conditions where higher contrast for the digitally presented objects are desirable.
The camera captures images (video and/or stills) of the surrounding environment by receiving reflected light from the surrounding environment off of the beamsplitter/combiner 203, which is in-line with the animal's see-through view of the surrounding. In some embodiments, the light emission surface of the display element 202 is enclosed by a light shield 402 (
In some embodiments, as shown, the camera 205 aperture is perpendicular to the direction in which the animal is looking and directly aligned with the display element 202, as shown in
In some embodiments, the beamsplitter/combiner 203 includes a coating on its surface that faces the camera 205, such that visible wavelength light is substantially transmitted while infrared light is substantially reflected; and the camera 205 captures images that include at least a portion of the infrared wavelength light. In such embodiments, the image light 233 includes visible wavelength light, and portion 237 of the visible wavelength light is transmitted by the beamsplitter/combiner 203. This may be useful to remote users through a networked connection 4 on remote interfaces and devices 3 to view the image presented by AW-HMD device 1 to the animal when in low-light conditions, for example.
As noted, the optical module 201 may contain a lens 207 or an array 204 of lenses 207, where light from the display element 202 is projected through the lens or lens array 204 onto the beamsplitter/combiner 203 to overlay objects onto the animal's view of the real world. Light control structures (not shown) can also be included to control the distribution of the light that is delivered by the display element 202. The light control structures can include, for example, diffusers, elliptical diffusers, prism films and lenticular lens arrays, prism arrays, cylindrical lenses, Fresnel lenses, refractive lenses, diffractive lenses or other structures that control the angular distribution of the image light 233.
Additionally, the optical module 201 is not limited to a side-mounted display nor to a monocular display. Other embodiments, for example, can be in a binocular display, whereby images and objects are displayed to both eyes simultaneously. Additionally, other embodiments may include a single, flexible reflexive surface, where images and objects are displayed directly onto that reflexive surface.
An embodiment of the optical system discussed with regard to
As shown in
Another consideration relating to integration of an optical module into an animal-worn form factor is the departure from traditional design principles that are based on a more-planar human face, to a design suitable for an animal with a less-planar face, such as a dog or other animal with a prominent snout. To account for these variants, the location of the optical array may need to be adjusted so that it can clear, in this present example, the snout of a canine. To enable this adjustment and to account for potential variants of an animal's facial features while also allowing for the best “eye-box” to be presented to the animal, in certain embodiments the mechanism used to mount the optical array 201 to the goggles 400 can allow for rotational adjustment of the optical array 201 relative to the goggles 400. With this rotation of the optical module 201, it is possible to adjust its positioning for protruding features, such as the animal's snout. To further account for such facial variations, a small portion 206 of the beamsplitter/combiner 203 can be cut off near an edge, as shown in
Moreover, the AW-HMD device 1 is not limited to being able to highlight objects in the physical world that are in the immediate and direct FOV of the optical module 201. The technology introduced here can also enable a remote user to tag or otherwise direct their input to subject matter outside of the animal's current FOV through the optical module 201 but within the FOV of a camera of the AW-HMD device 1. For example, the remote user can “tag” an area or object within the addressable FOV of the camera on the user interface 3 but outside the animal's FOV through the optical module 201, which may cause the AW-HMD device 1 to direct the animal (e.g., through visual, haptic or audible signal) to adjust its head position, to bring the tagged area/object within the animal's FOV through the optical module 201.
In the illustrated embodiment, the physical components of the AW-HMD device 1 include one or more of each of: a processor 901, a memory 902, an optical module 903, an eye-tracking video camera 904, a video camera 905 for imaging the animal's environment in the animal's line of sight, a communication subsystem 906, an audio subsystem 907, a tactile subsystem 908, and an olfactory subsystem 909, all coupled together (directly or indirectly) by an interconnect 910. Note that in some embodiments, one or more of these components may be located off the headset portion of the AW-HMD device 1, such as on a vest worn by the animal, as shown and described with reference to
The interconnect 910 may be or include one or more conductive traces, buses, point-to-point connections, controllers, adapters, wireless links and/or other conventional connection devices and/or media, at least some of which may operate independently of each other.
The processor(s) 901 individually and/or collectively control the overall operation of the AW-HMD device 1 and perform various data processing and control functions. For example, the processor(s) 901 may provide at least some of the computation and data processing functionality for generating and displaying computer-generated images to the animal and/or for providing other signals to the animal (e.g., auditory, olfactory or haptic). Each processor 901 can be or include, for example, one or more general-purpose programmable microprocessors, digital signal processors (DSPs), mobile application processors, microcontrollers, application specific integrated circuits (ASICs), programmable gate arrays (PGAs), or the like, or a combination of such devices.
Data and instructions (code) 911 that configure the processor(s) 901 to execute aspects of the mixed-reality visualization technique introduced here can be stored in the one or more memories 902. Each memory 902 can be or include one or more physical storage devices, which may be in the form of random access memory (RAM), read-only memory (ROM) (which may be erasable and programmable), flash memory, miniature hard disk drive, or other suitable type of storage device, or a combination of such devices.
The optical module 903 may include one or more active display elements (e.g., an OLED display) and associated optics, for displaying computer-generated images to the animal. The communication subsystem 906 enables the AW-HMD device 1 to receive data and/or commands from, and send data and/or commands to, a remote processing system, such as remote user interfaces and devices 3. The communication subsystem 906 can be or include one or more of, for example, a Wi-Fi transceiver, cellular transceiver (e.g., LTE/4G or 5G), Bluetooth or Bluetooth Low Energy (BLE) transceiver, baseband processor, a universal serial bus (USB) adapter, Ethernet adapter, cable modem, DSL modem, or the like, or a combination thereof.
The audio subsystem 907 can be or include one or more speakers and/or one or more microphones. The tactile subsystem 908 may be or include one or more haptic actuators to provide haptic signals to the animal, and/or one or more haptic sensors by which to provide haptic feedback to a remote human user. The olfactory subsystem 909 may be or include one or more olfactory delivery elements to provide olfactory signals to the animal, and/or one or more olfactory sensors by which to provide olfactory feedback to a remote human user.
II. Animal-Wearable First Person View System
As noted above, some embodiments of the technology introduced here enable direct viewing by a human of the animal's visual perspective (“first person view”). Thus, embodiments disclosed herein include an Animal-Wearable First Person View (AW-FPV) digital capture and communications system that enables direct observation of an animal's direct line of sight, visual communication back to the animal, as well as bidirectional audio communication between the device and a remote human handler. Such embodiments may include: 1) a video camera system for the direct observation by a human of an animal's visual perspective that can be attached to existing animal worn hardware, objects, devices, and the like, 2) a video and auditory communications system for the direct observation by a human of an animal's visual perspective, and bidirectional audio feedback that can be attached to existing animal worn hardware, objects, devices, and the like, and 3) a video system for the direct observation by a human of an animal's visual perspective, and projecting light emitting sources for visual communication back to the animals that can be attached to existing animal worn hardware, objects, devices, and the like.
The AW-FPV system described herein may be integrated with an AW-HMD such as described above. Alternatively, the AW-FPV system may be a separate device that can be used alone or in conjunction with an AW-HMD system, such as AW-HMD system 1 described above.
Some embodiments of the AW-FPV system include the following components: a ruggedized housing with simple screw mount designed to mount flush with animal worn goggles, and one or more cameras integrated into the housing and capable of capturing the visual perspective of the animal. The AW-FPV system (also called “AW-FPV device” herein) may further include one or more microprocessors, controllers, wireless transceivers, or the like, and memory to store software and data. The AW-FPV system may also have one or more of the following: speaker; microphone; infrared camera; visible-light and/or infrared light sources, thermal camera; accelerometer; and/or magnetometer; gimbal; fold mirror(s); visible light sources. A microphone or microphones may be used to provide ambient or active auditory signals to a remote human handler. A speaker or speakers enable auditory commands from a remote human handler (or a computer system) to be given to the animal or to other persons or animals in the vicinity of the device. An outward facing light emitting source enables display of visual cues back into the line of sight of the animal. Further, additional sensors may also accompany the system along with custom software to specifically address issues related to mammal specific or species specific visual perception, morphology, physiology and the like. Other features of the AW-FPV system may include digital and/or mechanical image stabilization to account for the particular morphology and movement patterns of mammals or specific species.
The FOV provided by the camera or cameras 1200, in some embodiments, may be fixed and in others may be variable dynamically (i.e., during operation). Additionally, in this context the FOV may relate to vertical or horizontal orientations. Moreover, in some embodiments, the cameras 1200 may also have mechanical image stabilization, such that the image captured by the camera is stable in both static and dynamic scenarios. In some embodiments, the cameras 1200 may also have dynamic pan and zoom functionality, controlled by the associated processing capability of the AW-FPV system. In other instances, pan and zoom functionality may be controlled by a human user through remote user interfaces presented on a remote processing device, via a wireless (or wired) network connection (similarly to as described above in connection with the AW-HMD embodiments). In other embodiments, the ability to manipulate pan and zoom, as well as camera orientation and resulting field of view, may be adjusted manually at the AW-FPV device 1101 device itself.
In some embodiments, the illuminating LEDs 1201 may be aligned to the field of view of the cameras 1200. In the case that a remote user interface is being used to remotely view the captured image and environment of the AW-FPV device 1101 through a network connection, the observer may choose to ensure that the illuminating lights match the FOV of the camera, such that the best possible image is presented to the remote user interface. In other instances, it may be desirable to set a narrow and bright path, or a more widely distributed illumination stream; more confined than the camera 1200 FOV, or well beyond the FOV of the camera 1200. In these instances, the user is able to make this dynamic adjustment to the AW-FPV device 1101 through a remote user interface, via a networked connection. In some embodiments, however, as described herein, a user of the AW-FPV system may wish to manipulate the illuminating perspective of the LEDs 1201, manually. In such a case, it may be possible for manual adjustment directly on the AW-FPV device 1101 itself, to adjust the LEDs 1201 to the desired illumination angle.
Some embodiments of the AW-FPV device 1101 may contain indicators 1203 both for functional and aesthetic purposes. Indicators 1203 are assumed to be LEDs herein to facilitate description, but can be any other type of indicator. The LED indicators 1203 can provide indication of the system's successful operation. Alternatively or additionally, the LED indicators 1203, may simply provide a useful yet aesthetic complement to the overall design of the AW-FPV device 1101, controlled either by a remote user interface 103 through a networked connection 104 and/or through manual manipulation at the AW-FPV device 1101. The use of functional and/or aesthetic LEDs 1203 are not mutually exclusive, nor is the number of LEDs 1203 limited to the number shown.
The optional microphone 1202 enables the AW-FPV device 1101 to discern cardioid audio patterns, omnidirectional audio, and/or both. Moreover, based on what is desirable, some embodiments may include more than one microphone 1202, such that appropriate audio coverage is captured.
Some embodiments of the AW-FPV device 1101 may include protective coverings 1204 of the elements described herein. Protective coverings 1204 allow for normal operation of the equipment they are meant to protect, which in this embodiment may be transparent covers to LEDs 1201 and cameras 1200. However, in some embodiments it may be desirable for the protective coverings 1204 to modify the light entering the cameras 1200 and/or emitted by the illuminating LEDs 1201, such that the image is modified in a desirable way to facilitate observation through a remote user interface.
The configuration illustrated in
Some embodiments of the AW-FPV device 1101 may include outwardly projecting light sources, such that a directed beam of light that can be pointed in a direction at the discretion of a user through a remote user interface through a networked connection. Moreover, through detection by the camera(s) 1201 and use of associated processing power in the AW-FPV device 1101 and/or the remote device, the directed light source can be automatically aimed toward objects of interest in the environment 102 (e.g., an object that has just entered the FOV), based, for example, on user-specified settings in the AW-FPV device 1101.
The AW-FPV device 1101, via a remote user interface for viewing information provided by the AW-FPV device 1101 via a networked connection, may also allow for dynamic manipulation of the speakers 1300, based on real-time scenarios. For example, it may be desirable for a remote user to toggle between frequencies delivered by the speaker 1300. Moreover, because of the AW-FPV device 1101 is able to detect environmental conditions through sensors and associative processing power, the AW-FPV device 1101 may also dynamically modulate the frequency and/or volume level delivered through the speakers 1300 based on real-time conditions.
In the illustrated embodiment, the housing for the AW-FPV device is constructed in two primary pieces: a primary housing 1302, and a back housing 1301. In this embodiment the primary housing 1302 and back housing 1301 are connected and mounted to the top portion of an animal worn goggle, such as that produced by Rex Specs, using two fasteners, each comprising a screw and a mounting plate. In other embodiments, it may be desirable for the housing of the AW-FPV device 1101 to be constructed in a uni-body fashion or in more than two primary components.
The AW-FPV device may also include additional components, which are not shown, such as a power supply (e.g., battery), microprocessor, memory, wireless transceiver and antenna. In some embodiments, those additional components may be located within or integral to the housing. For example, the antenna can be of a form that wraps around at least part of the housing of the AW-FPV device. The processor, memory and wireless transmitter can be part of a system-on-a-chip (SoC) mounted on a custom-built printed circuit board (PCB). The battery can be a custom rechargeable battery pack. In other embodiments, at least some of those additional components may be external to the housing; for example, some of all of those components may be mounted on or in a vest 421 worn by the animal 422, as shown in
Additional embodiments may also include snap and lock mechanisms with a cantilever, whereby it may be possible to manually retract the cantilever hook, place the AW-FPV device 1101 onto the surface of which it is to be mounted, in this case, Rex Specs goggles, and upon releasing the cantilever hooks, the hooks would naturally lock the AW-FPV device 1101 into position by locking into place by securing itself to the ribbing 1503 on the top of the Rex Specs goggles. Moreover, additional methods for securing the AW-FPV device 1101 may also be desirable such that the AW-FPV device 1101, can easily secure to an existing device, or object by use of clips. In such an embodiment, pressure mounted clips may be mounted to the AW-FPV device 1101, securing in a similar fashion instead of screws 1501 in
In the illustrated embodiments, the orientation of the front housing 1302, back housing 1301, mounting mechanisms, and overall design have been made with respect to the morphological characteristics of mammals, such that the principles do not adhere to traditional orthogonal human-worn devices. Moreover, the non-planar facial structure of most mammalian species indicates that any animal worn device, specifically relating to devices worn on the head, will tend to have a horizontal rotation. Thus, it may be desirable for some applications of the AW-FPV device 1101 to counter this horizontal rotation tendency of the mounting surface, by mounting the AW-FPV device 1101 in such a way that it can overcome the rotation, giving it a downward focused appearance. In other embodiments, in may be desirable that, instead of orienting the AW-FPV device 1101 in its entirety in a downward manner, the individual components making up the AW-FPV device 1101, such as the cameras 1200 and the like, be mounted within the AW-FPV device 1101 with the appropriate downward angle to accommodate for the specific head orientation of the animal such that the observer through a remote user interface 103 may mirror the viewing angle of the animal wearing the AW-FPV device 1101.
In certain embodiments, the remote observer of the environment being captured by the AW-FPV device 1101 may choose to manually change, control or configure cameras 1200 on the AW-FPV device 1101, and/or change, control or configure the radiance and intensity of the illuminating LEDs 1201, by using a control 1701 (for example, a button, slider, or toggle switch) displayed on the remote user interface 1703. Further, it may be desirable for the observer to modify the picture being displayed on the remote user interface such that the picture more closely matches the acuity and perception of an animal or species wearing the AW-FPV device 1101.
The remote device that generates the remote user interface may be a secondary HMD device, worn by a human, such as an AR, MR and/or virtual reality (VR) headset, such that the wearer of the secondary HMD device can be virtually immersed in the environment of the animal. The remote user interface may be interactive, such that the wearer of the secondary HMD device, through gesture, voice commands, or the like, can affect or adjust operation of the AW-FPV and/or the AW-HMD (e.g., to guide the animal, adjust device settings, etc.)
The AW-FPV device 1101 may include one or more processors to perform and/or control the various functions of the AW-FPV mentioned above, as well as one or more wireless transceivers, memory and other electronic components. To that extent, the block diagram of
Unless contrary to physical possibility, it is envisioned that (i) the methods/steps described herein may be performed in any sequence and/or in any combination, and that (ii) the components of respective embodiments may be combined in any manner.
The machine-implemented operations described above can be implemented by programmable circuitry programmed/configured by software and/or firmware, or entirely by special-purpose circuitry, or by a combination of such forms. Such special-purpose circuitry (if any) can be in the form of, for example, one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), system-on-a-chip systems (SOCs), etc.
Software or firmware to implement the techniques introduced here may be stored on a machine-readable storage medium and may be executed by one or more general-purpose or special-purpose programmable microprocessors. A “machine-readable medium”, as the term is used herein, includes any mechanism that can store information in a form accessible by a machine (a machine may be, for example, a computer, network device, cellular phone, personal digital assistant (PDA), manufacturing tool, any device with one or more processors, etc.). For example, a machine-accessible medium includes recordable/non-recordable media (e.g., read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; etc.), etc.
The term “logic”, as used herein, means: a) special-purpose hardwired circuitry, such as one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), or other similar device(s); b) programmable circuitry programmed with software and/or firmware, such as one or more programmed general-purpose microprocessors, digital signal processors (DSPs) and/or microcontrollers, system-on-a-chip systems (SOCs), or other similar device(s); or c) a combination of the forms mentioned in a) and b).
Certain embodiments of the technology introduced herein are summarized in the following numbered examples:
1. A head-mountable display device comprising: a head fitting designed to fit the head of an animal; and an output subsystem coupled to or integral with the head fitting and configured to output a signal to the animal.
2. A head-mountable display device as recited in example 1, wherein the output subsystem comprises an optical module configured to project an image into an eye of the animal.
3. A head-mountable display device as recited in any of example 1 or example 2, wherein the optical module comprises a display element that is at least partially transparent, on which to overlay augmented reality images and/or mixed reality images on the animal's view of the real world.
4. A head-mountable display device as recited in any of examples 1 through 3, wherein the optical module is configured to display images using shapes and colors optimized for visual perception capabilities of a particular animal species.
5. A head-mountable display device as recited in any of examples 1 through 4, wherein the optical module is configurable to generate images conforming to a plurality of different visual acuity and visual perception requirements for a plurality of different animal species.
6. A head-mountable display device as recited in any of examples 1 through 5, wherein the output subsystem comprises an audio output device.
7. A head-mountable display device as recited in any of examples 1 through 6, wherein the output subsystem comprises a tactile output device.
8. A head-mountable display device as recited in any of examples 1 through 7, wherein the output subsystem comprises an olfactory output device.
9. A head-mountable display device as recited in any of examples 1 through 8, wherein the output subsystem comprises at least two from the list consisting of: an optical module configured to project an image into an eye of the animal an audio output device; a tactile output device; or an olfactory output device.
10. A head-mountable display device as recited in any of examples 1 through 9, further comprising a camera arranged to perform eye tracking of an eye of the animal.
11. A head-mountable display device as recited in any of examples 1 through 10, further comprising a wireless receiver to receive a wireless communication signal via a wireless communication link in response to a user input from a remote user, the wireless communication signals for controlling a function of the head-mountable display device.
12. A head-mountable display device as recited in any of examples 1 through 11, wherein the output system includes an output device controllable by the remote user via the wireless communication signal, to provide the signal to the animal.
13. A head-mountable display device as recited in any of examples 1 through 12, wherein the signal to the animal is a visual signal.
14. A head-mountable display device as recited in any of examples 1 through 13, wherein the signal to the animal is an auditory signal.
15. A head-mountable display device as recited in any of examples 1 through 14, wherein the signal to the animal is a tactile signal.
16. A head-mountable display device as recited in any of examples 1 through 15, wherein the signal to the animal is an olfactory signal.
17. A head-mountable display device as recited in any of examples 1 through 16, further comprising: a camera arranged to acquire viewpoint images corresponding to a visual perspective of the animal; and a wireless transmitter configured to transmit image data representing the viewpoint images to a remote device.
18. A head-mountable display device comprising: a head fitting designed to fit the head of an animal; an optical module configured to project an image into an eye of the animal, wherein the optical module includes a display element that is at least partially transparent, on which to overlay augmented reality images and/or mixed reality images on the animal's view of the real world; and at least two from the list consisting of: an audio output device, a tactile output device and an olfactory output device.
19. A head-mountable display device as recited in example 18, further comprising a wireless receiver to receive a wireless communication signal via a wireless communication link in response to a user input from a remote user, the wireless communication signals for controlling a function of the head-mountable display device.
20. A head-mountable display device as recited in example 18 or example 19, wherein the function is one of a visual signal to the animal, an auditory signal to the animal, a tactile signal to the animal or an olfactory signal to the animal.
21. A head-mountable display device as recited in any of examples 18 through 20, further comprising: a camera arranged to acquire viewpoint images corresponding to a visual perspective of the animal; and a wireless transmitter configured to transmit image data representing the viewpoint images to a remote device.
22. A method of operation of a head-mountable display device designed to be worn by an animal, the method comprising: projecting a plurality of images into an eye of the animal from a display element mounted to the head of the animal; receiving, via a wireless communication link, a control signal from a remote user; and generating an output signal to the animal based on the control signal.
23. A method as recited in example 22, wherein the output signal is included in the plurality of images.
24. A method as recited in example 22 or example 23, wherein the output signal is a non-visual signal.
25. A method as recited in any of examples 22 through 24, wherein the output signal is an audible signal.
26. A method as recited in any of examples 22 through 25, wherein the output signal is a tactile signal.
27. A method as recited in any of examples 22 through 26, wherein the output signal is an olfactory signal.
28. A method as recited in any of examples 22 through 27, further comprising: acquiring viewpoint images corresponding to a visual perspective of the animal; and transmitting image data representing the viewpoint images from the device to a remote device.
29. A sensor module comprising: a housing; at least a portion of a fastener, the fastener configured to removably attach the housing to a head fitting designed to fit the head of an animal; and a first camera at least partially contained within the housing and positioned to acquire image data of a real world environment of the animal when in operation while the sensor module is attached to the head fitting and the head fitting is worn by the animal.
30. A sensor module as recited in example 29, wherein the head fitting comprises goggles.
31. A sensor module as recited in example 29 or 30, further comprising a second camera at least partially contained within the housing and positioned to acquire image data of the real world environment of the animal when in operation while the sensor module is attached to the head fitting and the head fitting is worn by the animal.
32. A sensor module as recited in any of examples 29 through 31, further comprising a microphone at least partially contained within the housing and positioned to acquire audio data of the real world environment of the animal when in operation while the sensor module is attached to the head fitting and the head fitting is worn by the animal
33. A sensor module as recited in any of examples 29 through 32, wherein the head fitting comprises goggles, the sensor module further comprising a plurality of light emission elements at least partially contained within the housing and positioned to illuminate a field of view of the animal when in operation while the sensor module is attached to the goggles and the goggles is worn by the animal.
34. A sensor module as recited in any of examples 29 through 33, further comprising an audio speaker.
35. A sensor module as recited in any of examples 29 through 34, further comprising a second camera at least partially contained within the housing and positioned to acquire image data of the real world environment of the animal when in operation while the sensor module is attached to the goggles and the goggles is worn by the animal; a plurality of light emission elements at least partially contained within the housing and positioned to illuminate a field of view of the animal when in operation while the sensor module is attached to the goggles and the goggles is worn by the animal; a microphone at least partially contained within the housing and positioned to acquire audio data of the real world environment of the animal when in operation while the sensor module is attached to the goggles and the goggles is worn by the animal; and an audio speaker.
36. A head-mountable first person viewing system comprising: a head fitting designed to fit the head of an animal; a first camera coupled to the head fitting and positioned to acquire image data of a real world environment of the animal when the head-mountable first person viewing system is worn by the animal; and a wireless transmitter configured to transmit the image data in real-time, for delivery to a remote device configured to display images of the real world environment of the animal, based on the image data.
37. A head-mountable first person viewing system as recited in example 36, wherein the head fitting comprises goggles.
38. A head-mountable first person viewing system as recited in example 36 or 37, wherein the camera is at least partially contained within a sensor module that is removably attached to the goggles.
39. A head-mountable first person viewing system as recited in any of examples 36 through 38 wherein the sensor module is removably attached to a top surface of the goggles.
40. A head-mountable first person viewing system as recited in any of examples 36 through 39, wherein the top surface of the goggles comprises a webbing, the webbing including spaced apart segments of solid material, and wherein the sensor module is removably attached to the goggles by fasteners, at least portions of which pass through air gaps between the segments of solid material in the webbing.
41. A head-mountable first person viewing system as recited in any of examples 36 through 40, further comprising a second camera positioned to acquire image data of the real world environment of the animal when in operation while the head fitting is worn by the animal; and a plurality of light emission elements positioned to illuminate a field of view of the animal when in operation while the head fitting is worn by the animal.
42. A head-mountable first person viewing system as recited in any of examples 36 through 41, further comprising a housing coupled to the head fitting, the housing at least partially containing the first and second cameras and the plurality of light emission elements.
43. A head-mountable first person viewing system as recited in any of examples 36 through 42, further comprising a microphone at least partially contained within the housing and positioned to acquire audio data of the real world environment of the animal when in operation while the head fitting is worn by the animal.
44. A head-mountable first person viewing system as recited in any of examples 36 through 43, further comprising an audio speaker.
45. A head-mountable input/output (I/O) system comprising: a head fitting designed to fit the head of an animal; an optical output subsystem coupled to or integral with the head fitting and configured to output images to the animal; a plurality of cameras, each at least partially contained within the housing and positioned to acquire image data of a real world environment of the animal when in operation while the head-mountable I/O system is worn by the animal; and a wireless transmitter configured to transmit the image data in real-time, for delivery to a remote device configured to display images of the real world environment of the animal, based on the image data.
46. A head-mountable I/O system as recited in example 45, wherein the head fitting comprises goggles.
47. A head-mountable I/O system as recited in example 45 or 46, wherein the camera is at least partially contained within a sensor module that is removably attached to the goggles.
48. A head-mountable I/O system as recited in any of examples 45 through 47, wherein the sensor module is removably attached to a top surface of the goggles.
49. A head-mountable I/O system as recited in any of examples 45 through 48, wherein the top surface of the goggles comprises a webbing, the webbing including spaced apart segments of solid material, and wherein the sensor module is removably attached to the goggles by fasteners, at least portions of which pass through air gaps between the segments of solid material in the webbing.
50. A head-mountable I/O system as recited in any of examples 45 through 49, further comprising a microphone at least partially contained within the housing and positioned to acquire audio data of the real world environment of the animal when in operation while the head fitting is worn by the animal; and a plurality of light emission elements at least partially contained within the housing and positioned to illuminate a field of view of the animal when in operation while the head fitting is worn by the animal.
51. A head-mountable I/O system as recited in any of examples 45 through 50.
Any or all of the features and functions described above can be combined with each other, except to the extent it may be otherwise stated above or to the extent that any such embodiments may be incompatible by virtue of their function or structure, as will be apparent to persons of ordinary skill in the art. Unless contrary to physical possibility, it is envisioned that (i) the methods/steps described herein may be performed in any sequence and/or in any combination, and that (ii) the components of respective embodiments may be combined in any manner.
Although the subject matter has been described in language specific to structural features and/or acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as examples of implementing the claims and other equivalent features and acts are intended to be within the scope of the claims.
This application is a continuation of U.S. patent application Ser. No. 16/178,384, filed on Nov. 1, 2018, which is a continuation-in-part of U.S. patent application Ser. No. 16/030,672, filed on Jul. 9, 2018, and which claims the benefit of U.S. provisional patent application No. 62/581,501, filed on Nov. 3, 2017. U.S. patent application Ser. No. 16/030,672, filed on Jul. 9, 2018, claims the benefit of U.S. provisional patent application No. 62/537,883, filed on Jul. 27, 2017. Each of the above-mentioned applications is incorporated by reference herein in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3373444 | Militello | Mar 1968 | A |
| 5732415 | Boyd | Mar 1998 | A |
| 6311645 | Brown | Nov 2001 | B1 |
| 7372370 | Stults et al. | May 2008 | B2 |
| 9516862 | Emilo | Dec 2016 | B1 |
| 9628895 | Malaviya | Apr 2017 | B2 |
| 10094855 | Fuhr et al. | Oct 2018 | B1 |
| 10521940 | Cricri et al. | Dec 2019 | B2 |
| 10555106 | Mehra | Feb 2020 | B1 |
| 10959037 | Mehra | Mar 2021 | B1 |
| 20040130681 | Aastuen | Jul 2004 | A1 |
| 20050151924 | Di et al. | Jul 2005 | A1 |
| 20070217586 | Marti et al. | Sep 2007 | A1 |
| 20080141681 | Arnold | Jun 2008 | A1 |
| 20080169923 | Belden et al. | Jul 2008 | A1 |
| 20080243385 | Yamamoto et al. | Oct 2008 | A1 |
| 20090171233 | Lanfermann et al. | Jul 2009 | A1 |
| 20120206335 | Osterhout et al. | Aug 2012 | A1 |
| 20130103360 | Hsu | Apr 2013 | A1 |
| 20140247951 | Malaviya | Sep 2014 | A1 |
| 20140304891 | Waters | Oct 2014 | A1 |
| 20140313225 | Lee et al. | Oct 2014 | A1 |
| 20150054716 | Hirabayashi et al. | Feb 2015 | A1 |
| 20150279022 | Shuster et al. | Oct 2015 | A1 |
| 20150351896 | D'Lima et al. | Dec 2015 | A1 |
| 20160047746 | Tokhtuev et al. | Feb 2016 | A1 |
| 20160058090 | Leung | Mar 2016 | A1 |
| 20160063767 | Lee | Mar 2016 | A1 |
| 20160097721 | Bolduc et al. | Apr 2016 | A1 |
| 20160147086 | Cazalet | May 2016 | A1 |
| 20170019602 | Dopilka | Jan 2017 | A1 |
| 20170072316 | Finfter | Mar 2017 | A1 |
| 20170092235 | Osman et al. | Mar 2017 | A1 |
| 20170108697 | El-Ghoroury et al. | Apr 2017 | A1 |
| 20170116885 | Wilczek | Apr 2017 | A1 |
| 20170135315 | Marmen et al. | May 2017 | A1 |
| 20170153672 | Shin et al. | Jun 2017 | A1 |
| 20170263032 | Cricri | Sep 2017 | A1 |
| 20170289687 | Zamir | Oct 2017 | A1 |
| 20180001188 | Patel | Jan 2018 | A1 |
| 20180092601 | Wagner et al. | Apr 2018 | A1 |
| 20180181196 | Lee et al. | Jun 2018 | A1 |
| 20180253856 | Price et al. | Sep 2018 | A1 |
| 20180348812 | Miller | Dec 2018 | A1 |
| 20190068850 | Peper et al. | Feb 2019 | A1 |
| 20190320244 | Albers et al. | Oct 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 102014001041 | Jul 2015 | DE |
| 2241181 | Oct 2010 | EP |
| 2634617 | Sep 2013 | EP |
| 2016025443 | Feb 2016 | WO |
| Entry |
|---|
| Applicant-Initiated Interview Summary dated Dec. 16, 2019 for U.S. Appl. No. 16/178,384 of Peper, Jr. et al., filed Nov. 1, 2018. |
| Applicant-Initiated Interview Summary dated Dec. 9, 2019 for U.S. Appl. No. 16/030,672 of Peper, Jr. et al., filed Jul. 9, 2018. |
| International Search Report and Written Opinion dated Oct. 2, 2018 for PCT Application No. PCT/US18/42889 of Command Sight, Inc. |
| K-9 Storm Intruder, Real-Time Encrypted Camera; retrieved online on Jun. 21, 2018 from url:http://www.k9storm.com/intruder; 3 pages. |
| K9 Vision System; retrieved online on Jun. 21, 2018 from url: https://prok9supplies.com/k9-vision-system-for-sale.html; 10 pages. |
| Non-Final Office Action dated Jul. 19, 2019 for U.S. Appl. No. 16/030,672 of Peper, Jr. et al., filed Jul. 9, 2018. |
| Von-Final Office Action dated Oct. 11, 2019 for U.S. Appl. No. 16/178,384 of Peper, Jr. et al., filed Nov. 1, 2018. |
| Notice of Allowance dated Feb. 18, 2020 for U.S. Appl. No. 16/178,384 of Peper, Jr. et al., filed Nov. 1, 2018. |
| Notice of Allowance dated Feb. 4, 2020 for U.S. Appl. No. 16/030,672 of Peper, Jr. et al., filed Jul. 9, 2018. |
| WolfPAK K9 ISR—Advanced Canine (K9) Camera System; MissionTEQ; retrieved online on Jun. 21, 2018 from url:https://missionteq.com/canine-camera-system; 7 pages. |
| International Search Report and Written Opinion dated Jan. 18, 2019 for PCT Application No. US2018/058917 of Command Sight, Inc., filed on Nov. 2, 2018, 12 pages. |
| K9 Vision System, Full Version; https://prok9supplies.com/k9-vision-system-k9-vision-system-full-version-kvfv4.html, 4 pages. |
| K9 HELM head protection for working dogs; Trident K9 Tactical Helmet; retrieved online from url: www.k9helm.com/trident-helmet, 2018, 4 pages. |
| Williams, Fiona , et al., “Development of a Head-mounted, Eye-tracking System for Dogs”, Journal of Neuroscience, vol. 194, Issue 2; retrieved online from url: http://eprints.lincoln.ad.uk/3673/1/Development_of_a_head-mounted%2C_eye-tracking_system_for_dogs.pdf, Jan. 15, 2011, pp. 259-265. |
| European Search Report dated Mar. 29, 2021 for European Patent Application No. 18838050.5, 7 pages. |
| Supplementary European Search Report dated Aug. 11, 2021 for European Patent Application No. 18874198.7, 8 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20200228686 A1 | Jul 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62581501 | Nov 2017 | US | |
| 62537883 | Jul 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16178384 | Nov 2018 | US |
| Child | 16828842 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16030672 | Jul 2018 | US |
| Child | 16178384 | US |
