Patent Application
20220300079
The disclosure below relates to technically inventive, non-routine solutions that are necessarily rooted in computer technology and that produce concrete technical improvements. In particular, the disclosure below relates to techniques for using ultra-wideband (UWB) to identify and control another device.
As recognized herein, most modern electronic devices are not equipped with features that allow sufficient fine-grain location tracking indoors. As also recognized herein, it is often difficult and complex for users to navigate through multiple layers of on-screen menus to locate the controls for a given Internet of Things (IoT) device from among many IoT devices that might be available just to control the desired IoT device. There are currently no adequate solutions to the foregoing computer-related, technological problems.
Accordingly, in one aspect a first device includes at least one processor, an ultra-wideband (UWB) transceiver accessible to the at least one processor, an orientation sensor accessible to the at least one processor, and storage accessible to the at least one processor. The storage includes instructions executable by the at least one processor to transmit, using the UWB transceiver, a first UWB signal to a second device different from the first device. The instructions are also executable to receive, using the UWB transceiver, a second UWB signal from the second device in response to the first UWB signal. The instructions are then executable to determine a location of the second device based on the second UWB signal, receive input from the orientation sensor, and determine, based on the input, that the first device is pointing at the second device based on a predetermined axis of the first device. The instructions are further executable to receive a command at the first device to control the second device and, based on the command and the determination that the first device is pointing at the second device based on the predetermined axis of the first device, transmit the command to the second device.
In some example implementations, the command may be generated based on identification, by the first device, of the first device being gestured in the air. So, for example, the instructions may be executable to use the orientation sensor to identify the first device being gestured in the air.
Also, in some example implementations, the first device may include a display. The command may then be generated based on input to a graphical user interface (GUI) presented on the display, and/or based on receipt of a touch signal at the display at a display location that is not presenting a selector.
The command itself may be transmitted over a network that does not use UWB, and/or using the UWB transceiver. Also note that the orientation sensor may include a gyroscope in certain examples.
Still further, in some example implementations the instructions may be executable to broadcast, using the UWB transceiver, the first UWB signal to plural other devices different from the first device. The instructions may also be executable to receive, using the UWB transceiver, the second UWB signal from the second device in response to the first UWB signal and a third UWB signal from a third device in response to the first UWB signal. The third device may be different from the first and second devices. The instructions may then be executable to determine the location of the second device based on the second UWB signal and determine the location of the third device based on the third UWB signal. In these implementations, the instructions may then be executable to receive input from the orientation sensor and determine, based on the input and based on the locations of the second and third devices, that the first device is pointing at the second device based on the predetermined axis of the first device. The instructions may then be executable to receive the command at the first device to control the second device and, based on the command and the determination that the first device is pointing at the second device based on the predetermined axis of the first device, transmit the command to the second device.
Also note that in some example embodiments, the first signal may be transmitted responsive to an identification of a change in the orientation of the first device as identified via the orientation sensor. Additionally, or alternatively, the first signal may be transmitted responsive to touch input to a display on the first device and/or responsive to input to illuminate the display.
In another aspect, a method includes transmitting, at a first device and using an ultra-wideband (UWB) transceiver, a first UWB signal to a second device different from the first device. The method also includes receiving, using the UWB transceiver, a second UWB signal from the second device generated in response to the first UWB signal. The method then includes determining, at the first device, a location of the second device based on the second UWB signal. Thereafter, the method includes receiving input from an inertial sensor on the first device and determining, at the first device based on the input, that the first device is pointing toward the second device. The method then includes receiving a command at the first device to control the second device and, based on the command and the determination that the first device is pointing toward the second device, transmitting the command to the second device.
In some examples, the method may include determining that the first device is pointing toward the second device based on a longitudinal axis of the first device being oriented toward the second device.
Also, in various example implementations, the command itself may be generated based on the first device being pointed toward the second device and/or based on input to a graphical user interface (GUI) presented on a display, where the GUI may be presented by the first device responsive to the first device determining that the first device is pointing toward the second device.
Also, in certain examples the command may be identified at the first device based on identification of the first device as being gestured in free space in a bi-directional manner, and the command may be an on/off command. Additionally, or alternatively, the command may be identified at the first device based on identification of the first device as being gestured, relative to a longitudinal axis, in a clockwise or counterclockwise manner, where the command may be to adjust a parameter along a scale.
Still further, in some examples the command may be transmitted based on permissions being set for a user of the first device to control the second device.
In still another aspect, at least one computer readable storage medium (CRSM) that is not a transitory signal includes instructions executable by at least one processor to transmit, from a first device and using an ultra-wideband (UWB) transceiver, a first UWB signal to a second device different from the first device. The instructions are also executable to receive, using the UWB transceiver, a second UWB signal from the second device in response to the first UWB signal. The instructions are then executable to determine, using the first device, a location of the second device based on the second UWB signal. The instructions are further executable to determine that the first device is oriented toward the second device. The instructions are then executable to receive a command at the first device to control the second device and, based on the command and the determination that the first device is oriented toward the second device, transmit the command to the second device.
In some example implementations, the second signal may indicate a device type associated with the second device, a current device state associated with the second device, and/or an indication of one or more commands that can be issued to the second device. Thus, in these implementations the command may be issued based on at least one thing that the second signal indicates.
The details of present principles, both as to their structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which:
Among other things, the detailed description below relates to use of UWB for location and direction tracking between two devices with high accuracy. A user may thus control smart home devices by pointing a UWB-enabled controlling device (such as a phone or fob) at a lamp or other smart home device to control it based on its location as known via UWB. Thus, the identification of the other device may be made through UWB detection and use of gyroscopic information to help identify the pointing to translate the user's intention for how to change the device.
In the case of a toggle device such as a lamp, pointing at the device may be used to toggle the state of the device (off to on, on to off). For more complex devices with multiple settings, the pointing action may be followed by a directional gesture (such as up, down, clockwise, counterclockwise) to communicate a command.
In addition, in some examples control of devices could be restricted by person, time, day, or other factors. For example, only adults might be allowed to adjust the thermostat, but all family members may be allowed to turn lights on or off.
In some examples, the controlling device can interface with Wi-Fi-based smart home controls/protocols to issue the command. In other examples, the controlling device may transmit the command via UWB.
In any case, in various implementations at least two devices may be at play—a “transmitting device” Tx and a “receiving device” Rx. The Rx may simply have a UWB “tag” which when activated may respond to the Tx. This response could be a simple acknowledgment or include more-detailed information related to the Rx such as device type, current state (e.g., on/off), and/or a description of allowed fields and commands. The Tx may then process the information from the Rx and use what is known about the Rx's location (as determined via UWB) and other Rx info to implement a control action. Again, the control action may be relayed via Wi-Fi-based control or another protocol already being used to control the Rx, or UWB-based communication may be used to issue the control action to bypass the Wi-Fi mechanism.
Prior to delving further into the details of the instant techniques, note with respect to any computer systems discussed herein that a system may include server and client components, connected over a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices including televisions (e.g., smart TVs, Internet-enabled TVs), computers such as desktops, laptops and tablet computers, so-called convertible devices (e.g., having a tablet configuration and laptop configuration), and other mobile devices including smart phones. These client devices may employ, as non-limiting examples, operating systems from Apple Inc. of Cupertino Calif., Google Inc. of Mountain View, Calif., or Microsoft Corp. of Redmond, Wash. A Unix® or similar such as Linux® operating system may be used. These operating systems can execute one or more browsers such as a browser made by Microsoft or Google or Mozilla or another browser program that can access web pages and applications hosted by Internet servers over a network such as the Internet, a local intranet, or a virtual private network.
As used herein, instructions refer to computer-implemented steps for processing information in the system. Instructions can be implemented in software, firmware or hardware, or combinations thereof and include any type of programmed step undertaken by components of the system; hence, illustrative components, blocks, modules, circuits, and steps are sometimes set forth in terms of their functionality.
A processor may be any general-purpose single- or multi-chip processor that can execute logic by means of various lines such as address lines, data lines, and control lines and registers and shift registers. Moreover, any logical blocks, modules, and circuits described herein can be implemented or performed with a general-purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic device such as an application specific integrated circuit (ASIC), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor can also be implemented by a controller or state machine or a combination of computing devices. Thus, the methods herein may be implemented as software instructions executed by a processor, suitably configured application specific integrated circuits (ASIC) or field programmable gate array (FPGA) modules, or any other convenient manner as would be appreciated by those skilled in those art. Where employed, the software instructions may also be embodied in a non-transitory device that is being vended and/or provided that is not a transitory, propagating signal and/or a signal per se (such as a hard disk drive, CD ROM or Flash drive). The software code instructions may also be downloaded over the Internet. Accordingly, it is to be understood that although a software application for undertaking present principles may be vended with a device such as the system 100 described below, such an application may also be downloaded from a server to a device over a network such as the Internet.
Software modules and/or applications described by way of flow charts and/or user interfaces herein can include various sub-routines, procedures, etc. Without limiting the disclosure, logic stated to be executed by a particular module can be redistributed to other software modules and/or combined together in a single module and/or made available in a shareable library.
Logic when implemented in software, can be written in an appropriate language such as but not limited to hypertext markup language (HTML)-5, Java/JavaScript, C# or C++, and can be stored on or transmitted from a computer-readable storage medium such as a random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), a hard disk drive or solid state drive, compact disk read-only memory (CD-ROM) or other optical disk storage such as digital versatile disc (DVD), magnetic disk storage or other magnetic storage devices including removable thumb drives, etc.
In an example, a processor can access information over its input lines from data storage, such as the computer readable storage medium, and/or the processor can access information wirelessly from an Internet server by activating a wireless transceiver to send and receive data. Data typically is converted from analog signals to digital by circuitry between the antenna and the registers of the processor when being received and from digital to analog when being transmitted. The processor then processes the data through its shift registers to output calculated data on output lines, for presentation of the calculated data on the device.
Components included in one embodiment can be used in other embodiments in any appropriate combination. For example, any of the various components described herein and/or depicted in the Figures may be combined, interchanged or excluded from other embodiments.
“A system having at least one of A, B, and C” (likewise “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.
The term “circuit” or “circuitry” may be used in the summary, description, and/or claims. As is well known in the art, the term “circuitry” includes all levels of available integration, e.g., from discrete logic circuits to the highest level of circuit integration such as VLSI and includes programmable logic components programmed to perform the functions of an embodiment as well as general-purpose or special-purpose processors programmed with instructions to perform those functions.
Now specifically in reference to
As shown in
In the example of
The core and memory control group 120 include one or more processors 122 (e.g., single core or multi-core, etc.) and a memory controller hub 126 that exchange information via a front side bus (FSB) 124. As described herein, various components of the core and memory control group 120 may be integrated onto a single processor die, for example, to make a chip that supplants the “northbridge” style architecture.
The memory controller hub 126 interfaces with memory 140. For example, the memory controller hub 126 may provide support for DDR SDRAM memory (e.g., DDR, DDR2, DDR3, etc.). In general, the memory 140 is a type of random-access memory (RAM). It is often referred to as “system memory.”
The memory controller hub 126 can further include a low-voltage differential signaling interface (LVDS) 132. The LVDS 132 may be a so-called LVDS Display Interface (LDI) for support of a display device 192 (e.g., a CRT, a flat panel, a projector, a touch-enabled light emitting diode display or other video display, etc.). A block 138 includes some examples of technologies that may be supported via the LVDS interface 132 (e.g., serial digital video, HDMI/DVI, display port). The memory controller hub 126 also includes one or more PCI-express interfaces (PCI-E) 134, for example, for support of discrete graphics 136. Discrete graphics using a PCI-E interface has become an alternative approach to an accelerated graphics port (AGP). For example, the memory controller hub 126 may include a 16-lane (x16) PCI-E port for an external PCI-E-based graphics card (including, e.g., one of more GPUs). An example system may include AGP or PCI-E for support of graphics.
In examples in which it is used, the I/O hub controller 150 can include a variety of interfaces. The example of
The interfaces of the I/O hub controller 150 may provide for communication with various devices, networks, etc. For example, where used, the SATA interface 151 provides for reading, writing, or reading and writing information on one or more drives 180 such as HDDs, SDDs or a combination thereof, but in any case, the drives 180 are understood to be, e.g., tangible computer readable storage mediums that are not transitory, propagating signals. The I/O hub controller 150 may also include an advanced host controller interface (AHCI) to support one or more drives 180. The PCI-E interface 152 allows for wireless connections 182 to devices, networks, etc. The USB interface 153 provides for input devices 184 such as keyboards (KB), mice and various other devices (e.g., cameras, phones, storage, media players, etc.).
In the example of
The system 100, upon power on, may be configured to execute boot code 190 for the BIOS 168, as stored within the SPI Flash 166, and thereafter processes data under the control of one or more operating systems and application software (e.g., stored in system memory 140). An operating system may be stored in any of a variety of locations and accessed, for example, according to instructions of the BIOS 168.
Additionally, the system 100 may include an ultra-wideband (UWB) transceiver 191 configured to transmit and receive data using UWB signals and UWB communication protocol(s), such as protocols set forth by the FiRa Consortium. As understood herein, UWB may use low energy, short-range, high-bandwidth pulse communication over a relatively large portion of the radio spectrum. Thus, for example, an ultra-wideband signal/pulse may be established by a radio signal with fractional bandwidth greater than 20% and/or a bandwidth greater than 500 MHz. UWB communication may occur by using multiple frequencies (e.g., concurrently) in the frequency range from 3.1 to 10.6 GHz in certain examples.
To transmit UWB signals consistent with present principles, the transceiver 191 itself may include one or more Vivaldi antennas and/or a MIMO (multiple-input and multiple-output) distributed antenna system, for example. It is to be further understood that various UWB algorithms, time difference of arrival (TDoA) algorithms, and/or angle of arrival (AoA) algorithms may be used the system 100 to determine the distance to and location of another UWB transceiver on another device that is in communication with the UWB transceiver on the system 100.
Still in reference to
Still further, though not shown for simplicity, in some examples the system 100 may include an audio receiver/microphone that provides input from the microphone to the processor 122 based on audio that is detected, such as via a user providing audible input to the microphone. The system 100 may also include a camera that gathers one or more images and provides the images and related input to the processor 122. The camera may be a thermal imaging camera, an infrared (IR) camera, a digital camera such as a webcam, a three-dimensional (3D) camera, and/or a camera otherwise integrated into the system 100 and controllable by the processor 122 to gather still images and/or video. Also, the system 100 may include a global positioning system (GPS) transceiver that is configured to communicate with at least one satellite to receive/identify geographic position information and provide the geographic position information to the processor 122.
It is to be understood that an example client device or other machine/computer may include fewer or more features than shown on the system 100 of
Turning now to
Now in reference to
Based on the inference of user intent, the smartphone 302 may then take one or more actions in conformance with further disclosure below, such as turning the lamp 304 on or off based on the pointing alone (possibly without the display on the smartphone 302 even being illuminated). Additionally, or alternatively, the smartphone 302 may turn the light on or off based on additional input to a graphical user interface (GUI) presented on a display of the smartphone 302.
As another example, gestures of the smartphone 302 in the air may be identified by an inertial sensor in the smartphone 302 to identify the first device as being gestured in free space in a bi-directional manner (e.g., up and down as indicated by the respective arrows 308, 310) while the smartphone 302 is still pointed toward the lamp 304 in order to turn the light bulb for the lamp 304 on or off (e.g., up or down gesture, respectively). Or if the user wished to adjust the brightness level of the bulb inside the lamp 304 while it is illuminated, the user may gesture the smartphone 302 in free space in a clockwise or counterclockwise manner around the longitudinal axis 306 while the smartphone 302 is still pointed toward the lamp 304 in order to adjust the bulb's brightness up or down along a brightness scale for the bulb.
As one more example, if the smart IoT device were a smart Bluetooth speaker rather than the lamp 304, a clockwise or counterclockwise gesture of the smartphone 302 in free space may adjust another parameter along a scale, such as volume level along a volume scale or treble level along a treble scale.
Continuing the detailed description in reference to
Beginning at block 400, the first device may identify a trigger to transmit/broadcast a UWB signal from its UWB transceiver. The trigger may be established by, for example, identification by the first device of movement of the first device and/or a change in the orientation of the first device as identified via its orientation sensor or another inertial sensor. As another example, the trigger may be established by identification of touch input to the first device's display (e.g., even if the display is not illuminated, as still sensed by the display's touch sensor(s)). As yet another example, the trigger may be established by identification of input to a power button on the first device and/or input to illuminate the first device's display. In this way, preliminary user action upon the first device may trigger UWB signal exchange with one or more other UWB-enabled devices so that the other devices and their directions may be identified in a seamless manner, so the first device is ready for ensuing command input to control those other devices whenever the user desires.
From block 400 the logic may then proceed to block 402. At block 402 the first device may transmit at least a first UWB signal to a second device (e.g., responsive to the trigger). In some examples, at block 402 the first device may broadcast the first UWB signal no more than a predetermined distance or radius from the first device by controlling the intensity with which its UWB signals are transmitted to provide a sort of UWB-fencing. In this way, all other devices within the predetermined distance may respond with their own respective UWB signals, but also the user may not unintentionally point at and control another UWB-enabled device beyond their field of view or object of focus that happens to also respond.
From block 402 the logic may then proceed to block 404. At block 404 the first device may, in response to the first UWB signal, receive back at least a second UWB signal(s) from a second device and possibly receive back other UWB signals from still other devices that also responded to the first UWB signal as they might be preconfigured to do. The second UWB signal (and any other UWB signals from any other responding devices) may be a simple acknowledgment from the second device. Additionally, or alternatively, the second signal may include additional data stored at the second device such as a device type associated with the second device (e.g., a lamp in the example of
The logic of
From block 406 the logic may then proceed to block 408 where the first device may receive input from its orientation or other inertial sensor, such as its gyroscope. Then at block 410 the first device may execute one or more gyroscope input processing algorithms to determine an orientation of the first device (e.g., relative to earth) so that, already knowing its longitudinal axis or another predetermined axis via preprogramming and also having already identified the locations of the other respective devices, the first device may determine that it is pointed/oriented along the axis toward the second device. Also note again that orientation of the first device may also be determined using the first device's UWB transceiver and UWB location tracking.
After block 410 the logic may proceed to block 412 where the first device may receive a command to control the second device. The command may be established in a number of different ways. For example, in some implementations the act of pointing the first device toward the second device may establish a toggle on/off command so that if the second device were on, pointing at the second device would turn the device off, and vice versa, without any additional input from the user. In other examples, touch sensors along the bezel of the first device may also be used to identify the device as being held along with using the device being pointed toward the second device to establish the toggle on/off command in order to further reduce the chance of erroneous commands being provided (e.g., when the device is not being held but might still be pointed toward the second device).
The command may also be established by the end-user gesturing the first device in the air as set forth above with respect to
As yet another example, the command received at block 412 may be established by touch input to the first device's display at a display location that is not presenting a visual selector of any kind that might be selectable by a user, such as an icon or hyperlink. Thus, in this example touch input to the display may establish the command even if none of the display were illuminated so long as the display's touch sensors are still active.
As but one more example, the command received at block 412 may be established by touch input to a visual selector presented as part of a graphical user interface presented on the first device's illuminated display. In some examples, the GUI may even be presented responsive to the first device determining that it is pointing toward the second device so that the user does not have to navigate other complex and cumbersome menus to reach the appropriate GUI controls for the second device (or other device at which the first device might be pointed).
From block 412 the logic may proceed to block 414. At block 414 the first device may, based on the command and the determination that the first device is pointing at the second device, transmit the command to the second device. The command may be transmitted over a network that does not use UWB, such as a Wi-Fi network, LAN, WAN, the Internet, or Bluetooth network. In such an instance, the command may be transmitted using predetermined communication protocols for controlling the second device, such as a predetermined Nest, Hue, or Nexia IoT device management protocol.
However, in other examples the command may be transmitted at block 414 via UWB using the first device's UWB transceiver. This may be possible in embodiments where the second device has already reported its device type, state, and/or available commands to the first device via the second signal(s).
Now in reference to
As shown in
However, once the first device is in fact pointed toward another UWB-based device,
Continuing the detailed description in reference to
Before describing each of the GUIs 700, 800 individually, further note that these GUIs may be combined into a single GUI in some examples. Or in other examples, the first device may present the GUI 700 based on the other device reporting via UWB its device state as currently off, while presenting the GUI 800 based on the other device reporting via UWB its device state as currently on. In this example use case, the other device will again be a smart IoT lamp such as the lamp 304.
Now in reference to the GUI 700 of
The predetermined activation time may be a threshold amount of time after the GUI 700 is presented, and/or after the first device determines it is pointed at another device, during which gestures in free space may be used to control the other device. The predetermined activation time may therefore be used so that additional gestures or even unintentional movement occurring beyond the activation time but while the first device is still pointed toward the other device will not result in an unintended command being sent to the other device. So, in examples where the activation time is used but has expired in a given instance, the user may point the device away from the other device and then back to the other device again to restart the activation time.
Now in reference to the GUI 800 of
Now describing
Beginning first with the option 902, it may be selected to set or enable the device to undertake present principles in the future. For example, the option 902 may be selected to set or configure the device to enable UWB control of IoT devices. E.g., selection of the option 902 may set or enable the device to execute the logic of
The GUI 900 may also include options 904-910 that may be respectively selectable to enable control of other IoT devices from the device of
As also shown in
Also note that while permissions are shown for types of end-users and/or particular end-users that may control various devices, permissions may be established based on other factors as well (such as time of day or date) but have been omitted from
Now in reference to
It may now be appreciated that present principles provide for an improved computer-based user interface that increases the functionality and ease of use of the devices disclosed herein. The disclosed concepts are rooted in computer technology for computers to carry out their functions.
It is to be understood that whilst present principals have been described with reference to some example embodiments, these are not intended to be limiting, and that various alternative arrangements may be used to implement the subject matter claimed herein. Components included in one embodiment can be used in other embodiments in any appropriate combination. For example, any of the various components described herein and/or depicted in the Figures may be combined, interchanged or excluded from other embodiments.
