Patent Grant
10414052
This application relates generally to robots, and in particular to robots configured for operation in commercial and industrial settings.
Robots are increasingly prevalent in society, particularly in entertainment and manufacturing contexts. However, the adoption of robots in these contexts has outpaced settings where frequent robot/human interactions are required. Examples of such settings include retail environments, security environments, and the like. The abilities of robots, for instance, the ability to communicate with building and object infrastructure, creates an unaddressed demand for the implementation of robots in commercial and industrial settings.
A mobile robot (“robot” hereinafter) configured for operation in commercial and industrial settings is described herein. The robot can perform a number of functions and operations in a variety of categories, including but not limited to security operations, infrastructure or maintenance operations, navigation or mapping operations, inventory management operations, and robot/human interaction operations.
In some embodiments, the robot can primarily function as a security-oriented system. The robot can patrol routes within a building (such as a workplace, store, or warehouse), and can identify suspicious activity or individuals, detect violations of security policies, and detect security anomalies. For instance, the robot can identify unauthorized individuals within a restricted area, or can identify individuals perform an unauthorized action. Likewise, the robot can identify a building object or system indicative of a potential security risk, such as an unlocked door or open window. The robot can coordinate with a central system of the building, with building infrastructure and security systems, with other robots, and with individuals (such as security personnel and a robot operator).
In response to detecting a potential security risk or a security policy violation, the robot can perform one or more security operations. For instance, the robot can attempt to obstruct an individual attempting to access an off-limits portion of a building, can monitor or follow the individual, can record video of the individual, and can inform security personnel of the individual. Likewise, the robot can perform a corrective security operation, for instance by locking a door left unlocked or by relocating secure materials left in an unauthorized location. In some embodiments, the robot can couple with a building infrastructure system or security system to perform a security operation. For instance, the robot can instruct one or more security cameras associated with a location of a security anomaly to record video, or can instruct smart doors within a proximity of a suspicious individual to lock, preventing the individual from leaving the area.
The robot can provide information to a central system of a building, for instance information identifying a location of the robot, a field of view of the robot, video captured by the robot, locations of individuals and objects detected by the robot, and the like. The central system can display the information received from the robot in an interface in conjunction with information received from other robots, infrastructure systems, and security systems. In some embodiments, the central system displays a map identifying a location of the robot, a location of other infrastructure and security systems, and a location of all detected individuals and objects. The central system can further display information associated with detected security violations, information describing a robot state, functions available to a user of the central system to perform, and a patrol route of the robot.
In some embodiments, the robot can generate or update a semantic map associated with a location and identifying one or more of: locations of objects, an identity of the objects, a state of the objects, and other characteristics of the objects. The semantic map can be generated by the robot by using a first set of sensors to identify obstacles and pathways within the location, and by using a second set of sensors to identify objects within the location, object locations, object states, and object characteristics. The semantic map can be used by the robot to perform various operations, such as security operations (for instance, identify an object as a potential security policy violation by determining if a detected object state varies from a previous detection of the object state), navigation operations, and the like. Likewise, the semantic map can be provided to a central system for use by or display to a user of the central system.
The robot may be equipped with RFID reader antennas configured to detected RFID tags within a location. For instance, the robot can navigate within a retail store, and can scan the store for RFID tags while navigating. In some embodiments, the robot can determine a location of detected RFID tags, for instance by detecting the RFID tags from a number of different locations, or by using an onboard direction of arrival antenna array. The robot can perform various functions based on the detected RFID tags, for instance updating a store inventory database to reflect the detected RFID tag or the location of the detected RFID tag, by providing the location of the object coupled to the detected RFID tag to a requesting customer, or by flagging the object coupled to the detected RFID tag to a store employee for relocation (for instance, in response to determining that the object is in a location other than the proper or historical location of the object).
The robot can include various hardware components configured to enable the performance of the security operations, infrastructure or maintenance operations, navigation or mapping operations, inventory management operations, robot/human interaction operations, and other operations described herein. The robot can include a navigation system and power source configured to enable the robot to move within a location. The robot can include various indicators or interaction mechanisms configured to communication information to or receive an input from a human. The robot can include various sensors, such as cameras, motion detectors, audio detectors, rangefinders, depth sensors, and the like to enable the robot to recognize and process the environment in which the robot operators. The robot can also include various transceiver components, such as a WiFi antenna, cellular or LTE antenna, RFID reader antennas, and the like.
The robot can further include a fabric housing coupled around a body of the robot. The fabric housing can be removable, and can enable sensors within the robot body to detect or capture signals that propagate through the fabric. The fabric can be a low-cost component relative to a hardened, plastic, or other solid exterior, beneficially enabling easy replacement of the fabric housing in the event of damage. Further, the fabric housing can enable the flow of air through the housing, beneficially aiding in the cooling of interior components of the robot.
The figures depict various embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.
System Overview
The robot 100 includes exterior fabric panels, as described below in greater detail. The fabric panels can be removably or fixedly coupled around a housing body of the robot. The robot also includes a plurality of sensors and other components enclosed within the housing body, described below in greater detail. A portion of the sensors can be located within the body of the robot, for instance behind or underneath the fabric panels such that the sensors can receive signals that propagate through the fabric panels. The structure, components, and functionality of the robot are described below in greater detail. It should be noted that in other embodiments, the robot can include different form factors, structures, components, and functionalities than those described herein.
The environment of
The central system 210 can be a central server or other computing system configured to provide instructions to the robots, to receive information from the robots, to access data corresponding to the robots or the other components of
The infrastructure system 220 (or infrastructure systems) can include one or more components of a building or environment in which the robots 100 are located. In some embodiments, the infrastructure systems are communicatively coupled to the network 200 or are otherwise capable of communication with the robots. Examples of system infrastructure systems include smart doors and windows, elevators, routers and other network components, power systems, smart lights, smart fire alarms and other sensors, connected furnace or air conditioning systems, “Internet of things” or IoT systems, and the like. In some embodiments, the infrastructure systems aren't capable of communication with via the network 200 or otherwise, for instance disconnected doors and windows, disconnected elevators, thermal vents, and the like. The infrastructure systems of
The security system 230 (or security systems) can also include one or more security components of a building or environment in which the robots 100 are located. Examples of security systems can include camera systems, access gates, door or window locks, audio detectors, motion detectors, security interfaces or displays, and the like. The security systems can be configured to communicate with the network 200 and/or robots, or may be disconnected systems (alarms that do not communicate and that must be manually reset). The security systems of
The remote access interface 240 is a computing system, such as a computer, a server, a phone or other mobile device, or any other system that enables a use to remotely access the network 200, and via the network, to interact with one or more other components of the environment of
The security personnel 250 can include guards located at guard stations, patrolling guards, access gate guards, or remote security personnel. For instance, the security personnel can include one or more robot operators configured to monitor and control the actions or movements of the robots 100. Robot operators can interact with robots via the central system 210 or via the remote access interface 240, and also direct other security personnel as needed (for instance, by instructing security personnel to investigation a security violation detected by the robots).
The network 200 may be the Internet, a local area network (such as a building-wide or site-wide network), a wireless network, a mesh network, or any other network or combination thereof that allows communication between the components of
The communication interface 310 is configured to provide a communicative interface between the components of the central system 210 and the other components of
The user interface 312 is configured to enable a user of the central system 210 (such as a security personnel 250 or a robot operator) to interact with the central system. For instance, the user interface can include a graphical user interface (or GUI) displayed on hardware corresponding to the central system. The GUI can include video streams from one or more security cameras; can include a location of one or more robots 100 or objects overlaid on a map of a building, floor, or site; can include data corresponding to one or more infrastructure systems; and the like. The user interface can enable a user of the central system to provide one or more instructions or commands or otherwise interact with one or more components of the environment of
The robot interface 314 is configured to enable the central system 210 (or a user of the central system) to interact with one or more robots 100. For instance, the robot interface can receive data from the one or more robots 100 via the communication interface 310 for display on the user interface 312. The robot interface can also provide data to the one or more robots via the communication interface, for instance in response to a command received from a user via the user interface. The robot interface can format data provided to and received from the robots into a format compliant with the robots and central system, respectively. The data received from, the instructions provided to, and the functions of the robots are described below in greater detail.
The infrastructure interface 316 is configured to enable the central system 210 (or a user of the central system) to interact with one or more infrastructure systems 220 via the communication interface 310. For instance, the infrastructure interface can lock one or more doors within a building, can cut power to a portion of a floor, and store received inventory information. Likewise, the security engine 318 is configured to enable the central system or a user of the central system to interact with one or more security systems 230 via the communication interface. For instance, the security engine can direct video cameras to a location within a building and receive footage from the cameras, can receive badge data from badge readers, and direct security personnel 250 to a building portion where motion was detected. The security engine can also implement one or more security policies or security operations, either independently from or at the direction of a user of the central system. Security policies and operations are described below in greater detail.
The security policy storage module 340 stores one or more security policies for implementation by the central system 210. A security policy can identify:
The semantic maps storage module 342 can store one or more semantic maps, for instance semantic maps generated by robots 100. The user interface 312 can access and display one or more semantic maps to a user of the central system 210, and one or more components of the central system can update the displayed semantic map for subsequent storage in the semantic maps storage module in response to receiving information from one or more robots, security personnel 250, or any other component of
The inventory storage module 344 stores information associated with objects within a building, within a floor, within a room or store, and the like. For instance, the inventory storage module can include a database detailing types of inventory, stock levels associated with each inventory type, and a location of each inventory object within a building. The inventory storage module can be updated, for instance, in response to receiving information from a robot 100 indicating that the location of one or more objects within a store has changed, that a missing object has been found, or that objects have been misplaced or lost. It should be noted that the inventory module can store locations, numbers, and types of objects in addition to retail objects or stock, for instance lab equipment, books or other reference materials, and the like. The inventory storage module can additionally store a status for objects (for instance, “misplaced”, “lost”, “stolen”, “broken”, “operational”, and the like).
The doors 410 and windows 412 of a building can be standard, disconnected, and/or non-electronic doors and windows. Alternatively, the doors and windows can be electronic and/or connected to other building infrastructure, for instance via the network 200. In embodiments where the doors and windows are electronic, a user of the central system 210, a robot 100, or a security personnel 250 can close, open, lock, or unlock the doors and windows, for instance manually (by the security personnel or robot) or electronically (by directly interfacing with the electronic subsystems of the doors and windows, or by providing instructions to the doors and windows via the network).
The elevators 414 can include one or more elevator within a building. The elevators can be connected to one or more components of the environment of
The lights 416 can include various lighting systems within a building, for instance lights within particular rooms, on particular floors, or building-wide. The lights can be connected to other building systems, for instance via the network 200, and can be remotely activated or deactivated (for instance by a user of the central system 210 or a robot 100) or manually activated or deactivated (for instance by security personal 250 or a robot toggling a light switch or interface).
The network 418 can include networking infrastructure within a building or portion of a building, or across multiple buildings or locations. The network can include standard networking systems, such as routers, switches, severs, transceivers, and the like. The network can be connected to the other components of
The sensors 420 can include fire alarms, smoke detectors, carbon monoxide detects, gas or toxin detectors, thermometers, and the like. Information from the sensors can be provided to the central system 210, for instance directly via the network 200 or via a robot 100.
The temperature system 422 can include a furnace, air conditioning unit, fans or other air flow systems, thermal ductwork and corresponding control mechanisms (such as valves), and other systems associated with building temperature. Instructions can be provided to the temperature system, for instance from a user of the central system 210 via the network 200 or from a robot 100 (either via the network or by directly connecting to the temperature system). Example instructions include but are not limited to lowering the temperature or raising the temperature for an entire building or for building portions, increasing air flow, controlling one or more duct valves, and the like. Information can also be received from the temperature system by the central system, a robot, or by another component of
The power system 424 can include power infrastructure associated with a building. For instance, the power system can include one or more generators, feeds from an external power source, solar panels, breakers, power distribution units, power lines within and external to a building, meter boxes, robot recharge stations, vehicle recharge stations, and the like. In some embodiments, the power system is communicatively coupled to one or more components of
The inventory 426 includes objects within a building, portion of a building, store, room, or other area. Examples of inventor include store inventory (objects for sale within the store), building equipment (such as lab equipment, computer equipment, and the like), sensitive objects (such as high security objects, expensive objects, and the like), vehicles, and the like. The inventory of a building can include tags or other receivers/transmitters (such as RFID tags) that can identify the object to which the tag is affixed to a system capable of reading the tag. Each object can include a unique identifier that is transmitted to a reader system, and, as described below, a robot 100 equipped with one or more RFID readers can identify and locate inventory within a building or building portion by navigating within a proximity to the corresponding object.
The cameras 510 can be located within or external to a building, for instance within high-trafficked passages, high-security rooms or locations, locations corresponding to advantageous vantage points, and the like. In some embodiments, the field of view of a camera is fixed, while in other embodiments, the cameras can rotate, pan, tilt, or track objects or individuals, for instance in response to an instruction from a user of the central system 210 or in response to the detection of a suspicious noise or detected motion. The cameras can record video for storage and subsequent viewing or processing, or can stream the video to the central system or a robot 100. In some embodiments, the cameras can identify objects or individuals within video captured by the cameras and can provide this information to other components of
The access gates 512 can include physical barriers (such as barrier arms, walls, doors, and the like) configured to prevent or inhibit the movement of individuals, vehicles, or other objects. The access gates can allow individuals or vehicles to pass through in response to verifying that the individuals or vehicles are authorized to pass, for instance in response to verifying an individual's or vehicle's credentials or access permissions. In some embodiments, a robot 100, a user of the central system 210, or the central system itself can activate an access gate to allow or prevent individuals or vehicles from passing through, for instance in response to an instruction from the user or robot, or in response to a detected security violation or suspicious activity elsewhere within a vicinity of the access gate.
The locks 514 can be located within doors, windows, cabinets, cupboards, safes, or within any other suitable building infrastructure or object. The locks can be activated or deactivated by instructions from a robot 100, a user of the central system 210, or by the central system itself, for instance via the network 200. The locks can also be activated in response to the occurrence of a security violation or suspicious activity within a proximity of the locks.
The audio detector 516 is configured to detect audio signals, and to provide information describing the detected audio to one or more components of the environment of
The badge readers 520 are configured to detect ID badges of individuals, to authenticate the identity, access credentials, or permissions of the individuals. The badge readers can provide information associated with the detected badges to components of the environment of
The aerial systems 522 include one or more unmanned aerial vehicles (UAVs) configured to fly within a space (such as a building floor, parking lot, and the like). The aerial systems can receive instructions from components of
The security displays 524 are configured to display security information to individuals, such as a security status of a building or building portion, indications of security violations, and access credentials required to be in a particular location. The displayed security information can be received from the central system 210, from a robot 100, or from any other component of
Mobile Robot Housing
The fabric panel 600 can be made of any type of fabric material, for instance cotton, nylon, polyester, vinyl, velvet, or any other suitable or stretchable fabric material or blend/combination thereof. By stretching, the fabric panel of
The fabric panel 600 can include a fabric mesh material, allowing for the flow of air into and out of the robot 100, beneficially enabling the cooling of components of the robot interior by allowing for the transfer of heat out of the robot and through the fabric. Further, fabric may be semi-transparent or translucent, allowing for cameras, motion sensors, laser sensors, or other components within the robot to at least partially see through the fabric. Finally, the fabric may allow sound waves to pass through the fabric, allowing microphones or other audio components to capture sound waves passing through the fabric, and allowing speakers within the robot to transmit sound through the fabric.
The fabric panel 600 can couple to the robot 100 in a number of ways. In the embodiment of
In some embodiments, the coupling means 620B and 622B include holes within the fabric of the fabric panel 600 and grommets located around the holes. Likewise, in some embodiments, the coupling means 620A and 622A include pins, latches, or other securing protrusions configured to reciprocally couple with the holes of the couplings means 620B and 622B when the holes are placed over the coupling means 620A and 620B. In other embodiments, the coupling means include buttons, zippers, Velcro, levers, buckles, or any other suitable means enabling the fabric panel to couple to the robot 100. In some embodiments, when the coupling means 620B are coupled to the coupling means 620A, and when the coupling means 622B are coupled to the coupling means when the fabric panel is coupled to the robot, the fabric of the panel is stretched and held in partial tension along the axis of the fabric panel from the opening 624 to the opening 626. The coupling means can be located on an inside surface of the fabric panel, beneficially enabling the fabric panel to couple to the robot 100 without the coupling means being visible (e.g., the coupling means may not include holes within the fabric panel, allowing for a seamless/holeless fabric exterior). In some embodiments, the coupling means include elastic bands (for instance, around an inside surface of the fabric panel adjacent to or within a threshold distance of the openings 624 and 626) that secure the fabric panel to the robot by tightening around reciprocal grooves within the head, body, and/or base. Such elastic bands can beneficially improve the ease of coupling the fabric panel to the robot, and can enable an installer to quickly align the fabric panel on the robot during installation by allowing the installer to move the fabric panel over the robot until the elastic bands snap into the reciprocal grooves.
While the fabric panel 600 covers certain sensors of the robot 100 when the fabric panel is coupled to the robot, such as the sensor array 612, the fabric panel can include one or more openings enabling other components of the robot to be exposed/not covered by the fabric. For example, the opening 628 corresponds to the sensor 610 such that when the fabric panel is coupled to the robot, the opening 628 is secured around the sensor 610 thereby exposing the sensor. Although only one opening is shown within the side of the fabric panel in the embodiment of
Although the fabric panel 600 of the embodiment of
In some embodiments, instead of one fabric panel 600 that secures around an entire exterior of the body 604 of the robot 100, several fabric panels can each be secured to a portion of a robot exterior. For instance, a first panel can secure to a left side exterior of the robot and a second panel can secure to a right side exterior of the robot. In such embodiments, each fabric panel can include an outside ring structure located around an outside perimeter of the panel.
The fabric panel 600 illustrated in
Mobile Robot Architecture
The navigation system 710 is configured to move the robot 100 around a space, such as a store, a room, a floor, a building, a parking lot, or any indoor or outdoor space. The navigation system can include a motor (such as an electric motor), an engine, or any other systems capable of moving the robot. The navigation system can also include wheels, rollers, and the like configured to enable the robot to smoothly move. The navigation system can move the robot in a number of directions, for instance, forwards, backwards, or sideways. In some embodiments, the base of the robot is holonomic, enabling the robot to move in any direction. In some embodiments, the navigation system can rotate the robot clockwise or counterclockwise, for instance as much as 360 degrees. In some embodiments, the navigation system can turn the robot in place, without requiring a significant (if any) turn radius.
The navigation system 710 can move the robot 100 in response to receiving navigation instructions, for instance from a user of the central system 210, from a security personnel 250, or from another robot. In some embodiments, the navigation system moves the robot as part of a patrol, routine, or security protocol. Navigation instructions can include an end location and can determine a route from a current location of the robot to the end location, for instance by detecting obstacles and/or paths from the current location to the end location, by selecting a path based on the detected obstacles and paths, and by moving the robot along the selected path until the robot arrives at the end location. In some embodiments, the navigation instructions can include a path, an ordered set of locations, an objective (e.g., “patrol the 4th floor”), or a map, and the navigation system can move the robot based on the navigation instructions.
The power system 712 can include a power source, such as a battery, fuel, solar panels, or any other suitable source of power storage. The power system can provide power to the other components of the robot 100, such as the navigation system 710. The power system can also include a recharging system, for instance a plug/socket or other recharging interface. In some embodiments, the robot can navigate to a recharge station and can electrically couple the recharging system to the recharge station, enabling the recharge station to refill the power source of the power system. For example, the recharging system of the robot can include a socket within the robot protected by a pivoting door, and the recharge station can include a plug component. In such embodiments, the robot can navigate to the recharge station and can align the pivoting door with the plug component. The robot can then move towards the plug component such that the plug component pushes the pivoting door inward into the robot, and such that the plug component is inserted into the socket. The plug component can provide power to the power source via the socket. The power system can further include infrastructure electrically coupling the power system to the recharging interface and the other components of the robot.
The communication interface 714 can communicatively couple the robot 100 to other components of the environment of
In some embodiments, the robot 100 can choose to transmit information via the communication interface 714 based on a communicative status of the robot. For instance, in embodiments where the robot is patrolling a route within a building, the robot can transmit video and audio data captured during the patrol when the communication interface is coupled to the network 200 via a WiFi connection. In some embodiments, if the WiFi connection is lost, the robot can communicatively couple to the network via a secondary connection, such as an LTE connection. In such embodiments, the robot can modify the data sent via the communication interface before transmitting, for instance by transmitting only audio data, by performing lossy compression on the video before transmitting, by reducing the frame rate or resolution of the video before transmitting, by transmitting select images instead of video, by transmitting video data corresponding only to key events (such as video of a suspected unauthorized individual), by transmitting full-resolution video based on features within the video (such as faces) and low-resolution video otherwise, and the like. In the event of a total disconnection from the network, the robot can store information for subsequent transmission (for instance, when the communication interface is coupled to the network via WiFi or when the robot is docked at a charge station). Likewise, the robot can select data to transmit or can perform compression operations on the data before transmission if the bandwidth available to the robot falls before a threshold required to stream the data in real-time or other threshold. In some embodiments, the robot can store all detected information and data, for instance at full-resolution, for a set amount of time, until the robot's available storage capacity falls below a threshold, until the data is transmitted, until an operator instructs the robot to delete the data, or until any other suitable criteria is satisfied.
The indicators 716 can include visual indicators (such as LEDs or other lights), audio indicators, haptic indicators, vibration components, and the like. The indicators are configured to communicate information or a status associated with the robot 100 to an entity external to the robot, such as a user or individual. For instance, the indicators can include a ring of LEDs around a top surface of the robot that indicate a direction of movement of the robot. Likewise, LEDs can function as a turn signal, indicating a future direction change of the robot. The robot can also display emoji or other indicators of emotion, can display text identifying a status or current operation of the robot (e.g., “patrolling”, “security alert”, “danger alert”, “intruder nearby”, etc.) In some embodiments, lights or LEDs on the robot can change color to indicate a status of the robot (for instance, LED indicators can be green to indicate that the robot is available to assist customers, can be blue to indicate that the robot is busy performing a task, or can be red to indicate that the robot has detected a violation of a security policy). The robot also includes a hardware display 720 (such as a monitor or screen) configured to communicate information associated with the robot. For instance, the hardware display can display a map, images, a request (e.g., the text “How can I help you today?”), or any other suitable information. In embodiments where the hardware display is a touch-screen display, a user can interact with the displayed information, for instance to get additional information, to configure the display of information, to instruct the robot to perform an operation, and the like.
The robot 100 includes a user interface 718 configured to display information to an individual or user, for instance in response to a received request. In some embodiments, the user interface displays a graphic user interface (“GUI”) including the information, for instance on the hardware display 720 or via the indicators 716. The GUI can display information detailing a status of the robot, information requesting credentials or other information from an individual, information providing instructions to a user (e.g., “follow me”), information associated with an environment or building in which the robot is located (e.g., store or product information), or any other suitable type of information. The user interface can also include infrastructure to enable a user or individual to interface with the robot. For instance, the user interface can include a keyboard or keyboard, one or more buttons or dials, a mouse, motion/gesture detection capabilities, and the like. In some embodiments, the user interface includes the hardware display, and the hardware display is a touch-sensitive display configured to receive inputs from an individual. Inputs received via the user interface can be provided to one or more other components of the robot, can configure the robot to perform various functions or to operate in various modes, and the like. In some embodiments, the user interface further configures a voice recognition engine configured to recognize a user's voice, enabling a user to speak to the robot (for instance, to request information, to command the robot to perform an security operation, or the like).
The cameras 722 can include any number of cameras or camera arrays. For example, the cameras can include cameras configured to capture images or videos directly or a distance in front of the robot, to the sides of the robot, behind the robot, or above the robot. In some embodiments, the cameras can move, pivot, tilt, pan, or rotate, for instance in response to a change in motion by the robot, in response to an instruction received by the robot, in response to detecting an object within a proximity of the robot, or in response to the performance of a security or other function by the robot.
The cameras 722 can capture images and video at any suitable resolution or frame rate. In some embodiments, an array of fixed cameras can capture video 360 degree video around the robot 100, while in other embodiments, one or more cameras can rotate around a central axis of the robot to capture 360 degree video. In some embodiments, camera pairs can capture 3D video, and in some embodiments, images or video captured by multiple cameras can be stitched together using one or more stitching operations to produce a stitched image or video. In addition to capturing images or video in the visible light spectrum, the cameras can capture images within the IR spectrum or can capture thermal images. Images and videos captured by the cameras can be provided to one or more other components of the environment of
The audio system 724 can include components to capture and emit audio signals. For instance, the audio system can include one or more microphones configured to capture audio signals, such as audio signals that pass through the fabric panel 600 of the robot 100. Likewise, the audio system can include speakers configured to playback audio or sound to individuals within a proximity of the robot. In some embodiments, audio captured by the audio system can be provided to a component of
The scanners 726 can detect objects, obstacles, paths, or individuals within a proximity of the robot 100. Examples of scanners include laser scanners or rangefinders, radar, sonar, ultrasonic sensors, motion detectors, 3D depth sensors, thermal scanners, infrared or ultraviolet scanners, and the like. One or more scanners can be located near a base of the robot, for instance to identify objects that might obstruct or disrupt movement of the robot. One or more scanners can be located near a top of the robot, for instance to identify the immediate surroundings of the robot, beneficially enabling the robot to “see” over objects that might otherwise obstruct the viewpoint of scanners located near the base of the robot. In addition to detecting obstacles, objects, and individuals, the scanners can track moving objects or individuals, for instance to follow the movement of suspicious individuals. In some embodiments, information captured by the scanners is used by the navigation system when selecting a path of motion for the robot. Likewise, information captured by the scanners can be used by or provided to other components of the robot or an entity external to the robot (such as a component of
The pressure sensor 728 can include a pressurized tube located around a base of the robot 100, and can detect objects or other obstructions within an immediate proximity of the robot in response to a determination that the robot has run into the object or obstruction. For instance, a robot can move down a hallway, and can inadvertently run into an individual's foot. In response to the contact with the user's foot, the pressure inside the tube increases, and the pressure sensor can detect the increase in pressure and determine that the robot has made contact with an obstruction.
In response to a detected increase in pressure within the pressurized tube, the pressure sensor 728 can determine that the robot 100 has come into contact with an obstruction, and can slow or stop the motion of the robot, for instance by instructing the navigation unit 710. The pressure sensor can determine the direction of the obstruction based on the current direction the robot was moving at the time of the detected pressure increase. In some embodiments, the pressure sensor includes a protective outer housing that, upon contact with an obstruction, exerts a reciprocal pressure on the pressurized tube for detection by the pressure sensor. The pressure sensor enables the robot to immediately detect contact with an obstruction and adjust a trajectory or movement of the robot, beneficially reducing the risk that the contact can cause damage or harm to the robot, an object, or an individual.
In some embodiments, the pressure sensor 728 can measure an amount of pressure exerted upon the pressurized tube. In such embodiments, the robot 100 can use the pressure sensor to apply an amount of force upon an object. For example, the robot may want to gently push open a door, and can move into the door until the pressure sensor detects that ten pounds of pressure are being exerted upon the pressurized tube. By measuring the amount of force applied by the robot onto an object using the pressure sensor, the robot can beneficially limit the amount of force applied to the object in order to reduce the risk that the application of force by the robot damages the object.
The robot 100 can include one or more reader antennas 730 configured to receive signals from entities external the robot (such as components of the environment of
The security system 734 configures the robot 100 to perform one or more security operations. For instance, the security system can configure the robot to investigation a suspicious sound, to obstruct the movement of an unidentified individual, or patrol a portion of a building. The security system can instruct other components of the robot during the performance of security operations, for instance by capturing video of an unidentified user using the cameras 722. The security system can receive instructions to perform the one or more security operations, and/or can receive information associated with one or more security protocols via an external source, such as the central system 210 or the security system 230. Likewise, the security system can provide information associated with the context, surroundings, or activity of the robot, or information associated with the performance of a security operation to an entity external to the robot. The security operations performed by the security system are described in greater detail below.
The semantic mapping system 736 is configured to generate or update a semantic map associated with a location or setting in which the robot 100 is located. For instance, the semantic mapping system can generate a map associated with a patrol route through a building floor as the robot moves through the space. The location of obstructions, and paths within the building floor can be detected by the scanners 726 and recorded onto the semantic map. Likewise, objects can be detected during the robot's movement (for instance, by the cameras 722), and information describing the detected objects and the location of the detected objects can be included within a semantic map. Semantic maps generated or updated by the semantic mapping system can be provided to an entity external to the robot, for instance the central system 210, for storage and/or subsequent use. The generation of semantic maps by the semantic mapping system is discussed in greater detail below.
The RFID system 738 can identify RFID tags within a space and determination the location of the identified RFID tags. For instance, for store inventory objects, each tagged with an RFID tag, the RFID system can, during the course of movement within a proximity of the objects, detect the presence and location of the RFID tag (and thus, the object to which the RFID tag is coupled). The detected RFID tags and corresponding objects can be mapped onto a semantic map, and/or information associated with the detected RFID tags and the locations thereof can be provided to an entity external to the robot 100, such as the central system 210, for subsequent storage. In some embodiments, the RFID system can include short-range RFID readers capable of reading LF RFID or HF RFID tags from within a threshold distance, while in other embodiments, the RFID system can include long-range RFID readers capable of reading UHF RFID tags from greater than the threshold distance away but within a greater, second threshold distance. It should be noted that the RFID system can also read other types of tags according to the principles described herein, including NFC tags or devices, BLE tags or devices, and the like. The RFID system is described below in greater detail.
In the embodiment of
The base 608 of the robot 100 can similarly rotate, for instance using the navigation system 710. Rotating the base beneficially enables the robot to face different directions, to adjust an angle or trajectory of the robot, and the like. Likewise, rotating the base also rotates the laser scanners 812, beneficially enabling the laser scanners to scan in different directions. It should be noted that in some embodiments, the base can rotate while the body and 604 and head 602 of the robot do not rotate. For instance, the base can rotate 90 degrees counterclockwise, and the body can synchronously rotate 90 degrees counterclockwise, resulting in a rotation of the base 90 degrees relative to the floor without the rotation of the body and head relative to the floor.
The robot 100 can also include additional components not illustrated in the embodiments of
The robot 100 can include one or more arms 740 for interacting with objects or individuals within a proximity of the robot. The arm can be an extendable boom arm, for instance that folds up against or retracts into the robot body when not in use. The robot arms can be used for instance to receive an object from or present an object to an individual, to press an elevator button, to move an object, to open or close windows or doors, to lock or unlock windows or doors, and the like. In some embodiments, a robot arm can include a camera, RFID reader, or other sensor. In such embodiments, the robot arm can be positioned or extended in order to provide a better position or vantage point for the camera, RFID reader, or other sensor. For example, an extendable robot arm equipped with a camera can be extended to capture video on the other side of an obstruction or wall, or to capture video of an area or crowd from a better vantage point. In embodiments where a robot arm is extended, the robot may reposition or balance itself to compensate for the shift in the center of gravity of the robot, for instance using inertial sensors (such as 3-axis gyroscopes, accelerometers, or magnetometers).
The robot 100 can also include one or more of: a spotlight or other lighting system for illuminating a room or area, strobe lights or other flashing lights to ward off intruders or otherwise deter particular behaviors of individuals, GPS receivers, sirens or alarms (which may be included within the audio system 724), beacon components configured to alert other robots, non-lethal deterrent components (such as tasers, tranquilizers, nets, tear gas, and the like), force or torque sensors (to detect the position and strength of contact between the robot and the environment or individuals within a proximity of the robot), civilian or police radios, fire extinguishers, emergency/assistance buttons or other interfaces (that enable users to call for help), emergency stop buttons or other interfaces (that enable users to immediately stop the movement of the robot), temperature sensors, light sensors, chemical/gas sensors, actuators configured to move any other components of the robot, or any other suitable components.
It should also be noted that the robot 100 includes component necessary to communicatively couple and control the components of the robot, including but not limited to: on-board computers, controllers, and processors; electric circuitry (e.g., motor drivers); computer memory; storage media (e.g., non-transitory computer-readable storage mediums, such as flash memory, hard drives, and the like); communication buses; cooling or heat dissipation systems; and the like.
Security Functions
As illustrated in the environment of
The security operations described below can be combined with additional security operations. For instance, during the performance of any of the security operations below, a robot 100 can perform one or more of the following operations: warning one or more individuals, obstructing the movement of one or more individuals, reporting individuals or suspicious activity to an operator or security personnel 250, reporting security violations or object/infrastructure status to an operator or security personnel, recording or streaming video to an operator or external entity, locking doors or windows, sounding or triggering an alarm, calling the police or fire department, using non-lethal force to incapacitate an individual, querying one or more database to identify an individual or object, disabling a network or power to a portion of a building, summoning additional robots to assist with the performance of security operations, tracking or following an object or individual, displaying a status or information associated with security or detected activity, requesting information or credentials from individuals, and the like.
As used herein, “suspicious activity” can refer to the identification of an unknown individual, an unauthorized individual, an individual performing an unauthorized or unusual activity, movement in areas or at times when movement is not expected, unusual network activity, sound in areas where sound is not expected, types of sound associated with security risks, objects in unexpected or unauthorized locations, unlocked or open doors or windows at unauthorized times, or any other activity associated with a potential security risk or threat or combination thereof.
Further, as used herein, “boundary” can refer to a perimeter of an area into which an unauthorized individual is not permitted to enter. Accordingly, the crossing of a boundary by an unauthorized individual may be considered to be a security violation, and the robot 100 can perform one or more security operations in response. Boundaries can be predetermined (e.g., associated with pre-defined areas, for instance of a floor or building), can be permanent or temporary (e.g., can exist for particular time period or during a security event or mode), can be learned by the robot based on behaviors of individuals or based on boundaries defined in similar locations, or can be defined by users, operators, or security personnel 250. For instance, a security guard can drive a line on the ground using a laser pointer, and the robot can detect the line and establish the line as a boundary for the purposes of implementing a security policy or performing one or more security operations.
A. Patrolling
The robot 100 can perform a security patrol, for instance within a building or building portion, within a parking lot or other outside area, within a shopping mall or store, or within any other suitable location or premises. The path of a patrolling robot can be predetermined (e.g., based on pre-selected patrol routes defined by a security policy), or can be dynamic (e.g., based on detected suspicious activity, based on how recently an area was patrolled, based on a sensitivity or security associated with any area, based on a viewpoint/vantage point of an area, or based on any other security criteria). The path of a patrolling robot can also be determined for unknown or unmapped areas, for instance by determining a series of waypoints for the robot to follow, by scanning the area to identify nearby paths and obstacles, and by selecting additional waypoints based on the identified paths and obstacles.
In some embodiments, the robot 100 patrols a route periodically, rotates between multiple routes, or selects routes to rotate based on a security level associated with a route or area associated with a route. A route can also be selected based on a time of day, a history of suspicious activity or security violations associated with the route, based on a current security status or violation associated with a route or location, or based on any other suitable criteria. In some embodiments, a route is selected based on a proximity to a recharge station or based on an amount of available power to the robot. In some embodiments, a robot can pause patrolling a patrol route when the amount of power available to the robot falls below a threshold amount of power, can recharge at a nearby recharge station, and can continue patrolling the patrol route after recharged. In embodiments where the patrol route cannot be paused (for instance, when suspicious activity is detected), the robot can communicate with the central system 210 to locate another robot with sufficient charge to perform the patrol. Patrol routes can be randomly selected, for instance to increase the unpredictability of a patrolling robot for a potential unauthorized individual. A patrol route can also be manually selected by an operator of the robot (for instance, the operator can control the path of the robot in real time using a control mechanism).
In some embodiments, robots 100 generate maps as they patrol an area, while in other embodiments, robots access previously generated maps for use in identifying a patrol route. While patrolling, the robot can update a map to identify a route patrolled by the robot, beneficially enabling a robot operator to see the path of the robot. In some embodiments, if a path is blocked for a robot patrolling along a predetermined route, the robot can use a previously generated map to identify an alternative route to bypass the blocked path and to continue on the patrol route (for instance, a shortest alternative route). The robot can update a map to include a location of an obstructed path, to include a location of identified objects or individuals, or to include any other information associated with the patrol route.
While patrolling, the robot 100 can identify suspicious activity, can identify individuals or objects in proximity of the patrol route, and can identify a status of objects or systems along the patrol route. For instance, the robot can identifying suspicious sounds or motion, can determine a location of the identified sounds or motion, and can report the sounds or motion to an external entity (such as the central system 210 or security personnel 250). Likewise, the robot can identify objects and object types while patrolling, and can store or provide the location of the identified objects to an external entity, for instance, in conjunction with a map of the route.
While patrolling, the robot 100 can perform one or more security operations in response to identifying suspicious activity or a security threat. For instance, a robot can flag the location of the identified activity or threat, can inform an external entity of the identifier activity or threat, or can perform any other security operation described herein.
The robot 100 can approach the individual, and can request security credentials for the individual. Alternatively, the robot can remotely scan an RFID badge of the individual to determine the user's security credentials. The robot can determine 915 if the individual is authorized to be in the location based on the determined security credentials and based on the security restrictions associated with the location. In the event that the individual is not authorized to be in the location, the robot can perform 920 a security action. For example, the robot can alert security personnel, follow and monitor the individual, emit a siren, obstruct the movement of the individual, and the like. After the security incident has been resolved (e.g., the individual has left or been apprehended), the robot can continue with the patrol, or can return to a charging station.
B. Video Recording/Streaming
As noted above, the cameras 722 of the robot 100 can capture, record, and stream images and video. Captured images and video can be stored at the robot for subsequent uploading to an external entity (for instance, in the event that the robot is out of range or unable to connect to the network 200). Likewise, captured images and video can be streamed to an external entity (such as the central system 210) for storage or viewing by an operator or other user. In some embodiments, the robot stores video for periodic upload (e.g., once a day or once an hour), for times when the robot is charging at a charge station, or during any other suitable times. Video captured by one or more robots can be synchronized by the central system 210, and can be further synchronized with other video (such as video captured by one or more security cameras) and/or other security information (such as identities of individuals identified within the video or located within an area associated with the video) for subsequent synchronized playback to a user.
In some embodiments, the robot 100 can stream video constantly (for instance, during all times of operation). The robot can also capture and/or stream video selectively, for instance in response to being instructed by the central system 210 or an operator. In some embodiments, the robot captures and streams video in response to one or more criteria being satisfied. Examples of such criteria include: detected movement, suspicious activity (e.g., an unidentified individual or an individual performing an unauthorized action), detected sound when no sound is expected (e.g., during night time operation), suspicious sound (e.g., broken glass), in response to a security incident detected by the robot or other robots, in response to the implementation one or more security policies, or based on any other suitable criteria. In some embodiments, the robot can provide images to the central system during normal operation, and can provide video to the central system only in response to the detection of a security anomaly, in response to a request or instruction from an operator, or in response to any other suitable criteria.
C. Individual/Group Monitoring
The robot 100 can be configured to detect and identify individuals. For instance, the robot can capture images or videos of the individuals using the cameras 722, and can perform facial recognition on the captured images or videos. Likewise, the robot can identify a height or size of the individual, or can scan a badge of the individual (for instance, using an RFID reader). Individuals can be identified by comparing detected characteristics of the individuals with an identity database. For instance, an image of a detected face can be compared against a database of detected facial characteristics associated with individual identities to identify an individual. The identity database can be local (e.g., a database of a building's occupants or company's employees), or can global (e.g., a police database of known criminals). The identity database can also include identities of individuals not permitted to be in a particular area (such as a blacklist).
The robot 100 can take various actions in response to identifying an individual. For instance, if a robot detects an individual, identifies the individual, and determines that the individual is authorized to be at the detected location (e.g., the individual is an employee of a company associated with the building in which the robot is located), then the robot can log the detection and identification of the individual with no further action. On the other hand, if the robot determines that the identified individual is not authorized to be in a particular location (e.g., the individual is an employee of the company, but is in an off-limits area of the company's building), the robot can warn the individual, can instruct the individual to leave the area, can obstruct the individual, can lock nearby doors, can trigger an alarm, can call other robots or security personnel 250 to investigate or intercept the individual, or can perform any other suitable security operation. Likewise, the robot can perform one or more of the security actions described herein in response to being unable to identify the individual.
The robot 100 can monitor multiple individuals. For instance, the robot can identify one or more individuals in a group of individuals, and can track the identified individuals within the group. In some embodiments, the robot can monitor very large groups, such as crowds at a conference or concert. The robot can monitor statistics associated with the group, for instance to determine a number of individuals within the group, to determine an approximate size or floor space required by the group, and to monitor objects carried by or associated with the group. In some embodiments, the robot can determine and monitor a mood of the group, for instance by determining if the group is angry and/or violent or likely to become violent (based on words spoken by members of the group associated with anger or violence, based on detected actions performed by members of the group associated with anger or violence, and the like). The robot can perform one or more security operations in response to monitoring the group, for instance by notifying security personnel 250 if the group becomes violent, by activating fire sprinklers or other group deterrents, band the like. Beneficially, the monitoring of individuals, for instance in an area, can be used by building planners to provide insight into the movement of people in the area for the purposes of space planning.
D. Motion Detection and Action Recognition
The robot 100 can detect motion within a proximity of the robot, for instance using the cameras 722 or using a dedicated motion detection sensor. One or more actions can be taken based on the detected motion. For instance, if the motion is detected in an area where no motion is expected (such as a secure room, a building during off-hours, or the like), the robot can notify security personnel 250, can interact with a building in which the robot is located (for instance, to lock nearby doors), or can investigate further. In some embodiments, the robot can determine if the detected motion is associated with an individual or is associated with an object (such as wind blowing papers off a desk), and one or more security actions can be taken in response to the determination.
In some embodiments, the robot 100 can identify an action associated with a detected movement, for instance if the detected movement is determined to be associated with the individual. For example, the robot can determine if the individual is running, walking, standing, carrying an object, holding a gun, attempting to access a location (e.g., trying to open a door or pass through a security gate), interacting with an object (such as a control panel or a desk drawer), taking a picture or recording video, placing an object (such as an item for sale within a store, or paperwork located on a desk) in a pocket or bag, and the like. The action can be identified by comparing information about the detected movement with a database of detected movements and corresponding action identifiers. The robot can perform one or more security operations based on the identified action. For instance, if an individual is running in an area with dangerous equipment, the robot can instruct the individual to slow down. Likewise, if an individual is holding a gun, the robot can attempt to obstruct the individual.
E. Object Recognition
The robot 100 can detect objects within a proximity of the robot, for instance by capturing images of the objects and analyzing the images to identify the objects within the images. The robot can perform one or more security operations in response to the detected objects. For instance, in the event that an object is not in a location it is supposed to be in, the robot can relocate the object or can alert security personnel of the location of the object. Likewise, in the event that an object is broken or malfunctioning (such as an overflowing sink), the robot can attempt the repair the object, or can notify security personnel of the object.
In some embodiments, the robot 100 can detect license plates within a parking garage or parking lot, and can flag license plates of vehicles that are not authorized to be in the parking garage or lot. For instance, a robot, during the course of a patrol of a parking garage, can scan the location of every license plate in the garage, can query a database of authorized license plates, and can flag scanned license plates that are not included within the database. The robot can then inform security personnel 250 of the unauthorized vehicle, can move to a location blocking the vehicle from moving, or can attach a wheel lock to one or more of the vehicle's wheels.
F. Escorting, Following, Observing, and Access Control
The robot 100 can escort one or more individuals located within a proximity of the robot to a new location. For instance, an individual can be authorized to be in a first portion of a building but not a second portion (such as a contract worker, an external maintenance individual, or an employee with limited security credentials). In such embodiments, the robot can escort the individual from one location to another location in order to ensure that the individual does not cross into an area the user is not authorized to be. For example, a maintenance worker can be working to repair networking equipment and may have to use the restroom. In such embodiments, the worker can inform the robot of the need to visit the restroom, and the robot, in response, can identify the location of the restroom to the worker, can project a path on the ground to the restroom, can project boundaries on the ground that the worker is not allowed to cross, can warn the worker if the worker crosses the boundary, can instruct the user to “follow me”, can wait while the worker uses the restroom, and can escort the worker back to the location of the networking equipment. In some embodiments, the robot can escort an individual for the entire time a worker is within a building or location, while in other embodiments, the robot can escort an individual between locations.
The robot 100 can also follow detected individuals, for instance closely (within a threshold distance) or from a distance (greater than a threshold distance). For instance, if an individual is detected within a building, and the robot is unable to verify the identity of the individual, the robot can follow the individual until the identity of the individual is verified, until security personnel 250 arrive, or until the individual leaves the building. It should be noted that escorting and following individuals for extended periods of time can be particularly tedious for humans to perform, and so implementing one or more robots to perform such tasks can free up humans for other tasks that robots are unable to perform. Further, by following or escorting an individual, the robot 100 can beneficially act as a deterrent for unauthorized behavior by the individual.
In some embodiments, individuals can be followed, escorted, and tracked by the robot 100 by performing facial detection on the individuals to verify that the correct individual is being followed, escorted, or tracked. The location of individuals being followed, escorted, or tracked can be monitored by the robot, can be provided to an external entity (such as the central system 210), or can be updated on a map of the area or building (for instance, in real-time). In addition, the robot can provide information to the individual, for instance in response to request from the individual.
In the event that an identified individual being followed, escorted, or tracked performs an unauthorized action or commits a security violation, the robot 100 can perform one or more security operations in response.
In response to the individual moving 1010 within a threshold distance of the identified boundary, the robot 100 can perform a preliminary security operation. For instance, the robot can move to intercept the individual to obstruct and prevent the individual from crossing the boundary. Likewise, the robot can issue a warning to the individual, can project the boundary on the ground to inform the individual of the area that individual cannot cross, and can inform necessary security personnel of the potential security violation. In response to the individual crossing the boundary, the robot can perform 1015 an enhanced security operation. For instance, the robot can emit a siren, can request the presence of security personnel, can call the police, or can lock nearby doors. In some embodiments, in response to notifying security personnel or an operator of the proximity of the individual to the boundary, or the crossing of the boundary by the individual, the robot can receive authorization for the individual to cross the boundary, and the robot can permit the individual to continue without performing further security operations. It should be emphasized that in some embodiments, the robot may perform an enhanced security operation in response to the individual moving within the threshold distance of the boundary, or may not perform a security operation until the individual crosses the boundary.
In addition, a robot 100 can act as a sentry to control access to a particular area, a building portion, a room, a doorway, a hallway, a floor, an elevator, and the like. The robot can stay within a proximity of the location being monitored, and can physically impede individuals from accessing the location. If an individual approaches the robot (for instance, within a threshold distance from a location of the robot), the robot can warn the user not to cross a boundary associated with the location being monitored, can project a boundary onto the floor around the location, can request credentials from the individual, by moving to mirror the individual's movement (to prevent the individual from bypassing the robot), or by performing any other suitable actions. In the event the individual persists and attempts to or succeeds in bypassing the robot, the robot can perform an enhanced security operation, for instance by notifying security personnel 250, by following the individual, by triggering an alarm, by locking nearby doors, or by performing any other security operation. It should be noted that “enhanced security operation” as used herein can refer to a first security operation more likely to result in preventing a security violation than a second security operation.
G. Interaction with Building Infrastructure
The robot 100 can interact with one or more hardware systems or other infrastructure of a building, campus, site, or the like. An example of such hardware systems includes a camera array.
Another example of a hardware system with which a robot 100 can interact is a building computing system.
Additional examples of hardware systems with which a robot 100 can interact include an alarm system, connected doors and windows, access gates, connected locks, security gates, computer systems, network systems or components, fire alarms, elevators, or any other connected object or system within a building or location. The robot can interact with one or more hardware systems of a building, for instance while patrolling a route within the building, in response to detecting suspicious activity, in response to detecting an unidentified or unauthorized individual, in response to determining that an individual violated a security policy, in response to detecting that a building object is misplaced or is in violation of a security policy, or in response to any other suitable criteria. In such instances, the robot can communicatively couple to a hardware system of the building (for instance, via the network 200) in order to leverage the capabilities or functionality of the hardware system to perform one or more security operations.
In response to identifying the object or individual, the robot determines 1310 if the identified object or individual violates one or more security policies. For example, the robot can determine that the individual or object violates a security policy if the individual is unidentified, if the individual is not permitted to be in a particular location or portion of a building, if the individual does not possess the proper security credentials (for instance, if the user is not in possession of an RFID badge), if the user is performing an unauthorized action, if the user is blacklisted from a location, if the user is in possession of an unauthorized object, if the object is misplaced or located in an unauthorized location, if a status of the object is unauthorized (e.g., if a door that is supposed to be locked is unlocked), if an object is broken or malfunctioning, or based on any other security policy violation criteria being satisfied by the individual or the object.
In response to determining that the individual or object violates a security policy, the robot 100 can connect 1315 to a hardware system of a building, and can perform 1320 one or more security operations using the connected hardware system. For instance, the robot can communicatively couple to a central system 210, an infrastructure system 220 (such as an elevator system or power system of the building), or a security system 230 (such as a camera array located within the building). In some embodiments, the robot communicatively couples to a system selected based on a type of security policy violation. For instance, if the robot determines that an individual's movement or location violates a security policy, the robot can communicatively couple to a camera array associated with the location of the individual to control the camera to capture video of the individual or to follow the movement of the individual. Likewise, if the robot determines that a malfunctioning network component violates a security policy, the robot can communicatively couple to a network controller or the central system in order to reset the network component.
In some embodiments, the robot 100 communicatively couples to the central system 210 upon detecting the security policy violation in order to inform a user of the central system or an operator of the robot of the security violation in order to allow the user or operator to take an action in response to the detected security violation. In some embodiments, the object determined to violate the security policy is the hardware system to which the robot communicatively couples. For example, if the robot determines that a door being unlocked violates a security policy, the robot can communicatively couple to the door in order to lock the door (for instance, in embodiments in which the door is connected to the network 200 is capable of directly coupling to the robot). In some embodiments, the hardware system to which the robot connects is another robot. In such embodiments, the robot can provide instructions to the second robot in order to perform one or more security operations. For instance, the set of robots can corner or obstruct an individual determined to have violated a security policy.
In some embodiments, in response to detecting that an individual violates a security policy, the robot 100 can control a camera array to adjust the orientation of the camera array to capture video of the individual. In such embodiments, if the individual outruns the robot or hides such that the robot does not have a direct line of sight to the individual, the robot can use the video footage captured by the camera and streamed to the robot to locate the individual. The robot can then approach the individual, can report the location of the individual to one or more security personnel 250, or can instruct a second camera to capture video of the user (for instance, in response to the individual moving outside of the range of the camera array).
In addition, the robot can perform 1320 security operations including but not limited to: sounding an alarm, triggering a fire sprinkler system, closing or locking doors or windows, locking access gates, accessing computer system databases, reporting the violation of the security policy and any associated information to a local authority (such as the police), halting elevator systems, resetting or freezing network systems, and the like. It should also be emphasized that a robot can communicatively couple with a building hardware system in order to perform one or more operations using the connected hardware system without requiring the detection of a security violation. For example, the robot can determine that the temperature of a room is below a threshold value, and connect to a furnace system in order to increase heat flow into the room. Likewise, the robot can detect smoke and can connect to windows and a building sprinkler system in order to open the windows (to vent the smoke) and to turn on the sprinkler system (in order to extinguish the fire). In some embodiments, the robot can determine that an object (such as an item for sale in a store) is misplaced, and can communicatively couple to an inventory database in order to update the location of the object and to a cashier's computer system in order to inform that cashier that the object is misplaced.
In some embodiments, the robot 100 can perform a security operation, and a building hardware system can perform a security operation or other action based on the result of the security operation performed by robot. For instance, the robot can verify the identity or the credentials of an individual (for instance, by performing a face detection operation on the individual, by scanning the individual's identity badge, or by receiving a password or code from the individual), can communicate the verified identity or credentials to a connected access gate or door, and the access gate or door can unlock or open in response to receiving the identity or credentials. Similarly, the robot can determine that an individual is suspicious, can inform a security camera of the detected suspicious individual, and the security camera can record video of the individual in response.
In some embodiments, the robot 100 can perform a security operation in response to a security operation performed by a hardware system of a building. For instance, if a security camera detects a suspicious individual, the security camera can deploy a mobile robot to approach or follow the individual, to request the user's credentials, or to warn the individual. Likewise, if an audio system detects a suspicious noise, if a connected door detects that an individual attempted to open the door, or if a connected badge reader scans a badge for an individual that is not authorized to access a location associated with the badge reader, a robot can be deployed to perform one or more security operations associated with the security operation performed by the hardware system. It should be noted that a robot and a hardware system can perform security operations as described herein autonomously or without the explicit instructions or involvement of a human (such as a security personnel 250 or an operator of the robot), beneficially reducing the amount of human interaction required to perform the security operations.
H. Anomaly Detection
The robot 100 can detect security anomalies, for instance while patrolling within a building. As used herein, a security anomaly refers to a circumstance that might represent a security risk. Security anomalies can be detected using one or more sensors or systems of the robot, or may be detected based on data received from one or more systems external to the robot, such as from the central system 210, an infrastructure system 220, or a security system 230. A security anomaly can be identified based on accessed information describing a characteristic of an environment contrary to an expected characteristic of the environment. Upon detecting a security anomaly, the robot can perform one or more security operations, for instance to contain or mitigate the security risk corresponding to the security anomaly.
An example of a security anomaly is the detection of a sound that may indicate a security risk. The detected sound may indicate a security anomaly based on the type of sound detected. For instance, if a robot 100 detects the sound of broken glass, the robot can identify the detected sound as indicative of a security risk, and thus a security anomaly. Examples of detected sounds that can indicate a security anomaly include but are not limited to: windows opening, glass breaking, doors opening or closing, gun shots, walking footsteps, running footsteps, car sounds, shouting, water running, hissing (as from gas), and sounds that are an above-threshold magnitude. Likewise, if a robot detects a human voice in an area where no individual is present, the robot can identify the detected voice as a security anomaly. In some embodiments, any detected sound above a threshold magnitude can be identified as a security anomaly (for instance, in secure environments where individuals are not authorized to enter).
In some embodiments, security anomalies can be detected using cameras of the robot 100 or of a security system 230. For instance, a security anomaly can be detected if the robot detects an individual at a time or place where no individuals are expected to be present, if the robot identifies an individual that is blacklisted or is otherwise not authorized to be a particular location, if the robot identifies an individual taking an action that is not expected or authorized, if the robot identifies a broken or open window, if the robot detects an unlocked or open door when the door is supposed to be closed or locked, if the robot detects an object that is out of place, or the like. In some embodiments, a security anomaly can be detected based on detected or intercepted communications, wireless signals, or the like.
In response to detecting a security anomaly, the robot 100 can perform one or more security operations. For example, in response to detecting a sound indicative of a security anomaly, the robot can investigate the detected sound, can inform the central system 210 of the detected sound, or can instruct nearby security cameras to record video of the location where the sound was detected. Likewise, in response to detecting an individual in a location where there individual is not authorized to be, the robot can warn, follow, or confront the individual; can request security personnel 250 to come investigate; and can lock nearby doors to prevent the escape of the individual. In addition, in response to detecting a broken window, the robot can update a semantic map to indicate that the window is broken, or can summon security personnel to come replace the window. In response to detecting an unlocked door (at a time when the door is supposed to be locked), the robot can lock the door (either remotely in the event that the door is connected to the network 200, or manually otherwise) and can instruct nearby robots, cameras, and security personnel to be on the lookout for unauthorized individuals responsible for the door being unlocked that might be in the area. In some embodiments, the central system 210 can, in response to the detection of a security anomaly by a robot, increase patrols to the location associated with the detected security anomaly for a threshold amount of time, for instance, for an hour.
I. Aerial Systems
As noted above, in some embodiments the robot 100 or the other components of
In some embodiments, the aerial robots can land on and perch on ledges, surfaces, or objects and disable flight mechanisms (such as propeller spinning, navigation components, and the like). Perching enables the aerial robots to perform various security operations (such as recording video) from a better vantage point than the robot 100, while also saving the power otherwise required to maintain altitude or otherwise fly. When the aerial robot subsequently needs to recharge or move, the aerial robot can enable flight mechanisms and fly away as needed.
In some embodiments, aerial vehicles can land on and couple to charging stations in order to charge the batteries of the aerial vehicles. The charge station can include a coupling component and the aerial vehicle can include a reciprocally coupling component, such that the aerial vehicle charges when the coupling component is coupled to the reciprocally coupling component. In some embodiments, the aerial vehicles autonomously fly to the nearest charge station when the battery power available to the aerial vehicle falls below a threshold. In some embodiments, the robot 100 can include a charge station and/or launch/landing pad, and the aerial vehicle can fly to the nearest robot to recharge as needed. In yet other embodiments, an aerial vehicle can be tethered (e.g., connected to a power source via a cable) to either a charge station, robot, or other power source, beneficially enabling the aerial vehicle to have an unlimited or much greater power supply at the cost of mobility.
In some embodiments, the aerial vehicle can be wirelessly recharged, for instance using inductive recharging. In such embodiments, the landing pad or charge station can include a first inductive coil and the aerial vehicle can include a second inductive coil such that an electromagnetic field created by the first inductive coil can create energy in the second inductive coil which can then be used to charge the batteries of the aerial vehicle. In some embodiments, aerial vehicles can be wirelessly charged while in flight or remote from charge stations.
In some embodiments, the aerial vehicles can be manually controlled (for instance, by a user of the central system 210 or other aerial vehicle operator). In such embodiments, the operator of the aerial vehicle can control the location, movement, flight, and functionality of the aerial vehicle. For example, if a robot 100 detects a security anomaly or unidentified individual, the robot can launch an aerial vehicle to investigate the security anomaly or unidentified individual. The location and/or field of view of aerial vehicles can be displayed to a user of the central system, for instance within a security interface map.
In some embodiments, the aerial vehicles can be launched and controlled by a robot 100, for instance, in response to a determination by the robot during the patrolling of a route that an obstruction (such as a wall or barrier) prevents the robot from being able to adequately observe or monitor an area. In such embodiments, the robot can launch an aerial vehicle from the robot or from an aerial vehicle launch pad or charge station, the aerial vehicle can scan the area and can provide information about the scan to the robot, the aerial vehicle can return to the point of origin (the robot, the launch pad, the charge station, etc.), and the robot can perform a security operation based on the information received from the aerial vehicle (if necessary) or can continue with patrolling the route.
The aerial vehicle can record sound or video while in flight, and can stream the recorded sound or video (for instance, either directly from the aerial vehicle to the central station 210 via the network, or by via the robot 100, which in turn can broadcast the recorded sound or video). Likewise, the aerial vehicle can record, store, and/or broadcast other data captured by other sensors, information related to security operations or the operation of the aerial vehicle, or any other suitable information as needed to the robot or the central system. The video, audio, or other information captured by the aerial vehicle can be played or displayed to a user of the central system, for instance within a security interface portion.
In some embodiments, the aerial vehicle can receive security operation instructions from the central system 210, and can perform one or more security operations in response. For instance, the aerial vehicle can patrol routes in response to an instruction received directly from the central system. Likewise, the aerial vehicle can scan RFID badges in a particular location in response to an instruction received from the robot 100, which in turn received the instruction to identify all individuals within the location. In such embodiments, the aerial vehicle can perform the functionalities of the robots or security systems described herein.
J. Central System Operation
The central system 210 can interact with one or more robots 100 and other infrastructure systems 220 and security systems 230 to perform one or more security operations. The central system can receive data from the one or more robots, infrastructure systems, and security systems; can aggregate, normalize, or otherwise format the received data; and can display the formatted data within an interface. Security interfaces are described below in greater detail. The central system can also request particular data from particular robots, infrastructure systems, or security systems; can instruct robots, infrastructure systems, and security systems to perform one or more security operations, either independently or in concert with other robots, infrastructure systems, or security systems; or can implement one or more security policies. It should be noted that the functions of the central system described herein can be performed in response to input from a user or operator of the central system, or without receiving inputs from a user or operator (for instance, autonomously or in response to detected security criteria that trigger the performance of security operations).
The central system 210 can aggregate sensor data from one or more robots 100, infrastructure systems, or security systems. For instance, the central security system can receive locations of individuals within a building from multiple robots and security cameras, and can combine the location information to determine refined locations of each individual within the building. Likewise, the central system can interpolate data received from multiple sources, for instance by averaging temperatures detected in each room in a building to determine an average temperature for each room. Finally, the central system can receive information describing a state of one or more objects in a location, for instance the power levels, capabilities, or locations of each robot within a building.
The central system 210 can manage multiple maps for a location, for instance stored building maps and blueprints, semantic maps describing a building or area, and simultaneous location and mapping (or “SLAM” maps) generated by a robot 100 or other system configured to scan a location or area. In some embodiments, the central system can coordinate between multiple maps of the same location, for instance by updating older maps to include the location of objects that newer maps indicated have moved, or by incorporating types of information present in a first map but not a second map into the second map (for instance, the location of windows, whether a door is locked or unlocked, or the location of security cameras).
The central system 210 can implement one or more security policies, such as a building security policy. For instance, a security policy can specify one or more security criteria, such as: patrol routes within a building, can specify a frequency of patrolling each route, can specify access credentials required for an individual to be in one or more portions of the building, can specify acceptable threshold values or ranges for sensors within the building (such as minimum or maximum temperatures, sound magnitudes, characteristics of video data captured by cameras), can specify acceptable or forbidden actions or activities for individuals within one or more building portions, can define geofences describing portions of a building that can be accessed or that are off-limits (for instance, by time, by access credential), and the like. In some embodiments, the security criteria are time-dependent, and can vary based on whether the current time falls within a time range specified for the security criteria.
In implementing the security policy, the central system 210 can instruct one or more robots 100 to perform a task (such as patrol a route or intercept an individual that isn't authorized to be in a particular location), can instruct security cameras to change viewpoint and/or capture video of a location associated with a potential violation of a security policy, can request sensor data detected by one or more infrastructure systems 220 or security systems 230, and the like. In some embodiments, security policies can be implemented in response to the detection of one or more security criteria. For instance, if an individual is detected in an off-limits portion of a building or a suspicious sound is detected, the central system can dispatch one or more robots or security personnel to investigate.
As noted above, the central system 210 can also instruct one or more robots 100 and/or one or more infrastructure systems 220 or security systems 230 to perform a security operation in response to a request from a robot, in response to receiving information from a robot (such as information indicative of a security violation or threat), or in response to a request or information from an infrastructure system or security system. The central system can prioritize security operations, and can instruct the one or more robots, infrastructure systems, and security systems to perform security operations based on the priority of each security operations. For instance, if the central system identifies ten security operations that need to be performed by five robots, the central system can prioritize the ten security operations, and can instruct the five robots to perform the five highest priority operations first, and when those operations are complete, can instruct the robots to perform the remaining five security operations. The central system can account for the capabilities and availability of each robot, infrastructure system, and security system before instructing these robots and systems to perform security operations. For instance, the central system can determine that of two robots, a first is able to perform a particular operation (such as an IR scan) and a second isn't, and can instruct the first robot to perform the operation (and not the second, even if the first robot is currently occupied).
The central system 210 can record and store data received from one or more robots 100, infrastructure systems 220, and security systems, for instance within log files in memory accessible to the central system. In addition, the central system can report or log any suspicious activity, detected security anomalies, detected threats or alerts, or any potential or actual violations of security policies, for instance to security personnel 250 or to an operator or user of the central system. The central system can also coordinate communications between components of the environment of
As noted above, an implementation of the central system 210 can present a security interface to an operator. The operator can review data provided by the robots 100, infrastructure systems 220, and security systems 230 and presented within the interface. The information presented by the security interface can be updated in real-time, for instance reflecting updated information received from the robots, infrastructure systems, and security systems. For example, the security interface can update the location of the robots within a building in real-time. The operator can coordinate the actions of multiple robots, can assign patrol routes to robots and can alter patrol routes as the robots are patrolling the routes, can dispatch robots to investigate security anomalies or suspicious activity, and can assign particular security operations to particular robots.
The central system 210 can notify security personnel 250 or other individuals or groups to inform the security personnel of security violations, suspicious activities, security anomalies, and the like. For instance, the central system can send text or SMS messages, emails, or voice calls to security personnel, to police, to a company's employees or managers, and the like. Likewise, the central system can field queries from robots 100, infrastructure systems 220, security systems 230, security personnel 250, users of a remote interface 240, and the like, and can provide information gathered by the central system in response to the queries. For example, the central system can provide a map of all RFID readings as detected by a fleet of robots to a requesting entity. Likewise, the central system can store and provide a WiFi heat map, a temperature map, and the like to users of the central system, and can generate (either periodically, automatically, or in response to user request) reports of anomalies, security events, patrol coverage, or building properties for review by users of the central system.
In some embodiments, the robots 100 may not include the necessary computing power to perform certain functions (or may include the necessary computing power but may not be able to perform such functions in a time-efficient manner), including both security operations (such as facial detection) and non-security operations (such as national language processing). In such embodiments, the robots can request that the central system 210 (which may include greater processing capabilities and resources) to instead perform such functions. For example, a robot can receive a request from a user asking “How do I get to the conference room?”, and can send the recorded voice sample to the central system, which can parse the voice signal and determine the user's intent. Continuing with this example, the central system can identify a set of navigation instructions to the conference room to the robot, and the robot can provide the navigation instructions to the user. Likewise, a robot can request that the centralized server compress data, perform video processing tasks, to stream video to other robots or entities, to perform machine learning tasks, or to perform any other processing- or resource-intensive tasks, and can provide the data to perform such tasks to the central system. The central system can also access a building's or company's databases or processing resources (such as servers or other computer systems), for instance to identify an individual within a building or to use the accessed processing resources to perform a computationally-intensive task.
The central system 210 can select 1415 a second security operation based on the data received from the robot while performing the security operation. For instance, if the data received from the robot is video, and the video indicates an unauthorized individual is located in a portion of the building, the second security operation can be an authentication operation (to authenticate the identity of the individual), a video recording operation (to record video of the individual), and the like. The central system can instruct 1420 a security system to perform the second security operation. In some embodiments, the security system is the robot, and the robot can immediately begin performing the second security operation, can perform the second security operation after completing the original security operation, or can slot the second security operation into the queue of security operations maintained by the robot (for instance, at the front of the queue, at a location within the queue corresponding to the priority of the second operation, or at the end of the queue). In other embodiments, the security system is a system other than the robot, which can likewise immediately perform the second security operation, or can queue the second security operation based on the priority of the second operation.
Although the central system 210 is described herein as a single entity, it should be noted that in some embodiments, the central system can include any number of systems operating in concert. For instance, the central system 210 can be implemented within a data center, by several communicatively coupled servers, or the like. Finally, it should be noted that in some embodiments, the tasks performed by the central system may instead be performed by other components of the embodiment of
K. Security Interface
As noted above, the central system 210 can display a security interface that includes various security information and data from the components of
The security interface can include a number of portions, and each interface portion can display a set of security information. For example, an interface portion can display a map of a building or area, robot status information, robot video streams, security camera video streams, security notifications, a list of individuals within an area, a list of security personnel 250 within an area, security policy violation information, video chat streams (for instance, for an individual speaking to a robot 100), communication information, infrastructure system status information, security system status information, a list of available security operations (for assignment to a robot, infrastructure system 220, security system 230, or security personnel), a list of security operations being performed or queued, security operation status information, and the like.
The first interface portion 1500 includes a local map illustrating various entities, objects, and notifications within a portion of a building. The location map can include a floorplan of the portion of the building, for example illustrating rooms, names of rooms, room numbers, hallways, doors and/or windows, and important building features. In the embodiment of
The level of certainty or uncertainty in the individual's location can also be visually represented within the security interface, for instance by adjusting an amount of fading or the color of the icon representative of the individual. The level of certainty or uncertainty in the individual's location can be based on the amount of time since the individual was last observed (e.g., wherein the uncertainty increases with time since last observation), a speed or activity of the individual when the individual was last observed (e.g., wherein the uncertainty increases with the speed of the individual or based on the activity type of the individual), or based on an identity of the individual (e.g., wherein the uncertainty increases faster for individuals with lower access credentials or with a history of security violations than for other individuals). In some embodiments, different icons, colors, or other representations can be used to display different categories of people, for instance identified individuals, unidentified individuals, individuals by categories of access credentials, individuals by activity type, individuals performing prohibited actions, individuals associated with security risks or threats, and the like.
In addition to robots, the first interface portion 1500 also includes other objects, such as the security camera 1512. The field of view of the security camera is also displayed (for instance, in a different color than the field of view 1514 of the robot 1510), and any individuals within the field of view of the security camera are indicated with icons. Likewise, the location of various tracked assets (such as the tracked asset 1526) can be indicated within the first interface portion using suitable icons. Tracked assets can include assets coupled to RFID tags, and examples of tracked assets can include lab equipment, computer equipment, and secure documents. The first interface portion can also include notifications, such as the alert notification 1522 and the warning notification 1524. The location of notifications within the local map can be based on the location of the entity or object associated with the notifications. In some embodiments, a user can view all tracked assets in a particular location within the security interface, for instance a listing of assets by name or identity, a time last seen, an expected or permitted location, and an actual or last seen location. The user can also use the security interface to add additional tracked items, to change the permissions of existing tracked items (for instance by changing the permitted locations for the items), and can remove tracked items.
As noted above, the security interface can be updated in real-time, as objects and entities move and events occur. For instance, the first interface portion 1500 can be updated to display the robot 1510 as it moves along the patrol path 1520. As the robot moves along the patrol path, the displayed robot icon can move to reflect the change in location of the robot, allowing a user of the security interface to be able to track the robot in real-time. Likewise, the location of individuals or objects detected by robots, security cameras, or other location-tracking mechanisms (such as GPS tracking devices, mobile phones, or RFID readers) can be updated as the individuals or objects move in real-time.
The security interface can also include a time bar 1508 enabling a user of the security interface to select a previously occurring viewing time. The security interface, in response to the user selecting a previously occurring viewing time, can adjust the various interface portions of the security interface to display information associated with the previously occurring viewing time. The user can then “play” the information within the security interface from the selected previously occurring viewing time, enabling the user to view changes in displayed information in simulated real-time (or augment real-time slowed down or sped up relative to real-time). Accordingly, a user can view security interface events as they occur in real-time, or can beneficially go back in time to re-watch security interface events in simulated real-time as needed.
The second interface portion 1502 displays a global map of a building, including robot icons for robots within the building, person icons for individuals within the building, and the like. The global map includes a window 1528 that corresponds to the current view of the local map within the first interface portion 1500. The global map can also update in real-time, for instance as the robots move within the building, and as the portion of the building illustrated in the local map changes.
The third interface portion 1504 illustrates text notifications corresponding to events occurring within the portion of the local map 1500, corresponding to information associated with objects or individuals within the building, or corresponding to any other information associated with the security interface. In the embodiment of
The fourth interface portion 1506 displays a robot interface. In the embodiment of
In some embodiments, a user of the security interface can interact with the security interface in various ways. For instance, a user can scroll the first interface portion 1500 to change the portion of the global map displayed within the local map. Likewise, a user can touch a portion of the global map of the second interface portion 1502 to relocate the window 1528 and thus relocate the portion of the global map shown within the local map. A user can also zoom in or out on the local map, can select a displayed icon for information associated with the entity or object corresponding to the icon (for instance, a user can select a robot icon or security camera icon to change the information displayed with the fourth interface portion 1506), and can select an icon corresponding to an entity or object to change a state of the entity or object (for instance, a user can select a door icon to open, close, lock, or unlock the corresponding door, or can select a robot icon to change a security operation being performed by or mode of the corresponding robot).
In some embodiments, a user can use the security interface to change the patrol route of a robot by selecting among a set of patrol routes, or by manually drawing a patrol route on the local map (upon which, a set of navigation instructions through a building portion corresponding to the drawn patrol route will be sent to the robot, and the robot can begin patrolling the drawn patrol route based on the navigation instructions). In addition, a user can select a building portion or room displayed within the local map for investigation by a robot. In response, the robot can investigate the building portion or room, and can provide information corresponding to the investigation (such as video data or sensor data) back to the security interface for display. A user can select a displayed robot icon or security camera icon to adjust a field of view or viewpoint of the robot or security camera, to view video captured by the robot or security camera, or to change the behavior of the robot or security camera. A user can also use the security interface to select a location for a robot to guard or act as sentry (for instance, by drawing a line within the local map), can select a guard or sentry behavior (e.g., “allow no one to pass”, “allow users with proper security credentials to pass”, etc.), and can select an active guard or sentry time (e.g., “guard between the hours of 1:00 am and 7:00 am). In some embodiments, the security interface can beneficially enable a user to manually control a robot, for instance using a keyboard or joystick mechanism to move the robot around a location displayed on a map and to adjust a viewpoint or field of view of the robot.
A user can interact with the security interface using any suitable means. For instance, a user can click on a displayed robot icon using a mouse or selecting gesture on a touch-screen display, a list of security options can be presented by the security interface in response, the user can select one of the presented security operations, and the security interface can instruct the robot to being performing the selected security option. Likewise, a user can scroll or zoom using a mouse or a scrolling gesture on a touch-screen display.
In some embodiments, an interface portion can display an identity of individuals in an area. For example, the interface portion can list the individuals who have been identified (for example, the name, position, and/or access credentials of the individuals), and can also list individuals who have not been identified (e.g., by a generic placeholder such as “Individual #1”, and an indication that the individual has not been identified). The interface portion can also list a location of each individual, for instance, a building or room in which each individual is located. Likewise, an interface portion can list security personnel in an area, and a location of each security personnel.
Each interface portion or each category of information shown within a security interface can be associated with an access level. Accordingly, the interface portions or categories of information shown to a user of the security interface can be limited based on the security credentials of the user. For instance, if a user is not authorized to view video streams from robots, the security interface can be configured to prevent video streams from being shown to the user.
The security interface can also differentiate displayed areas within a map based on a category of the area. For instance, off-limits rooms can be displayed in a first color, rooms available to high-security individuals can be displayed in a second color, and can display rooms available to everyone in a third color. The security interface can also display geofences within a map, for instance boundaries of areas not accessible to robots, and areas available to non-employee visitors. Finally, the security interface can differentiate areas of a map currently visible to robots and/or other sensors. For instance, the security interface can show areas of a map visible to robots in a first set of color tones, and can show areas of a map not visible to robots in a second set of color tones (for instance, a darker set of color tones), beneficially enabling a user of the security interface to quickly identify currently unmonitored portions of a building.
In some embodiments, the security interface can show security information received from or representative of more than one location. For instance, the security interface can receive security information from a plurality of different buildings, for example belonging to the same entity or company. In such embodiments, a user of the security interface can toggle the interface between locations. For example, a user can configure the security interface to show a map of robots, tracked assets, and security information within a first building, and can subsequently configure the security interface to show a map of robots, tracked assets, and security information within a second building. Each location can be associated with a different selectable icon displayed within the security interface, can be listed in a selectable drop-down list, and the like. The security interface can update with security information associated with a selected location in response to the user selection of a configuration option associated with the location.
A user of the security interface can also configure security policies using the security interface. For instance, a user can select security policy components, including one or more of a location, a time range for which the security policy will be active, one or more categories of individuals to whom the security policy applies (e.g., employees, managers, staff, visitors, etc.), permitted access credentials, security policy exceptions, actions taken in the event of security policy violations (e.g., notify security, revoke user credentials, lock nearby doors, etc.), and the like. The security policy configured by the user of the security interface can be stored and subsequently implemented, for instance by patrolling robots, security systems, and/or nearby doors and access gates. In some embodiments, the security interface can log the credentials, identity, and/or actions of users of the security interface, beneficially allowing the security interface to generate an auditable log of the users and user actions for subsequent review (for instance, for accountability and privacy considerations).
M. Additional Security Operations
The robot 100 can perform one or more additional security operations, either independently of or in concert with one or more of the security operations described above. In one embodiment, the robot can operate in a privacy mode, where data collected and security operations performed by the robot are not communicated to an external entity (to preserve the privacy of individuals in proximity to the robot, or to reduce the likelihood that such data is intercepted during transmission) unless one or more security criteria are triggered (such as the detection of suspicious activity, the identification of an unauthorized individual, and the like). Operating in such a privacy mode further beneficially enables the robot to operate in proximity to sensitive data or objects (such as secret lab equipment or confidential/classified information) without a risk of such sensitive data or objects being discovered by unauthorized entities within access to the robot or the central system 210. In some embodiments, the robot is configured to blur faces within captured images or videos to preserve the privacy of the individuals associated with the blurred faces. Such facial blurring may be performed by the robot on the captured images and videos only when the images or video is transmitted to an external entity. The robot can also beneficially blur computer monitors, documents, sensitive data or instruments, or any other objects or information as needed to satisfy security restrictions.
In some embodiments, the robot 100 can use one or more sensors, such as IR cameras, 3D depth cameras, ultrasonic sensors, and rangefinders to navigate within darkness. Doing so beneficially enables the robot to operate during times when the lights in an area are otherwise turned off. Accordingly, the absence of visible light in an area is not detrimental to the ability of the robot to perform security operations. As the majority of burglary occurs at night when employees or other individuals are not present, the ability of a robot to perform security operations in darkness beneficially prevents the need to power lights in an area. When the robot detects suspicious activity or when one or more security criteria are triggered while operating in darkness, the robot can activate a spotlight of the robot to highlight an object associated with the detected activity, can activate the buildings lights (e.g., by communicating with the central system 210 to turn on the lights for a particular area associated with the detected activity), or can call one or more security personnel 250 or other robots to the area associated with the detected activity.
In some embodiments, the robot 100 can enable a user within a proximity of the robot to interact with security personnel 250 or an operator of the robot. For instance, the robot can display real-time video of the operator or security personnel to the individual, and/or can transmit audio communications between the individual and the operation or security personnel. Doing so beneficially enables the individual to speak with a human associated with the robot, for instance in order to request authorization for access to a particular area, to authenticate the individual's credentials, or to enable the individual to request help or assistance.
The robot 100, in response to determining that the robot has lost connectivity with the network 200 during the patrolling of a route, can return to an earlier location with the route where the robot was connected to the network. Likewise, in some embodiments, the robot can move to a location further along the patrol route until connectivity is restored. While the robot is disconnected from the network, all data connected by the robot (related to security or otherwise) can be stored until connectivity is restored. When the robot is able to reconnect to the network, the robot can upload the stored data (for instance, to the central system 210).
In some embodiments, the robot 100 can monitor the environment in which the robot operates over time to detect potential security anomalies using one or more of the sensors of the robot. For instance, the robot can detect the temperature of particular rooms within a building, can detect smoke, can detect carbon monoxide levels, can detect light levels, can detect chemical compounds or gaseous substances, and the like. The robot can obtain and store these measurements, and can map out variations of these measurements over time and by location. If, during the course of a patrol, a robot records obtains a measurement that deviates by more than a threshold from historical measurements, the robot can flag these measurement as a security anomaly. For instance, if the robot detects the temperature of a room over time to be between 72 degrees Fahrenheit and 78 degrees Fahrenheit, the robot can identify a measure of the room of 68 degrees Fahrenheit as an indication of a security anomaly (for instance, as a result of someone leaving a window open). The detected security anomalies can subsequently be reported to the central system 210, or to an external entity (such as a building operator or facilities management team).
The robot 100 may detect fires, for instance visually using a camera 722, an IR sensors, a temperature sensor, or any other suitable measuring system. In response, the robot can attempt to distinguish the fire, for instance using an on-board fire extinguisher, can flag the fire for security personnel to handle, can flag the fire to the central system 210, can trigger a fire alarm, can trigger a building's sprinkler systems, or can call the fire department or other emergency services. The robot may also detect when an individual is smoking (for instance, either visually, by detecting the smoke in the air, or by detecting the presence of cigarette-related gases in the air), and can inform the individual that smoking is prohibited. In such instances, the robot can also inform the central security system. In emergency situations, the robot can enable an external entity (such as an operator of the robot) to communicate with individuals within a proximity of the robot, for instance to assist with evacuation, to hold or disable elevators, to unlike exit doors, to provide information to response personnel, to sound alarms, to map out a location of a fire, to identify exit routes, to obstruct or locate unauthorized or dangerous individuals, and the like.
In some embodiments, the robot 100 can include airbags configured to inflate when the robot is falling over in order to cushion the fall to the extent possible. The use of airbags to soften impact upon falling can beneficially reduce damage to the robots. In some embodiments, the robot can prevent “tailgating”, or the following of a first individual with proper security credentials by an unauthorized individual through a door or access gate open in response to the security credentials of the first individual.
In some embodiments, the robot 100 can be used, for instance, by a security personnel 250 or user of the central system 210 to locate an object or individual. In such embodiments, the central system can determine a last known location of the object or individual (for instance, by querying the security interface or by accessing data logs of object and individual locations), and can provide this location to the security personnel or user. Alternatively, the last known location of the individual can be used (for instance, by robots) to scan the area within a proximity of the last known location for the individual. When the current location of the individual is determined (for instance, in response to the deployment of a robot to the last known location and the subsequent scanning of the area), the current location can be provided, for instance to the requesting entity.
In some embodiments, the robot 100 can determine its location without using a map (for instance, in response to a mapping error). For instance, the robot can ask a nearby human for help, can use RFID tags found in a vicinity of an area and can identify a historical location of the RFID tags, can ask nearby robots for location information, can query the central system 210 for the robot's location, can use 3D information or 2D/map layout information to determine the location of the robot, and the like.
As noted above, the robot 100 can perform operations in addition to security operations. For instance, the robot can be located within an entrance or doorway and greet people as they enter or leave an area. The robot can request janitorial service in response to detect a mess within a proximity of the robot, and can act in self-defense in the event that someone tries to tamper with the robot or with another security system or infrastructure system.
In some embodiments, the robot 100 can delivery objects, packages, paperwork, and the like to an individual, for instance by querying a semantic map, the central system 210, another robot, or any other system or entity that tracks the location of individuals. In response to identifying a location of the individual, the robot can navigate to the individual, can verify the individual's identity (for instance, by scanning an RFID badge of the individual, by performing facial or voice recognition on the individual, or by any other suitable means), and can have the individual sign for the delivery before providing the object to the individual. In some embodiments, the robot can enable guests to check-in via the robot's touchscreen display. In other embodiments, the robot can enable individuals near the robot to report emergency situations, for instance a fire nearby, an unauthorized individual within the vicinity, a medical emergency, and the like.
It should be noted that in the examples of robot functionality described above, an exhaustive list of security operations that a robot can perform is not given for each example for the purposes of simplicity. However, it should be noted that any security operation described herein or otherwise can be performed by the robot in any context described herein, and that the lack of description of a particular security operation in a particular context does not indicate that the robot is unable to perform the security operation in that context.
Semantic Mapping
As noted above, the semantic mapping system 736 of the robot 100 can generate or update a semantic map associated with a location or setting in which the robot is located. As used herein, a “semantic map” refers to a map that includes one or more objects, entities, or individuals, and that includes a meaning, description, identity, or status of the identified objects, entities, or individuals. Semantic maps generated by the semantic mapping system can be stored locally by the robot, or can be uploaded to the central system 210 for storage in the semantic maps storage module 342.
A. Semantic Map Generation
In some embodiments, the robot 100 can generate a semantic map from scratch, while in other embodiments, the robot can update an existing semantic map or can generate a semantic map from a non-semantic map, for instance by adding semantic details (such as information describing the location, identity, status, or other characteristics of detected objects) to the non-semantic map. In embodiments, where the robot generates a semantic map from scratch, the robot first generates an area map identifying obstacles and pathways using, for instance, one or more of the scanners 726 (such as a laser rangefinder or other suitable depth sensor) and a simultaneous localization and mapping (or “SLAM”) algorithm. As used herein, such maps are referred to as “SLAM maps”.
The map 1700 also includes an unknown area 1704 that is not yet scanned by the robot 100. This area may be navigable, though because the robot has not yet scanned the area, potential obstacles within the area 1704 have not been identified. Unknown areas can be represented within the map using a second color, texture, or shade, allowing a viewer of the map to distinguish between the navigable areas 1702 and the unknown areas 1704. Finally, the map also includes obstacles 1706, representing areas the robot cannot navigate through without potentially being obstructed or impeded, or without damaging objects representative of the obstacles. In the embodiment of
In some embodiments, boundaries 1708 are included within the map 1700, for instance areas within a threshold distance of the obstacles 1706. Such boundaries can be indicated using a fourth color, texture, or shade. When the robot 100 uses a SLAM map to navigate an area (such as the floor of the building represented within the embodiment of
While SLAM or other areas maps (“non-semantic maps” hereinafter) can be useful for the purpose of robot navigation within an area, the rangefinders or depth sensors used by the robot 100 to generate the SLAM map do not generate information about what detected obstacles or objects are, or how the robot should handle or interact with the detected obstacles or objects. The robot can detect and classify or identify objects, and can determine a state or other characteristics of the objects. Information describing the classified/identified objects and the corresponding detected states or other characteristics of the objects are stored in conjunction with the non-semantic maps, in association with a location of the objects relative to the non-semantic map. The generation of a semantic map refers to the inclusion of information describing objects represented within the map (and their corresponding detected states and characteristics).
In some embodiments, objects can be detected and identified by the robot 100 using one or more sensors, for instance cameras, RFID readers, IR sensors, and the like. For example, the robot can capture an image of an object, and can apply image detection or classification algorithms to identify the object type. Likewise, the robot can analyze images of the object to determine a state of the object. For instance, if the robot detects a door, the robot can also detect whether the door is open or closed. In some embodiments, the identification of objects and object states or characteristics can be manually provided by a system administrator, a robot operator, an employee or individual associated with the robot, and the like. In some embodiments, the robot can access object identification and state information from a repository of object information, such as a building database identifying the location and expected state of various objects within the building. In some embodiments, object information can be received from other robots that have scanned locations and identified objects within the locations. Object information can also be received from accessing the central system 210, one or more infrastructure systems 220, or one or more security systems 230. For instance, for a smart door, the robot can query the state of the smart door (e.g., locked or unlocked) via the network 200. Likewise, the robot can query a building elevator system to determine the current floor of an elevator.
Examples of detected objects and corresponding object states and properties (if applicable) include:
In addition, a robot 100 can determine usage patterns of detected objects (such as any of the detected objects listed above), and information describing such usage patterns can be included within the semantic map. For instance, a robot can query a building system to determine the floors in a building most frequently visited by an elevator, can access a scheduling system of a building to determine historical usage patterns for conference rooms, or can monitor a kitchen area to determine the times of a day when the kitchen is the most crowded. Likewise, a robot can determine behavior or movement patterns of people within an area, and information describing such patterns can be included within the semantic map. For instance, a robot can query a security system to determine a list of people who frequently pass through an access gate, or can track individuals (for instance, by scanning the RFID badges of the individuals) to determine historical patterns of movement of the individuals.
In some embodiments, the robot 100 can determine a height of detected objects, and can generate a 3D semantic map based on the detected heights. For instance, the robot can, using rangefinders, cameras, or other sensors, determine the height of nearby walls, doors, windows, obstructions, and the like. Using these heights, the robot can generate a 3D model of a 2D semantic map, for instance by rendering a building's structure in 3D (e.g., walls, hallways, passage ways, and the like), and by projecting detected objects at a height relative to the rendered structure based on a detected height of the objects in the real world.
In the embodiment of
In some embodiments, the semantic map 1900 can display identification information as text within the semantic map. For example, room names, room numbers, the names of office owners, and the like can be displayed within or overlaid upon the semantic map. In some embodiments, the semantic map can highlight portions of the semantic map corresponding to identification information. For example, the portion of the map corresponding to the conference room 1906 can be highlighted a particular color, for instance a color corresponding to conference rooms, and the portion of the map corresponding to the restrooms 1920 and 1921 can be highlighted a different color, for instance a color corresponding to restrooms. In some embodiments, the patrol route 1950 can be displayed as a line or dotted line, as a colored path (the boundaries or width of which corresponds to the most likely path of the robot 100 as the robot patrols the route), or using any other suitable identifier. Portions of the semantic map can also be color-coded based on access credentials required to access the portions, based on a type of room or map portion, based on a current occupancy of rooms or map portions, or based on any other suitable criteria. In addition to displaying text or color representative of the identification information, the semantic map can include icons or other visual indicators representative of the identification information or various types of identification information, and can further include a legend within the display of the semantic map identifying the meaning of each color, icon, and the like used to represent the identifying information.
Although reference is made herein to the display of identification information by the semantic map 1900, it should be noted that in some embodiments, such information is not immediately visible to a viewer of the map. For instance, the semantic map may be displayed without text or other identifiers, and may display text and other identifying information only upon a mouseover of an object, a selection of an objection, a configuration of the display of the semantic map into a particular mode that displays the identifying information, and the like. In embodiments where the semantic map is not displayed (for instance, embodiments in which the robot 100 uses the semantic map to perform a task or security operation), the information identifying objects (and object location, state, and characteristics information) can be stored within metadata of the map, stored in a separate data construct corresponding to the map, or stored in any other suitable way such that the information is accessible to an entity viewing or using the semantic map.
It should be noted that although state information or object characteristic information is not illustrated in conjunction with the objects in semantic map 1900 of the embodiment of FIG. 19, the semantic map can include such information. State information or object characteristics information can be presented to a user viewing the semantic map (for instance, in an overlay interface or an interface adjacent to the semantic map), and can be displayed any time the semantic map is displayed, or in response to a request for such information (for instance, in response to the selection of an object displayed within the semantic map). For example, the semantic map can include information identifying the door 1904 as currently closed and unlocked, and identifying the windows 1905 as open. Likewise, the semantic map can store a maximum occupancy of the conference room 1906, an owner of the office 1908, individuals assigned to desks within the cubicle arrays 1922 and 1924, requisite credentials required to access the security area 1928, and the like.
B. Semantic Map Usage
In some embodiments, a floor plan for a building floor can be generated based on a semantic map generated by the robot 100. Likewise, a semantic map can be generated by the robot based on a building floor plan. As used herein, a floor plan can include a SLAM map. Locations of needed informational signage within the building floor can be identified using the semantic map, for instance wheel-chair ramp access signs, conference room names, restroom signs, and navigational signs (for example, indicating a direction to the kitchen). In some embodiments, a robot, security personnel 250, or any other individual can use a semantic map to identify any safety equipment (such as fire extinguishers) or building infrastructure (such as elevators) that require inspections soon, or whose inspections are past due.
A robot 100 can navigate within an area using a semantic map or a generated floor map, for instance by selecting a route that avoids obstacles (e.g., by a threshold distance), by selecting routes that avoid high-trafficked areas, by selecting routes that maximize the robot's exposure or proximity to high-value assets or other objects, and the like. In some embodiments, the robot can plan a route through an area (such as a building floor) in advance using the semantic map, or can dynamically adjust a route by querying the semantic map to identify an alternative route to a location (for instance, in the event that a route is blocked or in the event that suspicious activity or a security violation is detected).
Semantic maps can include permissible locations within map portions for detected objects. For instance, for high-value objects, such as computer equipment or sensitive documents, a semantic map can identify rooms in which the high-value objects are allowed to be taken. A robot 100 can detect an object within a building (for instance, while patrolling a route through the building), can query a semantic map of the building with an identifier identifying the detected object to determine portions of the building in which the detected object is permitted to be, and can determine whether the detected object is located in a location in which it is not permitted to be. In such instances, the robot can notify security personnel 250 to relocate the object to a permissible location and can monitor the object until the security personnel arrive. In some embodiments, the robot can physically relocate the object to a permissible location, for instance using one or more robotic arms 740. In some embodiments, the robot, while patrolling a route, can detect all objects within an area (for instance, by scanning RFID tags coupled to the objects), and can use the semantic map to identify any detected objects within the area that are not permitted to be within the area, to identify any detected objects that are misplaced (for instance, after an office move from a first location to a second location), or to identify any detected objects flagged as lost.
Semantic maps can be used by a robot 100 to determine if particular rooms are over capacity. For instance, if a semantic map indicates that a particular conference room as a maximum occupancy of 10 people, a robot can determine that the conference room is over-capacity in response to detecting more than 10 people within the room. In such instances, the robot can inform the occupants that the room is over-crowded, or can flag the room for manual inspection by security personnel 250. In some embodiments, the semantic map can be used to identify historically under-utilized rooms, for instance at particular times of the day. In such embodiments, the identified under-utilized rooms can be prioritized for future room reservations, for instance by suggesting such rooms to individuals attempting to schedule meetings.
Semantic maps can be used to identify objects or infrastructure in needed of maintenance or repair. For instance, a robot 100 can use a semantic map to identify light bulbs that need to be replaced, leaks that need to be fixed, cracks within walls that need to be fixed, trash cans that need to be emptied, and the like. In response to identifying the objects or infrastructure in need of maintenance or repair, the robot can request that maintenance or building personnel perform the maintenance or repair. It should be noted that in some embodiments, a robot can perform these tasks without the use of a semantic map (for instance, by detecting a burnt out light bulb or a full trash can while patrolling a route). Likewise, it should be noted that in some embodiments, the semantic map can identify a location of objects (such as light fixtures or trash cans) within an area, but not the current state of such objects, and the robot can patrol a route within a proximity of each of the locations of the objects to determine if the objects need maintenance or repair (for instance, by detecting whether a light bulb is burn out or a trash can is full). In some embodiments, robots can regularly, periodically, or in response to a manual request, patrol maintenance routes based on a location of maintenance-related objects identified by a semantic map. It should also be noted that as described herein, the robot can update a semantic map in response to detecting events or characteristics of a location associated with the semantic map (e.g., by indicating a location of a burnt out light bulb, a broken window, etc.)
The robot 100 can use locations of likely false positive detections identified by a semantic map to determine if a detected object or individual is the result of a false positive detection. For instance, if a robot scans a wall poster of a person and identifies the person within the poster as an individual, the robot can query a semantic map to see if the location of the poster is associated with frequent or possible false positive detections of individuals. In such embodiments, the robot can flag the potential false positive detection for manual classification by an operator of the robot, can dismiss the detected individual as a likely false positive, can perform additional scans (such as heat scanning, IR scanning, or depth detection) to aid in the classification, or can perform any other suitable action in response to the semantic map identifying the location as associated with false positive detections. In some embodiments, in response to one or more false positive detections within a particular location, the robot can institute a “time-out” period, where subsequent detections within the location are ignored or classified as also likely false positive detections.
The robot 100 can measure various signals within a location, for instance while moving through the location or patrolling a route, and can aggregate the measured signals over time to obtain average signal values for particular locations. For instance, the robot can measure temperature, WiFi signal strength, detected network activity, the location and strength of Bluetooth or RFID beacons and devices, carbon monoxide levels, and the like. The aggregated information can be included within a semantic map, for instance as a heat map, where color or shade is used to represent values of the aggregated information (e.g., the color red can be used to represent high levels of WiFi signal strength, the color blue can be used to represent low levels of WiFi signal strength, and colors in between can be used to represent middle levels of WiFi signal strength). The robot can use a semantic map that includes aggregated or historical signal information to perform one or more functions or security operations. For example, if the robot loses connectivity to the network 200, the robot can use the semantic map to identify a location within a floor that historically has strong WiFi signal strength, and can move to the identified location to reconnect with the network. The loss of connectivity implies a loss of the inability of a robot operator to control the robot; thus, the ability of the robot to autonomously relocate to reconnect to the network beneficially enables the robot to restore the ability of the operator to control the robot.
Continuing with the above example, the robot 100 can use a semantic map that includes aggregated or historical signal information to identify locations with abnormally high or low temperatures (which may indicate a security violation, a broken furnace or air conditioning unit, an open or broken window, or the like), to identify locations with dangerous levels of carbon monoxide, to identify areas with abnormally high levels of network activity (which may indicate a security violation), or to identify any other signals that vary from historical averages or ranges. In such embodiments, the robot can perform a security operation (for instance, investigating a cause of the abnormal signal values, requesting security personnel 250 to come investigate, or performing an action or security operation using an infrastructure system 220 or security system 230), can trigger a fire alarm or fire sprinklers (for instance, in the event that abnormal levels of smoke or carbon monoxide are detected, in order to evacuate the premises), can request maintenance personnel to investigate (for instance, in response to determining that a heating or cooling system is broken), or can update the semantic map to include the abnormal signal information.
In some embodiments, semantic maps can include information identifying an individual's historical movements or behaviors. In such embodiments, the robot 100 can identify an individual within a location and/or performing an activity, and can use a semantic map to determine if the user is in a location outside of the historical norms of the user's movements or if the user's activity is outside of the historical norms of the user's behavior. In both instances, the robot can determine that the user may be acting suspiciously, and may represent a security policy violation. In response, the robot can perform a security operation, such as following and monitoring the individual, requesting that a security system 230 record video of the individual, locking nearby doors, and the like.
The robot 100 can update a semantic map as the robot moves within an area, for instance as the robot patrols the area. For example, as the robot detects objects, object states, and object characteristics during a patrol route, the robot can update a semantic map with information representative of the detected objects, states, and characteristics. As the robot patrols a route, the robot can query such a semantic map to determine if a detected object is in a different location than indicated by the semantic map, to determine if the object is in a different state that indicated by the semantic map, or to determine if the characteristics of the object are different than indicated by the semantic map. For instance, if the robot detects a window and determines that the window is closed on a first patrol of a route, updates a semantic map to indicate that the window is closed, then subsequently determines that the window is open on a second patrol route, the robot can query the semantic map to determine that the state of the window has changed between patrol routes. In such instances, if robot uses a semantic map to determine if the location, state, or other characteristics of a detected object changes, the robot can identify the change as a potential security threat or policy violation, and can perform one or more security operations in response.
The robot 100 can use a semantic map to identify infrastructure systems 220 and security systems 230 closest to a particular location. For instance, if the robot detects a potential intruder within a building at a given location, the robot can query the semantic map to identify security cameras closest to the given location, and can instruct the security cameras to look for and record video of the potential intruder. Likewise, if the robot detects smoke or high levels of carbon monoxide, the robot can query the semantic map to identify the fire detectors closest to the robot, can request smoke detection information from the identified fire detectors, can query the semantic map to identify fire sprinklers closest to the robot in the event that the fire detectors also detect smoke, and can activate only the identified fire sprinklers (beneficially reducing the damage caused by more distance fire sprinklers while still targeting the locations in which the smoke originates). Accordingly, the robot can query a semantic map with a location to identify infrastructure or security systems closest to the location, and can perform an action (such as a security operation) using the identified infrastructure or security systems.
RFID Management
As noted above, the RFID system 738 of the robot 100 can identify and determine the location of RFID tags within a proximity of the robot. In some embodiments, the robot can autonomously identify RFID tags within an area, beneficially enabling the robot to take inventory of objects coupled to the RFID tags, to identify missing RFID tags, to identify moved or misplaced RFID tags, and to update an inventory database based on the identified RFID tags. Although reference will be made herein to retail embodiments (with retail inventory coupled to RFID tags), it should be noted that the principles described herein apply equally to any embodiment in which the location and tracking of RFID tags is desirable, such as warehouse environments, office environments, laboratory environments, high security environments, and the like. It should also be noted that while reference is made to RFID readers and tags herein, the description herein applies equally to any type of signal emission and detection devices.
A. RFID Tag Detection
The robot 100 navigates within a location (such as a retail store or building floor) and identifies RFID tags within the location. In some embodiments, any number of robots can identify RFID tags within the location, for instance one or more ground-based robots and/or one or more aerial systems. Robots with RFID reading capabilities can beneficially detect RFID tags coupled to objects that tend to be relocated or moved, can beneficially increase an RFID detection area by virtue of the movement of the robot, and unlike stationary or wall/ceiling-mount readers, aren't limited in coverage scope.
As noted above, RFID tags can be coupled to retail inventory (for instance, clothing, electronics, books, and the like), warehouse inventory (such as boxes, crates, and the like), important assets (such as lab equipment, computer equipment, and the like), or to any suitable object. RFID tags can be passive (providing identification information at short distances in response to receiving a signal from a reader's interrogating radio waves) or active (with a local power source configured to provide identification information at much longer distances, and without requiring the receipt of a signal from a reader). In some embodiments, RFID tags can be stickers applied to a surface or object, while in other embodiments, RFID tags can include larger circuitry, a housing, a power source, and the like.
As noted above, the robot 100 can include reader antennas 730 to identify or detect RFID tags, and can include direction of arrival (or “DoA”) antennas 732 to determine a location of the detected RFID tags. The RFID reader antennas of the robot can be distributed along a height of the robot, enabling the robot detect RFID tags at a plurality of heights. In some embodiments, the locations of the reader antennas are fixed, while in other embodiments, the reader antennas are configured move along a height of the robot, beneficially enabling the reader antennas to dynamically scan a plurality of heights for potential RFID tags. Likewise, the RFID DoA antennas can be fixed at various locations along a height of the robot, for instance at a top of the robot, on an exterior of the robot, or within the robot body. In other embodiments, the location of the DoA antennas can be changed dynamically. In embodiments where the robot can adjust a height of the reader antennas or DoA antennas, the reader or DoA antennas can be located on a moveable or extendible arm, or can be located on a moving surface within or on the outside of the robot.
In some embodiments, the reader and DoA antennas can be distributed around a circumference or perimeter of the robot 100. For instance, in the embodiment of
In some embodiments, the RFID antenna arrays can be located on a boom arm that can extend outward and away from the robot, beneficially providing coverage of locations outside the dimensions of a robot 100.
To locate items coupled to RFID tags, the robot 100 can scan areas in a direction and height corresponding to locations where items coupled to the RFID tags can be located. For example, for a circular rack within a store, the robot can circumnavigate around the rack, and can aim a reader antenna array at a height corresponding to the rack (or a threshold distance below the rack to compensate for the height of a hanger) to scan for RFID tags. Likewise, for a set of shelves, the robot can aim a reader antenna array at heights corresponding to each shelf, or can “sweep” the shelves by adjusting a height of the reader antenna array from a top of the shelves to the bottom (or vice versa). In some embodiments, the robot can perform a 3D scan of infrastructure within a store to identify potential locations for store inventory, and can aim reader antenna arrays based on the identified potential locations to scan for RFID tags. Likewise, the robot can access pre-existing 3D data for store infrastructure to identify the potential locations to scan for RFID tags. By identifying potential locations for RFID tags, the robot can also determine the location of items coupled to the RFID tags without the need for a DoA antenna array. For instance, if the robot identifies a set of cubby holes within a shelving structure, the robot can scan each cubby hole, and in response to scanning a cubby hole and identifying an RFID tag, the robot can determine that the location of the identified RFID tag is the scanned cubby hole.
B. RFID Tag Localization
The robot 100 can determine a location of a detected RFID tags in a number of ways. In some embodiments, the robot can determine a location of a detected tag by evaluating a proximity of the robot to potential tag locations. In such embodiments, if only one potential RFID tag location is within a threshold distance of the robot at the time the tag is detected, the robot can determine that the detected RFID tag is located at the potential RFID tag location. Likewise, if many potential RFID tag locations are within a threshold distance of the robot at the time the tag is detected, but only one of the potential RFID tag locations is historically associated with the detected RFID tag, the robot can determine that the detected RFID tag is located at the potential RFID tag location historically associated with the detected RFID tag.
The robot 100 can also determine a location of a detected RFID tag based on two instances of detecting the tag. For instance, the robot can detect the RFID tag a first time when navigating a first straight-line path within a proximity of the tag. The robot can determine a first location of the peak signal strength of the detected RFID tag, and can identify a first direction to the detected tag as the direction perpendicular to the navigation direction of the robot while navigating the first path. The robot can detect the RFID tag a second time when navigating a second straight-line path with a proximity of the tag, can determine a second location of the peak signal strength of the detected RFID tag while navigating the second path, and can identify a second direction to the detected tag as the direction perpendicular to the navigation direction of the robot while navigating the second path. The robot can determine that the location of the detected RFID tag is the location of the intersection of a first line defined by the first location (of the first peak signal strength) and the first direction and a second line defined by the second location (of the second peak signal strength) and the second direction. As used herein, “peak signal strength” can refer to peak received signal strength indicator or “RSSI”, or can refer to any suitable measure of signal strength.
In some embodiments, the robot 100 can determine a location of a detected RFID tag by detecting a first peak signal strength of the RFID tag, by moving in a first direction perpendicular to the direction of movement of the robot at the location of the first peak signal strength, and by determining a second peak signal strength of the RFID tag. In the embodiment that the second peak signal strength of the RFID tag is less than the first peak signal strength, the robot can return the location of the first peak signal strength and move in a direction opposite the first direction. The process can be repeated, allowing the robot to iteratively move closer to the detected RFID tag until the robot is able to reduce the distance between the robot and the RFID tag to less than a threshold distance. The robot can then identify the location of the detected RFID tag, or can associate the location of the robot with the RFID tag (for instance, if the threshold distance is less than a meter, a foot, or any other suitable distance).
Alternatively, the robot 100 can include two directive patch antennas pointed in different directions such that the beamwidths of the antennas overlap at least somewhat. When an RFID tag is detected, the robot can measure a signal strength for each directive patch antenna, and can move in a direction of the antenna that measures a greater signal strength. This process can be repeated, allowing the robot to incrementally move closer to the detected RFID tag. The robot can then identify the location of the detected RFID tag, or can associate the location of the robot with the RFID as described above.
In some embodiments, the robot 100 can build a Bayesian sensor model for potential RFID tag locations (for instance, for a rack, shelf, changing room area, and the like) that predicts RFID tag signal measurements based on the expected distances and angles between the RFID reader antennas of the robot and the RFID tags at the potential RFID tag locations. The Bayesian sensor model for a potential RFID tag location can be built based on historical RFID tag signals detected by the robot as the robot moves in relation to the potential RFID tag location. The Bayesian sensor model can map one or more of detected RFID tags, RFID signal strengths, and robot locations to likelihoods that the RFID tag is located at one or more particular locations. Over time, the robot can generate a Bayesian sensor model for every potential RFID tag location within a store, and can select a Bayesian sensor model for use in locating an RFID tag based on a proximity to a potential RFID tag location at the time the robot detects the RFID tag being localized.
When the robot subsequently detects an RFID tag in proximity to the potential RFID tag location, the robot can select a Bayesian sensor model from a set of Bayesian sensor models based on a proximity to potential RFID tag locations, and can query the Bayesian sensor model with one or more of the detected RFID tag, the signal strength of the detected RFID tag, and the location of the robot at the time the RFID tag was detected. The Bayesian sensor model can return a set of potential RFID tag locations and corresponding likelihoods that the detected RFID tag is located at each location, and the robot can select among the potential RFID tag locations to locate the detected RFID tag. For instance, by selecting the most likely potential RFID tag location or by selecting a set of the most likely potential RFID tag locations. In some embodiments, the robot can move to another location and re-detect the RFID tag, and can re-query the Bayesian model using the re-detected RFID tag signal to obtain a second set of potential RFID tag locations and corresponding likelihoods, can combine the probabilities of the first set of potential RFID tag locations and the second set of RFID tag locations, and can select a potential RFID tag location based on the combined probabilities (for instance by selecting the most likely potential RFID tag location).
In some embodiments, stationary/permanent reference RFID tags within a store can be used to localize other RFID tags. For instance, a reference RFID tag can be placed on each end of a shelf. When the robot 100 detects an RFID tag, the robot can also detect one or both of the reference RFID tags. The robot can identify the location of the detected RFID tag as located on the shelf based on the detected reference RFID tags. In some embodiments, the robot can localize the detected RFID tag based on the strength of the detected referenced RFID tag signals. For instance, if the robot weakly detects one of the reference RFID tags and doesn't detect the other, and strongly detects the RFID tag being localized, the robot can determine that the RFID tag being localized is not located on the shelf (since otherwise the reference RFID tags would be detected more strongly). Likewise, if the robot detects a first reference RFID tag strongly, detects the second reference RFID tag weakly, and detects the RFID tag being localized strongly, the robot can determine that the detected RFID tag being localized is located on the side of the shelf closest to the first reference RFID tag.
As noted above, in some embodiments, the robot 100 can include a DoA antenna array to determine a location of a detected RFID tag. DoA antenna arrays can include a plurality of DoA antennas spaced a distance apart such that the antennas can detect a difference in phase of received RFID signal. In some embodiments, each DoA antenna pair is spaced a distance that is less than or equal to a quarter of the RFID signal wavelength. The robot can estimate a direction of arrival of the RFID signal based on the difference in signal phase detected by a DoA antenna pair. However, the estimated direction of arrival of the RFID signal represents the actual direction of arrival for the RFID signal, or 180 degrees from/the opposite of the direction of arrival for the RFID signal. To unambiguously identify the direction of arrival, a third DoA antenna can be included within the DoA array. In such embodiments, the third DoA antenna can detect the RFID signal, and if the detected RFID signal by the third DoA antenna is stronger than the RFID signal detected by one or both of the DoA antenna pair, the estimated direction of arrival for the RFID signal can be determined to be the actual direction of arrival for the RFID signal. Alternatively, if the detected RFID signal by the third DoA antenna is weaker than the RFID signal detected by one or both of the DoA antenna pair, the actual direction of arrival for the RFID signal can be determined to be 180 degrees from (or the opposite of) the estimated direction of arrival.
A DoA antenna array can include any number of DoA antennas (for instance, three, four, six, eight, or more), and can be located within or external to the robot 100. For instance, the DoA antenna array of the robot 100 can include four DoA monopole antennas equally spaced around a top surface circumference of the robot. For example, in the embodiment of
In some embodiments, the DoA antenna array can include a first set of antennas to determine an azimuth direction of arrival for RFID signals (e.g., a horizontal angle of arrival relative to the horizon), and can include a second set of antennas to determine an elevation direction of arrival for RFID signals (e.g., a vertical angle of arrival relative to the horizon, or “elevational direction of arrival”). In such embodiments, the first set of antennas can be oriented perpendicular to the second set of antennas. Using two sets of DoA antennas can enable the robot 100 to determine an exact direction of a detected RFID signal (e.g., not just a direction of arrival angle parallel to the floor, but also including an angular displacement upward or downward relative to the floor).
In addition to detecting a direction of arrival for detected RFID signals, the robot 100 can determine a distance between the robot and the corresponding RFID tag. In some embodiments, the robot can determine a signal strength of the detected RFID signal (e.g., the “RSSI” of the detected signal), and can determine a distance between the robot and the corresponding RFID tag based on the determined signal strength. In some embodiments, the robot can perform frequency modulation continuous wave (“FMCW”) ranging on the detected RFID signal to determine a distance between the robot and the corresponding RFID tag.
In embodiments where the robot 100 can determine an azimuth direction of arrival for a detected RFID tag and a distance between the robot and the tag, the robot can identify a location of the RFID tag within a two-dimensional map (for instance, a floor plan of a retail store). Likewise, in embodiments where the robot can determine an azimuth direction of arrival, an elevational angle of arrival, and a distance to the RFID tag, the robot can identify a location of the RFID tag within a third-dimensional map (for instance, a three-dimensional rendering of a retail store).
C. Robot Behavior in RFID Environments
When scanning for RFID tags within a retail store, the robot 100 can plan a path through the store based on historical locations or RFID tags and/or based on known possible locations for RFID tags. For instance, if a store includes a set of infrastructure objects configured to hold store inventory, the robot can select a route through the store that comes within a threshold distance of each of the set of infrastructure objects. As used in this context, “threshold distance” refers to a distance within which the robot is likely to detect an RFID tag (for instance, at a 90% likelihood, a 95% likelihood, a 98% likelihood, or any other suitable likelihood). This threshold distance can also be based on the RFID reader antenna array strength, the transmission strength of each RFID tag, or any other suitable factor.
The robot 100 can also select a route through the store that comes within a threshold distance of each historical location of RFID tag within the store. The robot can also plan a path through a store based on sales data for the store. For example, if store sales registers indicate that a set of items has been sold, the robot can select a route through the store that comes within a threshold distance of the historical locations of each of the sold set of items. Likewise, the robot can select a path through a store that comes within a threshold distance of all items that sell above a threshold rate (for instance, a daily or hourly rate).
The robot 100 can additionally plan a path through a store to scan for RFID tags based on a goal of the scan. For instance, if the goal of the robot is to quickly scan for all possible RFID locations within the store or within a portion of the store, the robot can select a path through the store that maximizes the number of items the robot comes within a threshold distance of while ensuring that the robot can navigate the path in less than a threshold amount of time. Such a path can beneficially enable the robot to quickly identify RFID tags within the store or portion of the store, but at the cost of a potentially inaccurate or incomplete scan. Likewise, if the goal of the robot is to identify a maximum number of RFID tags within the store or store potion, the robot can select a path that minimizes the distance between the robot and historical locations of each RFID tag.
In some embodiments, the robot 100 can navigate down every aisle in a store, or can navigate every second aisle if the reader antenna array of the robot is strong enough to detect RFID tags across aisles. In some embodiments, the robot can select a path through a store based on values of items coupled to RFID tags. For instance, the robot can select a path through the store that comes within a threshold distance of each item in a set of high-value items, beneficially increasing the likelihood that the robot will detect the RFID tags coupled to the set of high-value items when navigating the selected path. The robot can also select a path based on locations where items waiting to be returned to the sales floor may be located. For instance, the selected path may come within a threshold distance of each changing room within a store, each return rack with the store, and sales counters within the store.
The robot 100 can additionally select a path through a store to scan for RFID tags based on information describing historical foot traffic through the store. For instance, the robot can select a path that comes within a threshold distance of each historical RFID location while avoiding areas of above-threshold foot traffic. Likewise, if a particular passage within the store is less than a threshold distance wide, the robot can select a path through the store that avoids the particular passage. In addition, the robot, while navigating a path through the store to scan for RFID tags, can detect people within the store, and in response to detecting people blocking a particular passage of the path within the store, the robot can wait for the people to move before proceeding, can request that the people move, or can select an alternative route that comes within a threshold distance of historical RFID locations or determined potential RFID locations.
The robot 100 can scan for expected RFID tags coupled to retail inventory in a location within a retail store, and in response to scanning the location for the expected RFID tags and not finding the tags, can request that additional inventory be brought to the store floor to replace the missing inventory. For example, the robot can identify the missing inventory to the central system 210, which in turn can instruct store personnel to restock the missing inventory. Alternatively, the robot or the central system can instruct another robot to retrieve replacement inventory from storage, or the robot can itself retrieve the replacement inventory from storage and can restock the retrieved replacement inventory at the location with the retail store. In some embodiments, the robot can determine that replacement inventory is not locally available, or is in low supply, and can order additional replacement inventory from a warehouse, distribution facility, or other inventory source.
In some embodiments, the robot 100 can scan inventory storage to identify inventory in low supply, and can order replacement inventory as needed (for instance, if the quantity of replacement inventory falls below a threshold). Replacement inventory can also be ordered based on a rate of sales of the inventory. For instance, if the replacement inventory corresponds to an above-threshold rate of sales, the robot can order replacement inventory, while if the replacement inventory corresponds to a below-threshold rate of sales, the robot can order replacement inventory when the available replacement inventory falls below a threshold quantity. In some embodiments, the robot can order replacement inventory based on a combination of rate of sales of the inventory and the available quantity of replacement inventory.
The robot 100, in response to scanning for RFID tags while navigating a first path through a store and detecting a large discrepancy between expected RFID tags and detected RFID tags, can select a second, more comprehensive path through the store that maximizes the likelihood that RFID tags present within the store will be scanned. Such a configuration beneficially enables a robot to navigate shorter paths through a store most of the time, while still enabling the robot to navigate comprehensive paths through the store when needed. In response to a large discrepancy between RFID tags detected while navigating the second, more comprehensive path in a store and expected RFID tags in the store, the robot can flag the missing items coupled to the expected-but-not-scanned RFID tags as potentially stolen, and can notify store employees of the discrepancy.
Shoppers within a retail store can interact with the robot 100, for instance via the display 720 or by talking to the robot. For instance, if the display is a touch-screen display, the display can display the text “How can I help you?”, along with selectable options including “help me find an item”, “talk to the store manager”, or “sale items”. In response to the selection of an option by a shopper, the display can take one or more actions or access and display appropriate information. In some embodiments, the shoppers can speak directly with the robot, which can identify what the shoppers are saying and can take actions or provide information as needed. For example, if a user selects the “help me find an item” or asks the robot to show the location of an item, the robot can query the store's inventory database to determine if the item is still available for sale, can say “follow me”, can navigate to the last known location of the item within the store, can scan nearby RFID tags to identify an exact location of the item, and can highlight the item for the shopper (for instance, using a laser pointer or projecting an arrow on the ground to indicate the location of the item). In some embodiments, the robot can access parse and process user requests autonomously, but in embodiments in which the robot is unable to parse and process a user request, the robot can provide the request to an operator of the robot, who can instruct the robot to perform an action or provide appropriate information in response.
In some embodiments, the robot 100 can scan for RFID tags within a store to identify locations of RFID tags within the store, and can compare the identified locations with a semantic map including information identifying last known locations of the RFID tags or historical locations of RFID tags. In embodiments where the identified locations of the RFID tags differ from the locations within the semantic map, the robot can move items coupled to the RFID tags within the store to the locations identified by the semantic map, can flag the items coupled to the RFID tags along with the identified locations of the RFID tags as misplaced or moved for a store employee to relocate the items coupled to the RFID tags, or can update the semantic map to reflect the identified locations of the RFID tags. In other words, during the course of moving through a store, the robot can identify items that have moved, and can update a semantic map to reflect the movement, beneficially producing a semantic map that includes up-to-date locations for store inventory.
In some embodiments, the robot 100 can scan for RFID tags within a doorway of a room, for instance in order to determine if an item coupled to an RFID tag is located within the room. If the robot detects the RFID tag within the room, the robot does not need to enter the room, as the item coupled to the RFID tag has already been determined to be in the room. However, if the robot does not detect the RFID tag within the room, the robot can enter the room in order to more comprehensively scan for the item. For instance, if the room is a conference room with a large conference table, and the item coupled to the RFID tag is a projector, the robot can enter the conference room, can circumnavigate the conference table in order to determine if the projector is located on the table, and in response to detecting the RFID tag, the robot can update an inventory or semantic map to indicate the detected location of the project, can store the location of the projector in a database, or can notify an operator of the location of the projector. If, after entering the room and more comprehensively scanning for the projector, the robot is unable to detect the RFID tag coupled to the projector, the robot can flag the projector as potentially lost, can inform an operator or building personnel of the missing projector, or the like. Accordingly, the movement/navigation of the robot can be adjusted based on the detection or absence of the detection of an RFID tag.
The robot detects 2505 an RFID tag while navigating the selected route. The RFID tag can be detected using, for instance, an array of RFID reader antennas. The robot can then localize 2510 the detected RFID tag. As noted above, the detected RFID tag can be localized by detecting the RFID tag from multiple locations and extrapolating the RFID tag location, by identifying a set of potential RFID tag locations and selecting among the potential RFID tag locations, or by correlating the detected RFID tag with corresponding historical RFID tag locations to identify the detected RFID tag location. In some embodiments, the robot can include a DoA antenna array, and can determine the location of the RFID tag using the DoA antenna array to determine an azimuth direction of arrival of a received RFID signal, and/or determining an elevational direction of arrival of the received RFID signal. The robot can further determine a distance between the robot and the detected RFID tag, for instance based on a signal strength of the detected RFID signal.
The robot identifies 2515 an object coupled to the RFID tag. For example, in retail environments, the object can be an object for sale within the retail store, while in warehouse environments, the object can be a box or crate of inventory. The robot performs 2520 an operation associated with the identified object based on the detected RFID tag. In some embodiments, the operation includes updating an inventory database to include the identified object coupled to the detected RFID tag. In other embodiments, the operation can include updating an inventory database with the determined location of the detected RFID tag, informing a customer of the location of the identified object coupled to the RFID tag (for instance, in response to a request from the customer for the location of the RFID tag), informing a store employee of the location of the identified object, flagging the identified object within an inventory database for relocation, moving the identified object to a location identified by an inventory database and corresponding to the identified object, adjusting the selected route of the robot based on the location of the detected RFID tag, or any other suitable operation based on one or more of the detected RFID tag, the identity of the object coupled to the RFID tag, and the determined location of the RFID tag.
Summary
The foregoing description of the embodiments of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.
Some portions of this description describe the embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like, for instance, within a robot or robot system. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules or engines, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.
Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described. Further, the functionalities described herein can be performed by a hardware processor or controller located within the robot.
Embodiments of the invention may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
Embodiments of the invention may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.
Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.
This application claims the benefit of and priority to U.S. Provisional Applications No. 62/293,229 filed Feb. 9, 2016, No. 62/301,545 filed Feb. 29, 2016, and No. 62/350,713 filed Jun. 15, 2016, each of which are incorporated by reference herein in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3088157 | Aguilar | May 1963 | A |
| 8255092 | Phillips et al. | Aug 2012 | B2 |
| 8428777 | Poursohi et al. | Apr 2013 | B1 |
| 9069356 | Papaefstathiou et al. | Jun 2015 | B2 |
| 9216509 | Renkis | Dec 2015 | B2 |
| 9323250 | Wang et al. | Apr 2016 | B2 |
| 9830798 | Poursohi | Nov 2017 | B2 |
| 20010037163 | Allard | Nov 2001 | A1 |
| 20050216126 | Koselka et al. | Sep 2005 | A1 |
| 20060112034 | Okamoto | May 2006 | A1 |
| 20070192910 | Vu et al. | Aug 2007 | A1 |
| 20100268383 | Wang et al. | Oct 2010 | A1 |
| 20110035054 | Gal et al. | Feb 2011 | A1 |
| 20110148633 | Kohlenberg | Jun 2011 | A1 |
| 20120083923 | Matsumoto et al. | Apr 2012 | A1 |
| 20120197464 | Wang et al. | Aug 2012 | A1 |
| 20120215383 | Yoon et al. | Aug 2012 | A1 |
| 20120313779 | Papaefstathiou | Dec 2012 | A1 |
| 20130014219 | Kraus et al. | Jan 2013 | A1 |
| 20140207282 | Angle | Jul 2014 | A1 |
| 20140266669 | Fadell | Sep 2014 | A1 |
| 20140366676 | Kim et al. | Dec 2014 | A1 |
| 20150170509 | Artes et al. | Jun 2015 | A1 |
| 20150205298 | Stoschek | Jul 2015 | A1 |
| 20150290808 | Renkis | Oct 2015 | A1 |
| 20150379358 | Renkis | Dec 2015 | A1 |
| 20160075027 | Renkis | Mar 2016 | A1 |
| 20160129592 | Saboo et al. | May 2016 | A1 |
| 20160334802 | Sogen | Nov 2016 | A1 |
| 20170201617 | So et al. | Jul 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 1593859 | Mar 2005 | CN |
| 102253673 | Nov 2011 | CN |
| 103294061 | Sep 2013 | CN |
| 2987689 | Sep 2013 | FR |
| 20150033443 | Apr 2015 | KR |
| Entry |
|---|
| English Translation for reference KR20150033443A (Year: 2015). |
| PCT Invitation to Pay Additional Fees, PCT Application No. PCT/US17/17133, dated Apr. 24, 2017, 2 pages. |
| PCT International Search Report and Written Opinion, PCT Application No. PCT/US17/17133, 27 pages. |
| United States Non Final Office Action, U.S. Appl. No. 15/426,050, dated Jul. 6, 2018, 16 pages. |
| United States Non Final Office Action, U.S. Appl. No. 15/426,052, dated Apr. 6, 2018, 11 pages. |
| United States Non Final Office Action, U.S. Appl. No. 15/426,054, dated Jul. 19, 2018, 19 pages. |
| United States Final Office Action, U.S. Appl. No. 15/426,050, dated Dec. 6, 2018, 12 pages. |
| United States Final Office Action, U.S. Appl. No. 15/426,054, dated Dec. 4, 2018, 19 pages. |
| United States Final Office Action, U.S. Appl. No. 15/426,052, dated Sep. 6, 2018, 6 pages. |
| United States Office Action, U.S. Appl. No. 15/426,050, dated Feb. 21, 2019, 13 pages. |
| United States Office Action, U.S. Appl. No. 15/426,054, dated Feb. 21, 2019, 22 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20170225336 A1 | Aug 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62293229 | Feb 2016 | US | |
| 62301545 | Feb 2016 | US | |
| 62350713 | Jun 2016 | US |
