The present invention relates to surveillance devices.
Historically, visual surveillance systems were designed as permanent additions to existing site infrastructure, and they were intended to protect assets and monitor activity in and around buildings and grounds by adding audio-video capture. They typically would extend existing security installations (comprised of specialty sensors for contact, motion, heat, chemicals, water, etc.), but they might also entail entirely new custom installations. Because of the significant investment of time and complexity of installing, such systems were generally permanent and unchanging without major redesign. As a result, cameras needed to be carefully placed at predefined, strategic locations within or on buildings or on poles and other structures that could provide good vantage points. And until recently sensors were hardwired to data collection nodes and monitoring centers, so reconfiguration was difficult and time consuming.
One advantage of such fixed-location/stationary camera installations is that camera positions can be precisely known, and therefore the relative positions of objects and events detected in the scene can be precisely inferred. Temporal synchronization is relatively straightforward as cameras operate on a common clock. And knowing the exact positions of cameras relative to one another facilitates reliable integration of events moving between the fields-of-view (FOVs) of different cameras.
The advent of low-cost, hi-res digital cameras and the emergence of robust wireless broadband technologies such as WiFi and 4G/5G enable more flexible positioning and repositioning of cameras. However, stationary cameras remain susceptible to occlusions (both permanent and transitory), adverse lighting effects (such as glare and shadowing), and lack of sufficient resolution at distance. As a consequence, some modern surveillance systems have come to incorporate mobile cameras—i.e., cameras mounted on moving platforms—which can provide more comprehensive data gathering and more detailed views of a particular area or situation.
Mobile platforms may be distinguished according to their navigation control paradigm (manually driven, tele-operated, or autonomously driven) and according to their operating environment (air, land, or sea). Waterborne vehicles are generally an exclusive concern of naval operations and have a different set of concerns from autonomous ground vehicles (AVGs). While airborne vehicles might ultimately enhance site surveillance, they are currently subject to shifting FAA regulations and local-varying legal operating restrictions.
Mobile camera platforms are capable of moving closer to events as needed and to attain better vantage points as circumstances permit. However it remains a major challenge to determine the exact location of the mobile platform and its cameras at any instant, especially while moving. While significant progress has been made using modern techniques such as sensor-fusion and SLAM (Simultaneous Localization and Mapping), large uncertainties can still be present due to drift of inertial sensors and lack of sufficient positional resolution of public GPS.
This lack of accurate positioning has prompted some vendors to develop and install proprietary differential positioning systems. But such proprietary differential positioning systems tend to be site-specific and are therefore brittle and costly to deploy. More recently, some vendors have announced visual surveillance products that are stationary or mobile, but these are not tightly integrated into a unified solution. They typically act as independent components with specific duties applicable to specific situations, and any integration happens through the backend.
More recently, there has been activity in so-called robot swarms. This trend has found particular appeal in the unmanned airborne vehicle (UAV) arena, and it can be viewed as an extension of classic parallel processing paradigms (in particular, SPMD—Single Program Multiple Data—models) to robotics. Swarms are generally motivated by biological examples (ants, bees, etc.), where multiple identical agents with the same capabilities realize a multiplicative advantage by each working on a small chunk of the problem. In surveillance, this typically means sub-regions of the area to be surveilled would be assigned to swarm members in 1-to-1 or 1-to-many-fashion.
Needless to say, coordination of a swarm's tasking can be complex. More significantly, since all agents have the same capabilities (both strengths and weaknesses), the swarm is not able to directly compensate for individual shortcomings. Instead, the swarm works through redundancy of effort, seeking to overwhelm the problem through brute-force rather than exploiting complementary capabilities. In addition, the swarm approach does not explicitly embody a means to rapidly deploy to a given site.
The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.
The following figures serve to illustrate various embodiments. It should be noted that modifications of size, placement, numbers of elements, etc. may be used to provide similar functionality. So for example, the one embodiment shows the pod employing a 3-sectioned sensor mast. But a 2-sectioned or single-sectioned sensor mast should be considered essentially equivalent as they are both intended to provide an advantageous viewpoint for the pod's surveillance sensors. The mast could also be telescoping, allowing for more rapid deployment in time-critical situations. Likewise, the erected mast is shown located on the forward deck in the starboard position. If it is deemed more advantageous or otherwise desirable to erect the mast (or even to add masts) at central or port positions, these variations should be considered equivalent insomuch as they provide good coverage for the pod's sensors.
In the figures, curved arrows refer to elements that are hidden or interior to what is shown, while straight arrows refer directly to what is visible. For assistance, a figure labelling taxonomy is provided in the following tables. This consists of leading “agent” labels and trailing “element” labels. Substituting 1 for x in the leading position refers to a pod element; substituting a 2 refers to a rover element. Thus, 1.1 indicates the pod sensor mast, while 2.1 indicates the rover sensor mast.
In general, the system includes a heterogeneous pair of autonomous agents, with complementary functionality, termed an Autonomous Surveillance Duo. A duo typically consists of a stationary surveillance platform termed a pod and a mobile surveillance platform termed a rover. The duo supports rapid and flexible deployment for ad-hoc surveillance situations, but it also supports seamless, flexible integration with existing security installs. The pod has a cargo bay, which houses hardware components integral to the pod's mission, including: electrical charging and storage units; computational and communication units; and environmental maintenance components, such as cooling fans. The bay also houses the rover itself during transport, and an integral trailer chassis with detachable tongue allows both the pod and the rover to be easily hauled by a service vehicle to the surveillance site. Once setup and calibrated, the pair is able to perform autonomous, joint surveillance of a designated zone on site. The respective strengths of each agent enhance the capabilities of the other. Thus, the precise geo-referencing of the pod's position assists in accurately locating the rover as it moves. And the rover's ability to move about and gain advantageous imaging positions provides a more complete assessment of the state of the zone than would otherwise be possible from a single stationary vantage point.
The pair of autonomous agents enables effective ad-hoc visual surveillance, which can be rapidly deployed and readily reconfigured.
The pair of autonomous agents may be used as “permanent” enhancements of existing/legacy surveillance installs (with expected service on the order of years).
The pair of autonomous agents may be used as semi-permanent surveillance of limited-duration events such as monitoring construction sites (with expected service on the order of months).
The pair of autonomous agents may be used as temporary surveillance of short-term, planned events such as outdoor concerts, fairs, etc. (with expected service on the order of days.
The pair of autonomous agents may be used as rapid-response deployment to surveil unplanned, emergency situations such as crime scenes or accident scenes (with expected service on the order of hours).
Rover autonomy facilitates programmatic coordination with its pod partner for successful duo surveillance. This permits security/surveillance personnel to attend to higher-level tasks rather than having to directly operate the rover, and in some cases it may reduce the number of personnel required.
Referring to
A fully deployed rover 2 is shown with its telescoping mast 2.1 raised and its sensor cap 2.1.1 mounted atop the mast. Here again the mast is hollow so that cables can be run up the center to provide power and data linkage to the cap. A stereo SLAM camera rig 2.2, located forward of the sensor mast, provides visual navigation input. Forward driving lights 2.3 are located at the front of the rover. The rover's body capsule 2.7 is also indicated. The rover 2 has a set of four wheels 2.0. The capsule houses all of the rover's hardware subsystems for surveillance, communications, navigation, and power management.
Referring to
Referring to
Referring to
Referring to
The rear elevation of
Referring to
Referring to
Referring to
The rear elevation of
Referring to
The elevation in
The front driving lights are shown with protruding cowlings both to indicate that shielding driving lights from weather and environmental elements is desirable and to assert that protrusion of elements at the vehicle's front is permitted, but not necessary. Thus, an alternative embodiment could employ recessed driving lights with lens covers.
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
The original panels 1.4A and additional panels 1.4B are mounted bilaterally (i.e., on both the port and starboard sides). In this configuration, the pod is oriented so that the sun's arc crosses from one set of side panels to the other. This configuration shows the pod oriented with its front (indicated by its forward deck 1.2.1) facing the sun's direction, so the sun crosses from port to starboard. The pod may also be oriented with its aft facing the sun.
Referring to
A similar configuration may be used for the rover's sensor cap.
Referring to
Referring to
At night, the headlights are preferably on during driving; the backup lights are additionally lit when the rover is reversing or surveilling; and the side surveillance lights are added when the rover is in surveillance mode.
Referring to
Referring to
Referring to
Referring to
Referring to
As illustrated and previously described, the system may include the autonomous surveillance duo comprising a stationary pod robot component (see 1 of
Duo surveillance may be a cooperative computation performed by a pod/rover pair. The pod/rover pair may detect and record significant zonal events within a common event database, and optional transmission of such event logs to an associated cloud for archiving and deeper analysis. The pod/rover pair may include heterogeneous, joint SLAM (simultaneous localization and mapping), which supports the progressive refinement of non-transient zonal features (landmarks, structures, terrain, vegetation, water bodies, etc.) within a common map database. Additional site-specific mission tasks such as: operator-specified “go-to-and-investigate” a specific location/event; traversing designated patrol routes; waypoint visitation; entry-point monitoring; gauge reading; and overall zonal situation awareness may be included.
The pod component may be positioned at a predefined, advantageous location within a designated zone to be surveilled, and it performs stationary surveillance of those zonal sub-regions that fall within range of its sensors—minimally visual and audio sensors. In one embodiment, the pod has a sectioned mast that is assembled and erected during deployment (see,
The pod performs as a local mission manager. Once positioned, it provides the origin of a local zonal coordinate system, and thus it supports integration of the data collected by both it and the rover. This integrated data provides a consistent assessment of the state of the surveilled zone. Because the pod is stationary, it can be accurately geo-referenced so that the data gathered by one zonal duo can be consistently integrated with the data of other duos operating in different zones on the site.
The pod also serves as a transport capsule so that the duo can be rapidly deployed to any desired surveillance location. The pod is effectively a trailer with hollow bay (see,
Once the pod is positioned, solar charging panels (see,
The pod further serves as a charging platform for both itself and the rover. With the pod is in its final position, the pod's rover-charging station (see,
For emergency situations, the pod can remain attached to the service vehicle (see,
The duo model also supports the case where only the pod (no rover) is deployed at the scene. The sensor mast and cap are erected, and the pod may be drawn behind the service vehicle and positioned at various locations where desired—as a sort of ersatz “rover”. This provides extremely fast deployment for time critical emergency situations.
The rover component is an autonomous ground vehicle (AGV) that initially disembarks from the pod and calibrates its map coordinate system with that of the pod using a combination of sensor modalities including vision, GPS, LIDAR, IMUs, and/or visual fiducials on the pod. The rover also localizes the pod charging station as indicated above so that it may reliably return and recharge as desired.
Stereo SLAM software and a stereo camera rig (see,
In addition to navigational cameras and sensors, the rover possesses a telescoping mast (see,
The rover monitors its own power levels and shares this data with the pod. Optionally, the rover has a solar panel embedded in its roof (see,
Daytime surveillance strategies are generally outlined previously. The pod and the rover exploit ambient illumination to perform visual surveillance and video capture during daylight hours. In areas of deep shadow, the rover may temporarily switch on lights for driving or to surveil.
Daytime power management strategies are also generally outlined previously. Both the pod and the rover maintain estimates of their individual power consumption rates and replenishment rates. The rover's data is forwarded to the pod whose power management subsystem estimates the optimal recharge scenario (timing and routing), based on the current rover tasking schedule, anticipated recharge rates, and/or predicted power usage curves.
During nighttime, the rover uses driving lights to visually navigate, and it may use additional illumination (see,
Because lights are generally used to surveil at night, power consumption should be more carefully monitored, and more conservative consumption strategies may be adopted. More especially, the solar panels will not be generating any additional power reserves. Apart from rover driving lamps, surveillance illumination may be event-driven. For example, lights for both the pod and the rover may be switched on by motion triggers and switched off with lack of motion. Another choice might be to use lower-powered IR LEDs as nighttime illumination.
As generally outlined previously, multiple duos may be used to adequately cover an entire site. This means that a duo may coordinate with other duos in neighboring zones to improve surveillance capability over the entire site and to overcome individual component failures. Some examples of inter-zone coordination are illustrated in
With event forwarding (see,
With pod failure (see,
With rover failure (see,
As previously described, the rover and pod enable flexible deployment of visual surveillance components where and when needed. It also facilitates the components being readily reconfigured as the surveillance requirements evolve over time. Nevertheless, the data and metadata preferably comply with existing surveillance data conventions, such as PSIM (Physical Security Information Management) and CSIM (Converged Security Information Management), so it readily integrates with existing installs.
Pairing of heterogeneous agents (pod-rover duo) integrally blends the individual strengths of stationary and mobile surveillance while mitigating their respective weaknesses. The stationary pod provides a reliable visual landmark and geo-referenced coordinate system with which the rover can be more precisely located. Conversely, the moving rover can surveil areas that are hidden from the pod's single stationary vantage point. In addition, both the pod and the rover offer surveillance of areas that may not be available with existing, fixed security installs.
Visual and instrumented triangulation of significant events by the duo pair enables more accurate localization of objects and events within the zone without the need for more expensive modalities like LIDAR. The stationary pod improves its GPS localization by integrating over time, and the pod in turn improves rover localization via differential GPS signaling.
A duo is designed to provide a fully functioning autonomous surveillance solution for a given zone 24/7. Complete and continuous coverage for an entire site can thus be achieved by adding more duos. This is in contrast to previous techniques that employ a patrol model—where agents roam about the site, but are only able to surveil what is in their current field of view.
Enablement of fully autonomous (unattended) charging of both the pod and rover delivers a completely self-sustaining surveillance solution that is cost-effective because it avoids the need for expensive human intervention. This is in contrast to previous techniques, which require periodic human intervention such as battery swaps to maintain functionality.
The design of the pod as a duo transport mechanism (with the rover riding securely inside the pod's cargo bay) supports rapid deployment and ad-hoc surveillance for temporary events and emergency situations.
Rapid deployment also supports rapid and cost effective recovery in the face of component failures (pod or rover). A replacement duo can be quickly rolled-out at minimal effort for urgent or time-critical situations.
Use of existing 4G (and later 5G, etc.) public cellular networks with data encryption provides robust bandwidth for secure transfer of surveillance data. Avoiding the need for a private network install provides increased flexibility, rapid deployment, and reduced complexity. Such high capacity networking also allows the pod and rover to work more efficiently together, and it supports site-wide cooperation among neighboring duos. It further allows offloading more intensive analyses to a cloud service as desired.
Use of open-source software and commercially available, off-the-shelf hardware components reduces delays due to software/component shortages and speeds delivery.
The removable towing tongue allows the duo to be hitched via standard ball hitch to a service vehicle for transport to the site when installed, but when removed and stowed inside the pod's cargo bay prevents easy theft of the duo.
The tow receiver channel that in one embodiment alternatively accommodates a removable towing tongue when deploying and a charge plate transmitter when deployed provides increased flexibility.
The pod door secures the pod bay and its contents when locked up, and when lowered serves as a loading ramp for the rover.
The inclined pod floor exploits gravity to facilitate unloading of the rover when deploying and to provide additional tension against the winch cable when transporting.
The longitudinal inner-track rails (on the door ramp and the inclined floor) guide the rover during loading/unloading and provide structural stiffness to the door (allowing it to be lighter). They also reduce the likelihood of the rover being stuck or damaged during transport and site deployment.
In another embodiment, a set of lights may be located on each of the four corners of the roof of the pod. Each of the lights may be movable and directable, as desired. Having the lights located in a such a position reduces the glare that may otherwise occur.
The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.
This patent application is a continuation of U.S. patent application Ser. No. 16/215,161, filed Dec. 10, 2018, which claims the benefit of U.S. Provisional Patent Application No. 62/699,024, filed Jul. 17, 2018, both of which are incorporated herein by reference in their entireties.
Number | Name | Date | Kind |
---|---|---|---|
7340076 | Stach et al. | Mar 2008 | B2 |
7852211 | Gunn | Dec 2010 | B2 |
7886646 | Bannasch | Feb 2011 | B2 |
7925049 | Zhu et al. | Apr 2011 | B2 |
8022812 | Beniyama et al. | Sep 2011 | B2 |
8134479 | Suhr et al. | Mar 2012 | B2 |
8174568 | Samarasekera et al. | May 2012 | B2 |
8238612 | Susca et al. | Aug 2012 | B2 |
8340901 | Fahn et al. | Dec 2012 | B2 |
8447863 | Francis, Jr. et al. | May 2013 | B1 |
8451165 | Puzella | May 2013 | B2 |
8464816 | Carrier | Jun 2013 | B2 |
8655513 | Vanek | Feb 2014 | B2 |
8660736 | Chen et al. | Feb 2014 | B2 |
8661605 | Svendsen et al. | Mar 2014 | B2 |
8663130 | Neubach et al. | Mar 2014 | B2 |
8705842 | Lee et al. | Apr 2014 | B2 |
8718838 | Kokkeby et al. | May 2014 | B2 |
8825387 | Mays et al. | Sep 2014 | B2 |
8874360 | Klinger et al. | Oct 2014 | B2 |
8918241 | Chen et al. | Dec 2014 | B2 |
8929604 | Platonov et al. | Jan 2015 | B2 |
8937410 | Comins et al. | Jan 2015 | B2 |
9026272 | Kokkeby et al. | May 2015 | B2 |
9117371 | Hutchings | Aug 2015 | B2 |
9157757 | Liao et al. | Oct 2015 | B1 |
20040167667 | Goncalves et al. | Aug 2004 | A1 |
20060055530 | Wang et al. | Mar 2006 | A1 |
20060056707 | Suomela et al. | Mar 2006 | A1 |
20060058921 | Okamoto | Mar 2006 | A1 |
20070027579 | Suzuki et al. | Feb 2007 | A1 |
20070061041 | Zweig | Mar 2007 | A1 |
20070100498 | Matsumoto et al. | May 2007 | A1 |
20070188328 | Mochizuki et al. | Aug 2007 | A1 |
20070271003 | Bang et al. | Nov 2007 | A1 |
20080077511 | Zimmerman | Mar 2008 | A1 |
20090012667 | Matsumoto et al. | Jan 2009 | A1 |
20090074545 | Lert, Jr. et al. | Mar 2009 | A1 |
20090207257 | Jung et al. | Aug 2009 | A1 |
20100076631 | Mian | Mar 2010 | A1 |
20100155156 | Finkelstein | Jun 2010 | A1 |
20110106312 | Chen et al. | May 2011 | A1 |
20120121161 | Eade et al. | May 2012 | A1 |
20130231779 | Purkayastha et al. | Sep 2013 | A1 |
20160026192 | Phillips | Jan 2016 | A1 |
20170094144 | Tomomasa | Mar 2017 | A1 |
20180005534 | Jesudason et al. | Jan 2018 | A1 |
20190043355 | Ferguson et al. | Feb 2019 | A1 |
20190373222 | Chiu | Dec 2019 | A1 |
Number | Date | Country | |
---|---|---|---|
20200358988 A1 | Nov 2020 | US |
Number | Date | Country | |
---|---|---|---|
62699024 | Jul 2018 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16215161 | Dec 2018 | US |
Child | 16941260 | US |