Patent Application
20190289263
Various embodiments concern computer programs and associated computer-implemented techniques for intelligently processing content generated by electronic devices such as security cameras, security lights, etc.
Surveillance is the monitoring of behavior, activities, or other changing information for the purpose of influencing, managing, or protecting people/items in a given environment. Generally, surveillance requires that the given environment be monitored by means of electronic devices such as security cameras, security lights, etc. For example, a variety of electronic devices may be distributed through the home environment to detect activities performed in/around the home.
Wireless security cameras have proved to be very popular among modern consumers due to their low installation costs and flexible installation options. Moreover, many wireless security cameras can be mounted in locations that were previously unavailable to wired security cameras. Thus, consumers can readily set up home security systems for seasonal monitoring/surveillance (e.g., of pools, yards, garages, etc.).
Various features and characteristics of the technology will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. Embodiments of the technology are illustrated by way of example and not limitation in the drawings, in which like references may indicate similar elements.
The drawings depict various embodiments for the purpose of illustration only. Those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technology. Accordingly, while specific embodiments are shown in the drawings, the technology is amenable to various modifications.
Surveillance is the monitoring of behavior, activities, or other changing information for the purpose of influencing, managing, or protecting people/items in a given environment. Surveillance often requires that the given environment be monitored by means of various electronic devices such as security cameras, security lights, etc. In some instances, surveillance systems (also referred to as “security systems”) are connected to a computer server via a network. Some content generated by a security system may be examined locally (i.e., by the security system itself), while other content generated by the security system may be examined remotely (e.g., by the computer server).
Generally, a network-connected surveillance system (also referred to as a “security system”) includes a base station and one or more electronic devices. The electronic device(s) can be configured to monitor various aspects of a surveilled physical environment (also referred to herein as a “surveilled environment”). For example, security cameras may be configured to record video upon detecting movement, while security lights may be configured to illuminate the surveilled environment upon detecting movement. Different types of electronic devices can create different types of content. Here, for example, the security cameras may generate audio data and/or video data, while the security lights may generate metadata specifying a time at which each illumination event occurred, a duration of each illumination event, etc.
The base station, meanwhile, may be responsible for transmitting the content generated by the electronic device(s) to a network-accessible computer server. Thus, each electronic device may provide data to the base station, which in turn provides at least some of the data to the network-accessible computer server.
Nowadays, security systems support features such as high-quality video recording, live video streaming, two-way audio transmission, cloud-based storage of recordings, instant alerts, etc. These features enable individuals to gain an in-depth understanding of what activities are occurring within the environment being surveilled. However, security systems having these features also experience drawbacks.
For example, once an event is detected by an electronic device, the security system may alert an administrator (e.g., a home owner). Certain alerts are not necessary in that the detected event does not pose any security risk. For instance, if the administrator observes that motion detection is triggered by movement of a bird, the administrator may determine that an alert is not needed. Conversely, if the administrator observes that motion detection is triggered by movement of a coyote, the administrator may determine that an alert is needed. Similar conclusions may be drawn for other routine events (e.g., mail delivery by postal worker). Administrators may simply ignore those alerts that are not needed (e.g., by simply deleting the corresponding notifications); however, an abundance of false positive notifications will tend to reduce the effectiveness of the security system as the administrator becomes overwhelmed by notifications and is not able to respond effectively.
Introduced herein is a technique for analyzing content generated by electronic devices in a network-connected security system to detect events and generate notifications based on the detected events that addresses the challenges discussed above. For example, a base station may employ cloud-based analytics to verify content (e.g., a video clip) before generating a notification or initiating a peer-to-peer stream to deliver the content to a user device. To verify the content, the base station may be required to contact the network-connected computer server, which can perform the processing needed to filter out unnecessary alerts. In some embodiments, the network-connected computer server is one of multiple network-connected computer servers that form a server system. The server system may balance the load amongst the multiple network-connected computer servers (e.g., by intelligently distributing images for processing) to ensure the verification process is completed with low latency. Similar cloud-based analytics can be employed on content generated by electronic devices to visually detect intruders, audibly detect sounds indicative of a break-in (e.g., an unrecognized voice or a window breaking), audibly detect sounds indicative of catastrophic events such as fire or earthquakes, etc. Further, such analysis may be performed in real time (or near real time) as content is generated so that an administrator is able to quickly respond to notifications of detected events.
References in this description to “an embodiment” or “one embodiment” means that the particular feature, function, structure, or characteristic being described is included in at least one embodiment. Occurrences of such phrases do not necessarily refer to the same embodiment, nor are they necessarily referring to alternative embodiments that are mutually exclusive of one another.
Unless the context clearly requires otherwise, the words “comprise” and “comprising” are to be construed in an inclusive sense rather than an exclusive or exhaustive sense (i.e., in the sense of “including but not limited to”). The terms “connected,” “coupled,” or any variant thereof is intended to include any connection or coupling between two or more elements, either direct or indirect. The coupling/connection can be physical, logical, or a combination thereof. For example, devices may be electrically or communicatively coupled to one another despite not sharing a physical connection.
The term “based on” is also to be construed in an inclusive sense rather than an exclusive or exhaustive sense. Thus, unless otherwise noted, the term “based on” is intended to mean “based at least in part on.”
The term “module” refers broadly to software components, hardware components, and/or firmware components. Modules are typically functional components that can generate useful data or other output(s) based on specified input(s). A module may be self-contained. A computer program may include one or more modules. Thus, a computer program may include multiple modules responsible for completing different tasks or a single module responsible for completing all tasks.
When used in reference to a list of multiple items, the word “or” is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of items in the list.
The sequences of steps performed in any of the processes described here are exemplary. However, unless contrary to physical possibility, the steps may be performed in various sequences and combinations. For example, steps could be added to, or removed from, the processes described here. Similarly, steps could be replaced or reordered. Thus, descriptions of any processes are intended to be open-ended.
The base station 105 and the one or more electronic devices 110 can be connected to each other via a local network 125. The local network 125 can be a local area network (LAN). In some embodiments, the local network 125 is a WLAN, such as a home Wi-Fi, created by one or more wireless accesses points (APs) 120. In some embodiments, functionality associated with base station 105 and/or wireless AP 120 are implemented in software instantiated at a wireless networking device. In other words, the system may include multiple wireless networking devices as nodes, wherein each of the wireless networking devices is operable as a wireless AP 120 and/or base station 105. The one or more electronic devices 110 and the base station 105 can be connected to each other wirelessly, e.g., over Wi-Fi, or using wired means. The base station 105 and the one or more electronic devices 110 can be connected to each other wirelessly via the one or more wireless APs 120, or directly with each other without the wireless AP 120, e.g., using Wi-Fi direct, Wi-Fi ad hoc or similar wireless connection technologies or via wired connections. Further, the base station 105 can be connected to the local network 125 using a wired means or wirelessly.
The one or more electronic devices 110 can be battery powered or powered from a wall outlet. In some embodiments, the one or more electronic devices 110 can include one or more sensors such as motion sensors that can activate, for example, the capturing of audio or video, the encoding of captured audio or video, and/or transmission of an encoded audio or video stream when motion is detected.
Cameras 110a may capture video, encode the video as a video stream, and wirelessly transmit the video stream via local network 125 for delivery to a user device 102. In some embodiments, certain cameras may include integrated encoder components. Alternatively, or in addition, the encoder component may be a separate device coupled to the camera 110a. For example, an analog camera may be communicatively coupled to the base station 105 and/or wireless AP 120 via an analog to digital encoder device (not shown in
Similarly, audio recording devices 110b may capture audio, encode the audio as an audio stream, and wirelessly transmit the audio stream via local network 125 for delivery to a user device 102. In some embodiments, certain audio recording devices may include integrated encoder components. Alternatively, or in addition, the encoder component may be a separate device coupled to the audio recording device 110b. For example, an analog audio recording device may be communicatively coupled to the base station 105 and/or wireless AP 120 via an analog to digital encoder device (not shown in
Although the example environment 100 illustrates various types of electronic devices 110a-d, the security system can include just a single type of electronic device (e.g., cameras 110a) or two or more different types of electronic devices 110 which can be installed at various locations of a building. The various electronic devices 110 of the security system may include varying features and capabilities. For example, some electronic devices 110 may be battery powered while another may be powered from the wall outlet. Similarly, some electronic devices 110 may connect wirelessly to the base station 105 while others rely on wired connections. In some embodiments, electronic devices of a particular type (e.g., cameras 110a) included in the security system may also include varying features and capabilities. For example, in a given security system, a first camera 110a may include an integrated night vision, audio recording, and motion sensing capabilities while a second camera 100a only includes video capture capabilities.
The base station 105 can be a computer system that serves as a gateway to securely connect the one or more electronic devices 110 to an external network 135, for example, via one or more wireless APs 120. The external network 135 may comprise one or more networks of any type including packet switched communications networks, such as the Internet, World Wide Web portion of the Internet, extranets, intranets, and/or various other types of telecommunications networks such as cellular phone and data networks, plain old telephone system (POTS) networks, etc.
The base station 105 can provide various features such as long range wireless connectivity to the electronic devices 110, a local storage device 115, a siren, connectivity to network attached storage (NAS), and enhance battery life for certain electronic devices 110, e.g., by configuring certain electronic devices 110 for efficient operation and/or by maintaining efficient communications between the base station 105 and such electronic devices 110. The base station 105 can be configured to store the content (e.g., audio and/or video) captured by some electronic devices 110 in any of the local storage device 115 or a network-accessible storage 148. The base station 105 can be configured to generate a sound alarm from the siren when an intrusion is detected by the base station 105 based on the video streams receive from cameras 110/112.
In some embodiments, the base station 105 can create its own network within the local network 125, so that the one or more electronic devices 110 do not overload or consume the network bandwidth of the local network 125. In some embodiments, the local network 125 can include multiple access points 120 to increase wireless coverage of the base station 105, which may be beneficial or required in cases where the electronic devices 110 are wirelessly connected and are spread over a large area.
In some embodiments the local network 125 can provide wired and/or wireless coverage to user devices (e.g., user device 102), for example, via APs 120. In the example environment 100 depicted in
In an example embodiment, when a user 103 sends a request (e.g., from user device 102), to access content (e.g., audio and/or video) captured by any of the electronic devices 110, the base station 105 receives the request and in response to receiving the request, obtains the encoded stream(s) from one or more of the electronic devices 110 and transmits the encoded stream to the user device 102 for presentation. Upon receiving the encoded stream at the user device 102, a playback application in the user device 102 decodes the encoded stream and plays the audio and/or video to the user 103, for example, via speakers and/or a display of the user devices 102.
As previously mentioned, in some embodiments, the base station 105 may include an encoding/transcoding component that performs a coding process on audio and/or video received from the electronic devices 110 before streaming to the user device 102. In an example embodiment, a transcoder at the base station 105 transcodes a stream received from an electronic device 100 (e.g., a video stream from a camera 110a), for example, by decoding the encoded stream and re-encoding the stream into another format to generate a transcoded stream that it then streams to the user device 102.
The audio and/or video stream received at the user device 102 may be a real-time stream and/or a recorded stream. For example, in some embodiments, a transcoder may transcode an encoded stream received from an electronic device 110 and stream the transcoded stream to the user device 102 in real time or near real time (i.e., within several seconds) as the audio and/or video is captured at the electronic device 110. Alternatively, or in addition, audio and/or video streamed by base station 105 to the user device 102 may be retrieved from storage such as local storage 115 or a network-accessible storage 148.
The base station 105 can stream audio and/or video to the user device 102 in multiple ways. For example, the base station 105 can stream to the user device 102 using peer-to-peer (P2P) streaming technique. In P2P streaming, when the playback application on the user device 102 requests the stream, the base station 105 and the user device 102 may exchange signaling information, for example via network 135 or a network-accessible server system 145, to determine location information of the base station 105 and the user device 102, to find a best path and establish a P2P connection to route the stream from the base station 105 to the user device 102. After establishing the connection, the base station 105 streams the audio and/or video to the user device 102, eliminating the additional bandwidth cost to deliver the audio and/or video stream from the base station 105 to a network-accessible server computer 146 in a network-accessible server system 145 and for streaming from the network-accessible server computer 146 to the user device 102. In some embodiments, a network-accessible server computer 146 in the network-accessible server system 145 may keep a log of available peer node servers to route streams and establish the connection between the user device 102 and other peers. In such embodiments, instead of streaming content, the server 146 may function as a signaling server or can include signaling software whose function is to maintain and manage a list of peers and handle the signaling between the base station 105 and the user device 102. In some embodiments, the server 146 can dynamically select the best peers based on geography and network topology.
In some embodiments, the network-accessible server system 145 is a network of resources from a centralized third-party provider using Wide Area Networking (WAN) or Internet-based access technologies. In some embodiments, the network-accessible server system 145 is configured as or operates as part of a cloud network, in which the network and/or computing resources are shared across various customers or clients. Such a cloud network is distinct, independent, and different from that of the local network 125.
In some embodiments, the local network 125 may include a multi-band wireless network comprising one or more wireless networking devices (also referred to herein as nodes) that function as wireless APs 120 and/or a base station 105. For example, with respect to the example environment 100 depicted in
The wireless networking devices of a multi-band wireless networking system can include radio components for multiple wireless bands, such as 2.5 GHz frequency band, low 5 GHz frequency band, and high 5 GHz frequency band. In some embodiments, at least one of the bands can be dedicated to the wireless communications among the wireless networking devices of the system. Such wireless communications among the wireless networking devices of the system are referred to herein as “backhaul” communications. Any other bands can be used for wireless communications between the wireless networking devices of the system and client devices such as cameras 110 connecting to the system. The wireless communications between the wireless networking devices of the system and client devices are referred to as “fronthaul” communications.
The electronic device 200 (also referred to as a “recording device”) can include one or more processors 202, a communication module 204, an optical sensor 206, a motion sensing module 208, a microphone 210, a speaker 212, a light source 214, and one or more storage modules 216.
The processor(s) 202 can execute instructions stored in the storage module(s) 216, which can be any device or mechanism capable of storing information. In some embodiments, a single storage module includes multiple computer programs for performing different operations (e.g., image recognition, noise reduction, filtering), while in other embodiments each computer program is hosted within a separate storage module.
The communication module 204 can manage communication between various components of the electronic device 200. The communication module 204 can also manage communications between the electronic device 200 and a base station, another electronic device, etc. For example, the communication module 204 may facilitate communication with a mobile phone, tablet computer, wireless access point (WAP), etc. As another example, the communication module 204 may facilitate communication with a base station responsible for communicating with a network-connected computer server; more specifically, the communication module 204 may be configured to transmit content generated by the electronic device 200 to the base station for processing. As further described below, the base station may examine the content itself or transmit the content to the network-connected computer server for examination.
The optical sensor 206 (also referred to as “image sensors”) can be configured to generate optical data related to the surveilled environment. Examples of optical sensors include charged-coupled devices (CCDs), complementary metal-oxide-semiconductors (CMOSs), infrared detectors, etc. In some embodiments, the optical sensor 206 is configured to generate a video recording of the surveilled environment responsive to, for example, determining that movement has been detected within the surveilled environment. In other embodiments, the optical data generated by the optical sensor 206 is used by the motion sensing module 208 to determine whether movement has occurred. The motion sensing module 208 may also consider data generated by other components (e.g., the microphone) as input. Thus, an electronic device 200 may include multiple optical sensors of different types (e.g., visible light sensors and/or IR sensors for night vision).
The microphone 210 can be configured to record sounds within the surveilled environment. The electronic device 200 may include multiple microphones. In such embodiments, the microphones may be omnidirectional microphones designed to pick up sound from all directions. Alternatively, the microphones may be directional microphones designed to pick up sounds coming from a specific direction. For example, if the electronic device 200 is intended to be mounted in a certain orientation (e.g., such that the camera 208 is facing a doorway), then the electronic device 200 may include at least one microphone arranged to pick up sounds originating from near the point of focus.
The speaker 212, meanwhile, can be configured to convert an electrical audio signal into a corresponding sound that is projected into the surveilled environment. Together with the microphone 210, the speaker 212 enables an individual located within the surveilled environment to converse with another individual located outside of the surveilled environment. For example, the other individual may be a homeowner who has a computer program (e.g., a mobile application) installed on her mobile phone for monitoring the surveilled environment.
The light source 214 can be configured to illuminate the surveilled environment. For example, the light source 214 may illuminate the surveilled environment responsive to a determination that movement has been detected within the surveilled environment. The light source 214 may generate metadata specifying a time at which each illumination event occurred, a duration of each illumination event, etc. This metadata can be examined by the processor(s) 202 and/or transmitted by the communication module 204 to the base station for further processing.
As previously discussed with respect to
The base station 300 can include one or more processors 302, a communication module 304, and one or more storage modules 306. In some embodiments, a single storage module includes multiple computer programs for performing different operations (e.g., image recognition, noise reduction, filtering), while in other embodiments each computer program is hosted within a separate storage module. Moreover, the base station 300 may include a separate storage module for each electronic device within its corresponding surveillance environment, each type of electronic device within its corresponding surveillance environment, etc.
Such a categorization enables the base station 300 to readily identify the content/data generated by security cameras, security lights, etc. The content/data generated by each type of electronic device may be treated differently by the base station 300. For example, the base station 300 may locally process sensitive content/data but transmit less sensitive content/data for processing by a network-connected computer server.
Thus, in some embodiments, the base station 300 processes content/data generated by the electronic devices, for example, to analyze the content to understand what events are occurring within the surveilled environment, while in other embodiments the base station 300 transmits the content/data to a network-connected computer server responsible for performing such analysis.
The communication module 304 can manage communication with electronic device(s) within the surveilled environment and/or the network-connected computer server. In some embodiments, different communication modules handle these communications. For example, the base station 300 may include one communication module for communicating with the electronic device(s) via a short-range communication protocol, such as Bluetooth® or Near Field Communication, and another communication module for communicating with the network-connected computer server via a cellular network or the Internet.
The base station 502 and the network-accessible server system 506 can be connected to one another via a computer network 504a. The computer network 504a may include a personal area network (PAN), local area network (LAN), wide area network (WAN), metropolitan area network (MAN), cellular network, the Internet, or any combination thereof.
The security management platform 508 may be responsible for parsing content/data generated by electronic device(s) arranged throughout a surveilled environment to detect occurrences of events within the surveilled environment. The security management platform 508 may also be responsible for creating interfaces through which an individual can view content (e.g., video clips and audio clips), initiate an interaction within someone located in the surveilled environment, manage preferences, etc.
As noted above, the security management platform 508 may reside in a network environment 500b. Thus, the security management platform 508 may be connected to one or more networks 504b-c. Similar to network 504a, networks 504b-c can include PANs, LANs, WANs, MANs, cellular networks, the Internet, etc. Additionally, or alternatively, the security management platform 508 can be communicatively coupled to computing device(s) over a short-range communication protocol, such as Bluetooth® or NFC.
The interface 510 is preferably accessible via a web browser, desktop application, mobile application, or over-the-top (OTT) application. Accordingly, the interface 510 may be viewed on a personal computer, tablet computer, personal digital assistant (PDA), mobile phone, game console, music player, wearable electronic device (e.g., a watch or fitness accessory), network-connected (“smart”) electronic device, (e.g., a television or home assistant device), virtual/augmented reality system (e.g., a head-mounted display), or some other electronic device.
As described above, one issue with security systems is the overabundance of alerts generated by the security systems. Consequently, individuals (e.g., administrators of the security systems) may lose interest in the security capabilities of these security systems due to too many undesired notifications.
These undesired notifications may be derived from several different sources. For example, in some embodiments, a security system may detect too many false instances of motion because it relies on a signal generated by an overly sensitive passive infrared sensor (PIS). As another example, in some embodiments a security system may detect too many false instances of audio because it relies on an overly sensitive audio sensor (which is configured to prompt recording by the security camera).
To reduce the quantity of notifications, a network-connected security system can be configured to filter those notifications deemed likely to be unnecessary. In some cases, the “filtering” of notifications may include receiving notifications and only forwarding a portion of the received notifications deemed necessary to a user. Alternatively, or in addition, “filtering” notifications may refer to detecting multiple events that would otherwise result in notifications to a user (e.g., detected motion) and only generating notifications based on a subset of the detected events for presentation to a user. For example, the base station can apply an algorithm that allows it to detect objects included in a video clip. Moreover, the base station can apply an algorithm that allows it to detect the scene depicted in the video clip. The base station can then remove undesired motion from the video clip. Said another way, the base station can ignore those movements that are indicative of events the corresponding individual does not wish to be notified about. Thereafter, the base station can generate notifications only for those events that survive the “filtering” process. Such action ensures that the corresponding individual is only notified about significant events.
Initially, one or more security cameras 110a generate content 602, for example, by capturing video and encoding the captured video into digital information. The content 602 may include, for example, one or more digital files including the encoded video.
The content 602 is then fed into an analytics system 604 for processing according to the introduced technique. In some embodiments, the step of feeding the content 602 into the analytics system 604 may include a camera 110a transmitting the generated content 602 over a computer network (e.g., a wired or wireless local network 125) to a base station 105. The base station 105 may then forward the received content 602 to a network-accessible server system 145 that implements the analytics system 604. Alternatively, or in addition, the camera 110a and/or base station 105 may include processing components that implement at least a portion of the analytics system 604.
In some embodiments, content 602 is fed into the analytics system 604 continually as it is generated. For example, in some embodiments, the camera 110a may generate a digital video stream that is transmitted to the analytics system 604 for processing by way of the base station 105.
In some embodiments, content 602 is continually generated by the camera 110a. For example, a camera 110a that is powered by a wall outlet may continually capture view, encode the captured video into a digital stream, and transmit that digital stream for processing by the analytics system.
Alternatively, the camera 110a may be configured to generate content 602 at periodic intervals and/or in response to detecting certain conditions or events. For example, the camera 110a may be equipped with, or in communication with, a motion detector that triggers the capturing and encoding of video when motion in the surveilled environment is detected. In response to receiving an indication of detected motion, the camera 110a may begin generating content 602 by capturing video and encoding the captured video. As another illustrative example, instead of transmitting a continuous stream of content 602, the video camera 110a may transmit small portions of content (e.g., short video clips or still images) at period intervals (e.g., every few seconds).
Generating content 602 at periodic intervals and/or in response to detected events may conserve energy at the camera 110a, which may be particularly beneficial for battery-powered cameras 110a. Generating content 602 at periodic intervals and/or in response to detected events may also reduce resource requirements to process the content, for example, when generating notifications. For example, in the case of a surveilled environment, the system may be configured based on an assumption that the video of the surveilled environment is of no interest to an administrator unless the video captures an object in motion.
In some embodiments, content 602 is fed into the analytics system 604 periodically (e.g., daily, weekly, or monthly) or in response to detected events. For example, even if the content 602 is continually generated, such content 602 may be held in storage (e.g., at local storage 115 or a NAS 148) before being released (periodically or in response to detected events) for analysis by the analytics system 604.
Thereafter, the analytics system 604 processes the received content 602 to perform the notification filtering technique described herein. For example, the analytics system 604 may process the received content 602 to detect whether an event has occurred that necessitates a notification to a user. As previously mentioned, processing of the received content 602 may be carried out by processors located at the base station 105, a network-accessible server system 145, or any combination thereof.
In some embodiments, processing of content 602 may include a content recognition process 606 to gain some level of understanding of the information captured in the content 602. For example, the content recognition process 606 may apply computer vision techniques to detect physical objects captured in the content 602.
The content recognition process 606 may further classify such detected objects. For example, given one or more classes of objects (e.g., humans, buildings, cars, animals, etc.), the content recognition process 606 may process the video content 602 to identify instances of various classes of physical objects occurring in the captured video of the surveilled environment.
In some embodiments, the content recognition process 606 may employ deep learning-based video recognition to classify detected objects. In an example deep learning-based video recognition process for detecting a face, raw image data is input as a matrix of pixels. A first representational layer may abstract the pixels and encode edges. A second layer may compose and encode arrangements of edges, for example, to detect objects. A third layer may encode identifiable features such as a nose and eyes. A fourth layer may recognize that the image includes a face based on the arrangement of identifiable features.
An example technique for classifying objects detected in images or video is the Haar Cascade classifier.
Notably, each stage in the example process 800 may apply increasing levels of processing which requiring increasingly more computational resources. A benefit of this cascade technique is that objects that are not faces are immediately eliminated as such at higher stages with relatively little processing. To be classified as a particular type of object (e.g., face detection), the content must pass each of the stages 802-808 of the classifier.
Note that the example Haar Cascade classifier process 800 depicted in
Returning to
The content recognition process 606 may also include recognizing the identity of detected objects (e.g., specific people). For example, the analytics system 604 may receive inputs (e.g., captured images/video) to learn the appearances of instances of certain objects (e.g., specific people) by building machine-learning appearance-based models. Instance segmentations identified based on processing of content 602 can then be compared against such appearance-based models to resolve unique identities for one or more of the detected objects. Identity recognition can be particularly useful in this context as it would allow the system to ignore the detection of certain known individuals in captured image (e.g., members of a household) while focusing notifications on unknown individuals and/or known unwanted individuals that more likely pose a security threat.
The content recognition process 606 may also include fusing information related to detected objects to gain a semantic understanding of the captured scene. For example, the content recognition process 606 may include fusing semantic information associated with a detected object with geometry and/or motion information of the detected object to infer certain information regarding the scene. Information that may be fused may include, for example, an object's category (i.e., class), identity, location, shape, size, scale, pixel segmentation, orientation, inter-class appearance, activity, and pose. As an illustrative example, the content recognition 606 process may fuse information pertaining to one or more detected objects to determine that a clip of video is capturing a known person (e.g., a neighbor) walking their dog past a house. The same process may be applied to another clip to determine that the other clip is capturing an unknown individual peering into a window of a surveilled house. The analytics system 604 can then use such information to generate notifications only for the scene that presents a heightened security risk (i.e., the unknown person looking in the window) despite motion being detected in both.
In some embodiments, labeled image data (e.g., historical video from one or more sources) may be input to train a neural network (or other machine-learning based models) as part of the content recognition process 606. For example, security experts may input previously captured video from a number of different sources as examples of certain classes of objects (e.g., car, building, cat, tree, etc.) to inform the content recognition process 606.
As alluded to above, after performing content recognition to, for example, detect objects or further to gain a semantic understanding of a captured scene, the analytics system 604 will utilize this information as part of an event detection process 608. Event detection may include detecting recognizable events (e.g., a person walking to the front door) and analyzing certain specifics regarding the detected event (e.g., person's identity, time of day, proximity to other detected events, and other contextual information) to determine if the detected event is indicative of a security threat that warrants a notification.
In some embodiments, determining that a detected event is indicative of a security threat may include comparing data associated with the detected event against a database of data associated with candidate threats, for example, defined based on input from industry security experts. As an illustrative example, by processing video content 602, the analytics system 604 may detect an event characterized by an unknown individual approaching the doorway to a residence. The analytics system 604 may then compare this semantic information regarding the detected event to a database of candidate threats. Based on the comparison, the analytics system 604 may identify a particular candidate threat that matches (within some threshold level of certainty) the detected event. In some embodiments, the process of comparing may include generating, by the analytics system 604, a pattern matching score (e.g., a value between 0 and 10) and identifying the detected event as indicative of the particular candidate security threat if the generated score satisfies a threshold criterion (e.g., above 7 on a scale of 0-10).
Alternatively, or in addition, the analytics system 604 may employ machine learning to analyze received content 602 to determine if the content is indicative of a security threat. For example, as previously discussed, the analytics system 604 may apply machine-learning-based behavioral analytics to learn the behavior of objects captured in video images and identify when the behavior of such objects as indicative of a security threat. Applying a machine-learning-based approach may be beneficial in certain instances as it may alleviate the need to develop complex threat detection rules that rely on preexisting knowledge and that are prone to incorrectly identifying unexpected or rare behavior.
If, based on the analysis of the received content 602, the analytics system 604 determines that the content is indicative of a security threat, the analytics system may apply a notification generation process 610 to generate one or more notifications for delivery to an administrative user at a user device 102.
In some embodiments, notification generation 610 may include generating one or more notifications 614, for example, in the form of messages that are then transmitted over a computer network to a user device 102 associated with an administrative user. Notifications may include emails, text messages (e.g., SMS, MMS, etc.), automated phone calls, alerts within interface 510, or any other communications medium appropriate for delivery at user device 102.
In some embodiments, notification generation may include transmitting processed and/or filtered content 616 for delivery at the client device. For example, in the case of video content 602, the analytics system may process the received video content 602 and forward content 616 based on the processing to the user device 102. Processed content 616 may include, for example, a shortened video clip that specifically depicts the activity upon which the security threat was identified. Processed content 616 may also include transformations to the original content to highlight the activity upon which the security threat was identified. For example, the analytics system 604 may be configured to process content 602 to remove detected motion from the content that is not indicative of a security threat.
Processed content 616 may be transmitted to the user device 102 in real time (or near real time) as the content 602 is generated by the camera 110a. For example, a camera 110a may transmit content 602 in the form of a continuous stream of video to analytics system 604. The analytics system 604 may process the received video stream as it is received and only forward portions of the video stream (i.e., processed/filtered content 616) as events are detected that are indicative of a security threat. This processing may occur in real time or near real time as the video is captured at the camera 110a so that an administrator user can effectively respond to the security threats.
Alternatively, or in addition, processed/filtered content 616 may represent time-shifted recordings that are delivered to the user device 102 after the events underlying the recordings have already occurred. For example, an administrator user that does not want to be bothered with the delivery of live streams throughout the day may elect instead to, for example, review the recordings for the day once at the end of the day.
In some embodiments, one or more of the electronic devices 110 may be configured to individually analyze certain content (e.g., captured video) and generate notifications. In such embodiments, an analytics system 604 operating apart from such an electronic device, may be configured to process such notifications as content 602 as part of a notification filtering process. Notifications received by the analytics system 604 for processing may be referred to as provisional notifications in that they are subject to filtering processes which may result in being forwarded to a user device or discarded/ignored.
As an illustrative example, the content 602 depicted in
In some embodiments, the analytics system 604 may consider user feedback 618 provided by one or more users. For example, if a user indicates that a certain video clip is undesirable, uninteresting, or otherwise not worthy of a notification, then the analytics system 604 may reduce/eliminate notifications related to future video clips having similar characteristics (e.g., generated at the same time, generated based on the same trigger, including the same visual/audible objects, etc.).
The analytics system 604 may also consider feedback provided by a cohort that includes the corresponding user. The cohort may include users that share a characteristic in common, such as geographical location, notification frequency, etc. For example, if the analytics system 604 considers feedback from users within a neighborhood, the analytics system 604 may know to filter those notifications pertaining to a cat that lives in the neighborhood. As another example, if the analytics system 604 considers feedback from users within a city, the analytics system 604 may know to filter those notifications pertaining to events triggered by weather (e.g., wind or rain).
Various programming models and associated techniques for processing and generating data can be applied by the analytics system 604 to process content 602 and generate notifications. For example, in some embodiments, analytics system 604 may utilize a distributed computing cluster to process content 602. Utilizing a distributed computing architecture can be particularly beneficial when processing large amounts of data such as content received from a security system or multiple security systems.
The distributed computing cluster 902 may represent a logical entity that includes sets of host machines (not shown in
Apache Hadoop™ software utilities can be employed to facilitate the development of filtering algorithm(s), the acquisition of data pertaining to surveilled environments, and the application of the filtering algorithm(s) to improve real-time analytics. Here, for example, the Apache Hadoop™ software utilities may consider content 602 (e.g., video clips) generated by electronic devices deployed in surveilled environments, as well as user feedback specifying which notifications are desired. The Apache Hadoop™ software utilities can also develop a classification model for classifying content by training a supervised machine learning algorithm. Various machine learning and/or artificial intelligence technologies can be employed to facilitate the development of the classification model.
Example process 1200 begins at step 1202 with receiving content 602 generated by an electronic device 110 located in a physical environment (e.g., a surveilled environment 400). As previously discussed, the electronic device 110 may be one of several electronic devices 110 associated with a network-connected security system. In some embodiments, the content 602 is received at step 1202 via a computer network, from a base station 105 associated with the network-connected security system.
In some embodiments, the network-connected security system is a video surveillance system and the electronic device 110 is a network-connected video camera 110a. In such embodiments the content 602 may include video files. Further, the optical parameter may be any of an optical parameter, an image processing parameter, or an encoding parameter.
In some embodiments, the content 602 received from the electronic device 110 at step 1202 may include a provisional notification generated by the electronic device 110. For example, the electronic device 110 may include processing resources to detect an event based on sensory information (e.g., video) and generate a notification based on the detected event.
Example process 1200 continues at step 1204 with processing the received content 602 to detect an event in the surveilled physical environment.
In the case of a video content (e.g., from a camera 110a), the processing of content at step 1204 to detect an event may include processing the video to detect one or more instances of physical objects in the physical environment and analyzing data associated with the detected one or more instances of physical objects to detect a scene captured by the video camera 110a. For example, as previously discussed, a content recognition process 606 (described with respect to
If the received content 602 includes a provisional notification (e.g., generated by an electronic device), the step of detecting the event at step 1204 may include processing the received provisional notification (e.g., by reading or interpreting a message included in the notification) and identifying the event (as detected by the electronic device 110) based on the processing.
Example process 1200 continues at step 1206 with determining if the detected event satisfies a specified criterion. As previously discussed, a purpose of processing content generated by a network-connected security system may be to determine if a notification to a user is necessary. In a security context, a notification is generally understood to be necessary when an event has occurred in a surveilled environment is abnormal, or more specifically indicative of a security risk. The specified criterion may therefore differ in various embodiments, but is generally established based on a need to selectively notify a user of activity in the surveilled environment that may be of interest to the user whether that activity is merely outside of a normal baseline or more specifically indicative of a security risk or threat.
In an illustrative embodiment, the step of determining if the detected event satisfies a specified criterion includes comparing data associated with the detected event against a database of data associated with a plurality of candidate security threats, generating a pattern matching score based on the detected event and a particular candidate security threat of the plurality of candidate security threats and identifying the detected event as indicative of the particular candidate security threat if the generated pattern matching score satisfies a threshold criterion. For example, pattern matching scores may be generated on a scale of 0-10 with a threshold criterion set at 7 to indicate that a detected event is indicative of a particular candidate security threat based on the comparison.
Depending on the computer system performing example process 1200, steps 1204 and 1206 may include transmitting the content 602 to another computing system for processing. For example, if a base station 105 is performing process 1200, steps 1204 and 1206 may include transmitting, by the base station 105, the content 602, via an external network, to an external computing system such as a network-accessible server system 145 to process the content 602 to detect an event and determine if the detected event satisfies a specified criterion.
Example process 1200 concludes at step 1208 with causing a notification to be presented at a user device 102 communicatively coupled to the network-connected security system if the detected event satisfies the specified criterion. In some embodiments, the notification is presented at the user device 102 in real time or near real time as the content 602 is generated by the electronic device 110. For example, a notification may be presented at a user device 102 within seconds or fractions of a second a portion of video content is captured at a camera 110a associated with a network-connected security system. The latency between content generation and presentation of the notification will depend on certain limitations in the system (e.g., processing resources, network speed, etc.).
In some embodiments, the notification presented at the user device 102 at step 1208 may include an alert message (e.g., emails, text messages (e.g., SMS, MMS, etc.), automated phone calls, alerts within interface 510) informing a user of the event. In other words, in some embodiments, step 1208 may include causing the alert message to be transmitted, via a computer network, to the client device 102.
In some embodiments, the notification presented at the user device 102 at step 1208 may include at least a portion of the content 602 generated by an electronic device 110. In such embodiments, step 1208 may include causing at least a portion of the content 602 received from the electronic device 110 to be transmitted, via a computer network, to the client device 102. In some embodiments, causing the at least a portion of the content 602 to be transmitted to the client device 102 may include initiating a peer-to-peer connection between the electronic device 110 and the client device 102 and causing the portion of content 602 to be transmitted via the peer-to-peer connection.
In cases where the content 602 received at step 1202 includes a provisional notification, step 1208 may include forwarding the received provisional notification if the event (e.g., as detected by the electronic device 110) satisfies the specified criterion.
The computer system 1300 may include one or more central processing units (“processors”) 1302, main memory 1306, non-volatile memory 1310, network adapter 1312 (e.g., network interface), video display 318, input/output devices 1320, control device 1322 (e.g., keyboard and pointing devices), drive unit 1324 including a storage medium 1326, and signal generation device 1330 that are communicatively connected to a bus 1316. The bus 1316 is illustrated as an abstraction that represents one or more physical buses and/or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 1316, therefore, can include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also referred to as “Firewire”).
The computer system 1300 may share a similar computer processor architecture as that of a desktop computer, tablet computer, personal digital assistant (PDA), mobile phone, game console, music player, wearable electronic device (e.g., a watch or fitness tracker), network-connected (“smart”) device (e.g., a television or home assistant device), virtual/augmented reality systems (e.g., a head-mounted display), or another electronic device capable of executing a set of instructions (sequential or otherwise) that specify action(s) to be taken by the computer system 1300.
While the main memory 1306, non-volatile memory 1310, and storage medium 1326 (also called a “machine-readable medium”) are shown to be a single medium, the term “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized/distributed database and/or associated caches and servers) that store one or more sets of instructions 1328. The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computer system 1300.
In general, the routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically comprise one or more instructions (e.g., instructions 1304, 1308, 1328) set at various times in various memory and storage devices in a computing device. When read and executed by the one or more processors 1302, the instruction(s) cause the computer system 1300 to perform operations to execute elements involving the various aspects of the disclosure.
Moreover, while embodiments have been described in the context of fully functioning computing devices, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms. The disclosure applies regardless of the particular type of machine or computer-readable media used to actually effect the distribution.
Further examples of machine-readable storage media, machine-readable media, or computer-readable media include recordable-type media such as volatile and non-volatile memory devices 1310, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD-ROMS), Digital Versatile Disks (DVDs)), and transmission-type media such as digital and analog communication links.
The network adapter 1312 enables the computer system 1300 to mediate data in a network 1314 with an entity that is external to the computer system 1300 through any communication protocol supported by the computer system 1300 and the external entity. The network adapter 1312 can include a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, and/or a repeater.
The network adapter 1312 may include a firewall that governs and/or manages permission to access/proxy data in a computer network and tracks varying levels of trust between different machines and/or applications. The firewall can be any number of modules having any combination of hardware and/or software components able to enforce a predetermined set of access rights between a particular set of machines and applications, machines and machines, and/or applications and applications (e.g., to regulate the flow of traffic and resource sharing between these entities). The firewall may additionally manage and/or have access to an access control list that details permissions including the access and operation rights of an object by an individual, a machine, and/or an application, and the circumstances under which the permission rights stand.
The techniques introduced here can be implemented by programmable circuitry (e.g., one or more microprocessors), software and/or firmware, special-purpose hardwired (i.e., non-programmable) circuitry, or a combination of such forms. Special-purpose circuitry can be in the form of one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc.
The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.
Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments may vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments.
The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.
This application is entitled to the benefit and/or right of priority of U.S. Provisional Application No. 62/644,847 (Attorney Docket No. 110729-8095.US00), titled, “ELASTIC PROCESSING FOR VIDEO ANALYSIS AND NOTIFICATION ENHANCEMENTS,” filed Mar. 19, 2018, the contents of which are hereby incorporated by reference in their entirety for all purposes. This application is therefore entitled to a priority date of Mar. 19, 2018.
| Number | Date | Country | |
|---|---|---|---|
| 62644847 | Mar 2018 | US |
