Patent Application
20080074496
The foregoing and other features of various embodiments of the invention will be apparent from the following, more particular description of such embodiments of the invention, as illustrated in the accompanying drawings, wherein like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements.
In describing the invention, the following definitions are applicable throughout (including above).
“Video” may refer to motion pictures represented in analog and/or digital form. Examples of video may include: television; a movie; an image sequence from a video camera or other observer; an image sequence from a live feed; a computer-generated image sequence; an image sequence from a computer graphics engine; an image sequences from a storage device, such as a computer-readable medium, a digital video disk (DVD), or a high-definition disk (HDD); an image sequence from an IEEE 1394-based interface; an image sequence from a video digitizer; or an image sequence from a network.
A “video sequence” may refer to some or all of a video.
A “video camera” may refer to an apparatus for visual recording. Examples of a video camera may include one or more of the following: a video imager and lens apparatus; a video camera; a digital video camera; a color camera; a monochrome camera; a camera; a camcorder; a PC camera; a webcam; an infrared (IR) video camera; a low-light video camera; a thermal video camera; a closed-circuit television (CCTV) camera; a pan, tilt, zoom (PTZ) camera; and a video sensing device. A video camera may be positioned to perform surveillance of an area of interest.
“Video processing” may refer to any manipulation and/or analysis of video, including, for example, compression, editing, surveillance, and/or verification.
A “frame” may refer to a particular image or other discrete unit within a video.
A “computer” may refer to one or more apparatuses and/or one or more systems that are capable of accepting a structured input, processing the structured input according to prescribed rules, and producing results of the processing as output. Examples of a computer may include: a computer; a stationary and/or portable computer; a computer having a single processor, multiple processors, or multi-core processors, which may operate in parallel and/or not in parallel; a general purpose computer; a supercomputer; a mainframe; a super mini-computer; a mini-computer; a workstation; a micro-computer; a server; a client; an interactive television; a web appliance; a telecommunications device with internet access; a hybrid combination of a computer and an interactive television; a portable computer; a personal digital assistant (PDA); a portable telephone; application-specific hardware to emulate a computer and/or software, such as, for example, a digital signal processor (DSP), a field-programmable gate array (FPGA), a chip, chips, or a chip set; an optical computer; a quantum computer; a biological computer; and an apparatus that may accept data, may process data in accordance with one or more stored software programs, may generate results, and typically may include input, output, storage, arithmetic, logic, and control units.
“Software” may refer to prescribed rules to operate a computer. Examples of software may include: software; code segments; instructions; applets; pre-compiled code; compiled code; computer programs; and programmed logic.
A “computer-readable medium” may refer to any storage device used for storing data accessible by a computer. Examples of a computer-readable medium may include: a magnetic hard disk; a floppy disk; an optical disk, such as a CD-ROM and a DVD; a magnetic tape; and a memory chip.
A “computer system” may refer to a system having one or more computers, where each computer may include a computer-readable medium embodying software to operate the computer. Examples of a computer system may include: a distributed computer system for processing information via computer systems linked by a network; two or more computer systems connected together via a network for transmitting and/or receiving information between the computer systems; and one or more apparatuses and/or one or more systems that may accept data, may process data in accordance with one or more stored software programs, may generate results, and typically may include input, output, storage, arithmetic, logic, and control units.
A “network” may refer to a number of computers and associated devices that may be connected by communication facilities. A network may involve permanent connections such as cables or temporary connections such as those made through telephone or other communication links. Examples of a network may include: an internet, such as the Internet; an intranet; a local area network (LAN); a wide area network (WAN); and a combination of networks, such as an internet and an intranet.
Exemplary embodiments of the invention are discussed in detail below. While specific exemplary embodiments are discussed, it should be understood that this is done for illustration purposes only. In describing and illustrating the exemplary embodiments, specific terminology is employed for the sake of clarity. However, the invention is not intended to be limited to the specific terminology so selected. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the invention. It is to be understood that each specific element includes all technical equivalents that operate in a similar manner to accomplish a similar purpose. Each reference cited herein is incorporated by reference. The examples and embodiments described herein are non-limiting examples.
The invention may be directed generally to the use of an automated video surveillance system employing video analytics for monitoring banking business processes. Video analytics may refer to the application of computer vision techniques to extract useful data or information from video streams or video sequences. Specifically, the invention may be applied to the banking business processes involving physical security, public safety, loss prevention, and business intelligence data gathering, including employee effectiveness.
Examples of applying embodiments of the invention to banking business processes involving physical security may include, for example, detection of: an intrusion within a banking facility; a vandalism act within or around a banking facility; unauthorized access to a secure area (known as “tailgating”); an illegal number of people in secure area, e.g. at least two people should be present when opening a safe, but there is only one; suspicious behavior around cars in a parking lot; suspicious behavior around an ATM; installation of a skimmer device on an ATM; teller withdrawal without a customer present; only one employee present for transactions requiring at least two employees; a customer reaching over the teller station and grabbing money; a person holding a gun; and/or detection of camera tampering.
Examples of applying embodiments of the invention to banking business processes involving public safety may include, for example, detection of: people slipping and falling on bank property; people running in a bank or a parking lot; an assault or abduction in a parking lot; a car speeding in a parking lot; a car driving in a wrong direction at the drive through teller window; an obstruction of a fire exit; an illegally parked vehicle; an abandoned package; a vehicle proceeding in the wrong direction in a drive-through teller window; and/or crowding in a bank area or outside of the bank.
Examples of applying embodiments of the invention to banking business processes involving business intelligence data gathering may include, for example: queue length monitoring; monitoring of cleanliness; monitoring of an unattended delivery; and/or measuring customer service process through people tracking by monitoring employee-customer interactions.
Intelligent video may be applied to these banking processes in at least two operating modes: standalone and integrated.
In the standalone mode, operation of the video analysis engine may be based solely on what is detected in the video, without any additional information. Standalone applications may include detecting vandalism, camera tampering, change in the appearance of an ATM, person loitering in an ATM vestibule, queue length monitoring, etc.
In the integrated mode, the intelligent surveillance system may combine what is detected in the video with additional external information. The external information may come, for example, from a card swipe reader, which would allow comparison of a number of card swipes with a number of people to detect tailgating; from an ATM, to know when a transaction starts and ends, so that a person standing for a long time at an ATM without initiating a transaction may be flagged as suspicious; or from a teller, to know when a transaction is taking place, so that the presence of a customer may be verified during a withdrawal.
The exemplary automated video surveillance system of the invention may employ low-level video analytics algorithms to process the video feeds from bank surveillance to extract all of the objects of interest, while ignoring any irrelevant background motion. These objects of interest may be described via a set of “video primitives,” which may be a text description of all of the objects and observable features within a video. These video primitives also may include, for example, descriptions of the objects, their locations, velocities, shape, colors, location of body parts, etc.
The video primitives may be analyzed in a real-time mode via an activity inference engine to determine if an activity of interest has occurred, and/or stored in a database for such further analysis. These activities may be presented as “alerts” to a user or gathered together to generate a report for a user.
In addition to, or instead of, this real-time mode, the system may also operate in an off-line mode, e.g., forensic mode, wherein queries may be applied to archived video primitives after the fact. In the off-line mode, the user may look for activities by mining the video primitives, instead of the entire video analytics again.
The exemplary automated video surveillance system of the invention may be implemented as disclosed in the following, for example, which are incorporated herein by reference: U.S. Patent Publication No. 2005-0146605-A1, “Video Surveillance System Employing Video Primitives”; U.S. Patent Publication No. 2005-0162515 A1, “Video Surveillance System”; U.S. Patent Publication No. 2005-0169367-A1, “Video Surveillance System Employing Video Primitives”; U.S. patent application Ser. No. 11/167,218, “Video Surveillance System Employing Video Primitives,” Attorney Docket No. 37112-219035; and U.S. patent application Ser. No. 11/300,581, “Video Surveillance System Employing Video Primitives,” Attorney Docket No. 37112-225921.
In terms of the architecture of the exemplary automated video surveillance system, a number of different embodiments may be used. For example, the various embodiments described in the following, which is incorporated herein by reference, may be used with the invention: U.S. patent application Ser. No. 11/165,182, “Detection of Change in Posture in Video,” Attorney Docket No. 37112-219109. In one embodiment, the video analytics engine of block 2 and the activity inference engine of block 6 may be co-resident on a single device (e.g., a computer or a chip, chips, or a chip set). The single device may be resident within the video camera 1, an encoder, a router, a digital video recorder (DVR), a network video encoder (NVR) or some other network video device. In another embodiment, the video analytics engine of block 2 and the activity inference engine of block 6 may be on separate devices. For example, the first device of the video analytics engine of block 2 may be a computer or a chip, chips, or a chip set and may be resident within the video camera 1, an encoder, a router, a digital video recorder (DVR), a network video encoder (NVR) or some other network video device. Meanwhile, the second device of the activity inference engine of block 6 may be a computer or a chip, chips, or a chip set and may be resident within an encoder, a router, a digital video recorder (DVR), a network video encoder (NVR) or some other network video device connected to the first device via a network. In this case, only the video primitives may need to be sent across the network for the proper operation of the system.
In block 2, the video analytics engine may process the video from the video camera 1 in real-time and produces video primitives. Examples of the type of algorithms that may be employed by the video analytics engine are disclosed in the following, for example, which are incorporated herein by reference: U.S. Pat. No. 6,625,310, “Video Segmentation Using Statistical Pixel Modeling”; U.S. Pat. No. 6,696,945, “Video Tripwire”; U.S. Patent Publication No. 2005-0146605 A1, “Video Surveillance System Employing Video Primitives”; U.S. Pat. No. 6,987,883, “Video Scene Background Maintenance Using Statistical Pixel Modeling”; U.S. Patent Publication No. 2005-0168574-A1, “Video-Based Passback Event Detection”; U.S. Patent Publication No. 2004-0151374-A1 “Video Segmentation Using Statistical Pixel Modeling”; U.S. Pat. No. 6,970,083, “Video Tripwire”; U.S. Patent Publication No. 2006-0066722-A1, “View Handling in Video Surveillance Systems”; U.S. Patent Publication No. 2006-0066719-A1, “Method of Finding Paths in Video”; U.S. Patent Publication No. 2006-0072010-A1 “Target Property Maps for Surveillance Systems”; U.S. Patent Publication No. 2005-0162515-A1, “Video Surveillance System”; U.S. Patent Publication No. 2005-0169367-A1, “Video Surveillance System Employing Video Primitives”; U.S. patent application Ser. No. 11/113,275, “Line Textured Target Detection and Tracking with Applications to ‘Basket-Run’ Detection,” Attorney Docket No. 37112-217049; U.S. patent application Ser. No. 11/132,213, “Periodic Motion Detection with Applications to Multi-Grabbing,” Attorney Docket No. 37112-217806; U.S. patent application Ser. No. 11/139,600, “Multi-State Target Tracking,” Attorney Docket No. 37112-218196; U.S. patent application Ser. No. 11/139,986, “Human Detection and Tracking for Security Applications,” Attorney Docket No. 37112-218471; U.S. patent application Ser. No. 11/167,218, “Video Surveillance System Employing Video Primitives,” Attorney Docket No. 37112-219035; U.S. patent application Ser. No. 11/165,182, “Detection of Change in Posture in Video,” Attorney Docket No. 37112-219109; U.S. patent application Ser. No. 11/165,435, “Target Detection and Tracking from Overhead Video Streams,” Attorney Docket No. 37112-219452; U.S. patent application Ser. No. 11/221,923, “Video Surveillance Using Spatial-Temporal Motion Analysis,” Attorney Docket No. 37112-222928; U.S. patent application Ser. No. 11/288,200, “Detection of Stationary Objects in Video,” Attorney Docket No. 37112-224862; and U.S. patent application Ser. No. 11/300,581, “Video Surveillance System Employing Video Primitives,” Attorney Docket No. 37112-225921.
For example, the video analytics engine of block 2 may determine the following information: detecting objects of interest; classifying objects in classes such as human, vehicle, or other; tracking of objects through a scene; determining if an object is inserted into a scene or removed from a scene; detecting an object moving against a flow of traffic; determining the “normal” size, shape, or speed of an object in an area; determining the “normal” path of an object through an area; determining if a customer reaches over into the teller area; determining if a person has slipped and fallen; and/or determining the density of a crowd in a scene.
The video analytics engine of block 2 may generate video primitives. The video primitives may be categorized in the following categories: time video primitives; blob video primitives; target video primitives; environmental video primitives; flow-control video primitives; and/or special purpose video primitives for banking business processes.
Each type of video primitive may contain generic primitive data. Generic primitive data may be generic identification information. Table 1 lists some exemplary generic primitive data that may include the following exemplary generic identification information.
Time video primitives may be generated periodically to provide a heartbeat to the system, even if nothing is happening in the video. Hence, no other video primitives are sent. The time video primitives may contain only generic primitive data.
Blob video primitives may be generated when a blob is detected. A blob may refer to a single frame instance of a spatially continuous moving target. Blob video primitives may contain generic primitive data and blob primitive data. The blob primitive data may be spatial descriptors. Table 2 lists some exemplary blob primitive data that may include the following exemplary information.
Target video primitives may describe a snapshot of a moving target and may contain generic primitive data and target primitive data. A target may refer to a complete description of a target over time (e.g., a sequence of blobs). Table 3 lists some exemplary target primitive data that may include the following exemplary information.
Environmental video primitives may describe an environmental change and may contain generic primitive data and environmental primitive data. Exemplary environmental primitive data may include, for example, a type of environmental change, e.g., lights on/off, camera moved; and/or a start and end time of change.
Flow-control video primitives may describe the detected motion for the flow within the video and may contain generic primitive data and flow-control primitive data. For flow-control video primitives, the scene may be decomposed into a series of grid elements, and the flow may be computed in each one. Exemplary flow-control primitive data may include, for example, a location of the grid element, e.g., a (x,y) coordinate of grid element; and/or a motion vector of the grid element, e.g., a (x,y) motion vector at that element.
Special purpose video primitives may be generated for special applications or for debugging. Examples of special purpose video primitives may include: a “basket run” video primitive; a high-value item stealing video primitive; a tide filter video primitive; an omni-directional video primitive; a reaching primitive; and/or a motion histogram video primitive.
A “basket run” video primitive may indicate the start and end point of detected line segments. This video primitive may be used, for example, for detection of a retail “basket run,” where a basket full of merchandise goes through the front door of a store without payment.
A high-value item stealing video primitive may indicate the position and direction, e.g., up and/or down of motion blocks. This may be used, for example, for detection of someone “cleaning out” a retail shelf. This video primitive may be used to count the number of times a customer has reached into a shelf.
A reaching primitive may indicate a person reaching into a restricted area from a wrong direction.
A tide filter video primitive may indicate the end position of the tide area for each row. This video primitive may be used for automatic detection of tidal areas in maritime environments.
An omni-directional video primitive may indicate warped image dimensions. This video primitive may be used for determining location and orientation of objects in an omni-directional image view.
A motion histogram video primitive may indicate a motion histogram for the whole image. This video primitive may be used for various anomaly detection rules.
In block 6, the activity inference engine may be considered to be a query engine that analyzes the video primitives based on one or more user-defined queries or rules from block 5 and determines activities of interest. This query language is described in the following, for example, which is incorporated herein by reference: U.S. Patent Publication No. 2005-0162515 A1, “Video Surveillance System.” The queries or rules include rule elements and combinators.
Rule elements detect properties and behaviors of targets. Table 4 lists some exemplary rule elements, along with examples thereof.
Combinators may combine rules and/or other combinators as parameters (known as sub-events). An event occurs if the combination succeeds. Table 5 lists exemplary combinators, along with examples thereof.
An exemplary automated video surveillance system may also track individuals across cameras. This may allow for the detection of unusual or suspicious behavior around an entire bank as targets are tracked from one camera to another. This tracking of targets using multiple cameras is described in the following, for example, which is incorporated herein by reference: U.S. patent application Ser. No. 11/098,579, “Wide-Area Site-Based Video Surveillance System,” Attorney Docket No. 37112-215813.
An exemplary embodiment of the integrated mode combining video analytics and additional data may convert the external information into external primitives, which then can be fed to activity inferencing together with the video primitives, as illustrated in
In another exemplary embodiment of the integrated mode, the video analytics and the additional data may be combined as illustrated in
Based on the exemplary embodiment of the invention, one of ordinary skill in the art would understand how particular banking business processes may be monitored by using the user-defined rule from block 5 in conjunction with the video primitives generated from block 2. The following is a list of exemplary banking business processes that may be monitored and exemplary rules that may be generated to monitor these banking business processes.
The following exemplary banking business processes involving physical security may be monitored by the following exemplary rules.
Intrusion detection within a banking facility may be detected when a person crosses perimeter, e.g., a tripwire, after normal hours. Exemplary rules are further discussed in the following, which are incorporated herein by reference U.S. Pat. No. 6,696,945, “Video Tripwire”; and U.S. Pat. No. 6,970,083, “Video Tripwire”.
Vandalism may be detected within or around a banking facility when an object, e.g., a poster or graffiti, is “inserted” onto a wall area. Unauthorized access to a secure area, e.g. “tailgating,” may be detected when two people enter a doorway area with only one access control event. Suspicious behavior around cars in a parking lot may be detected, for example, when a person visits multiple cars in a parking lot and lingers around each for a period of time. Suspicious behavior in an ATM vestibule may be detected, such as loitering or sleeping.
Suspicious behavior around an ATM may be detected, e.g. a person standing in front of the machine without a transaction, which may be an indication of installing a skimmer; or a person approaching another person who is conducting a transaction, which may indicate the person is trying to attack the person using the ATM. These applications may require the integrated mode, combining video analytics with additional non-video information (the presence or absence of a transaction).
Additional exemplary banking business processes involving physical security may be monitored, such as, for example, detection of a change in the appearance of an ATM, potentially indicating the installation of a skimmer; detection when a camera is tampered with, e.g. when the camera is moved, covered, or blinded; and/or detection of employees violating internal processes, such as a single employee carrying out a transaction requiring two people, for example, at a teller station, or at a safe.
Detection of suspicious transactions may also be monitored, such as a teller withdrawal without the presence of a customer. This application requires the integrated mode, combining video analytics with additional non-video information, i.e., the presence or absence of a transaction.
Embodiments of the invention may monitor banking business processes involving public safety. For example, embodiments of the invention may detect people slipping and falling on bank property. Exemplary rules related to detection of a slip and fall event are discussed in the following, which is incorporated herein by reference: U.S. patent application Ser. No. 11/165,182, “Detection of Change in Posture in Video,” Attorney Docket No. 37112-219109.
Embodiments of the invention may detect people running in a bank or a parking lot, for example, by detecting a person moving at speed faster than normal in the bank or parking lot area; and may also detect a car speeding in a parking lot by detecting a car moving faster than normal in a parking lot area. Exemplary rules related to detection of people or objects moving quickly are further discussed in the following, which is incorporated herein by reference: U.S. Patent Application Publication No. 2006-0072010-A1 “Target Property Maps for Surveillance Systems”.
Embodiments of the invention may detect an assault or abduction in a bank parking lot or at an ATM, for example, by detecting two or more people coming together from different directions and one running away afterward; or one person running towards another one and then both leaving in a different direction.
Embodiments of the invention may detect an obstruction of a fire exit, for example, by detecting an object inserted in fire door area.
Embodiments of the invention may detect crowding in a bank area or outside of the bank, for example, by detecting a high density crowd that appears for longer than “x” minutes in a bank area.
Embodiments of the invention may detect a vehicle driving in the wrong direction at the drive-through teller.
Embodiments of the invention may monitor banking business processes involving intelligence data gathering. For example, embodiments of the invention may track people through bank areas to determine, for example, how many people go through a bank area, for example, by counting alerts each time person enters or exits the area. Embodiments of the invention may track where people dwell on bank premises, for example, by counting how long people loiter in a particular area.
Embodiments of the invention may monitor queue length, for example, by counting how many people are in line. Exemplary rules for detecting counting people are further discussed in the following, which is incorporated herein by reference: U.S. patent application Ser. No. 11/165,435, “Target Detection and Tracking from Overhead Video Streams,” Attorney Docket No. 37112-219452.
Embodiments of the invention may monitor bank cleanliness, for example, by monitoring a bank area for spills or fallen merchandise or displays. Embodiments of the invention may monitor an unattended delivery, for example, when delivery people are not there at the right time, or move to unauthorized areas of the bank.
Embodiments of the invention may measure customer service processes through people tracking by monitoring employee-customer interactions, for example, by comparing the time a customer service employee spends with a customer.
FIG. 3Oa depicts an ATM with no skimmer device in area 341, and FIG. 3Ob depicts the system detecting a card skimmer device 342 placed on the ATM shown in FIG. 3Oa.
In block 8, the report generation engine accumulates events into a report that may provide an overview of banking business processes. For example, for the banking business process of intelligence data gathering, an example report may include people counting and queue length statistics. For people counting, if it is desired to count the number of people who enter or exit a bank, the report generation engine may be used to agglomerate individual entry and exit events (detected via a tripwire(s), for example) and provide a time-based histogram of consumer behavior. For queue length monitoring the automated video surveillance system may determine typical waiting times.
The invention has been described in detail with respect to preferred embodiments, and it will now be apparent from the foregoing to those skilled in the art that changes and modifications may be made without departing from the invention in its broader aspects. The invention, therefore, as defined in the appended claims, is intended to cover all such changes and modifications as fall within the true spirit of the invention.
This application is a non-provisional application of, and claims priority to, U.S. Provisional Application No. 60/846,337, filed Sep. 22, 2006, entitled “VIDEO ANALYTICS FOR BANKING BUSINESS PROCESS MONITORING,” of common assignee to the present application, and the contents of which is incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 60846337 | Sep 2006 | US |
