Patent Application
20060238616
The present invention relates generally to the field of video image processing. More specifically, the present invention pertains to devices and methods for interfacing with one or more monitoring devices within a video monitoring system.
Video monitoring systems are used in a variety of applications for monitoring objects within an environment. In security applications, for example, such systems are sometimes employed to track individuals or vehicles entering or leaving a building facility or security gate, or to monitor individuals within a store, office building, hospital, or other such setting where the health and/or safety of the occupants may be of concern. In the aviation industry, for example, such systems have been used to monitor the presence of individuals at key locations within an airport such as at a security gate or parking garage.
In certain systems, video image-processing modules may be employed to process video images that can be used to detect the occurrence of events within a region of interest. Such video image-processing modules are typically stand-alone modules that are hard-coded to run a number of algorithms or routines based on their intended use. In certain designs, for example, such modules may be provided as part of a network server software program that interacts with one or more monitoring devices (e.g. digital video monitors) based on a number of pre-programmed rules or constructs. A graphical user interface may be provided in certain systems to permit the user to interact with the software program in order to task the video monitoring system in a particular manner.
While many video image-processing modules provide a means to change settings relating to the implementation of algorithms already programmed therein, such devices typically lack the ability to easily add algorithmic functionality or analysis functions without having to re-code the entire software program. In some cases, the addition of new clients may require the user to re-formulate the entire software program to accommodate the additional clients. Such requirements may increase the complexity of the system, and may prevent future expansion of the system as new components are added to the existing system.
The present invention pertains to devices and methods for interfacing with one or more monitoring devices within a video monitoring system. A video monitoring system in accordance with an illustrative embodiment of the present invention may include a host application including one or more monitoring devices, a video image-processing appliance manager called by the host application, and one or more appliance modules called by the appliance manager and adapted to run a number of video image-processing algorithms or routines. The appliance manager can be configured to run an initiation algorithm for acquiring information about the type of monitoring devices employed by the system, the type of appliance modules to be tasked, as well as other parameters. Once initialized, the appliance manager can then be configured to run a video analysis algorithm that analyzes images acquired from the monitoring devices in order to detect the occurrence of one or more events contained within an event database. Once a configured event is detected, or at some other desired time established by the user, the appliance manager can be configured to close the video analysis algorithm and output a response notifying the user of the occurrence of the event.
The appliance manager can be configured to permit the modular incorporation of future appliance modules within the video monitoring system without having to re-code the host application software program. In certain embodiments, the video monitoring system may include a video motion detection module, a video motion tracking module, an object classification module, an event detection module, and an action dispatcher module. Other appliance modules can be added to the system to perform other tasks, as desired. In certain embodiments, the appliance manager can include a manager harness to permit the user to send various parameters to the appliance manager for evaluating or testing new or existing appliance modules, and/or for processing previously recorded video streams to detect the occurrence an event.
An illustrative method of interfacing with one or more monitoring devices within a video monitoring network may include the steps of providing a video image-processing appliance manager in communication with a host application including one or more monitoring devices, associating the appliance manager with one or more appliance modules each adapted to run a number of video image-processing algorithms or routines, analyzing video images received from at least one of the monitoring devices using one or more of the appliance modules, and then outputting a response to the host application and/or a user.
The following description should be read with reference to the drawings, in which like elements in different drawings are numbered in like fashion. The drawings, which are not necessarily to scale, depict selected embodiments and are not intended to limit the scope of the invention. Although examples of algorithms and processes are illustrated for the various elements, those skilled in the art will recognize that many of the examples provided have suitable alternatives that may be utilized.
A number of browsers or terminal stations 18 equipped with a user interface 20 (e.g. a graphical user interface) can be connected to the network 12 and tasked to control the operation of the DVSS's 14 in a particular manner. In some embodiments, a terminal monitor 22 (e.g. a closed-circuit television monitor) can also be provided in addition to, or in lieu of, the browsers or terminal stations 18 to view images acquired from one or more of the DVSS's 14. The browsers or terminal stations 18 can be configured to interact with a host application software program 24 that can be used to task the DVSS's 14 in a particular manner. Based on user input via the user interface 20, for example, the host application software program 24 can be used to change operational parameters of the DVSS's 14 (e.g. PTZ settings, ROI settings, resolution, frame rate, etc.), and/or to task a video image processing appliance manager 28 to perform certain user-defined tasks such as motion detection, motion tracking, etc. If, for example, the user desires to increase the resolution of images received by one or more of the DVSS's 14 in order to perform facial recognition of an individual within a region of interest, the host application software program 24 can be configured to accept a command causing one of the DVSS's 14 to zoom-in on the subject's face and increase the image capture rate. In the illustrative embodiment of
As can be further seen in
The host applications 34 can comprise separate components from the DVSS's (e.g. a stand-alone software package), or can be formed integral with one or more of the DVSS's and provided as a single component, if desired. In certain embodiments, for example, one or more of the DVSS's may comprise a physically separate video camera that is connected to an existing software-based host application adapted to run on the Internet, an intranet connection, and/or on an individual workstation equipped with a user interface 36. In such applications, each of the associated DVSS's can be connected to their associated host application 34 using an application program interface (API) or other suitable interface.
The host applications 34 may comprise one or more existing host application software programs contained on a network server, browser, terminal station, or other platform. The functionality provided by the existing host applications 34 will typically vary depending on their intended use. If, for example, the host applications 34 are adapted to interface with network-based control, access, and security systems/products, the host applications 34 may include an existing security software program that can be used to task a number of DVSS's to pan, tilt, and zoom to a tracked motion within a region of interest. Examples of other types of host applications 34 may include, but are not limited to, building management applications (e.g. HVAC control), life safety applications (e.g. fire protection, medical care, etc.), asset location applications, and energy management applications.
The appliance manager 32 will typically comprise a separate module from the host applications 34, allowing the appliance manager 32 to be linked with the user's existing system without having to significantly modify or reprogram the existing software to accommodate new DVSS clients as they are added to the system. It should be understood, however, that the appliance manager 32 and host applications 34 could be incorporated together as a single, stand-alone module, if desired.
The appliance manager 32 can be configured to maintain one or more business objects, which can include various information about the region or regions of interest to be monitored, any events configured by a configurator 44, as well as various configuration information about the host applications 34 connected to the system. In certain embodiments, for example, the appliance manager 32 can be configured to maintain a camera structure list and an event structure list containing information about the type of DVSS's employed and the type of events configured within the system. Such lists can be stored within a memory unit or database (e.g. database 38) and recalled each time the appliance manager 32 receives an initialization call 40 or detection call 42 from one or more of the host applications 34.
The video monitoring system may include a configurator 44 that can be used in the direct manipulation and configuration of images or other data received by the host applications 34. A tuning request/call 46 received from one or more of the host applications 34 can be used by the configurator 44 to tune the appliance manager 32 and/or other desired system components to function in a particular manner. If, for example, a user desires to increase the frame capture rate of one of the DVSS's (e.g. a video camera) field of view, the host application 34 can be configured to send a tuning request or call 46 to the appliance manager 32 that can be used by the configurator 44 to coordinate such a change. Once the change has been made, the appliance manager 32 can then be configured to pass the newly configured video stream through. In some embodiments, the configurator 44 can also be configured to send a response to the host application 34 and/or to the user indicating whether the tuning invocation succeeded or failed.
The appliance manager 32 can be connected to a database 38 that can be configured to store information received from the DVSS's as well as parameters received by the configurator 44, as directed by the appliance manager 32 and/or user via the user interface 36. In certain embodiments, the database 38 can be two separate databases residing at different servers, wherein one database can be linked to the host application 34 and the other database can be linked to the appliance manager 32. In other embodiments, the database 38 may comprise a single database, or multiple databases existing on a single server.
The appliance manager 32 can be configured to interact with a number of plug-in appliance modules each adapted to run various video image-processing algorithms or routines that can be used to perform certain user-defined image processing functions. In the illustrative embodiment of
The appliance manager 32 can be configured to permit the modular incorporation of future appliance modules 58, as desired. If, for example, the user desires to add a facial detection module or rapid eye detection module to the video monitoring system, the appliance manager 32 can be configured to accept a software call, network socket, physical port (e.g. a USB port, Firewire/IEEE 1394 port, parallel/serial port, etc.), and/or wireless port that can be used to add the additional modules. Since an appliance manager 32 is provided to interface with the host applications 34, the addition of future plug-in modules does not require the user to re-code or re-formulate the existing host applications 34. In addition, the appliance manager 32 may provide the user with the ability to change the implementation and/or features of existing functionality without significantly affecting the operation of the DVSS's.
The appliance manager 32 can be configured to run a video image-processing algorithm or routine that continuously monitors the camera structure list and configured events list to determine whether an event has been detected by one or more of the DVSS's. When an event contained within the event list is detected, the appliance manager 32 can be configured to transmit a result 60 back to the host application 34 along with an action request 62 tasking one or more of the DVSS's to operate in a particular manner. If, for example, the video motion tracking module 50 detects that an object is moving in a particular direction within a region of interest, the appliance manager 32 can be configured to provide a result 60 to the appropriate host application 34 informing it that the object is being tracked along with an action request 62 tasking one or more associated DVSS's to track the object.
Turning now to
Once the initialization algorithm 72 has been initialized (or if re-initialization at decision block 70 is unnecessary or undesired), the appliance manager 32 may next invoke a video analysis algorithm (block 74) that analyzes video images acquired from each host application 34 and/or DVSS 14 used in detecting the configured event. During this step, the appliance manager 32 can be configured to call one or more of the appliance modules 48, 50, 52, 54, as needed, to process the video images and/or other data received from the DVSS's tasked by the monitoring system. If the appliance manager 32 detects the occurrence of the configured event, or if some other user-defined event occurs, the appliance manager 32 can be configured to invoke a closing algorithm (block 76) and output a response to the host application 34 and/or a user. Otherwise, the appliance manager 32 can be configured to repeat the video analysis algorithm step 74 for each frame to be analyzed until an event is detected by one of the DVSS's.
Once the initialization function 82 has been invoked, the appliance manager 32 can be configured to receive a number of parameters relating to the type of images or other such data to be acquired. As shown by reference to block 84, for example, the appliance manager 32 can be configured to retrieve a device independent bitmap (DIB) header that contains image header information relating to the types of images that will be received in subsequent analyzing steps (e.g. during calls for the video analysis algorithm 74 of
As can be seen by reference to block 86 in
Once the DIB header 84 and video source name input parameters 86 are defined, the appliance manager 32 can then be configured to output a unique handle (block 88) that can later be used to identify a particular instance of video processing applicable to a particular video source. From this information, the appliance manager 32 may then set-up a number of internal variables and/or structures, which can later be recalled when the video analysis algorithm 74 is called by the handle.
Once a unique handle is created, the appliance manager 32 can be configured to analyze the ROI information from each DVSS to be tasked, any events associated with the ROI, and any actions associated with each event, as shown generally by reference to block 90. If there are any events configured within the event structure list, the appliance manager 32 considers processing that ROI, otherwise it ignores the processing of that ROI. With respect to each DVSS within the system, the appliance manager 32 may consider processing by the DVSS only if there is at least one ROI to be considered, thereby saving processing resources only for those DVSS's to be tasked to perform a particular function.
Once the appliance manager 32 has the requisite information about the ROI, it may then create a camera structure database and add the appropriate ROI information to that database, as shown generally by reference to block 92. After populating the camera structure database with information about each ROI, the appliance manager 32 may then add a camera node to a camera list, as show generally by reference to block 94. The camera list may comprise a list of those global variables pertaining to each DVSS tasked to process at least one ROI. The appliance manager 32 can also be configured to retrieve information pertaining to events associated to the DVSS, and then store the configured events as an event node, as shown generally by reference to block 96. After successfully creating an event node, the appliance manager 32 can then be configured to store the node in an event list, which similar to the camera list, contains global variables relating to those event configured within the system.
Once the events are retrieved and stored, the appliance manager 32 can be configured to scan the events within the event list and determine which appliance module(s) are required to execute the configured events, as shown generally by reference to block 98. If, for example, a DVSS within the video monitoring system is configured to classify a moving object as either a human actor or a vehicle, the appliance manager 32 can be configured to call the object classification module 52 to process images received from that DVSS. The appliance manager 32 can be configured to task each DVSS within the system beginning with a first DVSS client and following with each successive DVSS client until all DVSS clients adapted to run the configured event have been identified and provided with a unique identifier. The appliance manager 32 may then create the required components and interface pointers necessary to execute the configured events, as shown generally by reference to block 100.
Referring back to
For each ROI DIB extracted, the appliance manager 32 can then be configured to call the appropriate module or modules to process images received from the configured DVSS clients, as shown generally by reference to block 114. If, for example, the appliance manager 32 is to be tasked to detect object motion within a ROI, the appliance manager 32 can be configured to call an algorithm or routine within the video motion detection module 48 while passing the ROI DIB to that module 48. The video motion detection module 48 can then be configured to output parameters (e.g. minimum boundary rectangle (MBR) coordinates, patch, features, etc.) for each moving object detected, which are then consolidated and stored within memory.
In certain embodiments, the video motion detection module 48 can be configured to detect motion using a color and edge-based algorithm, which involves the generation of mean and variance images based on the color and edge information of each of the frames within a video sequence. A first level of processing may utilize color information in order to extract motion regions within the DVSS's field of view as a first cue to extract motion regions from a frame. To account for the shape and contour of the moving object, a second level of processing may utilize various edge information contained within each of the image frames. The resultant model is then thresholded using standard deviation information to conclude whether a particular pixel within the field of view belongs to the foreground or background, leading to the formation of both a color and edge foreground confidence map. The final, motion-segmented result can then be obtained by collating the information obtained from both the color and edge foreground confidence maps. While the fusion of color and edge-based algorithms can be used to detect motion of an object within a ROI, it should be understood that other methods and techniques can be employed.
After tasking the video motion detection module 48 and receiving parameters relating to each moving object, the appliance manager 32 can then be configured to provide this information to the video motion tracking module 50. If the video motion tracking module 50 determines that the object is a moving target, the module 50 can then be configured to output a number of parameters (e.g. Object ID's associated to MBR, direction, speed, future state, etc.). From this information, the appliance manager 32 can then create memory blocks for the object structures and then populate the memory with information returned by video motion detection and video motion tracking modules 48, 50, as shown generally by reference to blocks 116 and 118.
Once the memory is populated with information from the video motion detection and tracking modules 48, 50, the appliance manager 32 can next determine whether to call the object classification module 52 in order to compute the behavior of a tracked object within a ROI. If so, the appliance manager 32 may then call the object classification module 52, causing that module 52 to process information received from one or more of the DVSS clients and output a number of parameters relating to the identity of a tracked object. If, for example, the object classification module 52 is configured to determine whether an object is an individual, the module 52 can be configured to output a number of parameters (e.g. height, gender, etc.) relating to the individual. After receiving this information, the appliance manager 32 can then repeat the steps at blocks 116 and 118 to create memory blocks for each classified object and then populate the object variables with information received from the object classification module 52. If desired, a sequential classification algorithm or routine may be employed to provide a more consistent output from the object classification module 52.
The video analysis algorithm 102 may be performed using live video feeds from the DVSS clients, or using a previously recorded video clip using, for example, a manager harness, as described below with respect to
Once the appliance manager 32 has received the desired information from various appliance modules 48, 50, 52, 54, the appliance manager 32 can then be configured to close the video analysis algorithm 102 and invoke a closing algorithm, as indicated, for example, by reference to block 76 in
Once the camera handle is received at block 124, the appliance manager 32 next determines whether any camera node within the system corresponds to the camera handle, as indicated generally by reference to decision block 126. If no corresponding camera node is found, the appliance manager 32 may end the algorithm 120 (block 132). If, however, there is a corresponding camera node, the appliance manager 32 can be configured to delete that node from the camera list, as shown generally by reference to block 128. The appliance manager 32 can then be configured to decrease the camera count and release the handle if the camera count is zero, as indicated generally by reference to block 130. Once released, the appliance manager 32 may then end the algorithm 120, as shown generally by reference to block 132. The appliance manager 32 can then be configured to release any memory allocated for the image processing, and/or release the interface pointers of other appliance modules employed, if any.
Referring now to
A notification routine 136 invoked within the action dispatcher module 56 can be configured to send an alarm or other response to the user and/or one or more of the host applications 34. If, for example, the monitoring system detects a vehicle entering an entrance during the video analysis algorithm 102 described above with respect to
A send-to-video-station routine 138 invoked within the action dispatcher module 56 can be configured to send real-time and/or previously recorded images to a particular operator station, or to multiple operator stations belonging to a particular area. During such routine 138, the action dispatcher module 56 can be configured to output the station or area name associated with the triggered event. If, for example, an event is detected by “Video Camera 1” within the video monitoring system, the action dispatcher module 56 can be configured to transmit images from that video camera to one or more operator stations tasked to monitor that particular camera. The images can be transmitted upon the detection of an event (e.g. a motion detection event), or at some other desired time period. If desired, the appliance manager 32 can invoke a log message routine 140 within the action dispatcher module 56 that logs any errors and/or event messages as they occur.
Having thus described the several embodiments of the present invention, those of skill in the art will readily appreciate that other embodiments may be made and used which fall within the scope of the claims attached hereto. Numerous advantages of the invention covered by this document have been set forth in the foregoing description. It will be understood that this disclosure is, in many respects, only illustrative. Changes can be made with respect to various elements described herein without exceeding the scope of the invention.
