Patent Application
20150208042
The present invention is related to image processing and computer vision, and in particular to automatic commissioning of video analytic algorithms.
Intelligent video surveillance systems use image processing and computer vision techniques (i.e., video analytic software) to analyze video data provided by one or more video cameras. Based on the performed analysis, events are detected automatically without requiring an operator to monitor the data collected by the video surveillance systems.
However, the installation of intelligent video surveillance systems requires the video analytic software to be configured, including setting parameters associated with the video analytic software to optimize performance of the video analytic software in correctly identifying events in the analyzed video data. This process, known as commissioning the system, is time and labor intensive, typically requiring a technician to test different combinations of parameters.
According to an embodiment of the invention, a method of automatically commissioning an intelligent video system, includes receiving a target video that includes images and/or video data associated with the intelligent video system to be commissioned, installing one or more sets of selected parameters in the intelligent video system, testing performance of the one or more sets of selected parameters to find a best match set of parameters in response to an event output generated with respect to one of the one or more of sets of parameters by application of the one or more sets of selected parameters to video content, and receiving one or more ground truth events that represent correct analysis of the target video via at least one of an automated annotation, a manual annotation, and a third party provider, wherein the best match set of parameters is used to commission the intelligent video system for video surveillance.
In addition to one or more of the features described above, or as an alternative, further embodiments could include extracting a first set of one or more visual feature descriptors associated with the received target video.
In addition to one or more of the features described above, or as an alternative, further embodiments could include providing the one or more visual features descriptors to a parameter database that is comprised of a plurality of entries, each entry including a set of one or more stored visual feature descriptors and associated parameters tailored for the set of stored visual feature descriptors.
In addition to one or more of the features described above, or as an alternative, further embodiments could include searching the parameter database based on the extracted visual feature descriptors to locate one or more best matches between the extracted visual feature descriptors and the stored visual feature descriptors and retrieving the one or more sets of selected parameters associated with the one or more best matches.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that testing the performance of the one or more sets of parameters includes: A. analyzing the target video with video analytic software configured with one of the plurality of sets of parameters to generate an event output, B. comparing the event output generated with respect to one of the plurality of sets of parameters with the one or more ground truth events to calculate performance parameters that define the performance of the selected set of parameters, and C. selecting a subsequent set of parameters based on the performance parameters associated with the analyzed set of optimized values, D. repeating steps A through C until the generated performance parameters are satisfactory.
In addition to one or more of the features described above, or as an alternative, further embodiments could include uploading the target video to a storage location.
In addition to one or more of the features described above, or as an alternative, further embodiments could include notifying the third party provider regarding availability of the target video.
In addition to one or more of the features described above, or as an alternative, further embodiments could include downloading the target video to the third party provider.
In addition to one or more of the features described above, or as an alternative, further embodiments could include uploading one or more ground truth events to the storage location.
In addition to one or more of the features described above, or as an alternative, further embodiments could include downloading one or more ground truth events from the storage location.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that the one or more sets of selected parameters are provided by an algorithm designer.
According to an embodiment of the invention, an auto-commissioning system for automatically commissioning an intelligent video surveillance system, includes an input that receives target video that includes images and/or video data associated with the intelligent video system to be commissioned, a parameter selector testing performance of one or more sets of selected parameters to find a best match selected parameters in response to an event output generated with respect to the one or more sets of selected parameters by application of the one or more sets of selected parameters to video content; wherein the best match selected parameters are used to commission the intelligent video system for video surveillance, and at least one of manual annotation, automated annotation, and third party provider to identify one or more ground truth events that represent correct analysis of the target video.
In addition to one or more of the features described above, or as an alternative, further embodiments could include a video feature extractor that extracts a first set of one or more visual feature descriptors associated with the received target video.
In addition to one or more of the features described above, or as an alternative, further embodiments could include a parameter database that is comprised of a plurality of entries, each entry including a set of one or more stored visual feature descriptors and associated parameters tailored for the set of stored visual feature descriptors, wherein the parameter database is searched based on the first set of one or more visual feature descriptors to obtain one or more sets of selected parameters.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that testing the performance of the one or more sets of parameters includes: A. analyzing the target video with video analytic software configured with one of the plurality of sets of parameters to generate an event output, B. comparing the event output generated with respect to one of the plurality of sets of parameters with the one or more ground truth events to calculate performance parameters that define the performance of the selected set of parameters, and C. selecting a subsequent set of parameters based on the performance parameters associated with the analyzed set of optimized values, and D. repeating steps A through C until the generated performance parameters are satisfactory.
In addition to one or more of the features described above, or as an alternative, further embodiments could include a storage location to receive the target video.
In addition to one or more of the features described above, or as an alternative, further embodiments could include a communication to notify the third party provider regarding availability of the target video.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that the target video is downloaded from the storage location to the third party provider.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that one or more ground truth events is uploaded to the storage location.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that the one or more sets of selected parameters are provided by an algorithm designer.
According to an embodiment of the invention, a method of automatically commissioning an intelligent video system, includes receiving a target video that includes images and/or video data associated with the intelligent video system to be commissioned, installing one or more sets of selected parameters in the intelligent video system, testing performance of the one or more sets of selected parameters to find a best match set of parameters in response to an event output generated with respect to one of the one or more of sets of parameters by application of the one or more sets of selected parameters to video content, and receiving one or more ground truth events that represent correct analysis of the target video via a ground truth labelling tool, wherein the best match set of parameters is used to commission the intelligent video system for video surveillance.
In addition to one or more of the features described above, or as an alternative, further embodiments could include extracting a first set of one or more visual feature descriptors associated with the received target video.
In addition to one or more of the features described above, or as an alternative, further embodiments could include providing the one or more visual features descriptors to a parameter database that is comprised of a plurality of entries, each entry including a set of one or more stored visual feature descriptors and associated parameters tailored for the set of stored visual feature descriptors.
In addition to one or more of the features described above, or as an alternative, further embodiments could include searching the parameter database based on the extracted visual feature descriptors to locate one or more best matches between the extracted visual feature descriptors and the stored visual feature descriptors and retrieving the one or more sets of selected parameters associated with the one or more best matches.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that testing the performance of the one or more sets of parameters includes: A. analyzing the target video with video analytic software configured with one of the plurality of sets of parameters to generate an event output, B. comparing the event output generated with respect to one of the plurality of sets of parameters with the one or more ground truth events to calculate performance parameters that define the performance of the selected set of parameters, and C. selecting a subsequent set of parameters based on the performance parameters associated with the analyzed set of optimized values, D. repeating steps A through C until the generated performance parameters are satisfactory.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that the ground truth labelling tool is an automated ground truth labelling tool.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that the ground truth labelling tool is manual ground truth labelling tool.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that the ground truth labelling tool is a combination manual ground truth labelling tool and automated ground truth labelling tool.
In addition to one or more of the features described above, or as an alternative, further embodiments could include that the one or more sets of selected parameters are provided by an algorithm designer.
Technical function of the embodiments described above includes receiving one or more ground truth events that represent correct analysis of the target video from a third party ground truth provider or a manual or automated ground truth labelling tool, and receiving one or more ground truth events that represent correct analysis of the target video via a ground truth labelling tool.
Other aspects, features, and techniques of the invention will become more apparent from the following description taken in conjunction with the drawings.
The subject matter, which is regarded as the invention, is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which like elements are numbered alike in the several FIGURES:
The performance of video analytic software 18 is tailored for a particular application (i.e., the environment in which the intelligent video system is installed and/or the type of detection to be performed by the intelligent video system) by varying a plurality of parameters associated with video analytic software 18. These parameters may include thresholds for decision making, adaptation rates for adaptive algorithms, limits or bounds on acceptable computed values, etc. The process of selecting the parameters of video analytic software 18 during initialization of intelligent video surveillance system 10 is referred to as commissioning the system. Typically, commissioning an intelligent video surveillance system is done manually by a technician, who tests different combinations of parameter values until the video analytic software correctly interprets test data provided. However, this process is time-consuming and therefore expensive.
In the embodiment shown in
In general, auto-commissioning system 12 automatically extracts video feature descriptors (i.e., features that describe the video content) from the test video data. Video feature descriptors may include aspects such as illumination changes, presence of shadows, busy versus non-busy scene, background motion, camera vibration, etc. Visual feature descriptors may be binary in nature (e.g., either a scene is busy or non-busy), may be continuous and expressed by a range of values (e.g., illumination may be expressed as any number between 0-10), may be a probability distribution in space or time of some characteristic of the video (e.g., the distribution of shadows over some quantization of time), etc. For example, an outdoor surveillance system overlooking a park may include dramatic illumination changes, the presence of changing shadows, and be relatively busy. In contrast, a surveillance system employed in a museum hallway after hours may have no illumination changes, no shadows, and be non-busy. For each application, a different set of parameters will likely be employed to maximize performance.
Auto-commissioning system 12 extracts video feature descriptors from the test video data, and compares the extracted video feature descriptors to a parameter database that includes a plurality of combinations of video feature descriptors (describing different types of video data), each combination associated with parameters tailored to the combination of video feature descriptors. A best match between the extracted video feature descriptors and one of the plurality of combinations of video feature descriptors in the parameter database is determined, and the parameters associated with the best match is provided for provision to video analytic software 18.
In one embodiment, auto-commissioning system 12 is located in a centralized control room remote from intelligent video system 10. Test video data provided by intelligent video system 10 is communicated to centralized auto-commissioning system 12 for analysis, with parameters subsequently communicated from auto-commissioning system 12 to intelligent video system 10. Communication between devices may be wired or wireless, according to well known communication protocols (e.g., Internet, LAN). In other embodiments, auto-commissioning system 12 is portable/mobile (i.e., laptop or other mobile processing device), allowing a technician commissioning a system to connect auto-commissioning system 12 to intelligent video system 10 locally.
Parameter database 20 is created by combining information from a plurality of different sources. In the embodiment shown in
Inputs provided to parameter database 20 are provided in a format that allows for comparison and matching (search and retrieval) based on visual feature descriptors. Inputs may be loaded manually by a user via a user interface such as a keyboard, or may be uploaded from a remote system. In some instances, rather than manually enter a plurality of visual feature descriptors defining a particular application (i.e., busy, non-busy, motion, no motion, etc.), actual video content may be provided as an input and a visual feature extractor operating on a computer system analyzes and extracts desired video feature descriptors. Visual feature descriptors, whether manually entered or extracted from actual videos are then paired with parameters tailored or selected for the identified visual feature descriptors. Each entry stored in parameter database 20 relates visual features associated with a particular video scene with a corresponding plurality of parameters. Commissioning of an intelligent video system then becomes a matter of extracting video features associated with the field of view of the intelligent video system (i.e., test video data) and comparing the extracted video features with those stored in parameter database 20 to find a match or best fit. The parameters associated with the matching entry and then selected as the parameters for the intelligent video system.
In the embodiment shown in
Expert/commissioning knowledge is provided at step 28, and includes information collected from experts regarding parameters associated with various visual features. Expert knowledge may be provided via a user interface, or may be provided as a stored file. For example, an expert may provide a database entry with respect to a certain combination of visual features (e.g., moving shadows, changing illumination) and the experts' opinion regarding parameters best suited for the visual features provided. At step 30, the input provided by experts is organized into entries that include a plurality of visual features (i.e., set of visual features) describing a particular type of environment or conditions, and selected parameters for each set of visual features. Each entry is then stored to parameter database 20. In this way, the experience/human knowledge of a commissioning expert is organized as part of parameter database 20.
A collection of videos with expert defined parameters is provided as another input to parameter database 20. In one embodiment, the collection of video content represents problematic videos that required expert analysis to determine the best parameter values. Visual feature descriptors are extracted from each video at step 32, and the parameters previously selected by experts are paired with the extracted visual features to create database entries that include the plurality of video feature descriptors and selected parameters for each video content sample in the collection at step 34. The database entries are then stored in parameter database 20. The features and parameters in database 20 may change over time as features are added or eliminated and as parameters are added or eliminated with the change of video analytics algorithms.
Entries created by each of the plurality of different inputs are stored to parameter database 20 to allow for subsequent search and retrieval of database entries. In one embodiment, duplicative entries (i.e., those entries having the same visual feature descriptors as other entries) may be combined with one another by averaging the parameter values associated with each duplicative entry or by deleting one of the entries. Other well known methods of handling duplicate, missing, or contradictory information in databases may also be used such as clustering or functional approximation.
Auto-commissioning system 12 receives target video from intelligent video surveillance system 10. Feature extractor 42 extracts visual feature descriptors from the target video provided by intelligent video system 10. In the embodiment shown in
In addition to automatic extraction of visual feature descriptors, a user may manually provide input regarding visual feature descriptors associated with the target video via front-end GUI 44. The visual feature descriptors provided by a user via front-end GUI 44 are saved as a second visual feature descriptor. In other embodiments, visual feature descriptor input is provided only via automatic visual feature descriptor extraction, with no input required from the user.
If visual features descriptors are input from both a user and automatic visual feature extraction, then the first visual feature set and second visual feature set are combined by feature accumulator 46. The combination may include averaging of the first visual feature set with the second visual feature set, selection of maximum values within the first visual feature set and the second visual feature set, or other useful forms of combining the visual feature descriptor sets. The combined visual feature set is provided to parameter database 48 for search and retrieval of optimal parameters. In one embodiment, the combined visual features are organized into a vector as described with respect to
Parameter database 48 searches for stored entries matching or most closely matching the received visual feature descriptors. There are a number of search and retrieval algorithms that may be employed to match the visual feature descriptors associated with the target video to the visual feature descriptors stored in parameter database 48. For example, search and retrieval methods such as Kd-Tree searches and R-Tree search may be employed. The Kd-Tree search is described in detail by the following publication: Michael S. Lew, Nicu Sebe, Chabane Djeraba, Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computer, Communication, and Applications (TOMCCAP), Volume 2, Issue 1 (February 2006), Pages: 1-19. The R-tree search is described in detail by the following publication: V. S. Subrahmanian, “Principles of Multimedia Database Systems”, Morgan Kaufmann, January 1998. The results of the database search and retrieval are database entries having visual feature descriptors most closely matching the visual descriptors provided with respect to the target video.
In one embodiment, the parameters retrieved by parameter database 48 are provided directly to intelligent video surveillance system 10 for commissioning of the system. In the embodiment shown in
In the embodiment shown in
Based on the difference between the video analytic results and the ground truth results, parameter selector 52 calculates a performance value(s) with respect to the current set of selected parameters. Fast parameter selector 52 compares the current performance value with previously measured performance values associated with different sets of selected parameters, and uses the difference between the measured performance values (i.e., a parameter gradient) to select a next set of parameters to test.
For example, parameter selector 52 analyzes the target video with a first set of parameters retrieved by parameter database 48 and results are compared to the defined ground truth to define first performance values, and a second set of parameters retrieved by parameter database 48 (or selected based on the results of a previous set) and results are compared to the defined ground truth to define second performance values. The first and second set of performance values are compared to one another to define a parameter gradient that is used by parameter selector 52 to select a subsequent set of parameters to test. When the performance values indicate a threshold level of performance, the process ends and the selected parameters are provided to intelligent video surveillance system 10 for commissioning. When the performance values do not indicate a threshold level of performance, a new set of parameters are chosen by any of a number of well known optimization algorithms.
The system and method described with respect to
In one embodiment, ground truth labelling tool 50a is utilized to expedite and streamline ground truth labelling tasks. In certain embodiments, ground truth labelling tool 50a automates ground truth labelling, while in other embodiments, ground truth labelling tool 50a facilitates manual analysis and entry of ground truth labelling. In certain embodiments, ground truth labelling tool 50a utilizes a combination of automated and manual ground truth labelling and analysis. In certain embodiments, ground truth labelling tool 50a is utilized contemporaneously during initial parameter identification and configuration to parameter database 48.
In one embodiment, feature extractor 42, feature accumulator 46, and parameter database 48 can be selectively omitted with the selected set(s) of parameters from parameter database 48 being replaced with default values chosen by the algorithm designer.
In one embodiment, use of ground truth labelling tool 50a and parameter identification and configuration from parameter database 48 are performed during business hours, to facilitate operation of parameter selector 52 during non-business hours. Advantageously, results may be ready for review and verification during business hours the following day.
In one embodiment, the target video is uploaded to a storage location 60. Storage location 60 may be an FTP server or any other suitable storage location accessible to a third party ground truth provider 64. In certain embodiments, after the target video is uploaded to the storage location 60, a communication or notification is sent to the third party ground truth provider 64 via communication interface 62. Communication interface 62 may provide an electronic message or any other suitable notification. Third party ground truth provider 64 may download the target video from the storage location. Third party ground truth provider 64 may utilize any known techniques to label and identify ground truth, including but not limited to manual analysis, automated analysis, and any combination thereof. Ground truth events and files may be uploaded to another or same storage location 60 or 66. In certain embodiments, a communication or notification is sent via communication interface 62 to notify a user that the ground truth events are ready to be used in conjunction with the parameter selector 52.
In one embodiment, feature extractor 42, feature accumulator 46, and parameter database 48 can be selectively omitted with the selected set(s) of parameters from parameter database 48 being replaced with default values chosen by the algorithm designer.
In one embodiment, use of third party ground truth provider 64 allows for ground truth event identification to occur during non-business hours. Certain third party ground truth providers 64 may be in another time zone from the user to allow third party ground truth provider 64 to operate during local business hours to complement the work flow described. Accordingly, results may be ready for review and verification during business hours the following day.
While the invention has been described with reference to an exemplary embodiment(s), it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment(s) disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
This application is a continuation-in-part of U.S. patent application Ser. No. 13/175,540, the entire contents of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13175540 | Jul 2011 | US |
| Child | 14600388 | US |
