The present invention pertains to security and particularly to maintaining security in large and important facilities. More particularly, the invention pertains to video-based security.
The invention is a system for image-based synthesis of open dynamic social network. The system may automatically map and construct social networks from image data. It may also provide discovery of other networks and analysis of them. The system may recognize networks that span over long periods of time and great distances.
The present invention is a system for building a dynamic social network from image data. The ever present use of video sensors for monitoring appears to have transformed the security operations in large facilities such as airports and critically susceptible infrastructures. Commercial solutions that allow the monitoring of simple activities seem to be currently deployed in large numbers of facilities, and operators have come to rely on these tools for their daily security operations. While these solutions provide basic features, they appear unable to detect activities of interest related to security threats. Dangerous perpetrators tend to be aware of the presence of the cameras and that they are being watched, and consequently they act accordingly not to raise suspicion. To stop or hinder these people, security operators should be trained to spot unusual patterns of activities or suspicious behaviors.
The present system includes an approach for constructing or synthesizing a dynamic social network from image data. Aspects of such approach may be described in U.S. patent application Ser. No. 12/124,293, filed May 21, 2008. U.S. patent application Ser. No. 12/124,293, filed May 21, 2008, is hereby incorporated by reference.
The present invention is a system which may incorporate a paradigm for discovering motivation and intent of individuals and groups in secured public environments such as airports. The system may incorporate as much of associating as well as detection. It may synthesize social networks from image data (including data accumulated over a period of time).
Synthesizing may include various techniques of video analytics. “Video data” may be one kind of image data. The system may determine a dynamic social network via processing. Specific tools may be used to analyze aspects of the social network and the data. A focus of the present system may be to develop open dynamic social network (DSN) analysis driven by video analytic primitives and standoff biometric capabilities when available. Analyses of dynamic social networks already synthesized may enable a security operator to discover patterns of activities, behaviors, and relationships between people that are not currently being detected automatically via processing. Even though in the present system, a social network may be constructed from image data; sound, such as detected conversations among the people in the images, may be added to the system to aid in social network construction.
The system may include mapping and construction of social networks from video data, recognition of activities that span long periods of time, pattern discovery and analysis of discovered social networks
The system may aid operators and analysts in understanding and relating to activities that occur in disparate scenes and at different time scales. Also, the present system may much better handle the ever increasing amount of sensors and data (e.g., thousands of cameras in airports, petabytes of intelligence data, hundreds of hours of, e.g., UAVs, and so on) than other systems.
Modeling relationships with the system may serve primarily goals which include visualizing relationships between actors, events, objects and locales for uncovering underlying structures difficult to grasp from disparate data, studying factors that influence the relationships such as providing context to the analysis, and inferring implications from the relational data such as predictive capabilities, anomaly detection (e.g., change in the data flow), and so forth.
Video analytics may provide detection and tracking capabilities. There may be rule-based recognition of activities of interest which is highly dependent on the detection and tracking, and face association using high resolution imagery. Video surveillance may focus on providing accurate object detection and tracking (e.g., faces, people, vehicles, and other items) for analytics and indexing purposes, anomaly detection, and event ontology and recognition of activities of interest.
A paradigm of the present system may include discovering relationships among large groups of monitored people, and the motivation and intent of individuals and groups. It may further include inferring a contextual knowledge of the observed activity, and aggregating a large number of observations for augmenting situation awareness by operators and analysts. Typical social network analytic tools appear not to address deriving such information from video data. However, the present system may leverage existing tools and platforms for advancing its syntheses. Scenarios and applications well suited for this paradigm may involve public environments such as mass transportation sites which include airports, seaports, train stations, and so forth. Items of interest relative to the present system may include terrorist groups, organized crime, intelligence communities of hostile countries, and the like. The system may note a dangerous individual acting on his or her own, but working with, for example, suppliers or other support organizations to accomplish certain objectives without the suppliers or organizations being aware of such objectives.
Video-based inference of social networks may provide system operators and analysts time-sensitive information about people and their entourages. The system may assist analysts to understand complex patterns of activities and relationships of them. There may be a detection of changes in the entourages of key people, such as local representatives, military leaders, and so on, which may be “flags” indicating the dynamics of the social networks among them.
The system may provide automatic construction and identification of social networks from video analytics processed data. Contextual information and mission specific a priori knowledge may be integrated. This information may be complemented or supplemented with various sorts of intelligence.
There may be a situational understanding beyond the basic video analytics, which may be aided with an expansion of forensic analysis. Also, pattern finding and analysis of discovered social networks may aid in situational understanding. Persistent surveillance may be enabled with accurate contextual knowledge of the monitored activity. The situational understanding may include discovering motivation and intent of individuals and groups. Predictive capabilities for proactive/pre-emptive actions may be developed from the discovered information. A data framework may be provided for integration of a time-sensitive bottom-up information flow.
The system may incorporate social network analyses. Social networks may help in modeling patterns and/or relationships among interacting units. Modeling these relationships may serve primarily the following aspects. One aspect may be visualizing relationships between actors, events, objects and locales for uncovering underlying structures difficult to grasp from disparate data. Another aspect may be studying the factors that influence the relationships and provide context to the syntheses. Also, an aspect may include inferring implications from the relational data such as predictive capabilities and anomaly detection (e.g., change in the data flow).
Real-time video analytics may be combined with social network analysis. Predictive capabilities may be developed by combining social network analysis and video analytics. Principal elements and interactions may be noted in video analytics in
A video-based inference of social network (VISNET) may be implemented in observations of people. For instance, agents may pre-screen people with SPOT (screening passengers by observation techniques) to detect specific behaviors. SPOT may persistently be used on persons of interest. The application of SPOT may be effected at numerous airports and other sites. VISNET may assist operators and analysts in understanding complex patterns of activities and relationships among the observed people. VISNET may incorporate a large number of sensors for observing actors such as people that are interrelated. Changes may be detected in an entourage of key people associated with groups, associations and so forth. There may be automatic construction and identification of small social networks from video analytics data. Some of the construction and identification may result from an integration of contextual information and mission specific information from a prior knowledge. Completion and update of annotations of a manual on social networks developed from such construction and identification may be provided by intelligence organizations.
A module 51 containing past events and detection in the form of images is shown, whereas module 49 deals with the recent and on-the-fly items. Module 51 may deal with spatial, temporal association with past events. Two screens 52 and 53 may include images of persons of interest at two or more different times (i.e., several months or so). A person of interest is noted in both images with a body rectangle image 54 and a face rectangle image 55 around the full figure and face, respectively, in screen 52, and with a body rectangle 56 and face rectangle 57, respectively, in screen 53. The noted information from module 51 may be associated to indicate that the persons in both screens are the same person at the different times. That information 58 may go on to a social network synthesis/analysis mechanism 59.
The social network synthesis/analysis mechanism 49 and the social network synthesis/analysis mechanism 59 may each have a two-way connection with a social network construction module 62. Module 62 shows an example social network synthesized or built from an event. Also, a deception detection and time sensitive information inference module 61 may have a two-way connection with the social network construction module 62. Module 61 may output deception detection and time sensitive information to a support operator and informed actions module 63 and to the social network synthesis/analysis mechanism 59.
The social network construction module 62 reveals various aspects of the social structure that may evolve as a result from the trigger event of abandoned luggage as indicated by screen 65. The abandoned luggage may be connected with the security checkpoint occurrence of which screen 66 shows a body rectangle of the person concerned along with a screen 67 which shows a face rectangle of the person. The screen or screens may provide a person of interest “best signature”. There appears to be a match with the person by the luggage in screen 65. There are body and face rectangles in screens 68 and 69, respectively, of people in temporal proximity of the person of interest. Temporal proximity may involve multi-camera tracking in screen 71 of the person of interest or one appearing to be associated with the person of interest. Screen 72 is an example of a level-1 contact of the person of interest with another person. The level-1 contact may also involve multiple contacts as indicated by screens 73 and 74 showing spatial proximity of the other contacts, i.e., persons.
There may be level-2 contacts as shown in body rectangle screens 76 and 77. Also, shown are corresponding face rectangle screens 78 and 79, respectively, of screens 76 and 77, that stem from a level-1 contact shown in screen 75. Screens 81, 82 and 83 are examples of spatial proximity to the trigger event of image 65. Screen 81 shows a person looking at the abandoned luggage. Screen 82 shows a person going by the luggage. Still another person appears suspiciously near the luggage. Some instances of spatial proximity may include loitering. Each of the screens in block 62 may be synthesized and analyzed, and vice versa with reiteration for improving the synthesis, to construct whatever social network that happens to exist.
The image 54 of the person in the trigger event 1 may lead to a dynamic social network generation as shown in block 96. Model 97 encompasses the probe model image 54 with the body rectangle and image 55 with the face rectangle showing the person of interest. Model 97 has a connection with various persons of one or another kind of connection to generate the dynamic social network. Persons, and perhaps their relationships within the model 97, may be listed in block 98. Model 97 indicates a connection with another trigger event shown in image 65 of a person next to the abandoned suitcase. There appears to be a detection of initial spatial proximity of model 97 with persons in images 81, 82 and 83. There also appears to be a detection of initial temporal proximity of model 97 with a person in image 68. Persons of image 73 may be regarded as a level-1 contact and persons of image 74 may be regarded as a level-2 contact of model 97. There may be other level contacts. Images 72 and 76 may be of persons having spatial-temporal proximity to model 97.
Event 2 may also lead as a person in image 102 to a block 108. Image 102 may be a body portion or rectangle from image 65 of the person standing by the luggage. Block 108 shows a generation of a dynamic social network resulting from forensic analysis and other tools. Image 65 reveals the luggage as being the trigger event with a target 109 situated on it. Face images 55 and 57 may be associated with the person next to the luggage recorded as participating in the abandonment of the luggage. The person of image 55 may be associated with the May Year D checkpoint breach and abandoned luggage. The person of image 57 may have been spotted in July Year D and through face association may be identified as the same person of image 57. Observation of the person in image 55 may reveal a dynamic social network of other persons shown diagrammatically with an oval 111. Subsequent observation of the person in image 57 may reveal another dynamic social network of other persons shown diagrammatically with an oval 112. The persons connected to ovals 111 and 112 may be listed in a block 113 to the right of block 108. Analysis and correlation of various persons and aspects of their interaction in the networks of ovals 111 and 112 may be made.
Oval 111 may link the person of image 55 with persons in images 68, 72, 73, 74, 76, 81, 82 and 83, which are discussed herein. Oval 112 may link the person of image 57 through a person in image 114. With this intermediary, the person of image 57 may be linked to persons in images 116, 117, 118 and 119. Further linkage to persons 121, 122, 123 and 124, and 125 may be made via the persons in images 116, 117, 118 and 119, respectively.
A complexity analysis of dynamic social network generation in
A checkpoint breach 130 may be the trigger event 1 which indicates the activity leading to a dynamic social network generation which may be subject to a complexity analysis. A person 133 may participate in the breach 130. After the breach, person 133 may meet persons 131, 132, 134, 135 and 136. Some of these persons may or may not have had any previous relationship with person 133. One or more of these persons may begin a relationship with person 133 or vice versa. One or more, or none of the persons 131, 132, 134, 135 and 136 may be the same person as person 133 who appeared to be participating in breach 130.
Person 134 may participate by abandoning an object which is indicated by a node 141 and regarded as an abandoned object event node 141. The event node 141 may be linked to an object node 142 which is indicated to be a “bag 1”, such as luggage. Object node 142 may be linked to an event node 143 which is regarded as a “pick up object” node. A person 137 may participate by being the one who picked up the object, bag 1 or luggage.
The complexity of the network 140 may be equal to or less than O(Mp+Mp*Me+Mp*Me*Mo) where Mp is a coefficient of a number of persons, Me a number of events and Mo a number of objects. Since there are seven persons, three events and one object in network 140, then Mp=7, Me=3 and Mo=1. The complexity may be tabulated as O(7+7*3+7*3*1), or O(49).
The present system may have an automatic social network synthesis which may include a fusion of biometric identifications (IDs), video analytics, contextual information, data mining, and other items. The synthesis may include a predictive capability plus expanding the SPOT observational phase. Analytics methodologies, having tools for building dynamic social networks from non-co-located video observations, may be incorporated in the system. The methodologies may include uncertainty modeling, belief propagation and robust association of observations of people, objects, and so forth. Social network analysis may use incomplete observations, including dominant figure detection, importance cardinality, and group membership. Analyses may include a creation of strong and weak associations over short and long observations with temporal resolution. The present system may change detection, identify new people, note interactions and exchanges of people, and so on. Metrics may be used for quantification of performance and accuracy of the analyses and their results.
The present system may further include inferring and mapping of social networks from video data, video analytics, biometrics (e.g., iris standoff recognition and face association) from low resolution imagery, standoff irises, appearance modeling and activity recognition. The system may provide robust association across sensors and over long observations, and context-driven mapping and analysis. Some of the system components for improving expediency may include low resolution video, fast data indexing for forensic analyses and model social networks as an open dynamic arrangement. The system may be complemented with HSARPA programs (association of objects across non-overlapping cameras). There may be pre-emptive capabilities from the integration of social network knowledge with video analytics and vice versa.
Video analytics of the system may aid in people detection in crowded environments, processing information from a large network of cameras, activity recognition, association of objects across a network of uncalibrated sensors, and appearance modeling across uncalibrated non-overlapping cameras.
Association of objects by the system may aid in people association which includes robust appearance models and iris recognition at a distance. There may be face detection and association which includes feature-based descriptors for people and fingerprinting. There may be object detection and association which use uncalibrated sensors and object fingerprinting.
The system may map dynamic social networks (DSN). The open dynamic social network model may include an unknown a priori population of interest, and integrate objects, locales and events. It may augment current DSN models with observations inferred from video analytics, create open DSN from short term observations, and associate/aggregate social sub-networks from long-term observations.
There may be social network syntheses which may include derivation of social networks from incomplete observations based on incomplete video analytics, multi-valued edges and belief propagation in multi-modal graphs. There may be noted spatial and temporal frequencies of association among individuals, objects and locales. The syntheses may also include matching networks for predictive reasoning.
The DNS of the system may implement recognition of activities that span long periods of time and geographic areas. DSN of monitored people may be used to provide context to an analysis of observed activities. DSN may encode temporal frequencies for inferring the intent of activities carried on by a person (given past observations). Recognition of activities that span large geographical areas may be noted. DSN may be used for aggregating events that occurred at different locations, and for extracting social groups based on similar activities/events from disparate locations.
The system may provide for data indexing and event forensics. Some of the items may include association from incomplete observations, online indexing of visual data, indexing social networks, fast querying of observations, driving forensic analyses using the DSN topology and characteristics, and automatic populating DSN with people of interest.
Network analyses may cover for various gaps of information. There may be analyses of gaps in observations which include analyses from incomplete/uncertain observations. There may be multi-valued edges (e.g., similarities, locales, relationships, events, and so on). The analyses may consider belief propagation or probabilistic analysis in multi-modal networks and incomplete video analytics. Incomplete video analytics may rely on inferences made from incomplete measurements, identifying the network of interest from parts of it that were observed, defining indicators that can characterize the shape and nature of networks, and knowing what is not known such as identifying gaps in observations versus observed data. Network syntheses and analyses may be augmented with machine vision and learning and machine vision approaches. The approaches may include learning network structures for fast detection and recognition, learning methodology for analytics of large networks, and making inferences from partial observations. Metrics may be defined for assessing network analysis tools.
The system may deal with and overcome gaps in network mapping/data collection. It may shift from manual data collection to automatic association and mapping. There may be mapping networks/data collection from video analytics, which include robust extraction of objects, events, locales and context, and associations and relationships from spatio-temporal properties with large temporal and spatial scales. One example may include mapping a network corresponding to an unmanned aerial vehicle (UAV) monitoring and ground-based observations such as of objects, locales and events association. Another example may include mapping a network over a large geographic area and temporal scale, including event recognition over large period of time (e.g., days, weeks, months, years).
Dealing with the gaps may also include creating an open DSN from short term observations and associating and/or aggregating social sub-networks from long-term observations (e.g., gaps in observations). There may further be an open dynamic social network model having a hierarchical representation. There may be an unknown a priori population of interest which may be overcome with analyses of large scale networks. Integration of objects, locales, context and events may help remove gaps in network mapping and data collection. Metrics may be defined for assessing associated network mapping tools.
The system may accommodate gaps relative to its forensic tools. It may visualize relationships between actors, events and locales for uncovering underlying structures difficult to grasp from disparate data, which could include an integration of network visualization with video analytics and video as evidence to support facts, such as augmenting social network analysis tools adopted by the intelligence community.
Relative to online forensics, the system may shift from forensic analysis to pre-emptive/predictive analysis, which may include real-time backtracking of people of interest. Online forensics may augment network visualization with querying capabilities, and search for similar networks or spatio-temporal relationships.
The system may resort to cross-domain integration relative to forensics, which may include open source, broadcast news, and other video resources. Also, there may be multi-sensor intelligence resources where the data are different but the content is similar or relevant. There may be a data framework provided for collaboration across communities.
The system for constructing or synthesizing social networks from image data may be described in the context of a layered architecture. In
Video analytics information may show a cameras setting file for camera 1, camera 2, and so on. The setting file may provide frame rate, path to video files, names of video files, and other pertinent information that may be useful. Video analytics information for video description may provide time stamp dates, object ID for body and face rectangles, dimensions of the rectangles, and a number of time stamps which correspond to the number of frames.
Second, a middle layer 212 may include an actor-event multi-dimensional matrix. It may be a multi-mode sociomatrix, where rows index actors and columns index events. For instance, if there are n actors and m events, then the matrix may be an “n×m” matrix. An (i,j) cell of the matrix may be 1 if row actor i is involved with event j. Otherwise, the entry may be 0 if row actor i is not involved with event j. The row margin totals may indicate the number of events with which each actor is involved.
The matrix may a multi-dimensional matrix containing people, object and event information from a video analytics processor. The may be weighted relationship indices determined from information from the video analytics processor and put into the matrix for forming a basis of the social network.
Third, a high layer 213 may include a social network of actors and relationships. Actors and nodes may be regarded as equivalent terms. Relationships, edges and links may be regarded as equivalent terms. The term “some” may mean at least one. A relationship may result from, be indicated by, inferred from, or described by an event among the actors. For each relationship, link or edge, a weight may be assigned. The greater the weight, the tighter is the relationship. The weight may have correspondence to an importance factor of a respective event.
The low, middle and high layers may be regarded as first, second and third layers, respectively, or vice versa. The three, more or less, layers may be labeled with other terms as may appear fitting.
Events may often involve two or more actors. Relative to the architecture of the present system, a relationship of the two or more actors indicated at the high layer may be inferred from the actor-event matrix of the middle layer. The events may build a linkage among the actors. The events may be co-location, co-temporal, and/or other.
In proceeding from the middle layer 212 to the high layer 213, an importance factor may be determined for each event. A weighted frequency may be calculated for the relationship between two actors in the high layer. A basic frequency may be proportional to the number of times that two actors have a one (1) in the same columns of a table or matrix. The weighted frequency may be the basic frequency multiplied by an importance factor or weight of a relevant event. Attendance at some of the events may have a magnitude of importance which may be referred to as a “weight”.
In other words, the basic frequency may be a number of times that actors have been present at one or more of the same events. The weighted frequency of the relationship between the actors may be a product of the basic frequency and the weight assigned to the respective same event. The total of the weights assigned to all of the events of an actor-event matrix should be about one (1).
In sum, a dynamic social network may eventually be built from raw images or video with an approach that proceeds through the low layer 211, the middle layer 212 and the high layer 213, respectively, in the present system 210. With a sliding window in time, to reach back for information that has passed, the social network may be dynamic. Also, in that information is incomplete or updatable at a certain point in time, with the present system 210 in an on-line situation, data may continuously flow to the low layer for processing which may complete and/or update information already in the system, or bring in new information thus also resulting in the social network being dynamic.
Video or surveillance data may be extracted from the raw images or video at the low layer 211. The low layer may also handle image or video data for purposes of extracting and determining actors and events. A network of cameras may be used for collecting the data. For the same camera, or several cameras with overlapping fields of view, one may perform motion tracking of an association of an actor or actors. For different cameras, particularly with non-overlapping fields of view, there may be an identification (ID) association of an actor or actors between multiple cameras. The ID association may include face association, actor association, and/or biometrics association, such as standoff iris recognition. An association could instead be identification. Algorithms may be used for detecting and matching actors, faces, other biometrics, and more, especially with respect to identifying actors from one camera to another camera, actors at events, and actors associating and/or meeting with each or one another. Also, algorithms may be used to identify and/or associate events. The algorithms may be part of the video analytics at the low layer 211.
There may be detection, tracking, recognition, association and other sub-modules that provide information which can be put in a repository for data. Operations by the video analytics on the information from the sub-modules may be effected with appropriate algorithms.
The events under consideration may be co-spatial events and/or co-temporal events. For example, a co-spatial event may involve an object, such as luggage, abandoned by one actor and picked up by another actor. The luggage and persons may be regarded as actors, i.e., object and persons. The abandonment and picking up of the luggage may be regarded as one or two events to be analyzed. The event may be attended by both actors but not necessarily at the same time and thus be regarded as co-spatial. If both actors are aware of each other's actions, they may be considered as attending one and the same event. If that is not the case, for example, the first actor leaves or abandons luggage in an airport, intentionally or unintentionally, and the second actor, such as security guard, picks up the luggage for safety reasons and has little or no knowledge about the first actor, and that the picking up is not a planned or coordinated act relative to the first actor, then both actions regarded as two events. The luggage itself may be regarded as an actor. If both actors were playing a role relative the abandoning and pick-up of the luggage, then these actions may be considered as attending one event. This event appears to be of interest, especially in an airport setting, and may have a particular significant importance. In another setting, the event may be considered as having insignificant importance.
The video analytics of the low layer analysis may extract the events, determine who the actors are, and check features and match features of the actors. Numerous actors may be noted. There may be a meeting, i.e., an event, between two actors indicating a spatial-temporal co-location, that is, two actors being simultaneously at the same location for the event. However, in some situations, an event may be just co-spatial or co-temporal. The actors and events may be extracted from the video data at the low layer. Such data may be reduced to an actor-event matrix at the middle layer 212. Attending an event, such as a meeting, may be regarded as a logic function “AND” in the grid, table or matrix. For example, two actors relative to an event may be indicated by a one or zero unit in the respective box of the matrix 251 as may be noted in
There may be a spatial-temporal frequency. For instance, actors meeting five times within one week may establish a link. The link may be regarded as weak, moderate or strong. One meeting time between actors may be regarded and a weak or no link. The link may be from an event. Each event may have a weight. The weights of events may vary. An important event may have a significant weight. The abandoning of a piece of luggage at an airport may be an important event having a significant weight. If an event has no importance, it may have a weight of zero, and there may be no result of connection, relationship, edge or link between or among the actors of concern.
Matrix 280 may be an extended one, as shown in
The events may be extracted from raw data, where w1 may be a weight value assigned for one event, and other weights assigned for other events.
A maximum value of W may be 1. There may be one weight per event, and the weights of all of the events may add up to one. Link weights may be calculated for actors relative to the events as illustrated herein.
For implementation of the multi-layer architecture of the present system, a sliding window may be part of a dynamic social network at the high layer 213 of the present system 210 as it adds a dynamic feature to the network. A sliding window of frames or minutes may be regarded as a temporal window. There may be, for example, a twenty minute sliding window. Minutes may be regarded as more consistent measurement than frames. For instance, there may be a fast frame rate and a slow frame rate. There may be a same amount of time for different numbers of frames. The window may be slid back to examine earlier moments of an actor or event which after a period of observation became of interest.
In the present specification, some of the matter may be of a hypothetical or prophetic nature although stated in another manner or tense.
Although the invention has been described with respect to at least one illustrative example, many variations and modifications will become apparent to those skilled in the art upon reading the present specification. It is therefore the intention that the appended claims be interpreted as broadly as possible in view of the prior art to include all such variations and modifications.