Patent Application
20180293311
Object recognition may be performed on video and image content. For example, a surveillance video may be analyzed to determine the identification of a person appearing in the video. Image analysis techniques may be used to create a template indicating features associated with the person, such as facial features.
The drawings describe example embodiments. The following detailed description references the drawings, wherein:
In one implementation, object identification is performed by a database comparison operation. For example, a user defined database function may include executable instructions to perform object identification related to stored image and/or video data. The database comparison function may be executed by a processor associated with a database, such as a database engine and/or database analytics processor associated with the database engine. The processor may execute user defined functions to store object template information and then compare the stored object template information using database comparison capabilities. For example, a processor may execute a database operation to perform person recognition based on a database comparison of a first set of stored object templates to a stored target object template. The processor may output information related to the person recognition, such as a set of images displaying the top N closest results.
Performing object recognition analytics as a database operation allows for the data to be queried without first loading the data from the database to be analyzed by a software application. For example, the method may be faster due to the use of database comparison functionality. Both images and videos may be treated the same, such as where the same face template is associated with a video frame and an image because the database may store the relevant information such that the video itself is not reloaded at the time of comparison. Additionally, a database comparison operation may take advantage of parallel processing performed by the database, such as where the database automatically performs comparison functions related to different portions of stored data in parallel.
The processor 101 may be a database engine or an analytics engine associated with a database engine. The processor 101 may be a central processing unit (CPU), a semiconductor-based microprocessor, or any other device suitable for retrieval and execution of instructions. As an alternative or in addition to fetching, decoding, and executing instructions, the processor 101 may include one or more integrated circuits (ICs) or other electronic circuits that comprise a plurality of electronic components for performing the functionality described below. The functionality described below may be performed by multiple processors.
The database 109 may be any suitable data storage system for storing data, such as a relational database. The database 109 includes object template information 110. The object template information 110 may be, for example, templates for person identification, such as face templates. The object template information may include information associated with the source of the object template and the location of the object template within the input content. For example, the object template information 110 may include a file name, tracker number, video frame number, object bounding box coordinates, face bounding box coordinates, and object feature template. The object template information 110 may include information related to features associated with an object determined to be of a particular type, such as a person or face. The object template information 110 may include feature information related to a bounding box in an image or video where the bounding box surrounds an object of the particular type. The object template information 110 may include object template information of multiple types of objects. For example, a database system for person identification may store face object templates and clothing object templates.
The processor 101 may communicate with the machine-readable storage medium 102. The machine-readable storage medium 102 may be any suitable machine readable medium, such as an electronic, magnetic, optical, or other physical storage device that stores executable instructions or other data (e.g., a hard disk drive, random access memory, flash memory, etc.). The machine-readable storage medium 102 may be, for example, a computer readable non-transitory medium. The machine-readable storage medium 102 may include instructions, such as user defined database functions, that may be executed by the processor 101. For example, the machine-readable storage medium 102 may include object template creation instructions 103, object template storage instructions 104, target object template creation instructions 105, target object template storage instructions 106, object identification instructions 107, and output instructions 108.
In one implementation, a second processor communicates with the processor 101 via a network. For example, the second processor may query the processor 101 via a network related to the target object, and the second processor may output information related to received information from the processor 101 related to the object recognition information. For example, a software application may perform an operation to call a database function such that the processor 101 associated with a database system executes the instructions in the machine-readable storage medium 102 as a user defined database function. The software application may call the operation in an encapsulated manner without details as to how the comparison is performed by the database.
The object template creation instructions 103 may include instructions to create a set of object template information based on at least one of image and video information. For example, the image and video information may be received from any suitable source. In one implementation, the input information is received via a network from multiple sources, such as from multiple surveillance cameras and/or from surveillance information from multiple entities. The object template creation may be performed in any suitable manner. For example, the object template may be created based on local binary patterns within an area of the image or video.
The object template storage instructions 104 include instructions to store the created object template information in the storage 109 as the object template information 110. For example, the processor 101 may perform a search operation or other operation with respect to the stored object template. Additional information may be stored as other fields related to the object template, such as information about the source of the content including the object template. In one implementation, the source image and/or video frame is also stored in the storage 109.
The target object template creation instructions 105 include instructions to create an object template associated with a target object. For example, the processor 101 may receive a set of input content, such as via a network. The input content may be a video and/or image. The processor 101 or a different processor may analyze the input content to determine an object of a particular type within the input content and to create an object template representative of the object. For example, an object recognition application may be executed related to the input content to identify areas likely associated with the target object type, such as a face. The object template may be any suitable object template, such as a feature vector.
The target object template storage instructions 106 include instructions to store the target object template in the storage 109. Information about the source content and the object may be stored in the storage 109. In one implementation, additional information related to the object may be associated with the object template information in the storage 109, such as geographic information associated with the source content. For example, location information or a person identifier may be associated with the object template, and the associated information may be linked such that the object template may be searched based on the additional information and/or the additional information may be output in response to a query related to the object template. As an example, a user may query object templates related to a particular input object template and located in a certain place at a certain time in a surveillance video.
The object identification instructions 107 may include instructions to identify an object associated with the target object based on a database comparison operation of the stored set of object template information to the stored target object template information. For example, the database compare operation may compare object template information stored in the storage 109. The processor 101 may perform the comparison in any suitable manner. The processor 101 may take advantage of parallel capabilities of the database system 100 such that different sets of object template information 110 may be simultaneously compared to the target object template also stored in the storage 109.
In one implementation, the process of comparing and creating the object template occurs together, such as where the target object is an object extracted and stored in the storage 109. For example, the processor 101 may store a new object template and then compare the new object template to stored object template. Information about the comparison results may be stored in the storage 109, such as where similar object templates are linked together.
The output instructions 108 may include instructions to output information related to the object identification. For example, the information may be transmitted, displayed, and/or output to the storage 109 or another storage. In one implementation, images and/or videos may be displayed based on similarity to a target object template, such as where information associated with the most similar object template, top N most similar object templates, or top X % of similar object templates is displayed. A user may view the source images and/or video related to the object templates and provide feedback as to whether the objects are likely to be the same. For example, a user may compare a target face to faces in the output top N images. In one implementation, the results of the database comparison operation are used by a software operation to provide data to a user via a user interface.
Beginning at 200, a processor enrolls a set of object templates in a storage based on objects within input content. The input content may be any suitable input content, such as image or video content. Image and video content may be treated in a similar manner such that a stored object template associated with video content may be compared to an object template associated with image content. The input content may include video content such that an object template is associated with a set of adjacent video frames including the same object. For example, a single object template may be created for the object appearing in the adjacent frames.
The storage may be any suitable storage, such as a relational or XML database. The storage may be a database system including stored data and information related to stored user defined functions, such as functions that may be executed by the processor.
The processor may enroll the object templates in any suitable manner. Enrolling a set of object templates may include storing object templates received from another processor. In one implementation, the processor both analyzes input content to identify portions of the content to create an object template associated with it. The processor may ingest the input video and/or image data. The processor may then detect an object type within the ingested content, detect object landmark points, align the objects, and extract object features. The processor may create an object template based on the extracted object features. The object templates may be enrolled into the database system. The object templates may include any suitable information related to objects within the input content. For example, the object templates may include information about local binary patterns related to an area of the image or video. The information about the local binary patterns may be compared and classified.
Continuing to 201, a processor enrolls a target object template in the storage based on a target object in target content. The target content may be, for example, a video or image. Similarly to the input content, the object template may be created based on adjacent video frames including the same target object. The enrollment process may include creating the object template associated with the target object, and storing the object template in the storage such that it may later be compared. In one implementation, the processor ingests target content, identifies objects of a particular type within the content, creates object templates associated with the identified objects, and stores the created object templates. In one implementation, a software application performs image/video analysis and connects to the database to execute a user defined function related to creating an object template associated with the image/video analysis.
The target object may be a specific object in a user query and/or a set of objects within input content including multiple bounding boxes surrounding objects likely to be of a certain type is a target object. The target object may be automatically determined based on other triggering information, such as the actions of a user in a video.
The enrollment process may include linking the object template to additional information associated with the object. For example, time or location information may be associated with the input content and also associated with the object template. In one implementation, the enrollment process of the input object and/or target object includes associating the object template with data from additional sources related to the object template and/or input content. For example, the object template may be associated with an address associated with the surveillance information. In cases where an identity of an object template is known, name and other identifying information may be associated with the object template in the database.
Continuing to 202, a processor identifies an object within the input content associated with the target object based on a comparison of the stored set of object templates to the stored target object template. For example, a database comparison operation may be performed to compare information related to two object templates.
In one implementation, the processor both stores and compares an object template as a single step. For example, after enrolling an object template in the database, the processor may compare the object template to stored object templates. Information about links between the object templates may be stored. When the processor receives a query related to a target template, information about the linking object templates may be output.
The target object may be a user identified object related to a user query, or an object within selected content, such as where the processor iterates through each object identified within the selected content. In one implementation, the target object is automatically identified, such as where a processor ingests content from multiple surveillance videos and each identified object becomes a target object for search purposes after being stored. For example, an object may be enrolled and then compared, and then the next object may be enrolled and then compared.
In one implementation, the object identification is performed by multiple user defined database functions executing in parallel and/or a single user defined function taking advantage of parallel search capabilities of the database. For example, the target object template may be compared to a subset of stored data in parallel.
In one implementation, object identification includes both comparison to object templates and additional stored information. For example, a user may query for object templates related to an input object template and where the object template is associated with a particular location. A location field in the database may be searched and compared to the location criteria, and object templates fitting the location criteria may be compared to the target object template.
Continuing to 203, a processor outputs object recognition information related to the identified object. For example, the processor may output a top match, a top N ranked matches, a top % of matches to a query, and/or object templates with a similarity score above a threshold. The information may be output by transmitting, storing, or displaying the information. In one implementation, the output information is associated with a surveillance application for person identification. For example, a surveillance video including an incident of interest may be uploaded, and the output may include information related to object templates found in the uploaded video. In one implementation, the object template information is used for identification, and the output information includes information related to the object template, such as a name or address associated with the face template.
The information may be output to a software application calling the database comparison operation such that the software application processes and outputs information related to the received results of the database comparison operation.
Beginning at 500, a processor creates an object template for the next object. For example, the processor may move to the next bounding box identified as a potential object in an image or video or move to the next image or video frame. Continuing to 501, the processor stores the object template. The processor may store the object template in a database. Continuing to 502, the processor compares the newly stored object template to a set of previously stored object templates. For example, a database user defined function may compare the newly stored object template to previously stored object templates. Information about the comparison results may be output to a user and/or added to the storage to be associated with the newly stored object template. The processor may continue to the next object template.
Performing object identification using a database comparison operation may take advantage of the database comparison capabilities and allow for simpler object identification software application to rely on database operations.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US15/31463 | 5/19/2015 | WO | 00 |
