Patent Application
20150186374
The present application is based upon and claims priority to Chinese Patent Application No. 201310734306.5, filed Dec. 27, 2013, the entire contents of which are incorporated herein by reference.
The present disclosure relates to the crossing fields of radiation imaging, image understanding, machine learning, auxiliary examination and inspection, and the like, and in particular, to a retrieving system based on contents of fluoroscopic images picked up via radiation during security inspection.
Radiation fluoroscopic imaging technology is a major means for noninvasive inspection on articles, luggage, or the like, and has been currently widely used at air ports, customs, bus stations, large gatherings, and the like sites. The physical mechanism of radiation imaging causes the image feature presented to be notably different from a natural image. The difference directly causes the security inspection personnel not to perceptually understand image content. What is worse, the security inspection personnel's sensitivity on hazardous articles is extremely reduced. Therefore, the technical solution of globally or partially retrieving inspection fluoroscopic images to find similar images from a database, and giving annotation information such as similarity, target attribute (atomic number range, and weight and thickness ranges), and hazardous extent, is urgently desired. At present, in the field of security inspection, retrieval of the fluoroscopic images is mainly based on file name and time, without introducing any content information of the fluoroscopic images. With popularity of the radiation security inspection devices, a large amount of history image data is generated. If no convenient retrieval and comparison means is available, the large amount of history image data exerts no positive effects.
Currently, in the field of retrieval based on image contents, natural images and medical fluoroscopic images are mainly concentrated. Content retrieval of the natural images has gained rapid development in recent years, and a plurality of practical products have been developed, for example, Google image retrieving system, Baidu image retrieving system, and the like. Content retrieval of the medical fluoroscopic images mainly concerns accurate and precise search from the database for other patients having the same examination area, such that the doctor conveniently determines differences of pathological changes in the same examination area of different patients. However, such systems are still in the trial stage currently. Since the medical image data content is single, the data cannot be directly applied to the more widely used radiation fluoroscopic images.
In recent years, with rapid development of machine learning, mode recognition, and the like fields, such auxiliary means of quickly searching for fluoroscopic images having similar contents and providing these fluoroscopic images to the security inspection personnel for comparison and analysis has become a focus in the industry. However, due to complexity of the article categories and particularity of the fluoroscopic images, the field never sees fast development and promotion, and no relevant publication has been found.
In view of practical circumstances of fluoroscopic images picked up during security inspection—a variety categories of articles to be scanned, superimposition of article structural information, high natural expression information loss and low image resolution, and actual needs of helping security inspection personnel to better understand and inspect fluoroscopic images, the present disclosure provides a system capable of retrieving fluoroscopic images based on contents of the fluoroscopic images and outputting a diversified (similar contents but different article categories) retrieval result and relevant annotation information.
The retrieving system mainly implements the following functions: (1) fuzzy retrieval of image contents, providing a diversified (similar contents but different article categories) retrieval result; (2) parallel computation, achieving real-time return of the retrieval result; (3) providing annotation information of the retrieval result, such as similarity, target attribute, commonality, risk extent, and the like; (4) automatically regulating the retrieval model according to user's feedback information. The retrieving system may be mainly used to train security inspection personnel for better understanding of X-ray images, help the security inspection personnel to analyze image contents, and improve the inspection efficiency.
According to an embodiment of the present disclosure, a retrieving system based on contents of fluoroscopic images picked up during security inspection is provided. The retrieving system includes:
a pre-classifying module, configured to pre-classify fluoroscopic images, and classify the fluoroscopic images into texture images and non-texture images;
an image content feature extracting module, configured to perform feature extraction for contents of the fluoroscopic images;
an image representing module, configured to combine feature description vectors of the contents of the fluoroscopic images, and construct an image representation vector;
a retrieving module, configured to retrieve, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructing a result of preliminary candidates;
a diversified filtering module, configured to filter the result of preliminary candidates, select an image subset capable of covering a plurality of article categories, and thereby construct a diversified retrieval result;
a correlation feedback regulating module, configured to receive information feedback on the retrieval result from a user, and update the retrieval model; and
an interacting module, configured to display the retrieval result, and collect feedback on user's satisfaction of the retrieval result.
The retrieving system implements fuzzy retrieval of image contents, and provides a diversified (similar contents but different article categories) retrieval result; by means of parallel computation, achieves real-time return of the retrieval result; and provides annotation information of the retrieval result, such as similarity, target attribute, commonality, risk extent, and the like. According to user's feedback information, the retrieving system is also capable of automatically regulating the retrieval model.
Optionally, the image content feature extracting module is configured to: with respect to a texture image, establish an image pyramid, perform super pixel classification layer by layer, and establish a texture description vector using a super pixel as a unit; and with respect to a non-texture image, establish an image pyramid, perform feature point detection pixel by pixel and layer by layer, and establish a feature description vector for a detected feature point.
Optionally, the image representing module is configured to: with respect to a texture image, by using a codebook policy, collect statistics for all the texture description vectors established using a super pixel as a unit, establish a histogram, and use the histogram as a representation vector of the image; and with respect to a non-texture image, aggregate all feature description vectors established on feature points, thereby construct a feature vector set, and use the feature vector set as a representation vector of the image.
Optionally, the retrieving module is configured to: with respect to a texture image, represent the image using a feature vector; and with respect to a non-texture image, represent the image using a feature vector set.
Optionally, the interacting module employs a two-stage tree-like display solution: selecting a retrieval region on a scanning image via a mouse; upon selecting the retrieval region, displaying in real time two-stage tree-like retrieval results around the retrieval region, a first stage of displaying an output result from the diversified filtering module, and a second stage of displaying other similar images that are in the same article category as a previous stage; when the mouse slides over a node at either of the two stages, displaying a selection label for user's satisfaction on the retrieval results; and when feedback information is acquired, sending the feedback information to the correlation feedback regulating module.
According to another embodiment of the present disclosure, a retrieving method based on contents of fluoroscopic images is further provided. The retrieving method includes:
a pre-classification step: pre-classifying fluoroscopic images, and classifying the fluoroscopic images into texture images and non-texture images;
an image content feature extraction step: performing feature extraction for contents of the fluoroscopic images;
an image representation step: combining feature description vectors of the contents of the fluoroscopic images, and constructing an image representation vector;
a retrieval step: retrieving, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructing a result of preliminary candidates;
a diversified filtering step: filtering the result of preliminary candidates, selecting an image subset capable of covering a plurality of article categories, and thereby constructing a diversified retrieval result;
a correlation feedback regulation step: receiving information feedback on the retrieval result from a user, and updating the retrieval model; and
an interaction step: displaying the retrieval result, and collecting feedback on user's satisfaction of the retrieval result.
According to further another embodiment of the present disclosure, a retrieving system based on contents of fluoroscopic images is further provided. The retrieving system includes: one or more processors; a memory; and one or more modules stored in the memory and being configured to be executed by the one or more processors, the one or more modules having the following functions:
pre-classifying fluoroscopic images, and classifying the fluoroscopic images into texture images and non-texture images;
performing feature extraction for contents of the fluoroscopic images;
combining feature description vectors of the contents of the fluoroscopic images, and constructing an image representation vector;
retrieving, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructing a result of preliminary candidates;
filtering the result of preliminary candidates, selecting an image subset capable of covering a plurality of article categories, and thereby constructing a diversified retrieval result;
receiving information feedback on the retrieval result from a user, and updating the retrieval model; and
displaying the retrieval result, and collecting feedback on user's satisfaction of the retrieval result.
According to still another embodiment of the present disclosure, a security inspection device is further provided. The security inspection device includes a retrieving system based on contents of fluoroscopic images as recited above.
According to still another embodiment of the present disclosure, a non-transitory computer-readable storage medium is further provided. The non-transitory computer-readable storage medium have stored therein instructions that, when executed by one or more processors of a device, cause the device to perform:
pre-classifying fluoroscopic images, and classifying the fluoroscopic images into texture images and non-texture images;
performing feature extraction for contents of the fluoroscopic images;
combining feature description vectors of the contents of the fluoroscopic images, and constructing an image representation vector;
retrieving, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructing a result of preliminary candidates;
filtering the result of preliminary candidates, selecting an image subset capable of covering a plurality of article categories, and thereby constructing a diversified retrieval result;
receiving information feedback on the retrieval result from a user, and updating the retrieval model; and
displaying the retrieval result, and collecting feedback on user's satisfaction of the retrieval result
With reference to the embodiments described hereinafter, various details and aspects of the present disclosure would be clearly illustrated.
In the drawings:
To make the objectives, structures, and advantages of the present disclosure clearer, the present disclosure is further described in detail with reference to the attached drawings. For brevity of description, only one of multiple possible configurations is illustrated in the drawings and the specification. A person skilled in the art would understand that without departing from the spirit of the present disclosure, various modifications and replacements may be made to the following embodiments of the present disclosure.
Pre-Classifying Module
The pre-classifying module 101 pre-classifies fluoroscopic images, and classifies the fluoroscopic images into texture images and non-texture images. In the subsequent processing, the retrieving system separately processes the texture images and the non-texture images.
Image Content Feature Extracting Module
The image content feature extracting module 102 performs feature extraction for contents of the fluoroscopic images. Based on a pre-classification result, the texture images and the non-texture images are separately processed.
With respect to a texture image, the image content feature extracting module 102 establishes an image pyramid, performs super pixel classification layer by layer, and establishes a texture description vector using a super pixel as a unit.
With respect to a non-texture image, the image content feature extracting module 102 establishes an image pyramid, performs feature point detection pixel by pixel and layer by layer, and establishes a feature description vector for a detected feature point.
Image Representing Module
The image representing module 103 combines feature description vectors of the contents of the fluoroscopic images, and construct an image representation vector. The texture images and the non-texture images are separately processed.
With respect to a texture image, the image representing module 103, by using a codebook policy, collects statistics for all the texture description vectors established using a super pixel as a unit, establishes a histogram, and uses the histogram as a representation vector of the image.
With respect to a non-texture image, the image representing module 103 aggregates all feature description vectors established on feature points, thereby constructs a feature vector set, and uses the feature vector set as a representation vector of the image.
Retrieving Module
The retrieving module 104 retrieves, from a fluoroscopic image representation database based on a retrieval model, a plurality of images having a higher similarity to the image representation vectors under retrieval, and thereby constructs a result of preliminary candidates. By using the retrieval model, the result of preliminary candidates may be quickly obtained.
Diversified Filtering Module
The diversified filtering module 105 filters the result of preliminary candidates, selects an image subset capable of covering a plurality of article categories, and thereby constructs a diversified retrieval result.
Correlation Feedback Regulating Module
The correlation feedback regulating module 106 receives information feedback on the retrieval result from a user, and updates the retrieval model.
Interacting Module
The interacting module 107 displays the retrieval result, and collects feedback on user's satisfaction of the retrieval result.
According to another embodiment of the present disclosure, a retrieving method based on contents of fluoroscopic images picked up during security inspection is further provided. The retrieving method includes: a pre-classification step, an image content feature extraction step, an image representation step, a retrieval step, a diversified filtering step, a correlation feedback regulation step, and an interaction step.
Pre-Classification Step
In the pre-classification step, fluoroscopic images are pre-classified, and the fluoroscopic images are classified into texture images and non-texture images. In the subsequent processing, the retrieving system separately processes the texture images and the non-texture images.
Image Content Feature Extraction Step
In the image content feature extraction step: feature extraction is performed for contents of the fluoroscopic images. Based on a pre-classification result, the texture images and the non-texture images are separately processed.
With respect to a texture image, an image pyramid is established, super pixel classification is performed layer by layer, and a texture description vector is established using a super pixel as a unit.
With respect to a non-texture image, an image pyramid is established, feature point detection is performed pixel by pixel and layer by layer, and a feature description vector is established for a detected feature point.
Image Representation Step
In the image representation step, feature description vectors of the contents of the fluoroscopic images are combined, and an image representation vector is constructed. The texture images and the non-texture images are separately processed.
With respect to a texture image, by using a codebook policy, statistics for all the texture description vectors established using a super pixel as a unit are collected, a histogram is established, and the histogram is used as a representation vector of the image.
With respect to a non-texture image, all feature description vectors established on feature points are aggregated, thereby a feature vector set is constructed, and the feature vector set is used as a representation vector of the image.
Retrieval Step
In the retrieval step, a plurality of images having a higher similarity to the image representation vectors under retrieval are retrieved from a fluoroscopic image representation database based on a retrieval model, and thereby a result of preliminary candidates is constructed. By using the retrieval model, the result of preliminary candidates may be quickly obtained.
Diversified Filtering Step
In the diversified filtering step, the result of preliminary candidates is filtered, an image subset capable of covering a plurality of article categories is selected, and thereby a diversified retrieval result is constructed.
Correlation Feedback Regulation Step
In the correlation feedback regulation step, information feedback on the retrieval result is received from a user, and the retrieval model is updated.
Interaction Step
In the interaction step, the retrieval result is displayed, and feedback on user's satisfaction of the retrieval result is collected.
The retrieving system and method based on contents of fluoroscopic images picked up via radiation during security inspection according to the present disclosure implement fuzzy retrieval of image contents, and provide a diversified (similar contents but different article categories) retrieval result; by means of parallel computation, achieve real-time return of the retrieval result; and provide annotation information of the retrieval result, such as similarity, target attribute, commonality, risk extent, and the like. According to user's feedback information, the retrieving system is also capable of automatically regulating the retrieval model. The retrieving system and method according to the present disclosure may be mainly used to train security inspection personnel for better understanding of X-ray images, help the security inspection personnel to analyze image contents, and improve the inspection efficiency.
Running of the whole retrieving system/method may also be partitioned into four parts encircled by different dotted lines as illustrated in
Running mechanism of the above-described parts will be described hereinafter:
(1) Establishment and Update of the Fluoroscopic Image Representation Database
All fluoroscopic images, in a fluoroscopic image database, for which no image representation is constructed are firstly subjected to the pre-classification step to obtain a preliminary judgment (texture images or non-texture images) on the contents of the fluoroscopic images; then the image content feature extraction step is performed to obtain feature vectors describing the contents of the fluoroscopic images; and finally the image representation step is performed to construct image representations, and the image representations are stored in the fluoroscopic image representation database.
(2) Retrieval of the Fluoroscopic Images and Display of the Retrieval Result
The user submits to-be-retrieved images. Firstly, these images are subjected to the pre-classification step to obtain a preliminary judgment on image contents (texture images or non-texture images). Subsequently, the image content feature extraction step is performed to obtain feature vectors describing the contents of the fluoroscopic images. Finally, the image representation step is performed to construct image representation. After the image representations are obtained, the retrieval step is performed to obtain a set of fluoroscopic image representations having a higher similarity to each other from the fluoroscopic image representation database according to the retrieval model, and thereby obtain a result of preliminary candidates. Subsequently, the diversified filtering step is performed to select an image subset capable of covering a plurality of article categories from the result of preliminary candidates. A corresponding fluoroscopic image is selected and output from the fluoroscopic image database according to the index of the subset.
(3) Update of the Retrieval Model Based on User Information Feedback
Via the interaction step, the user evaluates satisfaction on the retrieval result returned by a retrieval engine, and feeds back an evaluation result to the retrieving system.
The retrieving system regulates model parameters according to the evaluation result feedback, and triggers an update process of the fluoroscopic image representation database.
(4) Update of the Fluoroscopic Image Database
The user submits a to-be-retrieved image to the fluoroscopic image database, to automatically trigger an establish process of the fluoroscopic image representation database, thereby achieving synchronization between the fluoroscopic image database and the fluoroscopic image representation database.
Without loss of generality, an embodiment of a specific algorithm is illustrated herein. However, it should be understood that under teachings of the present disclosure, a person skilled in the art may make variations or replacements to the specific algorithm in this embodiment without departing from the spirit of the present disclosure.
Pre-Classifying Module
The pre-classifying module randomly samples 50×50 image blocks on the to-be-retrieved image, and the coverage area of all the sampled image blocks needs to be over 60% of the total area of the image. The image blocks are subjected to a Histogram of Oriented Gradient (HOG) feature description, and are classified by a Support Vector Machine (SVM) classifier.
The HOG feature description is employed herein and an SVM classification model is employed for classification. However, the present disclosure is not limited to such combination. Any texture feature and classifier may be applied to the present disclosure.
Image Content Feature Extracting Module
Image content feature extraction is performed on the texture images and the non-texture images using different methods.
With respect to a texture image:
<1>Feature Description Using a Super Pixel as a Unit
Super pixel segmentation is performed for the images using the Statistical Region Merge (SRM) algorithm to establish a super pixel label graph. The images are sampled at an equal spacing, and an HOG feature description is established in a 20×20 region at the sampling site. Based on the super pixel label graph, the HOG feature descriptions at the sampling site of each of the super pixels are combined to construct the HOG feature descriptions of the super pixels.
<2>Establishment of a Super Pixel Feature Dictionary on a Training Image Set
Feature descriptions are established to all the super pixels in each of the training images. HOG feature description vectors of all the super pixels are aggregated to 2000 categories by using the KMeans algorithm. In this way, a super pixel feature dictionary having a capacity of 2000 is constructed. The dictionary would be used in the image representing module.
Herein, the SRM algorithm is used to perform super pixel segmentation, and HOG texture feature description is employed. However, the present disclosure is not limited to such combination. Any image segmentation algorithm and texture feature description algorithm may be applied to the present disclosure.
With respect to a non-texture image:
Feature point detection and feature description are performed on the image by using the Scale-Invariant Feature Transform (SIFT) algorithm, and thereby a feature description vector set is constructed.
Herein, the SIFT algorithm is used to perform feature detection and feature description. However, the present disclosure is not limited to such combination. Any feature point detection algorithm and feature description algorithm may be applied to the present disclosure.
Image Representing Module
The texture images and the non-texture images are represented using different image representation methods.
With respect to a texture image:
Based on the super pixel feature dictionary, feature descriptions of all the super pixels in the image are quantified, and thereby an image representation vector is constructed. The dimensionality of the representation vector is equal to the capacity of the super pixel feature dictionary, and is extremely sparse.
With respect to a non-texture image:
The set of the feature vectors of all the feature points is used as the image representation.
Retrieving Module
Euclidean distance is used for similarity comparison.
With respect to a texture image:
The image representation is indicated by a feature vector. Items relatively similar to the representation of the to-be-retrieved image are searched from the fluoroscopic image representation database using the K Nearest Neighbor (KNN) algorithm, and fluoroscopic image data is obtained from the fluoroscopic image database according to indexes of the items, and the obtained fluoroscopic image data is used as the result of preliminary candidates. A result of preliminary candidates having 200 elements is constructed.
With respect to a non-texture image:
The image representation is indicated by a feature vector set. With respect to each of the feature vectors in the feature vector set, items similar to the representation of the to-be-retrieved image are searched from the fluoroscopic image representation database using the KNN algorithm, and fluoroscopic image data is obtained from the fluoroscopic image database according to indexes of the items. If at least three feature vectors in the feature vector set point at the same fluoroscopic image, then the fluoroscopic image is added to the result of preliminary candidates. A result of preliminary candidates having 200 elements is constructed.
The similarity comparison herein employs the Euclidean distance. However, the present disclosure is not limited thereto. Any method for similarity measurement may be applied to the present disclosure, for example, Manifold Learning, and the like.
Diversified Filtering Module
Three images capable of maximally covering different article categories are selected from a preliminary search result using the structural SVM model.
Correlation Feedback Regulating Module
User's information feedback on the retrieval result is received, and the retrieval model is updated.
Interacting Module
A two-stage tree-like display solution is employed, as illustrated in
According to a further embodiment of the present disclosure, a security inspection device is disclosed. The security inspection device includes the retrieving system based on contents of fluoroscopic images as described above.
Although the present disclosure has been illustrated and described in detail with reference to the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary but not restrictive. The present disclosure is not limited to the disclosed embodiments. Other variations to the disclosed embodiments may be understood and effected by a person skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims.
In the claims and specification, the word “comprising” or “includes” does not exclude other elements or steps, and the infinite article “a” or “an” does not exclude a plurality. A single element or another unit may fulfill functions of several features recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201310734306.5 | Dec 2013 | CN | national |
