Patent Application
20220414395
The present invention relates generally to a system and method for image processing and recognizing a scene of an image. More specifically, the present invention is directed to an AI-based multi-media processing system for recognizing a scene of an image.
Existing solutions for image recognition and analysis often rely on computationally expensive methods such as object recognition methods, pixel-level segmentation, or scanning a detection window over an image, etc. During any object recognition, these existing approaches analyze the pixel data in the image to determine how the image should be segmented. The analysis requires significant processing time and can thus introduce latency or a time lag that can annoy users. As a result, it is generally impractical to use these solutions on mobile computing devices to analyze video streams being captured by those devices in real-time or images captured in the mobile computing devices.
Many existing vehicular video systems provide a little or no interpretation or analysis or recognition of images captured by them, such as images captured by current rear-view cameras. The systems may overlay the road geometry with images to highlight various aspects (e.g., a footprint of a vehicle when parking), but do so without interpreting the scene depicted by the images. Also, some systems may store or share images with other vehicles in compressed or reduced form, but they generally do not provide analysis or interpretation for the images. Other specialized vehicular video systems may identify specific driving-relevant features in the image, such as lane boundaries or crosswalks; they generally do not provide a holistic scene-level analysis to characterize the image.
A U.S. Pat. No. 9,129,161 assigned to Toyota Motor Corp. discloses a technology for inferring scenes from images. The invention further discloses a system for inferring scenes from images including a processor and memory storing instructions that, when executed, cause the system to determine partition regions for an image depicting a scene. The system receives image data including pixels forming the image and classifying pixels of the image into one or more pixel types based on one or more pixel-level features. The system basically determines each partition region and classifies a scene of the image based on the set of pixel characteristic data of each of the partition regions. Though the system classifies scenes in the image through pixels, however, the lack of re-stamping and deep learning makes the system a bit time lagging and less relevant.
A research paper “Object recognition from local scale-invariant features” assigned to D. G. Lowe discloses an object recognition system which is developed using a new class of local image features. The features share similar properties with neurons in the inferior temporal cortex that are used for object recognition in primate vision. Features are efficiently detected through a staged filtering approach that identifies stable points in scale space. The system uses advanced recognition techniques but lacks advanced re-stamping which leads to a little less distinctive or less processed image.
Another research paper “Face Description with Local Binary Patterns” assigned to T. Ahonen discloses an efficient facial image representation based on local binary pattern (LBP) texture features. The face image is divided into several regions from which the LBP feature distributions are extracted and concatenated into an enhanced feature vector to be used as a face descriptor. Though the system is improved on few aspects than the former one, however, the lack of a re-grouper for further classifying the descriptor by various labels leads to a bit unreliable system in terms of scene recognition and time constraint.
The present invention seeks to provide an improvement in the field of recognizing an image, more specifically, but not exclusively, in the field of deep neuro learning image recognition. It is also desirable to keep the cost of the packaging as low as possible while maintaining its durability. The invention is more advanced and precise image processing system with less time lagging feature.
Therefore to overcome the shortcomings of the prior-arts, there is a need to provide an AI-based image processing system. The system utilizes a deep neuro network with probability gating for precise categorization and highlighting the attention region and a background region separately. In view of the foregoing inventions, there is a need in the art for an advanced image processing system to overcome or alleviate the before mentioned shortcomings of the prior arts.
It is apparent now that numerous methods and systems are developed in the prior art that are adequate for various purposes. Furthermore, even though these inventions may be suitable for the specific purposes to which they address, accordingly, they would not be suitable for the purposes of the present invention as heretofore described. Thus, there is a need for an advanced image processing system that recognizes images or multimedia in real-time without lagging using a deep neuro network for labelling.
An image processing system for recognizing a scene of an image, the image processing system includes a pre-processing unit, a processing unit, and a post-processing unit. The pre-processing unit processes the image. The processing includes resizing the image and generating an attention region and a background region of the image. The pre-processing includes re-stamping the attention region on the background region to generate a fused image.
A processing unit for processing the fused image, the processing unit includes an encoder, a labeler, a deep neuro network, and a class re-grouper. The encoder is for generating a plurality of distinctive feature vectors of the fused image. The fused image with distinctive vectors is fed to a labeler for allocating a label from a bunch of pre-defined labels each of the plurality of distinctive feature vectors. The deep neuro network for predicting probability for each of the predefined labels and the class re-grouper for regrouping the pre-defined labels based on probability prediction to generate a plurality of parent labeled vectors.
The system uses feature extraction as a fundamental step to extract useful characteristics from the data, which in image processing is used to derive meaningful values from input images. The input images are marked with distinctive descriptors by an encoder. The descriptor could be a mixture of pixel properties such as color, texture, or shape of the image.
An encoder is first trained to generate a distinctive feature vector for different scene images. Based on these descriptions, they are classified into one of the pre-defined classes. Finally, a second regrouping network is trained to generate a more granular form of a label.
With smart phones embedding more and more sensors for various applications, the benefits are extended to scene or subject label recognition systems. This is done by combining one or more than one channel of multi-mode information with the original image. The input volume for the neuro network includes channels of AF value (auto-focus), EV value (exposure value) and motion value (from gyro or other motion sensors). During training, we introduced a drop-out and noisy scheme which randomly set one or more than one multi-mode channel to null or noisy value. This scheme made our neuro network robust to any setup of the multi-mode sensors, including lack of sensors on the devices. Ultimately, the feature vector extracted will take advantage of information beyond the still image and produces an accurate prediction for application.
A post-processing unit for producing scores based on the plurality of parent class outputs, wherein a final label is generated for to recognize the scene of the image based on the scores.
The primary objective of the invention is to provide a deep learning-based feature extraction model. The deep neuro network assigns a confidence value to pre-defined labels of the image to predict the probability for a group. Moreover, the deep neuro network utilizes the probability gating technique to predict the probability for a group of predefined labels by analyzing illumination, texture, structure, and subjects. Further, the deep neuro network is utilized for feature extraction by learning a plurality of descriptors.
The other objective of the invention is to provide a novel pre-processing technique of attention fusion that refocuses the subject from the background.
Another objective of the invention is to provide a scalable network pipeline is designed to further improve the accuracy of scene and subject label prediction that combines multiple mode
The yet another objective of the invention is to provide a class regrouping strategy that unites different appearances subjects into parent class label and a post-processing method that utilizes different or conditional thresholds for different classes.
Other objectives and aspects of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way for example, the features in accordance with embodiments of the invention.
To the accomplishment of the above and related objects, this invention may be embodied in the form illustrated in the accompanying drawings, attention being called to the fact, however, that the drawings are illustrative only, and that changes may be made in the specific construction illustrated and described within the scope of the appended claims.
Although, the invention is described above in terms of various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functionality described in one or more of the individual embodiments are not limited in their applicability to the particular embodiment with which they are described, but instead can be applied, alone or in various combinations, to one or more of the other embodiments of the invention, whether or not such embodiments are described and whether or not such features are presented as being a part of a described embodiment. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments.
The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent.
The objects and features of the present invention will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are, therefore, not to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
The present invention discloses an image processing system. The system classifies captured images into one of the defined scenes or objects. Images depicted as difficult or ambiguous are not labelled from our classification system. Our system introduced 1) a new pre-processing technique to improve subject-centered attention 2) hierarchical grouping system to merge the semantically similar subject of different appearances to a parent class 3) A probability gating technique to utilize interaction between subjects to improve label accuracy and consistency 4) an output confidence thresholding method for different class to generate the final label.
Image recognition is a process that analyzes and understands an image such as a photo or video. Each image includes a large number of pixels with each pixel corresponds to a specific location in the image. They correspond to different light intensity are typically represented in a tuple of three values, forming Red, Green, and Blue (RGB) color space. An image data may include information that describes a scene or object in the image for example a flower or a human. In addition, they could also convey semantic meanings comprehendible by human such as sunset, or snow.
Understanding an image requires disentangling information represented by an image. It has been developed to utilize information such as colors, textures, groups of objects, background scenes within images. A scene is a view of a real-world surrounding that may have no objects, or include multiple objects. As an example, in our application where Sky is a scene label, the image itself may not contain any physical object. A scene image may contain a large number of physical objects of various types such as humans, buildings, or cars. The location, size, and interaction between objects could also convey different semantics depending on the image captured. For a centered and apparent object with minimal background in an image, it is a straightforward process to identify scene labels.
However, an image with no dominant subject may be difficult to be categorized into an object label. As such, it could fall under the more abstract natural scenes, such as sunset, beach, or they could be an un-categorizable image with no distinct label. Scene understanding can be beneficial in various situations such as applying various image enhancement techniques based on the scene, or regrouping photo albums based on tags.
The proposed method goes through a hard resize technique that directly resizes the image to a square. This is followed by an image restamping strategy that takes the resized image, reduced in resolution, and placed it on top. As such, the overall image is covered with a similar background while the centered unrepeated part is the subject of attention. This image pre-processing method preserves background information and the centered subject is highlighted.
A processing unit 300 for processing the fused image, the processing unit 300 includes an encoder, a labeler, a deep neuro network, and a class re-grouper. The encoder is for generating a plurality of distinctive feature vectors of the fused image. The fused image with distinctive vectors is fed to a labeler for allocating pre-defined to labels each of the plurality of distinctive feature vectors. The deep neuro network for predicting probability for each of the predefined labels and the class re-grouper for regrouping the pre-defined labels based on probability prediction to generate a plurality of parent class outputs.
The system uses feature extraction as a fundamental step to extract useful characteristics from the data, which in image processing is used to derive meaningful values from input images. The input images are marked with distinctive descriptors by an encoder. The descriptor could be a mixture of pixel properties such as color, texture, or shape of the image.
An encoder is first trained to generate a distinctive feature vector for different scene images. Based on these descriptions, they are classified into one of the pre-defined classes. Finally, a second regrouping network is trained to generate a more granular form of a label.
With a smartphone embedding more and more sensors for various applications, the benefits are extended to scene or subject label recognition system. This is done by combining one or more than one channel of multi-mode information with the original image. The input volume for the neuro network includes channels of AF value (auto-focus), EV value (exposure value), and motion value (from gyro or another motion sensor). During training, we introduced a drop-out and noisy scheme which randomly sets one or more than one multi-mode channel to null or noisy value. This scheme made our neuro network robust to any setup of the multi-mode sensor, including lack of sensors on the devices. Ultimately, the feature vector extracted will take advantage of information beyond the still image and produces an accurate prediction for application.
A post-processing unit 400 for producing scores based on the plurality of parent class outputs, wherein a final label is generated to recognize the scene of the image based on the scores.
The system produces a score for each class. Based on the scores, the system generates a final label, which can be one of our targeted classes or ‘unclassifiable’. To alleviate ambiguous labeling due to multiple subjects and the indistinctive scene in the image, every class may have a different confidence threshold. For closely correlated labels, it is inevitable that there exists a scene with flowers surrounded greenery subjects—leaves, grass, etc. Therefore, the final label may be decided based on more than a single class threshold.
Primarily. the pre-processor unit includes an image identifier 204a which further includes an image compressor 206a that crop the image in a square shape. Alternatively, the image is compressed into smaller sizes as per the convenience or requirement of the system.
Further attention fusion processing is performed on the cropped image, and the attention region and a background region of the image is identified. A pixel reducer 208a reduces pixels of the attention region. A re-stamper 210 overlays the attention region on the background region and fuses them together to generate a fused image.
Primarily, the system is integrated on a smartphone for processing real-time images or already clicked images stored in the smartphone and recognizing the scenes on those images. Alternatively, the system is integrated into a camera or a tablet for processing the images.
The method goes through a hard resize technique that directly resizes the input image 202b to square 206b. This is followed by an image restamping strategy that takes the resized image, reduced the resized image in resolution 208b, and placed it on top. As such, the overall image is covered with a similar background while the centered unrepeated part is the subject of attention. This image pre-processing method preserves background information and centered subject is highlighted 210b.
A deep neuro network 306a uses deep neuro learning for predicting probability score for each of the labeled vectors. A class re-grouper 308a for grouping a plurality of labeled vectors based on probability values to generate one or more parent labeled vector.
The processing system processes the fused image and generates the parent labeled vector. The recognized scene is multimedia which either may be an image or a video. The system is an AI-based system that uses neuro learning.
Many different methods have been developed that are based on utilizing the texture information in an image, such as scale-invariant feature transform, a histogram of oriented gradients, local binary patterns (LBPs), and others. In our system, we focus on a deep learning method, that learns descriptors for various targeted classes in a feature extraction model.
With the smartphone embedding more and more sensors for various applications, the benefits are extended to scene or subject label recognition system. This is done by combining one or more than one channel of multi-mode information with the original image. The input volume 302b for the neuro network includes channels of AF value (auto-focus), EV value (exposure value), and motion value (from gyro or another motion sensor). During training, we introduced a drop-out and noisy scheme which randomly sets one or more than one multi-mode channel to null or noisy value. This scheme made our neuro network robust to any setup of the multi-mode sensors, including lack of sensors on the devices. Ultimately, the feature vector extracted 304b will take advantage of information beyond the still image and produces an accurate prediction for application.
The scene subject recognition system identifies pre-defined classes 306b, one of which has an unclassifiable label. The labels include scenes such as sunset, greenery scene, nature, or sky, and the subject could be a cat, dog, human, or food. The system regroups 308b subclasses subject which has different appearances to a parent labeled vector as shown in
The post-processing system generates scores through the confidence threshold in order to recognize the scene of multimedia which either may be an image or a video. The system is an AI-based system that uses neuro learning.
To alleviate ambiguous labeling due to multiple subjects and indistinctive the scene in the image, every class may have a different confidence threshold. For closely correlated labels, such as ‘flower’ and ‘greenery’, it is inevitable that there exists a scene with flowers surrounded greenery subjects—leaves, grass, etc.
Therefore, the final label 404b may be decided based on more than a single class threshold. For example, the system may require 80% confidence to output a ‘flower’ label when the second-highest label is ‘greenery’ and may require 90% accuracy otherwise.
Secondly, in the in-processing step 510, the feature extraction is performed on the fused image via a deep learning model and generating one or more parent labeled vectors. Further, the processing includes generating a plurality of distinctive feature vectors of the fused image 512. Followed by, assigning a label from a set of pre-defined labels to each of the plurality of distinctive feature vectors 514 to generate a labeled vector. This followed by predicting probability for each of the labeled vectors 516 and finally regrouping a plurality of labeled vectors based on probability value to generate one or more parent labeled vector 518.
Lastly, generating scores based on one or more parent labeled vector and a final label is generated 520 to recognize the scene of the image based on the scores. The final label is developed through the multi-class specific threshold.
While the various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not of limitation. Likewise, the figure may depict an example architectural or other configuration for the invention, which is done to aid in understanding the features and functionality that can be included in the invention. The invention is not restricted to the illustrated example architectures or configurations, but the desired features can be implemented using a variety of alternative architecture and configurations.
Although, the invention is described above in terms of various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functionality described in one or more of the individual embodiments are not limited in their applicability to the particular embodiment with which they are described, but instead can be applied, alone or in various combinations, to one or more of the other embodiments of the invention, whether or not such embodiments are described and whether or not such features are presented as being a part of a described embodiment. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments.
The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent.
