The present disclosure relates to systems for improving images using neural embedding techniques to reduce processing complexity and improve images or video. In particular, described is a method and system using neural embedding to provide classifiers that can be used to configure image processing parameters or camera settings.
Digital cameras typically require a digital image processing pipeline that converts signals received by an image sensor into a usable image. Processing can include signal amplification, corrections for Bayer masks or other filters, demosaicing, colorspace conversion, and black and white level adjustment. More advanced processing steps can include HDR in-filling, super resolution, saturation, vibrancy, or other color adjustments, tint or IR removal, and object or scene classification. Using various specialized algorithms, corrections can be made either on-board a camera, or later in post-processing of RAW images. However, many of these algorithms are proprietary, difficult to modify, or require substantial amounts of skilled user work for best results. In many cases, using traditional neural network methods is impractical due limited available processing power and high dimensionality of a problem. An imaging system may additionally make use of multiple image sensors to achieve its intended use-case. Such systems may process each sensor completely independently, jointly, or in some combination thereof. In many cases, processing each sensor independently is impractical due to the cost of specialized hardware for each sensor, whereas processing all sensors jointly is impractical due to limited system communication-bus bandwidth and high neural network input complexity. Methods and systems that can improve image processing, reduce user work, and allow updating and improvement are needed.
Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various figures unless otherwise specified.
In some of the following described embodiments, systems for improving images using neural embedding information or techniques to reduce processing complexity and improve images or video are described. In particular, a method and system using neural embedding to provide classifiers that can be used to configure image processing parameters or camera settings. In some embodiments, methods and systems for generating neural embeddings and using these neural embeddings for a variety of applications including: classification and other machine learning tasks, reducing bandwidth in imaging systems, reducing compute requirements in neural inference systems (and as a result power), identification and association systems such as database queries and object tracking, combining information from multiple sensors and sensor types, generating novel data for training or creative purposes, and reconstructing system inputs.
In some embodiments, an image processing pipeline including a still or video camera further includes a first portion of an image processing system arranged to use information derived at least in part from a neural embedding. A second portion of the image processing system can be used to modify at least one of an image capture setting, sensor processing, global post processing, local post processing, and portfolio post processing, based at least in part on neural embedding information.
In some embodiments, an image processing pipeline can include a still or video camera that includes a first portion of an image processing system arranged to reduce data dimensionality and effectively downsample an image, images, or other data using a neural processing system to provide neural embedding information. A second portion of the image processing system can be arranged to modify at least one of an image capture setting, sensor processing, global post processing, local post processing, and portfolio post processing, based at least in part on the neural embedding information.
In some embodiments, an image processing pipeline can include a first portion of an image processing system arranged for at least one of categorization, tracking, and matching using neural embedding information derived from a neural processing system. A second portion of the image processing system can be arranged to modify at least one of an image capture setting, sensor processing, global post processing, local post processing, and portfolio post processing, based at least in part on the neural embedding information.
In some embodiments, an image processing pipeline can include a first portion of an image processing system arranged to reduce data dimensionality and effectively downsample an image, images, or other data using a neural processing system to provide neural embedding information. A second portion of the image processing system can be arranged to preserve the neural embedding information within image or video metadata.
In some embodiments, an image capture device includes a processor to control image capture device operation. A neural processor is supported by the image capture device and can be connected to the processor to receive neural network data, with the neural processor using neural network data to provide at least two processing procedures selected from a group including sensor processing, global post processing, and local post processing.
After image capture, neural network based sensor processing (step 112A) can be used to provide custom demosaic, tone maps, dehazing, pixel failure compensation, or dust removal. Other neural network based processing can include Bayer color filter array correction, colorspace conversion, black and white level adjustment, or other sensor related processing.
Neural network based global post processing (step 114A) can include resolution or color adjustments, as well as stacked focus or HDR processing. Other global post processing features can include HDR in-filling, bokeh adjustments, super-resolution, vibrancy, saturation, or color enhancements, and tint or IR removal.
Neural network based local post processing (step 116A) can include red-eye removal, blemish removal, dark circle removal, blue sky enhancement, green foliage enhancement, or other processing of local portions, sections, objects, or areas of an image. Identification of the specific local area can involve use of other neural network assisted functionality, including for example, a face or eye detector.
Neural network based portfolio post processing (step 116A) can include image or video processing steps related to identification, categorization, or publishing. For example, neural networks can be used to identify a person and provide that information for metadata tagging. Other examples can include use of neural networks for categorization into categories such as pet pictures, landscapes, or portraits.
In more detail, the imaging system can include an optical system that is controlled and interacts with an electronics system. The optical system contains optical hardware such as lense and an illumination emitter, as well electronic, software or hardware controllers of shutter, focus, filtering and aperture. The electronics system includes a sensor and other electronic, software or hardware controllers that provide filtering, set exposure time, provide analog to digital conversion (ADC), provide analog gain, and act as an illumination controller. Data from the imaging system can be sent to the application layer for further processing and distribution and control feedback can be provided to a neural processing system (NPS).
The neural processing system can include a front-end module, a back-end module, user preference settings, portfolio module, and data distribution module. Computation for modules can be remote, local, or through multiple cooperative neural processing systems either local or remote. The neural processing system can send and receive data to the application layer and the imaging system.
In the illustrated embodiment, the front-end includes settings and control for the imaging system, environment compensation, environment synthesis, embeddings, and filtering. The back-end provides linearization, filter correction, black level set, white balance, and demosaic. User preferences can include exposure settings, tone and color settings, environment synthesis, filtering, and creative transformations. The portfolio module can receive this data an provide categorization, person identification, or geotagging. The distribution module can coordinate sending a receiving data from multiple neural processing systems and send and receive embeddings to the application layer. The application layer provides a user interface to custom settings, as well as image or setting result preview. Images or other data can be stored and transmitted, and information relating to neural processing systems can be aggregated for future use or to simplify classification, activity or object detection, or decision making tasks.
A wide range of still or video cameras can benefit from use neural network supported image or video processing pipeline system and method. Camera types can include but are not limited to conventional DSLRs with still or video capability, smartphone, tablet cameras, or laptop cameras, dedicated video cameras, webcams, or security cameras. In some embodiments, specialized cameras such as infrared cameras, thermal imagers, millimeter wave imaging systems, x-ray or other radiology imagers can be used. Embodiments can also include cameras with sensors capable of detecting infrared, ultraviolet, or other wavelengths to allow for hyperspectral image processing.
Cameras can be standalone, portable, or fixed systems. Typically, a camera includes processor, memory, image sensor, communication interfaces, camera optical and actuator system, and memory storage. The processor controls the overall operations of the camera, such as operating camera optical and sensor system, and available communication interfaces. The camera optical and sensor system controls the operations of the camera, such as exposure control for image captured at image sensor. Camera optical and sensor system may include a fixed lens system or an adjustable lens system (e.g., zoom and automatic focusing capabilities). Cameras can support memory storage systems such as removable memory cards, wired USB, or wireless data transfer systems.
In some embodiments, neural network processing can occur after transfer of image data to a remote computational resources, including a dedicated neural network processing system, laptop, PC, server, or cloud. In other embodiments, neural network processing can occur within the camera, using optimized software, neural processing chips, dedicated ASICs, custom integrated circuits, or programmable FPGA systems.
In some embodiments, results of neural network processing can be used as an input to other machine learning or neural network systems, including those developed for object recognition, pattern recognition, face identification, image stabilization, robot or vehicle odometry and positioning, or tracking or targeting applications. Advantageously, such neural network processed image normalization can, for example, reduce computer vision algorithm failure in high noise environments, enabling these algorithms to work in environments where they would typically fail due to noise related reduction in feature confidence. Typically, this can include but is not limited to low light environments, foggy, dusty, or hazy environments, or environments subject to light flashing or light glare. In effect, image sensor noise is removed by neural network processing so that later learning algorithms have a reduced performance degradation.
In certain embodiments, multiple image sensors can collectively work in combination with the described neural network processing to enable wider operational and detection envelopes, with, for example, sensors having different light sensitivity working together to provide high dynamic range images. In other embodiments, a chain of optical or algorithmic imaging systems with separate neural network processing nodes can be coupled together. In still other embodiments, training of neural network systems can be decoupled from the imaging system as a whole, operating as embedded components associated with particular imagers.
An example of a display system 206 is a high quality electronic display. The display can have its brightness adjusted or may be augmented with physical filtering elements such as neutral density filters. An alternative display system might comprise high quality reference prints or filtering elements, either to be used with front or back lit light sources. In any case, the purpose of the display system is to produce a variety of images, or sequence of images, to be transmitted to the imaging system.
The imaging system being profiled is integrated into the profiling system such that it can be programmatically controlled by the control and storage computer and can image the output of the display system. Camera parameters, such as aperture, exposure time, and analog gain, are varied and multiple exposures of a single displayed image are taken. The resulting exposures are transmitted to the control and storage computer and retained for training purposes.
The entire system is placed in a controlled lighting environment, such that the photon “noise floor” is known during profiling.
The entire system is setup such that the limiting resolution factor is the imaging system. This is achieved with mathematical models which take into account parameters, including but not limited to: imaging system sensor pixel pitch, display system pixel dimensions, imaging system focal length, imaging system working f-number, number of sensor pixels (horizontal and vertical), number of display system pixels (vertical and horizontal). In effect a particular sensor, sensor make or type, or class of sensors can be profiled to produce high-quality training data precisely tailored to an individual sensors or sensor models.
Various types of neural networks can be used with the systems disclosed with respect to
One neural network embodiment of particular utility is a fully convolutional neural network. A fully convolutional neural network is composed of convolutional layers without any fully-connected layers usually found at the end of the network. Advantageously, fully convolutional neural networks are image size independent, with any size images being acceptable as input for training or bright spot image modification. An example of a fully convolutional network 400 is illustrated with respect to
In the described embodiment of
Data packaging takes one or many training data sample(s), normalizes it according to a determined scheme, and arranges the data for input to the network in a tensor. Training data sample may comprise sequence or temporal data.
Preprocessing lambda allows the operator to modify the source input or target data prior to input to the neural network or objective function. This could be to augment the data, to reject tensors according to some scheme, to add synthetic noise to the tensor, to perform warps and deformation to the data for alignment purposes or convert from image data to data labels.
The network 516 being trained has at least one input and output 518, though in practice it is found that multiple outputs, each with its own objective function, can have synergetic effects. For example, performance can be improved through a “classifier head” output whose objective is to classify objects in the tensor. Target output data 508, source output data 518, and objective function 520 together define a network's loss to be minimized, the value of which can be improved by additional training or data set processing.
In some embodiments, neural network embeddings are useful because they can reduce the dimensionality of categorical variables and represent categories in the transformed space. Neural embeddings are particularly useful for categorization, tracking, and matching, as well as allowing a simplified transfer of domain specific knowledge to new related domains without needing a complete retraining of a neural network. In some embodiments, neural embeddings can be provided for later use, for example by preserving a latent vector in image or video metadata to allow for optional later processing or improved response to image related queries. For example, a first portion of an image processing system can be arranged to reduce data dimensionality and effectively downsample an image, images, or other data using a neural processing system to provide neural embedding information. A second portion of the image processing system can also be arranged for at least one of categorization, tracking, and matching using neural embedding information derived from the neural processing system. Similarly, neural network training system can include a first portion of a neural network algorithm arranged to reduce data dimensionality and effectively downsample an image or other data using a neural processing system to provide neural embedding information. A second portion of a neural network algorithm is arranged for at least one of categorization, tracking, and matching using neural embedding information derived from a neural processing system and a training procedure is used to optimize the first and second portions of the neural network algorithm.
In some embodiments, a training and inference system can include a classifier or other deep learning algorithm that can be combined with the neural embedding algorithm to create a new deep learning algorithm. The neural embedding algorithm can be configured such that its weights are trainable or non-trainable, but in either case will be fully differentiable such that the new algorithm is end-to-end trainable, permitting the new deep learning algorithm to be optimized directly from the objective function to the raw data input.
During inference, the above described algorithm (C) can be partitioned such that the embedding algorithm (A) executes on an edge or endpoint device, while the algorithm (B) can execute on a centralized computing resource (cloud, server, gateway device).
More specifically, as seen in
Bus mediation communication of neural networks such as discussed with respect to
As another example, a city, venue, or sports arena IP-camera system can be configured so that each camera outputs latent vectors that are stored or otherwise made available for video analytics. These latent vectors can be searched to identify objects, persons, scenes, or other image information without needing to provide real time searching of large amounts of image data. This allows performance of realtime video or image analysis on hundreds or thousands of cameras to find, for example, a red car associated with a certain person or scene, without needing access to large data pipeline and a large and expensive server.
As will be understood, the camera system and methods described herein can operate locally or in via connections to either a wired or wireless connect subsystem for interaction with devices such as servers, desktop computers, laptops, tablets, or smart phones. Data and control signals can be received, generated, or transported between varieties of external data sources, including wireless networks, personal area networks, cellular networks, the Internet, or cloud mediated data sources. In addition, sources of local data (e.g. a hard drive, solid state drive, flash memory, or any other suitable memory, including dynamic memory, such as SRAM or DRAM) that can allow for local data storage of user-specified preferences or protocols. In one particular embodiment, multiple communication systems can be provided. For example, a direct Wi-Fi connection (802.11b/g/n) can be used as well as a separate 4G cellular connection.
Connection to remote server embodiments may also be implemented in cloud computing environments. Cloud computing may be defined as a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned via virtualization and released with minimal management effort or service provider interaction, and then scaled accordingly. A cloud model can be composed of various characteristics (e.g., on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, etc.), service models (e.g., Software as a Service (“SaaS”), Platform as a Service (“PaaS”), Infrastructure as a Service (“IaaS”), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).
Reference throughout this specification to “one embodiment,” “an embodiment,” “one example,” or “an example” means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” “one example,” or “an example” in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, databases, or characteristics may be combined in any suitable combinations and/or sub-combinations in one or more embodiments or examples. In addition, it should be appreciated that the figures provided herewith are for explanation purposes to persons ordinarily skilled in the art and that the drawings are not necessarily drawn to scale.
The flow diagrams and block diagrams in the described Figures are intended to illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It will also be noted that each block of the block diagrams and/or flow diagrams, and combinations of blocks in the block diagrams and/or flow diagrams, may be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flow diagram and/or block diagram block or blocks.
Embodiments in accordance with the present disclosure may be embodied as an apparatus, method, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware-comprised embodiment, an entirely software-comprised embodiment (including firmware, resident software, micro-code, etc.), or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module,” or “system.” Furthermore, embodiments of the present disclosure may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.
Any combination of one or more computer-usable or computer-readable media may be utilized. For example, a computer-readable medium may include one or more of a portable computer diskette, a hard disk, a random access memory (RAM) device, a read-only memory (ROM) device, an erasable programmable read-only memory (EPROM or Flash memory) device, a portable compact disc read-only memory (CDROM), an optical storage device, and a magnetic storage device. Computer program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages. Such code may be compiled from source code to computer-readable assembly language or machine code suitable for the device or computer on which the code will be executed.
Many modifications and other embodiments of the invention will come to the mind of one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is understood that the invention is not to be limited to the specific embodiments disclosed, and that modifications and embodiments are intended to be included within the scope of the appended claims. It is also understood that other embodiments of this invention may be practiced in the absence of an element/step not specifically disclosed herein.
This application claims the benefit of U.S. Provisional Application Ser. No. 63/071,966, filed Aug. 28, 2020, and entitled CAMERA IMAGE OR VIDEO PROCESSING PIPELINES WITH NEURAL EMBEDDING, which is hereby incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
63071966 | Aug 2020 | US |