Patent Application
20180183998
The systems and methods disclosed herein are directed to reducing power in an imaging sensor, and more particularly, to controlling a sensor to selectively downscale certain portions of the sensor based on classification and segmentation of subject-free areas of an image.
Today, video capture processes and hardware is being pushed to the edge with high-resolutions and high frame-rates in stand-alone imaging systems and cameras that are included on mobile device, e.g., cell phones and tablets. As high-resolution resolution sensors that are used in such imaging systems/devices continue to increase to 16 megabytes and above for both video and still pictures, corresponding higher-end image processors are needed to effectively support the processing of the high through-put of such applications, and this can cause the system-on-chips (SOCs) preforming this processing to generate an undesired level of heat and cause large power consumption.
Providing less data from a sensor by downscaling can reduce heat generation. Downscaling at the sensor generally refers to forming groups of pixels to reduce the resolution of an image, or a portion of an image, and thus forming a smaller image that maybe easier to process or transmit. An imager may be configured to perform pixel binning on captured images. Pixel binning may be performed by forming groups of pixels and combining sampled values from the pixels in each group. The sampled values from the pixels may be combined by assigning weights to each pixel, scaling the sampled values by the corresponding weights, and summing the scaled values. The groups of pixels and pixel weights may be selected to produce binned images with even spatial distribution. The pixel binning operation may be performed by processing circuitry that receives captured image data from the imager. In some embodiments, the pixel binning operation may also be separated into a horizontal binning step and a vertical binning step that are performed by image readout circuitry during image readout. During the horizontal binning step, pixels of a particular row may be combined. During the vertical binning step, pixels of particular columns may be combined. Accordingly, intelligent use of downscaling techniques may be advantageous so that SOCs produce less heat and a device's power consumption is reduced.
In general, this disclosure is related to systems and techniques for simultaneously power reduction and performance improvement through selective sensor downscaling. Certain aspects of embodiments of systems and methods are disclosed herein. The one or more aspects may be included in various embodiments of such systems and methods. In addition, certain embodiments of the disclosed systems and methods may not include one or more of the disclosed aspects.
One innovation includes an imaging device including an image sensor having an array of imaging elements and readout circuitry for generating image information of a target scene, the image information comprising at least first and second image information. In some embodiments, the image sensor is configured to generate first image information of a first spatial resolution, the first image information including image data corresponding to a pixel values generated by a first portion of the array of imaging elements, and generate second image information of a second spatial resolution, the second image information including image data corresponding to pixel values generated by a second portion of the array of imaging elements, the second spatial resolution being less than the first spatial resolution, the image sensor being further configured to control the downscaling of the spatial resolution of portions of the array of imaging elements responsive to received downscaling control information. The system may also include a memory component, and a processor coupled to the memory component, the processor configured to receive an image of the target scene and classify content of the image based on predetermined criteria to identify candidate portions of the image for downscaling, determine downscaling control information for the candidate portions that for controlling downscaling of an image generated by the image sensor, and provide the downscaling control information to the image sensor.
Some other aspects of embodiments of such systems are disclosed below. In some embodiments, the image received by the processor is based on image data in the first image information and the second image information generated by the image sensor. The second image information may include meta-data identifying the downscaling of the image data in the second image information. The image sensor may be further configured to control downscaling of the second image information using the downscaling control information. In some embodiments, the image sensor is configured to perform vertical binning to generate the second image information responsive to the downscaling control information. The imaging device may further include an image processor in electronic communication with the image sensor and the processor, and the image processor may be configured to receive the first and second image information from the image sensor, upscale the image data in the second image information, and provide the image to the processor. In one aspect, the image information includes meta-data identifying the downscaling performed by the image sensor, and wherein the image processor is configured to upscale the image data in the second image information based on the meta-data. In another aspect, the image processor is in electronic communication with the processor, and wherein the image processor is configured to receive an IP control input from the processor and upscale the image data in the second image information based on the IP control input. In another aspect, the image processor comprises one or more filters configured to upscale the image data in the second image information. The processor may be further configured to provide an IP control input to the image processor, the IP control input identifying image processing operations for the image processor to perform on image data received from the image sensor.
In some embodiments, the image processor is configured to, responsive to the IP control input, downscale portions of image data received from the image sensor and subsequently upscale the downscaled portions. In some embodiments, the IP control input includes information that identifies portions of image data to upscale. The predetermined criteria used to determine candidate portions of an image of the target scene may include a dynamic range threshold value to identify portions of the content of the image that have a small range of pixel values. The predetermined criteria may include a sharpness threshold to identify portions of the content of the image that includes un-sharp pixel values. The predetermined criteria may include a contrast threshold to identify portions of the content of the image that includes pixel values indicative of out-of-focus image data content. The predetermined criteria may include a contrast threshold for identifying subject and background portions of the content of the image.
Another innovation is a method of processing image data, the method including receiving, by an electronic hardware processor, an image of a target scene that is generated at least in part by an image sensor, classifying, by the electronic hardware processor, content of the image based on predetermined criteria to identify candidate portions of the target scene for downscaling, generating, by the electronic hardware processor, downscaling control information that identifies one or more of the candidate portions of the target scene for downscaling by the image sensor, communicating the downscaling control information from the electronic hardware processor to the image sensor, and generating image information of the target scene by the image sensor based on the downscaling control information, the image information comprising first image information of a first spatial resolution, the first image information including image data corresponding to pixel values generated by a first portion of the array of imaging elements, and second image information of a second spatial resolution, the second image information including downscaled image data corresponding to pixel values generated by a second portion of the array of imaging elements, the second spatial resolution being less than the first spatial resolution.
In such methods, the image of the target scene received by the processor may be based on image information generated by the image sensor. In some embodiments, the second image information includes meta-data identifying the downscaled image data in the second image information. In some embodiments, generating the image information includes performing vertical binning to generate the second image information. Some embodiments of such methods further include receiving, at an image processor, the first and second image information from the image sensor, upscaling the image data in the second image information at the image processor, and providing, from the image processor, the image of the target scene to the electronic hardware processor. In some aspects of embodiments of such methods, the image information includes meta-data identifying downscaling performed by the image sensor, and wherein upscaling the image data in the second image information is based on the meta-data. In some aspects of embodiments of such methods, the image processor is in electronic communication with the electronic hardware processor, and the method further comprises receiving at the image processor an IP control input from the electronic hardware processor, and upscaling the image data in the second image information based on the IP control input. In some aspects, the IP control input includes information that identifies portions of a target scene to image data to upscale. In some aspects, at the image processor and responsive to the IP control input, downscaling portions of the image information image and subsequently upscaling the downscaled portions. In some aspects, the predetermined criteria includes at least one of a dynamic range threshold value for identifying portions of the content of the image that have a small range of pixel values, contrast threshold for identifying portions of the content of the image that includes pixel values indicative of out-of-focus image data content, or a sharpness threshold for identifying portions of the content of the image that includes un-sharp pixel values. Some embodiments of such methods further comprise generating the image data of the second image information by performing vertical binning on the image sensor.
Another innovation includes a non-transitory computer readable medium comprising instructions that when executed cause an electronic hardware processor to perform a method of image collection at an image sensor, the method including receiving, by the electronic hardware processor, an image of a target scene that generated at least in part by an image sensor comprising an array of imaging elements, classifying, at the electronic hardware processor, content of the received image based on predetermined criteria to identify candidate portions of the target scene for downscaling, generating, by the electronic hardware processor, downscaling control information that identifies one or more of the candidate portions of the target scene for downscaling by the image sensor, and communicating the downscaling control information from the electronic hardware processor to the image sensor for use by the image sensor to capture image information of the target scene, the image information including first image information of a first spatial resolution, the first image information including image data corresponding to pixel values generated by a first portion of the array of imaging elements, and second image information of a second spatial resolution, the second image information including downscaled image data corresponding to pixel values generated by a second portion of the array of imaging elements, the second spatial resolution being less than the first spatial resolution. In some embodiments, the method further includes generating an IP control input at the electronic hardware processor and providing the IP control input to an image processor, the IP control input including information for use by the image processor to upscale the image data of the second image information.
Another innovation is a method of processing image data, the method including receiving, at an electronic hardware processor from an image processor, an image of a target scene that was captured by an image sensor, classifying, by the electronic hardware processor, content of the image based on predetermined criteria to identify portions of the target scene for downscaling, generating, by the electronic hardware processor, downscaling control information that indicates one or more portions of the target scene for downscaling by the image sensor, providing the downscaling control information to the image sensor, generating, by the image sensor, image information of the target scene based on the downscaling control information, the image information includes first image information of a first spatial resolution, the first image information including image data corresponding to pixel values generated by a first portion of the array of imaging elements, and second image information of a second spatial resolution, the second image information including downscaled image data corresponding to pixel values generated by a second portion of the array of imaging elements, the second spatial resolution being less than the first spatial resolution. The method further includes providing the image information to the image processor; and upscaling the image data of the second image information, at the image processor.
The disclosed aspects will hereinafter be described in conjunction with the appended drawings and appendices, provided to illustrate and not to limit the disclosed aspects, wherein like designations denote like elements.
The systems and methods disclosed herein are directed to reducing power in an imaging sensor, for example, controlling a sensor to selectively downscale certain portions of the sensor based on classification and segmentation of subject-free areas of an image.
There has been a trend with the latest generation image sensors to increase resolution while also reducing the pixel size at the expense of some other performance parameters. Controlled downscaling of an image sensor may increase the performance of a video or still camera system machine by reducing the thermal energy produced and lowering bandwidth requirements. Binning is one example of downscaling. If you do not need all of the resolution provided in some portions of an image, then binning is an easy solution.
In some embodiments, binning may be performed by adding the charge of two or more pixels together. The charge in a target pixel then represents the illumination of the two or more pixels, that is, the photons received by the two or more pixels. In some embodiments, pixels may be binned vertically by shifting two image rows into the horizontal register without reading it after the first shift. In some embodiments, pixels may be binned horizontally by shifting the horizontal register two times into output node without resetting it after the first shift. Due to the architecture of most (if not all) sensors, horizontal binning cannot be done on the image sensor, and is instead performed in image sensor readout circuitry, or may be performed downstream during image processing operations, for example, performed by an image processor. Vertical binning, however, can be done on the sensor level. With vertical binning, the charge of multiple lines may be combined in the sensor before they are readout. For example, for a vertical binning of four lines (e.g., rows) of image data, a vertical shift (or transport) and summing of four lines occurs and then a horizontal transport of this binned line takes place. Then, the cycle of vertical shift of more four lines and horizontal readout starts again and repeats.
Combining of both vertical and horizontal binning results in square (or rectangular) image binning. For example, 2×2 binning is a combination of 2× vertical and 2× horizontal binning. One advantage of binning is higher signal-to-noise ratio (SNR) due to reduced read noise contributions and increased signal combinations. CCD read noise is added during each readout event and in normal operation, read noise will be added to each pixel. However, in binning mode, read noise is added to each super pixel, which has the combined signal from multiple pixels. In the ideal case, this produces SNR improvement equal to the binning factors. With binning you can increase pixel size arbitrarily (but reduce the spatial resolution). In the limit we could even read out the CCD as a single large pixel.
Binning also increases frame rate. Since the slowest step in the readout sequence is usually the digitization of a given pixel, binning can be used to increase the effective total frame rate of a given system. Thus, highly “binned” lower-resolution images can be obtained when a high speed frame rate is desired. By selectively downscaling subject-free, or non-important, parts of an image, an imaging system may decrease the amount (payload) of image data produced while maintaining quality in regions of interest. In some embodiments, the images are downscaled by vertical binning at the sensor and horizontal at the sensor (e.g., using the readout circuitry) before the data is communicated to an image processor. In other embodiments, further downscaling is performed by an image processor which receives image information from the sensor, the image information received having been vertically binned at the sensor. Based on the history of the image data being produced (for example, as provided to the viewfinder for still images, and one-or-more previous frames for video), one may detect the areas of the image that may be characterized as non-important. For example, areas of the image that are flat, out-of-focus, and/or far/background areas may be detected using various image processing techniques. For example, determining one or more portions of an image that has a dynamic range below a certain threshold value (flat image data), determining one or more portions of an image that have edge transitions below a certain threshold, that is, unsharp edges (out-of-focus image data), and/or determining one or more portions of an image that is background information (for example, by detecting sharp portions and un-sharp surrounding sharp portions, the unsharp portions being different than the sharp portions by a certain threshold value). Instructions for downscaling these “unimportant” areas can be passed to the sensor and be applied in the sensor to control binning, and/or this information may be supplied to an image processor to control downscaling (e.g., binning). This will effectively lead to less pixels being processed in the image chain from the sensor to (and thru at least some portions of) the image processor, thus saving significant power and increasing performance. Subsequently, the image processor, having the information of where downscaling may have occurred, may upscale the image data in the previously downscaled portions of the images (and other portions if desired) and still images or video may be provided at a desired resolution.
When performing the downscaling in the sensor, one gains additional benefits due to smaller bandwidth of the sensor-image processor interface. In some embodiments, horizontal binning may only be performed due to the rolling shutter. The ability to apply vertical binning, in the scenario of rolling shutter, may be dependent on the shake frequency. For example, vertical binning in case of high frequency shakes will cause blur and artifacts. If handled in the ISP, this can be avoided through proper processing. As in some implementations, when performing the downscaling in the front end image processor, one gains the ability to apply both horizontal and vertical downscaling, which can improve quality.
The example power reduction system 100 illustrated in
The image information 121 communicated from the image sensor 102 to the image processor 104 may include image data, for example, data that represents the signal information generated by the array of imaging elements of the image sensor 102 from light received from a target scene. The image information 102 may also include metadata that includes information of downscaling performed by the image sensor. For example, the metadata may include information that identifies a portion, or portions, of the image data as being downscaled (or not downscaled) by the image sensor 102, and indicate how it was downscaled, e.g., horizontally, vertically and/or to what extent: 2×, 3× 4×, 5×, 6×, 7×, 8×, 16×, 32×, etc. Note: the downscaling ratio does not have to be a factor of two. The metadata in the image information 121 is received by the image processor 104, and may be used by the image processor to upscale the previously downscaled portions of the image, or perform some other type of image processing operation on certain portions of the image data received in the image information 121. The metadata may include: (1) the scaling information, for example, block size and location, scaling factor, downscaling method (e.g., vertical and/or horizontal, bilinear, bicubic, or another method); (2) classifier and segmentation results; and (3) information regarding any processing done by the image sensor (e.g., the sensor chip) or done by filters preceding the image processor 104.
The image processor (or “media processor”) 104 may be a specialized digital signal processor (DSP) used for image processing applications in digital cameras, mobile phones or other devices. The image processor 104 may employ parallel computing with single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) technologies to increase speed and efficiency through processing using a number of processors that function asynchronously and independently. At any time, different processors may be executing different instructions on different pieces of data. The digital image processing engine can perform a range of tasks. In some embodiments, to increase the system integration on embedded devices, the image processor 104 maybe a system on a chip (SOC) with multi-core processor architecture.
The image processor 104 may include one or more filters 106a-c that are configured to perform image processing operations on image information 121 received from the image sensor 102, for example, the image processor operations may include filtering. The filters 106a-c can be configured in hardware, software, or a combination of hardware and software. “Filters” in this context is used to generally refer to a way of processing image data to change an aspect of the image data. For example, one or more of the filters 106a-c may be configured to downscale or upscale one or more portions of received image data, or all of the image data. The one or more of the filters 106a-c may also be configured to perform a variety of other image processing functionality on image data including, for example, low pass filtering, high pass filtering, median filtering, reconstruction filtering, and/or image enhancement filtering (changing dynamic range, brightness, contrast, or color). The image information 121 may include metadata and image data. The image data can include information representative of the light received by each imaging element in the image sensor when a target scene is captured. One or more portions of the image data may be downscaled image data, for example, that was downscaled by the image sensor 102 by vertical binning. In some embodiments, the metadata identifies portions of the image information that was downscaled by the image sensor 102. The metadata may be used by the image processor 104 to control the one or more filters 106a-c to perform downscaling of the image data, or to determine what portions of the image data should be upscaled because they were previously downscaled.
The power reduction system 100 also includes a classification and segmentation module 108 that may directly or indirectly receive image data from the sensor 102. The classification and segmentation module 108 may be configured to classify one or more portion of an image of the image data. In some embodiments, the classification and segmentation module 108 classifies one or more portions of the image data to be a particular category, for example, a subject portion, a background portion, an out-of-focus portion, a “flat” portion, and/or another “non-important” portion of the image. For example, an unimportant portion of the image may be a portion that is not as important as an identified subject portion of the image). The classification and segmentation module 108 also may determine segmentation information for the classified one or more portions of the image data, the segmentation information indicating which portion of an image should be segmented. Some representative examples of classifying and segmenting an image are illustrated in
Still referring to
The classification and segmentation module 108 may also provide a IP control input 133 to the image processor 104. The IP control input 133 may include information that the image processor 104 uses to upscale images that were downscaled by the image sensor 104. For example, in some embodiments an IP control input 133 may include information to identify which portions of the image data was downscaled by the image sensor 102, similar to the information discussed in reference to
Address generator circuitry 132 may generate signals on paths 134 as desired to control the functionality of imaging elements 128. For example, address generator circuitry 132 may generate reset signals on reset lines in paths 134, transfer signals on transfer lines in paths 134, and row select (e.g., row readout) signals on row select lines in paths 134 to control the operation of array 114. In some embodiments, address generator circuitry 132 and array 114 may be integrated together in a single integrated circuit.
Image readout circuitry 142 may include sample and hold circuitry, analog-to-digital converter circuitry, and line buffer circuitry, as examples. Image data from the image readout circuitry 142 may be provided to the image processor 104. As one example, circuitry 142 may be used to measure signals in imaging elements 128 and may be used to buffer the signals while analog-to-digital converters in circuitry 142 convert the signals to digital signals. In a typical arrangement, circuitry 142 reads signals from rows of imaging elements 128 one row at a time over lines 140. The digital signals read out by readout circuitry 142 may be representative of charges accumulated by imaging elements 128 in response to incident light. The digital signals (image information) 121 produced by the analog-to-digital converters of circuitry 142 may be conveyed to image processor circuitry 104, and may include image data and metadata.
When capturing target scenes, light enters the lens assembly 285 and is focused on the image sensor 102. In some embodiments, the lens assembly 205 is part of an autofocus lens system which can include multiple lenses and adjustable optical elements. In one aspect, the image sensor 102 utilizes a CMOS or CCD sensor. The lens assembly 285 is coupled to the actuator 212, and is moved by the actuator 212 relative to the image sensor 102. The actuator 112 is configured to move one or more optical elements of the lens assembly 205 in a series of one or more lens movements during an AF operation, for example, adjusting the lens position to change the focus of an image on the image sensor 102.
The display 280 is configured to display images captured by the image sensor 102 and may also be configured to be an user interface of the imaging system 200. In one implementation, display 280 can be configured to display one or more objects selected by a user, via an input device 190, which maybe a touchscreen incorporated on the display 280, of the imaging system 200. In some embodiments, the imaging system 200 may not include the display 280.
The input device 290 may take on many forms depending on the implementation. In some implementations, the input device 290 may be integrated with the display 280 so as to form a touch screen display. In other implementations, the input device 290 may include separate keys or buttons on the imaging system 200. These keys or buttons may provide input for navigation of a menu that is displayed on the display 280. In other implementations, the input device 290 may be an input port. For example, the input device 290 may provide for operative coupling of another device to the imaging system 200. The imaging system 100 may then receive input from an attached keyboard or mouse via the input device 290. In still other embodiments, the input device 290 may be remotely located device and communicate with the imaging system 200 over a communication network, e.g., a wireless network.
The working memory 270 may be utilized by the processor 205 to store data dynamically created during operation of the imaging system 200. For example, instructions from any of the modules stored in the memory 220 (discussed below) may be stored in working memory 270 when executed by the processor 205. The working memory 270 may also store dynamic run time data, such as stack or heap data utilized by programs executing on processor 205. The storage 275 may be utilized to store data created by the imaging system 100. For example, images captured via image sensor 102 may be stored on storage 275. Like the input device 290, the storage 275 may also be located remotely, i.e., not integral with the imaging system 200, and may receive captured images via the communication network.
The memory 220 may be considered a computer readable media and stores several modules. The modules store data values defining instructions for processor 105. These instructions configure the processor 205 to perform functions of imaging system 200. For example, in some aspects, memory 220 may be configured to store instructions that cause the processor 205 to perform method 200, or portions thereof, as described below and as illustrated in
In another aspect, the lens control module 227 can include instructions that configure the processor 205 to receive position information of optical elements of the lens assembly 285, along with other parameters that may be used to move the lens assembly 285. Lens position information may include the current position of the lens, and the processor 205 may provide signals to the actuator 212 to position optical elements of the lens assembly in a desired position, for example, at a desired zoom level, at a desired focus position, or in a rest or standby position. In some aspects, instructions in the lens control module 227 may represent one means for determining current lens position and moving the lens to target lens position.
The classification module 230 may include instructions that configure the processor 205 to classify portions of an image. For example, the classification module 230 may be configured to classify a portion of an image as a subject, a background, an out-of-focus area, or a “flat” area (for example, that has a low dynamic range). Image processing technologies used to perform these classifications may include various techniques that are used to distinguish portions of an image. Classification algorithms are based on the assumption that the image in question depicts one or more features, e.g., geometric parts in the case of a manufacturing classification system, or spectral regions in the case of remote sensing, and that each of these features belongs to one of several distinct and exclusive classes. The classes may be specified a priori by an analyst (as in supervised classification) or automatically clustered (e.g., as in unsupervised classification) into sets of prototype classes, where the analyst merely specifies the number of desired categories. Classification and segmentation have closely related objectives, as the former is another form of component labeling that can result in segmentation of various features in a scene. In some embodiments, an algorithm based on a convolutional neural networks is used. An illustration of their capabilities is given by the ImageNet Large Scale Visual Recognition Challenge; this is a benchmark in object classification and detection, with millions of images and hundreds of object classes. Performance of convolutional neural networks, on the ImageNet tests, is now close to that of humans. In other embodiments, other classification techniques may be used. For example, dynamic range determination, e.g., to determine “flat” portions of an image, edge detection, e.g., to determine one or more portions of an image that are out-of-focus, and a combination of dynamic range determination and edge detections, e.g., to determine a portion of an image depicting background in the image). Therefore, instructions in the classification module 230 may be one means for determining portions of an image that depict a certain type of image data, for example, a subject, background, out-of-focus imagery, and/or flat imagery. In some embodiments, the classification process may be a specific classification algorithm implemented on a processor. For example, the Snapdragon processor has a computer vision and video analytics framework that may perform face detection, object detection, flat/detail detection to classify subject, non-subject (unimportant) areas.
The segmentation module 235 includes instructions that configures the processor 205 to segment portions of the classified image into, for example, a subject portion and a background portion, a subject portion and an out-of-focus portion, or a subject portion and a “flat” portion. The functionality of classifying and segmentation may be overlapping, based on the implementation. For example, in some techniques of classifying, after classification is completed the segmentation may also be completed due to the specific technique used for classification. In other words, although classification and segmentation may generally be described as two processes, whether it is described as one process or two processes the result is the segmenting of an image into portions that have been determined to contain different subject matter, for the purpose of controlling the sensor 102 to downscale one or more certain portions of the image data to reduce the amount of data provided by the sensor, which will reduce power consumption.
The segmentation refers to identifying portions of the image that are classified as either the background portion or the subject portion. In some embodiments, the segmentation using a region growing methodology based on classification determinations. One region-growing method is the seeded region growing method. This method takes a set of seeds as input along with the image. The seeds mark each of the objects to be segmented, for example, each of the portions classified by the classification module 230. Clustering techniques can also be used for segmentation, clustering portions of an image into larger contiguous portions. While downscaling image data captured by the image sensor 102 can reduce power needed for processing, there may be detrimental performance in the image chain processing, and particularly in the downscaling on the sensor 102 and upscaling performed later in the image processor 104, if too many segments are identified each requiring control information for the downscaling and upscaling. Accordingly, the segmentation functionality may segment the image into portions of a certain minimum size. Therefore, instructions in the segmentation module 235 may be one means for determining portions of an image sensor that will be controlled to downscale image data collected from those portions of the image sensor, based on a previously determined classification of image data that was previously collected from the image sensor 102 (for example, a previous frame of a video stream of the same target scene). Various image segmentation processes (or algorithms) may be used to perform this image segmentation. In computer vision, image segmentation is the process of partitioning a digital image into multiple segments (sets of pixels). This segmentation may change the representation of an image into something that is more meaningful or useful for a specific purpose, or and easier to analyze, for example, important areas and unimportant areas. One or more of many known methods may be used to perform this segmentation, including but not limited to thresholding techniques, edge-based methods, regions-based methods, or connectivity-preserving relaxation-based segmentation methods. For example, a region-based methods may include seeded region growing, unseeded region growing, regions splitting and merging, and hierarchical clustering. In some embodiments, an unsupervised watershed algorithm can also be used. The segmentation process may be a specific segmentation algorithm implemented on a processor. For example, the Snapdragon processor has a computer vision and video analytics framework that may perform face detection, object detection, flat/detail detection to identify subject, non-subject (unimportant) areas.
The input processing module 238 may include instructions that configure the processor 205 to read input data from the input device 290. In one aspect, input processing module 238 may configure the processor 205 to detect objects within an image captured by the image sensor 102. In another aspect, input processing module 230 may configure processor 205 to receive a user input from input device 290 and identify a user selection or configuration based on the user manipulation of input device 290. Therefore, instructions in the input processing module 230 may represent one means for identifying or selecting one or more objects within an image.
The autofocus module 240 includes instructions that configure the processor 205 to determine an in-focus position of the lens assembly 285. For example, in some embodiments the autofocus module 240 includes instructions to perform auto-focus operations by determining an in-focus position of the lens assembly 285 based on determining the position of the lens assembly 285 where the contrast of images captured by the image sensor 102 is at a maximum value.
The control module 260 may be configured to control the operations of one or more of the modules in memory 220.
The operating system module 265 includes instructions that configure the processor 205 to manage the hardware and software resources of the imaging system 200.
The content of the image 300 illustrated in
In a second example,
Still referring to
This rectangle information may be provided to the image sensor 102 in any format necessary such that the image sensor 102 performs downscaling in the determined portions of the image data, in response to receiving the information. For example, information representative of the rectangles can be provided in the following representative format:
For example: for two rectangles, rectangle 905 with vertical downscaling at 2×, and rectangle 910 with vertical downscaling at 4×, a representative format may be:
A person of ordinary skill in the art will appreciate that the above format is merely representative of a possible format, many different types of formats may be used for the sensor control information, and the format of the sensor control information may depend on the configuration of the sensor 102.
Implementations disclosed herein provide systems, methods and apparatus for compact stereoscopic imaging systems. One skilled in the art will recognize that these embodiments may be implemented in hardware, software, firmware, or any combination thereof.
In some embodiments, the circuits, processes, and systems discussed above may be utilized in a wireless communication device. The wireless communication device may be a kind of electronic device used to wirelessly communicate with other electronic devices. Examples of wireless communication devices include cellular telephones, smart phones, Personal Digital Assistants (PDAs), e-readers, gaming systems, music players, netbooks, wireless modems, laptop computers, tablet devices, etc.
The wireless communication device may include one or more image sensors, one or more image signal processors, a memory including instructions or modules for carrying out the processes discussed above. The device may also have data, a processor loading instructions and/or data from memory, one or more communication interfaces, one or more input devices, one or more output devices such as a display device and a power source/interface. The wireless communication device may additionally include a transmitter and a receiver. The transmitter and receiver may be jointly referred to as a transceiver. The transceiver may be coupled to one or more antennas for transmitting and/or receiving wireless signals.
The wireless communication device may wirelessly connect to another electronic device (e.g., base station). Examples of wireless communication devices include laptop or desktop computers, cellular phones, smart phones, wireless modems, e-readers, tablet devices, gaming systems, etc. Wireless communication devices may operate in accordance with one or more industry standards such as the 3rd Generation Partnership Project (3GPP). Thus, the general term “wireless communication device” may include wireless communication devices described with varying nomenclatures according to industry standards.
The functions described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term “computer-readable medium” refers to any available medium that can be accessed by a computer or processor. By way of example, and not limitation, such a medium may comprise RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. It should be noted that a computer-readable medium may be tangible and non-transitory. The term “computer-program product” refers to a computing device or processor in combination with code or instructions (e.g., a “program”) that may be executed, processed or computed by the computing device or processor. As used herein, the term “code” may refer to software, instructions, code or data that is/are executable by a computing device or processor.
The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein can be implemented or performed by a machine, such as a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be a controller, microcontroller, or state machine, combinations of the same, or the like. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, a processor may also include primarily analog components. For example, any of the signal processing algorithms described herein may be implemented in analog circuitry. A computing environment can include any type of computer system, including, but not limited to, a computer system based on a microprocessor, a mainframe computer, a digital signal processor, a portable computing device, a personal organizer, a device controller, and a computational engine within an appliance, to name a few.
The methods disclosed herein comprise one or more steps or actions for achieving the described method. The method steps and/or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is required for proper operation of the method that is being described, the order and/or use of specific steps and/or actions may be modified without departing from the scope of the claims.
It should be noted that the terms “couple,” “coupling,” “coupled” or other variations of the word couple as used herein may indicate either an indirect connection or a direct connection. For example, if a first component is “coupled” to a second component, the first component may be either indirectly connected to the second component or directly connected to the second component. As used herein, the term “plurality” denotes two or more. For example, a plurality of components indicates two or more components.
The term “determining” encompasses a wide variety of actions and, therefore, “determining” can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” can include resolving, selecting, choosing, establishing and the like. The phrase “based on” does not mean “based only on,” unless expressly specified otherwise. In other words, the phrase “based on” describes both “based only on” and “based at least on.”
The previous description of the disclosed implementations is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these implementations will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
