Patent Application
20170206642
Thermal imaging cameras are used in a variety of situations. For example, thermal imaging cameras are often used during maintenance inspections to thermally inspect equipment. Example equipment may include rotating machinery, electrical panels, or rows of circuit breakers, among other types of equipment. Thermal inspections can detect equipment hot spots such as overheating machinery or electrical components, helping to ensure timely repair or replacement of the overheating equipment before a more significant problem develops.
Traditional images are captured using a single focus distance. This can lead to the misfocus of certain objects in a target scene if there is a wide range of distances between the camera and the objects in the scene. In the case of infrared imaging, an out of focus target can result in less accurate temperature measurement. Moreover, while comparing temperature or heat signature of your target component with other identical components in the field of view, it is important to ensure all the components are completely in focus for accurate comparisons.
To address this issue, a user must take multiple images of a scene with each image having different targets in focus. Then the user must view each image separately to accurately analyze components of the image that may be positioned at a variety of depths. This can make full scene analysis overly time-consuming and cumbersome. Additionally, acquiring such images for comparison, particularly algorithmic based comparison, can be challenging for imagery taken with a hand-held camera, since images can be displaced from one another due to hand-jitter and the like.
Aspects of the disclosure are directed toward systems and methods for combining aspects of a plurality of images taken at different focus distances in order to generate a final image having objects at a variety of depths remain in focus.
In some instances, a plurality of indexed images captured at a variety of different focus distances can be registered to one another. High frequency frames representative of the high frequency content in corresponding ones of the plurality of images can be generated, each of the high frequency frames being indexed and having like index as its corresponding image. In some examples, each of the images and corresponding high frequency frames can include a plurality of regions such that each region in a high frequency frame corresponds to a region in a corresponding image. In some such examples, a region in a high frequency frame is representative of the high frequency content of the like region in the corresponding image.
In some examples, methods can include the step of determining, for each of a plurality of regions in each of the high frequency frames, if the high frequency content in that region is valid. In some such examples, methods can further include determining, for each of the plurality of regions, the index of the high frequency frame having the greatest valid high frequency content among the plurality of high frequency frames. Methods can further include generating a final image that comprises, for each of the plurality of regions, the image data from the one of the plurality of images having the index corresponding to the index of the high frequency frame having the greatest valid high frequency content.
In various examples, images in the plurality of images can include infrared images, visible light images, or combined infrared and visible light images. Similarly, the final image can comprise image data from infrared, visible light, or combined infrared and visible light images.
In some examples determining whether or not the high frequency content in a region of a high frequency frame is valid comprises comparing the high frequency content of that region to the high frequency content of nearby regions. Nearby regions can include regions within the high frequency frame and/or within frames having an index within a threshold distance of the region such that the frames having such indices correspond to images having focus distances near the focus distance of the image corresponding to the high frequency frame containing the region. Comparing the high frequency content to nearby high frequency content can include determining whether or not the high frequency content is within some threshold value of the maximum high frequency content. Additionally or alternatively, determining whether or not high frequency content is valid comprises comparing the high frequency content to a predetermined validity threshold.
In some examples, methods can include generating a best focus frame, wherein the best focus frame comprises, for each region, the index of the high frequency frame having the greatest valid high frequency content in that region. The best focus frame can be used as a guide for populating each region in the final image with image data from the image having the index designated in that region of the best focus frame.
In various embodiments, methods can include capturing the plurality of images using an imaging device and/or recalling images from a memory. Capturing the plurality of images can include adjusting a focus distance of an imaging device while periodically capturing images so that each of the captured images has a different focus distance.
Embodiments of this disclosure are directed toward cameras or other imaging systems capable of performing such methods.
The following detailed description is exemplary in nature and is not intended to limit the scope, applicability, or configuration of the invention in any way. Rather, the following description provides some practical illustrations for implementing various embodiments of the present invention. Examples of constructions, materials, dimensions, and manufacturing processes are provided for selected elements, and all other elements employ that which is known to those of ordinary skill in the field of the invention. Those skilled in the art will recognize that many of the noted examples have a variety of suitable alternatives.
A thermal imaging camera may be used to detect heat patterns across a scene, including an object or objects, under observation. The thermal imaging camera may detect infrared radiation given off by the scene and convert the infrared radiation into an infrared image indicative of the heat patterns. In some embodiments, the thermal imaging camera may also capture visible light from the scene and convert the visible light into a visible light image. Depending on the configuration of the thermal imaging camera, the camera may include infrared optics to focus the infrared radiation on an infrared sensor and visible light optics to focus the visible light on a visible light sensor.
Various embodiments provide methods and systems for producing thermal images with reduced noise using averaging techniques. To further improve image quality and eliminate problems that may arise from averaging (e.g. blurring, ghosting, etc.), an image alignment process is performed on the thermal images prior to averaging.
In operation, thermal imaging camera 100 detects heat patterns in a scene by receiving energy emitted in the infrared-wavelength spectrum from the scene and processing the infrared energy to generate a thermal image. Thermal imaging camera 100 may also generate a visible light image of the same scene by receiving energy in the visible light-wavelength spectrum and processing the visible light energy to generate a visible light image. As described in greater detail below, thermal imaging camera 100 may include an infrared camera module that is configured to capture an infrared image of the scene and a visible light camera module that is configured to capture a visible light image of the same scene. The infrared camera module may receive infrared radiation projected through infrared lens assembly 104 and generate therefrom infrared image data. The visible light camera module may receive light projected through visible light lens assembly 106 and generate therefrom visible light data.
In some examples, thermal imaging camera 100 collects or captures the infrared energy and visible light energy substantially simultaneously (e.g., at the same time) so that the visible light image and the infrared image generated by the camera are of the same scene at substantially the same time. In these examples, the infrared image generated by thermal imaging camera 100 is indicative of localized temperatures within the scene at a particular period of time while the visible light image generated by the camera is indicative of the same scene at the same period of time. In other examples, thermal imaging camera may capture infrared energy and visible light energy from a scene at different periods of time.
Visible light lens assembly 106 includes at least one lens that focuses visible light energy on a visible light sensor for generating a visible light image. Visible light lens assembly 106 defines a visible light optical axis which passes through the center of curvature of the at least one lens of the assembly. Visible light energy projects through a front of the lens and focuses on an opposite side of the lens. Visible light lens assembly 106 can include a single lens or a plurality of lenses (e.g., two, three, or more lenses) arranged in series. In addition, visible light lens assembly 106 can have a fixed focus or can include a focus adjustment mechanism for changing the focus of the visible light optics. In examples in which visible light lens assembly 106 includes a focus adjustment mechanism, the focus adjustment mechanism may be a manual adjustment mechanism or an automatic adjustment mechanism.
Infrared lens assembly 104 also includes at least one lens that focuses infrared energy on an infrared sensor for generating a thermal image. Infrared lens assembly 104 defines an infrared optical axis which passes through the center of curvature of lens of the assembly. During operation, infrared energy is directed through the front of the lens and focused on an opposite side of the lens. Infrared lens assembly 104 can include a single lens or a plurality of lenses (e.g., two, three, or more lenses), which may be arranged in series. In some examples, the infrared lens assembly 104 may include lenses having diffractive or reflective properties or elements. Additional optical components such as mirrors (e.g., Fresnel mirrors) and the like may be included within or otherwise proximate to the infrared lens assembly 104.
As briefly described above, thermal imaging camera 100 includes a focus mechanism for adjusting the focus of an infrared image captured by the camera. In the example shown in
In some examples, thermal imaging camera 100 may include an automatically adjusting focus mechanism in addition to or in lieu of a manually adjusting focus mechanism. An automatically adjusting focus mechanism may be operatively coupled to at least one lens of infrared lens assembly 104 and configured to automatically move the at least one lens to various focus positions, e.g., in response to instructions from thermal imaging camera 100. In one application of such an example, thermal imaging camera 100 may use laser 110 to electronically measure a distance between an object in a target scene and the camera, referred to as the distance-to-target. Thermal imaging camera 100 may then control the automatically adjusting focus mechanism to move the at least one lens of infrared lens assembly 104 to a focus position that corresponds to the distance-to-target data determined by thermal imaging camera 100. The focus position may correspond to the distance-to-target data in that the focus position may be configured to place the object in the target scene at the determined distance in focus. In some examples, the focus position set by the automatically adjusting focus mechanism may be manually overridden by an operator, e.g., by rotating focus ring 114.
During operation of thermal imaging camera 100, an operator may wish to view a thermal image of a scene and/or a visible light image of the same scene generated by the camera. For this reason, thermal imaging camera 100 may include a display. In the examples of
Thermal imaging camera 100 can include a variety of user input media for controlling the operation of the camera and adjusting different settings of the camera. Example control functions may include adjusting the focus of the infrared and/or visible light optics, opening/closing a shutter, capturing an infrared and/or visible light image, or the like. In the example of
Infrared camera module 200 may be configured to receive infrared energy emitted by a target scene and to focus the infrared energy on an infrared sensor for generation of infrared energy data, e.g., that can be displayed in the form of an infrared image on display 108 and/or stored in memory. Infrared camera module 200 can include any suitable components for performing the functions attributed to the module herein. In the example of
Infrared sensor 220 may include one or more focal plane arrays (FPA) that generate electrical signals in response to infrared energy received through infrared lens assembly 104. Each FPA can include a plurality of infrared sensor elements including, e.g., bolometers, photon detectors, or other suitable infrared sensor elements. In operation, each sensor element, which may each be referred to as a sensor pixel, may change an electrical characteristic (e.g., voltage or resistance) in response to absorbing infrared energy received from a target scene. In turn, the change in electrical characteristic can provide an electrical signal that can be received by a processor 222 and processed into an infrared image displayed on display 108.
For instance, in examples in which infrared sensor 220 includes a plurality of bolometers, each bolometer may absorb infrared energy focused through infrared lens assembly 104 and increase in temperature in response to the absorbed energy. The electrical resistance of each bolometer may change as the temperature of the bolometer changes. With each detector element functioning as a sensor pixel, a two-dimensional image or picture representation of the infrared radiation can be further generated by translating the changes in resistance of each detector element into a time-multiplexed electrical signal that can be processed for visualization on a display or storage in memory (e.g., of a computer). Processor 222 may measure the change in resistance of each bolometer by applying a current (or voltage) to each bolometer and measure the resulting voltage (or current) across the bolometer. Based on these data, processor 222 can determine the amount of infrared energy emitted by different portions of a target scene and control display 108 to display a thermal image of the target scene.
Independent of the specific type of infrared sensor elements included in the FPA of infrared sensor 220, the FPA array can define any suitable size and shape. In some examples, infrared sensor 220 includes a plurality of infrared sensor elements arranged in a grid pattern such as, e.g., an array of sensor elements arranged in vertical columns and horizontal rows. In various examples, infrared sensor 220 may include an array of vertical columns by horizontal rows of, e.g., 16×16, 50×50, 160×120, 120×160, or 650×480. In other examples, infrared sensor 220 may include a smaller number of vertical columns and horizontal rows (e.g., 1×1), a larger number vertical columns and horizontal rows (e.g., 1000×1000), or a different ratio of columns to rows.
In certain embodiments a Read Out Integrated Circuit (ROIC) is incorporated on the IR sensor 220. The ROIC is used to output signals corresponding to each of the sensor pixels. Such ROIC is commonly fabricated as an integrated circuit on a silicon substrate. The plurality of detector elements may be fabricated on top of the ROIC, wherein their combination provides for the IR sensor 220. In some embodiments, the ROIC can include components discussed elsewhere in this disclosure (e.g. an analog-to-digital converter (ADC)) incorporated directly onto the FPA circuitry. Such integration of the ROIC, or other further levels of integration not explicitly discussed, should be considered within the scope of this disclosure.
As described above, the IR sensor 220 generates a series of electrical signals corresponding to the infrared radiation received by each infrared detector element to represent a thermal image. A “frame” of thermal image data is generated when the voltage signal from each infrared detector element is obtained by scanning all of the rows that make up the IR sensor 220. Again, in certain embodiments involving bolometers as the infrared detector elements, such scanning is done by switching a corresponding detector element into the system circuit and applying a bias voltage across such switched-in element. Successive frames of thermal image data are generated by repeatedly scanning the rows of the IR sensor 220, with such frames being produced at a rate sufficient to generate a video representation (e.g. 30 Hz, or 60 Hz) of the thermal image data.
The front end circuitry 202 includes circuitry for interfacing with and controlling the IR camera module 200. In addition, the front end circuitry 202 initially processes and transmits collected infrared image data to a processor 222 via a connection therebetween. More specifically, the signals generated by the IR sensor 220 are initially conditioned by the front end circuitry 202 of the thermal imaging camera 100. In certain embodiments, as shown, the front end circuitry 202 includes a bias generator 224 and a pre-amp/integrator 226. In addition to providing the detector bias, the bias generator 224 can optionally add or subtract an average bias current from the total current generated for each switched-in detector element. The average bias current can be changed in order (i) to compensate for deviations to the entire array of resistances of the detector elements resulting from changes in ambient temperatures inside the thermal imaging camera 100 and (ii) to compensate for array-to-array variations in the average detector elements of the IR sensor 220. Such bias compensation can be automatically controlled by the thermal imaging camera 100 or software, or can be user controlled via input to the output/control device 210 or processor 222. Following provision of the detector bias and optional subtraction or addition of the average bias current, the signals can be passed through a pre-amp/integrator 226. Typically, the pre-amp/integrator 226 is used to condition incoming signals, e.g., prior to their digitization. As a result, the incoming signals can be adjusted to a form that enables more effective interpretation of the signals, and in turn, can lead to more effective resolution of the created image. Subsequently, the conditioned signals are sent downstream into the processor 222 of the thermal imaging camera 100.
In some embodiments, the front end circuitry 202 can include one or more additional elements for example, additional sensors 228 or an ADC 230. Additional sensors 228 can include, for example, temperature sensors, visual light sensors (such as a CCD), pressure sensors, magnetic sensors, etc. Such sensors can provide additional calibration and detection information to enhance the functionality of the thermal imaging camera 100. For example, temperature sensors can provide an ambient temperature reading near the IR sensor 220 to assist in radiometry calculations. A magnetic sensor, such as a Hall Effect sensor, can be used in combination with a magnet mounted on the lens to provide lens focus position information. Such information can be useful for calculating distances, or determining a parallax offset for use with visual light scene data gathered from a visual light sensor.
An ADC 230 can provide the same function and operate in substantially the same manner as discussed below, however its inclusion in the front end circuitry 202 may provide certain benefits, for example, digitization of scene and other sensor information prior to transmittal to the processor 222 via the connection therebetween. In some embodiments, the ADC 230 can be integrated into the ROIC, as discussed above, thereby eliminating the need for a separately mounted and installed ADC 230.
In some embodiments, front end components can further include a shutter 240. A shutter 240 can be externally or internally located relative to the lens and operate to open or close the view provided by the IR lens assembly 104. As is known in the art, the shutter 240 can be mechanically positionable, or can be actuated by an electro-mechanical device such as a DC motor or solenoid. Embodiments of the invention may include a calibration or setup software implemented method or setting which utilize the shutter 240 to establish appropriate bias levels for each detector element.
Components described as processors within thermal imaging camera 100, including processor 222, may be implemented as one or more processors, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic circuitry, or the like, either alone or in any suitable combination. Processor 222 may also include memory that stores program instructions and related data that, when executed by processor 222, cause thermal imaging camera 100 and processor 222 to perform the functions attributed to them in this disclosure. Memory may include any fixed or removable magnetic, optical, or electrical media, such as RAM, ROM, CD-ROM, hard or floppy magnetic disks, EEPROM, or the like. Memory may also include a removable memory portion that may be used to provide memory updates or increases in memory capacities. A removable memory may also allow image data to be easily transferred to another computing device, or to be removed before thermal imaging camera 100 is used in another application. Processor 222 may also be implemented as a System on Chip that integrates some or all components of a computer or other electronic system into a single chip. These elements manipulate the conditioned scene image data delivered from the front end stages 204 in order to provide output scene data that can be displayed or stored for use by the user. Subsequently, the processor 222 (processing circuitry) sends the processed data to a display 108 or other output/control device 210.
During operation of thermal imaging camera 100, processor 222 can control infrared camera module 200 to generate infrared image data for creating an infrared image. Processor 222 can generate a digital “frame” of infrared image data. By generating a frame of infrared image data, processor 222 captures an infrared image of a target scene at substantially a given point in time. That is, in some examples, a plurality of pixels making up the infrared image may be captured simultaneously. In other embodiments, sets of one or more pixels may be captured serially until each pixel has been captured.
Processor 222 can capture a single infrared image or “snap shot” of a target scene by measuring the electrical signal of each infrared sensor element included in the FPA of infrared sensor 220 a single time. Alternatively, processor 222 can capture a plurality of infrared images of a target scene by repeatedly measuring the electrical signal of each infrared sensor element included in the FPA of infrared sensor 220. In examples in which processor 222 repeatedly measures the electrical signal of each infrared sensor element included in the FPA of infrared sensor 220, processor 222 may generate a dynamic thermal image (e.g., a video representation) of a target scene. For example, processor 222 may measure the electrical signal of each infrared sensor element included in the FPA at a rate sufficient to generate a video representation of thermal image data such as, e.g., 30 Hz or 60 Hz. Processor 222 may perform other operations in capturing an infrared image such as sequentially actuating a shutter 240 to open and close an aperture of infrared lens assembly 104, or the like.
With each sensor element of infrared sensor 220 functioning as a sensor pixel, processor 222 can generate a two-dimensional image or picture representation of the infrared radiation from a target scene by translating changes in an electrical characteristic (e.g., resistance) of each sensor element into a time-multiplexed electrical signal that can be processed, e.g., for visualization on display 108 and/or storage in memory. When displayed on a display 108, an infrared image can comprise a plurality of display pixels. Display pixels can have any defined relationship with corresponding sensor pixels. In some examples, each sensor pixel corresponds to a display pixel in an image representation of infrared data. In other examples, a plurality of sensor pixels may be combined (e.g., averaged) to provide infrared information for a single display pixel. In still other examples, a single sensor pixel may contribute to a plurality of display pixels. For example, a value from a single sensor pixel may be replicated at nearby pixels, such as in a simple upsampling procedure. In other examples, neighboring or otherwise nearby pixels may be averaged to create a new pixel value, such as in an interpolation procedure. Because relationships between display pixels and sensor pixels are defined with respect to camera operation, the generic term “pixel” may refer to the sensor pixel, the display pixel, or the data as it is processed from the sensor pixel to the display pixel unless otherwise stated. Processor 222 may perform computations to convert raw infrared image data into scene temperatures (radiometry) including, in some examples, colors corresponding to the scene temperatures.
Processor 222 may control display 108 to display at least a portion of an infrared image of a captured target scene. In some examples, processor 222 controls display 108 so that the electrical response of each sensor element of infrared sensor 220 is associated with a single pixel on display 108. In other examples, processor 222 may increase or decrease the resolution of an infrared image so that there are more or fewer pixels displayed on display 108 than there are sensor elements in infrared sensor 220. Processor 222 may control display 108 to display an entire infrared image (e.g., all portions of a target scene captured by thermal imaging camera 100) or less than an entire infrared image (e.g., a lesser port of the entire target scene captured by thermal imaging camera 100). Processor 222 may perform other image processing functions, as described in greater detail below.
Independent of the specific circuitry, thermal imaging camera 100 may be configured to manipulate data representative of a target scene so as to provide an output that can be displayed, stored, transmitted, or otherwise utilized by a user.
Thermal imaging camera 100 includes visible light camera module 206. Visible light camera modules are generally well known. For examples, various visible light camera modules are included in smartphones and numerous other devices. In some embodiments, visible light camera module 206 may be configured to receive visible light energy from a target scene and to focus the visible light energy on a visible light sensor for generation of visible light energy data, e.g., that can be displayed in the form of a visible light image on display 108 and/or stored in memory. Visible light camera module 206 can include any suitable components for performing the functions attributed to the module herein. In the example of
Visible light sensor 242 may include a plurality of visible light sensor elements such as, e.g., CMOS detectors, CCD detectors, PIN diodes, avalanche photo diodes, or the like. The number of visible light sensor elements may be the same as or different than the number of infrared light sensor elements.
In operation, optical energy received from a target scene may pass through visible light lens assembly 106 and be focused on visible light sensor 242. When the optical energy impinges upon the visible light sensor elements of visible light sensor 242, photons within the photodetectors may be released and converted into a detection current. Processor 222 can process this detection current to form a visible light image of the target scene.
During use of thermal imaging camera 100, processor 222 can control visible light camera module 206 to generate visible light data from a captured target scene for creating a visible light image. The visible light data may include luminosity data indicative of the color(s) associated with different portions of the captured target scene and/or the magnitude of light associated with different portions of the captured target scene. Processor 222 can generate a “frame” of visible light image data by measuring the response of each visible light sensor element of thermal imaging camera 100 a single time. By generating a frame of visible light data, processor 222 captures visible light image of a target scene at a given point in time. Processor 222 may also repeatedly measure the response of each visible light sensor element of thermal imaging camera 100 so as to generate a dynamic thermal image (e.g., a video representation) of a target scene, as described above with respect to infrared camera module 200. In some examples, the visible light camera module 206 may include its own dedicated processor or other circuitry (e.g., ASIC) capable of operating the visible light camera module 206. In some such embodiments, the dedicated processor is in communication with processor 222 for providing visible light image data (e.g., RGB image data) to processor 222. In alternative embodiments, a dedicated processor for the visible light camera module 206 may be integrated into processor 222.
With each sensor element of visible light camera module 206 functioning as a sensor pixel, processor 222 can generate a two-dimensional image or picture representation of the visible light from a target scene by translating an electrical response of each sensor element into a time-multiplexed electrical signal that can be processed, e.g., for visualization on display 108 and/or storage in memory.
Processor 222 may control display 108 to display at least a portion of a visible light image of a captured target scene. In some examples, processor 222 controls display 108 so that the electrical response of each sensor element of visible light camera module 206 is associated with a single pixel on display 108. In other examples, processor 222 may increase or decrease the resolution of a visible light image so that there are more or fewer pixels displayed on display 108 than there are sensor elements in visible light camera module 206. Processor 222 may control display 108 to display an entire visible light image (e.g., all portions of a target scene captured by thermal imaging camera 100) or less than an entire visible light image (e.g., a lesser port of the entire target scene captured by thermal imaging camera 100).
In some embodiments, one or both of infrared 200 and visible light 206 camera modules for acquiring IR and VL image data may be included in an image acquisition module 280. The image acquisition module may be in wired or wireless communication with a processing module 290 that includes a processor such as 222. Processing module 290 may receive image data from the image acquisition module 280 and perform subsequent processing steps as will be described herein. In some examples, processing module 290 may include portable processing devices, such as a smartphone, a tablet, a stand-alone computer such as a laptop or desktop PC, or the like. In some such embodiments, various components of front end circuitry 202 may be included in the image acquisition module 280, the processing module 290, or both.
In these and other examples, processor 222 may control display 108 to concurrently display at least a portion of the visible light image captured by thermal imaging camera 100 and at least a portion of the infrared image captured by thermal imaging camera 100. Such a concurrent display may be useful in that an operator may reference the features displayed in the visible light image to help understand the features concurrently displayed in the infrared image, as the operator may more easily recognize and distinguish different real-world features in the visible light image than the infrared image. In various examples, processor 222 may control display 108 to display the visible light image and the infrared image in side-by-side arrangement, in a picture-in-picture arrangement, where one of the images surrounds the other of the images, or any other suitable arrangement where the visible light and the infrared image are concurrently displayed.
For example, processor 222 may control display 108 to display the visible light image and the infrared image in a combined arrangement. In such an arrangement, for a pixel or set of pixels in the visible light image representative of a portion of the target scene, there exists a corresponding pixel or set of pixels in the infrared image, representative of substantially the same portion of the target scene. In various embodiments, the size and/or resolution of the IR and VL images need not be the same. Accordingly, there may exist a set of pixels in one of the IR or VL images that correspond to a single pixel in the other of the IR or VL image, or a set of pixels of a different size. Similarly, there may exist a pixel in one of the VL or IR images that corresponds to a set of pixels in the other image. Thus, as used herein, corresponding does not require a one-to-one pixel relationship, but may include mismatched sizes of pixels or groups of pixels. Various combination techniques of mismatched sized regions of images may be performed, such as up- or down-sampling one of the images, or combining a pixel with the average value of a corresponding set of pixels. Other examples are known and are within the scope of this disclosure.
Thus, corresponding pixels need not have a direct one-to-one relationship. Rather, in some embodiments, a single infrared pixel has a plurality of corresponding visible light pixels, or a visible light pixel has a plurality of corresponding infrared pixels. Additionally or alternatively, in some embodiments, not all visible light pixels have corresponding infrared pixels, or vice versa. Such embodiments may be indicative of, for example, a picture-in-picture type display as previously discussed. Thus, a visible light pixel will not necessarily have the same pixel coordinate within the visible light image as does a corresponding infrared pixel. Accordingly, as used herein, corresponding pixels generally refers pixels from any image (e.g., a visible light image, an infrared image, a combined image, a display image, etc.) comprising information from substantially the same portion of the target scene. Such pixels need not have a one-to-one relationship between images and need not have similar coordinate positions within their respective images.
Similarly, images having corresponding pixels (i.e., pixels representative of the same portion of the target scene) can be referred to as corresponding images. Thus, in some such arrangements, the corresponding visible light image and the infrared image may be superimposed on top of one another, at corresponding pixels. An operator may interact with user interface 208 to control the transparency or opaqueness of one or both of the images displayed on display 108. For example, the operator may interact with user interface 208 to adjust the infrared image between being completely transparent and completely opaque and also adjust the visible light image between being completely transparent and completely opaque. Such an exemplary combined arrangement, which may be referred to as an alpha-blended arrangement, may allow an operator to adjust display 108 to display an infrared-only image, a visible light-only image, of any overlapping combination of the two images between the extremes of an infrared-only image and a visible light-only image. Processor 222 may also combine scene information with other data, such as radiometric data, alarm data, and the like. In general, an alpha-blended combination of visible light and infrared images can comprise anywhere from 100 percent infrared and 0 percent visible light to 0 percent infrared and 100 percent visible light. In some embodiments, the amount of blending can be adjusted by a user of the camera. Thus, in some embodiments, a blended image can be adjusted between 100 percent visible light and 100 percent infrared.
Additionally, in some embodiments, the processor 222 can interpret and execute commands from user interface 208, and/or output/control device 210. This can involve processing of various input signals and transferring those signals to the front end circuitry 202 via a connection therebetween. Components (e.g. motors, or solenoids) proximate the front end circuitry 202 can be actuated to accomplish the desired control function. Exemplary control functions can include adjusting the focus (e.g., via motorized motion of one or more lenses), opening/closing a shutter, triggering sensor readings, adjusting bias values, etc. Moreover, input signals may be used to alter the processing of the image data that occurs in the processor 222.
Processor can further include other components to assist with the processing and control of the infrared imaging camera 100. For example, as discussed above, in some embodiments, an ADC can be incorporated into the processor 222. In such a case, analog signals conditioned by the front-end stages 204 are not digitized until reaching the processor 222. Moreover, some embodiments can include additional on board memory for storage of processing command information and scene data, prior to transmission to the display 108 or the output/control device 210.
An operator may interact with thermal imaging camera 100 via user interface 208, which may include buttons, keys, or another mechanism for receiving input from a user. The operator may receive output from thermal imaging camera 100 via display 108. Display 108 may be configured to display an infrared-image and/or a visible light image in any acceptable palette, or color scheme, and the palette may vary, e.g., in response to user control. In some examples, display 108 is configured to display an infrared image in a monochromatic palette such as grayscale. In other examples, display 108 is configured to display an infrared image in a color palette such as, e.g., amber, ironbow, blue-red, or other high contrast color scheme. Combinations of grayscale and color palette displays are also contemplated. In some examples, the display being configured to display such information may include processing capabilities for generating and presenting such image data. In other examples, being configured to display such information may include the ability to receive image data from other components, such as processor 222. For example, processor 222 may generate values (e.g., RGB values, grayscale values, or other display options) for each pixel to be displayed. Display 108 may receive such information and map each pixel into a visual display.
While processor 222 can control display 108 to concurrently display at least a portion of an infrared image and at least a portion of a visible light image in any suitable arrangement, a picture-in-picture arrangement may help an operator to easily focus and/or interpret a thermal image by displaying a corresponding visible image of the same scene in adjacent alignment.
A power supply (not shown) delivers operating power to the various components of thermal imaging camera 100 and, in some examples, may include a rechargeable or non-rechargeable battery and a power generation circuit.
During operation of thermal imaging camera 100, processor 222 controls infrared camera module 200 and visible light camera module 206 with the aid of instructions associated with program information that is stored in memory to generate a visible light image and an infrared image of a target scene. Processor 222 further controls display 108 to display the visible light image and/or the infrared image generated by thermal imaging camera 100.
In some instances during operation, a target scene can include objects located at a variety of distances from the camera. Moreover, such objects may be objects of interest to a user capturing the image and it is desirable to have all of such objects in focus within a captured image. This allows each object of interest to be more accurately analyzed. However, in some such instances, the differences in distance to the multiple objects may be too large to reasonably focus the camera so that each object is in focus, and acquiring multiple images may be necessary.
As previously described, in some examples, a camera can be configured to capture a plurality of images. Such images can be captured at a variety of intervals, such as the camera's frame rate or some other interval, such as one that is selectable by a user. In alternative embodiments, images can be captured at irregular intervals, such as whenever the camera receives a prompt from a user to capture an image (e.g., via depressing trigger control 112).
Additionally, in some instances, the camera can be used to capture a plurality of images while simultaneously adjusting the focus of the camera. For example, the camera can be used to acquire a plurality of images while a motor drives one or more lenses (e.g., in a lens assembly), thereby adjusting the focus of the camera. In other examples, the camera can be used to acquire a plurality of images while a user manually adjusts the position of one or more lenses to adjust the focus of the camera, such as by actuating focus ring 114.
The resulting set of images comprises a plurality of images of a target scene having different focus distances associated therewith. Depending on the contents of the target scene, different objects may appear to be in-focus in some images and out of focus in other images having a significantly different focus distance. Accordingly, an object of interest located at a first depth might be in focus in one image, while an object of interest located at a second depth might be in focus in a different image. Such images can be combined in order to form an image in which both objects are in focus.
In some embodiments, during motion of the lens assembly, a plurality of images can be acquired. In the illustrated embodiment, markers 402 indicate times during the lens motion at which images are captured. As previously discussed, markers 402 may coincide with the frame rate of the camera, an adjustable image acquisition rate, or with manual image capture commands from a user. In some examples, markers 402 may coincide with a subset of images, for example, every nth frame of the original framerate. In some such examples, the camera may discard unused frames.
Since the lens is moving (i.e., Velocity≠0), between times t0 and t3, images captured at each of markers 402 will each have a different focus distance associated therewith. The images captured at markers 402 can be indexed according to capture order. For instance, the first image taken at t0 can be indexed as Image 1, the second image indexed as Image 2, and so on. In some examples, the time at which the image was taken during the lens movement process can be recorded. For example, an image taken at t1 can be associated with time t1 and likewise with other images.
Additionally or alternatively, focus distance information corresponding to the focus distance at which each image is captured can be recorded and associated with respective images. Focus distance information can include the actual focus distance of the camera and/or the position of the lens assembly within the camera when the image is captured. In some embodiments, a camera can determine the focus distance information based on a measured position of the lens assembly at the time of capturing an image, for example, via position sensors or the like. Additionally or alternatively, the camera can use parameters such as lens assembly motion information to determine focus distance information. For example, referring to the velocity profile of
In various examples, motion of the lens assembly can be automated or manually driven via a motor, or manually moved via mechanical actuation of a component by a user. Various properties such as position and/or velocity of the motion can be determined in order to determine focus distance information associated with each captured image. In some embodiments, a camera may perform an automated process to move the lens assembly and acquire images automatically. In some such embodiments, the lens assembly is moved to an initial location (e.g., a nearest focal position), and moved through a range of positions to a final location (e.g., a furthest focal position) while acquiring images during the motion of the lens assembly. Various aspects may be adjustable by a user, such as a starting position, an ending position, a movement velocity, an image capture rate, and the like. As described above, various parameters, such as an image index, acquisition time, and/or focus distance information can be recorded and associated with the acquired images. Additionally, the execution of an image acquisition scheme such as described can be initiated by a camera operator.
As described above, alternatively to automated image capture, any other methods for acquiring a plurality of images taken at a plurality of focus distances may be performed. For instance, any steps of adjusting the camera focus and acquiring images may be performed manually and not according to any particular prescribed schedule. In some such examples, associated time and/or focus distance information can be stored with captured images. Images can be indexed at the time of image capturing or after the images have been captured.
Once a plurality of indexed images of a target scene having different focus distance characteristics have been captured, such images can be processed and combined to create a single image as described above. Such an image can include data from the plurality of images so that portions of the scene located at different distances from the camera may be presented in focus. Such image processing and combining can be performed whether or not the images are captured according to any particular prescribed acquisition technique.
In an exemplary process, each image of the plurality of indexed images is processed to generate a corresponding high frequency frame (HF Frame). As shown, Image 1 is processed to generate HF Frame 1, Image 2 is processed to generate HF Frame 2, etc. Thus, N images are processed to create a corresponding N high frequency frames. In general, each high frequency frame is representative of the high frequency content at corresponding locations in its corresponding image. For instance, a given region in HF Frame 1 is representative of the amount of high frequency content in a corresponding region in Image 1. In various embodiments, regions can correspond to pixels, sets of pixels, or other sections in an image.
Generating a high frequency frame from an image can be performed in a number of ways. In some examples, different edge detection/enhancement techniques, such as those described in U.S. patent application Ser. No. 14/837,757, filed Aug. 27, 2014, and U.S. patent applicant Ser. No. 14/222,153, filed Mar. 21, 2014, each of which is assigned to the assignee of the instant application and is hereby incorporated by reference in its entirety, may be used to generate a high frequency frame, such as a Sobel Operator, a Roberts Cross Operator, or other known operations. Other ways of determining the high frequency content of various regions of an image are possible. In general, each high frequency frame will include regions comprising a value representative of the high frequency content of corresponding regions in the respective acquired image. In some embodiments, the total high frequency content of each image is analyzed and used to define a Reference Frame. That is, in some examples, the Reference Frame is the image in the plurality of images (Image 1, Image 2 . . . Image N) having the highest frequency content, thereby designating an index for the Reference Frame (i.e., the index of the image determined to be the Reference Frame).
In some embodiments, the high frequency frames in the plurality of high frequency frames (e.g., HF Frame 1, HF Frame 2 . . . HF Frame N) are registered relative to one another. In some examples, the registration is performed on the high frequency frames. In an alternative embodiment, the images (e.g., Image 1, Image 2 . . . Image N) are registered relative to one another prior to generating the high frequency frames. Thus, the high frequency frames are already registered due to the registration of the images. In some embodiments, images are registered to the determined Reference Frame.
When registering images, some images are shifted from their original coordinates. For example, an image may be shifted by one or more regions to the left in order to be properly registered with other images. However, this can create blank regions within the image. For instance, if an image shifted one region left, the rightmost regions of the newly registered image may have no data associated wherewith, since these regions were previously outside the extent of the image. In such an example, in some embodiments, the blank regions are populated using the data in the region immediately adjacent to the blank region, thereby eliminating any blank regions in the registered images.
In various embodiments, regions within a high frequency frame (e.g., 4 and 5 in HF Frame n) need not have the same size or shape as one another. In preferred embodiments, each of a plurality of regions in one high frequency frame exists in each of the other high frequency frames. For example, shaded Region 5 in HF Frame n is similarly present in HF Frame n+1 and HF Frame n−1. Like regions in different high frequency frames have like size and shape, and correspond to like portions of the target scene when the high frequency frames are registered. Thus, in some embodiments, regions are not necessarily frame specific, but rather define a portion of the target scene. Such regions are thus present in each of the plurality of images and each of the corresponding high frequency frames.
In general, higher high frequency content tends to correspond to a more in-focus portion of an image, as it is generally indicative of sharp edges. Accordingly, in some embodiments, high frequency frames can be analyzed to determine which parts of which of the plurality of images are the most “in focus.” That is, for each region, it can be determined which high frequency frame has the greatest high frequency content. In some embodiments, an index can be assigned to each region, the assigned index being the index of the high frequency frame having the greatest high frequency content in that region. Such indices can be used to construct a final image wherein each region in the final image comprises image data from the corresponding region in the original image having that index.
For example, with reference to
In some examples, an intermediate step can include creating a best focus frame prior to generating a final image. The best focus frame can be a frame having a plurality of regions, each region corresponding to a region in each image and in each high frequency frame. The value associated with each region in the best focus frame is the index of the high frequency frame having the greatest high frequency content in the respective region. Thus, each region in the best focus frame is populated with an index designating a particular high frequency frame, and thus a particular image. Then, at least conceptually, the final image can be constructed by, for each region, replacing the index, n, in the best focus frame with the image data from the respective region from the image having index n. In theory, this image will include, in each region, the best-focused imaged data from the plurality of images.
In some embodiments, the plurality of images comprises a plurality of infrared images having different focus distances. In some examples, the each of the plurality of infrared images is captured with an associated visible light image having a like index as its associated infrared image. Each visible light image can be registered with its associated infrared image by any appropriate image registration process.
As described, the plurality of infrared images are registered and used to generate a plurality of high frequency frames, which are subsequently analyzed in order to generate a best focus frame comprising a plurality of indices as described. The indices are used to generate a final infrared image. In some examples, the best focus frame can be used to generate a final visible light image associated with the final infrared image. That is, the index in each of the plurality of regions of the best focus frame can be used to generate a visible light image comprising, in each of the plurality of regions, the visible light image data from that region in the visible light image having the index referenced in the best focus frame. The resulting final visible light image then includes visible light image data corresponding region-by-region to the infrared image data in the final infrared image. The final infrared image and final visible light image can be combined according to any of a variety of known processes, such as via alpha blending, picture-in-picture display, and the like.
The method further includes registering the images relative to one another (704). As described previously, in some examples, a reference frame can be determined and used as the frame to which other images are registered. The method includes, for each image, generating a high frequency frame representing high frequency content in a plurality of regions in the associated image (706). Thus, the high frequency content in a plurality of regions in each image is determined.
The method further includes, for each region, determining the index of the high frequency frame having the greatest high frequency content (708). Once such an index is determined for each region, a final image can be generated using, for each region, image data from the like region of the image having the determined index (710). The final image includes image data from one or more of the captured images, and the image from which the image data in each region in the final image is selected is determined based on the high frequency content of that image in that region.
While, as described previously, higher frequency content corresponds to a more in-focus image, in some instances, relatively large high frequency content may be present in an out-of-focus image as well. For example, a sharp edge in one region of an in-focus image can contribute to high frequency content in that region. However, as the edge becomes less focused as the focus distance is adjusted, the edge may blur and spread into adjacent regions, thus lowering the high frequency content of the original region and increasing the high frequency content of such adjacent regions. For instance, with reference to
This can lead to problems in creating the final image. One such example can occur near the location of a sharp edge in a nearby frame. In such an example, a certain region contains the defocused image of a nearby edge. The defocused location may have a large high frequency value as compared to sharpened background information. The high frequency content of this blurred edge should be ignored, but instead may undesirably contribute to the final image. For example, with reference to
Accordingly, in some embodiments, methods can include determining the validity of high frequency content in the high frequency frames. If high frequency content in a certain region in a certain high frequency frame is determined to be invalid, such high frequency content may be excluded when determining which high frequency frame has the greatest high frequency content for that region (e.g., step 708 in
In some examples, determining the validity of high frequency content in a region comprises comparing the amount the high frequency content in that region (i.e., the value of the high frequency frame in that region) to the amount of high frequency content in nearby regions.
The high frequency content of the selected region is compared to the high frequency content of proximate regions (908). Proximate regions can be considered to be proximate to the selected region in a spatial context, wherein the proximate regions are in the same frame as the selected region and within a certain neighborhood as the selected region. Additionally or alternatively, proximate regions can be considered to be proximate to the selected region based on index. That is, a region from a frame captured within a certain number of frames prior or subsequent to the capturing of the frame of the selected region can be considered proximate.
After comparing to the proximate regions, it can be determined whether or not the high frequency content is sufficiently close to the maximum high frequency content within the proximate regions (910). That is, the high frequency content of the selected region can be compared to a local maximum within the set of regions defined to be proximate the selected region. If the high frequency content of the selected region is not sufficiently close to the local maximum value, the selected region is considered invalid (912). Otherwise, it can be determined whether or not all regions have been analyzed (914). If not, a new region can be selected (900) and the process can be repeated. Otherwise, the validity determination is complete (916). It will be appreciated that alternatives to the method of
An exemplary validity determination can be demonstrated with reference to
If the threshold is met, the high frequency content of Region 5 is compared to the high frequency content of proximate regions. Which regions constitute proximate regions can vary between embodiments. In some examples, such a parameter can be set by a user. For instance, in an exemplary embodiment, regions proximate to a particular region in a given high frequency frame can include any immediately adjacent region in the same frame or an adjacent frame. Used with respect to Region 5 in HF Frame n in the example of
Determining whether or not the high frequency content of a particular region is sufficiently close to the local maximum can be performed in a variety of ways. In some examples, an invalidity scaling parameter is used to determine whether or not the high frequency content of the particular region is sufficiently close to the local maximum. For instance, an invalidity scaling parameter of 75% (or of 0.75) can be used, indicating that the high frequency content of the region is sufficiently close to the local maximum if it is at least 75% the local maximum value. In another embodiment, statistical analysis can be used to determine if the high frequency content is sufficiently close to the local maximum, such as within the 75% percentile of high frequency values of the proximate regions.
Allowing for regions that do not include the maximum high frequency content in the surrounding neighborhood of regions to be considered valid can help prevent undesired invalid region designations. For example, if a focused edge extending through multiple neighboring regions is present in a single image, valid high frequency content is likely present such regions in the corresponding high frequency frames. Deeming non-maximum regions within a neighborhood to be invalid may result in some regions that include such an edge being considered invalid. However, allowing for regions that have high frequency content sufficiently close to the local maximum may allow such regions to be considered valid despite possibly not having locally maximum high frequency content. In some embodiments, the requirements for what constitutes “sufficiently close,” including the type of comparison (e.g., threshold, statistical, and the like) and/or the comparison value (e.g., an invalidity scaling parameter) are selectable by a user in order to adjust the sensitivity of the validity determination process.
Once it is determined whether or not the high frequency content of a region (e.g., Region 5 in HF Frame n) is sufficiently close to the local maximum, a check is performed to determine if all regions have been analyzed (e.g., step 914). If so, the validity determination process is complete. If not, a new region can be selected (e.g., Region 4 in HF Frame n) and the process is repeated.
Designating valid and invalid regions in the high frequency frames can greatly impact the outcome of generating a final image.
The method of
In other embodiments, for each of the high frequency frames, a corresponding validity mask can be generated. The validity mask can be an array of binary values, each value associated with a corresponding region in the corresponding high frequency frame. That is, if a region in a high frequency frame is considered invalid, then the corresponding region in the corresponding validity mask is a zero. Otherwise, the corresponding region in the corresponding validity mask is a one.
The method of
Disregarding invalid high frequency content can help eliminate errors due to ghosting, haloing, or other artifacts when constructing a final image. Referring to a previous example with further reference to
Thus, in some examples, if Region 5 in HF Frame n and Region 7 in HF Frame n+1 are considered proximate to one another, the high frequency content in Region 5 of HF Frame n can render the comparatively lower high frequency content in Region 7 in HF Frame n+1 to be invalid. Accordingly, even if the high frequency content (due to artifacts) in Region 7 of HF Frame n+1 is greater than the high frequency content in Region 7 in HF Frame n−10 corresponding to a focused object in Image n−10, the image data from Image n−10 may be used to construct the final image, Region 7 in HF Frame n+1 is deemed invalid. That is, the high frequency content of Region 7 in HF Frame n+1 is not considered when determining the index of high frequency frame having the greatest high frequency content in Region 7.
As described elsewhere herein, the total high frequency content of the plurality of images (e.g., via analysis of the corresponding high frequency frames) can be used to determine a Reference Frame having the greatest total high frequency content. In some embodiments, the Reference Frame can refer to an index, thus being associated with both an image and a corresponding high frequency frame having like indices. Accordingly, in some embodiments, the method can include step of determining the total high frequency content of each image (1014) and setting the frame with the greatest high frequency content as the Reference Frame (1016). Image registration can comprise registering images relative to the Reference Frame (1018). Steps 1014, 1016, and 1018 can be performed as a sequence to achieve image registration, and may be used in addition to or in lieu of step 1004.
In some examples, generating a final image such as in step 1012 comprises generating a best focus frame comprising, for each region, the index of the frame having the greatest high frequency content.
The best focus frame of
As described, in generating the final image, for each region in the final image, image data from the like region of the image having the index of the high frequency frame having the greatest high frequency content in that region, which is the value in the best focus frame in that region. Since the value of the upper-left region in the best focus frame is “1”, the image data from the upper-left region in Image 1 is used to generate the upper-left region in the final image. As shown, the image data in the upper-left region of Image 1 is represented by “A.” Thus, the image data in the upper-left region of the final image is also “A.” Similarly, since the value of the lower-right region in the best focus frame is “3”, the image data from the lower-right region in Image 3 (“AA”) is used as the image data in the lower-right region in the final image.
Referring back to
In some embodiments, dilation processes fill these holes in the best focus frame. For example, index values in regions bordering such holes can be applied to such holes using standard dilation processes. In some embodiments, a series of dilation processes can be performed until all of the holes are filled, or dilation processes no longer fill remaining holes. In some such instances, all remaining holes are filled with the index of the determined Reference Frame (e.g., as determined in step 1016). Such steps can be used to fill holes in the best focus frame (1022) prior to generating the final image (1012).
As described elsewhere herein, in some examples, the acquired plurality of images (e.g., from step 1002) includes a plurality of infrared images each being captured with an associated visible light image. In some examples, steps 1004-1010 and steps 1020 and 1022 can be performed using the infrared images. In some such examples, step 1012 can include generating a final infrared image using, for each region, image data from the like region of the infrared image having the determined index referenced in the best focus frame. In some embodiments, step 1012 can be repeated using visible light images. That is, in some embodiments, the best focus frame generated in step 1020 can be used to generate a final visible light image associated with the final infrared image and comprising, for each region, image data from the like region of the visible light image having the determined index referenced in the best focus frame.
In some embodiments, to save on processing time, images can be downscaled (1024) prior to performing various processing steps. For example, as shown in
If a downscaling processes has been performed, in some embodiments, the processes further includes upscaling the images (1026), which can be performed, for example, prior to generating the final image (1012). In some examples, upscaling the image comprises performing the reverse processes as downscaling the images. This helps recover resolution lost due to the original downscaling process.
It will be appreciated that the methods described herein can be carried out in a variety of ways and by a variety of devices. For example, in some embodiments, methods herein described can be executed using a camera capable of acquiring a plurality images at a plurality of focus distances. The camera can execute an operation to both acquire the plurality of images having a plurality of focus distances, such as via methods described herein, and process the images to generate a final image. In some embodiments, a user may actuate automated execution of methods described herein. For example, a user can command the camera to adjust the focus by moving a lens assembly (such as IR lens assembly 104) from a first position (associated with a first focus) to a second position (associated with a second focus). The camera can move the lens assembly automatically via, for example, a motor. The camera can acquire a plurality of images while the lens is in a plurality of different positions between the first position and the second position during the lens motion, thus acquiring a plurality of images having a plurality of focus distances. Since some objects located at different depths in the scene will appear to be in focus in images having different focus distances, not all objects will appear in focus in a single image. Thus, an image viewing a live view of the target scene, for example, via display 108, may observe a scene in which not all objects are in focus. The user may initiate a through-focus process such as those described herein and achieve a fully focused image despite observing at least a partially unfocused image on the display and without having to perform any manual focusing procedures. In some cases, a user may observe a target scene via display 108 that is at least partially misfocused, and in response, trigger a through-focus process such as those described herein in order to generate a more fully focused final image. The final image that is generated can be saved in memory and/or displayed to the user.
In addition to such automated operation, in some examples, a user can manually perform the image acquisition process by repeatedly capturing images and adjusting the focus of the camera, and the camera can subsequently process the images to generate a final image.
Additionally or alternatively, automatically or manually captured images having different focus distances can be transferred to a separate computer or other processing device, such as a smartphone, tablet, or the like, for subsequent processing. The separate device can be capable of performing processes described herein in order to generate a final image from the plurality of images.
In various embodiments, images can include infrared images, visible light images, or combinations thereof, such as blended images and/or images displayed as a picture-in-picture display. In some embodiments, corresponding infrared and visible light images are captured during image acquisition. In some such embodiments, the visible light images can be used to register the infrared images relative to one another or vice versa.
The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a non-transitory computer-readable storage medium containing instructions. Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed. Computer readable storage media may include random access memory (RAM), read only memory (ROM), a hard disk, optical media, or other computer readable media.
Various embodiments have been described. Such examples are non-limiting, and do not define or limit the scope of the invention in any way. Rather, these and other examples are within the scope of the following claims.
| Number | Date | Country | |
|---|---|---|---|
| 62279415 | Jan 2016 | US |
