YCBCR pulsed illumination scheme in a light deficient environment

Information

  • Patent Grant
  • 11070779
  • Patent Number
    11,070,779
  • Date Filed
    Monday, July 27, 2020
    4 years ago
  • Date Issued
    Tuesday, July 20, 2021
    3 years ago
Abstract
The disclosure extends to methods, systems, and computer program products for producing an image in light deficient environments with luminance and chrominance emitted from a controlled light source.
Description
BACKGROUND

Advances in technology have provided advances in imaging capabilities for medical use. One area that has enjoyed some of the most beneficial advances is that of endoscopic surgical procedures because of the advances in the components that make up an endoscope.


The disclosure relates generally to electromagnetic sensing and sensors in relation to creating a video stream having chrominance and luminance pulses from a controlled light source. The features and advantages of the disclosure will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by the practice of the disclosure without undue experimentation. The features and advantages of the disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out herein.





BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive implementations of the disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. Advantages of the disclosure will become better understood with regard to the following description and accompanying drawings.



FIG. 1 illustrates a graphical representation of the operation of a pixel array in accordance with the principles and teachings of the disclosure;



FIG. 2 illustrates a graphical representation of a pixel array for a plurality of frames in accordance with the principles and teachings of the disclosure;



FIG. 3A illustrates a schematic of an embodiment of an operation sequence of chrominance and luminance frames in accordance with the principles and teachings of the disclosure;



FIG. 3B illustrates a schematic of an embodiment of an operation sequence of chrominance and luminance frames in accordance with the principles and teachings of the disclosure;



FIG. 3C illustrates a schematic of an embodiment of an operation sequence of chrominance and luminance frames in accordance with the principles and teachings of the disclosure;



FIG. 4 illustrates an embodiment of sensor and emitter modulation in accordance with the principles and teachings of the disclosure;



FIG. 5 illustrates an embodiment of sensor and emitter patterns in accordance with the principles and teachings of the disclosure;



FIG. 6A illustrates an embodiment of sensor and emitter patterns in accordance with the principles and teachings of the disclosure;



FIG. 6B illustrates an embodiment of sensor and emitter patterns in accordance with the principles and teachings of the disclosure;



FIG. 7 illustrates a graphical representation of the operation of a pixel array having pixels of differing pixel sensitivities in accordance with the principles and teachings of the disclosure;



FIG. 8 illustrates a graphical representation of the operation of a pixel array having pixels of differing pixel sensitivities in accordance with the principles and teachings of the disclosure;



FIG. 9 illustrates a flow chart of the operation of a pixel array in accordance with the principles and teachings of the disclosure;



FIG. 10 illustrates a flow chart of the operation of a pixel array in accordance with the principles and teachings of the disclosure;



FIG. 11 illustrates a flow chart of the operation of a pixel array in accordance with the principles and teachings of the disclosure;



FIG. 12A illustrates a graphical representation of the operation of a pixel array in accordance with the principles and teachings of the disclosure;



FIG. 12B illustrates a graphical representation of the operation of a pixel array in accordance with the principles and teachings of the disclosure;



FIG. 13 illustrates an embodiment of supporting hardware in accordance with the principles and teachings of the disclosure;



FIGS. 14A and 14B illustrate an implementation having a plurality of pixel arrays for producing a three dimensional image in accordance with the teachings and principles of the disclosure;



FIGS. 15A and 15B illustrate a perspective view and a side view, respectively, of an implementation of an imaging sensor built on a plurality of substrates, wherein a plurality of pixel columns forming the pixel array are located on the first substrate and a plurality of circuit columns are located on a second substrate and showing an electrical connection and communication between one column of pixels to its associated or corresponding column of circuitry; and



FIGS. 16A and 16B illustrate a perspective view and a side view, respectively, of an implementation of an imaging sensor having a plurality of pixel arrays for producing a three dimensional image, wherein the plurality of pixel arrays and the image sensor are built on a plurality of substrates.





DETAILED DESCRIPTION

The disclosure extends to methods, systems, and computer based products for digital imaging that may be primarily suited to medical applications. In the following description of the disclosure, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific implementations in which the disclosure may be practiced. It is understood that other implementations may be utilized and structural changes may be made without departing from the scope of the disclosure.


Luminance-chrominance based color spaces date back to the advent of color television, when color image transmission was required to be compatible with older monochrome CRTs. The luminance component corresponds to the (color-agnostic) brightness aspect of the image data. The color information is carried in the remaining two channels. The separation of image data into the luminance and chrominance components is still an important process in modern digital imaging systems, since it is closely related to the human visual system.


The human retina contains arrays of two basic photoreceptor cell types; rods and cones. The rods provide the brightness information and have about a factor-20 greater overall spatial density than the cones. The cones are much less sensitive and there are three basic types, having peak responses at three different wavelengths. The spectral response of the rods, which peaks in the green region, is the basis for computing luminance color-space conversion coefficients. Since rods have the greater density, the spatial resolution of an image representation is much more important for the luminance component than for either chrominance component. Camera designers and image processing engineers seek to account for this fact in several ways, e.g., by spatially filtering the chrominance channels to reduce noise and by affording greater relative system bandwidth to luminance data.


In describing the subject matter of the disclosure, the following terminology will be used in accordance with the definitions set out below.


It must be noted that, as used in this specification, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise.


As used herein, the terms “comprising,” “including,” “containing,” “characterized by,” and grammatical equivalents thereof are inclusive or open-ended terms that do not exclude additional, unrecited elements or method steps.


As used herein, the phrase “consisting of” and grammatical equivalents thereof exclude any element or step not specified.


As used herein, the phrase “consisting essentially of” and grammatical equivalents thereof limit the scope of a claim, if any, to the specified materials or steps and those that do not materially affect the basic and novel characteristic or characteristics of the claimed disclosure.


As used herein, the term “proximal” shall refer broadly to the concept of a portion nearest an origin.


As used herein, the term “distal” shall generally refer to the opposite of proximal, and thus to the concept of a portion farther from an origin, or a furthest portion, depending upon the context.


Referring now to the figures, FIG. 1 illustrates the basic timing of single frame capture by a conventional CMOS sensor. Co-pending U.S. patent application Ser. No. 13/952,518 entitled CONTINUOUS VIDEO IN A LIGHT DEFICIENT ENVIRONMENT is hereby incorporated by this reference into this disclosure as if fully set forth herein. It will be appreciated that the x direction corresponds to time and the diagonal lines indicate the activity of an internal pointer that reads out each frame of data, one line at time. The same pointer is responsible for resetting each row of pixels for the next exposure period. The net integration time for each row is equivalent, but they are staggered in time with respect to one another due to the rolling reset and read process. Therefore, for any scenario in which adjacent frames are required to represent different constitutions of light, the only option for having each row be consistent is to pulse the light between the readout cycles. More specifically, the maximum available period corresponds to the sum of the blanking time plus any time during which optical black or optically blind (OB) rows are serviced at the start or end of the frame.


An example illumination sequence is a repeating pattern of four frames (R-G-B-G). As for the Bayer pattern of color filters, this provides for greater luminance detail than chrominance. This approach is accomplished by strobing the scene with either laser or light-emitting diodes at high speed, under the control of the camera system, and by virtue of a specially designed CMOS sensor with high speed readout. The principal benefit is that the sensor can accomplish the same spatial resolution with significantly fewer pixels compared with conventional Bayer or 3-sensor cameras. Therefore, the physical space occupied by the pixel array may be reduced. The actual pulse periods may differ within the repeating pattern, as illustrated in FIG. 2. This is useful for, e.g., apportioning greater time to the components that require the greater light energy or those having the weaker sources. As long as the average captured frame rate is an integer multiple of the requisite final system frame rate, the data may simply be buffered in the signal processing chain as appropriate.


The facility to reduce the CMOS sensor chip-area to the extent allowed by combining all of these methods is particularly attractive for small diameter (˜3-10 mm) endoscopy. In particular, it allows for endoscope designs in which the sensor is located in the space-constrained distal end, thereby greatly reducing the complexity and cost of the optical section, while providing high definition video. A consequence of this approach is that to reconstruct each final, full color image, requires that data be fused from three separate snapshots in time. Any motion within the scene, relative to the optical frame of reference of the endoscope, will generally degrade the perceived resolution, since the edges of objects appear at slightly different locations within each captured component. In this disclosure, a means of diminishing this issue is described which exploits the fact that spatial resolution is much more important for luminance information, than for chrominance.


The basis of the approach is that, instead of firing monochromatic light during each frame, combinations of the three wavelengths are used to provide all of the luminance information within a single image. The chrominance information is derived from separate frames with, e.g., a repeating pattern such as Y-Cb-Y-Cr. While it is possible to provide pure luminance data by a shrewd choice of pulse ratios, the same is not true of chrominance. However, a workaround for this is presented in this disclosure.


In an embodiment, as illustrated in FIG. 3A, an endoscopic system 300a may comprise a pixel array 302a having uniform pixels and the system 300a may be operated to receive Y (luminance pulse) 304a, Cb (ChromaBlue) 306a and Cr (ChromaRed) 308a pulses.


In an embodiment, as illustrated in FIG. 3B, an endoscopic system 300b may comprise a pixel array 302b having uniform pixels and the system may be operated to receive Y (luminance pulse) 304b, λY+Cb (Modulated ChromaBlue) 306b and δY+Cr (Modulated ChromaRed) 308b pulses.


In an embodiment, as illustrated in FIG. 3C, an endoscopic system 300c may comprise a pixel array 302c having checker patterned (alternating) pixels and the system may be operated to receive Y (luminance pulse) 304c, λY+Cb (Modulated ChromaBlue) 306c and δY+Cr (Modulated ChromaRed) 308c pulses. Within the luminance frames, the two exposure periods are applied for the purpose of extending the dynamic range (YL and YS, corresponding to the long and short exposures).



FIG. 4 illustrates the general timing relationship within a 4-frame cycle, between pulsed mixtures of three wavelengths and the readout cycle of a monochrome CMOS sensor.


Essentially there are three monochromatic pulsed light sources under the fast control of the camera and a special design of monochromatic CMOS image sensor which enables high final progressive video rates of 60 Hz or more. Periodic sequences of monochromatic red, green and blue frames are captured, e.g., with an R-G-B-G pattern, and assembled into sRGB images in the image signal processor chain (ISP). The light-pulse and sensor readout timing relationship is shown in FIG. 5. In order to provide pure luminance information in the same frame, all three sources are pulsed in unison with light energies that are modulated according to the color transformation coefficients that convert from RGB space to YCbCr (as per the ITU-R BT.709 HD standard):







[



Y




Cb




Cr



]

=


[



R




G




B



]



[



0.183


0.614


0.062





-
0.101




-
0.339



0.439




0.439



-
0.399




-
0.040




]






It will be appreciated that other color space conversion standards may be implemented by the disclosure, including but not limited to, ITU-R BT.709 HD standard, ITU-R BT.601 standard, and ITU-R BT.2020 standard.


If white balance is being performed in the illumination domain, then this modulation is imposed in addition to the white balance modulation.


To complete a full color image requires that the two components of chrominance also be provided. However, the same algorithm that was applied for luminance cannot be directly applied for chrominance images since it is signed, as reflected in the fact that some of the RGB coefficients are negative. The solution to this is to add a degree of luminance of sufficient magnitude that all of the final pulse energies become positive. As long as the color fusion process in the ISP is aware of the composition of the chrominance frames, they can be decoded by subtracting the appropriate amount of luminance from a neighboring frame. The pulse energy proportions are given by:

Y=0.183·R+0.614·G+0.062·B
Cb=λ·Y−0.101·R−0.339·G+0.439·B
Cr=δ·Y+0.439·R−0.399·G−0.040·B


where







λ


0.339
0.614


=
0.552







δ


0.399
0.614


=
0.650




The timing for the general case is shown in FIG. 6A. It turns out that if the λ factor is equal to 0.552; both the red and the green components are exactly cancelled, in which case the Cb information can be provided with pure blue light. Similarly, setting δ=0.650 cancels out the blue and green components for Cr which becomes pure red. This particular example is illustrated in FIG. 6B, which also depicts λ and δ gas integer multiples of ½8. This is a convenient approximation for the digital frame reconstruction (see later discussion).


Referring now to FIG. 7, there is illustrated a general timing diagram for this process. The exposure periods for the two flavors of pixel are controlled by two internal signals within the image sensor, depicted as TX1 and TX2 in the figure. In fact, it is possible to do this at the same time as extending the dynamic range for the luminance frame, where it is most needed, since the two integration times can be adjusted on a frame by frame basis (see FIGS. 3a-3c). The benefit is that the color motion artifacts are less of an issue if all the data is derived from two frames versus three. There is of course a subsequent loss of spatial resolution for the chrominance data but that is of negligible consequence to the image quality for the reasons discussed earlier.


An inherent property of the monochrome wide dynamic range array is that the pixels that have the long integration time must integrate a superset of the light seen by the short integration time pixels. Co-pending U.S. patent application Ser. No. 13/952,564 entitled WIDE DYNAMIC RANGE USING MONOCHROMATIC SENSOR is hereby incorporated by this reference into this disclosure as if fully set forth herein. For regular wide dynamic range operation in the luminance frames, that is desirable. For the chrominance frames it means that the pulsing must be controlled in conjunction with the exposure periods so as to provide, e.g., λY+Cb from the start of the long exposure and switch to δY+Cr at the point that the short pixels are turned on (both pixel types have their charges transferred at the same time). During color fusion, this would be accounted for. FIG. 8 shows the specific timing diagram for this solution.


A typical ISP involves first taking care of any necessary sensor and optical corrections (such as defective pixel elimination, lens shading etc.), then in turn; white balance, demosaic/color fusion and color correction.


Before finally applying gamma to place the data in the standard sRGB space, there might typically be some operations (e.g., edge enhancement) and/or adjustments (e.g., saturation) performed in an alternative color space such as YCbCr or HSL. FIG. 9 depicts a basic ISP core that would be appropriate for the R-G-B-G pulsing scheme. In this example, the data is converted to YCbCr in order to apply edge enhancement in the luminance plane and conduct filtering of the chrominance, then converted back to linear RGB.


In the case of the Y-Cb-Y-Cr pulsing scheme, the image data is already in the YCbCr space following the color fusion. Therefore, in this case it makes sense to perform luminance and chrominance based operations up front, before converting back to linear RGB to perform the color correction etc. See FIG. 10.


The color fusion process is more straightforward than de-mosaic, which is necessitated by the Bayer pattern, since there is no spatial interpolation. It does require buffering of frames though in order to have all of the necessary information available for each pixel, as indicated in FIG. 11. FIG. 12A shows the general situation of pipelining of data for the Y-Cb-Y-Cr pattern which yields 1 full color image per two raw captured images. This is accomplished by using each chrominance sample twice. In FIG. 12B the specific example of a 120 Hz frame capture rate providing 60 Hz final video is drawn.


The linear Y, Cb and Cr components for each pixel may be computed thus:












Y
i

=


2

m
-
4


+

(


x

i
,

n
-
1



-
K

)











{






Cb
i

=


2

m
-
1


+

(


x

i
,
n


-
K

)

-

λ
·

(


x

i
,

n
-
1



-
K

)















Cr
i

=


2

m
-
1


+

(


x

i
,

n
-
2



-
K

)

-

δ
·

(


x

i
,

n
-
1



-
K

)







}






when





n

=



Cb







frame









{





Cb
i

=


2

m
-
1


+

(


x

i
,

n
-
2



-
K

)

-

λ
·

(


x

i
,

n
-
1



-
K

)











Cr
i

=


2

m
-
1


+

(


x

i
,
n


-
K

)

-

δ
·

(


x

i
,

n
-
1



-
K

)












}






when





n

=



Cr







frame





Where xi,n is the input data for pixel i in frame n, m is the pipeline bit-width of the ISP and K is the ISP black offset level at the input to the color fusion block, (if applicable). Since chrominance is signed it is conventionally centered at 50% of the digital dynamic range (2m-1).


If two exposures are used to provide both chrominance components in the same frame as described earlier, the two flavors of pixel are separated into two buffers. The empty pixels are then filled in using, e.g., linear interpolation. At this point, one buffer contains a full image of δY+Cr data and the other; δY+Cr+λY+Cb. The δY+Cr buffer is subtracted from the second buffer to give λY+Cb. Then the appropriate proportion of luminance data from the Y frames is subtracted out for each.


Implementations of the disclosure may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Implementations within the scope of the disclosure may also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are computer storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, implementations of the disclosure can comprise at least two distinctly different kinds of computer-readable media: computer storage media (devices) and transmission media.


Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.


A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. In an implementation, a sensor and camera control unit may be networked in order to communicate with each other, and other components, connected over the network to which they are connected. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.


As can be seen in FIG. 13, various computer system components, program code means in the form of computer-executable instructions or data structures that can be transferred automatically from transmission media to computer storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile computer storage media (devices) at a computer system. RAM can also include solid state drives (SSDs or PCIx based real time memory tiered Storage, such as FusionIO). Thus, it should be understood that computer storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.


Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined herein is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as examples.


Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, control units, camera control units, hand-held devices, hand pieces, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. It should be noted that any of the above mentioned computing devices may be provided by or located within a brick and mortar location. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.


Further, where appropriate, functions described herein can be performed in one or more of: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) or field programmable gate arrays can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the following description to refer to particular system components. As one skilled in the art will appreciate, components may be referred to by different names. This document does not intend to distinguish between components that differ in name, but not function.



FIG. 13 is a block diagram illustrating an example computing device 100. Computing device 100 may be used to perform various procedures, such as those discussed herein. Computing device 100 can function as a server, a client, or any other computing entity. Computing device can perform various monitoring functions as discussed herein, and can execute one or more application programs, such as the application programs described herein. Computing device 100 can be any of a wide variety of computing devices, such as a desktop computer, a notebook computer, a server computer, a handheld computer, camera control unit, tablet computer and the like.


Computing device 100 includes one or more processor(s) 102, one or more memory device(s) 104, one or more interface(s) 106, one or more mass storage device(s) 108, one or more Input/Output (I/O) device(s) 110, and a display device 130 all of which are coupled to a bus 112. Processor(s) 102 include one or more processors or controllers that execute instructions stored in memory device(s) 104 and/or mass storage device(s) 108. Processor(s) 102 may also include various types of computer-readable media, such as cache memory.


Memory device(s) 104 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 114) and/or nonvolatile memory (e.g., read-only memory (ROM) 116). Memory device(s) 104 may also include rewritable ROM, such as Flash memory.


Mass storage device(s) 108 include various computer readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., Flash memory), and so forth. As shown in FIG. 13, a particular mass storage device is a hard disk drive 124. Various drives may also be included in mass storage device(s) 108 to enable reading from and/or writing to the various computer readable media. Mass storage device(s) 108 include removable media 126 and/or non-removable media.


I/O device(s) 110 include various devices that allow data and/or other information to be input to or retrieved from computing device 100. Example I/O device(s) 110 include digital imaging devices, electromagnetic sensors and emitters, cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.


Display device 130 includes any type of device capable of displaying information to one or more users of computing device 100. Examples of display device 130 include a monitor, display terminal, video projection device, and the like.


Interface(s) 106 include various interfaces that allow computing device 100 to interact with other systems, devices, or computing environments. Example interface(s) 106 may include any number of different network interfaces 120, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interface(s) include user interface 118 and peripheral device interface 122. The interface(s) 106 may also include one or more user interface elements 118. The interface(s) 106 may also include one or more peripheral interfaces such as interfaces for printers, pointing devices (mice, track pad, etc.), keyboards, and the like.


Bus 112 allows processor(s) 102, memory device(s) 104, interface(s) 106, mass storage device(s) 108, and I/O device(s) 110 to communicate with one another, as well as other devices or components coupled to bus 112. Bus 112 represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.


For purposes of illustration, programs and other executable program components are shown herein as discrete blocks, although it is understood that such programs and components may reside at various times in different storage components of computing device 100, and are executed by processor(s) 102. Alternatively, the systems and procedures described herein can be implemented in hardware, or a combination of hardware, software, and/or firmware. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein.



FIGS. 14A and 14B illustrate a perspective view and a side view, respectively, of an implementation of a monolithic sensor 2900 having a plurality of pixel arrays for producing a three dimensional image in accordance with the teachings and principles of the disclosure. Such an implementation may be desirable for three dimensional image capture, wherein the two pixel arrays 2902 and 2904 may be offset during use. In another implementation, a first pixel array 2902 and a second pixel array 2904 may be dedicated to receiving a predetermined range of wave lengths of electromagnetic radiation, wherein the first pixel array is dedicated to a different range of wave length electromagnetic radiation than the second pixel array.



FIGS. 15A and 15B illustrate a perspective view and a side view, respectively, of an implementation of an imaging sensor 3000 built on a plurality of substrates. As illustrated, a plurality of pixel columns 3004 forming the pixel array are located on the first substrate 3002 and a plurality of circuit columns 3008 are located on a second substrate 3006. Also illustrated in the figure are the electrical connection and communication between one column of pixels to its associated or corresponding column of circuitry. In one implementation, an image sensor, which might otherwise be manufactured with its pixel array and supporting circuitry on a single, monolithic substrate/chip, may have the pixel array separated from all or a majority of the supporting circuitry. The disclosure may use at least two substrates/chips, which will be stacked together using three-dimensional stacking technology. The first 3002 of the two substrates/chips may be processed using an image CMOS process. The first substrate/chip 3002 may be comprised either of a pixel array exclusively or a pixel array surrounded by limited circuitry. The second or subsequent substrate/chip 3006 may be processed using any process, and does not have to be from an image CMOS process. The second substrate/chip 3006 may be, but is not limited to, a highly dense digital process in order to integrate a variety and number of functions in a very limited space or area on the substrate/chip, or a mixed-mode or analog process in order to integrate for example precise analog functions, or a RF process in order to implement wireless capability, or MEMS (Micro-Electro-Mechanical Systems) in order to integrate MEMS devices. The image CMOS substrate/chip 3002 may be stacked with the second or subsequent substrate/chip 3006 using any three-dimensional technique. The second substrate/chip 3006 may support most, or a majority, of the circuitry that would have otherwise been implemented in the first image CMOS chip 3002 (if implemented on a monolithic substrate/chip) as peripheral circuits and therefore have increased the overall system area while keeping the pixel array size constant and optimized to the fullest extent possible. The electrical connection between the two substrates/chips may be done through interconnects 3003 and 3005, which may be wirebonds, bump and/or TSV (Through Silicon Via).



FIGS. 16A and 16B illustrate a perspective view and a side view, respectively, of an implementation of an imaging sensor 3100 having a plurality of pixel arrays for producing a three dimensional image. The three dimensional image sensor may be built on a plurality of substrates and may comprise the plurality of pixel arrays and other associated circuitry, wherein a plurality of pixel columns 3104a forming the first pixel array and a plurality of pixel columns 3104b forming a second pixel array are located on respective substrates 3102a and 3102b, respectively, and a plurality of circuit columns 3108a and 3108b are located on a separate substrate 3106. Also illustrated are the electrical connections and communications between columns of pixels to associated or corresponding column of circuitry.


It will be appreciated that the teachings and principles of the disclosure may be used in a reusable device platform, a limited use device platform, a re-posable use device platform, or a single-use/disposable device platform without departing from the scope of the disclosure. It will be appreciated that in a re-usable device platform an end-user is responsible for cleaning and sterilization of the device. In a limited use device platform the device can be used for some specified amount of times before becoming inoperable. Typical new device is delivered sterile with additional uses requiring the end-user to clean and sterilize before additional uses. In a re-posable use device platform a third-party may reprocess the device (e.g., cleans, packages and sterilizes) a single-use device for additional uses at a lower cost than a new unit. In a single-use/disposable device platform a device is provided sterile to the operating room and used only once before being disposed of.


Additionally, the teachings and principles of the disclosure may include any and all wavelengths of electromagnetic energy, including the visible and non-visible spectrums, such as infrared (IR), ultraviolet (UV), and X-ray.


In an embodiment, a method for digital imaging for use with an endoscope in ambient light deficient environments may comprise: actuating an emitter to emit a plurality of pulses of electromagnetic radiation to cause illumination within the light deficient environment, wherein said pulses comprise a first pulse that is within a first wavelength range that comprises a first portion of electromagnetic spectrum, wherein said pulses comprise a second pulse that is within a second wavelength range that comprises a second portion of electromagnetic spectrum, wherein said pulses comprise a third pulse that is with is a third wavelength range that comprises a third portion of electromagnetic spectrum; pulsing said plurality of pulses at a predetermined interval; sensing reflected electromagnetic radiation from said pulses with a pixel array to create a plurality of image frames, wherein said pixel array is read at an interval that corresponds to the pulse interval of said laser emitter; and creating a stream of images by combining the plurality of image frames to form a video stream. In an embodiment, said first pulse comprises chrominance red. In an embodiment, said second pulse comprises chrominance blue. In an embodiment, said third pulse comprises a luminance pulse. In an embodiment, said luminance pulse is created by pulsing a red pulse and a blue pulse and a green pulse. In such an embodiment, said red pulse is modulated relative to the blue and green pulse such that the red pulse has a positive chrominance value. In an embodiment, said blue pulse is modulated relative to the red and green pulse such that the blue pulse has a positive chrominance value. In an embodiment, said green pulse is modulated relative to the blue and red pulse such that the green pulse has a positive chrominance value. In an embodiment, the method further comprises modulating the plurality of pulses by a value such that the chrominance value of each pulse is positive. In an embodiment, the method further comprises removing pulse modulation values from during image stream construction. In such an embodiment, the process of modulating comprises adding a luminance value to the plurality of pulses. In an embodiment, the luminance value for modulation is an integer that is a multiple of (½)8. In an embodiment, a luminance value for modulation of 0.552 cancels out red chrominance and green chrominance. In an embodiment, a luminance value for modulation of 0.650 cancels out blue chrominance and green chrominance. In an embodiment, the method further comprises reducing noise while creating the stream of image frames. In an embodiment, the method further comprises adjusting white balance while creating the stream of mage frames. In an embodiment, said third pulse is a luminance pulse that is pulses twice as often as the first and second pulses. In an embodiment, said luminance pulse is sensed by long exposure pixel and short exposure pixels within a pixel array. In an embodiment, the method further comprises sensing data generated by a plurality of pixel arrays and combining said data into a three dimensional image stream.


It will be appreciated that various features disclosed herein provide significant advantages and advancements in the art. The following embodiments are exemplary of some of those features.


In the foregoing Detailed Description of the Disclosure, various features of the disclosure are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the disclosure requires more features than are expressly recited in each claim, if any. Rather, inventive aspects lie in less than all features of a single foregoing disclosed embodiment.


It is to be understood that the above-described arrangements are only illustrative of the application of the principles of the disclosure. Numerous modifications and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the disclosure.


Thus, while the disclosure has been shown in the drawings and described above with particularity and detail, it will be apparent to those of ordinary skill in the art that numerous modifications, including, but not limited to, variations in size, materials, shape, form, function and manner of operation, assembly and use may be made without departing from the principles and concepts set forth herein.


Further, where appropriate, functions described herein can be performed in one or more of: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the following description to refer to particular system components. As one skilled in the art will appreciate, components may be referred to by different names. This document does not intend to distinguish between components that differ in name, but not function.

Claims
  • 1. A system for digital imaging in an ambient light deficient environment comprising: an emitter configured to emit a pulse of electromagnetic radiation;an imaging sensor comprising an array of pixels for sensing electromagnetic radiation, wherein the imaging sensor comprises pixels having a plurality of pixel sensitivities, and wherein the plurality of pixel sensitivities comprises a long exposure and a short exposure; anda control unit comprising a processor and wherein the control unit is in electrical communication with the imaging sensor and the emitter;wherein the control unit is configured to synchronize the emitter and the imaging sensor to produce a plurality of image reference frames;wherein the imaging sensor is configured to produce a sequence of frames comprising: a luminance frame comprising luminance image data of both long exposure pixel data and short exposure pixel data in a single luminance frame; anda chrominance frame comprising chrominance data;wherein the luminance frame and the chrominance frame are combined to form a color image frame.
  • 2. The system of claim 1, wherein the sequence of image reference frames produced by the imaging sensor comprises: a red chrominance frame; anda blue chrominance frame;wherein at least one of the red chrominance frame and the blue chrominance frame does not include long exposure pixel data and short exposure pixel data.
  • 3. The system of claim 2, wherein the luminance frame is represented in the pattern twice as often as the red chrominance frame and blue chrominance frame.
  • 4. The system of claim 1, wherein the sequence of image reference frames produced by the imaging sensor comprises: a red chrominance frame of long exposure pixel data and short exposure pixel data, anda blue chrominance frame of long exposure pixel data and short exposure pixel data.
  • 5. The system of claim 1, wherein the emitter produces a pulsing pattern of luminance, chrominance blue, luminance, chrominance red.
  • 6. The system of claim 1, wherein the emitter produces a pulsing pattern of luminance, chrominance blue combined with chrominance red, luminance, chrominance blue combined with chrominance red.
  • 7. The system of claim 1, wherein the emitter comprises a plurality of sources that each emits a pulse of a portion of electromagnetic spectrum.
  • 8. The system of claim 7, wherein the plurality of sources is configured to be actuated simultaneously and wherein the plurality of sources is configured to produce a pulse of a predetermined interval.
  • 9. The system of claim 1, wherein the pulse is adjusted to provide luminance information during the luminance frame by matching to color space conversion coefficients.
  • 10. The system of claim 1, wherein the pulse is adjusted to provide chrominance information during the chrominance frame by matching to color space conversion coefficients.
  • 11. The system of claim 1, wherein the controller is configured to use chrominance frames more than once to reconstruct the color image frame.
  • 12. The system of claim 1, wherein a luminance coefficient is added to chrominance frames by an image signal processor and wherein the luminance coefficient is an integer that is a multiple of (½)8.
  • 13. The system of claim 1, wherein the imaging sensor comprises uniform pixels configured to be read individually, wherein the uniform pixels can be read after a plurality of durations wherein the plurality of durations produces a long exposure and a short exposure.
  • 14. The system of claim 1, wherein the imaging sensor is a monochrome sensor.
  • 15. The system of claim 1, wherein a pulse of electromagnetic radiation emitted by the emitter is of a wavelength that is not visible to humans.
  • 16. The system of claim 1, wherein the emitter emits a plurality of electromagnetic wavelengths, wherein at least one of the plurality of electromagnetic wavelengths are visible to humans and that are not visible to humans.
  • 17. The system of claim 1, wherein the plurality of electromagnetic wavelengths is emitted at differing magnitudes, wherein the differing magnitudes correspond to the imaging sensor's sensitivity to differing wavelengths.
  • 18. The system of claim 1, wherein the system further comprises an endoscope for accessing the ambient light deficient environment comprising a lumen and a hand piece.
  • 19. The system of claim 18, wherein the imaging sensor is disposed within a distal portion of the lumen of the endoscope.
  • 20. The system of claim 18, wherein the imaging sensor is disposed within the hand piece.
  • 21. The system of claim 1, wherein said pixel array comprises a plurality of subsets of pixels, wherein the plurality of subsets of pixels each have different sensitivities, wherein variations in sensitivity for different subsets of pixels are accomplished by separate, global exposure periods.
  • 22. The system of claim 1, wherein the emitter comprises one or more laser bundles.
CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No. 16/398,114, filed Apr. 29, 2019, which is a continuation of U.S. application Ser. No. 15/701,264, filed on Sep. 11, 2017 (now U.S. Pat. No. 10,277,875), which is a continuation of U.S. application Ser. No. 15/369,170, filed on Dec. 5, 2016 (now U.S. Pat. No. 9,762,879, issued Sep. 12, 2017), which is a division of U.S. application Ser. No. 13/952,570, filed on Jul. 26, 2013 (now U.S. Pat. No. 9,516,239, issued Dec. 6, 2016) and claims the benefit of U.S. Provisional Patent Application No. 61/676,289, filed on Jul. 26, 2012, and U.S. Provisional Patent Application No. 61/790,487, filed on Mar. 15, 2013, and U.S. Provisional Patent Application No. 61/790,719, filed on Mar. 15, 2013 and U.S. Provisional Patent Application No. 61/791,473, filed on Mar. 15, 2013, which are hereby incorporated by reference herein in their entireties, including but not limited to those portions that specifically appear hereinafter, the incorporation by reference being made with the following exception: In the event that any portion of the above-referenced applications is inconsistent with this application, this application supersedes said above-referenced applications.

US Referenced Citations (512)
Number Name Date Kind
3666885 Hemsley et al. May 1972 A
4011403 Epstein et al. Mar 1977 A
4356534 Hattori Oct 1982 A
4363963 Ando Dec 1982 A
4433675 Konoshima Feb 1984 A
4436095 Kruger Mar 1984 A
4473839 Noda Sep 1984 A
4644403 Sakai et al. Feb 1987 A
4651226 Motoori et al. Mar 1987 A
4692606 Sakai et al. Sep 1987 A
4740837 Yanagisawa et al. Apr 1988 A
4741327 Yabe May 1988 A
4742388 Cooper et al. May 1988 A
4745471 Takamura et al. May 1988 A
4773396 Okazaki Sep 1988 A
4782386 Ams et al. Nov 1988 A
4786965 Yabe Nov 1988 A
4832003 Yabe May 1989 A
4845555 Yabe et al. Jul 1989 A
4853772 Kikuchi Aug 1989 A
4853773 Hibino et al. Aug 1989 A
4865018 Kanno et al. Sep 1989 A
4866526 Ams et al. Sep 1989 A
4884133 Kanno et al. Nov 1989 A
4884134 Tsuji et al. Nov 1989 A
4908701 Udagawa Mar 1990 A
4918521 Yabe et al. Apr 1990 A
4924856 Noguchi May 1990 A
4938205 Nudelman Jul 1990 A
4942473 Zeevi et al. Jul 1990 A
4947246 Kikuchi Aug 1990 A
4953539 Nakamura et al. Sep 1990 A
4959710 Uehara et al. Sep 1990 A
4963960 Takami Oct 1990 A
5001556 Nakamura et al. Mar 1991 A
5016975 Sasaki et al. May 1991 A
5021888 Kondou et al. Jun 1991 A
5047846 Uchiyama et al. Sep 1991 A
RE33854 Adair Mar 1992 E
5103497 Hicks Apr 1992 A
5111804 Funakoshi May 1992 A
5133035 Hicks Jul 1992 A
5187572 Nakamura et al. Feb 1993 A
5188094 Adair Feb 1993 A
5196938 Blessinger Mar 1993 A
5200838 Nudelman et al. Apr 1993 A
5220198 Tsuji Jun 1993 A
5228430 Sakamoto Jul 1993 A
5233416 Inoue Aug 1993 A
5241170 Field, Jr. et al. Aug 1993 A
5255087 Nakamura et al. Oct 1993 A
5264925 Shipp et al. Nov 1993 A
5313306 Kuban et al. May 1994 A
5325847 Matsuno Jul 1994 A
5365268 Minami Nov 1994 A
5402768 Adair Apr 1995 A
5408268 Shipp Apr 1995 A
5411020 Ito May 1995 A
5427087 Ito et al. Jun 1995 A
5454366 Ito et al. Oct 1995 A
5494483 Adair Feb 1996 A
5523786 Parulski Jun 1996 A
5550595 Hannah Aug 1996 A
5558841 Nakagawa et al. Sep 1996 A
5594497 Ahem et al. Jan 1997 A
5627584 Nishikori et al. May 1997 A
5658238 Suzuki et al. Aug 1997 A
5665959 Fossum et al. Sep 1997 A
5704836 Norton et al. Jan 1998 A
5730702 Tanaka et al. Mar 1998 A
5734418 Danna Mar 1998 A
5748234 Lippincott May 1998 A
5749830 Kaneko et al. May 1998 A
5754313 Pelchy et al. May 1998 A
5783909 Hochstein Jul 1998 A
5784099 Lippincott Jul 1998 A
5857963 Pelchy et al. Jan 1999 A
5887049 Fossum Mar 1999 A
5924978 Koeda et al. Jul 1999 A
5929901 Adair et al. Jul 1999 A
5949483 Fossum et al. Sep 1999 A
5957834 Mochida Sep 1999 A
5986693 Adair et al. Nov 1999 A
5995136 Hattori et al. Nov 1999 A
6023315 Harrold et al. Feb 2000 A
6038067 George Mar 2000 A
6043839 Adair et al. Mar 2000 A
6139489 Wampler et al. Oct 2000 A
6141505 Miyata et al. Oct 2000 A
6142930 Ito et al. Nov 2000 A
6166768 Fossum et al. Dec 2000 A
6184922 Saito et al. Feb 2001 B1
6184940 Sano Feb 2001 B1
6215517 Takahashi et al. Mar 2001 B1
6222175 Krymski Apr 2001 B1
6239456 Berezin et al. May 2001 B1
6272269 Naum Aug 2001 B1
6275255 Adair et al. Aug 2001 B1
6292220 Ogawa et al. Sep 2001 B1
6294775 Seibel et al. Sep 2001 B1
6310642 Adair et al. Oct 2001 B1
6320331 Iida et al. Nov 2001 B1
6331156 Haefele et al. Dec 2001 B1
6389205 Muckner et al. May 2002 B1
6416463 Tsuzuki et al. Jul 2002 B1
6429953 Feng Aug 2002 B1
6444970 Barbato Sep 2002 B1
6445022 Barna et al. Sep 2002 B1
6445139 Marshall et al. Sep 2002 B1
6464633 Hosoda Oct 2002 B1
6466618 Messing et al. Oct 2002 B1
6485414 Neuberger Nov 2002 B1
6512280 Chen et al. Jan 2003 B2
6567115 Miyashita et al. May 2003 B1
6627474 Barna et al. Sep 2003 B2
6631230 Campbell Oct 2003 B1
6659940 Adler Dec 2003 B2
6665013 Fossum et al. Dec 2003 B1
6677992 Matsumoto et al. Jan 2004 B1
6687534 Tsujita Feb 2004 B2
6690466 Miller et al. Feb 2004 B2
6692431 Kazakevich Feb 2004 B2
6707499 Kung et al. Mar 2004 B1
6772181 Fu et al. Aug 2004 B1
6773392 Kikuchi et al. Aug 2004 B2
6791739 Ramanujan et al. Sep 2004 B2
6796939 Hirata et al. Sep 2004 B1
6799065 Niemeyer Sep 2004 B1
6809358 Hsieh et al. Oct 2004 B2
6836288 Lewis Dec 2004 B1
6838653 Campbell et al. Jan 2005 B2
6841947 Berg-johansen Jan 2005 B2
6847399 Ang Jan 2005 B1
6856712 Fauver et al. Feb 2005 B2
6873363 Barna et al. Mar 2005 B1
6879340 Chevallier Apr 2005 B1
6899675 Cline et al. May 2005 B2
6900829 Orzawa et al. May 2005 B1
6906745 Fossum et al. Jun 2005 B1
6921920 Kazakevich Jul 2005 B2
6933974 Lee Aug 2005 B2
6947090 Komoro et al. Sep 2005 B2
6961461 MacKinnon et al. Nov 2005 B2
6970195 Bidermann et al. Nov 2005 B1
6977733 Denk et al. Dec 2005 B2
6982740 Adair et al. Jan 2006 B2
6998594 Gaines et al. Feb 2006 B2
6999118 Suzuki Feb 2006 B2
7009634 Iddan et al. Mar 2006 B2
7009648 Lauxtermann et al. Mar 2006 B2
7030904 Adair et al. Apr 2006 B2
7037259 Hakamata et al. May 2006 B2
7068878 Crossman-Bosworth et al. Jun 2006 B2
7071979 Ohtani et al. Jul 2006 B1
7079178 Hynecek Jul 2006 B2
7102682 Baer Sep 2006 B2
7105371 Fossum et al. Sep 2006 B2
7106377 Bean et al. Sep 2006 B2
7119839 Mansoorian Oct 2006 B1
7151568 Kawachi et al. Dec 2006 B2
7159782 Johnston et al. Jan 2007 B2
7184084 Glenn Feb 2007 B2
7189226 Auld et al. Mar 2007 B2
7189961 Johnston et al. Mar 2007 B2
7194129 Reinhart Mar 2007 B1
7208983 Imaizumi et al. Apr 2007 B2
7252236 Johnston et al. Aug 2007 B2
7258663 Doguchi et al. Aug 2007 B2
7261687 Yang Aug 2007 B2
7280139 Pahr et al. Oct 2007 B2
7298938 Johnston Nov 2007 B2
7312879 Johnston Dec 2007 B2
7319478 Dolt et al. Jan 2008 B2
7355155 Wang Apr 2008 B2
7356198 Chauville et al. Apr 2008 B2
7365768 Ono et al. Apr 2008 B1
7369140 King et al. May 2008 B1
7369176 Sonnenschein et al. May 2008 B2
7385708 Ackerman et al. Jun 2008 B2
7455638 Ogawa et al. Nov 2008 B2
7470229 Ogawa et al. Dec 2008 B2
7476197 Wiklof et al. Jan 2009 B2
7532760 Kaplinsky et al. May 2009 B2
7540645 Choi May 2009 B2
7544163 MacKinnon et al. Jun 2009 B2
7545434 Bean et al. Jun 2009 B2
7564935 Suzuki Jul 2009 B2
7567291 Bechtel et al. Jul 2009 B2
7573516 Krymski et al. Aug 2009 B2
7573519 Phan et al. Aug 2009 B2
7583872 Seibel et al. Sep 2009 B2
7608807 Hick et al. Oct 2009 B2
7616238 Avni et al. Nov 2009 B2
7630008 Sarwari Dec 2009 B2
7744528 Wallace et al. Jun 2010 B2
7783133 Dunki-Jacobs et al. Aug 2010 B2
7784697 Johnston et al. Aug 2010 B2
7791009 Johnston et al. Sep 2010 B2
7792378 Liege et al. Sep 2010 B2
7794394 Frangioni Sep 2010 B2
7813538 Carroll et al. Oct 2010 B2
7901974 Venezia et al. Mar 2011 B2
7914447 Kanai Mar 2011 B2
7916193 Fossum Mar 2011 B2
7935050 Luanava et al. May 2011 B2
7944566 Xie May 2011 B2
7952096 Rhodes May 2011 B2
7969097 Van De Ven Jun 2011 B2
7995123 Lee et al. Aug 2011 B2
8018589 MacKinnon et al. Sep 2011 B2
8040394 Fossum et al. Oct 2011 B2
8054339 Fossum et al. Nov 2011 B2
8059174 Mann et al. Nov 2011 B2
8100826 MacKinnon et al. Jan 2012 B2
8159584 Iwabuchi et al. Apr 2012 B2
8193542 Machara Jun 2012 B2
8194061 Wang et al. Jun 2012 B2
8212884 Seibel et al. Jul 2012 B2
8213698 Wang Jul 2012 B2
8231522 Endo et al. Jul 2012 B2
8300111 Iwane Oct 2012 B2
8372003 St. George et al. Feb 2013 B2
8382662 Soper et al. Feb 2013 B2
8396535 Wang et al. Mar 2013 B2
8423110 Barbato et al. Apr 2013 B2
8471938 Altice, Jr. et al. Jun 2013 B2
8476575 Mokhuatyuk Jul 2013 B2
8482823 Cheng Jul 2013 B2
8493474 Richardson Jul 2013 B2
8493564 Brukilacchio et al. Jul 2013 B2
8523367 Ogura Sep 2013 B2
8537203 Seibel et al. Sep 2013 B2
8559743 Liege et al. Oct 2013 B2
8582011 Dosluoglu Nov 2013 B2
8602971 Farr Dec 2013 B2
8605177 Rossi et al. Dec 2013 B2
8610808 Prescher et al. Dec 2013 B2
8614754 Fossum Dec 2013 B2
8625016 Fossum et al. Jan 2014 B2
8638847 Wang Jan 2014 B2
8648287 Fossum Feb 2014 B1
8649848 Crane et al. Feb 2014 B2
8668339 Kabuki et al. Mar 2014 B2
8675125 Cossairt et al. Mar 2014 B2
8698887 Makino et al. Apr 2014 B2
8836834 Hashimoto et al. Sep 2014 B2
8848063 Jo et al. Sep 2014 B2
8858425 Farr et al. Oct 2014 B2
8885034 Adair et al. Nov 2014 B2
8941308 Briggs Jan 2015 B2
9182337 Kamee et al. Nov 2015 B2
9349764 Lee et al. May 2016 B1
9509917 Blanquart et al. Nov 2016 B2
9516239 Blanquart et al. Dec 2016 B2
9634878 Bench et al. Apr 2017 B1
9762879 Blanquart et al. Sep 2017 B2
9777913 Talbert et al. Oct 2017 B2
10084944 Henley et al. Sep 2018 B2
10251530 Henley et al. Apr 2019 B2
10277875 Blanquart et al. Apr 2019 B2
10568496 Blanquart et al. Feb 2020 B2
10785461 Blanquart et al. Sep 2020 B2
10911649 Henley et al. Feb 2021 B2
10917562 Richardson et al. Feb 2021 B2
20010016064 Tsuruoka et al. Aug 2001 A1
20010017649 Yaron Aug 2001 A1
20010030744 Chang Oct 2001 A1
20010055462 Seibel Dec 2001 A1
20020045801 Niida et al. Apr 2002 A1
20020054219 Jaspers May 2002 A1
20020064341 Fauver et al. May 2002 A1
20020080248 Adair et al. Jun 2002 A1
20020080359 Denk et al. Jun 2002 A1
20020140844 Kurokawa et al. Oct 2002 A1
20020158976 Vni et al. Oct 2002 A1
20020158986 Baer Oct 2002 A1
20030007087 Hakamata et al. Jan 2003 A1
20030007686 Roever Jan 2003 A1
20030107664 Suzuki Jun 2003 A1
20030189663 Dolt et al. Oct 2003 A1
20030189705 Pardo Oct 2003 A1
20040082833 Adler et al. Apr 2004 A1
20040170712 Sadek El Mogy Sep 2004 A1
20040215059 Homan et al. Oct 2004 A1
20050009982 Inagaki et al. Jan 2005 A1
20050010081 Doguchi et al. Jan 2005 A1
20050027164 Barbato et al. Feb 2005 A1
20050038322 Banik Feb 2005 A1
20050041571 Ichihara et al. Feb 2005 A1
20050052680 Okamura Mar 2005 A1
20050113641 Bala May 2005 A1
20050122530 Denk et al. Jun 2005 A1
20050151866 Ando et al. Jul 2005 A1
20050169375 Pai Aug 2005 A1
20050200291 Naugler, Jr. et al. Sep 2005 A1
20050234302 MacKinnon et al. Oct 2005 A1
20050237384 Jess et al. Oct 2005 A1
20050261552 Mori et al. Nov 2005 A1
20050267328 Blumzvig et al. Dec 2005 A1
20050267329 Konstorum et al. Dec 2005 A1
20050277808 Sonnenschein et al. Dec 2005 A1
20050288546 Sonnenschein et al. Dec 2005 A1
20060038823 Arcas Feb 2006 A1
20060069314 Farr Mar 2006 A1
20060087841 Chern et al. Apr 2006 A1
20060106284 Shouji et al. May 2006 A1
20060197664 Zhang et al. Sep 2006 A1
20060202036 Wang et al. Sep 2006 A1
20060221250 Rossbach et al. Oct 2006 A1
20060226231 Johnston et al. Oct 2006 A1
20060264734 Kimoto et al. Nov 2006 A1
20060274335 Wittenstein Dec 2006 A1
20070010712 Negishi Jan 2007 A1
20070010713 Negishi Jan 2007 A1
20070029629 Yazdi Feb 2007 A1
20070041448 Miller Feb 2007 A1
20070066868 Shikii Mar 2007 A1
20070083085 Bimkrant et al. Apr 2007 A1
20070092283 Sugihara Apr 2007 A1
20070129601 Johnston et al. Jun 2007 A1
20070147033 Ogawa et al. Jun 2007 A1
20070182723 Imai et al. Aug 2007 A1
20070182842 Sonnenschein et al. Aug 2007 A1
20070225560 Avni et al. Sep 2007 A1
20070244364 Luanava et al. Oct 2007 A1
20070244365 Wiklof Oct 2007 A1
20070274649 Takahashi et al. Nov 2007 A1
20070276187 Wiklof et al. Nov 2007 A1
20070279486 Bayer et al. Dec 2007 A1
20070285526 Mann Dec 2007 A1
20070293720 Bayer Dec 2007 A1
20080045800 Farr Feb 2008 A2
20080049132 Suzuki Feb 2008 A1
20080088719 Jacob et al. Apr 2008 A1
20080107333 Mazinani et al. May 2008 A1
20080136953 Barnea Jun 2008 A1
20080158348 Karpen et al. Jul 2008 A1
20080164550 Chen et al. Jul 2008 A1
20080165360 Johnston Jul 2008 A1
20080167523 Uchiyama et al. Jul 2008 A1
20080192131 Kim et al. Aug 2008 A1
20080208077 Iddan et al. Aug 2008 A1
20080218598 Harada et al. Sep 2008 A1
20080218615 Huang et al. Sep 2008 A1
20080218824 Johnston et al. Sep 2008 A1
20080249369 Seibel et al. Oct 2008 A1
20080287742 St. George et al. Nov 2008 A1
20090012361 MacKinnon et al. Jan 2009 A1
20090012368 Banik Jan 2009 A1
20090021588 Border et al. Jan 2009 A1
20090021618 Schwarz et al. Jan 2009 A1
20090024000 Chen Jan 2009 A1
20090028465 Pan Jan 2009 A1
20090074265 Huang et al. Mar 2009 A1
20090091645 Trimeche et al. Apr 2009 A1
20090137893 Seibel et al. May 2009 A1
20090147077 Tani et al. Jun 2009 A1
20090154886 Lewis et al. Jun 2009 A1
20090160976 Chen et al. Jun 2009 A1
20090189530 Ashdown et al. Jul 2009 A1
20090208143 Yoon et al. Aug 2009 A1
20090227847 Tepper et al. Sep 2009 A1
20090232213 Jia Sep 2009 A1
20090259102 Koninckx et al. Oct 2009 A1
20090268063 Ellis-Monaghan et al. Oct 2009 A1
20090274380 Wedi Nov 2009 A1
20090292168 Farr Nov 2009 A1
20090309500 Reisch Dec 2009 A1
20090316116 Melville et al. Dec 2009 A1
20090322912 Blanquart Dec 2009 A1
20100004513 MacKinnon et al. Jan 2010 A1
20100026722 Kondo Feb 2010 A1
20100049180 Wells et al. Feb 2010 A1
20100069713 Endo et al. Mar 2010 A1
20100102199 Negley et al. Apr 2010 A1
20100121142 OuYang et al. May 2010 A1
20100121143 Sugimoto et al. May 2010 A1
20100123775 Shibasaki May 2010 A1
20100134608 Shibasaki Jun 2010 A1
20100134662 Bub Jun 2010 A1
20100135398 Wittmann et al. Jun 2010 A1
20100137684 Shibasaki et al. Jun 2010 A1
20100149421 Lin et al. Jun 2010 A1
20100157037 Iketani et al. Jun 2010 A1
20100157039 Sugai Jun 2010 A1
20100165087 Corso et al. Jul 2010 A1
20100171429 Garcia et al. Jul 2010 A1
20100182446 Matsubayashi Jul 2010 A1
20100198009 Farr et al. Aug 2010 A1
20100198134 Eckhouse et al. Aug 2010 A1
20100201797 Shizukuishi et al. Aug 2010 A1
20100208056 Olsson et al. Aug 2010 A1
20100228089 Hoffman et al. Sep 2010 A1
20100261961 Scott et al. Oct 2010 A1
20100274082 Iguchi et al. Oct 2010 A1
20100274090 Ozaki et al. Oct 2010 A1
20100305406 Braun et al. Dec 2010 A1
20100309333 Smith et al. Dec 2010 A1
20110028790 Farr et al. Feb 2011 A1
20110034769 Adair et al. Feb 2011 A1
20110051390 Lin et al. Mar 2011 A1
20110063483 Rossi et al. Mar 2011 A1
20110122301 Yamura et al. May 2011 A1
20110149358 Cheng Jun 2011 A1
20110181709 Wright et al. Jul 2011 A1
20110181840 Cobb Jul 2011 A1
20110184239 Wright et al. Jul 2011 A1
20110184243 Wright et al. Jul 2011 A1
20110208004 Feingold et al. Aug 2011 A1
20110212649 Stokoe et al. Sep 2011 A1
20110237882 Saito Sep 2011 A1
20110237884 Saito Sep 2011 A1
20110245605 Jacobsen et al. Oct 2011 A1
20110245616 Kobayashi Oct 2011 A1
20110255844 Wu et al. Oct 2011 A1
20110274175 Sumitomo Nov 2011 A1
20110279679 Samuel et al. Nov 2011 A1
20110288374 Hadani et al. Nov 2011 A1
20110291564 Huang Dec 2011 A1
20110292258 Adler et al. Dec 2011 A1
20110295061 Haramaty et al. Dec 2011 A1
20120004508 McDowall et al. Jan 2012 A1
20120014563 Bendall Jan 2012 A1
20120016200 Seto et al. Jan 2012 A1
20120029279 Kucklick Feb 2012 A1
20120033118 Lee et al. Feb 2012 A1
20120041267 Benning et al. Feb 2012 A1
20120041534 Clerc et al. Feb 2012 A1
20120050592 Oguma Mar 2012 A1
20120078052 Cheng Mar 2012 A1
20120098933 Robinson et al. Apr 2012 A1
20120104230 Eismann et al. May 2012 A1
20120113506 Gmitro et al. May 2012 A1
20120120282 Goris May 2012 A1
20120140302 Xie et al. Jun 2012 A1
20120155761 Matsuoka Jun 2012 A1
20120157774 Kaku Jun 2012 A1
20120172665 Alyn et al. Jul 2012 A1
20120194686 Liu et al. Aug 2012 A1
20120197080 Murayama Aug 2012 A1
20120200685 Kawasaki et al. Aug 2012 A1
20120209071 Bayer et al. Aug 2012 A1
20120242975 Min et al. Sep 2012 A1
20120262621 Sato et al. Oct 2012 A1
20120281111 Jo et al. Nov 2012 A1
20120296238 Chernov et al. Nov 2012 A1
20120319586 Riesebosch Dec 2012 A1
20120327270 Shirakawa et al. Dec 2012 A1
20130016200 Ovod Jan 2013 A1
20130018256 Kislev et al. Jan 2013 A1
20130035545 Ono Feb 2013 A1
20130053642 Mizuyoshi et al. Feb 2013 A1
20130070071 Peltie et al. Mar 2013 A1
20130126708 Blanquart May 2013 A1
20130127934 Chiang May 2013 A1
20130135589 Curtis et al. May 2013 A1
20130144120 Yamazaki Jun 2013 A1
20130155215 Shimada et al. Jun 2013 A1
20130155305 Shintani Jun 2013 A1
20130158346 Soper et al. Jun 2013 A1
20130184524 Shimada et al. Jul 2013 A1
20130211217 Yamaguchi et al. Aug 2013 A1
20130242069 Kobayashi Sep 2013 A1
20130244453 Sakamoto Sep 2013 A1
20130274597 Byrne et al. Oct 2013 A1
20130289347 Ito et al. Oct 2013 A1
20130292571 Mukherjee et al. Nov 2013 A1
20130296651 Ito et al. Nov 2013 A1
20130296652 Farr Nov 2013 A1
20130300837 DiCarlo et al. Nov 2013 A1
20130342690 Williams et al. Dec 2013 A1
20140012078 Coussa Jan 2014 A1
20140022365 Yoshino Jan 2014 A1
20140031623 Kagaya Jan 2014 A1
20140005532 Choi et al. Feb 2014 A1
20140052004 D'Alfonso et al. Feb 2014 A1
20140066711 Farin et al. Mar 2014 A1
20140073852 Banik et al. Mar 2014 A1
20140073853 Swisher et al. Mar 2014 A1
20140078278 Lei Mar 2014 A1
20140088363 Sakai et al. Mar 2014 A1
20140094649 Ito Apr 2014 A1
20140104466 Fossum Apr 2014 A1
20140110485 Toa et al. Apr 2014 A1
20140142383 Blumenzweig et al. May 2014 A1
20140160318 Blanquart et al. Jun 2014 A1
20140163319 Blanquart et al. Jun 2014 A1
20140198249 Tanaka et al. Jul 2014 A1
20140203084 Wang Jul 2014 A1
20140225215 Chien et al. Aug 2014 A1
20140267655 Richardson et al. Sep 2014 A1
20140267851 Rhoads Sep 2014 A1
20140267890 Lelescu et al. Sep 2014 A1
20140268860 Talbert et al. Sep 2014 A1
20140275764 Shen et al. Sep 2014 A1
20140288365 Henley et al. Sep 2014 A1
20140300698 Wany Oct 2014 A1
20140316197 St. George et al. Oct 2014 A1
20140316199 Kucklick Oct 2014 A1
20140354788 Yano Dec 2014 A1
20140364689 Adair et al. Dec 2014 A1
20150023611 Salvador et al. Jan 2015 A1
20150237245 Renard et al. Aug 2015 A1
20150271370 Henley et al. Sep 2015 A1
20160183775 Blanquart et al. Jun 2016 A1
20170085853 Blanquart et al. Mar 2017 A1
20170230574 Richardson et al. Aug 2017 A1
20190028621 Henley et al. Jan 2019 A1
20190133416 Henley et al. May 2019 A1
20190174058 Richardson et al. Jun 2019 A1
20190253685 Blanquart et al. Aug 2019 A1
20200292160 Talbert et al. Sep 2020 A1
Foreign Referenced Citations (69)
Number Date Country
1520696 Aug 2004 CN
101079966 Nov 2007 CN
201239130 May 2009 CN
101449575 Jun 2009 CN
101634749 Jan 2010 CN
101755448 Jun 2010 CN
102469932 May 2012 CN
103185960 Jul 2013 CN
0660616 Jun 1995 EP
0904725 Mar 1999 EP
1079255 Feb 2001 EP
1116473 Jul 2001 EP
1637062 Mar 2006 EP
1712177 Oct 2006 EP
1819151 Aug 2007 EP
2359739 Aug 2011 EP
2371268 Aug 2011 EP
2478693 Jul 2012 EP
3459431 Mar 2014 EP
236893 Aug 2020 IL
H04-039789 Apr 1992 JP
H07-240931 Sep 1995 JP
1995-240931 Mar 1997 JP
2000-051150 Feb 2000 JP
2000-199863 Jul 2000 JP
2000270230 Sep 2000 JP
2001190489 Jul 2001 JP
2001-308531 Nov 2001 JP
2002-020816 Jan 2002 JP
2002-028125 Jan 2002 JP
2002-045329 Feb 2002 JP
2002-112961 Apr 2002 JP
2005-204741 Aug 2005 JP
2007-029746 Feb 2007 JP
2007143963 Jun 2007 JP
2007-240931 Sep 2007 JP
2008514304 May 2008 JP
2008-153313 Jul 2008 JP
2008264539 Nov 2008 JP
2008295929 Dec 2008 JP
2009-537283 Oct 2009 JP
2010-17377 Jan 2010 JP
2010-068992 Apr 2010 JP
2010-125284 Jun 2010 JP
2010-158415 Jul 2010 JP
2011-055327 Mar 2011 JP
2011514605 May 2011 JP
2011514605 May 2011 JP
2012-000160 Jan 2012 JP
2012024450 Feb 2012 JP
2013-27432 Feb 2013 JP
2011-267098 Jun 2013 JP
2014514782 Jun 2014 JP
5682812 Jan 2015 JP
2015525642 Sep 2015 JP
A61-62440 Jun 2017 JP
6804488 Dec 2020 JP
2015001195 Jan 2016 MX
346174 Mar 2017 MX
1996005693 Feb 1996 WO
2006037034 Apr 2006 WO
2009120228 Jan 2009 WO
2009018613 Feb 2009 WO
2009045235 Apr 2009 WO
2009115885 Sep 2009 WO
2009120228 Oct 2009 WO
2012043771 Apr 2012 WO
2012043771 May 2012 WO
2012137845 Oct 2012 WO
Non-Patent Literature Citations (2)
Entry
Jack, Keith “Video Demystified: A Handbook for the Digital Engineer,” 2007 Fifth Edition. ISBN: 978-0-7506-8395-1, p. 21.
Blumenfeld, et al. Three-dimensional image registration of MR proximal femur images for the analysis of trabecular bone parameters. Oct. 2008. [retrieved on Jul. 30, 2014] Retrieved from the internet: <URL: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2673590/>.
Related Publications (1)
Number Date Country
20200358994 A1 Nov 2020 US
Provisional Applications (4)
Number Date Country
61791473 Mar 2013 US
61790487 Mar 2013 US
61790719 Mar 2013 US
61676289 Jul 2012 US
Divisions (1)
Number Date Country
Parent 13952570 Jul 2013 US
Child 15369170 US
Continuations (3)
Number Date Country
Parent 16398114 Apr 2019 US
Child 16940189 US
Parent 15701264 Sep 2017 US
Child 16398114 US
Parent 15369170 Dec 2016 US
Child 15701264 US