Patent Application
20250183220
The present disclosure relates to an electronic component and a device including the electronic component.
Japanese Patent Application Laid-Open No. 2006-191465 discusses an electronic device including a Peltier element and an image sensor.
According to some embodiments of the present disclosure, an electronic component includes a cooling member, a semiconductor substrate, and a base member in which the cooling member and the semiconductor substrate are placed, wherein the cooling member is arranged between the base member and the semiconductor substrate, wherein the semiconductor substrate includes a first electrode, wherein the base member includes a second electrode, wherein the first electrode and the second electrode are connected by a conductive wire, and wherein balls are formed on both of a bonded portion of the first electrode and the conductive wire, and a bonded portion of the second electrode and the conductive wire.
Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
In order to prevent interference between a bonding capillary and a substrate when performing wire bonding of the substrate on which an image sensor is arranged and a package base member in which the substrate is to be placed, it is difficult to reduce the size of an electronic component. The technique of the present disclosure provides an electronic component that allows wire bonding between a substrate and a package base member in which the substrate is to be placed to be desirably performed.
The following exemplary embodiments are provided to embody the technical idea of the present disclosure and are not intended to limit the present disclosure. The sizes and the positional relationship of members illustrated in the drawings are sometimes exaggerated to clarify the description. In the following description, the same components are assigned the same reference numerals, and the description thereof will be sometimes omitted.
Hereinafter, various exemplary embodiments, features, and aspects of the present disclosure will be described in detail with reference to the drawings. In the following description, terms (e.g., “up”, “down”, “right”, “left”, and other terms including these terms) indicating specific directions and positions are used as appropriate. These terms are used to facilitate the understanding of the exemplary embodiments to be described with reference to the drawings. The technical scope of the present disclosure is not limited by the meanings of these terms.
In this specification, a “planar view” refers to a view from a direction vertical to a light incidence surface of a semiconductor layer. A cross-sectional view refers to a view of a surface in the direction vertical to the light incidence surface of the semiconductor layer. In a case where the light incidence surface of the semiconductor layer is a rough surface when viewed microscopically, a planar view is defined based on the light incidence surface of the semiconductor layer that when viewed macroscopically.
The semiconductor layer has a first surface that light enters, and a second surface on the opposite side of the first surface. In this specification, a depth direction refers to a direction extending from the first surface to the second surface of the semiconductor layer on which a photodiode (PD) is arranged. A “depth” of a certain point or a certain region in the semiconductor layer means a distance of the point or the region from the first surface. When a point (or region) Z1 with a distance (depth) d1 from the first surface and a point (or region) Z2 with a distance (depth) d2 from the first surface exist, and d1>d2 is satisfied, such a state is sometimes represented as “the point Z1 is deeper than the point Z2” or “the point Z2 is shallower than the point Z1”. When there additionally exists a point (or region) Z3 with a distance (depth) d3 from the first surface, and d1>d3>d2 is satisfied, such a state is sometimes represented as “the point Z3 exists between the depths of the points Z1 and Z2” or “the point Z3 exists between the points Z1 and Z2 in the depth direction”.
An electronic component according to a first exemplary embodiment of the present disclosure will be described with reference to
In an electronic component equipped with an image sensor, noise is sometimes generated due to a temperature change in the image sensor. For noise reduction, a cooling member for cooling a semiconductor chip on which the image sensor is placed can be further placed in a package in which the semiconductor chip is to be placed. As an example of a member to be used in the cooling of the semiconductor chip, there is a Peltier element. In the following exemplary embodiment, an electronic component including an image sensor and a Peltier element will be described, but the cooling member is not limited to the Peltier element and may be a cooling fin or a heat pipe, for example.
The electronic component includes a package 100 including abase member 101, and an optical member 102 bonded to a frame member of the base member 101 by an optical member adhesive 108. A Peltier element 106 is bonded to a bottom portion of the base member 101 by a Peltier adhesive 105, and furthermore, a semiconductor substrate 103 is bonded by a semiconductor substrate adhesive 107. The semiconductor substrate 103 is electrically connected with the base member 101 by a conductive wire 109.
The base member 101 is formed of alumina or ceramic, such as aluminum nitride, as a main material. This is because such a material has high thermal conductivity, so that heat generated in the Peltier element 106 can be easily released to the outside of the package 100.
The optical member 102 is formed of glass, crystal, or sapphire, for example. The crystal and the sapphire also function as a low-pass filter (LPF). The sapphire has high strength and can be made thin as compared with the crystal. In other words, the sapphire is advantageous in reducing the size of the entire package 100. Because a linear expansion coefficient of the sapphire is nearly equal to a linear expansion coefficient of the alumina, if the optical member 102 is made of sapphire in a case where the base member 101 is made of alumina, bonding reliability is high.
The semiconductor substrate 103 is a silicon substrate, for example, and is provided with a pixel region 104 in which a plurality of image sensors is arranged in an array. The image sensor may be, for example, a complementary metal-oxide semiconductor (CMOS) image sensor or may be an avalanche diode. In a case where the image sensor is an avalanche diode, the avalanche diode may be a single photon avalanche diode (SPAD).
The optical member adhesive 108 is an epoxy adhesive, for example. The optical member adhesive 108 may be an ultraviolet cure adhesive or a thermosetting adhesive, but the optical member adhesive 108 is desirably an adhesive with low moisture permeability that can secure airtightness, because the inside of the base member 101 is kept under N2 atmosphere or in a vacuum state for heat insulation.
In order to keep low moisture permeability, it is desirable that an adhesive thickness of the optical member adhesive 108 is thin and an adhesive width of the optical member adhesive 108 is wide. For example, the thickness of the optical member adhesive 108 is desirably 20 micrometers (μm) to 30 μm or less. The adhesive width can be set from the viewpoint of the size reduction, bonding reliability, and moisture permeability of the package 100.
The Peltier adhesive 105 and the semiconductor substrate adhesive 107 are desirably made of material with high thermal conductivity, such as silver paste. Because a space generated between bonded surfaces hinders thermal conduction, these adhesives are desirably applied to the bonded surfaces as wide as possible, or the entire surfaces. Adhesive thicknesses of the Peltier adhesive 105 and the semiconductor substrate adhesive 107 are each about 100 μm or less, and are desirably about 20 μm to 30 μm in particular.
At the center of the electronic component, the pixel region 104 of the rectangular semiconductor substrate 103 is arranged, and a plurality of conductive wires 109 is provided on each side. When a length in a longer direction of the electronic component is denoted by x, and a length in a shorter direction is denoted by y, the electronic component is assumed to have a size with about 20 millimeters (mm) of the lengths x and y, but the size of the electronic component is not limited to this.
The configuration of an electrode for connecting to the outside of the package 100 may be the LGA or the LCC, or may be a pin grid array (PGA). From the viewpoint of reducing the size, the LGA is desirable because a low height can be realized as compared with other configurations. Alternatively, a configuration obtained by combining the LGA and the LCC may be employed.
In a case where the LGA is employed as the configuration of the package 100, it can be manufactured by using a reflow furnace, and improvement in productivity can be expected as compared with other methods. At this time, in order to prevent the Peltier element 106 from being damaged due to the melting of solder contained in the Peltier element 106 in the reflow processing, the temperature of the reflow furnace to be used in the manufacturing of the electronic component according to the present exemplary embodiment needs to be low temperature (200° C. (Celsius) or less). For this reason, the bonding with the outside is implemented using material with a low melting point, such as resin reinforced solder.
As a terminal array of the LGA, a portion with no terminal is desirably provided at the center portion as illustrated in
A first electrode 110 provided on the semiconductor substrate 103 and a second electrode 111 provided on the base member 101 are connected by the conductive wire 119. When a region of the bottom portion of the base member 101 in which the frame member is provided is regarded as a first region, and a region in which the Peltier element 106 is placed is regarded as a second region, the second electrode 111 is arranged between the first region and the second region. In
In the electronic component in which the Peltier element 106 is placed together with the semiconductor substrate 103 as in the present exemplary embodiment, the conductive wire 119 connecting the semiconductor substrate 103 and the base member 101 desirably has a fixed length in order to prevent a return heat flow from the Peltier element 106, and the second electrode 111 is desirably provided at a low position to ensure a wire length.
A distance in a vertical direction between the surface on which the second electrode 111 is provided and an uppermost part of the conductive wire 119 is 1 mm or more. The distance in the vertical direction of the conductive wire 119 is determined based on the thickness of the Peltier element 106 and the chip thickness of the semiconductor substrate 103. In the present exemplary embodiment, in order to maintain the thickness of the semiconductor substrate 103, backgrinding is not performed, and the semiconductor substrate 103 has a thickness of about 0.7 mm to 0.8 mm. In a case where backgrinding is not performed, thermal conductivity in a direction parallel to the flat surface of the semiconductor substrate 103 becomes high, and an effect of uniformly keeping a temperature distribution within the surface of the image sensor becomes high, which is more desirable. In the case of performing backgrinding on the semiconductor substrate 103, by setting the substrate thickness to about 0.5 mm, the distance in the vertical direction is ensured.
Similarly, the distance in the vertical direction of the conductive wire 119 can be also extended by increasing the thickness of the Peltier element 106. Nevertheless, if the Peltier element 106 is thick, a capillary to be used in wire bonding might interfere with the base member 101.
The conductive wire 119 is desirably thinner than a wire to be generally used in an image sensor not equipped with the Peltier element 106. For example, the size of the conductive wire 119 is desirably Φ15 μm. This is because when wire thermal resistance becomes high, it is possible to prevent a heat flow from returning to the semiconductor substrate 103 from the Peltier element 106 via the base member 101. Furthermore, in order to enhance thermal resistance, a wire to be used as the conductive wire 119 is desirably a wire with thermal conductivity smaller than 300 W/mK (Watts per meter Kelvin) that contains a gold alloy or aluminum as a main material, rather than a gold wire. To prevent a wire breakage, a wire diameter of the conductive wire 119 is determined based on balance between an allowable current amount, resistance, and inductance.
On the other hand, because power consumption of the Peltier element 106 is large, a wire diameter of a wire electrically connecting the Peltier element 106 and the base member 101 is desirably thick. A member connecting the Peltier element 106 and the base member 101 is not limited to the wire and may be a conductive adhesive, such as silver paste, or solder.
The minimum clearance between the semiconductor substrate 103 and the capillary varies depending on the size of the semiconductor substrate 103 and the position of the second electrode 111. In a case where the Peltier element 106 with the same thickness B as that in
As illustrated in
A space desirable in a case where normal wire bonding is performed will be further described with reference to
By forming a ball on the first electrode 110 and performing pressure bonding while applying heat and ultrasonic waves, the first electrode 110 and the conductive wire 109 are bonded. After that, a bend point is formed in the conductive wire 109, and the conductive wire 109 is drawn out by a desirable length. By pressure-bonding the conductive wire 109 to the second electrode 111 while maintaining tension in such a manner as to prevent the drawn the conductive wire 109 from getting loose, a stitch is formed at the bonded portion. In a case where the height of the second electrode 111 is lower than the height of the semiconductor substrate 103, if a wall surface of the base member 101 exists on a trajectory drawn by the capillary, the capillary and the base member 101 interfere with each other. To prevent the interference, a space in the horizontal direction is desirable between the second electrode 111 and the end portion of the base member 101, and this leads to an increase in the size of the electronic component.
An electronic component according to a second exemplary embodiment will be described with reference to
A Peltier element 106 to be mounted on the electronic component according to the present exemplary embodiment will be described with reference to
One of the upper and lower heatsinks is a heat absorption surface 201, and the other one is a heat generation surface 202. In the electronic component according to the present exemplary embodiment of the present disclosure, the heat absorption surface 201 faces the semiconductor substrate 103 and the heat generation surface 202 faces the base member 101. The heatsinks are made of alumina or aluminum nitride, and their thermal conductivity may be increased by processing, such as gold plating processing. If the materials of the base member 101 and the heatsinks are the same, a linear expansion coefficient difference is not generated, which is advantageous in terms of stress. The Peltier element 106 has a conducting space including electrodes 203 and 204 for conduction with the outside. For example, the electrode 203 is an electrode connected with a power source, and the electrode 204 is an electrode connected with the ground.
When a voltage V is applied to the Peltier element 106, a current I flows. A heat absorption surface temperature is denoted by Tc and a heat generation surface temperature is denoted by Th. When a Seebeck coefficient is denoted by α, an internal resistance is denoted by R, a thermal conductivity is denoted by λ, and a temperature difference between the heat absorption surface temperature and the heat release surface temperature is denoted by ΔT, a heat absorption amount Qc is represented by the following equation.
In other words, the thermal conductivity λ and the internal resistance R are desirably small in order to increase the heat absorption amount Qc of the Peltier element 106.
Variations of the Peltier element 106 will be described with reference to
In an electronic component illustrated in
In an electronic component illustrated in
In an electronic component illustrated in
In a standard Peltier element, the smaller an area of a heatsink is, the smaller the thermal conductivity λ can be. Nevertheless, because the heat absorption amount Qc accordingly decreases, it is important to select a Peltier element including an appropriate number of P-type semiconductors and an appropriate number of N-type semiconductors.
In a case where a targeted cooling temperature of the image sensor is denoted by Tc, a temperature of the base member 101 on the heat release side is denoted by Th, and a thermal resistance of a plurality of conductive wires 109 is denoted by Rw, heat Pw returning to the semiconductor substrate 103 from the base member 101 through the entire conductive wire 109 is represented by (Th−Tc)/Rw. To keep the cooling temperature Tc of the image sensor constant, the sum of power consumption Ps of the image sensor and the heat Pw that transmits through the conductive wire 109 is to be equal to the heat absorption amount Qc of the Peltier element 106.
In order to save power consumption Pp of the Peltier element 106 under such a condition, it is desirable to select a Peltier element 106 including P-type semiconductors and N-type semiconductors of which the numbers make COP=Qc/Pp, which indicates a ratio between the heat absorption amount Qc and the power consumption of the Peltier element 106, the largest. In a case where a widely mass-produced bismuth telluride Peltier element is used in the cooling of the image sensor, as illustrated in
Nevertheless, in a case where the first electrode 110 is bonded at a position overlapping, in a planar view, a hollow region where the Peltier element 106 is not arranged below the semiconductor substrate 103, heat and ultrasonic waves may not be easily transmitted to a bonded portion from a stage for heating that is provided below the package 100.
Because the Peltier element 106 is small and a region holding the semiconductor substrate 103 is narrow, when wire bonding is performed, load may be applied to the bonded surface of the semiconductor substrate 103 and the Peltier element 106 according to the principle of the wire bonding. Furthermore, temperature unevenness may be generated on the surface of the semiconductor substrate 103.
A modified example of the Peltier element 106 will be further described with reference to
Also in this case, the cooling capacity, the thermal resistance, and the COP remain unchanged from those of the Peltier elements 106 illustrated in
In such a configuration, wire bonding can be easily performed, and it is possible to prevent temperature unevenness and ensure strength.
A copper (Cu) alloy is desirably used as the heat spreader 205. The heat spreader 205 desirably has a thickness of about 0.5 mm, and desirably exists over the entire region below the semiconductor substrate 103. In this case, in order to prevent the thickness from being increased, the thickness of the semiconductor substrate 103 may be decreased by backgrinding.
Even with such a configuration, it is possible to facilitate wire bonding and also obtain effects of preventing temperature unevenness and ensuring strength.
In the configuration illustrated in
In a case where a resin frame is used as the frame portion 206, because its heat conductivity is lower than those of a ceramic frame or a metal ceramic, for example, heat is less likely to be transmitted through the resin frame than these frames. As compared with a case where a hollow space exists without the frame portion 206 provided below the semiconductor substrate 103, heat inflow from the heat generation surface of the Peltier element 106 increases. For this reason, the frame portion 206 desirably has a width as thin as possible. Specifically, when the width of the first electrode 110 is about 50 μm to 100 μm, it is desirable that the width of the frame portion 206 is set up to a size of about ten times the electrode size (about 0.5 mm to 1 mm) in such a manner that a slight deviation can be allowed.
As an example of the arrangement of the Peltier element 106,
In
As illustrated in
An electronic component according to a third exemplary embodiment will be described with reference to
A configuration example of the conductive wire 109 and the second conductive wire 112 according to the present exemplary embodiment will be described.
In the above-described configuration example, the conductive wire 109 and the second conductive wire 112 may be made of the same material or may be made of different materials. As described in the first exemplary embodiment, in order to enhance the effect of suppressing a heat flow by increasing thermal resistance, a wire with thermal conductivity smaller than 300 W/mK that contains a gold alloy or aluminum as a main material is desirable rather than a gold wire.
An electronic component according to a fourth exemplary embodiment will be described with reference to
In the electronic component illustrated in the first exemplary embodiment, the base member 101 and the optical member 102 are sealed by an adhesive. For heat insulation of the inside of the package, vacuum sealing is performed in some cases. For the sake of upper temperature limits of the pixel region 104 and the Peltier element 106, sealing cannot be performed at high temperature. In view of the foregoing, seam weld sealing may be performed in which the base member 101 is pressure-bonded to an optical member 302 equipped with a metal frame 301 by applying heat and load to the optical member 302. By the seam weld sealing, vacuum sealing can be performed within the upper temperature limits of the pixel region 104 and the Peltier element 106.
An electronic component according to the fifth exemplary embodiment will be described with reference to
In a case where the LGA is employed as a package configuration, the package is mounted on a secondary substrate by reflow processing. Here, the secondary substrate is the PCB substrate 401, for example. The PCB substrate 401 may have an opening portion as illustrated in
As described above, in order to prevent the Peltier element 106 from being damaged due to the melting of solder contained in the Peltier element 106 in the reflow processing, a material with a low melting point, such as resin reinforced solder that can be used in a reflow furnace at low temperature of 200° C. or less, more desirably 180° C. or less, is used.
The weight of the electronic component according to the present exemplary embodiment is heavy because the Peltier element 106 is mounted. Accordingly, there is a concern of a short circuit (
As illustrated in
As a variation of the spacer 404, as illustrated in
In contrast, as illustrated in
A photoelectric conversion system according to a sixth exemplary embodiment will be described with reference to
Any of the electric components (hereinafter also referred to as photoelectric conversion apparatuses) described in the above-described first to fifth exemplary embodiments can be applied as a photoelectric conversion apparatus to various photoelectric conversion systems. Examples of photoelectric conversion systems to which the electric component can be applied include a digital still camera, a digital camcorder, a monitoring camera, a copier, a facsimile, a mobile phone, an in-vehicle camera, and an observation satellite. A camera module including an optical system, such as a lens, and an imaging apparatus is also included in the examples of photoelectric conversion systems. As an example of these photoelectric conversion systems,
The photoelectric conversion system exemplified in
The photoelectric conversion system further includes a signal processing unit 1007 serving as an image generation unit that generates an image by processing an output signal output by the imaging apparatus 1004. The signal processing unit 1007 performs an operation of outputting image data after performing various types of correction and compression as desired. The signal processing unit 1007 may be formed on a semiconductor substrate on which the imaging apparatus 1004 is provided or may be formed on a semiconductor substrate different from the semiconductor substrate on which the imaging apparatus 1004 is provided. The imaging apparatus 1004 and the signal processing unit 1007 may be formed on the same semiconductor substrate.
The photoelectric conversion system further includes a memory unit 1010 for temporarily storing image data, and an external interface unit (external I/F unit) 1013 for communicating with an external computer. The photoelectric conversion system further includes a recording medium 1012, such as a semiconductor memory, for recording or reading out captured image data, and a recording medium control interface unit (recording medium control I/F unit) 1011 for performing recording onto or readout from the recording medium 1012. The recording medium 1012 may be built into the photoelectric conversion system, or may be detachably attached to the photoelectric conversion system.
The photoelectric conversion system further includes an overall control/calculation unit 1009 that controls various types of calculation and the entire digital still camera, and a timing signal generation unit 1008 that outputs various timing signals to the imaging apparatus 1004 and the signal processing unit 1007. The timing signals may be input from the outside. The photoelectric conversion system preferably includes at least the imaging apparatus 1004 and the signal processing unit 1007 that processes an output signal output from the imaging apparatus 1004.
The imaging apparatus 1004 outputs an imaging signal to the signal processing unit 1007. The signal processing unit 1007 outputs image data after performing predetermined signal processing on the imaging signal output from the imaging apparatus 1004. The signal processing unit 1007 generates an image using the imaging signal.
In this manner, according to the present exemplary embodiment, a photoelectric conversion system to which the photoelectric conversion apparatus (imaging apparatus) according to any of the above-described exemplary embodiments is applied can be realized.
A photoelectric conversion system and a movable body according to a seventh exemplary embodiment will be described with reference to
The photoelectric conversion system 1300 is connected with a vehicle information acquisition apparatus 1320 and can acquire vehicle information, such as a vehicle speed, a yaw rate, or a rudder angle. In addition, an electronic control unit (ECU) 1330 is connected to the photoelectric conversion system 1300. The ECU 1330 serves as a control apparatus that outputs a control signal for generating braking force for a vehicle based on a determination result obtained by the collision determination unit 1318. The photoelectric conversion system 1300 is also connected with an alarm apparatus 1340 that issues an alarm to a driver based on a determination result obtained by the collision determination unit 1318. For example, in a case where the determination result obtained by the collision determination unit 1318 indicates a high likelihood of collision, the ECU 1330 performs vehicle control to avoid collision or reduce damages by braking, releasing an accelerator, or decreasing engine output. The alarm apparatus 1340 issues an alarm to a user by issuing an alarm, such as sound, displaying warning information on a screen of a car navigation system, or vibrating a seatbelt or a steering wheel.
In the present exemplary embodiment, the photoelectric conversion system 1300 captures an image of the periphery of the vehicle, such as the front side or the rear side, for example.
The above description has been given of an example in which control is performed in such a manner as not to collide with another vehicle. The photoelectric conversion system can also be applied to the control for performing automatic operation by following another vehicle, or the control for performing automatic operation in such a manner as not to drift out of a lane. Furthermore, the photoelectric conversion system can be applied to a movable body (moving apparatus), such as a vessel, an aircraft, or an industrial robot, for example, in addition to a vehicle, such as an automobile. This movable body includes either one or both of a drive force generation unit that generates drive force to be mainly used for the movement of the movable body, and a rotator to be mainly used for the movement of the movable body. The drive force generation unit can be an engine, a motor, or the like. The rotator can be a tire, a wheel, a screw of a ship, a propeller of a flight vehicle, or the like. Moreover, the photoelectric conversion system can be applied to a device that extensively uses object recognition, such as an intelligent transport system (ITS), in addition to a movable body.
A photoelectric conversion system according to an eighth exemplary embodiment will be described with reference to
As illustrated in
The optical system 402 includes one or a plurality of lenses. The optical system 402 forms an image on a light receiving surface (sensor portion) of the photoelectric conversion apparatus 408 while guiding image light (incident light) from the subject to the photoelectric conversion apparatus 408.
The photoelectric conversion apparatus described in each of the above exemplary embodiments is applied to the photoelectric conversion apparatus 408, and a distance signal indicating a distance obtained from a light receiving signal output from the photoelectric conversion apparatus 408 is supplied to the image processing circuit 1404.
The image processing circuit 1404 performs image processing for constructing a distance image, based on the distance signal supplied from the photoelectric conversion apparatus 408. Then, a distance image (image data) obtained by the image processing is supplied to the monitor 405 and displayed thereon, or supplied to the memory 406 and stored (recorded) therein.
By applying the above-described electric component or photoelectric conversion apparatus, the distance image sensor 1401 having the above-described configuration can acquire a more accurate distance image in accordance with improvement of pixel characteristics, for example.
A photoelectric conversion system according to a ninth exemplary embodiment will be described with reference to
The endoscope 1100 includes a lens barrel 1101 having a region to be inserted into a body cavity of the patient 1132 by a predetermined length from a distal end, and a camera head 1102 connected to a base end of the lens barrel 1101. In the example illustrated in
An opening portion into which an objective lens is fitted is provided at the distal end of the lens barrel 1101. A light source apparatus is connected to the endoscope 1100, and light generated by the light source apparatus is guided to the distal end of the lens barrel 1101 by a light guide extended inside the lens barrel 1101, and emitted onto an observation target in the body cavity of the patient 1132 via the objective lens. The endoscope 1100 may be a forward-viewing endoscope, or may be a forward-oblique viewing endoscope or a side-viewing endoscope.
An optical system and a photoelectric conversion apparatus are provided inside the camera head 1102. Reflected light (observation light) from an observation target is condensed by the optical system to the photoelectric conversion apparatus. The observation light is photoelectrically converted by the photoelectric conversion apparatus, and an electric signal corresponding to the observation light, i.e., image signal corresponding to an observed image, is generated. The photoelectric conversion apparatus (imaging apparatus) described in each of the above exemplary embodiments can be used as the photoelectric conversion apparatus. The image signal is transmitted to a camera control unit (CCU) 1135 as RAW data. The image signal is transmitted to a camera control unit (CCU) 1135 as RAW data.
The CCU 1135 includes a central processing unit (CPU) or a graphics processing unit (GPU), and comprehensively controls operations of the endoscope 1100 and a display device 1136. Furthermore, the CCU 1135 receives an image signal from the camera head 1102, and performs various types of image processing, such as development processing (demosaic processing), on the image signal to display an image that is based on the image signal.
Based on the control from the CCU 1135, the display device 1136 displays an image that is based on the image signal on which image processing has been performed by the CCU 1135.
The light source apparatus includes a light source, such as a light emitting diode (LED), and supplies the endoscope 1100 with irradiation light at the time of capturing an image of an operative site.
An input apparatus 1137 is an input interface with the endoscopic operation system. A user can input various types of information and instructions to the endoscopic operation system via the input apparatus 1137.
A processing tool control apparatus 1138 controls the driving of an energy processing tool 1112 for cauterizing or cutting a tissue, or sealing a blood vessel.
The light source apparatus that supplies the endoscope 1100 with irradiation light at the time of capturing an image of an operative site can include, for example, an LED, a laser light source, or a white light source including a combination of these. In a case where a white light source includes a combination of red-green-blue (RGB) laser light sources, because output intensity and an output timing of each color (each wavelength) can be controlled highly accurately, white balance of a captured image can be adjusted in the light source apparatus. In this case, by emitting laser light from each RGB laser light source onto an observation target in a time division manner, and controlling the driving of an image sensor of the camera head 1102 in synchronization with the emission timing, an image corresponding to each of RGB can also be captured in a time division manner. According to the method, a color image can be obtained without providing a color filter in the image sensor.
The driving of the light source apparatus may be controlled in such a manner as to change the intensity of light to be output, every predetermined time. By acquiring images in a time division manner by controlling the driving of the image sensor of the camera head 1102 in synchronization with the change timing of the light intensity, and combining the images, it is possible to generate a high dynamic range image without so-called blocked up shadows and blown out highlights.
The light source apparatus may be configured to supply light in a predetermined wavelength band adapted to special light observation. In the special light observation, for example, wavelength dependency of light absorption in body tissues is utilized. Specifically, by emitting light in a narrow band as compared with irradiation light (i.e., white light) in normal observation, an image of a predetermined tissue, such as a blood vessel of a superficial portion of a mucous membrane, is captured with high contrast. Alternatively, in special light observation, fluorescent observation for obtaining an image by fluorescence generated by emitting excitation light may be performed. In the fluorescent observation, fluorescence from a body tissue can be observed by emitting excitation light onto the body tissue, or a fluorescent image can be obtained by locally injecting reagent, such as indocyanine green (ICG), into a body tissue and emitting excitation light suitable for a fluorescence wavelength of the reagent, onto the body tissue. The light source apparatus can be configured to supply narrow-band light and/or excitation light adapted to such special light observation.
A photoelectric conversion system according to a tenth exemplary embodiment will be described with reference to
The eyeglasses 1600 further include a control apparatus 1603. The control apparatus 1603 functions as a power source that supplies power to the photoelectric conversion apparatus 1602 and the above-described display device. The control apparatus 1603 controls operations of the photoelectric conversion apparatus 1602 and the display device. In the lens 1601, an optical system for condensing light to the photoelectric conversion apparatus 1602 is formed.
From a captured image of an eyeball obtained by image capturing using infrared light, a line of sight of a user with respect to a displayed image is detected. An arbitrary known method can be applied to line-of-sight detection that uses a captured image of an eyeball. As an example, a line-of-sight detection method that is based on a Purkinje image obtained by reflection of irradiation light on a cornea can be used.
More specifically, line-of-sight detection processing that is based on the pupil center corneal reflection is performed. By calculating an eye vector representing the direction (rotational angle) of an eyeball, based on an image of a pupil and a Purkinje image that are included in a captured image of the eyeball, using the pupil center corneal reflection, a line of sight of a user is detected.
The display device of the present exemplary embodiment includes the photoelectric conversion apparatus including a light receiving element, and a displayed image of the display device may be controlled based on line-of-sight information on the user from the photoelectric conversion apparatus.
Specifically, in the display device, a first eyeshot region viewed by the user, and a second eyeshot region other than the first eyeshot region are determined based on the line-of-sight information. The first eyeshot region and the second eyeshot region may be determined by a control apparatus of the display device, or the first eyeshot region and the second eyeshot region determined by an external control apparatus may be received. In a display region of the display device, a display resolution of the first eyeshot region may be controlled to be higher than a display resolution of the second eyeshot region. In other word, a resolution of the second eyeshot region may be lower than a resolution of the first eyeshot region.
The display region includes a first display region and a second display region different from the first display region. Based on the line-of-sight information, a region with high priority may be determined from the first display region and the second display region. The first display region and the second display region may be determined by a control apparatus of the display device, or the first display region and the second display region determined by an external control apparatus may be received. A resolution of a region with high priority may be controlled to be higher than a resolution of a region other than the region with high priority. In other words, a resolution of a region with relatively low priority may be set to a low resolution.
Artificial intelligence (AI) may be used for determining the first eyeshot region and the region with high priority. The AI may be a model configured to estimate an angle of a line and a distance to a target existing in the line of sight, from an image of an eyeball, using teaching data including an image of the eyeball, and a direction in which the eyeball in the image actually gazes. An AI program may be included in the display device, may be included in the photoelectric conversion apparatus, or may be included in an external apparatus. In a case where an external apparatus includes an AI program, the AI program is transmitted to the display device via communication.
In a case where display control is performed based on line-of-sight detection, the present exemplary embodiment can be desirably applied to smart glasses further including a photoelectric conversion apparatus that captures an image of the outside. The smart glasses can display external information obtained by image capturing, in real time.
The exemplary embodiments described above can be appropriately changed without departing from the technical idea. The disclosure in this specification is not limited to matters described in this specification, and includes all matters that can be identified from this specification and the drawings accompanying this specification. The disclosure in this specification includes a complementary set of individual concepts described in this specification. More specifically, if “A is larger than B” is described in this specification, even if the description “B is not larger than A” is omitted, this specification is assumed to disclose that “B is not larger than A”. This is because, in a case where “A is larger than B” is described, the case is premised based on a case where “B is not larger than A”.
While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of priority from Japanese Patent Applications No. 2023-202720, filed Nov. 30, 2023, and No. 2024-159060, filed Sep. 13, 2024, which are hereby incorporated by reference herein in their entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2023-202720 | Nov 2023 | JP | national |
| 2024-159060 | Sep 2024 | JP | national |
