Patent Application
20230410306
The present invention is generally directed to the field of ophthalmic imaging systems. More specifically, it is directed to techniques for facilitating user operation of an ophthalmic imaging system.
There are various type of ophthalmic examination systems, including ophthalmoscopes (or fundus cameras), Optical Coherence Tomography (OCT), and other ophthalmic imaging systems. One example of ophthalmic imaging is slit-Scanning or Broad-Line fundus imaging (see for example, U.S. Pat. Nos. 4,170,398, 4,732,466, PCT Publication No. 2012059236, US Patent Application No. 2014/0232987, and US Patent Publication No. 2015/0131050, the contents of all of which are hereby incorporated by reference), which is a technique for achieving high resolution in vivo imaging of the human retina. By illuminating a strip of the retina in a scanning fashion, the illumination stays out of the viewing path, which enables a clearer view of much more of the retina than the annular ring illumination used in traditional fundus cameras.
To obtain a good image, it is desirable for the illumination to pass unabated through the pupil and reach the fundus of an eye. This requires careful alignment of the eye with the ophthalmic imager (or other ophthalmic examination system). Various technical means have been developed to help determine the position of a patient's eye relative to the ophthalmic imaging device. However, conveying such three dimensional positioning information to a system operator (e.g., human operator or ophthalmic photographer) in an intuitive matter so that he/she may make quick use of the positioning information without requiring complex mental calculations, or mental translations from one reference plane to another, has been difficult. Generally, such systems provide a live video feed, or image/video stream, of a patient's eye on a display (viewable by the system operator) and add graphical positioning cues overlaid on the live video feed that may be interpreted by the system operator to determine positioning information of the ophthalmic imaging system relative to the patient's eye. The system operator needs to monitor the patient's eye while interpreting the system's positioning cues to determine how to adjust the position of the system and when proper alignment is achieved. Consequently, much training is generally needed to achieve a high level of competency in using such systems.
It is an object of the present invention to provide tools to facilitate the alignment of an eye with an ophthalmic examination system.
It is a further object of the present invention to provide a graphical user interface that conveys intuitive alignment information to a system operator to permit alignment of an ophthalmic imaging device to a patient's eye with reduced training.
The above objects are met in a method/system for aiding a system operator to align an ophthalmic imaging device for imaging/scanning a portion of a patient's eye, such as the fundus. A preferred embodiment eliminates the need for a live feed of a patients eye. Instead various graphics, or graphic combinations, are used to convey three-dimensional (3D) information. For example, a first distinctive graphic (e.g., a dotted circle) whose size is indicative of a predefined target axial position for a pupil of an eye, is displayed/provided. A second distinctive graphic (e.g., a solid, round graphic, such as a sphere or circle) may be used to represent a patient's pupil (e.g., a pupil graphic). The displayed size of the second graphic relative to the displayed size of the first graphic is indicative of a currently observed/determined axial position of the pupil relative to the predefined target axial position (or the ophthalmic device).
Additional graphics may then be used to convey full x, y, z axis positioning information of the patient's pupil relative to the ophthalmic imaging system. In one embodiment, a cross-hair graphic may be used to convey translational (e.g., x-y axis) positioning information of the ophthalmic device relative to the pupil graphic (e.g., relative to the patient's pupil), or vice versa. z-axis information may be conveyed by illustrating the first graphic (e.g., a z-position, (round) target/reference graphic) in combination with the second graphic (e.g. the pupil graphic whose size varies with axial distance from the target graphic). For example, the size of the pupil graphic may change relative to the (e.g., fixed) size of the z-position round (target/reference) graphic, or vice versa, to represent depth information. For example, if the patient's pupil is closer to the ophthalmic device than desired (e.g., than the target z-position), the pupil graphic may be made larger than the z-position graphic, and if the patient's pupil is farther from the ophthalmic device than desired, the pupil graphic may be displayed smaller than the z-position graphic. The pupil graphic may be made to match (e.g., have a displayed size that matches) the size of the z-position graphic when the patient's pupil is within a predefined range of axial positions suitable for proper imagining. This approach of conveying depth information by use of size corresponds better to (e.g., maps much more closely to) human perception of distance/depth than other 2-dimensional guidance options.
Furthermore, since a pupil graphic is used, rather than a live feed, the size (and optionally the position) of the pupil graphic may be kept constant even while a patient's pupil momentarily moves, such as due to tremors. This eliminates unnecessary adjustments by a system operator.
Additionally, the color of the pupil graphic and/or the color of the z-position graphic may change (e.g., to match and/or to blink and/or to predefined colors or graphic patterns) to indicate when an axial position for proper alignment is achieved.
Other objects and attainments together with a fuller understanding of the invention will become apparent and appreciated by referring to the following description and claims taken in conjunction with the accompanying drawings.
Several publications may be cited or referred to herein to facilitate the understanding of the present invention. All publications cited or referred to herein, are hereby incorporated herein in their entirety by reference.
The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Any embodiment feature mentioned in one claim category, e.g. system, can be claimed in another claim category, e.g. method, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However, any subject matter resulting from a deliberate reference back to any previous claims can be claimed as well, so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims.
Priority application U.S. Ser. No. 63/120,525 contain at least one color drawing and is herein incorporated by reference.
In the drawings wherein like reference symbols/characters refer to like parts:
Ophthalmic photographers need to position the pupil very precisely relative to an ophthalmic imaging device when attempting to capture retinal images, or other ophthalmic images. A discussion of ophthalmic imaging devices, such as a fundus cameras, suitable for use with the present invention is provided below. Proper alignment relies on horizontal (x-axis), vertical (y-axis) and depth (z-axis) adjustments of the acquisition device relative to the patient. Given the difficulty in managing multiple planes of adjustment simultaneously, a user interface (e.g., a graphical user interface, GUI) that provides guidance as to the direction and magnitude of needed adjustments for proper alignment is generally provided to increase the ease-of-use of acquisition devices by ophthalmic photographers.
In contrast to the approach of
By way of example,
The ophthalmic imaging device will generally include a means/mechanism for determining the position of a patient's eye relative to the ophthalmic imaging device. The specific method/mechanism for determining this relative position is not critical to the present invention, but one exemplary method is provided here.
At least two iris cameras are needed to cover all three degrees of freedom (x,y,z) at any given time. Offset information is extracted by detecting the patient's pupil and locating the center of the pupil and then comparing it to stored and calibrated reference values of pupil centers. For example, iris camera Cam3 (located at the 270° position) maps the x coordinate of the patient's pupil center to the column coordinate of the iris camera image (which is comprised of rows and columns of pixels), while the z-coordinate is mapped to the row coordinate of the camera image. As the patient or the instrument moves laterally (e.g., right to left or left to right), the image moves laterally (e.g., right to left), and as the instrument is moved closer or farther away from the patient, the image of the pupil will move up or down in
Although translational movement (e.g., along the x-axis and y axis) of the eye relative to cross-hairs 402 is intuitive to an operator, the use of vertical movement of the image in combination with a horizontal overlay guide 400 might not be optimal for conveying an intuitive understanding of axial motion (e.g., along the z-axis) and axial positioning information.
Optionally in an alternate embodiment, the portion of the sphere 13 that is within the predefined target axial position 11 (e.g., within the plane of dotted circle 11), may be displayed brighter (as indicated by bright spot 14) than the portion of the sphere 13 that is not within the predefined target axial position, as indicated by less bright region of sphere 13. For example, the portion of sphere 13 that is darker than bright spot 14 may be farther away from the axial position defined by dotted circle 11 and the ophthalmic device. Further alternatively, the color distribution of sphere 13 may be such that the portion of sphere 13 that is closer to the ophthalmic device is made lighter or brighter or of a different color than that the portion of sphere 13 that is farther from the ophthalmic device. In general, different colors could be used instead of different brightness levels.
A third graphic, e.g. cross-hairs 15 (or Cartesian plane, or other graphic indicative of a plane normal to the axial direction/axis), provides/indicates a predefined reference position on the plane. In the present example, the center region of the cross-hairs may indicate the predefined reference position on the plane, and a translational position of the sphere 13 on the display 10 may be indicative of a current translational position of the pupil on the plane. In the present example, cross-hairs 15 has two horizontal lines 15a and whose separation may indicate a desired positioning range for optimal imaging on the xy plan. Alternatively, double horizontal dash lines 15a/15b may be an alternate axial information indicator, as explained above in reference to the semi-transparent band 400 of
In the present embodiment, the center of the sphere 13 is always within the dotted circle 11, and preferably maintained aligned with the center of the dotted circle 13. In this manner, both dotted circle 11 and sphere 13 are move in tandem about the display/screen in response to translational motion (e.g. changes in the x and y axes) of the pupil, while the size, intensity, and/or color change of sphere 13 indicates its axial position relative to the dotted circle 11.
Thus, based on the positioning information provided by the GUI in
With reference to
The progress in the z-axis indicator (pupil sphere 13) across
While a live view of the eye may communicate distance through observed size changes, the scale of visual, video size is too small to provide useful feedback to the user. Also, small eye movements (e.g., tremors) can be exaggerated with a close-up live view, leading to unstable feedback to the system user. The illustrated sphere 13, by contrast, is completely configurable by the system to provide scaled size and stable positioning feedback that is readily perceived and responded to by the user. Thus, since the system determines the size of the displayed sphere 13, it can filter out (or mask) momentary movements, or tremors, from the patient. For example, a timer may be provided such that if a change in distance of the pupil relative to the ophthalmic imaging device is of a duration lower than a predefined threshold, the displayed size of sphere 13 remains unchanged.
The present system may further be coupled with, and augment, an automatic imaging device alignment system. For example, the present system may provide fine tuning to the automatic image device alignment system. Or if the automatic image device alignment system requires that the device be within a specific position range of the patient's eye for proper operation, the present system may quickly bring the imaging device to within this target position range needed by the automatic image device alignment system for proper operation.
An exemplary automatic image device alignment system is herein presented.
Pupil detection is integral to alignment guidance during fundus image acquisition and automated fundus image capture. A deep learning algorithm for real-time tracking of pupils at greater than 25 frames per second (fps) is herein presented. In the present example, 13,674 eye images that provide off-axis views of patients' pupils were collected using prototype software on a CLARUS™ 500 (ZEISS, Dublin, CA). This dataset was divided into 3 parts:
To reduce operation time, a constringed single-shot detector (SSD) inspired by single-shot multi-box detection technique (as is known in the art) was used, comprised of three feature extraction and three candidate box prediction layers. The confidence score was used to predict at most one box out of 944 candidate output boxes.
The final model/algorithm achieved accuracies of 95.1% and 98.3% on mydriatic and non-mydriatic images of Dataset3. By comparison, the model/algorithm developed in step I (without hard-negative training) achieved accuracies of 91.7% and 95.6%.
Thus, the present model/algorithm was shown to provide robust, real-time pupil detection for alignment guidance, with accuracies greater than 95% in detecting the correct pupil location within 400 μm of manual annotations while also operating at a frame rate greater than the camera acquisition. The present GUI system may then be used to verify the present model's results and achieve greater levels of alignment.
Hereinafter is provided a description of various hardware and architectures suitable for the present invention.
Fundus Imaging System
Two categories of imaging systems used to image the fundus are flood illumination imaging systems (or flood illumination imagers) and scan illumination imaging systems (or scan imagers). Flood illumination imagers flood with light an entire field of view (FOV) of interest of a specimen at the same time, such as by use of a flash lamp, and capture a full-frame image of the specimen (e.g., the fundus) with a full-frame camera (e.g., a camera having a two-dimensional (2D) photo sensor array of sufficient size to capture the desired FOV, as a whole). For example, a flood illumination fundus imager would flood the fundus of an eye with light, and capture a full-frame image of the fundus in a single image capture sequence of the camera. A scan imager provides a scan beam that is scanned across a subject, e.g., an eye, and the scan beam is imaged at different scan positions as it is scanned across the subject creating a series of image-segments that may be reconstructed, e.g., montaged, to create a composite image of the desired FOV. The scan beam could be a point, a line, or a two-dimensional area such a slit or broad line. Examples of fundus imagers are provided in U.S. Pat. Nos. 8,967,806 and 8,998,411.
From the scanner LnScn, the illumination beam passes through one or more optics, in this case a scanning lens SL and an ophthalmic or ocular lens OL, that allow for the pupil of the eye E to be imaged to an image pupil of the system. Generally, the scan lens SL receives a scanning illumination beam from the scanner LnScn at any of multiple scan angles (incident angles), and produces scanning line beam SB with a substantially flat surface focal plane (e.g., a collimated light path). Ophthalmic lens OL may then focus the scanning line beam SB onto an object to be imaged. In the present example, ophthalmic lens OL focuses the scanning line beam SB onto the fundus F (or retina) of eye E to image the fundus. In this manner, scanning line beam SB creates a traversing scan line that travels across the fundus F. One possible configuration for these optics is a Kepler type telescope wherein the distance between the two lenses is selected to create an approximately telecentric intermediate fundus image (4-f configuration). The ophthalmic lens OL could be a single lens, an achromatic lens, or an arrangement of different lenses. All lenses could be refractive, diffractive, reflective or hybrid as known to one skilled in the art. The focal length(s) of the ophthalmic lens OL, scan lens SL and the size and/or form of the pupil splitting mirror SM and scanner LnScn could be different depending on the desired field of view (FOV), and so an arrangement in which multiple components can be switched in and out of the beam path, for example by using a flip in optic, a motorized wheel, or a detachable optical element, depending on the field of view can be envisioned. Since the field of view change results in a different beam size on the pupil, the pupil splitting can also be changed in conjunction with the change to the FOV. For example, a 45° to 60° field of view is a typical, or standard, FOV for fundus cameras. Higher fields of view, e.g., a widefield FOV, of 60°-120°, or more, may also be feasible. A widefield FOV may be desired for a combination of the Broad-Line Fundus Imager (BLFI) with another imaging modalities such as optical coherence tomography (OCT). The upper limit for the field of view may be determined by the accessible working distance in combination with the physiological conditions around the human eye. Because a typical human retina has a FOV of 140° horizontal and 80°-100° vertical, it may be desirable to have an asymmetrical field of view for the highest possible FOV on the system.
The scanning line beam SB passes through the pupil Ppl of the eye E and is directed towards the retinal, or fundus, surface F. The scanner LnScn1 adjusts the location of the light on the retina, or fundus, F such that a range of transverse locations on the eye E are illuminated. Reflected or scattered light (or emitted light in the case of fluorescence imaging) is directed back along as similar path as the illumination to define a collection beam CB on a detection path to camera Cmr.
In the “scan-descan” configuration of the present, exemplary slit scanning ophthalmic system SLO-1, light returning from the eye E is “descanned” by scanner LnScn on its way to pupil splitting mirror SM. That is, scanner LnScn scans the illumination beam from pupil splitting mirror SM to define the scanning illumination beam SB across eye E, but since scanner LnScn also receives returning light from eye E at the same scan position, scanner LnScn has the effect of descanning the returning light (e.g., cancelling the scanning action) to define a non-scanning (e.g., steady or stationary) collection beam from scanner LnScn to pupil splitting mirror SM, which folds the collection beam toward camera Cmr. At the pupil splitting mirror SM, the reflected light (or emitted light in the case of fluorescence imaging) is separated from the illumination light onto the detection path directed towards camera Cmr, which may be a digital camera having a photo sensor to capture an image. An imaging (e.g., objective) lens ImgT may be positioned in the detection path to image the fundus to the camera Cmr. As is the case for objective lens ObjL, imaging lens ImgL may be any type of lens known in the art (e.g., refractive, diffractive, reflective or hybrid lens). Additional operational details, in particular, ways to reduce artifacts in images, are described in PCT Publication No. WO2016/124644, the contents of which are herein incorporated in their entirety by reference. The camera Cmr captures the received image, e.g., it creates an image file, which can be further processed by one or more (electronic) processors or computing devices (e.g., the computer system of
In the present example, the camera Cmr is connected to a processor (e.g., processing module) Proc and a display (e.g., displaying module, computer screen, electronic screen, etc.) Dspl, both of which can be part of the image system itself, or may be part of separate, dedicated processing and/or displaying unit(s), such as a computer system wherein data is passed from the camera Cmr to the computer system over a cable or computer network including wireless networks. The display and processor can be an all in one unit. The display can be a traditional electronic display/screen or of the touch screen type and can include a user interface for displaying information to and receiving information from an instrument operator, or user. The user can interact with the display using any type of user input device as known in the art including, but not limited to, mouse, knobs, buttons, pointer, and touch screen.
It may be desirable for a patient's gaze to remain fixed while imaging is carried out. One way to achieve this is to provide a fixation target that the patient can be directed to stare at. Fixation targets can be internal or external to the instrument depending on what area of the eye is to be imaged. One embodiment of an internal fixation target is shown in
Slit-scanning ophthalmoscope systems are capable of operating in different imaging modes depending on the light source and wavelength selective filtering elements employed. True color reflectance imaging (imaging similar to that observed by the clinician when examining the eye using a hand-held or slit lamp ophthalmoscope) can be achieved when imaging the eye with a sequence of colored LEDs (red, blue, and green). Images of each color can be built up in steps with each LED turned on at each scanning position or each color image can be taken in its entirety separately. The three, color images can be combined to display the true color image, or they can be displayed individually to highlight different features of the retina. The red channel best highlights the choroid, the green channel highlights the retina, and the blue channel highlights the anterior retinal layers. Additionally, light at specific frequencies (e.g., individual colored LEDs or lasers) can be used to excite different fluorophores in the eye (e.g., autofluorescence) and the resulting fluorescence can be detected by filtering out the excitation wavelength.
The fundus imaging system can also provide an infrared reflectance image, such as by using an infrared laser (or other infrared light source). The infrared (IR) mode is advantageous in that the eye is not sensitive to the IR wavelengths. This may permit a user to continuously take images without disturbing the eye (e.g., in a preview/alignment mode) to aid the user during alignment of the instrument. Also, the IR wavelengths have increased penetration through tissue and may provide improved visualization of choroidal structures. In addition, fluorescein angiography (FA) and indocyanine green (ICG) angiography imaging can be accomplished by collecting images after a fluorescent dye has been injected into the subject's bloodstream. For example, in FA (and/or ICG) a series of time-lapse images may be captured after injecting a light-reactive dye (e.g., fluorescent dye) into a subject's bloodstream. It is noted that care must be taken since the fluorescent dye may lead to a life-threatening allergic reaction in a portion of the population. High contrast, greyscale images are captured using specific light frequencies selected to excite the dye. As the dye flows through the eye, various portions of the eye are made to glow brightly (e.g., fluoresce), making it possible to discern the progress of the dye, and hence the blood flow, through the eye.
Computing Device/System
In some embodiments, the computer system may include a processor Cpnt1, memory Cpnt2, storage Cpnt3, an input/output (I/O) interface Cpnt4, a communication interface Cpnt5, and a bus Cpnt6. The computer system may optionally also include a display Cpnt7, such as a computer monitor or screen.
Processor Cpnt1 includes hardware for executing instructions, such as those making up a computer program. For example, processor Cpnt1 may be a central processing unit (CPU) or a general-purpose computing on graphics processing unit (GPGPU). Processor Cpnt1 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory Cpnt2, or storage Cpnt3, decode and execute the instructions, and write one or more results to an internal register, an internal cache, memory Cpnt2, or storage Cpnt3. In particular embodiments, processor Cpnt1 may include one or more internal caches for data, instructions, or addresses. Processor Cpnt1 may include one or more instruction caches, one or more data caches, such as to hold data tables. Instructions in the instruction caches may be copies of instructions in memory Cpnt2 or storage Cpnt3, and the instruction caches may speed up retrieval of those instructions by processor Cpnt1. Processor Cpnt1 may include any suitable number of internal registers, and may include one or more arithmetic logic units (ALUs). Processor Cpnt1 may be a multi-core processor; or include one or more processors Cpnt1. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.
Memory Cpnt2 may include main memory for storing instructions for processor Cpnt1 to execute or to hold interim data during processing. For example, the computer system may load instructions or data (e.g., data tables) from storage Cpnt3 or from another source (such as another computer system) to memory Cpnt2. Processor Cpnt1 may load the instructions and data from memory Cpnt2 to one or more internal register or internal cache. To execute the instructions, processor Cpnt1 may retrieve and decode the instructions from the internal register or internal cache. During or after execution of the instructions, processor Cpnt1 may write one or more results (which may be intermediate or final results) to the internal register, internal cache, memory Cpnt2 or storage Cpnt3. Bus Cpnt6 may include one or more memory buses (which may each include an address bus and a data bus) and may couple processor Cpnt1 to memory Cpnt2 and/or storage Cpnt3. Optionally, one or more memory management unit (MMU) facilitate data transfers between processor Cpnt1 and memory Cpnt2. Memory Cpnt2 (which may be fast, volatile memory) may include random access memory (RAM), such as dynamic RAM (DRAM) or static RAM (SRAM). Storage Cpnt3 may include long-term or mass storage for data or instructions. Storage Cpnt3 may be internal or external to the computer system, and include one or more of a disk drive (e.g., hard-disk drive, HDD, or solid-state drive, SSD), flash memory, ROM, EPROM, optical disc, magneto-optical disc, magnetic tape, Universal Serial Bus (USB)-accessible drive, or other type of non-volatile memory.
I/O interface Cpnt4 may be software, hardware, or a combination of both, and include one or more interfaces (e.g., serial or parallel communication ports) for communication with I/O devices, which may enable communication with a person (e.g., user). For example, I/O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device, or a combination of two or more of these.
Communication interface Cpnt5 may provide network interfaces for communication with other systems or networks. Communication interface Cpnt5 may include a Bluetooth interface or other type of packet-based communication. For example, communication interface Cpnt5 may include a network interface controller (NIC) and/or a wireless NIC or a wireless adapter for communicating with a wireless network. Communication interface Cpnt5 may provide communication with a WI-FI network, an ad hoc network, a personal area network (PAN), a wireless PAN (e.g., a Bluetooth WPAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), the Internet, or a combination of two or more of these.
Bus Cpnt6 may provide a communication link between the above-mentioned components of the computing system. For example, bus Cpnt6 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand bus, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or other suitable bus or a combination of two or more of these.
Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.
Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.
While the invention has been described in conjunction with several specific embodiments, it is evident to those skilled in the art that many further alternatives, modifications, and variations will be apparent in light of the foregoing description. Thus, the invention described herein is intended to embrace all such alternatives, modifications, applications and variations as may fall within the spirit and scope of the appended claims.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/083764 | 12/1/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63120525 | Dec 2020 | US |
